123456789_123456789_123456789_123456789_123456789_

Class: RDoc::Parser::Ruby

Relationships & Source Files
Super Chains via Extension / Inclusion / Inheritance
Class Chain:
self, ::RDoc::Parser
Instance Chain:
self, RubyTools, ::RDoc::TokenStream, ::RDoc::Parser
Inherits: RDoc::Parser
Defined in: lib/rdoc/parser/ruby.rb

Overview

Extracts code elements from a source file returning a ::RDoc::TopLevel object containing the constituent file elements.

This file is based on rtags

RubyParser understands how to document:

  • classes

  • modules

  • methods

  • constants

  • aliases

  • private, public, protected

  • private_class_function, public_class_function

  • module_function

  • attr, attr_reader, attr_writer, attr_accessor

  • extra accessors given on the command line

  • metaprogrammed methods

  • require

  • include

Method Arguments

The parser extracts the arguments from the method definition. You can override this with a custom argument definition using the :call-seq: directive:

##
# This method can be called with a range or an offset and length
#
# :call-seq:
#   my_method(Range)
#   my_method(offset, length)

def my_method(*args)
end

The parser extracts yield expressions from method bodies to gather the yielded argument names. If your method manually calls a block instead of yielding or you want to override the discovered argument names use the :yields: directive:

##
# My method is awesome

def my_method(&block) # :yields: happy, times
  block.call 1, 2
end

Metaprogrammed Methods

To pick up a metaprogrammed method, the parser looks for a comment starting with '##' before an identifier:

##
# This is a meta-programmed method!

add_my_method :meta_method, :arg1, :arg2

The parser looks at the token after the identifier to determine the name, in this example, :meta_method. If a name cannot be found, a warning is printed and 'unknown is used.

You can force the name of a method using the :method: directive:

##
# :method: some_method!

By default, meta-methods are instance methods. To indicate that a method is a singleton method instead use the :singleton-method: directive:

##
# :singleton-method:

You can also use the :singleton-method: directive with a name:

##
# :singleton-method: some_method!

You can define arguments for metaprogrammed methods via either the :call-seq:, :arg: or :args: directives.

Additionally you can mark a method as an attribute by using :attr:, :attr_reader:, :attr_writer: or :attr_accessor:. Just like for :method:, the name is optional.

##
# :attr_reader: my_attr_name

Hidden methods and attributes

You can provide documentation for methods that don't appear using the :method:, :singleton-method: and :attr: directives:

##
# :attr_writer: ghost_writer
# There is an attribute here, but you can't see it!

##
# :method: ghost_method
# There is a method here, but you can't see it!

##
# this is a comment for a regular method

def regular_method() end

Note that by default, the :method: directive will be ignored if there is a standard rdocable item following it.

Constant Summary

Class Attribute Summary

::RDoc::Parser - Inherited

.parsers

An Array of arrays that maps file extension (or name) regular expressions to parser classes that will parse matching filenames.

Class Method Summary

::RDoc::Parser - Inherited

.alias_extension

::RDoc::Alias an extension to another extension.
.binary?

Determines if the file is a “binary” file which basically means it has content that an ::RDoc::RDoc parser shouldn't try to consume.
.can_parse

Return a parser that can handle a particular extension.
.can_parse_by_name

Returns a parser that can handle the extension for #file_name.
.check_modeline

Returns the file type from the modeline in #file_name
.for

Finds and instantiates the correct parser for the given #file_name and content.
.new

Creates a new ::RDoc::Parser storing top_level, #file_name, content, options and stats in instance variables.
.parse_files_matching

Record which file types this parser can understand.
.process_directive

Processes common directives for CodeObjects for the C and Ruby parsers.
.remove_modeline

Removes an emacs-style modeline from the first line of the document.
.use_markup

If there is a markup: parser_name comment at the front of the file, use it to determine the parser.
.zip?

Checks if file is a zip file in disguise.

Instance Attribute Summary

::RDoc::Parser - Inherited

#file_name

The name of the file being parsed.

Instance Method Summary

RubyTools - Included

#add_token_listener

Adds a token listener obj, but you should probably use token_listener.
#get_tk

Fetches the next token from the scanner.
#get_tk_until

Reads and returns all tokens up to one of tokens.
#get_tkread

Retrieves a String representation of the read tokens.
#peek_read

Peek equivalent for get_tkread.
#peek_tk

Peek at the next token, but don't remove it from the stream.
#remove_token_listener

Removes the token listener obj
#reset

Resets the tools.
#skip_tkspace

Skips whitespace tokens including newlines if skip_nl is true.
#token_listener

Has obj listen to tokens.
#unget_tk

Returns tk to the scanner.

::RDoc::TokenStream - Included

#add_token

::RDoc::Alias for TokenStream#add_tokens.
#add_tokens

Adds tokens to the collected tokens.
#collect_tokens

Starts collecting tokens.
#pop_token

Remove the last token from the collected tokens.
#start_collecting_tokens

::RDoc::Alias for TokenStream#collect_tokens.
#token_stream

Current token stream.
#tokens_to_s

Returns a string representation of the token stream.

Constructor Details

.new(top_level, file_name, content, options, stats) ⇒ Ruby

Creates a new Ruby parser.

[ GitHub ] 

  


# File 'lib/rdoc/parser/ruby.rb', line 162



def initialize(top_level, file_name, content, options, stats)
  super

  @size = 0
  @token_listeners = nil
  @scanner = RDoc::RubyLex.new content, @options
  @scanner.exception_on_syntax_error = false
  @prev_seek = nil
  @markup = @options.markup
  @track_visibility = :nodoc != @options.visibility

  @encoding = nil
  @encoding = @options.encoding if Object.const_defined? :Encoding

  reset
end


  







Instance Method Details



  

    #collect_first_comment  
  



  

    

Look for the first comment in a file that isn't a shebang line.


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 225



def collect_first_comment
  skip_tkspace
  comment = ''
  comment.force_encoding @encoding if @encoding
  first_line = true
  first_comment_tk_class = nil

  tk = get_tk

  while TkCOMMENT === tk
    if first_line and tk.text =~ /\A#!/ then
      skip_tkspace
      tk = get_tk
    elsif first_line and tk.text =~ /\A#\s*-\*-/ then
      first_line = false
      skip_tkspace
      tk = get_tk
    else
      break if first_comment_tk_class and not first_comment_tk_class === tk
      first_comment_tk_class = tk.class

      first_line = false
      comment << tk.text << "\n"
      tk = get_tk

      if TkNL === tk then
        skip_tkspace false
        tk = get_tk
      end
    end
  end

  unget_tk tk

  new_comment comment
end


  








  

    #error(msg)  
  



  

    

Aborts with msg


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 300



def error(msg)
  msg = make_message msg

  abort msg
end


  








  

    #get_bool  
  



  

    

Looks for a true or false token.  Returns false if TkFALSE or TkNIL are found.


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 310



def get_bool
  skip_tkspace
  tk = get_tk
  case tk
  when TkTRUE
    true
  when TkFALSE, TkNIL
    false
  else
    unget_tk tk
    true
  end
end


  








  

    #get_class_or_module(container, ignore_constants = false)  
  



  

    
Look for the name of a class of module (optionally with a leading 



or


with 



separated named) and return the ultimate name, the associated





container, and the given name (with the ::).


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 329



def get_class_or_module container, ignore_constants = false
  skip_tkspace
  name_t = get_tk
  given_name = ''

  # class ::A -> A is in the top level
  case name_t
  when TkCOLON2, TkCOLON3 then # bug
    name_t = get_tk
    container = @top_level
    given_name << '::'
  end

  skip_tkspace false
  given_name << name_t.name

  while TkCOLON2 === peek_tk do
    prev_container = container
    container = container.find_module_named name_t.name
    container ||=
      if ignore_constants then
        RDoc::Context.new
      else
        c = prev_container.add_module RDoc::NormalModule, name_t.name
        c.ignore unless prev_container.document_children
        @top_level.add_to_classes_or_modules c
        c
      end

    record_location container

    get_tk
    skip_tkspace false
    name_t = get_tk
    given_name << '::' << name_t.name
  end

  skip_tkspace false

  return [container, name_t, given_name]
end


  








  

    #get_class_specification  
  



  

    

Return a superclass, which can be either a constant of an expression


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 374



def get_class_specification
  case peek_tk
  when TkSELF then return 'self'
  when TkGVAR then return ''
  end

  res = get_constant

  skip_tkspace false

  get_tkread # empty out read buffer

  tk = get_tk

  case tk
  when TkNL, TkCOMMENT, TkSEMICOLON then
    unget_tk(tk)
    return res
  end

  res += parse_call_parameters(tk)
  res
end


  








  

    #get_constant  
  



  

    

Parse a constant, which might be qualified by one or more class or module names


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 402



def get_constant
  res = ""
  skip_tkspace false
  tk = get_tk

  while TkCOLON2 === tk or TkCOLON3 === tk or TkCONSTANT === tk do
    res += tk.name
    tk = get_tk
  end

  unget_tk(tk)
  res
end


  








  

    #get_constant_with_optional_parens  
  



  

    

Get a constant that may be surrounded by parens


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 419



def get_constant_with_optional_parens
  skip_tkspace false

  nest = 0

  while TkLPAREN === (tk = peek_tk) or TkfLPAREN === tk do
    get_tk
    skip_tkspace
    nest += 1
  end

  name = get_constant

  while nest > 0
    skip_tkspace
    tk = get_tk
    nest -= 1 if TkRPAREN === tk
  end

  name
end


  








  

    #get_symbol_or_name  
  



  

    

Extracts a name or symbol from the token stream.


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 506



def get_symbol_or_name
  tk = get_tk
  case tk
  when TkSYMBOL then
    text = tk.text.sub(/^:/, '')

    if TkASSIGN === peek_tk then
      get_tk
      text << '='
    end

    text
  when TkId, TkOp then
    tk.name
  when TkAMPER,
       TkDSTRING,
       TkSTAR,
       TkSTRING then
    tk.text
  else
    raise RDoc::Error, "Name or symbol expected (got #{tk})"
  end
end


  








  

    #look_for_directives_in(context, comment)  
  



  

    

Look for directives in a normal comment block:



# :stopdoc:
# Don't display comment from this point forward



This routine modifies its comment parameter.


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 552



def look_for_directives_in context, comment
  @preprocess.handle comment, context do |directive, param|
    case directive
    when 'method', 'singleton-method',
         'attr', 'attr_accessor', 'attr_reader', 'attr_writer' then
      false # handled elsewhere
    when 'section' then
      context.set_current_section param, comment.dup
      comment.text = ''
      break
    end
  end

  remove_private_comments comment
end


  








  

    #make_message(message)  
  



  

    

Adds useful info about the parser to message


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 571



def make_message message
  prefix = "#{@file_name}:"

  prefix << "#{@scanner.line_no}:#{@scanner.char_no}:" if @scanner

  "#{prefix} #{message}"
end


  








  

    #new_comment(comment)  
  



  

    

Creates a comment with the correct format


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 582



def new_comment comment
  c = RDoc::Comment.new comment, @top_level
  c.format = @markup
  c
end


  








  

    #parse_alias(context, single, tk, comment)  
  



  

    

Parses an alias in context with comment


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 654



def parse_alias(context, single, tk, comment)
  offset  = tk.seek
  line_no = tk.line_no

  skip_tkspace

  if TkLPAREN === peek_tk then
    get_tk
    skip_tkspace
  end

  new_name = get_symbol_or_name

  @scanner.lex_state = :EXPR_FNAME

  skip_tkspace
  if TkCOMMA === peek_tk then
    get_tk
    skip_tkspace
  end

  begin
    old_name = get_symbol_or_name
  rescue RDoc::Error
    return
  end

  al = RDoc::Alias.new(get_tkread, old_name, new_name, comment,
                       single == SINGLE)
  record_location al
  al.offset = offset
  al.line   = line_no

  read_documentation_modifiers al, RDoc::ATTR_MODIFIERS
  context.add_alias al
  @stats.add_alias al

  al
end


  








  

    #parse_attr(context, single, tk, comment)  
  



  

    

Creates an ::RDoc::Attr for the name following tk, setting the comment to comment.


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 592



def parse_attr(context, single, tk, comment)
  offset  = tk.seek
  line_no = tk.line_no

  args = parse_symbol_arg 1
  if args.size > 0 then
    name = args[0]
    rw = "R"
    skip_tkspace false
    tk = get_tk

    if TkCOMMA === tk then
      rw = "RW" if get_bool
    else
      unget_tk tk
    end

    att = create_attr context, single, name, rw, comment
    att.offset = offset
    att.line   = line_no

    read_documentation_modifiers att, RDoc::ATTR_MODIFIERS
  else
    warn "'attr' ignored - looks like a variable"
  end
end


  








  

    #parse_attr_accessor(context, single, tk, comment)  
  



  

    

Creates an ::RDoc::Attr for each attribute listed after tk, setting the comment for each to comment.


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 623



def parse_attr_accessor(context, single, tk, comment)
  offset  = tk.seek
  line_no = tk.line_no

  args = parse_symbol_arg
  rw = "?"

  tmp = RDoc::CodeObject.new
  read_documentation_modifiers tmp, RDoc::ATTR_MODIFIERS
  # TODO In most other places we let the context keep track of document_self
  # and add found items appropriately but here we do not.  I'm not sure why.
  return if @track_visibility and not tmp.document_self

  case tk.name
  when "attr_reader"   then rw = "R"
  when "attr_writer"   then rw = "W"
  when "attr_accessor" then rw = "RW"
  else
    rw = '?'
  end

  for name in args
    att = create_attr context, single, name, rw, comment
    att.offset = offset
    att.line   = line_no
  end
end


  








  

    #parse_call_parameters(tk)  
  



  

    

Extracts call parameters from the token stream.


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 697



def parse_call_parameters(tk)
  end_token = case tk
              when TkLPAREN, TkfLPAREN
                TkRPAREN
              when TkRPAREN
                return ""
              else
                TkNL
              end
  nest = 0

  loop do
    case tk
    when TkSEMICOLON
      break
    when TkLPAREN, TkfLPAREN
      nest += 1
    when end_token
      if end_token == TkRPAREN
        nest -= 1
        break if @scanner.lex_state == :EXPR_END and nest <= 0
      else
        break unless @scanner.continue
      end
    when TkCOMMENT, TkASSIGN, TkOPASGN
      unget_tk(tk)
      break
    when nil then
      break
    end
    tk = get_tk
  end

  get_tkread_clean "\n", " "
end


  








  

    #parse_class(container, single, tk, comment)  
  



  

    

Parses a class in context with comment


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 736



def parse_class container, single, tk, comment
  offset  = tk.seek
  line_no = tk.line_no

  declaration_context = container
  container, name_t, given_name = get_class_or_module container

  cls =
    case name_t
    when TkCONSTANT
      parse_class_regular container, declaration_context, single,
        name_t, given_name, comment
    when TkLSHFT
      case name = get_class_specification
      when 'self', container.name
        parse_statements container, SINGLE
        return # don't update offset or line
      else
        parse_class_singleton container, name, comment
      end
    else
      warn "Expected class name or '<<'. Got #{name_t.class}: #{name_t.text.inspect}"
      return
    end

  cls.offset = offset
  cls.line   = line_no

  cls
end


  








  

    #parse_comment(container, tk, comment)  
  



  

    

Generates an RDoc::Method or ::RDoc::Attr from comment by looking for :method: or :attr: directives in comment.


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 951



def parse_comment container, tk, comment
  return parse_comment_tomdoc container, tk, comment if @markup == 'tomdoc'
  column  = tk.char_no
  offset  = tk.seek
  line_no = tk.line_no

  text = comment.text

  singleton = !!text.sub!(/(^# +:?)(singleton-)(method:)/, '\1\3')

  co =
    if text.sub!(/^# +:?method: *(\S*).*?\n/i, '') then
      parse_comment_ghost container, text, $1, column, line_no, comment
    elsif text.sub!(/# +:?(attr(_reader|_writer|_accessor)?): *(\S*).*?\n/i, '') then
      parse_comment_attr container, $1, $3, comment
    end

  if co then
    co.singleton = singleton
    co.offset    = offset
    co.line      = line_no
  end

  true
end


  








  

    #parse_comment_tomdoc(container, tk, comment)  
  



  

    

Creates an RDoc::Method on container from comment if there is a Signature section in the comment


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1033



def parse_comment_tomdoc container, tk, comment
  return unless signature = RDoc::TomDoc.signature(comment)
  offset  = tk.seek
  line_no = tk.line_no

  name, = signature.split %r%[ \(]%, 2

  meth = RDoc::GhostMethod.new get_tkread, name
  record_location meth
  meth.offset    = offset
  meth.line      = line_no

  meth.start_collecting_tokens
  indent = TkSPACE.new 0, 1, 1
  indent.set_text " " * offset

  position_comment = TkCOMMENT.new 0, line_no, 1
  position_comment.set_text "# File #{@top_level.relative_name}, line #{line_no}"
  meth.add_tokens [position_comment, NEWLINE_TOKEN, indent]

  meth.call_seq = signature

  comment.normalize

  return unless meth.name

  container.add_method meth

  meth.comment = comment

  @stats.add_method meth
end


  








  

    #parse_constant(container, tk, comment, ignore_constants = false)  
  



  

    

Parses a constant in context with comment.  If ignore_constants is true, no found constants will be added to ::RDoc::RDoc.


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 848



def parse_constant container, tk, comment, ignore_constants = false
  offset  = tk.seek
  line_no = tk.line_no

  name = tk.name
  skip_tkspace false

  return unless name =~ /^\w+$/

  eq_tk = get_tk

  if TkCOLON2 === eq_tk then
    unget_tk eq_tk
    unget_tk tk

    container, name_t, = get_class_or_module container, ignore_constants

    name = name_t.name

    eq_tk = get_tk
  end

  unless TkASSIGN === eq_tk then
    unget_tk eq_tk
    return false
  end

  if TkGT === peek_tk then
    unget_tk eq_tk
    return
  end

  value = ''
  con = RDoc::Constant.new name, value, comment

  body = parse_constant_body container, con

  return unless body

  value.replace body
  record_location con
  con.offset = offset
  con.line   = line_no
  read_documentation_modifiers con, RDoc::CONSTANT_MODIFIERS

  @stats.add_constant con
  container.add_constant con

  true
end


  








  

    #parse_meta_attr(context, single, tk, comment)  
  



  

    

Parses a meta-programmed attribute and creates an ::RDoc::Attr.



To create foo and bar attributes on class C with comment “My attributes”:



class C

  ##
  # :attr:
  #
  # My attributes

  my_attr :foo, :bar

end



To create a foo attribute on class C with comment “My attribute”:



class C

  ##
  # :attr: foo
  #
  # My attribute

  my_attr :foo, :bar

end


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1153



def parse_meta_attr(context, single, tk, comment)
  args = parse_symbol_arg
  rw = "?"

  # If nodoc is given, don't document any of them

  tmp = RDoc::CodeObject.new
  read_documentation_modifiers tmp, RDoc::ATTR_MODIFIERS

  if comment.text.sub!(/^# +:?(attr(_reader|_writer|_accessor)?): *(\S*).*?\n/i, '') then
    rw = case $1
         when 'attr_reader' then 'R'
         when 'attr_writer' then 'W'
         else 'RW'
         end
    name = $3 unless $3.empty?
  end

  if name then
    att = create_attr context, single, name, rw, comment
  else
    args.each do |attr_name|
      att = create_attr context, single, attr_name, rw, comment
    end
  end

  att
end


  








  

    #parse_meta_method(container, single, tk, comment)  
  



  

    

Parses a meta-programmed method


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1185



def parse_meta_method(container, single, tk, comment)
  column  = tk.char_no
  offset  = tk.seek
  line_no = tk.line_no

  start_collecting_tokens
  add_token tk
  add_token_listener self

  skip_tkspace false

  singleton = !!comment.text.sub!(/(^# +:?)(singleton-)(method:)/, '\1\3')

  name = parse_meta_method_name comment, tk

  return unless name

  meth = RDoc::MetaMethod.new get_tkread, name
  record_location meth
  meth.offset = offset
  meth.line   = line_no
  meth.singleton = singleton

  remove_token_listener self

  meth.start_collecting_tokens
  indent = TkSPACE.new 0, 1, 1
  indent.set_text " " * column

  position_comment = TkCOMMENT.new 0, line_no, 1
  position_comment.value = "# File #{@top_level.relative_name}, line #{line_no}"
  meth.add_tokens [position_comment, NEWLINE_TOKEN, indent]
  meth.add_tokens @token_stream

  parse_meta_method_params container, single, meth, tk, comment

  meth.comment = comment

  @stats.add_method meth

  meth
end


  








  

    #parse_method(container, single, tk, comment)  
  



  

    

Parses a normal method defined by def


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1290



def parse_method(container, single, tk, comment)
  singleton = nil
  added_container = false
  name = nil
  column  = tk.char_no
  offset  = tk.seek
  line_no = tk.line_no

  start_collecting_tokens
  add_token tk

  token_listener self do
    prev_container = container
    name, container, singleton = parse_method_name container
    added_container = container != prev_container
  end

  return unless name

  meth = RDoc::AnyMethod.new get_tkread, name
  meth.singleton = single == SINGLE ? true : singleton

  record_location meth
  meth.offset = offset
  meth.line   = line_no

  meth.start_collecting_tokens
  indent = TkSPACE.new 0, 1, 1
  indent.set_text " " * column

  token = TkCOMMENT.new 0, line_no, 1
  token.set_text "# File #{@top_level.relative_name}, line #{line_no}"
  meth.add_tokens [token, NEWLINE_TOKEN, indent]
  meth.add_tokens @token_stream

  parse_method_params_and_body container, single, meth, added_container

  comment.normalize
  comment.extract_call_seq meth

  meth.comment = comment

  @stats.add_method meth
end


  








  

    #parse_method_dummy(container)  
  



  

    

Parses a method that needs to be ignored.


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1369



def parse_method_dummy container
  dummy = RDoc::Context.new
  dummy.parent = container
  dummy.store  = container.store
  skip_method dummy
end


  








  

    #parse_method_or_yield_parameters(method = nil, modifiers = RDoc::METHOD_MODIFIERS)  
  



  

    

Extracts yield parameters from method


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1479



def parse_method_or_yield_parameters(method = nil,
                                     modifiers = RDoc::METHOD_MODIFIERS)
  skip_tkspace false
  tk = get_tk
  end_token = get_end_token tk
  return '' unless end_token

  nest = 0

  loop do
    case tk
    when TkSEMICOLON then
      break if nest == 0
    when TkLBRACE, TkfLBRACE then
      nest += 1
    when TkRBRACE then
      nest -= 1
      if nest <= 0
        # we might have a.each { |i| yield i }
        unget_tk(tk) if nest < 0
        break
      end
    when TkLPAREN, TkfLPAREN then
      nest += 1
    when end_token then
      if end_token == TkRPAREN
        nest -= 1
        break if nest <= 0
      else
        break unless @scanner.continue
      end
    when TkRPAREN then
      nest -= 1
    when method && method.block_params.nil? && TkCOMMENT then
      unget_tk tk
      read_documentation_modifiers method, modifiers
      @read.pop
    when TkCOMMENT then
      @read.pop
    when nil then
      break
    end
    tk = get_tk
  end

  get_tkread_clean(/\s+/, ' ')
end


  








  

    #parse_method_parameters(method)  
  



  

    

Capture the method's parameters. Along the way, look for a comment containing:



# yields: ....



and add this as the block_params for the method


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1535



def parse_method_parameters method
  res = parse_method_or_yield_parameters method

  res = "(#{res})" unless res =~ /\A\(/
  method.params = res unless method.params

  return if  method.block_params

  skip_tkspace false
  read_documentation_modifiers method, RDoc::METHOD_MODIFIERS
end


  








  

    #parse_method_params_and_body(container, single, meth, added_container)  
  



  

    

Parses the parameters and body of meth


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1338



def parse_method_params_and_body container, single, meth, added_container
  token_listener meth do
    @scanner.continue = false
    parse_method_parameters meth

    if meth.document_self or not @track_visibility then
      container.add_method meth
    elsif added_container then
      container.document_self = false
    end

    # Having now read the method parameters and documentation modifiers, we
    # now know whether we have to rename #initialize to ::new

    if meth.name == "initialize" && !meth.singleton then
      if meth.dont_rename_initialize then
        meth.visibility = :protected
      else
        meth.singleton = true
        meth.name = "new"
        meth.visibility = :public
      end
    end

    parse_statements container, single, meth
  end
end


  








  

    #parse_module(container, single, tk, comment)  
  



  

    

Parses an ::RDoc::NormalModule in container with comment


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1550



def parse_module container, single, tk, comment
  container, name_t, = get_class_or_module container

  name = name_t.name

  mod = container.add_module RDoc::NormalModule, name
  mod.ignore unless container.document_children
  record_location mod

  read_documentation_modifiers mod, RDoc::CLASS_MODIFIERS
  mod.add_comment comment, @top_level
  parse_statements mod

  @stats.add_module mod
end


  








  

    #parse_require(context, comment)  
  



  

    

Parses an ::RDoc::Require in context containing comment


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1569



def parse_require(context, comment)
  skip_tkspace_comment
  tk = get_tk

  if TkLPAREN === tk then
    skip_tkspace_comment
    tk = get_tk
  end

  name = tk.text if TkSTRING === tk

  if name then
    @top_level.add_require RDoc::Require.new(name, comment)
  else
    unget_tk tk
  end
end


  








  

    #parse_rescue  
  



  

    

Parses a rescue


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1590



def parse_rescue
  skip_tkspace false

  while tk = get_tk
    case tk
    when TkNL, TkSEMICOLON then
      break
    when TkCOMMA then
      skip_tkspace false

      get_tk if TkNL === peek_tk
    end

    skip_tkspace false
  end
end


  








  

    #parse_statements(container, single = NORMAL, current_method = nil, comment = new_comment(''))  
  



  

    

The core of the Ruby parser.


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1610



def parse_statements(container, single = NORMAL, current_method = nil,
                     comment = new_comment(''))
  raise 'no' unless RDoc::Comment === comment
  comment.force_encoding @encoding if @encoding

  nest = 1
  save_visibility = container.visibility

  non_comment_seen = true

  while tk = get_tk do
    keep_comment = false
    try_parse_comment = false

    non_comment_seen = true unless TkCOMMENT === tk

    case tk
    when TkNL then
      skip_tkspace
      tk = get_tk

      if TkCOMMENT === tk then
        if non_comment_seen then
          # Look for RDoc in a comment about to be thrown away
          non_comment_seen = parse_comment container, tk, comment unless
            comment.empty?

          comment = ''
          comment.force_encoding @encoding if @encoding
        end

        while TkCOMMENT === tk do
          comment << tk.text << "\n"

          tk = get_tk

          if TkNL === tk then
            skip_tkspace false # leading spaces
            tk = get_tk
          end
        end

        comment = new_comment comment

        unless comment.empty? then
          look_for_directives_in container, comment

          if container.done_documenting then
            throw :eof if RDoc::TopLevel === container
            container.ongoing_visibility = save_visibility
          end
        end

        keep_comment = true
      else
        non_comment_seen = true
      end

      unget_tk tk
      keep_comment = true

    when TkCLASS then
      parse_class container, single, tk, comment

    when TkMODULE then
      parse_module container, single, tk, comment

    when TkDEF then
      parse_method container, single, tk, comment

    when TkCONSTANT then
      unless parse_constant container, tk, comment, current_method then
        try_parse_comment = true
      end

    when TkALIAS then
      parse_alias container, single, tk, comment unless current_method

    when TkYIELD then
      if current_method.nil? then
        warn "Warning: yield outside of method" if container.document_self
      else
        parse_yield container, single, tk, current_method
      end

    # Until and While can have a 'do', which shouldn't increase the nesting.
    # We can't solve the general case, but we can handle most occurrences by
    # ignoring a do at the end of a line.

    when TkUNTIL, TkWHILE then
      nest += 1
      skip_optional_do_after_expression

    # 'for' is trickier
    when TkFOR then
      nest += 1
      skip_for_variable
      skip_optional_do_after_expression

    when TkCASE, TkDO, TkIF, TkUNLESS, TkBEGIN then
      nest += 1

    when TkSUPER then
      current_method.calls_super = true if current_method

    when TkRESCUE then
      parse_rescue

    when TkIDENTIFIER then
      if nest == 1 and current_method.nil? then
        keep_comment = parse_identifier container, single, tk, comment
      end

      case tk.name
      when "require" then
        parse_require container, comment
      when "include" then
        parse_extend_or_include RDoc::Include, container, comment
      when "extend" then
        parse_extend_or_include RDoc::Extend, container, comment
      end

    when TkEND then
      nest -= 1
      if nest == 0 then
        read_documentation_modifiers container, RDoc::CLASS_MODIFIERS
        container.ongoing_visibility = save_visibility

        parse_comment container, tk, comment unless comment.empty?

        return
      end
    else
      try_parse_comment = nest == 1
    end

    if try_parse_comment then
      non_comment_seen = parse_comment container, tk, comment unless
        comment.empty?

      keep_comment = false
    end

    unless keep_comment then
      comment = new_comment ''
      comment.force_encoding @encoding if @encoding
      container.params = nil
      container.block_params = nil
    end

    consume_trailing_spaces
  end

  container.params = nil
  container.block_params = nil
end


  








  

    #parse_symbol_arg(no = nil)  
  



  

    

Parse up to no symbol arguments


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1770



def parse_symbol_arg(no = nil)
  skip_tkspace_comment

  case tk = get_tk
  when TkLPAREN
    parse_symbol_arg_paren no
  else
    parse_symbol_arg_space no, tk
  end
end


  








  

    #parse_symbol_in_arg  
  



  

    

Returns symbol text from the next token


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1844



def parse_symbol_in_arg
  case tk = get_tk
  when TkSYMBOL
    tk.text.sub(/^:/, '')
  when TkSTRING
    eval @read[-1]
  when TkDSTRING, TkIDENTIFIER then
    nil # ignore
  else
    warn("Expected symbol or string, got #{tk.inspect}") if $DEBUG_RDOC
    nil
  end
end


  








  

    #parse_top_level_statements(container)  
  



  

    

Parses statements in the top-level container


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1861



def parse_top_level_statements container
  comment = collect_first_comment

  look_for_directives_in container, comment

  throw :eof if container.done_documenting

  @markup = comment.format

  # HACK move if to RDoc::Context#comment=
  container.comment = comment if container.document_self unless comment.empty?

  parse_statements container, NORMAL, nil, comment
end


  








  

    #parse_visibility(container, single, tk)  
  



  

    

Determines the visibility in container from tk


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1879



def parse_visibility(container, single, tk)
  vis_type, vis, singleton = get_visibility_information tk, single

  skip_tkspace_comment false

  case peek_tk
    # Ryan Davis suggested the extension to ignore modifiers, because he
    # often writes
    #
    #   protected unless $TESTING
    #
  when TkNL, TkUNLESS_MOD, TkIF_MOD, TkSEMICOLON then
    container.ongoing_visibility = vis
  else
    update_visibility container, vis_type, vis, singleton
  end
end


  








  

    #parse_yield(context, single, tk, method)  
  



  

    

Determines the block parameter for context


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1900



def parse_yield(context, single, tk, method)
  return if method.block_params

  get_tkread
  @scanner.continue = false
  method.block_params = parse_method_or_yield_parameters
end


  








  

    #read_directive(allowed)  
  



  

    

Directives are modifier comments that can appear after class, module, or method names. For example:



def fred # :yields: a, b



or:



class MyClass # :nodoc:



We return the directive name and any parameters as a two element array if the name is in allowed.  A directive can be found anywhere up to the end of the current line.


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1922



def read_directive allowed
  tokens = []

  while tk = get_tk do
    tokens << tk

    case tk
    when TkNL, TkDEF then
      return
    when TkCOMMENT then
      return unless tk.text =~ /\s*:?([\w-]+):\s*(.*)/

      directive = $1.downcase

      return [directive, $2] if allowed.include? directive

      return
    end
  end
ensure
  unless tokens.length == 1 and TkCOMMENT === tokens.first then
    tokens.reverse_each do |token|
      unget_tk token
    end
  end
end


  








  

    #read_documentation_modifiers(context, allowed)  
  



  

    

Handles directives following the definition for context (any ::RDoc::CodeObject) if the directives are allowed at this point.



See also Markup::PreProcess#handle_directive


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1955



def read_documentation_modifiers context, allowed
  directive, value = read_directive allowed

  return unless directive

  @preprocess.handle_directive '', directive, value, context do |dir, param|
    if %w[notnew not_new not-new].include? dir then
      context.dont_rename_initialize = true

      true
    end
  end
end


  








  

    #remove_private_comments(comment)  
  



  

    

Removes private comments from comment


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1987



def remove_private_comments comment
  comment.remove_private
end


  








  

    #scan  
  



  

    

Scans this Ruby file for Ruby constructs


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 1994



def scan
  reset

  catch :eof do
    begin
      parse_top_level_statements @top_level

    rescue StandardError => e
      bytes = ''

      20.times do @scanner.ungetc end
      count = 0
      60.times do |i|
        count = i
        byte = @scanner.getc
        break unless byte
        bytes << byte
      end
      count -= 20
      count.times do @scanner.ungetc end

      $stderr.puts <<-EOF

#{self.class} failure around line #{@scanner.line_no} of
#{@file_name}

      EOF

      unless bytes.empty? then
        $stderr.puts
        $stderr.puts bytes.inspect
      end

      raise e
    end
  end

  @top_level
end


  








  

    #skip_for_variable  
  



  

    

skip the var [in] part of a 'for' statement


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 2079



def skip_for_variable
  skip_tkspace false
  get_tk
  skip_tkspace false
  tk = get_tk
  unget_tk(tk) unless TkIN === tk
end


  








  

    #skip_method(container)  
  



  

    

Skips the next method in container


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 2090



def skip_method container
  meth = RDoc::AnyMethod.new "", "anon"
  parse_method_parameters meth
  parse_statements container, false, meth
end


  








  

    #skip_optional_do_after_expression  
  



  

    

while, until, and for have an optional do


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 2037



def skip_optional_do_after_expression
  skip_tkspace false
  tk = get_tk
  end_token = get_end_token tk

  b_nest = 0
  nest = 0
  @scanner.continue = false

  loop do
    case tk
    when TkSEMICOLON then
      break if b_nest.zero?
    when TkLPAREN, TkfLPAREN then
      nest += 1
    when TkBEGIN then
      b_nest += 1
    when TkEND then
      b_nest -= 1
    when TkDO
      break if nest.zero?
    when end_token then
      if end_token == TkRPAREN
        nest -= 1
        break if @scanner.lex_state == :EXPR_END and nest.zero?
      else
        break unless @scanner.continue
      end
    when nil then
      break
    end
    tk = get_tk
  end

  skip_tkspace false

  get_tk if TkDO === peek_tk
end


  








  

    #skip_tkspace_comment(skip_nl = true)  
  



  

    

Skip spaces until a comment is found


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 2099



def skip_tkspace_comment(skip_nl = true)
  loop do
    skip_tkspace skip_nl
    return unless TkCOMMENT === peek_tk
    get_tk
  end
end


  








  

    #warn(message)  
  



  

    

Prints message to $stderr unless we're being quiet


  



  [ GitHub ]


  

  


# File 'lib/rdoc/parser/ruby.rb', line 2155



def warn message
  @options.warn make_message message
end