123456789_123456789_123456789_123456789_123456789_

Class: RDoc::RubyLex

Relationships & Source Files
Namespace Children
Exceptions:
Error
Super Chains via Extension / Inclusion / Inheritance
Class Chain:
self, Exception2MessageMapper
Instance Chain:
self, IRB
Inherits: Object
Defined in: lib/rdoc/ruby_lex.rb

Overview

Ruby lexer adapted from irb.

The internals are not documented because they are scary.

Class Method Summary

Constructor Details

.new(content, options) ⇒ RubyLex

Creates a new lexer for content. options is an Options, only +tab_width is used.

[ GitHub ] 

  


# File 'lib/rdoc/ruby_lex.rb', line 82



def initialize(content, options)
  lex_init

  if /\t/ =~ content then
    tab_width = options.tab_width
    content = content.split(/\n/).map do |line|
      1 while line.gsub!(/\t+/) {
        ' ' * (tab_width*$&.length - $`.length % tab_width)
      }  && $~
      line
    end.join("\n")
  end

  content << "\n" unless content[-1, 1] == "\n"

  set_input StringIO.new content

  @base_char_no = 0
  @char_no = 0
  @exp_line_no = @line_no = 1
  @here_readed = []
  @readed = []
  @rests = []
  @seek = 0

  @here_header = false
  @indent = 0
  @indent_stack = []
  @lex_state = :EXPR_BEG
  @space_seen = false

  @continue = false
  @line = ""

  @skip_space = false
  @readed_auto_clean_up = false
  @exception_on_syntax_error = true

  @prompt = nil
  @prev_seek = nil
  @ltype = nil
end


  







Class Method Details



  

    .tokenize(ruby, options)  
  



  

    

Returns an Array of ruby tokens.  See .new for a description of options.


  



  [ GitHub ]


  

  


# File 'lib/rdoc/ruby_lex.rb', line 65



def self.tokenize ruby, options
  tokens = []

  scanner = RDoc::RubyLex.new ruby, options
  scanner.exception_on_syntax_error = true

  while token = scanner.token do
    tokens << token
  end

  tokens
end