Class: RDoc::RubyLex

Relationships & Source Files
Namespace Children
Exceptions: Error
Super Chains via Extension / Inclusion / Inheritance
Class Chain: self, Exception2MessageMapper
Instance Chain: self, IRB
Inherits:	Object
Defined in:	lib/rdoc/ruby_lex.rb

Overview

Ruby lexer adapted from irb.

The internals are not documented because they are scary.

Class Method Summary

.new(content, options) ⇒ RubyLex constructor

Creates a new lexer for content.
.tokenize(ruby, options)

Returns an Array of ruby tokens.

Constructor Details

.new(content, options) ⇒ `RubyLex`

Creates a new lexer for content. options is an Options, only +tab_width is used.

[ GitHub ]

# File 'lib/rdoc/ruby_lex.rb', line 83


def initialize(content, options)
  lex_init

  if /\t/ =~ content then
    tab_width = options.tab_width
    content = content.split(/\n/).map do |line|
      1 while line.gsub!(/\t+/) {
        ' ' * (tab_width*$&.length - $`.length % tab_width)
      }  && $~
      line
    end.join("\n")
  end

  content << "\n" unless content[-1, 1] == "\n"

  set_input StringIO.new content

  @base_char_no = 0
  @char_no = 0
  @exp_line_no = @line_no = 1
  @here_readed = []
  @readed = []
  @rests = []
  @seek = 0

  @here_header = false
  @indent = 0
  @indent_stack = []
  @lex_state = :EXPR_BEG
  @space_seen = false

  @continue = false
  @line = ""

  @skip_space = false
  @readed_auto_clean_up = false
  @exception_on_syntax_error = true

  @prompt = nil
  @prev_seek = nil
  @ltype = nil
end

Class Method Details

.tokenize(ruby, options)

Returns an Array of ruby tokens. See .new for a description of options.