Module: RDoc::TokenStream
Relationships & Source Files | |
Extension / Inclusion / Inheritance Descendants | |
Included In:
| |
Defined in: | lib/rdoc/token_stream.rb |
Overview
A TokenStream is a list of tokens, gathered during the parse of some entity (say a method). Entities populate these streams by being registered with the lexer. Any class can collect tokens by including TokenStream
. From the outside, you use such an object by calling the start_collecting_tokens method, followed by calls to add_token and pop_token.
Class Method Summary
-
.to_html(token_stream)
Converts #token_stream to HTML wrapping various tokens with
<span>
elements.
Instance Method Summary
-
#add_token(*tokens)
Alias for #add_tokens.
-
#add_tokens(*tokens)
(also: #add_token)
Adds
tokens
to the collected tokens. -
#collect_tokens
(also: #start_collecting_tokens)
Starts collecting tokens.
-
#pop_token
Remove the last token from the collected tokens.
-
#start_collecting_tokens
Alias for #collect_tokens.
-
#token_stream
Current token stream.
-
#tokens_to_s
Returns a string representation of the token stream.
Class Method Details
.to_html(token_stream)
Converts #token_stream to HTML wrapping various tokens with <span>
elements. The following tokens types are wrapped in spans with the given class names:
- TkCONSTANT
-
'ruby-constant'
- TkKW
-
'ruby-keyword'
- TkIVAR
-
'ruby-ivar'
- TkOp
-
'ruby-operator'
- TkId
-
'ruby-identifier'
- TkNode
-
'ruby-node'
- TkCOMMENT
-
'ruby-comment'
- TkREGEXP
-
'ruby-regexp'
- TkSTRING
-
'ruby-string'
- TkVal
-
'ruby-value'
Other token types are not wrapped in spans.
# File 'lib/rdoc/token_stream.rb', line 29
def self.to_html token_stream token_stream.map do |t| next unless t style = case t when RDoc::RubyToken::TkCONSTANT then 'ruby-constant' when RDoc::RubyToken::TkKW then 'ruby-keyword' when RDoc::RubyToken::TkIVAR then 'ruby-ivar' when RDoc::RubyToken::TkOp then 'ruby-operator' when RDoc::RubyToken::TkId then 'ruby-identifier' when RDoc::RubyToken::TkNode then 'ruby-node' when RDoc::RubyToken::TkCOMMENT then 'ruby-comment' when RDoc::RubyToken::TkREGEXP then 'ruby-regexp' when RDoc::RubyToken::TkSTRING then 'ruby-string' when RDoc::RubyToken::TkVal then 'ruby-value' end text = CGI.escapeHTML t.text if style then "<span class=\"#{style}\">#{text}</span>" else text end end.join end
Instance Method Details
#add_token(*tokens)
Alias for #add_tokens.
# File 'lib/rdoc/token_stream.rb', line 63
alias add_token add_tokens
#add_tokens(*tokens) Also known as: #add_token
Adds tokens
to the collected tokens
# File 'lib/rdoc/token_stream.rb', line 59
def add_tokens(*tokens) tokens.flatten.each { |token| @token_stream << token } end
#collect_tokens Also known as: #start_collecting_tokens
Starts collecting tokens
# File 'lib/rdoc/token_stream.rb', line 68
def collect_tokens @token_stream = [] end
#pop_token
Remove the last token from the collected tokens
# File 'lib/rdoc/token_stream.rb', line 77
def pop_token @token_stream.pop end
#start_collecting_tokens
Alias for #collect_tokens.
# File 'lib/rdoc/token_stream.rb', line 72
alias start_collecting_tokens collect_tokens
#token_stream
Current token stream
# File 'lib/rdoc/token_stream.rb', line 84
def token_stream @token_stream end
#tokens_to_s
Returns a string representation of the token stream
# File 'lib/rdoc/token_stream.rb', line 91
def tokens_to_s token_stream.compact.map { |token| token.text }.join '' end