Module: Prism
Overview
The Prism Ruby parser.
"Parsing Ruby is suddenly manageable!"
- You, hopefully
Constant Summary
-
BACKEND =
# File 'lib/prism.rb', line 137
The FFI backend is used on other Ruby implementations.
:FFI
-
DUMP_OPTIONS_KEYS =
private
# File 'lib/prism/ffi.rb', line 602
The set of options that are understood by the parsing APIs. Note that raise_error is not listed here because it is deleted from the options hash by raise_error_format_type before the options are dumped.
[:attach_comments, :command_line, :encoding, :filepath, :freeze, :frozen_string_literal, :line, :main_script, :partial_script, :scopes, :version].freeze
-
VERSION =
# File 'lib/prism/ffi.rb', line 266
The version constant is set by reading the result of calling pm_version.
LibRubyParser.pm_version.read_string.freeze
Class Method Summary
-
.dump(source, **options)
Mirror the
dumpAPI by using the serialization API. -
.dump_file(filepath, **options)
Mirror the
dump_fileAPI by using the serialization API. -
.find(callable)
Given a Method, UnboundMethod, Proc, or
Thread::Backtrace::Location, returns thePrismnode representing it. -
.lex(code, **options)
Mirror the
lexAPI by using the serialization API. -
.lex_compat(source, **options) ⇒ LexCompat::Result
Returns a parse result whose value is an array of tokens that closely resembles the return value of
Ripper.lex. -
.lex_file(filepath, **options)
Mirror the
lex_fileAPI by using the serialization API. -
.load(source, serialized, freeze) ⇒ ParseResult
Load the serialized AST using the source as a reference into a tree.
-
.parse(code, **options)
Mirror the
parseAPI by using the serialization API. -
.parse_comments(code, **options)
Mirror the
parse_commentsAPI by using the serialization API. -
.parse_failure?(code, **options) ⇒ Boolean
Mirror the
parse_failure?API by using the serialization API. -
.parse_file(filepath, **options)
Mirror the
parse_fileAPI by using the serialization API. -
.parse_file_comments(filepath, **options)
Mirror the
parse_file_commentsAPI by using the serialization API. -
.parse_file_failure?(filepath, **options) ⇒ Boolean
Mirror the
parse_file_failure?API by using the serialization API. -
.parse_file_success?(filepath, **options) ⇒ Boolean
Mirror the
parse_file_success?API by using the serialization API. -
.parse_lex(code, **options)
Mirror the
parse_lexAPI by using the serialization API. -
.parse_lex_file(filepath, **options)
Mirror the
parse_lex_fileAPI by using the serialization API. -
.parse_stream(stream, **options)
Mirror the
parse_streamAPI by using the serialization API. -
.parse_success?(code, **options) ⇒ Boolean
Mirror the
parse_success?API by using the serialization API. -
.profile(source, **options)
Mirror the
profileAPI by using the serialization API. -
.profile_file(filepath, **options)
Mirror the
profile_fileAPI by using the serialization API. -
.scope(locals: [], forwarding: [])
Create a new scope with the given locals and forwarding options that is suitable for passing into one of the
Prism.*methods that accepts thescopesoption. -
.dump_options(options)
private
Convert the given options into a serialized options string.
-
.dump_options_command_line(options)
private
Return the value that should be dumped for the command_line option.
-
.dump_options_version(version)
private
Return the value that should be dumped for the version option.
-
.version_string_to_number(version)
private
Converts a version string like "4.0.0" or "4.0" into a number.
- .dump_common(string, options) private Internal use only
- .lex_common(string, code, options) private Internal use only
- .parse_comments_common(string, code, options) private Internal use only
- .parse_common(string, code, options) private Internal use only
- .parse_file_success_common(string, options) private Internal use only
- .parse_lex_common(string, code, options) private Internal use only
-
.raise_error(string, options, format_type)
private
Internal use only
Parse the given source and format any errors that are encountered into an appropriate exception, then raise it.
-
.raise_error_format_type(options)
private
Internal use only
Extract the raise_error option from the given options hash and convert it into the format type that should be used when formatting errors, or nil if raising is disabled.
Class Method Details
.dump(source, **options)
Mirror the dump API by using the serialization API.
# File 'lib/prism/ffi.rb', line 270
def dump(source, **) LibRubyParser::PrismSource.with_string(source) { |string| dump_common(string, ) } end
.dump_common(string, options) (private)
# File 'lib/prism/ffi.rb', line 420
def dump_common(string, ) # :nodoc: if (format_type = raise_error_format_type()) raise_error(string, , format_type) end LibRubyParser::PrismBuffer.with do |buffer| LibRubyParser.pm_serialize_parse(buffer.pointer, string.pointer, string.length, ()) dumped = buffer.read dumped.freeze if .fetch(:freeze, false) dumped end end
.dump_file(filepath, **options)
Mirror the dump_file API by using the serialization API.
# File 'lib/prism/ffi.rb', line 275
def dump_file(filepath, **) LibRubyParser::PrismSource.with_file(filepath, ) { |string| dump_common(string, ) } end
.dump_options(options) (private)
Convert the given options into a serialized options string.
# File 'lib/prism/ffi.rb', line 606
def () unknown_keys = .keys - DUMP_OPTIONS_KEYS raise ArgumentError, "unknown keyword: #{unknown_keys.first}" unless unknown_keys.empty? template = +"" values = [] template << "L" if (filepath = [:filepath]) values.push(filepath.bytesize, filepath.b) template << "A*" else values << 0 end template << "l" values << .fetch(:line, 1) template << "L" if (encoding = [:encoding]) name = encoding.is_a?(Encoding) ? encoding.name : encoding values.push(name.bytesize, name.b) template << "A*" else values << 0 end template << "C" values << (.fetch(:frozen_string_literal, false) ? 1 : 0) template << "C" values << () template << "C" values << ([:version]) template << "C" values << ([:encoding] == false ? 1 : 0) template << "C" values << (.fetch(:main_script, false) ? 1 : 0) template << "C" values << (.fetch(:partial_script, false) ? 1 : 0) template << "C" values << (.fetch(:freeze, false) ? 1 : 0) template << "C" values << (.fetch(:attach_comments, false) ? 1 : 0) template << "L" if (scopes = [:scopes]) values << scopes.length scopes.each do |scope| locals = nil forwarding = 0 case scope when Array locals = scope when Scope locals = scope.locals scope.forwarding.each do |forward| case forward when :* then forwarding |= 0x1 when :** then forwarding |= 0x2 when :& then forwarding |= 0x4 when :"..." then forwarding |= 0x8 else raise ArgumentError, "invalid forwarding value: #{forward}" end end else raise TypeError, "wrong argument type #{scope.class.inspect} (expected Array or Prism::Scope)" end template << "L" values << locals.length template << "C" values << forwarding locals.each do |local| name = local.name template << "L" values << name.bytesize template << "A*" values << name.b end end else values << 0 end values.pack(template) end
.dump_options_command_line(options) (private)
Return the value that should be dumped for the command_line option.
# File 'lib/prism/ffi.rb', line 547
def () command_line = .fetch(:command_line, "") raise ArgumentError, "command_line must be a string" unless command_line.is_a?(String) command_line.each_char.inject(0) do |value, char| case char when "a" then value | 0b000001 when "e" then value | 0b000010 when "l" then value | 0b000100 when "n" then value | 0b001000 when "p" then value | 0b010000 when "x" then value | 0b100000 else raise ArgumentError, "invalid command_line option: #{char}" end end end
.dump_options_version(version) (private)
Return the value that should be dumped for the version option.
# File 'lib/prism/ffi.rb', line 565
def (version) case version when "current" version_string_to_number(RUBY_VERSION) || raise(CurrentVersionError, RUBY_VERSION) when "latest" 0 # Handled in pm_parser_init when "nearest", nil dump = version_string_to_number(RUBY_VERSION) return dump if dump if RUBY_VERSION < "3.3" version_string_to_number("3.3") else 0 # Handled in pm_parser_init end else version_string_to_number(version) || raise(ArgumentError, "invalid version: #{version}") end end
.find(callable)
Given a Method, UnboundMethod, Proc, or Thread::Backtrace::Location,
returns the Prism node representing it. On CRuby, this uses node_id for
an exact match. On other implementations, it falls back to best-effort
matching by source location line number.
: (Method | UnboundMethod | Proc | Thread::Backtrace::Location callable) -> Node?
.lex(code, **options)
Mirror the lex API by using the serialization API.
# File 'lib/prism/ffi.rb', line 280
def lex(code, **) LibRubyParser::PrismSource.with_string(code) { |string| lex_common(string, code, ) } end
.lex_common(string, code, options) (private)
# File 'lib/prism/ffi.rb', line 435
def lex_common(string, code, ) # :nodoc: format_type = raise_error_format_type() LibRubyParser::PrismBuffer.with do |buffer| LibRubyParser.pm_serialize_lex(buffer.pointer, string.pointer, string.length, ()) result = Serialize.load_lex(code, buffer.read, .fetch(:freeze, false)) raise_error(string, , format_type) if format_type && result.failure? result end end
.lex_compat(source, **options) ⇒ LexCompat::Result
Returns a parse result whose value is an array of tokens that closely
resembles the return value of Ripper.lex.
For supported options, see .parse.
: (String source, **untyped options) -> ::Prism::LexCompat::Result
.lex_file(filepath, **options)
Mirror the lex_file API by using the serialization API.
# File 'lib/prism/ffi.rb', line 285
def lex_file(filepath, **) LibRubyParser::PrismSource.with_file(filepath, ) { |string| lex_common(string, string.read, ) } end
.load(source, serialized, freeze) ⇒ ParseResult
Load the serialized AST using the source as a reference into a tree.
: (String source, String serialized, ?bool freeze) -> ::Prism::ParseResult
# File 'lib/prism.rb', line 84
def self.load(source, serialized, freeze = false) Serialize.load_parse(source, serialized, freeze) end
.parse(code, **options)
Mirror the parse API by using the serialization API.
# File 'lib/prism/ffi.rb', line 290
def parse(code, **) LibRubyParser::PrismSource.with_string(code) { |string| parse_common(string, code, ) } end
.parse_comments(code, **options)
Mirror the parse_comments API by using the serialization API.
# File 'lib/prism/ffi.rb', line 349
def parse_comments(code, **) LibRubyParser::PrismSource.with_string(code) { |string| parse_comments_common(string, code, ) } end
.parse_comments_common(string, code, options) (private)
# File 'lib/prism/ffi.rb', line 456
def parse_comments_common(string, code, ) # :nodoc: if (format_type = raise_error_format_type()) raise_error(string, , format_type) end LibRubyParser::PrismBuffer.with do |buffer| LibRubyParser.pm_serialize_parse_comments(buffer.pointer, string.pointer, string.length, ()) Serialize.load_parse_comments(code, buffer.read, .fetch(:freeze, false)) end end
.parse_common(string, code, options) (private)
# File 'lib/prism/ffi.rb', line 447
def parse_common(string, code, ) # :nodoc: format_type = raise_error_format_type() serialized = dump_common(string, ) result = Serialize.load_parse(code, serialized, .fetch(:freeze, false)) raise_error(string, , format_type) if format_type && result.failure? result end
.parse_failure?(code, **options) ⇒ Boolean
Mirror the parse_failure? API by using the serialization API.
# File 'lib/prism/ffi.rb', line 376
def parse_failure?(code, **) !parse_success?(code, **) end
.parse_file(filepath, **options)
Mirror the parse_file API by using the serialization API. This uses
native strings instead of Ruby strings because it allows us to use mmap
when it is available.
# File 'lib/prism/ffi.rb', line 297
def parse_file(filepath, **) LibRubyParser::PrismSource.with_file(filepath, ) { |string| parse_common(string, string.read, ) } end
.parse_file_comments(filepath, **options)
Mirror the parse_file_comments API by using the serialization
API. This uses native strings instead of Ruby strings because it allows us
to use mmap when it is available.
# File 'lib/prism/ffi.rb', line 356
def parse_file_comments(filepath, **) LibRubyParser::PrismSource.with_file(filepath, ) { |string| parse_comments_common(string, string.read, ) } end
.parse_file_failure?(filepath, **options) ⇒ Boolean
Mirror the parse_file_failure? API by using the serialization API.
# File 'lib/prism/ffi.rb', line 386
def parse_file_failure?(filepath, **) !parse_file_success?(filepath, **) end
.parse_file_success?(filepath, **options) ⇒ Boolean
Mirror the parse_file_success? API by using the serialization API.
# File 'lib/prism/ffi.rb', line 381
def parse_file_success?(filepath, **) LibRubyParser::PrismSource.with_file(filepath, ) { |string| parse_file_success_common(string, ) } end
.parse_file_success_common(string, options) (private)
# File 'lib/prism/ffi.rb', line 479
def parse_file_success_common(string, ) # :nodoc: format_type = raise_error_format_type() success = LibRubyParser.pm_serialize_parse_success_p(string.pointer, string.length, ()) raise_error(string, , format_type) if format_type && !success success end
.parse_lex(code, **options)
Mirror the parse_lex API by using the serialization API.
# File 'lib/prism/ffi.rb', line 361
def parse_lex(code, **) LibRubyParser::PrismSource.with_string(code) { |string| parse_lex_common(string, code, ) } end
.parse_lex_common(string, code, options) (private)
# File 'lib/prism/ffi.rb', line 467
def parse_lex_common(string, code, ) # :nodoc: format_type = raise_error_format_type() LibRubyParser::PrismBuffer.with do |buffer| LibRubyParser.pm_serialize_parse_lex(buffer.pointer, string.pointer, string.length, ()) result = Serialize.load_parse_lex(code, buffer.read, .fetch(:freeze, false)) raise_error(string, , format_type) if format_type && result.failure? result end end
.parse_lex_file(filepath, **options)
Mirror the parse_lex_file API by using the serialization API.
# File 'lib/prism/ffi.rb', line 366
def parse_lex_file(filepath, **) LibRubyParser::PrismSource.with_file(filepath, ) { |string| parse_lex_common(string, string.read, ) } end
.parse_stream(stream, **options)
Mirror the parse_stream API by using the serialization API.
# File 'lib/prism/ffi.rb', line 302
def parse_stream(stream, **) format_type = raise_error_format_type() LibRubyParser::PrismBuffer.with do |buffer| # The largest number of bytes a single character can occupy in any # encoding Ruby supports. IO#gets(limit) may return up to # (max_enc_len - 1) bytes more than requested to avoid splitting a # multi-byte character, so we reserve that much headroom (plus the NUL # terminator) to guarantee the result fits in the caller's buffer. This # mirrors MAX_ENC_LEN in ext/prism/extension.c. max_enc_len = 6 source = +"" callback = -> (string, size, _) { raise "Expected size to be > #{max_enc_len}, got: #{size}" if size <= max_enc_len line = String.try_convert(stream.gets(size - max_enc_len)) if !line.nil? # A misbehaving `gets` may ignore the limit; never write past the # buffer (one byte is reserved for the NUL terminator). line = line.byteslice(0, size - 1) if line.bytesize > size - 1 source << line string.write_string("#{line}\x00", line.bytesize + 1) end } eof_callback = -> (_) { stream.eof? } pm_source = LibRubyParser.pm_source_stream_new(nil, callback, eof_callback) begin LibRubyParser.pm_serialize_parse_stream(buffer.pointer, pm_source, ()) result = Prism.load(source, buffer.read, .fetch(:freeze, false)) if format_type && result.failure? LibRubyParser::PrismSource.with_string(source) do |string| raise_error(string, , format_type) end end result ensure LibRubyParser.pm_source_free(pm_source) if pm_source && !pm_source.null? end end end
.parse_success?(code, **options) ⇒ Boolean
Mirror the parse_success? API by using the serialization API.
# File 'lib/prism/ffi.rb', line 371
def parse_success?(code, **) LibRubyParser::PrismSource.with_string(code) { |string| parse_file_success_common(string, ) } end
.profile(source, **options)
Mirror the profile API by using the serialization API.
# File 'lib/prism/ffi.rb', line 391
def profile(source, **) LibRubyParser::PrismSource.with_string(source) do |string| if (format_type = raise_error_format_type()) raise_error(string, , format_type) end LibRubyParser::PrismBuffer.with do |buffer| LibRubyParser.pm_serialize_parse(buffer.pointer, string.pointer, string.length, ()) nil end end end
.profile_file(filepath, **options)
Mirror the profile_file API by using the serialization API.
# File 'lib/prism/ffi.rb', line 405
def profile_file(filepath, **) LibRubyParser::PrismSource.with_file(filepath, ) do |string| if (format_type = raise_error_format_type()) raise_error(string, , format_type) end LibRubyParser::PrismBuffer.with do |buffer| LibRubyParser.pm_serialize_parse(buffer.pointer, string.pointer, string.length, ()) nil end end end
.raise_error(string, options, format_type) (private)
Parse the given source and format any errors that are encountered into an appropriate exception, then raise it. If the source parses without any errors, then return nil.
# File 'lib/prism/ffi.rb', line 522
def raise_error(string, , format_type) # :nodoc: LibRubyParser::PrismBuffer.with do |buffer| level = LibRubyParser.pm_serialize_parse_errors_format(buffer.pointer, string.pointer, string.length, (), format_type) return if level == -1 encoding_name, _, = buffer.read.partition("\0") case level when 0 # syntax error = SyntaxError.new(.force_encoding(encoding_name)) error.instance_variable_set(:@path, .fetch(:filepath, "")) raise error when 1 # argument raise ArgumentError, .force_encoding(encoding_name) when 2 # load error = LoadError.new(.force_encoding(Encoding.find("locale"))) error.instance_variable_set(:@path, nil) raise error else raise "Unknown error level: #{level}" end end end
.raise_error_format_type(options) (private)
Extract the raise_error option from the given options hash and convert it into the format type that should be used when formatting errors, or nil if raising is disabled.
# File 'lib/prism/ffi.rb', line 490
def raise_error_format_type() # :nodoc: case (value = .delete(:raise_error)) when nil, false nil when true # When given true, mirror the behavior of CRuby itself: color when # $stderr is a terminal (unless NO_COLOR is set), bold styling when # NO_COLOR is set, plain otherwise. This policy is duplicated in # ext/prism/extension.c (build_options_i) and the two must stay in # sync. if $stderr.respond_to?(:tty?) && $stderr.tty? no_color = ENV["NO_COLOR"] (no_color.nil? || no_color.empty?) ? :PM_ERRORS_FORMAT_COLOR : :PM_ERRORS_FORMAT_STYLE else :PM_ERRORS_FORMAT_PLAIN end when :plain :PM_ERRORS_FORMAT_PLAIN when :style :PM_ERRORS_FORMAT_STYLE when :color :PM_ERRORS_FORMAT_COLOR when Symbol raise ArgumentError, "invalid raise_error value: #{value}" else raise TypeError, "wrong argument type #{value.class} (expected Symbol)" end end
.scope(locals: [], forwarding: [])
Create a new scope with the given locals and forwarding options that is
suitable for passing into one of the Prism.* methods that accepts the
scopes option.
: (?locals: Array, ?forwarding: Array) -> ::Prism::Scope
.version_string_to_number(version) (private)
Converts a version string like "4.0.0" or "4.0" into a number. Returns nil if the version is unknown.
# File 'lib/prism/ffi.rb', line 586
def version_string_to_number(version) case version when /\A3\.3(\.\d+)?\z/ 1 when /\A3\.4(\.\d+)?\z/ 2 when /\A3\.5(\.\d+)?\z/, /\A4\.0(\.\d+)?\z/ 3 when /\A4\.1(\.\d+)?\z/ 4 end end