123456789_123456789_123456789_123456789_123456789_

Encodings

The Basics

A character encoding, often shortened to encoding, is a mapping between:

Some character sets contain only 1-byte characters; US-ASCII, for example, has 256 1-byte characters. This string, encoded in US-ASCII, has six characters that are stored as six bytes:

s = 'Hello!'.encode(Encoding::US_ASCII)  # => "Hello!"
s.encoding                               # => #<Encoding:US-ASCII>
s.bytes                                  # => [72, 101, 108, 108, 111, 33]

Other encodings may involve multi-byte characters. UTF-8, for example, encodes more than one million characters, encoding each in one to four bytes. The lowest-valued of these characters correspond to ASCII characters, and so are 1-byte characters:

s = 'Hello!' # => "Hello!"
s.bytes      # => [72, 101, 108, 108, 111, 33]

Other characters, such as the Euro symbol, are multi-byte:

s = "\u20ac" # => "€"
s.bytes      # => [226, 130, 172]

The Encoding Class

Encoding Objects

Ruby encodings are defined by constants in class Encoding. There can be only one instance of Encoding for each of these constants. Method Encoding.list returns an array of Encoding objects (one for each constant):

Encoding.list.size        # => 103
Encoding.list.first.class # => Encoding
Encoding.list.take(3)
# => [#<Encoding:ASCII-8BIT>, #<Encoding:UTF-8>, #<Encoding:US-ASCII>]

Names and Aliases

Method Encoding#name returns the name of an Encoding:

Encoding::ASCII_8BIT.name  # => "ASCII-8BIT"
Encoding::WINDOWS_31J.name # => "Windows-31J"

An Encoding object has zero or more aliases; method Encoding#names returns an array containing the name and all aliases:

Encoding::ASCII_8BIT.names
# => ["ASCII-8BIT", "BINARY"]
Encoding::WINDOWS_31J.names
#=> ["Windows-31J", "CP932", "csWindows31J", "SJIS", "PCK"]

Method Encoding.aliases returns a hash of all alias/name pairs:

Encoding.aliases.size # => 71
Encoding.aliases.take(3)
# => [["BINARY", "ASCII-8BIT"], ["CP437", "IBM437"], ["CP720", "IBM720"]]

Method Encoding.name_list returns an array of all the encoding names and aliases:

Encoding.name_list.size # => 175
Encoding.name_list.take(3)
# => ["ASCII-8BIT", "UTF-8", "US-ASCII"]

Method name_list returns more entries than method list because it includes both the names and their aliases.

Method Encoding.find returns the Encoding for a given name or alias, if it exists:

Encoding.find("US-ASCII")       # => #<Encoding:US-ASCII>
Encoding.find("US-ASCII").class # => Encoding

Default Encodings

Method Encoding.find, above, also returns a default Encoding for each of these special names:

Method Encoding.default_external returns the default external Encoding:

Encoding.default_external # => #<Encoding:UTF-8>

Method Encoding.default_external= sets that value:

Encoding.default_external = Encoding::US_ASCII # => #<Encoding:US-ASCII>
Encoding.default_external                      # => #<Encoding:US-ASCII>

Method Encoding.default_internal returns the default internal Encoding:

Encoding.default_internal # => nil

Method Encoding.default_internal= sets the default internal Encoding:

Encoding.default_internal = Encoding::US_ASCII # => #<Encoding:US-ASCII>
Encoding.default_internal                      # => #<Encoding:US-ASCII>

Compatible Encodings

Method Encoding.compatible? returns whether two given objects are encoding-compatible (that is, whether they can be concatenated); returns the Encoding of the concatenated string, or nil if incompatible:

rus = "\u{442 435 441 442}"
eng = 'text'
Encoding.compatible?(rus, eng) # => #<Encoding:UTF-8>

s0 = "\xa1\xa1".force_encoding(Encoding::ISO_8859_1) # => "\xA1\xA1"
s1 = "\xa1\xa1".force_encoding(Encoding::EUCJP)      # => "\x{A1A1}"
Encoding.compatible?(s0, s1)                         # => nil

String Encoding

A Ruby String object has an encoding that is an instance of class Encoding. The encoding may be retrieved by method String#encoding.

The default encoding for a string literal is the script encoding; see Script Encoding.

's'.encoding # => #<Encoding:UTF-8>

The default encoding for a string created with method String.new is:

In either case, any encoding may be specified:

s = String.new(encoding: Encoding::UTF_8)         # => ""
s.encoding                                        # => #<Encoding:UTF-8>
s = String.new('foo', encoding: Encoding::BINARY) # => "foo"
s.encoding                                        # => #<Encoding:BINARY (ASCII-8BIT)>

The encoding for a string may be changed:

s = "R\xC3\xA9sum\xC3\xA9"              # => "Résumé"
s.encoding                              # => #<Encoding:UTF-8>
s.force_encoding(Encoding::ISO_8859_1)  # => "R\xC3\xA9sum\xC3\xA9"
s.encoding                              # => #<Encoding:ISO-8859-1>

Changing the assigned encoding does not alter the content of the string; it changes only the way the content is to be interpreted:

s                                 # => "R\xC3\xA9sum\xC3\xA9"
s.force_encoding(Encoding::UTF_8) # => "Résumé"

The actual content of a string may also be altered; see Transcoding a String.

Here are a couple of useful query methods:

s = "abc".force_encoding(Encoding::UTF_8)         # => "abc"
s.ascii_only?                                     # => true
s = "abc\u{6666}".force_encoding(Encoding::UTF_8) # => "abc晦"
s.ascii_only?                                     # => false

s = "\xc2\xa1".force_encoding(Encoding::UTF_8)    # => "¡"
s.valid_encoding?                                 # => true
s = "\xc2".force_encoding(Encoding::UTF_8)        # => "\xC2"
s.valid_encoding?                                 # => false

Symbol and Regexp Encodings

The string stored in a Symbol or Regexp object also has an encoding; the encoding may be retrieved by method Symbol#encoding or Regexp#encoding.

The default encoding for these, however, is: