Module: NKF
| Relationships & Source Files | |
| Defined in: | ext/nkf/nkf.c |
Overview
NKF - Ruby extension for Network Kanji Filter
Description
This is a Ruby Extension version of nkf (Network Kanji Filter). It converts the first argument and returns converted result. Conversion details are specified by flags as the first argument.
Nkf is a yet another kanji code converter among networks, hosts and terminals. It converts input kanji code to designated kanji code such as ISO-2022-JP, Shift_JIS, EUC-JP, UTF-8 or UTF-16.
One of the most unique faculty of nkf is the guess of the input kanji encodings. It currently recognizes ISO-2022-JP, Shift_JIS, EUC-JP, UTF-8 and UTF-16. So users needn't set the input kanji code explicitly.
By default, X0201 kana is converted into X0208 kana. For X0201 kana, SO/SI, SSO and ESC-(-I methods are supported. For automatic code detection, nkf assumes no X0201 kana in Shift_JIS. To accept X0201 in Shift_JIS, use -X, -x or -S.
Flags
-b -u
Output is buffered (DEFAULT), Output is unbuffered.
-j -s -e -w -w16 -w32
Output code is ISO-2022-JP (7bit JIS), Shift_JIS, EUC-JP, UTF-8N, UTF-16BE, UTF-32BE. Without this option and compile option, ISO-2022-JP is assumed.
-J -S -E -W -W16 -W32
Input assumption is JIS 7 bit, Shift_JIS, EUC-JP, UTF-8, UTF-16, UTF-32.
-J
Assume JIS input. It also accepts EUC-JP. This is the default. This flag does not exclude Shift_JIS.
-S
Assume Shift_JIS and X0201 kana input. It also accepts JIS. EUC-JP is recognized as X0201 kana. Without -x flag, X0201 kana (halfwidth kana) is converted into X0208.
-E
Assume EUC-JP input. It also accepts JIS. Same as -J.
-t
No conversion.
-i_
Output sequence to designate JIS-kanji. (DEFAULT B)
-o_
Output sequence to designate ASCII. (DEFAULT B)
-r
de/encrypt ROT13/47
-h --hiragana --katakana --katakana-hiragana
[-h1 --hiragana] Katakana to Hiragana conversion.
[-h2 --katakana] Hiragana to Katakana conversion.
[-h3 --katakana-hiragana] Katakana to Hiragana and Hiragana to Katakana conversion.
-T
Text mode output (MS-DOS)
-l
ISO8859-1 (Latin-1) support
-f[m [- n]]
Folding on m length with n margin in a line.
Without this option, fold length is 60 and fold margin is 10.
-F
New line preserving line folding.
-Z
Convert X0208 alphabet (Fullwidth Alphabets) to ASCII.
[-Z -Z0] Convert X0208 alphabet to ASCII.
[-Z1] Converts X0208 kankaku to single ASCII space.
[-Z2] Converts X0208 kankaku to double ASCII spaces.
[-Z3] Replacing Fullwidth >, <, ", & into '>', '<', '"', '&' as in HTML.
-X -x
Assume X0201 kana in MS-Kanji. With -X or without this option, X0201 is converted into X0208 Kana. With -x, try to preserve X0208 kana and do not convert X0201 kana to X0208. In JIS output, ESC-(-I is used. In EUC output, SSO is used.
-B
Assume broken JIS-Kanji input, which lost ESC. Useful when your site is using old B-News Nihongo patch.
[-B1] allows any char after ESC-( or ESC-$.
[-B2] forces ASCII after NL.
-I
Replacing non iso-2022-jp char into a geta character (substitute character in Japanese).
-d -c
Delete \r in line feed, Add \r in line feed.
-m
MIME ISO-2022-JP/ISO8859-1 decode. (DEFAULT) To see ISO8859-1 (Latin-1) -l is necessary.
[-mB] Decode MIME base64 encoded stream. Remove header or other part before conversion.
[-mQ] Decode MIME quoted stream. '_' in quoted stream is converted to space.
[-mN] Non-strict decoding. It allows line break in the middle of the base64 encoding.
[-m0] No MIME decode.
-M
MIME encode. Header style. All ASCII code and control characters are intact. Kanji conversion is performed before encoding, so this cannot be used as a picture encoder.
[-MB] MIME encode Base64 stream.
[-MQ] Perform quoted encoding.
-l
Input and output code is ISO8859-1 (Latin-1) and ISO-2022-JP. -s, -e and -x are not compatible with this option.
-L
new line mode Without this option, nkf doesn't convert line breaks.
[-Lu] unix (LF)
[-Lw] windows (CRLF)
[-Lm] mac (CR)
--fj --unix --mac --msdos --windows
convert for these system
--jis --euc --sjis --mime --base64
convert for named code
--jis-input --euc-input --sjis-input --mime-input --base64-input
assume input system
--ic=input codeset --oc=output codeset
Set the input or output codeset.
NKF supports following codesets and those codeset name are case insensitive.
[ISO-2022-JP] a.k.a. RFC1468, 7bit JIS, JUNET
[EUC-JP (eucJP-nkf)] a.k.a. AT&T JIS, Japanese EUC, UJIS
[eucJP-ascii] a.k.a. x-eucjp-open-19970715-ascii
[eucJP-ms] a.k.a. x-eucjp-open-19970715-ms
[CP51932] Microsoft Version of EUC-JP.
[Shift_JIS] SJIS, MS-Kanji
[Windows-31J] a.k.a. CP932
[UTF-8] same as UTF-8N
[UTF-8N] UTF-8 without BOM
[UTF-8-BOM] UTF-8 with BOM
[UTF-16] same as UTF-16BE
[UTF-16BE] UTF-16 Big Endian without BOM
[UTF-16BE-BOM] UTF-16 Big Endian with BOM
[UTF-16LE] UTF-16 Little Endian without BOM
[UTF-16LE-BOM] UTF-16 Little Endian with BOM
[UTF-32] same as UTF-32BE
[UTF-32BE] UTF-32 Big Endian without BOM
[UTF-32BE-BOM] UTF-32 Big Endian with BOM
[UTF-32LE] UTF-32 Little Endian without BOM
[UTF-32LE-BOM] UTF-32 Little Endian with BOM
[UTF8-MAC] NKDed UTF-8, a.k.a. UTF8-NFD (input only)
--fb-html, xml, perl, java, subchar
Specify the way that nkf handles unassigned characters. Without this option, --fb-skip is assumed.
--prefix= escape character target character ..
When nkf converts to Shift_JIS, nkf adds a specified escape character to specified 2nd byte of Shift_JIS characters. 1st byte of argument is the escape character and following bytes are target characters.
--no-cp932ext
Handle the characters extended in CP932 as unassigned characters.
--no-best-fit-chars
When Unicode to Encoded byte conversion, don't convert characters which is not round trip safe. When Unicode to Unicode conversion, with this and -x option, nkf can be used as UTF converter. (In other words, without this and -x option, nkf doesn't save some characters)
When nkf convert string which related to path, you should use this option.
--cap-input
Decode hex encoded characters.
--url-input
Unescape percent escaped characters.
==== --
Ignore rest of -option.
Constant Summary
-
ASCII =
# File 'ext/nkf/nkf.c', line 490rb_enc_from_encoding(rb_nkf_enc_get("US-ASCII"))
-
AUTO =
# File 'ext/nkf/nkf.c', line 486Qnil -
BINARY =
# File 'ext/nkf/nkf.c', line 489rb_enc_from_encoding(rb_nkf_enc_get("BINARY"))
-
EUC =
# File 'ext/nkf/nkf.c', line 492rb_enc_from_encoding(rb_nkf_enc_get("EUC-JP"))
-
GEM_VERSION =
# File 'ext/nkf/nkf.c', line 505
Version of nkf library
rb_str_new_cstr(NKF_GEM_VERSION)
-
JIS =
# File 'ext/nkf/nkf.c', line 491rb_enc_from_encoding(rb_nkf_enc_get("ISO-2022-JP"))
-
NKF_RELEASE_DATE =
# File 'ext/nkf/nkf.c', line 503
Release date of nkf
rb_str_new2(NKF_RELEASE_DATE)
-
NKF_VERSION =
# File 'ext/nkf/nkf.c', line 501
Version of nkf
rb_str_new2(NKF_VERSION)
-
NOCONV =
# File 'ext/nkf/nkf.c', line 487Qnil -
SJIS =
# File 'ext/nkf/nkf.c', line 493rb_enc_from_encoding(rb_nkf_enc_get("Shift_JIS"))
-
UNKNOWN =
# File 'ext/nkf/nkf.c', line 488Qnil -
UTF16 =
# File 'ext/nkf/nkf.c', line 495rb_enc_from_encoding(rb_nkf_enc_get("UTF-16BE"))
-
UTF32 =
# File 'ext/nkf/nkf.c', line 496rb_enc_from_encoding(rb_nkf_enc_get("UTF-32BE"))
-
UTF8 =
# File 'ext/nkf/nkf.c', line 494rb_enc_from_encoding(rb_utf8_encoding())
-
VERSION =
# File 'ext/nkf/nkf.c', line 499
Full version string of nkf
rb_str_new2(RUBY_NKF_VERSION)
Class Method Summary
-
.guess(str) ⇒ Encoding
mod_func
Returns guessed encoding of str by nkf routine.
-
.nkf(opt, str) ⇒ String
mod_func
Convert str and return converted result.
Class Method Details
.guess(str) ⇒ Encoding (mod_func)
Returns guessed encoding of str by nkf routine.
# File 'ext/nkf/nkf.c', line 190
static VALUE
rb_nkf_guess(VALUE obj, VALUE src)
{
reinit();
input_ctr = 0;
input = (unsigned char *)StringValuePtr(src);
i_len = RSTRING_LENINT(src);
guess_f = TRUE;
kanji_convert( NULL );
guess_f = FALSE;
return rb_enc_from_encoding(rb_nkf_enc_get(get_guessed_code()));
}
.nkf(opt, str) ⇒ String (mod_func)
Convert str and return converted result.
Conversion details are specified by opt as ::String.
require 'nkf'
output = NKF.nkf("-s", input)# File 'ext/nkf/nkf.c', line 136
static VALUE
rb_nkf_convert(VALUE obj, VALUE opt, VALUE src)
{
VALUE tmp;
reinit();
nkf_split_options(StringValueCStr(opt));
if (!output_encoding) rb_raise(rb_eArgError, "no output encoding given");
switch (nkf_enc_to_index(output_encoding)) {
case UTF_8_BOM: output_encoding = nkf_enc_from_index(UTF_8); break;
case UTF_16BE_BOM: output_encoding = nkf_enc_from_index(UTF_16BE); break;
case UTF_16LE_BOM: output_encoding = nkf_enc_from_index(UTF_16LE); break;
case UTF_32BE_BOM: output_encoding = nkf_enc_from_index(UTF_32BE); break;
case UTF_32LE_BOM: output_encoding = nkf_enc_from_index(UTF_32LE); break;
}
output_bom_f = FALSE;
incsize = INCSIZE;
input_ctr = 0;
input = (unsigned char *)StringValuePtr(src);
i_len = RSTRING_LENINT(src);
tmp = rb_str_new(0, i_len*3 + 10);
output_ctr = 0;
output = (unsigned char *)RSTRING_PTR(tmp);
o_len = RSTRING_LENINT(tmp);
*output = '\0';
/* use _result_ begin*/
result = tmp;
kanji_convert(NULL);
result = Qnil;
/* use _result_ end */
rb_str_set_len(tmp, output_ctr);
if (mimeout_f)
rb_enc_associate(tmp, rb_usascii_encoding());
else
rb_enc_associate(tmp, rb_nkf_enc_get(nkf_enc_name(output_encoding)));
return tmp;
}