Close menu

GB 18030


Source: http://en.wikipedia.org/wiki/GB_18030
Updated: 2017-08-28T19:03Z
GB 18030
MIMEGB18030
Alias(es)Code page 54936
StandardGB 18030-2005, GB 18030-2000
Language(s)zh
Preceded byGBK, GB2312

GB 18030 is a Chinese government standard, described as Information technology — Chinese coded character set and defines the required language and character support necessary for software in China. GB18030 is the registered Internet name for the official character set of the People's Republic of China (PRC) superseding GB2312.[1] As a Unicode Transformation Format[a] (i.e. an encoding of all Unicode code points), it is compatible with legacy encodings including GB2312, CP936,[b] and GBK 1.0, GB18030 supports both simplified and traditional Chinese characters.

In addition to the "GB18030 character encoding", this standard contains requirements about which scripts must be supported, font support, etc.[2]

History

The GB18030 character set is formally called "Chinese National Standard GB 18030-2005: Information technology — Chinese coded character set". GB abbreviates Guójiā Biāozhǔn (国家标准), which means national standard in Chinese. The standard was published by the China Standard Press, Beijing, November 8, 2005. Only a portion of the standard is mandatory.[2] Since May 1, 2006, support for the mandatory subset is officially required for all software products sold in the PRC.

Different Unicode mappings between GB 18030 versions
GB byte
sequence
Unicode code point
GB 18030-2000 GB 18030-2005
A8 BC (ḿ)
U+E7C7
U+1E3F ḿ
81 35 F4 37U+1E3F ḿ
U+E7C7
An older version of the standard, known as "Chinese National Standard GB 18030-2000: Information Technology — Chinese ideograms coded character set for information interchange — Extension for the basic set", was published on March 17, 2000. The encoding scheme stays the same in the new version, and the only difference in GB-to-Unicode mapping is that GB 18030-2000 mapped the character
A8 BC
(ḿ) to a private use code point U+E7C7, and character
81 35 F4 37
(without specifying any glyph) to U+1E3F (ḿ), whereas GB 18030-2005 swaps these two mapping assignments.[3]:534 More code points are now associated with characters due to update of Unicode, especially the appearance of CJK Unified Ideographs Extension B. Some characters used by ethnic minorities in China, such as Mongolian characters and Tibetan characters (GB 16959-1997 and GB/T 20542-2006), have been added as well, which accounts for the renaming of the standard.

Compared with its ancestors, GB 18030's mapping to Unicode has been modified for the 81 characters that were provisionally assigned a Unicode Private Use Area code point (U+E000–F8FF) in GBK 1.0 and that have later been encoded in Unicode.[4] This is specified in Appendix E of GB 18030.[3]:534[5]:499 There are 24 characters in GB 18030-2005 that are still mapped to Unicode PUA.[6]

Private use characters in GB-to-Unicode mappings
GB byte
sequence
Unicode code point (blue = private use)
GBK 1.0[7][3]:534 GB 18030
-2005[6]
Unicode 4.1
A6 D9[8]:108U+E78DU+FE10
A6 DAU+E78EU+FE12
A6 DBU+E78FU+FE11
A6 DCU+E790U+FE13
A6 DDU+E791U+FE14
A6 DEU+E792U+FE15
A6 DFU+E793U+FE16
A6 ECU+E794U+FE17
A6 EDU+E795U+FE18
A6 F3U+E796U+FE19
A8 BCU+E7C7U+1E3F ḿ
A8 BFU+E7C8U+01F9 ǹ
A9 89U+E7E7U+303E
A9 8AU+E7E8U+2FF0
A9 8BU+E7E9U+2FF1
A9 8CU+E7EAU+2FF2
A9 8DU+E7EBU+2FF3
A9 8EU+E7ECU+2FF4
A9 8FU+E7EDU+2FF5
A9 90U+E7EEU+2FF6
A9 91U+E7EFU+2FF7
A9 92U+E7F0U+2FF8
A9 93U+E7F1U+2FF9
A9 94[8]:173U+E7F2U+2FFA
A9 95U+E7F3U+2FFB
FE 50U+E815U+2E81
FE 51U+E816U+20087 𠂇
FE 52U+E817U+20089 𠂉
FE 53U+E818U+200CC 𠃌
FE 54U+E819U+2E84
FE 55U+E81AU+3473
FE 56U+E81BU+3447
FE 57U+E81CU+2E88
FE 58U+E81DU+2E8B
FE 59U+E81EU+9FB4
FE 5AU+E81FU+359E
FE 5BU+E820U+361A
FE 5CU+E821U+360E
FE 5DU+E822U+2E8C
FE 5EU+E823U+2E97
FE 5FU+E824U+396E
FE 60U+E825U+3918
FE 61U+E826U+9FB5
FE 62U+E827U+39CF
FE 63U+E828U+39DF
FE 64U+E829U+3A73
FE 65U+E82AU+39D0
FE 66U+E82BU+9FB6
FE 67U+E82CU+9FB7
FE 68U+E82DU+3B4E
FE 69U+E82EU+3C6E
FE 6AU+E82FU+3CE0
FE 6BU+E830U+2EA7
FE 6CU+E831U+215D7 𡗗
FE 6DU+E832U+9FB8
FE 6EU+E833U+2EAA
FE 6FU+E834U+4056
FE 70U+E835U+415F
FE 71U+E836U+2EAE
FE 72U+E837U+4337
FE 73U+E838U+2EB3
FE 74U+E839U+2EB6
FE 75U+E83AU+2EB7
FE 76U+E83BU+2298F 𢦏
FE 77U+E83CU+43B1
FE 78U+E83DU+43AC
FE 79U+E83EU+2EBB
FE 7AU+E83FU+43DD
FE 7BU+E840U+44D6
FE 7CU+E841U+4661
FE 7DU+E842U+464C
FE 7EU+E843U+9FB9
FE 80U+E844U+4723
FE 81U+E845U+4729
FE 82U+E846U+477C
FE 83U+E847U+478D
FE 84U+E848U+2ECA
FE 85U+E849U+4947
FE 86U+E84AU+497A
FE 87U+E84BU+497D
FE 88U+E84CU+4982
FE 89U+E84DU+4983
FE 8AU+E84EU+4985
FE 8BU+E84FU+4986
FE 8CU+E850U+499F
FE 8DU+E851U+499B
FE 8EU+E852U+49B7
FE 8FU+E853U+49B6
FE 90U+E854U+9FBA
FE 91U+E855U+241FE 𤇾
FE 92U+E856U+4CA3
FE 93U+E857U+4C9F
FE 94U+E858U+4CA0
FE 95U+E859U+4CA1
FE 96U+E85AU+4C77
FE 97U+E85BU+4CA2
FE 98U+E85CU+4D13
FE 99U+E85DU+4D14
FE 9AU+E85EU+4D15
FE 9BU+E85FU+4D16
FE 9CU+E860U+4D17
FE 9DU+E861U+4D18
FE 9EU+E862U+4D19
FE 9FU+E863U+4DAE
FE A0U+E864U+9FBB

As a national standard

The mandatory part of GB 18030-2005 consists of 1 byte and 2 byte encoding, together with 4 byte encoding for CJK Unified Ideographs Extension A. The corresponding Unicode code points of this subset, including provisional private assignments, lie entirely in the BMP.[3]:3 These parts correspond to the fully mandatory GB 18030-2000.[2]:2

Most major computer companies had already standardised on some version of Unicode as the primary format for use in their binary formats and OS calls. However, they mostly had only supported code points in the BMP originally defined in Unicode 1.0, which supported only 65,536 codepoints and was often encoded in 16 bits as UCS-2.

In a move of historic significance for software supporting Unicode, the PRC decided to mandate support of certain code points[which?] outside the BMP.[citation needed] This means that software can no longer get away with treating characters as 16 bit fixed width entities (UCS-2). Therefore, they must either process the data in a variable width format (such as UTF-8 or UTF-16), which are the most common choices, or move to a larger fixed width format (such as UCS-4 or UTF-32). Microsoft made the change from UCS-2 to UTF-16 with Windows 2000.

Mapping

GB 18030 defines a one (ASCII), two (extended GBK), or four-byte (UTF) encoding. The two-byte codes are defined in a lookup table, while the four-byte codes are defined sequentially (hence algorithmically) to fill otherwise unencoded parts in UCS. GB 18030 inherits the bad aspects of GBK, most notably needing special code to safely find ASCII characters in a GB18030 sequence.

GB 18030 encoding[3]:3[5]:252[9]
GB 18030 code points[c] Unicode
byte 1 (MSB) byte 2 byte 3 byte 4
00
7F
128
0000
007F
80
invalid[d]
81
FE
40
FE
except
7F
[e]
23940
0080
FFFF
except
D800
DFFF
[f]
81
84
30
39
81
FE
30
39
39420
85
— (12600) reserved for future character extension
86
8F
— (126000) reserved for future ideographic extension
unassigned
D800
DFFF
[g]
90
E3
30
39
81
FE
30
39
1048576 1000010FFFF
E4
FC
— (315000) reserved for future standard extension
FD
FE
— (25200) user-defined
FF
invalid
Total 1112064

The one- and two-byte code points are essentially GBK with the euro sign, PUA mappings for unassigned/user-defined points, and vertical punctuations.The four byte scheme can be thought of as consisting of two units, each of two bytes. Each unit has a similar format to a GBK two byte character but with a range of values for the second byte of 0x30–0x39 (the ASCII codes for decimal digits). The first byte has the range 0x81 to 0xFE, as before. This means that a string search routine that is safe for GBK should also be reasonably safe for GB18030 (in much the same way that a basic byte-oriented search routine is reasonably safe for EUC).

This gives a total of 1,587,600 (126×10×126×10) possible 4 byte sequences, which is easily sufficient to cover Unicode's 1,112,064 (17×65536 − 2048 surrogates) assigned, reserved, and noncharacter code points.

Unfortunately, to further complicate matters there are no simple rules to translate between a 4 byte sequence and its corresponding code point. Instead, codes are allocated sequentially (with the first byte containing the most significant part and the last the least significant part) only to Unicode code points that are not mapped in any other manner. For example:

U+00DE (Þ) → 81 30 89 37U+00DF (ß) → 81 30 89 38U+00E0 (à) → A8 A4U+00E1 (á) → A8 A2U+00E2 (â) → 81 30 89 39U+00E3 (ã) → 81 30 8A 30

An offset table is used in the WHATWG and W3C version of GB 18030 to efficiently translate code points.[10] ICU[9] and glibc use similar range definitions to avoid wasting space on large sequential blocks.

Support

Encoding

Windows 2000 can support the GB18030 encoding if GB18030 Support Package[11] is installed. Windows XP can support it natively. The open source PostgreSQL database supports GB18030 through its full support for UTF-8, i.e. by converting it to and from UTF-8. Similarly Microsoft SQL Server supports GB18030 by conversion to and from UTF-16.

More specifically, supporting the GB18030 encoding on Windows means that Code Page 54936 is supported by MultiByteToWideChar and WideCharToMultiByte. Due to the backward compatibility of the mapping, many files in GB18030 can be actually opened successfully as the legacy Code Page 936, that is GBK, even if the Code Page 54936 is not supported. However, that is only true if the file in question contains only GBK characters. Loading will fail or cause corrupted result if the file contains characters that do not exist in GBK (see § Technical details for examples).

GNU glibc's gconv, the character codec library used on most Linux distributions, supports GB 18030-2000 since 2.2,[12] and GB 18030-2005 since 2.14;[13] glibc notably includes non-PUA mappings for GB 18030-2005 in order to achieve round-trip conversion.[14] GNU libiconv, an alternative iconv implementation frequently used on non-glibc UNIX-like environments like Cygwin, supports GB 18030 since version 1.4.[15]

Glyphs

The GB18030 Support Package for Windows contains SimSun18030.ttc, a TrueType font collection file which combines two Chinese fonts, SimSun-18030 and NSimSun-18030. The SimSun 18030 font includes all the characters in Unicode 2.1 plus new characters found in the Unicode CJK Unified Ideographs Extension A block, but despite its name, it does not contain glyphs for all GB 18030 characters, as all (about a million) Unicode code points up to U+10FFFF can be encoded as GB 18030. GB 18030 compliance certification only requires correct handling and recognition of glyphs in the mandatory (two-byte) Chinese part.[2]:4

Other CJK font families like HAN NOM[16] and Hanazono Mincho[17] provide wider coverage for Unicode CJK Extension blocks than SimSun-18030 or even Simsun (Founder Extended), but they don't support all code points defined in Unicode 5.0.0 either.

See also

Notes

  1. ^ Note that GB18030 omits surrogates; see #Mapping.
  2. ^ with the exception of the euro sign which is given a single byte code of 0x80 in Microsoft's later versions of CP936/GBK and a two byte code of A2 E3 in GB18030
  3. ^ Including the 66 Unicode noncharacters
  4. ^ ICU seems to erroneously consider this code point valid, which is in neither versions of the published standards. WHATWG assigns this byte to U+20AC (GBK Euro Sign) in its general-use gbk/gb18030 decoder.
  5. ^ For a finer division of this range see GBK#Encoding.
  6. ^ Some code points are encoded with two bytes (upper row), the others with four bytes (lower row). U+FFFF is encoded as
    84 31 A4 39
    on page 239 of the 2005 standard, although the standard gives as far as
    84 39 FE 39
    for BMP mapping.
  7. ^ These are surrogate code points; they have no meaning outside of UTF-16 encoding.

References

  1. ^ Anthony Fok (2002-03-15). "Application of IANA Charset Registration for GB18030". IANA Character Set Registrations. Retrieved 2016-12-05. 
  2. ^ a b c d CESI (2009-07-08). "GB18030 符合性问与答" [GB18030 compliance FAQ]. CESI Certification Center. Archived from the original on 2016-09-28. Retrieved 2016-10-12. Page 4 同时达到以下两个要求的产品,为符合GB 18030-2005强制部分的产品:①产品可以正确输入、输出、处理GB 18030-2005强制部分规定的全部汉字字符;②产品可以正确识别GB 18030-2005强制性部分规定的全部汉字字符对应的编码。[A product compliant with the mandatory part of GB 18030 must be able to correctly a) input, output and process all Chinese characters defined in the mandatory set; b) recognize encodings for characters in the mandatory set.] 
  3. ^ a b c d e Standardization Administration of China (SAC) (2005-11-18). GB 18030-2005: Information Technology—Chinese coded character set. 
  4. ^ "Unicode FAQ on GB 18030". ICU Project. Retrieved 10 September 2016. 
  5. ^ a b Standardization Administration of China (SAC) (2000-03-17). GB 18030-2000: Information Technology—Chinese coded character set for information interchange — Extension for the basic set. 
  6. ^ a b Lunde, Ken (2006). "L2/06-394 Update on GB 18030:2005". Unicode Technical Committee Document Registry. Retrieved 28 September 2016. 
  7. ^ "Group:GBK外字". GlyphWiki. Retrieved 11 September 2016. 
  8. ^ a b Lunde, Ken (December 2008). CJKV Information Processing. O'Reilly Media, Inc. ISBN 978-0-596-51447-1. Retrieved 11 September 2016. 
  9. ^ a b Authoritative mapping table between GB18030-2000 and Unicode. ICU – International Components for Unicode. 2001-02-21. Accessed 2016-09-04.
  10. ^ "Encoding Standard # gb18030-index". WHATWG. Retrieved 2016-09-24. 
  11. ^ Microsoft. "GB18030 Support Package". Archived from the original on 2012-06-05. 
  12. ^ Drepper, Ulrich. "GB18030 iconv module for glibc.". glibc git. Retrieved 29 November 2016. 
  13. ^ Drepper, Ulrich. "Update GB18030 to 2005 version". glibc git. Retrieved 29 November 2016. 
  14. ^ Weimer, Florian; O'Donell, Carlos. "Status of GB18030 tables (#19575)". Sourceware Bugzilla. Retrieved 29 November 2016. 
  15. ^ "NEWS - libiconv.git - libiconv". git.savannah.gnu.org. Retrieved 2016-10-13. 
  16. ^ VietUnicode. "/hannom". sourceforge.net. Retrieved 2016-10-13. 
  17. ^ "Hanazono fonts". fonts.jp. Retrieved 2016-10-13. 

External links

Text is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. By using this site, you agree to the Terms of Use and Privacy Policy. Wikipedia is a registered trademark of the Wikimedia Foundation, Inc., a non-profit organization.

Also On Wow

    Advertisement

    Trending Now