Character Encodings: a Useful Overview and New Features for Software of RP Photonics

One should not confuse the issues of character encodings and fonts; nowadays, those aspects are normally handled separately. An encoding determines what kind of character (e.g. a certain letter) is associated with a certain bit sequence, while a font determines how that character looks when it is displayed on a screen or printed. By using different fonts, one can have different graphical looks of the letter “A”, for example.

For characters, the ASCII encoding was used in the early days, providing only 128 usable different characters, represented with a single byte (a set of 8 bits), where only the lower 7 bits were used. That is of course only sufficient for the simplest purposes.

Due to the need for more different characters, one soon started to use extended character ranges where the full 256 characters which can be encoded with 8 bits were utilized. However, many different versions of such ANSI character sets have been used (and are still used); some of them contain primarily things like accented letters (e.g. for the French language), while others contain letters of other languages or more symbols of certain types, e.g. box drawing symbols. Basically, the problem is that overall people in this world need far more than 256 different characters. In Windows, different ANSI character sets are characterized with so-called code pages (originally introduced by IBM). For example, in parts of Europe it is common that Windows systems use code page 1252 “Western European”. Note that for countries like Japan, where more than 256 characters are needed, 8 bits are not sufficient for representing one character.

Although the use of different ANSI code pages made possible the use of a vast variety of characters, this approach has serious limitations. In particular, when displaying e.g. the content of the text file with such an encoding, this works correctly only when the corresponding code page is known and correctly taken into account. Unfortunately, plain text files often do not contain any information on the used code page, so that additional information (e.g. manually transmitted by the user) is often needed to display such files properly; if such information is available, it may or may not be correct. In addition, one can of course not convert files from one code page to another without information loss, unless the file contains only characters which occur in both code pages.

In order to solve such problems, the Unicode system (i.e., Unicode character definitions in conjunction with various encoding systems) has been developed. Here, a huge number of different characters (each one represented by a so-called code point) can be encoded – essentially any characters which one would normally require.

Of course, one byte (8 bits) is not sufficient for encoding an arbitrary Unicode character. Now there are a couple of different encoding schemes for handling Unicode characters in computer memory or files. A common scheme is UTF-16, where 16 bits (2 bytes) are normally used for one character, but for some characters one in fact uses 2 of those items, i.e., 32 bits in total, called surrogate pairs. This scheme is quite practical for use in CPUs and main memories. Another common encoding scheme for Unicode characters is UTF-8, where the most common characters are encoded with a single bytes (8 bits), most other characters need two bytes, and some need even three or four bytes.

The content of a computer file, for example, can only be correctly handled if the used encoding (e.g. ANSI or Unicode, and which particular scheme) is known. Unfortunately, that information is often lost or corrupted e.g. by wrong assumptions on an encoding. This problem has been solved for many Unicode files by writing a so-called byte order mark (BOM), consisting of a few bytes, at the beginning of a file. A program finding such a BOM in a text file can use it to determine (a) that it is a Unicode file and (b) what UTF encoding exactly has been used. However, it is common e.g. for web pages that one uses Unicode by without a BOM; the used encoding is indicated elsewhere (in an HTTP header transmitted before the page content). It is then often difficult or impossible for software to determine what encoding has been used for some file; asking the user may be impractical or not work because the user doesn't know that either.

Of course, an encoding which does not use a fixed number of bytes for one character implies some technical difficulties. For example, in order to find the 23rd character in such a byte sequence, one has to scan it from the beginning in order to properly take into account the number of bytes for each previous character. Due to such difficulties, it is often not easy to modify old software such that it can handle Unicode data. That problem can in principle be solved by using UTF-32 encoding, where each character is represented by 32 bits (4 bytes), but that is rarely used because it does not present a memory-efficient way for storing text.