www.sbin.org

Xlib Programming Manual (O'Reilly & Associates, Inc.)

Internationalization

An internationalized application is one that runs, without changes to the binary, in any given "locale." Among other things, this means that a program must display all text in the user's language, accept input of all text in that same language, and display times, dates, and numbers in the user's accustomed format.

The internationalization of terminal-based programs is a problem that has been satisfactorily solved where terminals exist that can display and accept input for a particular language. The ANSI-C library contains mechanisms for this terminal-based internationalization, and R5 internationalization is based on these mechanisms. This chapter begins with a detailed overview of the goals, concepts, and techniques of internationalization, starting with ANSI-C internationalization and progressing to the new R5 internationalization features. After the overview, each section covers an individual topic in X internationalization. Internationalized text input with R5 is a large subject and is given its own chapter following this one.

Internationalization is implemented with a separate set of functions for handling keyboard input and drawing text, that are new in Release 5. All the input and drawing techniques shown in previous chapters continue to work, but they do not support internationalization. So it is up to you which set of functions to use depending on your needs.

Also note that the internationalization features of R5 are not self contained, and therefore may not work on all systems. If you do not have the ANSI-C internationalization features, you may be able to make do with alternatives provided by Xlib and by contributed libraries, but these have not been thoroughly tested and you may encounter difficulties. In ANSI-C internationalization, the C library reads a "localization database" customized for each locale. Many systems (systems sold in the U.S., at least) support ANSI-C internationalization, but do not ship databases for any but a default locale.

One more warning and disclaimer is required. These internationalization features are new in Release 5, and therefore there is no experience in their use. So the coverage in this book probably does not yet answer every question you might have, nor present a foolproof procedure for writing an internationalized application. We hope to add more practical instructions once we know better what to tell you.

A final point of terminology: the word "internationalization" contains 20 letters. In the MIT X documentation and elsewhere, you may find it abbreviated as i18n--the letter "i" followed by 18 letters and the letter "n."

An Overview of Internationalization

• An internationalized application must display all text in the user's native or preferred language. This includes prompts, error messages, and text displayed by buttons, menus, and other widgets. The obvious approach to this sort of internationalization is to remove all strings that will be displayed from the source code of the application and put them instead in a file that will be read in when the application starts up. Then it is a relatively simple matter to translate the file of strings to other languages and have the application read the appropriate one at startup. Many X applications that use the X resource manager to provide an app-defaults file are already internationalized in this way, though some still have non-internationalized error messages. Another approach to the internationalization of strings is the message catalog facility defined by the X/Open Portability Guide, Issue 3 (often known as XPG3).  The three functions catopen, catgets, and catclose, provide a simple mechanism for retrieving numbered strings from a plain text file. These functions are available on some systems, but are not part of any formal standard, and are not universally available.
• An internationalized application must display times, dates, numbers, etc. in the format that the user is accustomed to. Where an American user sees a date in the form month/day/year, an English user should see day/month/year, and a German user should see day.month.year. And where an American user sees the number 1,234.56, a French user should see 1.234,56. The definition of "alphabetical order" is a similar customary usage that varies from country to country. In Spain, for example, the string "ch" is treated as a single letter that comes after "c." So while the strings "Chile" and "Colombia" are in alphabetical order for an American user, they are out of order for a Spanish user. These and related problems of local customs are resolved with the ANSI-C setlocale mechanism. Calling this function causes the ANSI-C library to read a database of localization information. Other functions in the C library (such as printf for displaying numbers and strcoll for comparing strings) use the information in this database so that they can behave correctly in the current locale. The R5 internationalization mechanisms are built upon this setlocale mechanism. It is described in more detail in the next section.
• An internationalized program must be capable of displaying all the characters used in the user's language, and must allow the user to generate all these characters as input. For terminal-based applications, this can be thought of as a hardware issue: a French user's terminal must be capable of displaying the accented characters used in French, and there must be some way to generate those characters from the keyboard. With X and bit-mapped displays, character display is not a problem--simply a matter of finding the required font or fonts. For languages like Chinese, fonts with many characters are required, but X supports 16-bit fonts, which is large enough for almost all languages. Keyboard input for Chinese and other ideographic Asian languages is another matter, however. When there are more characters in a language than there are keys on a keyboard, some sort of "input method" is required for converting multiple keystrokes into a single character. Ideographic languages require complex input methods, and often there is more than one standard method for a language. An internationalized application must support any input method chosen by the user. R5 provides this capability; it is described in Chapter 10, "Internationalization."
• An internationalized program must operate regardless of the encoding of characters in the user's language. A program (or operating system) that ignores or truncates the eightth bit of every character won't work in Europe, because the accented characters used in many European languages are represented with numbers greater than 127. An application that assumes that every character is 8 bits long won't work in Japan where there are many thousands of ideographic characters. Furthermore, common Japanese usage intermixes 16-bit Japanese characters with 8-bit Latin characters, so it is not even safe to assume that characters are of a uniform width. When internationalizing an application, two areas of particular difficulty are string manipulation (how, for example, can you iterate through the characters of a string when those characters have differing widths) and text input and output. (How, for example, do you display a Japanese string that contains characters from different fonts?)One approach to the encoding problem is to side-step it by defining a universal encoding used everywhere. The Latin-1 encoding is suitable for English and most western European languages, and this shared encoding dramatically simplifies the problem of porting applications to work in many European countries. But this approach does not work outside of Europe, and while ANSI-C provides some rudimentary internationalized string manipulation functions, it leaves issues of text input and output to the terminal hardware or terminal driver software. It is here that R5 makes its real contribution to internationalization--in an extension to the setlocale model, an internationalized X application reads a localization file at startup that contains information about the text encoding used in the locale. This information allows X to correctly parse strings into characters and figure out how to display them. There are a number of issues surrounding character encoding in internationalized applications, and it is possible to explore them in full and confusing detail. In practice, though, most of the string encoding details are hidden by the operating system, or with X internationalization, by Xlib. Section 10.1.2, "Text Representation in an Internationalized Application" explains some of the basics of text encoding in more detail.

The following sections continue this introduction to internationalization with a description of the ANSI-C setlocale mechanism and a further discussion of character encoding and text representation issues.

Internationalization with ANSI-C

The first step in any internationalized application is to establish the locale--to cause the localization database to be read in. This is done with the C library function setlocale. It takes two arguments: a locale category and the locale name. The locale name specifies the database that should be used to localize the program, and the locale category specifies which behaviors (for example, the collation sequence of the alphabet or the formatting of times and dates) of the program should be changed. setlocale will most often be used as shown below:

setlocale(LC_ALL, "");

language[_territory[.codeset]]

The category LC_ALL instructs setlocale to set all internationalization behavior defined by ANSI-C to operate in the given locale. The locale may also be specified for each category individually. The standard categories (other, non-standard, categories may also be defined) and the aspects of program behavior that they control are listed below:

LC_COLLATE

This category defines the collation sequence used by the ANSI-C library functions strcoll and strxfrm which are used to order strings alphabetically.

LC_CTYPE

This category defines the behavior of the character classification and case conversion macros (such as isspace and tolower) defined in the header file <ctype.h>. Different languages will have different classifications for characters. Not all characters have uppercase equivalents, for example, and characters with codes between 128 and 255 which are non-printing in ASCII are important alphabetic characters in many European languages.

LC_MONETARY

This category does not affect the behavior of any C library functions. The problem of formatting monetary quantities was deemed too intricate for any standard library function, so the library simply provides a way for an application to look up any of the localized parameters it needs to do its own formatting of monetary quantities. The ANSI-C function localeconv returns a pointer to a structure of type lconv that contains the parameters (such as decimal separator, currency symbol, and flags that indicate whether the currency symbol should appear before or after positive and negative quantities, etc.) needed for numeric and monetary formatting in the current locale.

LC_NUMERIC

This category affects the decimal separator used by printf (and its variants), scanf (and its variants), gcvt (and related functions), strtod, and atof. It also affects the values in the lconv structure returned by localeconv.

LC_TIME

This category affects the behavior of the time and date formatting functions strftime and strptime. It defines such things as the names of the days of the week and their standard abbreviations in the language of the locale.

Text Representation in an Internationalized Application

We've been using the term encoding rather loosely. Before we consider text representation any further, some definitions are appropriate. A character is an abstract element of text, distinct from a font glyph, which is the actual image that gets displayed. A character set is simply a set of characters; there are no numbers associated with those characters. We are all familiar with the character set used by ASCII. The Latin-1 character set used by many Western European Latin-based languages is an extension of ASCII that contains the accented characters required by many of those languages. An encoding is any numeric representation of the characters in a character set. The term codeset is sometimes used as a synonym for encoding. A charset (not the same as a character set) is an encoding in which all characters have the same number of bits. ASCII is a 7-bit encoding, for example, and is therefore a charset. Figure 10-1 diagrams the relationship between character sets, charsets, fonts, and font glyphs.

Character sets, encodings, charsets, fonts, and glyphs

 The last two fields of an X font name specify a charset. By definition, the index of a font glyph in the font is the same as the encoding of the corresponding character in that charset. When the encoding of a locale is a charset, this obviously simplifies matters a great deal: text in the locale can be displayed using glyphs from a single font, and the character encoding can be used directly as the index of the corresponding font glyph.

Not all languages can be represented with a single charset, however. Japanese text, for example, commonly requires Japanese ideographic characters, Japanese phonetic characters, and Latin characters. Each of these character sets has its own standard fixed-width encoding, and is therefore a charset. Note, however, that the ideographic charset is 16-bits wide while the phonetic and Latin charsets are 8-bits wide. Full Japanese text display requires a font for each charset, and Japanese text representation requires a "super-encoding" that combines each of the component encodings. There are, in fact, several encodings commonly used for Japanese text. What they have in common is the use of "shift sequences" to indicate which charset the following character belongs to.

It is crucial to the concept of a locale that each locale has a single well-defined encoding. Many languages have only a single standardized encoding. If a language can be encoded in more than one standard way, each encoding defines a locale of its own, and the name of the encoding is part of the name of the locale.

ISO8859-1 and Other Encodings

ISO8859-1 contains a superset of the ASCII characters. Every character in the ASCII character set has the same encoding in Latin-1 as it does in ASCII. (But Latin-1 does not define any control characters such as linefeed, backspace or the bell character.) Because it is an 8-bit encoding, Latin-1 strings can be represented using the usual C null-terminated array of char. Because the characters are a uniform 8 bits and because strings do not contain embedded shift states, it is possible to use Latin-1 strings with the standard C string manipulation routines (strlen, strcat, etc.) In conjunction with the ANSI-C internationalization facilities, the careful design of ISO8859-1 means that most programs originally written for ASCII use can easily be ported for use in most Western European countries.

But it is not so simple once we try to go beyond Western Europe and Latin-based alphabets. Japanese text, for example, commonly uses (at least within the computer industry) words written in the Latin alphabet along with phonetic characters from the katakana and hiragana alphabets and ideographic kanji characters. Each of these types of text has its own charset (8- or 16-bit), but they must be combined into a single encoding for Japanese text. This is done with shift sequences, bytes embedded in the running text which control the character set in which the following character will be interpreted. It is possible to use "locking shifts" which modify the interpretation of the next and subsequent characters, but this scheme is infrequently used because it makes strings of text very difficult to manipulate.

Compound Text is another text representation that is used in X applications. Compound Text strings identify their encoding using embedded escape sequences (they can also have multiple sub-strings with multiple encodings) and are therefore locale-independent. The Compound Text representation was standardized as part of X11R4 for use as a text interchange format for interclient communication. It is often used to encode text properties and for the transfer of text via selections, and is not intended for text representation internal to an application. There are new R5 routines that convert X property values to and from the Compound Text representation. Note that Compound Text is not the same thing as the Compound Strings used by the Motif widget set.

Multi-byte Strings and Wide-character Strings

As an alternative to these multi-byte strings, ANSI-C defines a wide-character type, wchar_t, in which each character has a fixed size and occupies one array element in the string. (The wchar_t is 2 bytes on some systems, 4 bytes on others, and may be 1 byte on systems that support nothing but the default C locale.) ANSI-C defines functions to convert between multi-byte and wide-character strings: mblen, mbstowcs, mbtowc, wcstombs, and wctomb.  As you can see here, and as you will see with the R5 internationalized text input and output functions, "multi-byte" is commonly abbreviated "mb" in function names, and "wide character" is abbreviated "wc." Multi-byte strings are usually more compact than wide-character strings, but wide-character strings are easier to work with. Note that ANSI-C does not provide wide-character string manipulation functions. There is, however, a contributed library of wide character functions that is shipped with the MIT R5 release; see the directory contrib/lib/Xwchar.

In an internationalized application, you must take care to handle all strings properly. Unfortunately the ANSI-C library does not provide adequate functions or conventions for sophisticated internationalized text manipulation. Note, though, that many applications can do internationalized text input and output without performing any manipulations on that text. The following list gives a few guidelines for handling internationalized strings:

• Multi-byte strings are null-terminated. There is no single convention for the termination of wide character strings, but strings passed to wcstombs are null-terminated. As was the case before R5, X text output and input functions take and return strings with a count of the characters they contain.
• If an encoding is state-dependent (i.e., if it uses locking shifts) multi-byte strings are assumed to begin in the default shift state of the encoding. There is no convention for the shift state at the end of a string, so when concatenating two strings, the first may need to be reset to the default shift state in order to guarantee correct interpretation of the second. In practice, state-dependent encodings are rarely used.
• None of the C library string-handling functions work with wide-character strings.
• The following C string-handling functions may be safely used with multi-byte strings (in a state-independent encoding): strcat, strcmp, strcpy, strlen, strncmp. Note that the string comparison routines are only useful to check for byte-for-byte equality. To compare strings for sorting, use strcoll.
• Multi-byte strings can be written to file or output streams. Assuming a terminal that operates in the current locale, printing a multi-byte string to stdout or stderr will cause the correct text to be displayed.
• Multi-byte strings can be read from files or from the stdin input stream. If the file is encoded in the current locale, or the terminal operates in the locale, then the strings that are read will be meaningful.

Internationalization Using X

The sections below cover these topics as follows:

• Section 10.2, "Locale Management in X" describes the X locale management functions.
• Section 10.3, "Internationalized Text Output in X" describes internationalized text output with R5.
• Section 10.4, "String Encoding Changes for Internationalization" describes string encoding changes in various Xlib functions.
• Section 10.5, "Internationalized Interclient Communication" describes interclient communication using internationalized properties and interlocale string conversions.
• Section 10.6, "Localization of Resource Databases" describes the localization of resource databases.
• Section 10.7, "Summary: Writing an Internationalized Application" describes changes to the X Toolkit to support internationalization.

Locale Management in X

Immediately after calling setlocale, an application should call XSupportsLocale() to determine if the Xlib implementation supports the current locale. This function takes no arguments and return a Bool. If this function returns False, an application will typically print a "Locale not supported" message and exit.

After verifying that the locale is supported, an application should call XSetLocaleModifiers(). A "locale modifier" can be thought of as an extension to the name of a locale; it specifies more information about the desired localized behavior of an application. R5 as shipped by MIT recognizes one locale modifier, used to specify the input method (see Chapter 10, "Internationalization") to be used for internationalized text input for the locale.

XSetLocaleModifiers() allows the programmer to specify a list of modifiers (usually none) which will be concatenated with a list of user-specified modifiers from an operating system environment variable (XMODIFIERS in POSIX). The strings passed to XSetLocaleModifiers() and set in the XMODIFIERS environment variable are a series of concatenated "@category=value" strings. Thus to specify that the "Xwnmo" input method should be used by an application, a user might set the XMODIFIERS as follows:

setenv XMODIFIERS @im=_XWNMO

Establishing the locale of an X application

#include <stdio.h>
#include <X11/Xlib.h>
/*
 * include <locale.h> or the non-standard X substitutes
 * depending on the X_LOCALE compilation flag
 */
#include <X11/Xlocale.h>
main(argc, argv)
int argc;
char *argv[];
{
    char *program_name = argv[0];
    /*
     * The error messages in this program are all in English.
     * In a truly internationalized program, they would not be
     * hardcoded; they would be looked up in a database of some sort.
     */
    if (setlocale(LC_ALL, "") == NULL) {
        (void) fprintf(stderr, "%s: cannot set locale., program_name);
        exit(1);
    }
    if (!XSupportsLocale()) {
        (void) fprintf(stderr, "%s: X does not support locale %s.,
                       program_name, setlocale(LC_ALL, NULL));
        exit(1);
    }
    if (XSetLocaleModifiers("") == NULL) {
        (void) fprintf(stderr, "%s: Warning: cannot set locale modifiers.,
                       program_name);
    }
        .
        .
        .
}

Internationalized Text Output in X

R5 bases its new text output routines on a new Xlib abstraction, the XFontSet. An XFontSet is bound to the locale in which it is created, and contains all the fonts needed to display text in that locale, or all the independent charsets used in the encoding of that locale. Technical Japanese text, for example, often mixes Latin with Japanese characters, so for a Japanese locale, fonts might be required with the charsets jisx0208.1983-0 for Kanji ideographic characters, jisx0201.1976-0 for Kana phonetic characters, and iso8859-1 for Latin characters.

Drawing internationalized text in R5 is conceptually very similar to drawing text in X11R4--there are routines that allow you to query font metrics, measure strings, and draw strings. The new R5 functions use an XFontSet rather than an XFontStruct or a font specified in a graphics context. The drawing and measuring routines interpret text in the encoding of the locale of the fontset, and correctly map wide or multi-byte characters to the corresponding font glyph (or glyphs).

Creating and Manipulating Fontsets

XCreateFontSet() returns a list of the charsets for which no font could be found, and a default string that will be drawn in place of characters from the missing charset or charsets. The list of missing charsets should be freed with a call to XFreeStringList(). The returned default string should not be freed by the programmer. Example 10-2 shows how to create an XFontSet.

Creating an XFontSet

XFontSet fontset;
char **missing_charsets;
int num_missing_charsets = 0;
char *default_string;
int i;
        .
        .
        .
fontset = XCreateFontSet(dpy,
                         "-misc-fixed-*-*-*-*-*-130-75-75-*-*-*-*",
                         &missing_charsets, &num_missing_charsets,
                         &default_string);
/*
 * if there are charsets for which no fonts can
 * be found, print a warning message.
 */
if (num_missing_charsets > 0) {
    (void)fprintf(stderr, "%s: The following charsets are missing:,
                  program_name);
    for(i=0; i < num_missing_charsets; i++)
        (void)fprintf(stderr, "%s: %s, program_name,
                      missing_charsets[i]);
    (void)fprintf(stderr, "%s: The string %s will be used in place,
                  program_name, default_string);
    (void)fprintf(stderr, "%s: of any characters from those sets.,
                  program_name);
    XFreeStringList(missing_charsets);
}
        .
        .
        .

The following routines also use or operate on font sets:

XFreeFontSet()

Frees an XFontSet and all information associated with it.

XFontsOfFontSet()

Returns the list of XFontStructs and font names associated with an XFontSet.

XBaseFontNameListOfFontSet()

Returns a string containing the comma-separated base font name list for the given FontSet.

XLocaleOfFontSet()

Returns the name of the locale of the specified XFontSet.

Querying Fontset Metrics

The XFontSetExtents() structure

typedef struct {
    XRectangle max_ink_extents;          /* over all drawable characters */
    XRectangle max_logical_extents;      /* over all drawable characters */
} XFontSetExtents;

Context Dependencies in Displayed Text

The function XContextDependentDrawing() returns True if the locale associated with a font set includes context dependencies in text drawing. An internationalized application could use this function to check if it can take the various shortcuts allowed in non-context dependent locales. If XSupportsLocale() returns True, then any context dependencies in the text of a locale are correctly handled by the text-measuring and text-displaying routines described below.

There is another, more difficult, kind of context dependency in languages such as Hebrew and Arabic which are drawn right-to-left except for numbers which are drawn left-to-right. In this case it is not valid to assume that characters that are adjacent in a string will be adjacent when displayed. R5 does not make any provisions for handling this sort of text with mixed drawing directions.

Measuring Strings

Xmb/XwcTextEscapement()

Return the number of pixels the given string would require in the x dimension if drawn.

Xmb/XwcTextExtents()

Return the text escapement as the value of the function, and also return a bounding box for all the ink in the string, and a bounding box for all the ink plus intercharacter and interline spacing.

There is another pair of text extent functions that are useful when there are context dependencies in the displayed text. Xmb/XwcTextPerCharExtents() return the escapement and extents of a string as the above functions do, but also return the ink extents and the logical extents of each character in the string. These extents are measured relative to the drawing origin of the string, not the origin of the particular glyph. Note that these extents are returned for each character of the string, not for each font glyph displayed. If a sequence of characters map to a single glyph, each of those characters will have identical extent rectangles. Similarly if a single character requires several font glyphs to display, its extents will be the combined extents of those glyphs. The dimensions of the rectangle are independent of the drawing direction of the character.

Example 10-4 in the next section shows a use of XmbTextExtents() and XmbTextPerCharExtents().

Drawing Internationalized Text

Xmb/XwcDrawString()

Draw the specified string. The foreground pixels of each font glyph are drawn, but the background pixels of each glyph are not.

Xmb/XwcDrawImageString()

Draw the specified string. Both the foreground and background pixels of each glyph are drawn.

Xmb/XwcDrawText()

Draw text with complex spacing or font set changes. These routines draw text described in an array of XmbTextItem or XwcTextItem structures. These structures are shown in Example 10-4.

The XmbTextItem() and XwcTextItem() structures

typedef struct {
    char        *chars;                 /* pointer to string */
    int         nchars;                 /* number of bytes in string */
    int         delta;                  /* pixel delta between strings */
    XFontSet    font_set;               /* fonts, None means don't change */
} XmbTextItem;
typedef struct {
    wchar_t     *chars;                 /* pointer to wide char string */
    int         nchars;                 /* number of wide characters */
    int         delta;                  /* pixel delta between strings */
    XFontSet    font_set;               /* fonts, None means don't change */
} XwcTextItem;

Centering and drawing a multi-byte string

#include <X11/Xlib.h>
/*
 * This function draws a specified multi-byte string centered in
 * a specified region of a window.
 */
void DrawCenteredMbString(dpy, w, fontset, gc,
                          str, num_bytes, x, y, width, height)
Display *dpy;
Window w;
XFontSet fontset;
GC gc;
char *str;
int num_bytes;
int x, y, width, height;
{
    XRectangle boundingbox;
    XRectangle dummy;
    int originx, originy;
   /*
    * Figure out how big the string will be.
    * We should be able to pass NULL instead of &dummy, but
    * XmbTextExtents is buggy in the Xsi implementation.
    * Also, it should return the escapement of the string, but doesn't.
    */
    (void) XmbTextExtents(fontset, str, num_bytes,
                          &dummy, &boundingbox);
   /*
    * The string we want to center may be drawn left-to-right,
    * right-to-left, or some of both, so computing the
    * drawing origin is a little tricky.  The bounding box's x
    * and y coordinates are the upper left hand corner and are
    * relative to the drawing origin.
    * if boundingbox.x is 0, the string is pure left-to-right.
    * If it is equal to -boundingbox.width then the string is pure
    * right-to-left, but it may not be either of these, so what
    * we've got to do is choose the origin so that the bounding box
    * is centered in the window without assuming that the origin is
    * at one end or another of the string.
    */
    originx = x + (width - boundingbox.width)/2 - boundingbox.x;
    originy = y + (height - boundingbox.height)/2 - boundingbox.y;
   /*
    * now draw the string
    */
    XmbDrawImageString(dpy, w, fontset, gc,
                       originx, originy,
                       str, num_bytes);
}

String Encoding Changes for Internationalization

Some of the issues that must be considered are the appropriate encoding for color and font names passed to the X server, the encoding of bitmap files, the encoding of strings selected in one client and copied to another, and the encoding of resource values and names. When making decisions on questions like these, the designers of X internationalization had several choices. They could specify that particular strings were:

• In the encoding of the locale.
• In the COMPOUND_TEXT encoding, in which each string is encoded along with the name of its encoding.
• In the STRING encoding, which is Latin-1 plus the newline and tab control characters.
• In ASCII, which as the encoding of the C language, is actually fairly portable.
• In an implementation-dependent encoding.
• Not in any encoding, and are simply interpreted as a sequence of bytes.

Note that the above list refers to the COMPOUND_TEXT and STRING encodings. These capitalized names refer to the Atom names used in the ICCCM to specify the type of a "Property." The ICCCM also specifies a selection conversion target Atom, TEXT, which simply means a string in whatever encoding is convenient for the selection owner.

Sometimes the best choice of encodings is ASCII. It may seem unfair to non-English locales that the ASCII encoding should be singled out for special treatment, but for strings that are to be shared between X client and X server (such as Display, Property, and font and color names) some standard encoding must be specified. Because ASCII is widespread and is the usual encoding for C programming, it is a natural choice. In many cases, though, it is not the specific ASCII encoding that is important, but the fact that there is some common encoding for all the characters used by ASCII. R5 never actually refers to ASCII. Instead, it defines the X Portable Character Set as a set of basic characters that must exist in all locales supported by Xlib. Those characters are:

a..z A..Z 0..9
!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
<space>, <tab>, and <newline>

String-encoding issues arise throughout Xlib, and particularly so for functions that involve X properties and resource databases. The internationalization of client-to-window-manager and client-to-client communication via properties is described in 10.5 below and the internationalization of X resource databases is discussed in 10.6. Here we itemize the remaining changes to the Xlib specification that involve string encodings. Table 10-1 lists Xlib functions and the encodings of the strings that are passed in and out of them. These are not so much changes to the Xlib specification as clarifications of it to make the encodings explicit.
 

Table 10-1 : String Encodings Used by Various Xlib Functions

Function	String Encoding
XDrawImageString() XDrawString() XQueryTextExtents() XTextExtents() XTextWidth() XTextItem structureXChar2b structure	No encoding; "characters" are treated as glyph indexes into the font, independent of locale.
XServerVendor() ServerVendor() macro	If the X server uses the Latin Portable Character Encoding, this function will return a string in the Host Portable Character Encoding; otherwise the encoding is implementation-dependent.
XOpenDisplay() XDisplayName() DisplayName() macro XDisplayString() DisplayString() macro	Display names in the Host Portable Character Encoding are supported; additional encodings are implementation dependent.
XAllocNamedColor() XLookupColor() XStoreNamedColor() XParseColor()	Color names in the Host Portable Character Encoding are supported; Xlib implementations may support additional encodings, and may look up color names in locale-specific databases before passing them to the server.
XLoadFont() XLoadQueryFont()	Font names in the Host Portable Character Encoding are supported; implementations may support additional encodings.
XListFonts() XListFontsWithInfo()	Font patterns in the Host Portable Character Encoding are supported; implementations may support additional encodings. Returned strings are in the Host Portable Character Encoding if the server returns strings in the Latin Portable Character Encoding; otherwise the encoding is implementation-dependent.
XSetFontPath() XGetFontPath()	The encoding and interpretation of the font path is implementation-dependent.
XParseGeometry() XGeometry() XWMGeometry()	Geometry strings in the Host Portable Character Encoding are supported; implementations may support additional encodings.
XInternAtom()	Atom names in the Host Portable Character Encoding are supported; implementations may support additional encodings.
XGetAtomName()	The returned atom name is in the Host Portable Character Encoding if the server returns a value in the Latin Portable Character Encoding.
XStringToKeysym()	Keysym names in the Host Portable Character Encoding are supported; implementations may support additional encodings.
XKeysymToString()	The returned string is in the Host Portable Character Encoding.
XInitExtension() XQueryExtension()	Extension names in the Host Portable Character Encoding are supported; implementations may support additional encodings.
XListExtensions()	The returned strings are in the Host Portable Character Encoding if the server returns strings in the Latin Portable Character Encoding.
XReadBitmapFile()	The bitmap file is parsed in the encoding of the current locale.
XWriteBitmapFile()	The file is written in the encoding of the current locale.
XFetchBytes() XFetchBuffer() XStoreBytes() XStoreBuffer()	No encoding; data in cut buffers is treated as uninterpreted bytes.
XGetErrorDatabaseText()	Name and message arguments in the Host Portable Character Encoding are supported; implementations may support additional encodings. The default_string argument is encoded in the current locale, and the returned text is also in encoded in the current locale.
XGetErrorText()	The returned text is in the current locale.
XSetWMProperties() XSetStandardProperties() XStoreName() XSetIconName() XSetCommandP() XSetClassHint()	Strings in the Host Portable Character Encoding are supported; implementations may support additional encodings. The strings are set as the values of a property of type STRING.
XFetchName() XGetIconName() XGetCommand() XGetClassHint()	Returned strings are in the Host Portable Character Encoding if the data returned by the server is in the Latin Portable Character Encoding.

 

Internationalized Interclient Communication

When writing an internationalized application it is not safe to assume that all interclient communication with text properties will be done with Latin-1 or ASCII strings. R5 provides some new functions that do not make this assumption. The first is a convenience routine for communication with window managers. XmbSetWMProperties() is a function very similar to XSetWMProperties(), except that the window_name and icon_name arguments are multi-byte strings (rather than XTextProperty pointers) in the encoding of the locale. If these strings can be converted to the STRING encoding (Latin-1 plus newline and tab), then their corresponding WM_NAME and WM_ICON_NAME properties are created with type STRING. If this conversion cannot be performed, the strings are converted to Compound Text (this conversion can always be done, by the definition of Compound Text), and the properties are created with type COMPOUND_TEXT. Note that there is no wide-character version of this function.

Since X properties have a single contiguous block of data as their value, they cannot directly represent types such as char **. But sometimes such a complex type must be represented (imagine a text editor setting a property to a set of disjointed selected strings). To allow this, X11R4 defined the XTextProperty structure (shown in Example 10-6) and the functions XStringListToTextProperty() and XTextPropertyToStringList().

The XTextProperty structure

typedef struct {
        unsigned char *value;   /* property data */
        Atom encoding;          /* type of property */
        int format;             /* 8, 16, or 32 */
        unsigned long nitems;   /* number of items in value */
} XTextProperty;

The XICCEncodingStyle type

typedef enum {
        XStringStyle,           /* STRING */
        XCompoundTextStyle,     /* COMPOUND_TEXT */
        XTextStyle,             /* text in owner's encoding (current locale) */
        XStdICCTextStyle        /* STRING, else COMPOUND_TEXT */
} XICCEncodingStyle;

• XStringStyle specifies that the text should be converted to the STRING encoding, and the encoding field of the returned XTextProperty should be set to the Atom STRING. Note that text cannot always be converted to this type without loss of data--only characters that are in the Latin-1 character set will be convertible.
• XCompoundTextStyle specifies that the text should be converted to the Compound Text encoding and the encoding field of the returned XTextProperty should be set to the Atom COMPOUND_TEXT.
• XTextStyle specifies that the text should be left unconverted in the encoding of the current locale. The encoding field of the returned XTextProperty structure is set to an Atom which names that encoding.
• XStdICCTextStyle specifies that the text should be converted to STRING if that conversion is possible and otherwise it should be converted to Compound Text. The encoding field of the returned XTextProperty will be set to the Atom STRING or COMPOUND_TEXT depending on which conversion was performed.

The other two routines, Xmb/XwcTextPropertyToTextList(), perform the conversion in the opposite direction. They are passed an XTextProperty (obtained with a call to XGetTextProperty(), perhaps) and return an array of pointers to char* or an array of pointers to wchar_t*. These routines do not require an argument of type XICCEncodingStyle; they always convert from the encoding of the property to the encoding of the current locale if such a conversion is possible. The application is responsible for freeing the memory allocated by these functions. To free the array of multi-byte strings (and the strings themselves) returned by XmbTextPropertyToTextList() use XFreeStringList(), which is a pre-R5 function. To free the array of wide-character strings (and the strings themselves) allocated by XwcTextPropertyToTextList() use the new function XwcFreeStringList().

These four functions return an integer. The possible values and their meanings are as follows:

Success

The conversion is completely successful; all characters were converted.

XNoMemory

There was not enough memory available to perform the conversion.

XLocaleNotSupported

The current locale is not supported. By definition, no conversions are possible to or from the encoding of an unsupported locale. This error code will never be returned if XSupportsLocale() has returned True for the current locale.

XConverterNotFound

No converter could be found between the encoding of the text property and the current locale. There is always a converter for converting between STRING and COMPOUND_TEXT and encoding of the current locale (if that locale is supported, of course), so Xmb/wcTextListToTextProperty() never returns this error code, and Xmb/XwcTextPropertyToTextList() will never return it if the text property is in the STRING or COMPOUND_TEXT encodings.

any value > 0

There were unconvertible characters in the string, and the return value indicates how many. Even when the current locale is supported, and an appropriate converter is found, it is by no means guaranteed that all the characters of the string can be converted. If two locales use the same character set but simply encode those characters differently, then strings will be fully convertible between the locales. But imagine trying to convert from French text to ASCII--any accented characters would be unconvertible because they simply do not exist in the ASCII character set. When converting between languages as dissimilar as Arabic and Korean, for example, there will be no convertible characters.  Note that the return value Success has a value of 0, and the other return values, XNoMemory, XLocaleNotSupported, and XConverterNotFound all have negative values. Therefore any positive return value indicates unconvertible characters.

                Xmb/XwcTextListToTextProperty()

                Xmb/XwcTextPropertyToTextList()

When there are unconvertible characters in a string, the conversion functions substitute a locale-dependent default string (encoded in the current locale). The value of the default string may be queried with XDefaultString(), and may be the empty string (""). There is no way to set the value of the default string. The default string is independent of the default string used by the R5 text-drawing routines when an XFontSet does not contain all the characters needed to represent text in a locale.

Localization of Resource Databases

When a resource file or string are parsed into an XrmDatabase(), that parsing is done in the current locale, and the database is bound to that locale even if the current locale changes. We can speak of the "locale of the database" in the same way that we speak of the "locale of the XFontSet." To determine the locale of a database, call XrmLocaleOfDatabase().

The internationalization of resources requires additions to the Xlib specification to make explicit the encoding and interpretation of the strings that are passed in and out of the Xrm functions. Table 10-4 lists the resource manager functions that have been respecified.
 

Table 10-4 : String Encoding and Locale Changes to Xrm Functions

Function	String Encoding and Locale Changes
XrmStringToQuark()XrmStringToQuarkList()XrmStringToBindingQuarkList()	Quark names in the Host Portable Character Encoding are supported; implementations may support additional encodings.
XrmQuarkToString()	No specified encoding; the returned string is equal byte-for-byte to the string originally passed to one of the string-to-quark routines.
XrmGetFileDatabase()	The file is parsed in the current locale.
XrmGetStringDatabase()	The string is parsed in the current locale.
XrmPutLineResource()	The line is parsed in the locale of the database. The resource name part of the line and the colon are in the Host Portable Character Encoding or some implementation-dependent encoding.
XrmPutFileDatabase()	The resource file is written in the locale of the database. Resource names in the Host Portable Character Encoding, and resource values in the encoding of the locale of the database are supported; implementations may support additional encodings.
XrmPutResource()	Resource specifiers and types in the Host Portable Character Encoding are supported; implementations may support additional encodings. The resource value is stored as uninterpreted bytes.
XrmQPutResource()	The resource value is stored as uninterpreted bytes.
XrmPutStringResource()	Resource specifiers in the Host Portable Character Encoding are supported; implementations may support additional encodings. The resource value is stored as uninterpreted bytes. The resource type is set to the quark for the string "String" encoded in the Host Portable Character Encoding.
XrmQPutStringResource()	The resource value is stored as uninterpreted bytes. The resource type is set to the quark for the string "String" encoded in the Host Portable Character Encoding.
XrmGetResource()	Resource names and classes in the Host Portable Character Encoding are supported; implementations may support additional encodings.
XrmMergeDatabases()	The database values and types are merged as uninterpreted bytes regardless of the locales of the databases. The locale of the target database is not changed.
XResourceManagerString()	The RESOURCE_MANAGER property is converted from STRING encoding to the encoding of the current locale in the same way that XmbTextPropertyToTextString performs conversions.
XrmParseCommand()	The option strings in the XrmOptionDescList are compared byte-for-byte with the characters in argv, independent of locale. The name argument and the resource specifier strings in the XrmOptionDescList are in the Host Portable Character Encoding or in an additional implementation-dependent encoding. The resource values are stored in the database as uninterpreted bytes, and all database entries are created with their type set to the quark for the string "String" in the Host Portable Character Encoding.
XGetDefault()	The use of this function is discouraged.

 

Summary: Writing an Internationalized Application

• Set the locale desired by the user by calling setlocale with the empty string ("") as the locale name argument. Verify that the locale is supported by Xlib with XSupportsLocale(). Set the X locale modifiers as desired by the user by passing the empty string to XSetLocaleModifiers(). In an X Toolkit application, use XtSetLanguageProc to register a procedure to set the locale. The default language procedure (which is not actually registered by default) performs all of the above functions.
• Use ANSI-C functions such as strcoll and strftime which make use of the current setting of the locale. Avoid the superseded functions that do not.
• Place all strings which will be displayed by the application in an X resource file. Use X Resource Manager functions in the application to look those strings up.
• Do not assume that the strings your application handles have a uniform state-independent encoding. Treat them as multi-byte strings or convert them to wide-character strings.
• Create an XFontSet for the locale and use it with the new R5 text output functions to measure and display multi-byte and wide-character strings.
• Use XmbSetWMProperties() to set the essential properties for communication with the window manager.
• Use the new R5 property routines to convert from or to the encoding of the current locale when setting or reading text properties.
• Pay attention to the encoding of strings such as Atom and Display names, font and color names, resource names, and resource values specifications.
• Use the new X input method mechanisms to get correctly encoded multi-byte and wide-character input. Chapter 11, "Internationalized Text Input " explains how to do this.