Download - Российский комитет Программы ЮНЕСКО ...

A Hidden Component: The Universal Declaration of Human Rights
Hidden inside the language identification instrument is a set of training texts
for the software. Considering that the richness and quality of training texts
is the most critical in language identification task we used a set of translated
texts from the Universal Declaration of Human Rights (UDHR) provided by
the UN Higher Commission for Human Rights (UNHCHR).
Of note is that not all translated UDHR texts are provided with encoding;
some are available only as image files. Image files can be read by humans but not
directly by computers, necessitating that we transform images into text data.
Table 1 illustrates how many transformed texts are given in image format (322
languages were available at the date of the first search, in early 2004). More than
two hundred languages use Latin script, with or without diacritics, and only
three of them were given in PDF or GIF file format. In contrast to this, among
languages using so-called Abugida script 3 , not a single language was presented in
the form of encoded text. This fact might itself point to the existence of a digital
language divide, or in this particular case, a “digital script divide”.
Table 1. Number of available UDHR texts from the UNHCHR website by format
Latin Cyrillic Other Abjad Abugida Hanzi All Total
alphabet
others
Encoded 253 10 1 1 0 3 0 268
PDF 2 4 2 3 10 0 4 25
GIF 1 3 0 9 15 0 1 29
Total 256 17 3 13 29 3 5 322
Other alphabets: Greek, Armenian and Georgian; Abjad: Arabic and Hebrew; Abugida: Amharic
and all Brahmi origin scripts used in South and Southeast Asia; Hanzi: Chinese, Japanese and
Korean; All others: Assyrian, Canadian syllabics, Ojibwa, Cree, Mongolian and Yi.
Around the same time as we launched the Language Observatory Project, Eric
Miller launched UDHR-in-Unicode project. The objective of this project was
to demonstrate the use of Unicode for a wide variety of languages, using the
Universal Declaration of Human Rights (UDHR) as a representative text.
Currently, UDHR-in-Unicode is housed on the Unicode Consortium website
and the texts are used in the study of natural language processing.
3
Abugida scripts are syllabic scripts, most of which are generated from Indian Brahmi scripts and currently
used in South and Southeast Asian regions. Another important Abugida script is Amharic.
47