Download - Российский комитет Программы ЮНЕСКО ...

Yoshiki MIKAMI
Leader, Language Observatory Project;
Professor, Nagaoka University of Technology;
(Nagaoka, Japan)
Katsuko T. NAKAHIRA
Member, Language Observatory Project;
Assistant Professor, Nagaoka University of Technology
(Nagaoka, Japan)
Measuring Linguistic Diversity on the Web
The Language Observatory
The Language Observatory [1] Project was founded in 2003. The main
objective of the project is to observe the real state of language use on the web.
When the first workshop of the project was held on 21 February 2004, the
Russian Committee of the UNESCO Information for All Programme kindly
reported it in Russian. Then we received several responses from various
language communities around the world. This really encouraged us.
How It Works
The Language Observatory is designed to measure the use of each language on
the World Wide Web. Measurement is done by counting the number of written
pages on the Web in each language.
The observatory consists of two major components. The first is a data collection
instrument from the Web, a crawler robot developed at the University of Milan.
It can collect millions of Web pages per day.
The second component is a language identification instrument. We have
developed software to identify language, script and encoding properties of
Web pages with high accuracy and maximum coverage. The first version of
the identification algorithm LIM (Language Identification Module) was
developed in 2002 [2] and implemented in 2004. The most recent updated
version is called G2LI. You can use it on the Web.
According to a recent verification examination G2LI is capable of identifying
184 languages in ISO Language Code (ISO 639-1) with an average accuracy of
94%. In addition to a wide coverage of languages, it can identify various types
of legacy encodings 2 , which are still extensively used by many non-Latin-script
user communities.
2
Legacy encodings are non-standardised, and often proprietary encodings.
46