YEARS OF EUROPEAN ONLINE ANNÉES DE EN LIGNE ...

allows for identifying the concepts of a document. In fact, both approaches
should be used subsequently to arrive at a clear and solid classiication of texts.
the documents actually are instantiation of different concepts.
text-mining technology is not limited to the basic ideas which are expressed
at the surface of a given text, but discovers other ones as well. thus
documents can be related to each other although the key terms used are different.
for example, a document handling the legislation on ‘environment protection’
can be classiied together with documents treating problems of ‘air pollution’.
Linguistic statistics offer basic methods for the distinction of relevant and
less- or non-relevant components. Regularities concerning the frequency of
words and word forms in a given language allow the description of linguistic
relations. this approach combined with a probabilistic linguistic model is one
of the fundaments of text mining.
One of the most important methods analyses the differences of the vocabulary
of a given text with a reference text collection which is supposed to
represent the general vocabulary of a language. After determining the frequencies
of all word forms or tokens in both the text and the reference collection,
there are four classes which help to identify a word:
(1) words which exist in the analysed text, but are not part of the reference
collection: there is a very high probability that these words belong to the
speciic vocabulary of the domain which the text deals with;
(2) words which exist both in the analysed text and the reference collection,
but with the relative frequency in the analysed text being higher than in
the general collection: if a predeined threshold value is exceeded, it is
probable that these words also belong to the speciic terminology of the
domain which the text speaks about;
(3) words which exist both in the analysed text and the reference collection
and whose relative frequencies are more or less the same: in general these
words are necessary for the functioning of a language, they do not regard
speciic subjects;
(4) words which exist in the reference collection with higher relative frequencies
than in the analysed text: with a very high probability these words do
not contain terms of the subject matter treated by the analysed document.
the advantage of this method is that the key terms of a given text can be
identiied with a relatively high probability. the next step could consist of the
classiication of the document with documents which deal with similar sub-
01_2007_5222_txt_ML.indd 156 6-12-2007 15:14:05