Eckhard Bick - VISL

2.2.6 Recall: Quantifying the problems
In this chapter, I will attempt a quantitative evaluation of the performance of the
morphological analyser module, i.e. that part of the system most prominent in section
2. Obviously, performance at this level only reflects lexical/morphological coverage
and the efficiency of the morphological heuristics and derivation modules. For an
evaluation of the performance of the parsing system as a whole, see chapters 3.9 and
8.1.
Since it does not include contextual disambiguation, the morphological
analyser will have very low precision figures, directly reflecting the inherent
morphological ambiguity of the Portuguese language. Thus, on average, every word
form is assigned 2 morphological/PoS readings (cp. chapter 3.2). However, recall 66 is
high at the morphological analyser level, and can be measured in a meaningful way.
Assuming a reasonably good lexicon coverage and quantification before analytical
heuristics 67 (as explained in 2.2.4.7), most cases of recall failure for a given word
form will be cases of "no reading" rather than "wrong" reading, and the statistics
below will be based upon the assumption that, if non-heuristic readings are found,
the correct one will be among them. With this slight simplification it becomes
possible to judge recall by quantifying "unanalysable", no-reading words. 68 This
allows automatic extraction of the problematic words for closer inspection, reducing
inspection work load from 100% to about 0.5%.
The sample in (1) consists of a 131.981 word corpus of literature and
commentaries, containing 604 unanalysable words. For comparison, language
specified percentages for loan word frequency in a larger sample (629.364 words,
2599 unanalysables) from the mixed Borba-Ramsey corpus of Brazilian Portuguese
are given in parentheses.
66
Basically, disambiguation improves precision and reduces recall, ultimately - at 100% precision -, recall will become
"correctness", i.e. the percentage of correct readings.
67
The only kind of heuristics that does have a bearing on the numbers below, are some rules for orthographical regional
variation, but the respective figures are included in table (1). Capitalized names are here not regarded as "unanalysable",
and not included. Name heuristics, involving up to 2% of word forms in running text, is described in detail in 2.2.4.4.
68
The management of “unanalysable” words is discussed in detail in chapter 2.2.4.7.
- 96 -