Eckhard Bick - VISL

no lexicon. By combining supervised and unsupervised learning, accuracies of up to
96.8% have subsequently be described (Brill, 1996). Results for languages other than
English seem to confirm the 97% mark as a kind of upper ceiling for the performance of
probabilistic PoS taggers. Thus, the Morphy system described in (Lezius et. al., 1996)
achieved an accuracy of 95.9% for a tag set of 51 tags, using a lexicon of 21.500 words
(about 100.000 word forms). Lezius cites 5 other German taggers or morphology
systems with accuracy rates in the range between 92.8 - 96.7% 91 (ibd. p. 370).
For probabilistic (syntactic) parsing, performance is considerably lower, and such
systems have not so far been able to replace manual annotation as a means of syntactic
parsing. For standard PCFGs, which augment standard CFGs with probabilistic
applicability constraints, accuracies of about 35% are supposed to be typical. Better
results are achieved by conditioning production probabilities not only on the terminal in
question, but also on the rule that generated it, as well as one or more subsequent words.
On the short sentences of the MIT Voyager corpus, an accuracy of 87.5% is reported
(Marcus, 1993). Some parsers make use of lexical information: For the SPATTER
parser (Magermann, 1995) 84% accuracy is claimed for recovering labelled constituents
in WSJ text. In (Collins, 1996) head-dependent relations between pairs of words are
modelled in a probabilistic fashion, yielding 85% precision and recall on the same
material. For longer sentences, systems do not fare as well: (Carroll and Briscoe, 1995)
describes experiments with a probabilistic LR parser trained and tested on the Susannecorpus
(average sentence length: 20 tokens), which first had been relabelled with
CLAWS-II tags using the Acquilex HMM-tagger (Elsworthy, 1994). Here, for
bracketings matching the treebank, a recall of 73.56% and a precision of 39.82 is
reported for the highest ranked 3 analyses of each sentence. 43.8% of sentences had the
correct analysis ranked among the top 10. Parse fails amounted to 25.9% and time-outs
to 0.2%. Nearly a third of all test sentences received more than one hundred different
analyses, 5.8% were assigned more than 100.000 parses.
3.5.2 Generative Grammar:
All or nothing - the competence problem
91 For larger tag sets with hundreds of tags (presumably including inflexional information), considerably lower accuracy
rates - around 80% - are cited for those members of the group of German taggers, that have this option. Of course, as
Elworthy (1995) points out, what is important for performance, may not so much be the size of the tag set used, but the type
of information encoded. From the point of view of disambiguation one might argue that larger tag sets leave more
ambiguities to resolve, but they also provide more and better context to do so (for example, in the shape of inflexional
agreement information). The relatively constant performance of different versions of CLAWS (Leech et. al., 1994), with tag
set size varying by nearly a factor of three, seem to corroborate this assumption.
- 139 -