Eckhard Bick - VISL

3.5.1 Probabilistics: The 'fire a linguist' approach
Most probabilistic NLP systems address part of speech tagging by automatic training
and base themselves upon Hidden Markov Models (HMM), a mathematical model,
where a surface-sequence of symbols is stochastically generated by an underlying
("hidden") Markov process with a state- and/or transition-dependent symbol generator.
A Markov Model consists of a finite number of states and describes processes (or
sequences) as transitions (probability labelled arcs) between these states:
(1)
start
0.2 0.1
0.5 0.4 0.6
2
0.3
1 3
The MM in (1) has three states and a stop-state (Ø). When in state 1, for example, the
MM has a 50% probability of staying there, a 20% probability of moving to state 2, and
a 30% probability of moving to state three. The probability for a given sequence can be
computed as the product of the individual transition probabilities. Thus, the sequence
1132 is assigned the probability 0.5 x 0.3 x 0.1 x 0.4 = 0.006. Since transition
probabilities only depend on which state the process is in at a given point in time, such
a MM is called a first order Markov Model. If the model's states represented the words
of a language, sequences could be used to model utterances in that language, and
transition probabilities could be computed as bigram frequencies in a text corpus.
However, the lack of "contextual memory" in a first order MM makes it impossible to
describe long distance correlations like subject-predicate agreement or valency. In
theory, using higher order Markov Models can be used to somewhat soften this
problem. In a n-th order MM, the networks history of the last n-1 states is taken into
- 134 -
0.9