Eckhard Bick - VISL

As shown in the diagram (yellow boxes), PALMORF - or rather its preprocessor and
heuristics modules - is quite capable of “meddling” with its data. Still, orthographic
intervention as such (*) is used only heuristically, where no ordinary analysis has
been found, and the altered word forms are marked 'ALT', so they can be identified
later, for example for output statistics, and for the sake of general corpus fidelity.
Affected areas are luso-brazilian orthographic variation (e.g. oi/ou digraphs, ct -> t,
cp -> p), typographically based accentuation errors (e.g. 7-bit-ASCII vs. 8-bit-ASCII
input) and some common spelling errors (e.g. cão -> ção, çao -> ção).
2.2.2.2 Preprocessing
local
disambiguation
Unlike post-analysis heuristics, preprocessor intervention (+) applies to all input, and
is close to being a general parsing necessity. Among other things, a natural and
unavoidable step in all NLP is the decision of what to tag. Obviously, in a word
based tagger and a sentence based parser, this amounts to establishing word and
sentence boundaries.
First, the preprocessor strives to establish what is not a word, and marks it by
prefixing a $-sign: $. - $, - $( - $) - $% -$78.7 - $± - $” - $7:20 etc. Of these, some
are later treated as words anyway. Thus, numbers will be assigned the word class
NUM and a syntactic function, $% will be treated as a noun (N), $7:20 as a time
adverbial. Punctuation is treated in four ways:
(a) as sentence delimiter. Ordinarily, it is the DELIMITERS list of the CG rule
file that determines which punctuation marks are treated as sentence boundaries (e.g.
$. and $:, but not $- and $,). However, the preprocessor can add sentence delimiters
() where it identifies sentence-final abbreviations, or - for instance - instead of
double line feeds around punctuation-free headlines.
(b) as a regular non-word. Such punctuation is shown in the analysis file
without a tag (e.g. $: or $!), but can still be referred to by CG-rules.
(c) as tag-bearing “words”. This is unusual in a Constraint Grammar, but $%
(as a noun) is an example, and $, as a co-ordinator (like the conjunction ‘e’) is
another one.
- 18 -
OUTPUT
orthographic
variation*
accentuation errors*
spelling errors*
propria heuristics+
non-propria heuristics+