Suffix and Prefix Arrays for Gappy Phrase Discovery

Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Suffix and Prefix Arrays for Gappy Phrase
Discovery
Dale Gerdemann
Department of Linguistics
Computational Linguistics Section
D-72074 Tübingen
First Tübingen Workshop on Machine Learning, 2010

Outline
Gappy Phrase
Discovery
Gerdemann
Introduction
1 Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion

Outline
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
1 Introduction
2 Terminology
Toy Corpus
Examples
Bridge
Partners
Conclusion

Outline
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
1 Introduction
2 Terminology
3 The Algorithm
Conclusion

Outline
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
1 Introduction
2 Terminology
3 The Algorithm
4 Toy Corpus Examples

Outline
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
1 Introduction
2 Terminology
3 The Algorithm
4 Toy Corpus Examples
5 Bridge Partners

Outline
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
1 Introduction
2 Terminology
3 The Algorithm
4 Toy Corpus Examples
5 Bridge Partners
6 Conclusion

Introduction
Gappy Phrase
Discovery
Gerdemann
Goal: Discovery of recurrent patterns in texts.
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion

Introduction
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Goal: Discovery of recurrent patterns in texts.
Examples:
more than
never even
no idea what
of the
. . .

Introduction
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Goal: Discovery of recurrent patterns in texts.
Examples:
more than
never even
no idea what
of the
. . .
The approach ignores traditional linguistic categories Det, N,
NP, etc. Such linguistic knowledge can be rediscovered, and
applied to multi-word expressions, rather than to
whitespace-delimited “words.”

Gappy Phrases
Gappy Phrase
Discovery
Gerdemann
Recurrent patterns often occur with slight variation.
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion

Gappy Phrases
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Recurrent patterns often occur with slight variation.
Gappy phrase discovery is an approach for finding sequences of
two phrases separated from each other by a distance not
greater than d.
Bridge
Partners
Conclusion

Gappy Phrases
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Recurrent patterns often occur with slight variation.
Gappy phrase discovery is an approach for finding sequences of
two phrases separated from each other by a distance not
greater than d.
Examples:
from one X to the other

Gappy Phrases
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Recurrent patterns often occur with slight variation.
Gappy phrase discovery is an approach for finding sequences of
two phrases separated from each other by a distance not
greater than d.
Examples:
from one X to the other
upside [|-] down

Gappy Phrases
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Recurrent patterns often occur with slight variation.
Gappy phrase discovery is an approach for finding sequences of
two phrases separated from each other by a distance not
greater than d.
Examples:
from one X to the other
upside [|-] down
åäèí è ñúùè, åäíà è ñúùà, åäíî è ñúùî, åäíè è ñúùè

Gappy Phrases
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Recurrent patterns often occur with slight variation.
Gappy phrase discovery is an approach for finding sequences of
two phrases separated from each other by a distance not
greater than d.
Examples:
from one X to the other
upside [|-] down
åäèí è ñúùè, åäíà è ñúùà, åäíî è ñúùî, åäíè è ñúùè
Bézier curve . . . control point

Gappy Phrases
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Recurrent patterns often occur with slight variation.
Gappy phrase discovery is an approach for finding sequences of
two phrases separated from each other by a distance not
greater than d.
Examples:
from one X to the other
upside [|-] down
åäèí è ñúùè, åäíà è ñúùà, åäíî è ñúùî, åäíè è ñúùè
Bézier curve . . . control point
at the top of [her|his|his shrill little|its] voice

Gappy Phrases
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Recurrent patterns often occur with slight variation.
Gappy phrase discovery is an approach for finding sequences of
two phrases separated from each other by a distance not
greater than d.
Examples:
from one X to the other
upside [|-] down
åäèí è ñúùè, åäíà è ñúùà, åäíî è ñúùî, åäíè è ñúùè
Bézier curve . . . control point
at the top of [her|his|his shrill little|its] voice
autumn , when the leaves are [|getting] brown

Bidirectional Examples
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Übungsaufgabe [|dem geneigten Leser] überlassen
als [|leichte|triviale|untergeordneter] Übungsaufgabe
Beweis [für unser|habe ich Ihnen] bereits
Beweis [|geschieht] durch
Beweis [|nun schon] zu
den [|geforderten] Beweis
Höhe [|eines Fahnenmastes|eines Hochhauses] bestimmen
Höhe [sowie|wissen und nicht] die Länge
aus [einer bekannten|seiner] Höhe

Bidirectional Examples
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Übungsaufgabe [|dem geneigten Leser] überlassen
als [|leichte|triviale|untergeordneter] Übungsaufgabe
Beweis [für unser|habe ich Ihnen] bereits
Beweis [|geschieht] durch
Beweis [|nun schon] zu
den [|geforderten] Beweis
Höhe [|eines Fahnenmastes|eines Hochhauses] bestimmen
Höhe [sowie|wissen und nicht] die Länge
aus [einer bekannten|seiner] Höhe
To group these together, suffix arrays are used for the forward
case, and prefix arrays are used for the backward case.

Tokenizing by Character
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Many interesting alternations can be found by tokenizing
characters (here also converting capitalization into markup):
〈cap〉zum 〈cap〉beispiel könn[|t]en 〈cap〉sie
Toy Corpus
Examples
Bridge
Partners
Conclusion

Tokenizing by Character
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Many interesting alternations can be found by tokenizing
characters (here also converting capitalization into markup):
〈cap〉zum 〈cap〉beispiel könn[|t]en 〈cap〉sie
But many junk alternations are also found:
versch[ieb|l|w]u
Zeitverschiebung, 1x
Phasenverschiebung, 1x
Rotverschiebung, 3x
verschluckt,
1x
verschwunden,
1x

Tokenizing by Character
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Many interesting alternations can be found by tokenizing
characters (here also converting capitalization into markup):
〈cap〉zum 〈cap〉beispiel könn[|t]en 〈cap〉sie
But many junk alternations are also found:
versch[ieb|l|w]u
Zeitverschiebung, 1x
Phasenverschiebung, 1x
Rotverschiebung, 3x
verschluckt,
1x
verschwunden,
1x
Challenge: filter out junk; retain interesting cases.

Motivation
Gappy Phrase
Discovery
Gerdemann
Lexicography and language research
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion

Motivation
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
Lexicography and language research
Machine translation
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion

Motivation
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Lexicography and language research
Machine translation
Plagerism detection, detection of cut-and-paste problems and
detection of other intended repetitions
Conclusion

Motivation
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Lexicography and language research
Machine translation
Plagerism detection, detection of cut-and-paste problems and
detection of other intended repetitions
Corpus construction/consistency checking

Motivation
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Lexicography and language research
Machine translation
Plagerism detection, detection of cut-and-paste problems and
detection of other intended repetitions
Corpus construction/consistency checking
Text categorization (see MA thesis of Maria Tchalakova)

Motivation
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Lexicography and language research
Machine translation
Plagerism detection, detection of cut-and-paste problems and
detection of other intended repetitions
Corpus construction/consistency checking
Text categorization (see MA thesis of Maria Tchalakova)
Morphology induction

Motivation
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Lexicography and language research
Machine translation
Plagerism detection, detection of cut-and-paste problems and
detection of other intended repetitions
Corpus construction/consistency checking
Text categorization (see MA thesis of Maria Tchalakova)
Morphology induction
German compound noun splitting, Chinese segmentation.

Terminology
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The sequence “from one” is the left part, “to the other” is the
right part and the set of values for X is the bridge. The
parameter d specifies the length of the longest sequence in X .
Procedurely, the part that is found first is called the initial part.
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion

Terminology
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
The sequence “from one” is the left part, “to the other” is the
right part and the set of values for X is the bridge. The
parameter d specifies the length of the longest sequence in X .
Procedurely, the part that is found first is called the initial part.
A phrase is a sequence of tokens which has occurrences in a
text. A phrase is saturated or maximal if it has at least two
occurrences and cannot be extended without losing some
occurrences. Suppose, for example, that mumbo and mumbo
jumbo each have 5 occurrences, but any extended phrase, such
as speak mumbo jumbo has fewer than 5 occurrences. Then
mumbo jumbo is maximal, but mumbo is not.

Terminology
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
The sequence “from one” is the left part, “to the other” is the
right part and the set of values for X is the bridge. The
parameter d specifies the length of the longest sequence in X .
Procedurely, the part that is found first is called the initial part.
A phrase is a sequence of tokens which has occurrences in a
text. A phrase is saturated or maximal if it has at least two
occurrences and cannot be extended without losing some
occurrences. Suppose, for example, that mumbo and mumbo
jumbo each have 5 occurrences, but any extended phrase, such
as speak mumbo jumbo has fewer than 5 occurrences. Then
mumbo jumbo is maximal, but mumbo is not.
Maximal phrases are interesting for two reasons:
more efficient to only use maximal phrases as initial part
non-maximal phrases may be linguistically incomplete

Combinatorics
Gappy Phrase
Discovery
Gerdemann
A text of length n has a maximum of ( )
n+1
2 n-grams.
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion

Combinatorics
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
A text of length n has a maximum of ( )
n+1
2 n-grams.
But the number of maximal phrases with at least 2 occurrences
is no greater than n. Yamamoto and Church (2001) prove this
for right maximal phrases, for maximal phrases there are even
fewer. A phrase is right maximal if it cannot be extended to
the right without losing some of its occurrences.
Conclusion

Combinatorics
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
A text of length n has a maximum of ( )
n+1
2 n-grams.
But the number of maximal phrases with at least 2 occurrences
is no greater than n. Yamamoto and Church (2001) prove this
for right maximal phrases, for maximal phrases there are even
fewer. A phrase is right maximal if it cannot be extended to
the right without losing some of its occurrences.
The set of all right right maximal phrases is easy to represent
with a suffix tree or a suffix array. To get just the maximal
phrases requires some filtering. Abouelhoda et al. (2004) use a
Burrows and Wheeler transformation table for this purpose.

Combinatorics
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
A text of length n has a maximum of ( )
n+1
2 n-grams.
But the number of maximal phrases with at least 2 occurrences
is no greater than n. Yamamoto and Church (2001) prove this
for right maximal phrases, for maximal phrases there are even
fewer. A phrase is right maximal if it cannot be extended to
the right without losing some of its occurrences.
The set of all right right maximal phrases is easy to represent
with a suffix tree or a suffix array. To get just the maximal
phrases requires some filtering. Abouelhoda et al. (2004) use a
Burrows and Wheeler transformation table for this purpose.
Term and document frequencies are calculated for equivalence
classes of phrases. Statistical measures for determining
“interesting” phrases are based on these frequencies.

Terminology for Gappy Phrases
Gappy Phrase
Discovery
Gerdemann
Introduction
A gappy phrase can be extended in 4 different ways, by
extending the left or right parts to the left or right.
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion

Terminology for Gappy Phrases
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
A gappy phrase can be extended in 4 different ways, by
extending the left or right parts to the left or right.
It is possible that the left and right parts are individually
maximal, and yet the combination l . . . r is not.
Bridge
Partners
Conclusion

Terminology for Gappy Phrases
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
A gappy phrase can be extended in 4 different ways, by
extending the left or right parts to the left or right.
It is possible that the left and right parts are individually
maximal, and yet the combination l . . . r is not.
Example: accept. . . changes
it is recommended that one accept all the defaults
cut, copy, paste, undo changes and so on
The user then may accept or reject all changes or
the user can accept or reject the changes individually or
in order to turn off the function or accept or reject changes

Nearness Condition
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
How many occurrences of row. . . your boat are there in row,
row, row your boat? The nearness condition states that only
the nearest occurrence of “row” counts, so that there is only
one occurrence.
Toy Corpus
Examples
Bridge
Partners
Conclusion

Nearness Condition
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
How many occurrences of row. . . your boat are there in row,
row, row your boat? The nearness condition states that only
the nearest occurrence of “row” counts, so that there is only
one occurrence.
It is not clear that that the nearness condition is always
desirable. For the phrase nail the X into the wall, it would seem
reasonable for one instantiation of X to be nail.

Nearness Condition
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
How many occurrences of row. . . your boat are there in row,
row, row your boat? The nearness condition states that only
the nearest occurrence of “row” counts, so that there is only
one occurrence.
It is not clear that that the nearness condition is always
desirable. For the phrase nail the X into the wall, it would seem
reasonable for one instantiation of X to be nail.
The nearness condition is tricky to implement, involving two
binary searches. For example, when “row” is found, a binary
search is made to find the closest instance of “your boat,” and
then a second binary search to the left is performed to see if
there is a closer instance of “row.” For an alternative method,
see Apostolico and Satta (2009).

Nearness Condition Continued
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Example where nearness condition interacts with maximality:
The following step-by-step procedure shows
The following brief step-by-step example
The following is a step-by-step procedure
The following provides a step-by-step procedure
Conclusion

Nearness Condition Continued
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Example where nearness condition interacts with maximality:
The following step-by-step procedure shows
The following brief step-by-step example
The following is a step-by-step procedure
The following provides a step-by-step procedure
It appears that step is maximal, since it occurs with several
right contexts: {procedure,example,-}

Nearness Condition Continued
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Example where nearness condition interacts with maximality:
The following step-by-step procedure shows
The following brief step-by-step example
The following is a step-by-step procedure
The following provides a step-by-step procedure
It appears that step is maximal, since it occurs with several
right contexts: {procedure,example,-}
But the instances with following procedure or example fail the
nearness condition. The correct maximal phrase for the right
part is step-by-step.

Overlap
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Assuming that “to the” and “the ground” are both maximal,
then here we have overlap:
to the ground
Toy Corpus
Examples
Bridge
Partners
Conclusion

Overlap
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Assuming that “to the” and “the ground” are both maximal,
then here we have overlap:
to the ground
Note that the overlap alternates with non-overlapping
instances:
to the one on the ground
to the pail on the ground

Overlap
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Assuming that “to the” and “the ground” are both maximal,
then here we have overlap:
to the ground
Note that the overlap alternates with non-overlapping
instances:
to the one on the ground
to the pail on the ground
Another example:
really like to know
really like to get to know

More Overlap
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Commonly involved in overlap:
determiners: the, a
prepositions: of, on
infinitive marker: to
punctuation: comma
Bridge
Partners
Conclusion

More Overlap
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Commonly involved in overlap:
determiners: the, a
prepositions: of, on
infinitive marker: to
punctuation: comma
More unusual examples:
long click the
long click or short click the
extra credit card
old men ’s gym

More Overlap
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Commonly involved in overlap:
determiners: the, a
prepositions: of, on
infinitive marker: to
punctuation: comma
More unusual examples:
long click the
long click or short click the
extra credit card
old men ’s gym
Current version deals with overlap in asymmetric, ad hoc way.
Possibly better approach in Apostolico and Satta (2009).

Lcp-Interval Tree for: mining engineering
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
e
g
ing
in
e e e g g g i i i i m n n n n n r $
e n r i $ n n n n i e g g g i i e
r g i n e e g g i n e i $ n n n
i i n e n e $ n i r n e g g g
n n g e g r e g n i e n $ i
. . . . . . . . . . . . . .
ɛ
n
ng

Lcp-Interval Tree for: mining engineering
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
e
g
ing
in
e e e g g g i i i i m n n n n n r $
e n r i $ n n n n i e g g g i i e
r g i n e e g g i n e i $ n n n
i i n e n e $ n i r n e g g g
n n g e g r e g n i e n $ i
. . . . . . . . . . . . . .
ɛ
n
ng
Red nodes are for binary search through possibly large
alphabet; black nodes represent longest-common-prefix
intervals (Kim et al., 2008).

The KS Algorithm (Greatly Simplified)
Gappy Phrase
Discovery
Gerdemann
Given a text as a sequence of integers, build an
lcp-interval tree (triple of arrays) in linear time.
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion

The KS Algorithm (Greatly Simplified)
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Given a text as a sequence of integers, build an
lcp-interval tree (triple of arrays) in linear time.
Traverse interval tree. Upon encountering a (left-maximal)
black node, record set of positions of phrases in set p1.
Recursively build lcp-interval tree for the set of positions
within distance d to the right of a position in p1. (left
case is similar)
Conclusion

The KS Algorithm (Greatly Simplified)
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Given a text as a sequence of integers, build an
lcp-interval tree (triple of arrays) in linear time.
Traverse interval tree. Upon encountering a (left-maximal)
black node, record set of positions of phrases in set p1.
Recursively build lcp-interval tree for the set of positions
within distance d to the right of a position in p1. (left
case is similar)
Traverse the recursively built interval tree. Upon
encountering a black node, record set of positions of
phrases in set p2. For each position in p2, find (using
binary search) a corresponding position in p1 that satisfies
the nearness condition. Record these positions in p3. For
each position in t ∈ p3 find (binary search again) the
corresponding position s ∈ p2 and record the pair (s, t) in
p4. The set p4 now contains indices of the left and right
parts of candidate gappy phrases.

The KS Algorithm (Greatly Simplified)
Gappy Phrase
Discovery
Gerdemann
Introduction
Filter out non-maximal gapppy phrases.
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion

The KS Algorithm (Greatly Simplified)
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Filter out non-maximal gapppy phrases.
Record term frequencies and document frequencies for the
gappy phrase as and also for the left and right parts as
standalone phrases. These frequencies could be used to
filter out cases that occur by chance according to some
probability model.

The KS Algorithm (Greatly Simplified)
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Filter out non-maximal gapppy phrases.
Record term frequencies and document frequencies for the
gappy phrase as and also for the left and right parts as
standalone phrases. These frequencies could be used to
filter out cases that occur by chance according to some
probability model.
Note: Finding tf and df for the right part involves binary
search through the interval tree. So it is important to use
a data structure that is adapted to a large alphabet. (Kim
et al., 2008)

A Toy Corpus
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
A corpus of 4 texts, with a sentinel at the end of each:
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&
Toy Corpus
Examples
Bridge
Partners
Conclusion

A Toy Corpus
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
A corpus of 4 texts, with a sentinel at the end of each:
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&
A regular expression for tokenizing:
\$\s |
\#\s |
\%\s |
\&\s |
[\w\W]

A Toy Corpus
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
A corpus of 4 texts, with a sentinel at the end of each:
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&
A regular expression for tokenizing:
\$\s |
\#\s |
\%\s |
\&\s |
[\w\W]
Other characters, such as punctuation and paragraph
boundaries, can also be treated like sentinels.

Initial Phrase Discovery
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Non-gappy phrase discovery finds the “phrase” se:
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&

Initial Phrase Discovery
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Non-gappy phrase discovery finds the “phrase” se:
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&
Note se is maximal; se is supermaximal.
A phrase is supermaximal if it is maximal, and not a proper
substring of another maximal phrase.

leftward Search Space
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Select prefix end points within distance d of the initial phrase.
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&
Bridge
Partners
Conclusion

leftward Search Space
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Select prefix end points within distance d of the initial phrase.
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&
These prefixes are represented by the following numbers, and
the numbers are sorted rather than the prefixes themselves.
tor #3
tort #4
torto #5
tortoi #6
. . . the #57
etc

leftward Search Space
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
We discover that se is preceded 3 times by ho, and 2 times by
the longer phrase ho,
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&

leftward Search Space
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
We discover that se is preceded 3 times by ho, and 2 times by
the longer phrase ho,
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&
Note: If any phrase were found from rtoi, tho and hou, then
this phrase would have to be rejected. Such a phrase would be
rediscovered when searching leftward from se .

Overlapping leftward Search Space
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
What happens when the leftward search spaces overlap?
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&
Bridge
Partners
Conclusion

Overlapping leftward Search Space
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
What happens when the leftward search spaces overlap?
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&
If d = 4, we get the following search space:
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&

Overlapping leftward Search Space
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
What happens when the leftward search spaces overlap?
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&
If d = 4, we get the following search space:
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&
The phrase t is found twice within this search space. These 2
occurrences need to be matched with corresponding right parts.

Too close to the edge
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Should we “discover” that ea combines with
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&
th to the left?

Too close to the edge
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Should we “discover” that ea combines with
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&
th to the left?
To rule this out, we need to say that “%” and other sentinels
are illegal as part of a bridge alternative.

Too close to the edge
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Should we “discover” that ea combines with
tortoise that supports the earth$
put the cart before the horse#
a comedy to those who think%
eat you out of house and home&
th to the left?
To rule this out, we need to say that “%” and other sentinels
are illegal as part of a bridge alternative.
Additional illegal bridge elements can be specified with a
regular expression.

Reversing the Problem
Gappy Phrase
Discovery
Gerdemann
It is trivial to adapt the algorithm to find pairs of phrases that
coocur in the bridge.
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion

Reversing the Problem
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
It is trivial to adapt the algorithm to find pairs of phrases that
coocur in the bridge.
have a [concussion|favorite word|life|. . . |thesis] or
life and thesis occur
in same bridge
Bridge
Partners
Conclusion

Reversing the Problem
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
It is trivial to adapt the algorithm to find pairs of phrases that
coocur in the bridge.
have a [concussion|favorite word|life|. . . |thesis] or
life and thesis occur
in same bridge
Definition: two phrases occurring as different alternatives in a
bridge are called bridge partners.
than the [life|thesis] of a
life and thesis occur
in same bridge again

Reversing the Problem
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
It is trivial to adapt the algorithm to find pairs of phrases that
coocur in the bridge.
have a [concussion|favorite word|life|. . . |thesis] or
life and thesis occur
in same bridge
Definition: two phrases occurring as different alternatives in a
bridge are called bridge partners.
than the [life|thesis] of a
life and thesis occur
in same bridge again
It can’t be a coincidence, can it?

Random Sample Output from 6000 lines
Science humor collected by Joachim Verhagen
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
(asked|trying)/[(was~to explain), (who was~to)]
(making|the)/[(,~light of the), (of~use of)]
(had to|have)/[(I~also), (I~come up with)]
(matter|the problem)/[(that~cannot be), (the nature of
(don’t|really don’t)/[(I~care), (I~know .), (I~need to
(made|told)/[(I~it to), (were~to use)]
(obvious|possible)/[(, it is~that), (it is~. "), (some
(a specific|the)/[(do~job), (with~focus)]
(answer|one)/[(be the~to), (the correct~.)]
(building|bulb)/[(hold the~and), (on the~at the top)]
(for|to get)/[(enough~all), (enough~it to)]
(You|you)/[("~need to), (" Do~believe in), ((~could),
(first|top)/[(His~-), (of the~ten)]
(statement|story)/[(a true~.), (true~:)]
(certain|sure)/[(I can’t be~.), (make~that), (never~wh

Big, Little, Large, Small
See: Lynne Murphy, Semantic Relations and the Lexicon
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
(big|great)/[(a~deal), (is a~place)]
(bigger|equal)/[(are~, choose), (may be~, or)]
(big|real)/[(Coulomb got a~charge out of the), (the~deal), (you could ge
(big|large)/[(a~ball of fire), (a~charge), (a~lump)]
(bigger|greater)/[(have a~chance of), (times~than)]
(big|good)/[(After a~meal), (a~deal), (a~f * *)]
(little|lot)/[(a~faster), (be a~more)]
(little|lot of)/[(a~fun .), (a~heat)]
(bit of|little)/[(a~fun .), (a~help from)]
(a little|the)/[(. What~acorn), (for~known), (lecture ,~old)]
(large chunk|piece)/[(a~of potassium), (such a~of)]
(a large|the)/[(at~university , the), (in~empty), (to~engineering), (to~
(large|real)/[(a~beaker), (a~charge)]
(large|small)/[(a~South), (a~beaker), (a~fence), (a~group of), (a~order)
(late|small)/[(too~)), (too~for it)]
(short|small)/[(a~step), (so~that you)]
(a small|the)/[(of~South), (puts~fence around)]
(problems|small children)/[(had~with him), (he had~with)]

Jason Riggle’s use of the word “set”
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
(generator|set)/[(candidate~and), (candidate~with a), (on the candidate~
(number of|set of)/[(in the~nodes), (the~ERCs that), (the~actual), (the~
(machine|set of constraints)/[(a~, it is), (the entire~.), (the intersec
(pair|set)/[(, another~of), (a~of nodes), (a new~of nodes at), (each~of
(ranked set|set)/[(a~of constraints), (takes a~of), (with a~of)]
(constraints|set)/[(how the~of), (the~corresponds to), (the~that make up
(set of|sets of)/[(finite~candidates), (ranked~constraints), (the~cost v
(infinite set|set)/[(the~of candidates), (the~of input), (the~of paths),
(constraints|set of constraints)/[(by the~.), (intersected~.), (ranked~.
(cost|set)/[(, once the~of), (and finds the~of), (find the~of)]
(most harmonic|set of)/[(the~final states), (the~paths through M), (the~
(number|set)/[(exactly the~of), (get the same~of violations), (in the~of
(set|sets)/[(finite~. If), (finite~of candidates), (finite candidate~.),
(set|string)/[(of a~of), (the empty~and), (with the empty~as)]
(machine|set H)/[(added to the~.), (in the~. The), (of the nodes in the~
(same set|set)/[(exactly the~of), (the~of contenders), (the~of nodes), (
(cascade|set)/[(a~of finite), (of a~of), (with a~of)]
(range|set)/[(generates an infinite~of), (in a~of), (of the infinite~of
(optimal|set of)/[(in the~candidates), (the~candidates .), (the~paths th
(set of|single)/[(a~finite state), (a~optimal), (the~cheapest)]
(bound|set)/[(particular~of), (to a~of), (violation~. The)]
(representation|set)/[(, but the~of), (a finite~of), (the size of the~of

Applications of Bridge Partners
Gappy Phrase
Discovery
Gerdemann
Introduction
Could bridge partners be used as features for machine
learning algorithms?
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion

Applications of Bridge Partners
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Could bridge partners be used as features for machine
learning algorithms?
Could bridge partners be useful for named entity
recognition? A capitalized word with 3rd, sing, masc
pronoun as bridge partner is likely to be the name of a
male person.
(Ñòåôàí|Òîé)/[(. ~ ñå êà÷è â), (Ìàêñ . ~ ñå), (óäîâîëñòâèå . ~ áåøå)]
Conclusion

Applications of Bridge Partners
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Could bridge partners be used as features for machine
learning algorithms?
Could bridge partners be useful for named entity
recognition? A capitalized word with 3rd, sing, masc
pronoun as bridge partner is likely to be the name of a
male person.
(Ñòåôàí|Òîé)/[(. ~ ñå êà÷è â), (Ìàêñ . ~ ñå), (óäîâîëñòâèå . ~ áåøå)]
Could bridge partners be useful for learning morphology?
Note, for example, that corresponding singular and plural
forms tend to occur as partners:
(grammar|grammars)/[(. Gerdemann and Van Noord call the resulting˜for
such cases), (the˜defined by), (the˜won’t)]
(model|models)/[(Gerdemann and Van Noord’s˜is), (computational˜of
Optimality Theory), (finite state˜of OT)]

Applications of Bridge Partners (Continued)
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
Could educational programs use bridge partners as indicators
that a learner relates the concepts? Consider the pair machine
and set of constraints:
(machine|set of constraints)/[(a˜, it is), (the entire˜.), (the intersected˜.)]
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion

Applications of Bridge Partners (Continued)
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Could educational programs use bridge partners as indicators
that a learner relates the concepts? Consider the pair machine
and set of constraints:
(machine|set of constraints)/[(a˜, it is), (the entire˜.), (the intersected˜.)]
Does this mean that Jason’s concept of a machine is similar to
his concept for a set of constraints?
Bridge
Partners
Conclusion

Applications of Bridge Partners (Continued)
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Could educational programs use bridge partners as indicators
that a learner relates the concepts? Consider the pair machine
and set of constraints:
(machine|set of constraints)/[(a˜, it is), (the entire˜.), (the intersected˜.)]
Does this mean that Jason’s concept of a machine is similar to
his concept for a set of constraints?
Alternatives:
LSA Use a term-document matrix with Latent
Semantic Analysis for dimensionality reduction.
Disadvantage: One needs a large number of texts
from the learner.
Proximity A simple indicator of relatedness is mention of
both concepts within a given radius. Ex: Bézier
curve and control point in the OpenOffice user
manual.

Conclusions
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Extraction of gapped phrases is feasible using suffix/prefix
arrays.
Toy Corpus
Examples
Bridge
Partners
Conclusion

Conclusions
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Extraction of gapped phrases is feasible using suffix/prefix
arrays.
Linear time?
Bridge
Partners
Conclusion

Conclusions
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Extraction of gapped phrases is feasible using suffix/prefix
arrays.
Linear time?
Gappped phrases should be maximal/saturated.
Conclusion

Conclusions
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Extraction of gapped phrases is feasible using suffix/prefix
arrays.
Linear time?
Gappped phrases should be maximal/saturated.
There are various kinds of gapped phrases.

Conclusions
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Extraction of gapped phrases is feasible using suffix/prefix
arrays.
Linear time?
Gappped phrases should be maximal/saturated.
There are various kinds of gapped phrases.
Gap fillers (bridge partners) are also interesting.

Conclusions
Gappy Phrase
Discovery
Gerdemann
Introduction
Terminology
The Algorithm
Toy Corpus
Examples
Bridge
Partners
Conclusion
Extraction of gapped phrases is feasible using suffix/prefix
arrays.
Linear time?
Gappped phrases should be maximal/saturated.
There are various kinds of gapped phrases.
Gap fillers (bridge partners) are also interesting.
Work is needed primarily on filtering out uninteresting
cases,

For Further Reading I
Gappy Phrase
Discovery
Gerdemann
Appendix
For Further
Reading
References
Mohamed Ibrahim Abouelhoda, Stefan Kurtz, and Enno
Ohlebusch. Replacing suffix trees with enhanced suffix
arrays. J. of Discrete Algorithms, 2(1):53–86, 2004.
Alberto Apostolico and Giorgio Satta. Discovering subword
associations in strings in time linear in the output size. J. of
Discrete Algorithms, 7(2):227–238, 2009.
Dong Kyue Kim, Minhwan Kim, and Heejin Park. Linearized
suffix tree: an efficient index data structure with the
capabilities of suffix trees and suffix arrays. Algorithmica, 52
(3):350–377, 2008.
Mikio Yamamoto and Kenneth W. Church. Using suffix arrays
to compute term frequency and document frequency for all
substrings in a corpus. Comput. Linguist., 27(1):1–30, 2001.