The.Algorithm.Design.Manual.Springer-Verlag.1998

War Story: String 'em Up
``I've tried the fastest computer in our department, but our program is too slow,'' Dimitris complained.
``It takes forever on strings of length only 2,000 characters. We will never get up to 50,000.''
For interactive SBH to be competitive as a sequencing method, we had to be able to sequence long
fragments of DNA, ideally over 50 kilobases in length. If we couldn't speed up the program, we would be
in the embarrassing position of having a biological technique invented by computer scientists fail
because the computations took too long.
We profiled our program and discovered that almost all the time was spent searching in this data
structure, which was no surprise. For each of the possible concatenations, we did this k-1 times. We
needed a faster dictionary data structure, since search was the innermost operation in such a deep loop.
``What about using a hash table?'' I suggested. ``If we do it right, it should take O(k) time to hash a kcharacter
string and look it up in our table. That should knock off a factor of , which will mean
something when 2,000.''
Dimitris went back and implemented a hash table implementation for our dictionary. Again, it worked
great up until the moment we ran it.
``Our program is still too slow,'' Dimitris complained. ``Sure, it is now about ten times faster on strings
of length 2,000. So now we can get up to about 4,000 characters. Big deal. We will never get up to
50,000.''
``We should have expected only a factor ten speedup,'' I mused. ``After all, . We need a
faster data structure to search in our dictionary of strings.''
``But what can be faster than a hash table?'' Dimitris countered. ``To look up a k-character string, you
must read all k characters. Our hash table already does O(k) searching.''
``Sure, it takes k comparisons to test the first substring. But maybe we can do better on the second test.
Remember where our dictionary queries are coming from. When we concatenate ABCD with EFGH, we
are first testing whether BCDE is in the dictionary, then CDEF. These strings differ from each other by
only one character. We should be able to exploit this so that each subsequent test takes constant time to
perform....''
``We can't do that with a hash table,'' Dimitris observed. ``The second key is not going to be anywhere
near the first in the table. A binary search tree won't help, either. Since the keys ABCD and BCDE differ
according to the first character, the two strings will be in different parts of the tree.''
file:///E|/BOOK/BOOK/NODE39.HTM (3 of 5) [19/1/2003 1:28:38]