a1 Introd uction
In this paper we shall present a promising approach to a classic search problem: given a single word w and a large set of words W , quickly deciding which of the words in W most closely resembles w , measured by some metric of similarity, such as minimum edit distance.
Performing this type of search quickly is important for many applications, not least in natural language processing. Spelling correctors, Optical Character Recognition applications and syntactic parsers, among others, all rely on quick approximate matching of some string against a pattern of strings.
The standard edit distance algorithm given in most textbooks on the subject does not scale well to large search problems. Certainly, calculating the edit distance between two individual words can be done quickly, even with diﬀerent costs for diﬀerent character substitutions and insertions. The ubiquitous textbook algorithm for ﬁnding the minimum edit distance (MED) between two words based on the dynamic programming method takes quadratic time in the length of the longer word.
However, ﬁnding the closest match between an input word w and a list of, say 1,000,000 words, is a much more demanding task. The strategy of calculating the edit distance between every word on the list and theinput word is likely to take too much time. Naturally, if we perform the search by cal-
culating the edit distance between each word on a list and our target word w , there are a number of possible optimizations that can be made, such as aborting the calculation for a word-pair as soon as the MED comparison exceeds the lowest MED encountered so far. Even so, the search space remains too large for this strategy to be of practical use.
1.1 A more general problem
In addressing this problem, we shall investigate a much more general problem: that of ﬁnding the closest approximate match between a word w and words encoded in ﬁnitestate automaton A . The problem is more general because a ﬁnite automaton can not only encode a ﬁnite number of words (as a deterministic acyclic automaton), but an inﬁnite number of words, or a ‘pattern.’
Considering the problem of ﬁnding an approximate match to word w against some automaton A instead of a word list L has many advantages. First, a ﬁnite word list can always be converted into an acyclic deterministic ﬁnite automaton. This means that a good solution to the problem of matching against an automaton is also a good solution to the word list case. Second, a ﬁnite automaton can represent an inﬁnite number of strings— i.e. a pattern of strings—by virtue of th
eISSN: 1135-5948 © 2009 Sociedad Española para el Procesamiento del Lenguaje Natural
fact that it can contain cycles. Some natural languages, for instance, allow very long compound words, the patterns of which can be compactly modeled as a cyclic automaton. Third: natural language morphological analyzers are often implemented as ﬁnite-state transducers (Beesley and Karttunen, 2003). Creating a deterministic minimal ﬁnite-state automatonbyextractingthedomainorrange from a ﬁnite-state transducer is trivial: one simply ignores the input or the output labels, and determinizes and minimizes the resulting automaton. This means, for instance, that if one has access to a morphology of a language representedasatransducer, thatmorphology can easily be used as a spelling corrector.
2 Solution based o n informed search
1Most previous solutions to this problem have been based on simple brute-force breadth or depth-ﬁrst search through the automaton A , comparing paths in the automaton with the wordathandandabortingsearchesonapath as soon as the required number of changes in thewordatapointinthesearchreachessome speciﬁed cutoﬀ.
Our observation, however, has been that ﬁnite-state automata contain useful information that can be extracted cheaply and used proﬁtably as a reliable guide in the search process, avoiding a brute-force strategy and the exponential growth of the search complexity it entails. In particular, as we shall show, one piece of information that can be extracted from an automaton with little effort is knowledge about what kinds of symbols can be encountered in the future. That is, for each state in an automaton A , we can extract information about all the possible future symbols that can be encountered for the next n steps, given any n .
When performing searches with some type of additional information to guide the search process, the preferred family of algorithms to use is usually some variant of the A*algorithm (Hart, Nilsson, and Raphael, 1968). This was an obvious choice for us as well. Theadditionalinformationweuse—the
1The most prominent ones given in the literature are Oﬂazer (1996) who presents a depth-ﬁrst search algorithm, and Schulz and Mihov (2002), which is essentially the same algorithm.possible symbols that can be seen in future paths that extend out from some state in the automaton—can function as a guess about proﬁtable paths to explore, and thus ﬁts well in the A*-paradigm. The A*-algorithm essentially requires that we have access to two types of costs during our search: a cost of the path in a partial exploration (g ), and a guess about the future cost (h ), which may not be an overestimate for the heuristic to be admissible. At every stepofchoosingwhichpartialpathtoexpand next, we take into account the combined actual cost so far (g ), and the guess (h ), yielding f = g +h .
Searching an automaton for matches against a word with edit distance naturally yieldsg ,whichisthenumberofchangesmade so far in reaching a state s in an automaton comparing against the word w . The guess h is based on the heuristic we already brieﬂy introduced.
2.2 The search algorithm
For our initial experiments, we have considered the restricted problem of ﬁnding the shortest Levenshtein distance between w and paths in an automaton A . This is the case of MED where insertion, substitution, and deletion all cost 1 unit. However, the algorithm can easily be extended to capture varying insertion, substitution, and deletion costs through so-called confusion matrices, and even to context-dependent costs.
Essentially, the search begins with a single node represented by the start state and the word at position 0 (the starting position). We then consider each possible edge in the automaton from the state v . If the edge matches the current word position symbol, we create a new node by advancing to the target state v, advancing pos (the word position counter pos) by 1, recalculating the costs f = g +h and storing this as a new node to the agenda marked as x:x (which indicates no change was made). We also consider the cases where we insert a symbol (0:x), delete a symbol (x:0), and (if the edge currently inspected does not match the symbol in the word at position pos) substituting a symbol (x:y) with another one. When we are done with the node, we ﬁnd the node with the lowest score so far, expand that node, and keep going until we ﬁnd a solution. See ﬁgure 1 for a partially expanded search of the wor
d58 Procesamiento del Lenguaje Natural, núm. 43 (2009)
f=3 g=1 h=2d:f[a,c,d,f,o,r] [o,g]
7Input word: datstate=0
f=1 g=1 h=0
f=4 g=1 h=3 pos=0 state=1 d:0
f=3 g=1 h=2
f=1 g=1 h=00:d
fstate=3[s][a,c,d,f,g,o,r,s,t] [o,g,s] [g,s]
f = 0 g = 0 h =0 pos = 0 state = 0pos=1 state=2
f=2 g=0 h=2
f=2 g=1 h=1 pos=0d:d
Figure 2: An automaton where information about the possible symbols of future paths of length n (for the case n =2) is stored in each state
Figure 1: Initial steps in searching for the approximate match against the word dat and the automaton depicted in ﬁgure 3. The node withtheoutgoingarrowwillbeexpandednext. To reach that node from the initial node, we moved from state 0 to state 3 in the automaton, and thus had to substitute a d in the input for a c in the automaton, costing one unit, reﬂected in the g -value. The h value for that node is 0 since all the subsequent letters in the remainder of the word (a and t are in the state’s symbol table. For this illustration, we assume the heuristic h (8 ), seen in ﬁgure 3.
dat against the automata in ﬁgure 3.
3 The heuristics
There are two obvious criteria for creating a useful heuristic h for the approximate search through a graph: the heuristic must be fast to calculate, either beforehand or on the ﬂy, and, if calculated prior to the search, it must be compact—i.e. take up a linear amount of space proportional to the number of states in A we want to search.Figure 3: An automaton where information about the possible symbols of future paths of length n (for the case n = 8 ) is stored in each state. Calculating this for the set of states is accomplished in linear time by algorithm 2.1.
As already mentioned, we have chosen as our heuristic function to store, for each state, a list of symbols that can be seen somewhere along a path of length n starting from that state. Figures 2 and 3 illustrate this by two acyclic ﬁnite automata that encode a number of words, and where every state has been marked with information about all possible future symbols n steps ahead, where n is 2 (ﬁgure 2), and 8 (ﬁgure 3).
During the search of the graph this list is consulted and compared against the symbols in the current word we have yet to match. Thediscrepanciesbetweenthesymbolsinth
e59 Procesamiento del Lenguaje Natural, núm. 43 (2009)
state and the symbols yet to be matched in the current word is used as our cost heuristic. That is, for each of a maximum of n symbols remaining to be matched in the word w at the current position of the search, for any symbol not found stored in the current state, we add a cost of 1 to the heuristic function h . Thisreﬂectsanestimate(whichiscorrect) that some time in the future any symbols not present in the path we are exploring will have to be either considered substitutions or deletions with respect to the input word.
For example, suppose we were matching the word cag r at position 1, i.e. the symbols remaining to be matched are ag r , and supposeweareinthestatemarkedwithanasteriskinﬁgures2and3. Nowusingourheuristic function where n is 2 (ﬁgure 2), gives us an estimated h -value of 1, since the two following symbols to be matched are a and g , and the state marked with an asterisk does not contain g . For the same position, using the heuristic where n is 8 (ﬁgure 3), the h -value becomes 2, since neither g nor r are found in subsequent paths, as is seen from the list of symbols stored in the state.
3.1 Consistency of h
It is easy to see that h for any n is an admissible heuristic for A*2-search in that it never overestimates the cost to reach the goal.
3Certainly if there is a discrepancy of i symbols between future paths and the remainder of the word for some number of steps, those symbols will have to be produced by insertion or substitution (each of which cost 1 in our model), and hence h cannot overestimate the cost to the goal.
This meansthat goals inthe search will be found in order of cost, which is of course desirable, since we know that the ﬁrst solution we ﬁnd is the shortest one. Naturally, we can keep exploring and ﬁnd more solutions if desired in increasing order of cost. For a spell checking application, for instance, we would keep the search going until either we reach some cutoﬀ where the cost has gotten too high, or we ﬁnd some desired number of
2*See e.g. Pearl (1984) for extensive discussions ab out what kinds of heuristics are suitable for A.
3)+ h (pThe estimate h is also co nsistent in that it fulﬁls the triangle equality h (p ) = cost(p, a, p), i.e. the estimated cost to the goal is alway s smaller than the combined actual cost to any p oint preachable from p and the new estimated cost from pto the goal.approximate matches to suggest as corrected spellings.
4 P recalculating h
For any serious application we will want to precalculate, for each state, the symbols that can be subsequently matched along some path of length n .
For any ﬁnite n this is a straightforward task: for each state we simply perform a depth-ﬁrst search with a depth threshold of n states, and store all symbols we encounter on an edge in the state we started the search from. This procedure is given in Algorithm 2.2.
The case where n = 8 is more diﬃcult. Here we want to know, for each state, all the symbols that can possibly be encountered in the future, no matter how long the path. If we knew that the automaton we are dealing with were acyclic, this could be solved—although at a cost of some computationtime—bythesamealgorithmasforﬁnite n by limiting searches to the number of states in A . Since we do not want to limit ourselves to searching acyclic graphs only, we have developed a separate algorithm for marking future symbols on the states in the case where n = 8.
4.1 The sp ecial case n =8The solution for this case is based on the observation that we can divide an automatoninto itsstronglyconnected components— groups of states where each state has a path to each other in the group. Naturally, all states in the same strongly connected component (SCC) share the same future symbols for n = 8 .4Interestingly, we do not need to split the graph into its SCCs ﬁrst, and then mark the states. We can in fact do both at the same time—calculate the SCC of the graph, and mark each state with the possible symbols that can follow. We do so by an adaptation ofthewell-knownalgorithmbyTarjan(1972) for dividing a graph into SCCs. The core of Tarjan’s original algorithm is to perform, by recursion, a depth-ﬁrst search (DFS) on the graph, while keeping a separate stack to store the vertices of the search performed so far. In addition, each vertex is given an index (low) to mark when it was discovered.
4A very thorough analysis of this remarkably simple algorithm is given in Aho, Hop croft, and Ullma
n60 Procesamiento del Lenguaje Natural, núm. 43 (2009)
Fast approximate string matching with finite automata
We now turn to the question of adapting this algorithm to mark each state with all its possible future symbols, presented in Algorithm 2.1. The crux to the marking the states at the same time as performing the SCC DFS is that a) for each edge between v and vdiscovered, the parent vertex v inherits the symbols at vas well as the edge symbol e.sy m used to get from v to v, and b) after we encounter the root of a SCC, we copy the properties of the root vertex v to all the child vertices v.h0h1h2
15 Choosing n
Havingnowvariouspossiblen atourdisposal to use as the guiding heuristic in the search of solutions, the question about which n ,or which combinations of n to use as a heuristic is largely an empirical one. Our tests indicate that using a combination of n = 2 and n = 8 is far superior to any other combination of n -values. When used in combination, the algorithm always chooses the larger of the two guesses, Max (h (2),h(8 )), as the h -value.
Naturally, using two heuristics requires two symbol vectors to be stored for each state, where each vector takes up |S| bits of space, for a sum total of 2|S||A| bits, where |A| is the number of states in the automaton and |S| the alphabet size of the automaton. Given that word lists can be compressed into a very small number of states, the combined h is probably the best choice even though it takes up twice as much storage as a single heuristic. If a single heuristic is needed because of space concerns, our results indicate that h (8 ) is the best overall choice, although for applications where a small number of errors are expected, such as spelling correction, n =2orn = 3 may be slightly better.
Table 1 shows the number of nodes expanded as well as the number of nodes inserted on the agenda of the search graph (by Algorithm 2.3) for 100 randomly chosen misspelled words (all with shortest edit distance 3 or less), using various combinations of h , against a dictionary of 556,368 words.
We used these results (because they most likely reﬂect average use of the algorithm) to settle for using the combination of Max (h (2),h(8)) as our best heuristic h .
(1974).NI 6092 1892 1548 1772 1904 622 NE 3295 1143 909 1049 1193 89
Table 1: Average number of nodes inserted on the agenda (NI) and nodes expanded (NE) when performing a search of 100 randomly perturbed words against a dictionary of Spanish containing 556,368 words, encoded as a ﬁnite-state automaton. Each search terminatedafterﬁndingthe5closestmatches. The average length of the words was 6.6 letters and the average edit distance to each match was 2.18. The diﬀerent heuristics used were h0: no heuristic, i.e. h is always zero; h2: only consider n = 8; h: only consider n =2; h3: only consider n =3; h, only consider n =4, and h: the heuristic was max(n =2,n = 8) and ties were resolved in the priority queue by giving priority to the highest word position. The priority queue strategy for all except h5was purposely adversarial to the search: i.e. nodes expanded were extracted LILO in the case of f -score ties. Note that the nodes expanded measure is much more important since the time to insert a node in a priority queue is generally constant, while the time taken to extract the minimum is on the order of log (n ), with n being the number of elements in the queue.
5.1 Priority queue strategy
The number of nodes explored before ﬁnding solutions is aﬀected not only by the particular heuristic used, but also by the choice of tiebreaking in the search strategy. In many cases, we will ﬁnd unexpanded fringe nodes in the priority queue with the exact same f value. How to choose a node for expansion in the case of ties has great bearing on the speed of the search as well, as is seen in table 1. In our experiments, the strategy of breaking ties so that the node that has advanced farthest in the word to be matched is given priority has proven to be far superior to any of the tiebreaking strategies we explored.
The reason for this seems natural: if we are exploring possible approximate matches, then, other things being equal (i.e. the f score), one should follow the path that has moved the longest toward the end of the word.
61 Procesamiento del Lenguaje Natural, núm. 43 (2009
foma: regex Spanish2; 47432 states, 126535 arcs, 556368 paths. foma: apply med apply med> wuighuiwrwegfwfw
-sig-ui´er-e--mos wuighuiwrwegfwfw Cost[f]: 10
elapsed time: 0.100000s Nodes inserted: 56293 Nodes expanded: 20549 apply med> 0 1000 2000 3000 4000 5000 6000 7000 8000Average times for MED search for two algorithms Current algorithmSchulz & Mihov
1 2 3 4 5 6 7 8 9 10
DFigure 4: Example search against a dictionary in our implementation.
We have implemented the algorithms described as an extension to the freely availableﬁnitestatetoolkit, foma (Hulden, 2009). The algorithms were written in C. We tested them against various automata, including a lexicon of English (represented as a cyclic ﬁnite-state automaton), and several acyclic lexicons of Spanish of between 500,000 and 1,000,000 words. The timing tests in ﬁgure 5 were achieved by generating 100 random words of each minimum edit distance 1 through 10 from one of our lexica, and then searching for the closest match with the algorithm. Although discovery of closest matches of edit distance more than 4 are probablynotusefulforspellingcorrectionapplications, other applications (such as Optical Character Recognition) may need to ﬁnd closest matches that take into account far more perturbances than spelling correction applications need. For instance, ﬁgure 4 depicts our interface in the search of a word that results in a match of edit distance 10, where clearly the match would not be useful for spelling correction since the input word is mangled beyond recognition from its closest match in the dictionary.
7 Discussion and further work
The main result that the algorithm points to is that an informed search strategy such as we have outlined above is clearly superior to any uninformed strategy when searching word graphs such as ﬁnite automata for approximate matches. In particular, the heuristic of the strategy we have used is fast to compute ahead of time—taking only linearFigure 5: Average wall clock times in our implementation of the new algorithm for ﬁnding minimum edit distances for 1000 words of edit distance, compared to running a Schulz and Mihov (2002) search on the same words. The 1000 words consisted of 100 words of each edit distance, i.e. 100 words in each group 1 through 10. The automaton which the randomly generated words were matched against was the FreeLing Spanish dictionary of 556,368 words which we ﬁrst converted into a deterministic acyclic automaton.
timeandspaceinthesizeoftheautomaton— and often results in a dramatic reduction of the search space that needs to be explored in order to ﬁnd approximate matches. For applications such as spelling correction, the method is very space and time-eﬃcient, since one can encode large ﬁnite lists of words quite compactly as a ﬁnite automaton. Finding matches for minimum edit distance 5 in all our experiments against large dictionaries never took more than 70 milliseconds. For edit distances 1–3, the granularity of the system clock (one millisecond) was such that the timing results were always 0. As such, our implementation is already practically usable for many applications.
It also appears that the new strategy scales well to much larger problems. The seemingly exponential growth in the search time seen toward the end of the graph in ﬁgure 5 is probably partly an artifact resulting from the large number of errors in the word and the fact that most of the words in the automaton we matched against were much shorter than the input words were. That is, since almost every character in the words we testedthatwereofMED9or10wasanerror, there seems to be no way to avoid exploring a very large part of the entire search space. However, in preliminary tests against lexica that contain much longer words, this growth
62 Procesamiento del Lenguaje Natural, núm. 43 (2009)average time(msec
Fast approximate string matching with finite automata
does not occur as quickly.
The fastest previously known method (to us) for performing the same type of approximate search, that of Schulz and Mihov (2002), which we had used before and was easily available in foma, shows much poorer performance as seen in the graph in ﬁgure 5. Implementations may of course vary in practical details and so an exact comparison is diﬃcult. However, in comparing our algorithm to Schulz & Mihov we tried to minimize this disturbance in two ways: 1) Schulz & Mihov perform a preliminary step of constructing what they call a Levenshtein automaton for some edit distance n and the input word beforehand; we have not included this construction time in our ﬁgures; 2) Schulz & Mihov is essentially a simultaneous depth-ﬁrst search of two graphs, the Levenshtein automaton and the lexicon automaton A , and this search is performed on the exact same data structure as in the new algorithm (the internal automaton data structure of foma). These two points we believe make the algorithm comparison more meaningful than it would under other circumstances.
7.2 Complex distance metrics
All our experiments with the approximate matching were done with Levenshtein edit distance. But the algorithm can easily be adapted to a variety of distance metrics. One ofthemoreusefulmetricsistocalculatecosts from a confusion matrix where each individual substitution, insertion, or deletion can have a diﬀerent cost. For instance, for Spanish spelling correction, one would want to specify lower costs than ordinarily for substituting c for s, and b for v because of their phonetic proximity or equality for speakers.
One can also concoct context-dependent cost schemes. Again, drawing on an example for Spanish, it may be beneﬁcial to give a lowcosttosubstitutingl asy, butonlyifpreceded by a deletion of an l . This would represent the replacement of ll with y , as these are phonetically similar for many speakers. For all such extensions, the only modiﬁcation the basic algorithm needs is to keeping track of the h -score estimate given that the costs may diﬀer for diﬀerent operations of word perturbation.8 Conclusion
We have presented an algorithm for ﬁnding approximate matches of a word to paths in a ﬁnite automaton. The algorithm is a variant of the popular A*search algorithm where the main heuristic to guide the search is to record, for each state in the automaton, every symbol that can be encountered in paths extending from that state for n steps. Precalculating this for the special case where n = 8 proved to be an interesting problem in its own right, which we solved by adapting Tarjan’s(1972)algorithmforﬁndingstrongly connected components in a graph in linear time.
We expect the algorithm to be useful for a number of purposes, and have shown that it performs very eﬃciently in suggesting spelling corrections for misspelled word, even against very large dictionaries.
Aho, Alfred V., John E. Hopcroft, and Jeffrey D. Ullman. 1974. The Design and Analysis of Computer Algorithms. Addison-Wesley.
Beesley, Kenneth and Lauri Karttunen. 2003. Finite-State Morphology. CSLI, Stanford.
Hart, P.E., N.J. Nilsson, and B. Raphael. 1968. A formal basis for the heuristic determinationofminimumcostpaths. IEEE transactions on Systems Science and Cybernetics, 4(2):100–107.
Hulden, Mans. 2009. Foma: a ﬁnite-state compiler and library. In EACL 2009 Proceedings, pages 29–32.
Oﬂazer, Kemal. 1996. Error-tolerant ﬁnite-state recognition with applications to morphological analysis and spelling correction. Computational Linguistics, 22(1):73–89.
Pearl, Judea. 1984. Heuristics: intelligent search strategies for computer problem solving. Addison-Wesley.
Schulz, Klaus and Stoyan Mihov. 2002. Fast string correction with Levenshtein automata. International Journal on Document Analysis and Recognition, 5(1):67– 85.
Tarjan, Robert E. 1972. Depth-First Search and Linear Graph Algorithms. SIAM Journal on Computing, 1(2):146–160
.63 Procesamiento del Lenguaje Natural, núm. 43 (2009)
Algorithm 2.1: MarkVinf (v ) v .index ← indexv.low ← index index ← index +1 Push (v ) for each edge(v → v⎧ ⎪)v .sy ms ← v .sy ms ∪ v.sy ms ∪ edge.sym if v .index =0thenMarkVinf (v) v.low ← Min (v.low,
v.l ow ) else if v is on stack then v.low ← Min (v.low, v
rep eat Pop(v⎧ ⎪⎪ ⎨⎪ ⎪ ⎩) v.sy ms ← v .sy ms until v =
⎧ ⎨⎩vhead.syms ← vhead.syms∪ edge LimitDFS (v) d ← d -
1do⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩.index) if v.low = v .index
Algorithm 2.2: MarkVfin (V, n) for each vhead ∈ Vd ← 0 LimitDFS (vhead
)LimitDFS (v ) d ← d +1 if d>n
d ← d - 1 Return
for each edge(v → v
)Algorithm 2.3: Search (word, A) agenda(v,pos,g,h) ← (start state, 0, 0, cal cul ate h ())while agenda not empty
(pos,v,cost) ← remove-cheapest(agenda) for each edge(v → v⎧ ⎪do⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩do)with sym e
⎧ ⎪⎪ ⎪ ⎪ ⎪ ⎪ ⎨⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩if v is ﬁnal and pos is end of word then Solution (node )if word(pos) = edge then Add (v, pos +1, cost) else Add (v, pos +1, cost +substcost, cal cul ate h ()) Add (v, pos, cost +del cost, cal cul ate h ()) Add (v , pos +1, cost +inscost, cal cul ate h ())
64 Procesamiento del Lenguaje Natural, núm. 43 (2009