Breaking the Variance: Approximating the Hamming Distance in 1/∈ Time per Alignment

The algorithmic tasks of computing the Hamming distance between a given pattern P of length m and each location in a text T of length n is one of the most fundamental algorithmic tasks in string algorithms. Karloff [IPL 1999] showed that if one is willing to suffer a 1+∈ approximation, then it is possible to solve the problem with high probability, in Õ(n/∈2) time. Due to related lower bounds for computing the Hamming distance of two strings in the one-way communication complexity model, it is strongly believed that obtaining an algorithm for solving the approximation version cannot be done much faster as a function of 1/∈. We show here that this belief is false by introducing a new Õ(n/∈) time algorithm that succeeds with high probability. The main idea behind our algorithm, which is common in sparse recovery problems, is to reduce the variance of a specific randomized experiment by (approximately) separating heavy hitters from non-heavy hitters. However, while known sparse recovery techniques work very well on vectors, they do not seem to apply here, where we are dealing with mismatches between pairs of characters. We introduce two main algorithmic ingredients. The first is a new sparse recovery method that applies for pair inputs (such as in our setting). The second is a new construction of hash/projection functions, for which have which allows us to count the number of projections that induce mismatches between two characters exponentially faster than brute force. We expect that these algorithmic techniques will be of independent interest.