Distilling Reliable Information From Unreliable Theories

Suppose a domain expert gives us a domain theory which is meant to classify examples as positive or negative examples of some concept. Now suppose, as is often the case, that the expert specifies parts of the theory which might be in need of repair, as opposed to those parts of the theory which are certainly not in need of repair. We say that such a theory is partially mutable. There might be some non-empty set of examples each of which has a classification in the partially mutable theory which is invariant under all possible sets of repairs to unreliable components of the theory. We call such examples stable. We present an efficient algorithm for identifying stable examples for a large class of first-order clausal theories with negation and recursion. We further show how to use stability to arbitrate between the theory and a noisy oracle to improve classification accuracy. We present experimental results on some flawed theories which illustrate the approach.