Committee-Based Sample Selection for Probabilistic Classifiers

Shlomo Argamon-Engelson, Ido Dagan

Research output: Contribution to journalArticlepeer-review

93 Scopus citations


In many real-world learning tasks it is expensive to acquire a sufficient number of labeled examples for training. This paper investigates methods for reducing annotation cost by sample, selection. In this approach, during training the learning program examines many unlabeled examples and selects for labeling only those that are most informative at each stage. This avoids redundantly labeling examples that contribute little new information. Our work follows on previous research on Query By Committee, and extends the committee-based paradigm to the context of probabilistic classification. We describe a family of empirical methods for committee-based sample selection in probabilistic classification models, which evaluate the informativeness of an example by measuring the degree of disagreement between several model variants. These variants (the committee) are drawn randomly from a probability distribution conditioned by the training set labeled so far. The method was applied to the real-world natural language processing task of stochastic part-of-speech tagging. We find that all variants of the method achieve a significant reduction in annotation cost, although their computational efficiency differs. In particular, the simplest variant, a two member committee with no parameters to tune, gives excellent results. We also show that sample selection yields a significant reduction in the size of the model used by the tagger.

Original languageEnglish
Pages (from-to)335-360
Number of pages26
JournalJournal of Artificial Intelligence Research
StatePublished - 1999


Dive into the research topics of 'Committee-Based Sample Selection for Probabilistic Classifiers'. Together they form a unique fingerprint.

Cite this