We consider the problem of minimizing the number of misclassifications of a weighted voting classifier, plus a penalty proportional to the number of nonzero weights. We first prove that its optimum is at least as hard to approximate as the minimum disagreement halfspace problem for a wide range of penalty parameter values. After formulating the problem as a mixed integer program (MIP), we show that common "soft margin" linear programming (LP) formulations for constructing weighted voting classsifiers are equivalent to an LP relaxation of our formulation. We show that this relaxation is very weak, with a potentially exponential integrality gap. However, we also show that augmenting the relaxation with certain valid inequalities tightens it considerably, yielding a linear upper bound on the gap for all values of the penalty parameter that exceed a reasonable threshold. Unlike earlier techniques proposed for similar problems (Bradley and Mangasarian (1998) , Weston et al. (2003) ), our approach provides bounds on the optimal solution value.
Bibliographical noteFunding Information:
This material is based upon work funded in part by the U.S. Department of Homeland Security under Grant Award Number 2008-DN-077-ARI001-02, the Daniel Rose Technion-Yale Initiative for Research on Homeland Security and Counter-Terrorism, and the Council of Higher Education, State of Israel. We thank Rob Schapire for helpful discussions, and also thank the anonymous referees for comments that helped improve the presentation of these results. The first author would also like to thank Martin Milanic, Ilan Newman, and Asaf Levin for their comments.
- Computational complexity
- Hardness of approximation
- Integrality gap
- Machine learning
- Weighted voting classification