Sparse weighted voting classifier selection and its linear programming relaxations

Noam Goldberg; Jonathan Eckstein

doi:10.1016/j.ipl.2012.03.004

Sparse weighted voting classifier selection and its linear programming relaxations

Noam Goldberg, Jonathan Eckstein

Research output: Contribution to journal › Article › peer-review

13 Scopus citations

Abstract

We consider the problem of minimizing the number of misclassifications of a weighted voting classifier, plus a penalty proportional to the number of nonzero weights. We first prove that its optimum is at least as hard to approximate as the minimum disagreement halfspace problem for a wide range of penalty parameter values. After formulating the problem as a mixed integer program (MIP), we show that common "soft margin" linear programming (LP) formulations for constructing weighted voting classsifiers are equivalent to an LP relaxation of our formulation. We show that this relaxation is very weak, with a potentially exponential integrality gap. However, we also show that augmenting the relaxation with certain valid inequalities tightens it considerably, yielding a linear upper bound on the gap for all values of the penalty parameter that exceed a reasonable threshold. Unlike earlier techniques proposed for similar problems (Bradley and Mangasarian (1998) [4], Weston et al. (2003) [14]), our approach provides bounds on the optimal solution value.

Original language	English
Pages (from-to)	481-486
Number of pages	6
Journal	Information Processing Letters
Volume	112
Issue number	12
DOIs	https://doi.org/10.1016/j.ipl.2012.03.004
State	Published - 30 Jun 2012
Externally published	Yes

Bibliographical note

Funding Information:
This material is based upon work funded in part by the U.S. Department of Homeland Security under Grant Award Number 2008-DN-077-ARI001-02, the Daniel Rose Technion-Yale Initiative for Research on Homeland Security and Counter-Terrorism, and the Council of Higher Education, State of Israel. We thank Rob Schapire for helpful discussions, and also thank the anonymous referees for comments that helped improve the presentation of these results. The first author would also like to thank Martin Milanic, Ilan Newman, and Asaf Levin for their comments.

Funding

This material is based upon work funded in part by the U.S. Department of Homeland Security under Grant Award Number 2008-DN-077-ARI001-02, the Daniel Rose Technion-Yale Initiative for Research on Homeland Security and Counter-Terrorism, and the Council of Higher Education, State of Israel. We thank Rob Schapire for helpful discussions, and also thank the anonymous referees for comments that helped improve the presentation of these results. The first author would also like to thank Martin Milanic, Ilan Newman, and Asaf Levin for their comments.

Funders	Funder number
Council of Higher Education, State of Israel
Daniel Rose Technion-Yale Initiative for Research on Homeland Security
U.S. Department of Homeland Security	2008-DN-077-ARI001-02

Keywords

Computational complexity
Hardness of approximation
Integrality gap
Machine learning
Sparsity
Weighted voting classification

Access to Document

10.1016/j.ipl.2012.03.004

Cite this

@article{218899265470484bb7324711c7d14f66,

title = "Sparse weighted voting classifier selection and its linear programming relaxations",

abstract = "We consider the problem of minimizing the number of misclassifications of a weighted voting classifier, plus a penalty proportional to the number of nonzero weights. We first prove that its optimum is at least as hard to approximate as the minimum disagreement halfspace problem for a wide range of penalty parameter values. After formulating the problem as a mixed integer program (MIP), we show that common {"}soft margin{"} linear programming (LP) formulations for constructing weighted voting classsifiers are equivalent to an LP relaxation of our formulation. We show that this relaxation is very weak, with a potentially exponential integrality gap. However, we also show that augmenting the relaxation with certain valid inequalities tightens it considerably, yielding a linear upper bound on the gap for all values of the penalty parameter that exceed a reasonable threshold. Unlike earlier techniques proposed for similar problems (Bradley and Mangasarian (1998) [4], Weston et al. (2003) [14]), our approach provides bounds on the optimal solution value.",

keywords = "Computational complexity, Hardness of approximation, Integrality gap, Machine learning, Sparsity, Weighted voting classification",

author = "Noam Goldberg and Jonathan Eckstein",

note = "Funding Information: This material is based upon work funded in part by the U.S. Department of Homeland Security under Grant Award Number 2008-DN-077-ARI001-02, the Daniel Rose Technion-Yale Initiative for Research on Homeland Security and Counter-Terrorism, and the Council of Higher Education, State of Israel. We thank Rob Schapire for helpful discussions, and also thank the anonymous referees for comments that helped improve the presentation of these results. The first author would also like to thank Martin Milanic, Ilan Newman, and Asaf Levin for their comments.",

year = "2012",

month = jun,

day = "30",

doi = "10.1016/j.ipl.2012.03.004",

language = "אנגלית",

volume = "112",

pages = "481--486",

journal = "Information Processing Letters",

issn = "0020-0190",

publisher = "Elsevier B.V.",

number = "12",

}

TY - JOUR

T1 - Sparse weighted voting classifier selection and its linear programming relaxations

AU - Goldberg, Noam

AU - Eckstein, Jonathan

N1 - Funding Information: This material is based upon work funded in part by the U.S. Department of Homeland Security under Grant Award Number 2008-DN-077-ARI001-02, the Daniel Rose Technion-Yale Initiative for Research on Homeland Security and Counter-Terrorism, and the Council of Higher Education, State of Israel. We thank Rob Schapire for helpful discussions, and also thank the anonymous referees for comments that helped improve the presentation of these results. The first author would also like to thank Martin Milanic, Ilan Newman, and Asaf Levin for their comments.

PY - 2012/6/30

Y1 - 2012/6/30

N2 - We consider the problem of minimizing the number of misclassifications of a weighted voting classifier, plus a penalty proportional to the number of nonzero weights. We first prove that its optimum is at least as hard to approximate as the minimum disagreement halfspace problem for a wide range of penalty parameter values. After formulating the problem as a mixed integer program (MIP), we show that common "soft margin" linear programming (LP) formulations for constructing weighted voting classsifiers are equivalent to an LP relaxation of our formulation. We show that this relaxation is very weak, with a potentially exponential integrality gap. However, we also show that augmenting the relaxation with certain valid inequalities tightens it considerably, yielding a linear upper bound on the gap for all values of the penalty parameter that exceed a reasonable threshold. Unlike earlier techniques proposed for similar problems (Bradley and Mangasarian (1998) [4], Weston et al. (2003) [14]), our approach provides bounds on the optimal solution value.

AB - We consider the problem of minimizing the number of misclassifications of a weighted voting classifier, plus a penalty proportional to the number of nonzero weights. We first prove that its optimum is at least as hard to approximate as the minimum disagreement halfspace problem for a wide range of penalty parameter values. After formulating the problem as a mixed integer program (MIP), we show that common "soft margin" linear programming (LP) formulations for constructing weighted voting classsifiers are equivalent to an LP relaxation of our formulation. We show that this relaxation is very weak, with a potentially exponential integrality gap. However, we also show that augmenting the relaxation with certain valid inequalities tightens it considerably, yielding a linear upper bound on the gap for all values of the penalty parameter that exceed a reasonable threshold. Unlike earlier techniques proposed for similar problems (Bradley and Mangasarian (1998) [4], Weston et al. (2003) [14]), our approach provides bounds on the optimal solution value.

KW - Computational complexity

KW - Hardness of approximation

KW - Integrality gap

KW - Machine learning

KW - Sparsity

KW - Weighted voting classification

UR - http://www.scopus.com/inward/record.url?scp=84859523996&partnerID=8YFLogxK

U2 - 10.1016/j.ipl.2012.03.004

DO - 10.1016/j.ipl.2012.03.004

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???

AN - SCOPUS:84859523996

SN - 0020-0190

VL - 112

SP - 481

EP - 486

JO - Information Processing Letters

JF - Information Processing Letters

IS - 12

ER -

Sparse weighted voting classifier selection and its linear programming relaxations

Abstract

Bibliographical note

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this