Adversarial laws of large numbers and optimal regret in online classification

Noga Alon; Omri Ben-Eliezer; Yuval Dagan; Shay Moran; Moni Naor; Eylon Yogev

doi:10.1145/3406325.3451041

Adversarial laws of large numbers and optimal regret in online classification

Noga Alon
, Omri Ben-Eliezer
, Yuval Dagan
, Shay Moran
, Moni Naor
, Eylon Yogev

Department of Computer Science and Artificial Intelligence

Princeton University
Tel Aviv University
Harvard University
Massachusetts Institute of Technology
Technion-Israel Institute of Technology
Alphabet Inc.
Weizmann Institute of Science
Boston University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

46 Scopus citations

Abstract

Laws of large numbers guarantee that given a large enough sample from some population, the measure of any fixed sub-population is well-estimated by its frequency in the sample. We study laws of large numbers in sampling processes that can affect the environment they are acting upon and interact with it. Specifically, we consider the sequential sampling model proposed by Ben-Eliezer and Yogev (2020), and characterize the classes which admit a uniform law of large numbers in this model: these are exactly the classes that are online learnable. Our characterization may be interpreted as an online analogue to the equivalence between learnability and uniform convergence in statistical (PAC) learning. The sample-complexity bounds we obtain are tight for many parameter regimes, and as an application, we determine the optimal regret bounds in online learning, stated in terms of Littlestone's dimension, thus resolving the main open question from Ben-David, Pál, and Shalev-Shwartz (2009), which was also posed by Rakhlin, Sridharan, and Tewari (2015).

Original language	English
Title of host publication	STOC 2021 - Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing
Editors	Samir Khuller, Virginia Vassilevska Williams
Publisher	Association for Computing Machinery
Pages	447-455
Number of pages	9
ISBN (Electronic)	9781450380539
DOIs	https://doi.org/10.1145/3406325.3451041
State	Published - 15 Jun 2021
Event	53rd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2021 - Virtual, Online, Italy Duration: 21 Jun 2021 → 25 Jun 2021

Publication series

Name	Proceedings of the Annual ACM Symposium on Theory of Computing
ISSN (Print)	0737-8017

Conference

Conference	53rd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2021
Country/Territory	Italy
City	Virtual, Online
Period	21/06/21 → 25/06/21

Bibliographical note

Publisher Copyright:
© 2021 ACM.

Funding

Noga Alon is supported in part by National Science Foundation (NSF) grant DMS-1855464, US-Israel Binational Science Foundation (BSF) grant 2018267, and by the Simons Foundation. Research partially conducted while Omri Ben-Eliezer was at Weizmann Institute of Science, supported in part by a Israel Science Foundation (ISF) grant no. 950/15. Shay Moran is a Robert J. Shillman Fellow and was supported in part by the ISF (grant No. 1225/20), by an Azrieli Faculty Fellowship, and by BSF grant 2018385. Moni Naor is Supported in part by ISF grants (no. 950/15 and 2686/20) and by the Simons Foundation Collaboration on the Theory of Algorithmic Fairness. Incumbent of the Judith Kleeman Professorial Chair. Eylon Yogev is supported in part by ISF grants 484/18, 1789/19, Len Blavatnik and the Blavatnik Foundation, and The Blavatnik Interdisciplinary Cyber Research Center at Tel Aviv University.

Funders	Funder number
Blavatnik Foundation
Simons Foundation Collaboration on the Theory of Algorithmic Fairness	1789/19, 484/18
National Science Foundation	DMS-1855464
Simons Foundation
Bonfils-Stanton Foundation	2018267
Weizmann Institute of Science
United States-Israel Binational Science Foundation
Israel Science Foundation	2686/20, 1225/20, 2018385, 950/15
Tel Aviv University

Keywords

Littlestone dimension
adversarial robustness
online learning
random sampling
robust sampling

Access to Document

10.1145/3406325.3451041

Cite this

Alon, N., Ben-Eliezer, O., Dagan, Y., Moran, S., Naor, M., & Yogev, E. (2021). Adversarial laws of large numbers and optimal regret in online classification. In S. Khuller, & V. V. Williams (Eds.), STOC 2021 - Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing (pp. 447-455). (Proceedings of the Annual ACM Symposium on Theory of Computing). Association for Computing Machinery. https://doi.org/10.1145/3406325.3451041

Alon, Noga ; Ben-Eliezer, Omri ; Dagan, Yuval et al. / Adversarial laws of large numbers and optimal regret in online classification. STOC 2021 - Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing. editor / Samir Khuller ; Virginia Vassilevska Williams. Association for Computing Machinery, 2021. pp. 447-455 (Proceedings of the Annual ACM Symposium on Theory of Computing).

@inproceedings{c934432ad92f42e4b274466c44fb6467,

title = "Adversarial laws of large numbers and optimal regret in online classification",

abstract = "Laws of large numbers guarantee that given a large enough sample from some population, the measure of any fixed sub-population is well-estimated by its frequency in the sample. We study laws of large numbers in sampling processes that can affect the environment they are acting upon and interact with it. Specifically, we consider the sequential sampling model proposed by Ben-Eliezer and Yogev (2020), and characterize the classes which admit a uniform law of large numbers in this model: these are exactly the classes that are online learnable. Our characterization may be interpreted as an online analogue to the equivalence between learnability and uniform convergence in statistical (PAC) learning. The sample-complexity bounds we obtain are tight for many parameter regimes, and as an application, we determine the optimal regret bounds in online learning, stated in terms of Littlestone's dimension, thus resolving the main open question from Ben-David, P{\'a}l, and Shalev-Shwartz (2009), which was also posed by Rakhlin, Sridharan, and Tewari (2015).",

keywords = "Littlestone dimension, adversarial robustness, online learning, random sampling, robust sampling",

author = "Noga Alon and Omri Ben-Eliezer and Yuval Dagan and Shay Moran and Moni Naor and Eylon Yogev",

note = "Publisher Copyright: {\textcopyright} 2021 ACM.; 53rd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2021 ; Conference date: 21-06-2021 Through 25-06-2021",

year = "2021",

month = jun,

day = "15",

doi = "10.1145/3406325.3451041",

language = "אנגלית",

series = "Proceedings of the Annual ACM Symposium on Theory of Computing",

publisher = "Association for Computing Machinery",

pages = "447--455",

editor = "Samir Khuller and Williams, \{Virginia Vassilevska\}",

booktitle = "STOC 2021 - Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing",

}

Alon, N, Ben-Eliezer, O, Dagan, Y, Moran, S, Naor, M & Yogev, E 2021, Adversarial laws of large numbers and optimal regret in online classification. in S Khuller & VV Williams (eds), STOC 2021 - Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing. Proceedings of the Annual ACM Symposium on Theory of Computing, Association for Computing Machinery, pp. 447-455, 53rd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2021, Virtual, Online, Italy, 21/06/21. https://doi.org/10.1145/3406325.3451041

Adversarial laws of large numbers and optimal regret in online classification. / Alon, Noga; Ben-Eliezer, Omri; Dagan, Yuval et al.
STOC 2021 - Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing. ed. / Samir Khuller; Virginia Vassilevska Williams. Association for Computing Machinery, 2021. p. 447-455 (Proceedings of the Annual ACM Symposium on Theory of Computing).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Adversarial laws of large numbers and optimal regret in online classification

AU - Alon, Noga

AU - Ben-Eliezer, Omri

AU - Dagan, Yuval

AU - Moran, Shay

AU - Naor, Moni

AU - Yogev, Eylon

PY - 2021/6/15

Y1 - 2021/6/15

N2 - Laws of large numbers guarantee that given a large enough sample from some population, the measure of any fixed sub-population is well-estimated by its frequency in the sample. We study laws of large numbers in sampling processes that can affect the environment they are acting upon and interact with it. Specifically, we consider the sequential sampling model proposed by Ben-Eliezer and Yogev (2020), and characterize the classes which admit a uniform law of large numbers in this model: these are exactly the classes that are online learnable. Our characterization may be interpreted as an online analogue to the equivalence between learnability and uniform convergence in statistical (PAC) learning. The sample-complexity bounds we obtain are tight for many parameter regimes, and as an application, we determine the optimal regret bounds in online learning, stated in terms of Littlestone's dimension, thus resolving the main open question from Ben-David, Pál, and Shalev-Shwartz (2009), which was also posed by Rakhlin, Sridharan, and Tewari (2015).

AB - Laws of large numbers guarantee that given a large enough sample from some population, the measure of any fixed sub-population is well-estimated by its frequency in the sample. We study laws of large numbers in sampling processes that can affect the environment they are acting upon and interact with it. Specifically, we consider the sequential sampling model proposed by Ben-Eliezer and Yogev (2020), and characterize the classes which admit a uniform law of large numbers in this model: these are exactly the classes that are online learnable. Our characterization may be interpreted as an online analogue to the equivalence between learnability and uniform convergence in statistical (PAC) learning. The sample-complexity bounds we obtain are tight for many parameter regimes, and as an application, we determine the optimal regret bounds in online learning, stated in terms of Littlestone's dimension, thus resolving the main open question from Ben-David, Pál, and Shalev-Shwartz (2009), which was also posed by Rakhlin, Sridharan, and Tewari (2015).

KW - Littlestone dimension

KW - adversarial robustness

KW - online learning

KW - random sampling

KW - robust sampling

UR - https://www.scopus.com/pages/publications/85108173431

U2 - 10.1145/3406325.3451041

DO - 10.1145/3406325.3451041

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85108173431

T3 - Proceedings of the Annual ACM Symposium on Theory of Computing

SP - 447

EP - 455

BT - STOC 2021 - Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing

A2 - Khuller, Samir

A2 - Williams, Virginia Vassilevska

PB - Association for Computing Machinery

T2 - 53rd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2021

Y2 - 21 June 2021 through 25 June 2021

ER -

Alon N, Ben-Eliezer O, Dagan Y, Moran S, Naor M, Yogev E. Adversarial laws of large numbers and optimal regret in online classification. In Khuller S, Williams VV, editors, STOC 2021 - Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing. Association for Computing Machinery. 2021. p. 447-455. (Proceedings of the Annual ACM Symposium on Theory of Computing). doi: 10.1145/3406325.3451041

Adversarial laws of large numbers and optimal regret in online classification

Abstract

Publication series

Conference

Bibliographical note

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this