Approximating probability distributions by RELu networks

Manuj Mukherjee; Aslan Tchamkerten; Mansoor Yousefi

doi:10.1109/ITW46852.2021.9457598

Approximating probability distributions by RELu networks

Manuj Mukherjee, Aslan Tchamkerten, Mansoor Yousefi

Bar-Ilan University - The Alexander Kofkin Faculty of Engineering

Institut Polytechnique de Paris

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

How many neurons are needed to approximate a target probability distribution using a neural network with a given input distribution and approximation error? This paper examines this question for the case when the input distribution is uniform, and the target distribution belongs to the class of histogram distributions. We obtain a new upper bound on the number of required neurons, which is strictly better than previously existing upper bounds. The key ingredient in this improvement is an efficient construction of the neural nets representing piecewise linear functions. We also obtain a lower bound on the minimum number of neurons needed to approximate the histogram distributions.

Original language	English
Title of host publication	2020 IEEE Information Theory Workshop, ITW 2020
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781728159621
DOIs	https://doi.org/10.1109/ITW46852.2021.9457598
State	Published - 11 Apr 2021
Event	2020 IEEE Information Theory Workshop, ITW 2020 - Virtual, Riva del Garda, Italy Duration: 11 Apr 2021 → 15 Apr 2021

Publication series

Name	2020 IEEE Information Theory Workshop, ITW 2020

Conference

Conference	2020 IEEE Information Theory Workshop, ITW 2020
Country/Territory	Italy
City	Virtual, Riva del Garda
Period	11/04/21 → 15/04/21

Bibliographical note

Publisher Copyright:
© 2021 IEEE.

Access to Document

10.1109/ITW46852.2021.9457598

Cite this

@inproceedings{a24185f7f7b84779980f67121563e178,

title = "Approximating probability distributions by RELu networks",

abstract = "How many neurons are needed to approximate a target probability distribution using a neural network with a given input distribution and approximation error? This paper examines this question for the case when the input distribution is uniform, and the target distribution belongs to the class of histogram distributions. We obtain a new upper bound on the number of required neurons, which is strictly better than previously existing upper bounds. The key ingredient in this improvement is an efficient construction of the neural nets representing piecewise linear functions. We also obtain a lower bound on the minimum number of neurons needed to approximate the histogram distributions.",

author = "Manuj Mukherjee and Aslan Tchamkerten and Mansoor Yousefi",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE.; 2020 IEEE Information Theory Workshop, ITW 2020 ; Conference date: 11-04-2021 Through 15-04-2021",

year = "2021",

month = apr,

day = "11",

doi = "10.1109/ITW46852.2021.9457598",

language = "אנגלית",

series = "2020 IEEE Information Theory Workshop, ITW 2020",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2020 IEEE Information Theory Workshop, ITW 2020",

address = "ארצות הברית",

}

Mukherjee, M, Tchamkerten, A & Yousefi, M 2021, Approximating probability distributions by RELu networks. in 2020 IEEE Information Theory Workshop, ITW 2020., 9457598, 2020 IEEE Information Theory Workshop, ITW 2020, Institute of Electrical and Electronics Engineers Inc., 2020 IEEE Information Theory Workshop, ITW 2020, Virtual, Riva del Garda, Italy, 11/04/21. https://doi.org/10.1109/ITW46852.2021.9457598

Approximating probability distributions by RELu networks. / Mukherjee, Manuj; Tchamkerten, Aslan; Yousefi, Mansoor.
2020 IEEE Information Theory Workshop, ITW 2020. Institute of Electrical and Electronics Engineers Inc., 2021. 9457598 (2020 IEEE Information Theory Workshop, ITW 2020).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Approximating probability distributions by RELu networks

AU - Mukherjee, Manuj

AU - Tchamkerten, Aslan

AU - Yousefi, Mansoor

PY - 2021/4/11

Y1 - 2021/4/11

N2 - How many neurons are needed to approximate a target probability distribution using a neural network with a given input distribution and approximation error? This paper examines this question for the case when the input distribution is uniform, and the target distribution belongs to the class of histogram distributions. We obtain a new upper bound on the number of required neurons, which is strictly better than previously existing upper bounds. The key ingredient in this improvement is an efficient construction of the neural nets representing piecewise linear functions. We also obtain a lower bound on the minimum number of neurons needed to approximate the histogram distributions.

AB - How many neurons are needed to approximate a target probability distribution using a neural network with a given input distribution and approximation error? This paper examines this question for the case when the input distribution is uniform, and the target distribution belongs to the class of histogram distributions. We obtain a new upper bound on the number of required neurons, which is strictly better than previously existing upper bounds. The key ingredient in this improvement is an efficient construction of the neural nets representing piecewise linear functions. We also obtain a lower bound on the minimum number of neurons needed to approximate the histogram distributions.

UR - http://www.scopus.com/inward/record.url?scp=85113292729&partnerID=8YFLogxK

U2 - 10.1109/ITW46852.2021.9457598

DO - 10.1109/ITW46852.2021.9457598

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85113292729

T3 - 2020 IEEE Information Theory Workshop, ITW 2020

BT - 2020 IEEE Information Theory Workshop, ITW 2020

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2020 IEEE Information Theory Workshop, ITW 2020

Y2 - 11 April 2021 through 15 April 2021

ER -

Approximating probability distributions by RELu networks

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this