On the practical computational power of finite precision RNNs for language recognition

Gail Weiss; Yoav Goldberg; Eran Yahav

doi:10.18653/v1/p18-2117

On the practical computational power of finite precision RNNs for language recognition

Gail Weiss, Yoav Goldberg, Eran Yahav

Department of Computer Science

Technion-Israel Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

158 Scopus citations

Abstract

While Recurrent Neural Networks (RNNs) are famously known to be Turing complete, this relies on infinite precision in the states and unbounded computation time. We consider the case of RNNs with finite precision whose computation time is linear in the input length. Under these limitations, we show that different RNN variants have different computational power. In particular, we show that the LSTM and the Elman-RNN with ReLU activation are strictly stronger than the RNN with a squashing activation and the GRU. This is achieved because LSTMs and ReLU-RNNs can easily implement counting behavior. We show empirically that the LSTM does indeed learn to effectively use the counting mechanism.

Original language	English
Title of host publication	ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers)
Publisher	Association for Computational Linguistics (ACL)
Pages	740-745
Number of pages	6
ISBN (Electronic)	9781948087346
DOIs	https://doi.org/10.18653/v1/p18-2117
State	Published - 2018
Event	56th Annual Meeting of the Association for Computational Linguistics, ACL 2018 - Melbourne, Australia Duration: 15 Jul 2018 → 20 Jul 2018

Publication series

Name	ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)
Volume	2

Conference

Conference	56th Annual Meeting of the Association for Computational Linguistics, ACL 2018
Country/Territory	Australia
City	Melbourne
Period	15/07/18 → 20/07/18

Bibliographical note

Publisher Copyright:
© 2018 Association for Computational Linguistics

Funding

The research leading to the results presented in this paper is supported by the European Union’s Seventh Framework Programme (FP7) under grant agreement no. 615688 (PRIME), The Israeli Science Foundation (grant number 1555/15), and The Allen Institute for Artificial Intelligence.

Funders	Funder number
Allen Institute for Artificial Intelligence
Israeli Science Foundation	1555/15
Office of Intelligence
Seventh Framework Programme	615688
Israel Science Foundation
Seventh Framework Programme

Access to Document

10.18653/v1/p18-2117

Cite this

Weiss, G., Goldberg, Y., & Yahav, E. (2018). On the practical computational power of finite precision RNNs for language recognition. In ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers) (pp. 740-745). (ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers); Vol. 2). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/p18-2117

Weiss, Gail ; Goldberg, Yoav ; Yahav, Eran. / On the practical computational power of finite precision RNNs for language recognition. ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers). Association for Computational Linguistics (ACL), 2018. pp. 740-745 (ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)).

@inproceedings{baf98288910a41109823939f5ea92b96,

title = "On the practical computational power of finite precision RNNs for language recognition",

abstract = "While Recurrent Neural Networks (RNNs) are famously known to be Turing complete, this relies on infinite precision in the states and unbounded computation time. We consider the case of RNNs with finite precision whose computation time is linear in the input length. Under these limitations, we show that different RNN variants have different computational power. In particular, we show that the LSTM and the Elman-RNN with ReLU activation are strictly stronger than the RNN with a squashing activation and the GRU. This is achieved because LSTMs and ReLU-RNNs can easily implement counting behavior. We show empirically that the LSTM does indeed learn to effectively use the counting mechanism.",

author = "Gail Weiss and Yoav Goldberg and Eran Yahav",

note = "Publisher Copyright: {\textcopyright} 2018 Association for Computational Linguistics; 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018 ; Conference date: 15-07-2018 Through 20-07-2018",

year = "2018",

doi = "10.18653/v1/p18-2117",

language = "אנגלית",

series = "ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)",

publisher = "Association for Computational Linguistics (ACL)",

pages = "740--745",

booktitle = "ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers)",

address = "ארצות הברית",

}

Weiss, G, Goldberg, Y & Yahav, E 2018, On the practical computational power of finite precision RNNs for language recognition. in ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers). ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), vol. 2, Association for Computational Linguistics (ACL), pp. 740-745, 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, 15/07/18. https://doi.org/10.18653/v1/p18-2117

On the practical computational power of finite precision RNNs for language recognition. / Weiss, Gail; Goldberg, Yoav; Yahav, Eran.
ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers). Association for Computational Linguistics (ACL), 2018. p. 740-745 (ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers); Vol. 2).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - On the practical computational power of finite precision RNNs for language recognition

AU - Weiss, Gail

AU - Goldberg, Yoav

AU - Yahav, Eran

PY - 2018

Y1 - 2018

N2 - While Recurrent Neural Networks (RNNs) are famously known to be Turing complete, this relies on infinite precision in the states and unbounded computation time. We consider the case of RNNs with finite precision whose computation time is linear in the input length. Under these limitations, we show that different RNN variants have different computational power. In particular, we show that the LSTM and the Elman-RNN with ReLU activation are strictly stronger than the RNN with a squashing activation and the GRU. This is achieved because LSTMs and ReLU-RNNs can easily implement counting behavior. We show empirically that the LSTM does indeed learn to effectively use the counting mechanism.

AB - While Recurrent Neural Networks (RNNs) are famously known to be Turing complete, this relies on infinite precision in the states and unbounded computation time. We consider the case of RNNs with finite precision whose computation time is linear in the input length. Under these limitations, we show that different RNN variants have different computational power. In particular, we show that the LSTM and the Elman-RNN with ReLU activation are strictly stronger than the RNN with a squashing activation and the GRU. This is achieved because LSTMs and ReLU-RNNs can easily implement counting behavior. We show empirically that the LSTM does indeed learn to effectively use the counting mechanism.

UR - http://www.scopus.com/inward/record.url?scp=85051119405&partnerID=8YFLogxK

U2 - 10.18653/v1/p18-2117

DO - 10.18653/v1/p18-2117

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85051119405

T3 - ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)

SP - 740

EP - 745

BT - ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers)

PB - Association for Computational Linguistics (ACL)

T2 - 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018

Y2 - 15 July 2018 through 20 July 2018

ER -

Weiss G, Goldberg Y, Yahav E. On the practical computational power of finite precision RNNs for language recognition. In ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers). Association for Computational Linguistics (ACL). 2018. p. 740-745. (ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)). doi: 10.18653/v1/p18-2117

On the practical computational power of finite precision RNNs for language recognition

Abstract

Publication series

Conference

Bibliographical note

Funding

Access to Document

Other files and links

Fingerprint

Cite this