Abstract
We present an algorithm for extraction of a probabilistic deterministic finite automaton (PDFA) from a given black-box language model, such as a recurrent neural network (RNN). The algorithm is a variant of the exact-learning algorithm L*, adapted to a probabilistic setting with noise. The key insight is the use of conditional probabilities for observations, and the introduction of a local tolerance when comparing them. When applied to RNNs, our algorithm often achieves better word error rate (WER) and normalised distributed cumulative gain (NDCG) than that achieved by spectral extraction of weighted finite automata (WFA) from the same networks. PDFAs are substantially more expressive than n-grams, and are guaranteed to be stochastic and deterministic - unlike spectrally extracted WFAs.
Original language | English |
---|---|
Journal | Advances in Neural Information Processing Systems |
Volume | 32 |
State | Published - 2019 |
Event | 33rd Annual Conference on Neural Information Processing Systems, NeurIPS 2019 - Vancouver, Canada Duration: 8 Dec 2019 → 14 Dec 2019 |
Bibliographical note
Publisher Copyright:© 2019 Neural information processing systems foundation. All rights reserved.
Funding
The authors wish to thank Rémi Eyraud for his helpful discussions and comments, and Chris Ham-merschmidt for his assistance in obtaining the results with FLEXFRINGE . The research leading to the results presented in this paper is supported by the Israeli Science Foundation (grant No.1319/16), and the European Research Council (ERC) under the European Union’s Seventh Framework Programme (FP7-2007-2013), under grant agreement no. 802774 (iEXTRACT). The authors wish to thank R?mi Eyraud for his helpful discussions and comments, and Chris Hammerschmidt for his assistance in obtaining the results with FLEXFRINGE. The research leading to the results presented in this paper is supported by the Israeli Science Foundation (grant No.1319/16), and the European Research Council (ERC) under the European Union's Seventh Framework Programme (FP7-2007-2013), under grant agreement no. 802774 (iEXTRACT).
Funders | Funder number |
---|---|
FLEXFRINGE | |
Israeli Science Foundation | 1319/16 |
European Commission | |
Seventh Framework Programme | 802774, FP7-2007-2013 |