File Download
There are no files associated with this item.
Supplementary
-
Citations:
- Appears in Collections:
Conference Paper: Non-autoregressive neural machine translation
Title | Non-autoregressive neural machine translation |
---|---|
Authors | |
Keywords | Machine translation Non-autoregressive Transformer Fertility NMT |
Issue Date | 2018 |
Citation | 6th International Conference on Learning Representations (ICLR), Vancouver, Canada, 30 April - 3 May 2018 How to Cite? |
Abstract | Existing approaches to neural machine translation condition each output word on previously generated outputs. We introduce a model that avoids this autoregressive property and produces its outputs in parallel, allowing an order of magnitude lower latency during inference. Through knowledge distillation, the use of input token fertilities as a latent variable, and policy gradient fine-tuning, we achieve this at a cost of as little as 2.0 BLEU points relative to the autoregressive Transformer network used as a teacher. We demonstrate substantial cumulative improvements associated with each of the three aspects of our training strategy, and validate our approach on IWSLT 2016 English–German and two WMT language pairs. By sampling fertilities in parallel at inference time, our non-autoregressive model achieves near-state-of-the-art performance of 29.8 BLEU on WMT 2016 English–Romanian. |
Persistent Identifier | http://hdl.handle.net/10722/261953 |
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Gu, J | - |
dc.contributor.author | Bradbury, J | - |
dc.contributor.author | Xiong, C | - |
dc.contributor.author | Li, VOK | - |
dc.contributor.author | Socher, R | - |
dc.date.accessioned | 2018-09-28T04:50:53Z | - |
dc.date.available | 2018-09-28T04:50:53Z | - |
dc.date.issued | 2018 | - |
dc.identifier.citation | 6th International Conference on Learning Representations (ICLR), Vancouver, Canada, 30 April - 3 May 2018 | - |
dc.identifier.uri | http://hdl.handle.net/10722/261953 | - |
dc.description.abstract | Existing approaches to neural machine translation condition each output word on previously generated outputs. We introduce a model that avoids this autoregressive property and produces its outputs in parallel, allowing an order of magnitude lower latency during inference. Through knowledge distillation, the use of input token fertilities as a latent variable, and policy gradient fine-tuning, we achieve this at a cost of as little as 2.0 BLEU points relative to the autoregressive Transformer network used as a teacher. We demonstrate substantial cumulative improvements associated with each of the three aspects of our training strategy, and validate our approach on IWSLT 2016 English–German and two WMT language pairs. By sampling fertilities in parallel at inference time, our non-autoregressive model achieves near-state-of-the-art performance of 29.8 BLEU on WMT 2016 English–Romanian. | - |
dc.language | eng | - |
dc.relation.ispartof | International Conference on Learning Representations (ICLR) | - |
dc.subject | Machine translation | - |
dc.subject | Non-autoregressive | - |
dc.subject | Transformer | - |
dc.subject | Fertility | - |
dc.subject | NMT | - |
dc.title | Non-autoregressive neural machine translation | - |
dc.type | Conference_Paper | - |
dc.identifier.email | Li, VOK: vli@eee.hku.hk | - |
dc.identifier.authority | Li, VOK=rp00150 | - |
dc.identifier.hkuros | 292171 | - |