File Download
Links for fulltext
(May Require Subscription)
- Scopus: eid_2-s2.0-85090174020
- WOS: WOS:000535866904074
- Find via
Supplementary
- Citations:
- Appears in Collections:
Conference Paper: Episodic memory in lifelong language learning
Title | Episodic memory in lifelong language learning |
---|---|
Authors | |
Issue Date | 2019 |
Citation | 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada, 8-14 December 2019. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019 How to Cite? |
Abstract | © 2019 Neural information processing systems foundation. All rights reserved. We introduce a lifelong language learning setup where a model needs to learn from a stream of text examples without any dataset identifier. We propose an episodic memory model that performs sparse experience replay and local adaptation to mitigate catastrophic forgetting in this setup. Experiments on text classification and question answering demonstrate the complementary benefits of sparse experience replay and local adaptation to allow the model to continuously learn from new datasets. We also show that the space complexity of the episodic memory module can be reduced significantly (~50-90%) by randomly choosing which examples to store in memory with a minimal decrease in performance. We consider an episodic memory component as a crucial building block of general linguistic intelligence and see our model as a first step in that direction. |
Persistent Identifier | http://hdl.handle.net/10722/296219 |
ISSN | 2020 SCImago Journal Rankings: 1.399 |
ISI Accession Number ID |
DC Field | Value | Language |
---|---|---|
dc.contributor.author | de Masson D'Autume, Cyprien | - |
dc.contributor.author | Ruder, Sebastian | - |
dc.contributor.author | Kong, Lingpeng | - |
dc.contributor.author | Yogatama, Dani | - |
dc.date.accessioned | 2021-02-11T04:53:05Z | - |
dc.date.available | 2021-02-11T04:53:05Z | - |
dc.date.issued | 2019 | - |
dc.identifier.citation | 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada, 8-14 December 2019. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019 | - |
dc.identifier.issn | 1049-5258 | - |
dc.identifier.uri | http://hdl.handle.net/10722/296219 | - |
dc.description.abstract | © 2019 Neural information processing systems foundation. All rights reserved. We introduce a lifelong language learning setup where a model needs to learn from a stream of text examples without any dataset identifier. We propose an episodic memory model that performs sparse experience replay and local adaptation to mitigate catastrophic forgetting in this setup. Experiments on text classification and question answering demonstrate the complementary benefits of sparse experience replay and local adaptation to allow the model to continuously learn from new datasets. We also show that the space complexity of the episodic memory module can be reduced significantly (~50-90%) by randomly choosing which examples to store in memory with a minimal decrease in performance. We consider an episodic memory component as a crucial building block of general linguistic intelligence and see our model as a first step in that direction. | - |
dc.language | eng | - |
dc.relation.ispartof | Advances in Neural Information Processing Systems 32 (NeurIPS 2019) | - |
dc.title | Episodic memory in lifelong language learning | - |
dc.type | Conference_Paper | - |
dc.description.nature | link_to_OA_fulltext | - |
dc.identifier.scopus | eid_2-s2.0-85090174020 | - |
dc.identifier.isi | WOS:000535866904074 | - |
dc.identifier.issnl | 1049-5258 | - |