File Download
There are no files associated with this item.
Links for fulltext
(May Require Subscription)
- Publisher Website: 10.1613/JAIR.1.15280
- Scopus: eid_2-s2.0-85179132521
- Find via
Supplementary
-
Citations:
- Scopus: 0
- Appears in Collections:
Article: Graphmax for Text Generation
Title | Graphmax for Text Generation |
---|---|
Authors | |
Issue Date | 27-Nov-2023 |
Publisher | AI Access Foundation |
Citation | Journal of Artificial Intelligence Research, 2023, v. 78, p. 823-848 How to Cite? |
Abstract | In text generation, a large language model (LM) makes a choice of each new word based only on the former selection of its context using the softmax function. Nevertheless, the link statistics information of concurrent words based on a scene-specific corpus is valuable in choosing the next word, which can help to ensure the topic of the generated text to be aligned with the current task. To fully explore the co-occurrence information, we propose a graphmax function for task-specific text generation. Using the graph-based regularization, graphmax enables the final word choice to be determined by both the global knowledge from the LM and the local knowledge from the scene-specific corpus. The traditional softmax function is regularized with a graph total variation (GTV) term, which incorporates the local knowledge into the LM and encourages the model to consider the statistical relationships between words in a scene-specific corpus. The proposed graphmax is versatile and can be readily plugged into any large pre-trained LM for text generation and machine translation. Through extensive experiments, we demonstrate that the new GTV-based regularization can improve performances in various natural language processing (NLP) tasks in comparison with existing methods. Moreover, through human experiments, we observe that participants can easily distinguish the text generated by graphmax or softmax. |
Persistent Identifier | http://hdl.handle.net/10722/347132 |
ISSN | 2023 Impact Factor: 4.5 2023 SCImago Journal Rankings: 1.614 |
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Liu, Bin | - |
dc.contributor.author | Yin, Guosheng | - |
dc.date.accessioned | 2024-09-18T00:30:32Z | - |
dc.date.available | 2024-09-18T00:30:32Z | - |
dc.date.issued | 2023-11-27 | - |
dc.identifier.citation | Journal of Artificial Intelligence Research, 2023, v. 78, p. 823-848 | - |
dc.identifier.issn | 1076-9757 | - |
dc.identifier.uri | http://hdl.handle.net/10722/347132 | - |
dc.description.abstract | In text generation, a large language model (LM) makes a choice of each new word based only on the former selection of its context using the softmax function. Nevertheless, the link statistics information of concurrent words based on a scene-specific corpus is valuable in choosing the next word, which can help to ensure the topic of the generated text to be aligned with the current task. To fully explore the co-occurrence information, we propose a graphmax function for task-specific text generation. Using the graph-based regularization, graphmax enables the final word choice to be determined by both the global knowledge from the LM and the local knowledge from the scene-specific corpus. The traditional softmax function is regularized with a graph total variation (GTV) term, which incorporates the local knowledge into the LM and encourages the model to consider the statistical relationships between words in a scene-specific corpus. The proposed graphmax is versatile and can be readily plugged into any large pre-trained LM for text generation and machine translation. Through extensive experiments, we demonstrate that the new GTV-based regularization can improve performances in various natural language processing (NLP) tasks in comparison with existing methods. Moreover, through human experiments, we observe that participants can easily distinguish the text generated by graphmax or softmax. | - |
dc.language | eng | - |
dc.publisher | AI Access Foundation | - |
dc.relation.ispartof | Journal of Artificial Intelligence Research | - |
dc.rights | This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. | - |
dc.title | Graphmax for Text Generation | - |
dc.type | Article | - |
dc.identifier.doi | 10.1613/JAIR.1.15280 | - |
dc.identifier.scopus | eid_2-s2.0-85179132521 | - |
dc.identifier.volume | 78 | - |
dc.identifier.spage | 823 | - |
dc.identifier.epage | 848 | - |
dc.identifier.eissn | 1943-5037 | - |
dc.identifier.issnl | 1076-9757 | - |