Research on Text Generation Based on LSTM

Lifen Li ； Tianyu Zhang

《International Core Journal of Engineering》 7卷5期 (2021/05) Pp. 525-535

https://doi.org/10.6919/ICJE.202105_7(5).0067

摘要

Text generation is an important direction in the field of natural language processing (NLP). In the era of pre-training model, transformer improved pre-training text generation model still can not achieve relatively ideal results, and at the same time, there is no efficient language model to automatically evaluate the quality of generated text. As an improved variant of cyclic neural network (RNN), long-term memory network (LSTM) is characterized by its long-term dependence, and it performs very well in the task of processing long sequences. The LSTM has many improved variants for different tasks, including the gated loop unit (GRU) and the LSTM with a peephole connection, all of which have better performance than the LSTM in specific tasks. However, it is not clear whether these improved variants have better performance in the field of text generation. Therefore, an exploratory text generation experiment is conducted to solve this problem. By comparing the generated text quality of standard LSTM with LSTM's improved variant GRU and LSTM model with visual hole connection, the evaluation results of LSTM model in long text field are obviously better than those of the other two models through three evaluation indexes: confusion degree, BERT score and BLEURT. Finally, we draw a conclusion and research direction that the native LSTM in the field of long text still has very superior performance. In the future, we can design a pre-training model based on LSTM for text generation. Future language models can be designed to guide the optimization and improvement of language models through large-scale evaluation using automated evaluation indicators such as BERT score and BLEURT, which are close to manual evaluation, so as to design language models that can generate higher quality text.

關鍵字

LSTM ； Peephole Connection ； GRU ； BERT Score ； BLUERT

參考文獻

Li Su. The market approach and reflection of the development of machine journalis- Taking Autamated Insights as an example[J]. press. 2015(18):56-61(in Chinese)

Wan Xiaojun, Feng Yansong, Sun Weiwei. Research Progress and Trend of Automatic Text Generation. CCF Chinese Information Technology Professional Committee (in Chinese)

Hochreiter, Sepp & Schmidhuber, Jürgen. (1997). Long Short-term Memory. Neural computation. 9. 1735-80. 10.1162/neco.1997.9.8.1735.

Gers F A, Schmidhuber J. Recurrent nets that time and count[C]// Neural Networks, 2000. IJCNN 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on. IEEE, 2000.

Cho, Kyunghyun & van Merriënboer, Bart & Gulcehre, Caglar & Bougares, Fethi & Schwenk, Holger & Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. 10.3115/v1/D14-1179.