Towards Supervised Extractive Text Summarization via RNN-based Sequence Classification

Towards Supervised Extractive Text Summarization via RNN-based Sequence Classification

Eduardo Brito Fraunhofer IAIS, Sankt Augustin, Germany
Fraunhofer Center for Machine Learning, Germany
B-IT, University of Bonn, Bonn, Germany
   Max Lübbering Fraunhofer IAIS, Sankt Augustin, Germany
Fraunhofer Center for Machine Learning, Germany
B-IT, University of Bonn, Bonn, Germany
   David Biesner Fraunhofer IAIS, Sankt Augustin, Germany
Fraunhofer Center for Machine Learning, Germany
B-IT, University of Bonn, Bonn, Germany
   Lars Patrick Hillebrand Fraunhofer IAIS, Sankt Augustin, Germany
Fraunhofer Center for Machine Learning, Germany
B-IT, University of Bonn, Bonn, Germany
   Christian Bauckhage Fraunhofer IAIS, Sankt Augustin, Germany
Fraunhofer Center for Machine Learning, Germany
B-IT, University of Bonn, Bonn, Germany
Abstract

This article briefly explains our submitted approach to the DocEng’19 competition on extractive summarization[3]. We implemented a recurrent neural network based model that learns to classify whether an article’s sentence belongs to the corresponding extractive summary or not. We bypass the lack of large annotated news corpora for extractive summarization by generating extractive summaries from abstractive ones, which are available from the CNN corpus.

Keywords:
neural networks, extractive text summarization

1 Introduction

The DocEng ’19 competition focused on automatic extractive text summarization. Participants were provided with a corpus of 50 news articles from the CNN-corpus[4]. These articles contained corresponding extractive and abstractive summaries aimed to train and test a system to perform the summarization task. The gold standard summaries contained around 10% of the original text, with a minimum of 3 sentences. After submission, the methods were tested on a larger test set consisting of 1000 articles randomly chosen from the CNN-corpus. The limited available training data was one of the major challenges of this competition, which prevented any deep learning approach from being successful if no external corpus was incorporated to the training set.

2 Approach

Figure 1: Our RNN-based sequence classifier (based on [5]). All word embeddings from each sentence are averaged to generate a sentence embedding. Sentence embeddings are then used for the bidirectional RNN at sentence level. At the top, the sigmoid activation based classification layer decides whether a sentence is included in the summary based on the content richness of the sentence, its salience with respect to the document and its novelty respect to the accumulated summary representation.

Our work is based on the SummaRuNNer model [5]. It consists of a two-layer bi-directional Gated Recurrent Unit (GRU) Recurrent Neural Network (RNN) which treats the summarization problem as a binary sequence classification problem, where each sentence is classified sequentially as sentence to be included or not in the summary. However, we introduced two modifications to the original SummaRuNNer architecture, leading to better results while reducing complexity:

  1. Our model operates directly on a sentence level (instead of at word level within each sentence). We compute sentence vector representations by means of the the Flair library. [1]111https://github.com/zalandoresearch/flair. These sentence embeddings substitute the bottom layer of the SummaRuNNer architecture.

  2. We do not consider the position of each sentence (absolute or relative) for the logistic layer.

The resulting architecture is displayed on Figure 1. Our code to generate extractive summaries according to the instructions established for the competition is publicly available222https://jira.iais.fraunhofer.de/stash/users/dbiesner/repos/doceng2019_fraunhofer.

3 Data

In contrast to [5], we trained our model only on CNN articles from the CNN/Daily Mail corpus[2]. Due to the limited number of provided news articles, we automatically annotated a large corpus of CNN articles from which an abstractive summary was available. In a similar approach to [5], we calculated the ROUGE-1 F1 score between each sentence and its article’s abstractive summary. Finally for each article, we sorted the sentences having the highest ROUGE-1 F1 score and picked the top sentences.

Score Precision Recall F1
Sentence matching gold standard 0.375 0.357 0.358
ROUGE-1 0.384 0.206 0.261
ROUGE-2 0.141 0.094 0.094
Table 1: Evaluation on the 50 labeled news articles provided by the competition organizers

4 Evaluation

We evaluated our model on the provided labeled CNN news articles with three different metrics: sentences from the generated summary matching the gold standard summary, ROUGE-1 and ROUGE-2. The achieved scores with our trained model after 20 epochs are displayed on Table 1.

5 Conclusion

Our approach achieved the second best performance among the compared methods in the competition, although the F1-score difference between both approaches is not statistically significant[3]. Additionally, the performance of these approaches is hardly better than some of the ”traditional algorithms” that were presented as baselines, which are much simpler than ours. Moreover, the real value of the different approaches on the various use cases of automatic text summarization cannot be covered with the current evaluation since the valuable properties of the summaries vary depending on the use case. For instance, coherence is important if the summary will be read by a final user while it is not if the summary is ”just” a preprocessing step within an indexing pipeline. Therefore, it would be interesting to assess the different techniques on several downstream tasks to obtain a better overview about which algorithms are most suitable.

References

  • [1] A. Akbik, D. Blythe, and R. Vollgraf (2018) Contextual string embeddings for sequence labeling. In COLING 2018, 27th International Conference on Computational Linguistics, pp. 1638–1649. Cited by: item 1.
  • [2] K. M. Hermann, T. Kočiský, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom (2015) Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS), External Links: Link Cited by: §3.
  • [3] R. D. Lins, R. F. Mello, and S. Simske (2019) DocEng’19 competition on extractive text summarization. In Proceedings of the ACM Symposium on Document Engineering 2019, DocEng ’19, New York, NY, USA, pp. 4:1–4:2. External Links: ISBN 978-1-4503-6887-2, Link, Document Cited by: Towards Supervised Extractive Text Summarization via RNN-based Sequence Classification, §5.
  • [4] R. D. Lins, H. Oliveira, L. Cabral, J. Batista, B. Tenorio, R. Ferreira, R. Lima, G. de França Pereira e Silva, and S. J. Simske (2019) The cnn-corpus: a large textual corpus for single-document extractive summarization. In Proceedings of the ACM Symposium on Document Engineering 2019, DocEng ’19, New York, NY, USA, pp. 16:1–16:10. External Links: ISBN 978-1-4503-6887-2, Link, Document Cited by: §1.
  • [5] R. Nallapati, F. Zhai, and B. Zhou (2017) Summarunner: a recurrent neural network based sequence model for extractive summarization of documents. In Thirty-First AAAI Conference on Artificial Intelligence, Cited by: Figure 1, §2, §3.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
398421
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description