Keyphrase Generation for Scientific Articles using GANs
In this paper, we present a keyphrase generation approach using conditional Generative Adversarial Networks (GAN). In our GAN model, the generator outputs a sequence of keyphrases based on the title and abstract of a scientific article. The discriminator learns to distinguish between machine-generated and human-curated keyphrases. We evaluate this approach on standard benchmark datasets. Our model achieves state-of-the-art performance in generation of abstractive keyphrases and is also comparable to the best performing extractive techniques. We also demonstrate that our method generates more diverse keyphrases and make our implementation publicly available111Code is available at https://github.com/avinsit123/keyphrase-gan.
Keyphrases are employed to capture the most salient topics of a long document and are indexed in databases for convenient retrieval. Researchers annotate their scientific publications with high quality keyphrases to ensure discoverability in large scientific repositories. Keyphrases could either be extractive (part of the document) or abstractive. Keyphrase generation is the process of predicting both extractive and abstractive keyphrases from a given document. This process is similar to abstractive summarization but instead of a summary the models generate keyphrases.
Researchers have achieved considerable success in the field of abstractive summarization using conditional-GANs . There has also been growing interest in deep learning models for keyphrase generation [6, 1]. Inspired by these advances, we propose a new GAN architecture for keyphrase generation where the generator produces a sequence of keyphrases from a given document and the discriminator distinguishes between human-curated and machine-generated keyphrases.
Proposed Adversarial Model
As with most GAN architectures, our model also consists of a generator (G) and discriminator (D), which are trained in an alternating fashion .
Generator - Given a document , where is the token, the generator produces a sequence of keyphrases: , where each keyphrase is composed of tokens . We employ catSeq model  for the generation process, which uses an encoder-decoder framework: the encoder being a bidirectional Gated Recurrent Unit (bi-GRU) and the decoder a forward GRU. To incorporate the out-of-vocabulary words, we use a copying mechanism . We also make use of attention mechanism to help the generator identify the relevant components of the source text.
Discriminator - We propose a new hierarchical-attention model as the discriminator, which is trained to distinguish between human-curated and machine-generated keyphrases. The first layer of this model consists of bi-GRUs. The first bi-GRU encodes the input document as a sequence of vectors: . The other bi-GRUs, which have the same weight parameters, encode each keyphrase as a vector: . We then use an attention-based approach  to build context vectors for each keyphrase, where is a weighted average over . By concatenating and , we get a contextualized representation of keyphrase .
The second layer of the discriminator is another bi-GRU which consumes the document representation and the keyphrase representations . The final state of this layer is passed through one fully connected layer () and sigmoid transformation to get the probability that a given keyphrase sequence is human-curated.
GAN training - For a given dataset (S), which contain the documents and corresponding keyphrases, we first pre-train the generator (G) using Maximum Likelihood Estimation. We then use this generator to produce machine-generated keyphrases for all documents in S. These generated keyphrases along with the curated keyphrases are used to train the first version of the discriminator (D).
We then employ policy gradient reinforcement learning to train the subsequent versions of G. We freeze the weight parameters of D and use it for reward calculation to train a new version of G. The reward for each keyphrase is obtained from the last states of the second bi-GRU layer in D (see Figure 1). The gradient update is given as:
where B is a baseline obtained by greedy decoding of keyphrase sequence. The resulting generator is then used to create new training samples for D. This process is continued till G converges.
Experiments and Results
We trained the proposed GAN model on KP20k dataset  which consists of 567,830 samples for training, 20,000 each for testing and validation. Each sample consists of an abstract, title, and the corresponding keyphrases of a scientific article. We evaluated the model on four datasets: Inspec, NUS, KP20k, and Krapivin, which contain 600, 211, 20,000, and 800 test samples respectively. For training G, we used Adagrad optimizer with learning rate 0.0005. We compare our proposed approach against 2 baseline models - catSeq , RL-based catSeq Model  in terms of F1 scores as explained in . The results, summarized in Table 1, are broken down in terms of performance on extractive and abstractive keyphrases.
For extractive keyphrases, our proposed model performs better than the pre-trained catSeq model on all datasets but is slightly worse than catSeq-RL except for on Krapivin where it obtains the best F1@M of 0.37. On the other hand, for abstractive keyphrases, our model performs better than the other two baselines on three of four datasets suggesting that GAN models are more effective in generation of keyphrases.
We also evaluated the models in terms of -nDCG@5 . The results are summarized in Table 2. Our model obtains the best performance on three out of the four datasets. The difference is most prevalent in KP20k, the largest of the four datasets, where our GAN model (at 0.85) is nearly 5% better than both the other baseline models.
In this paper, we propose new GAN architecture for keyphrase generation. The proposed model obtains state-of-the-art performance in generating abstractive keyphrases. To our knowledge, this is the first work that applies GANs to keyphrase generation problem.
-  (2019) Neural keyphrase generation via reinforcement learning with adaptive rewards. In ACL, Cited by: Introduction, Experiments and Results.
-  (2008-01) Novelty and diversity in information retrieval evaluation. Proc. of the 31st ACM SIGIR, pp. 659–666. External Links: Cited by: Experiments and Results.
-  (2014) Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680. Cited by: Proposed Adversarial Model.
-  (2016) Incorporating copying mechanism in sequence-to-sequence learning. arXiv preprint arXiv:1603.06393. Cited by: Proposed Adversarial Model.
-  (2015) Effective approaches to attention-based neural machine translation. In EMNLP, Cited by: Proposed Adversarial Model.
-  (2017) Deep keyphrase generation. arXiv preprint arXiv:1704.06879. Cited by: Introduction, Experiments and Results.
-  (2018) Learning to encode text as human-readable summaries using generative adversarial networks. CoRR abs/1810.02851. External Links: Cited by: Introduction.
-  (2018) Generating diverse numbers of diverse keyphrases. ArXiv abs/1810.05241. Cited by: Proposed Adversarial Model, Experiments and Results.