Enhancing Drug-Drug Interaction Extraction from Texts by Molecular Structure Information
We propose a novel neural method to extract drug-drug interactions (DDIs) from texts using external drug molecular structure information. We encode textual drug pairs with convolutional neural networks and their molecular pairs with graph convolutional networks (GCNs), and then we concatenate the outputs of these two networks. In the experiments, we show that GCNs can predict DDIs from the molecular structures of drugs in high accuracy and the molecular information can enhance text-based DDI extraction by 2.39 percent points in the F-score on the DDIExtraction 2013 shared task data set.
When drugs are concomitantly administered to a patient, the effects of the drugs may be enhanced or weakened, which may also cause side effects. These kinds of interactions are called Drug-Drug Interactions (DDIs). Several drug databases have been maintained to summarize drug and DDI information such as DrugBank [Law et al.(2014)Law, Knox, Djoumbou, Jewison, Guo, Liu, Maciejewski, Arndt, Wilson, Neveu et al.], Therapeutic Target database [Yang et al.(2016)Yang, Qin, Li, Tao, Zhou, Yu, Xu, Chen, Zhu, and Chen], and PharmGKB [Thorn et al.(2013)Thorn, Klein, and Altman]. Automatic DDI extraction from texts is expected to support the maintenance of databases with high coverage and quick update to help medical experts. Deep neural network-based methods have recently drawn a considerable attention [Liu et al.(2016)Liu, Tang, Chen, and Wang, Sahu and Anand(2017), Zheng et al.(2017)Zheng, Lin, Luo, Zhao, Li, Zhang, Yang, and Wang, Lim et al.(2018)Lim, Lee, and Kang] since they show state-of-the-art performance without manual feature engineering.
In parallel to the progress in DDI extraction from texts, Graph Convolutional Networks (GCNs) have been proposed and applied to estimate physical and chemical properties of molecular graphs such as solubility and toxicity [Duvenaud et al.(2015)Duvenaud, Maclaurin, Iparraguirre, Bombarell, Hirzel, Aspuru-Guzik, and Adams, Li et al.(2016)Li, Tarlow, Brockschmidt, and Zemel, Gilmer et al.(2017)Gilmer, Schoenholz, Riley, Vinyals, and Dahl].
In this study, we propose a novel method to utilize both textual and molecular information for DDI extraction from texts. We illustrate the overview of the proposed model in Figure 1. We obtain the representations of drug pairs in molecular graph structures using GCNs and concatenate the representations with the representations of the textual mention pairs obtained by convolutional neural networks (CNNs). We trained the molecule-based model using interacting pairs mentioned in the DrugBank database and then trained the entire model using the labeled pairs in the text data set of the DDIExtraction 2013 shared task (SemEval-2013 Task 9) [Segura Bedmar et al.(2013)Segura Bedmar, Martínez, and Herrero Zazo]. In the experiment, we show GCNs can predict DDIs from molecular graphs in a high accuracy. We also show molecular information can enhance the performance of DDI extraction from texts in 2.39 percent points in F-score.
The contribution of this paper is three-fold:
We propose a novel neural method to extract DDIs from texts with the related molecular structure information.
We apply GCNs to pairwise drug molecules for the first time and show GCNs can predict DDIs between drug molecular structures in a high accuracy.
We show the molecular information is useful in extracting DDIs from texts.
2.1 Text-based DDI Extraction
Our model for extracting DDIs from texts is based on the CNN model by \newcitezeng2014relation. When an input sentence is given, We prepare word embedding of and word position embeddings and that correspond to the relative positions from the first and second target entities, respectively. We concatenate these embeddings as in Equation (1), and we use the resulting vector as the input to the subsequent convolution layer:
where denotes the concatenation. We calculate the expression for each filter with the window size .
where is the number of windows, and are the weight and bias of CNN, and indicates max pooling [Boureau et al.(2010)Boureau, Ponce, and LeCun].
We convert the output of the convolution layer into a fixed-size vector that represents a textual pair as follows:
where is the number of filters.
We get a prediction by the following fully connected neural networks:
where and are weights and and are bias terms.
2.2 Molecular Structure-based DDI Classification
We represent drug pairs in molecular graph structures using two GCN methods: CNNs for fingerprints (NFP) [Duvenaud et al.(2015)Duvenaud, Maclaurin, Iparraguirre, Bombarell, Hirzel, Aspuru-Guzik, and Adams] and Gated Graph Neural Networks (GGNN) [Li et al.(2016)Li, Tarlow, Brockschmidt, and Zemel]. They both convert a drug molecule graph into a fixed size vector by aggregating the representation of an atom node in . We represent atoms as nodes and bonds as edges in the graph.
NFP first obtains the representation by the following equations [Duvenaud et al.(2015)Duvenaud, Maclaurin, Iparraguirre, Bombarell, Hirzel, Aspuru-Guzik, and Adams].
where is the representation of in the -th step, is the neighbors of , and is a weight parameter. is initialized by the atom features of . is the degree of a node and is a sigmoid function. NFP then acquires the representation of the graph structure
where is a weight matrix.
GGNN first obtains the representation by using Gated Recurrent Unit (GRU)-based recurrent neural networks \citepli2015gated as follows:
where is a weight for the bond type of each edge . GGNN then acquires the representation of the graph structure.
where and are linear layers and is the element-wise product.
We obtain the representation of a molecular pair by concatenating the molecular graph representations of drugs and , i.e., .
We get a prediction as follows:
where and are weights and and are bias terms.
2.3 DDI Extraction from Texts Using Molecular Structures
We realize the simultaneous use of textual and molecular information by concatenating a text-based and molecule-based vectors: . We normalize molecule-based vectors. We then use instead of in Equation 7.
In training, we first train the molecular-based DDI classification model. The molecular-based classification is performed by minimizing the loss function . We then fix the parameters for GCNs and train text-based DDI extraction model by minimizing the loss function .
3 Experimental Settings
In this section, we explain the textual and molecular data and task settings and training settings.
3.1 Text Corpus and Task Setting
We followed the task setting of Task 9.2 in the DDIExtraction 2013 shared task [Segura Bedmar et al.(2013)Segura Bedmar, Martínez, and Herrero Zazo, Herrero-Zazo et al.(2013)Herrero-Zazo, Segura-Bedmar, Martínez, and Declerck] for the evaluation. This data set is composed of documents annotated with drug mentions and their four types of interactions: Mechanism, Effect, Advice and Int. For the data statistics, please refer to the supplementary materials.
The task is a multi-class classification task, i.e., to classify a given pair of drugs into the four interaction types or no interaction. We evaluated the performance with micro-averaged precision (P), recall (R), and F-score (F) on all the interaction types. We used the official evaluation script provided by the task organizers.
As preprocessing, we split sentences into words using the GENIA tagger [Tsuruoka et al.(2005)Tsuruoka, Tateishi, Kim, Ohta, McNaught, Ananiadou, and Tsujii]. We replaced the drug mentions of the target pair with DRUG1 and DRUG2 according to their order of appearance. We also replaced other drug mentions with DRUGOTHER. We did not employ negative instance filtering unlike other existing methods, e.g., \citetliu2016drug, since our focus is to evaluate the effect of the molecular information on texts.
We linked mentions in texts to DrugBank entries by string matching. We lowercased the mentions and the names in the entries and chose the entries with the most overlaps. As a result, 92.15% and 93.09% of drug mentions in train and test data set matched the DrugBank entries.
3.2 Data and Task for Molecular Structures
We extracted 255,229 interacting (positive) pairs from DrugBank. We note that, unlike text-based interactions, DrugBank only contains the information of interacting pairs; there are no detailed labels and no information for non-interacting (negative) pairs. We thus generated the same number of pseudo negative pairs by randomly pairing drugs and removing those in positive pairs. To avoid overestimation of the performance, we also deleted drug pairs mentioned in the test set of the text corpus. We split positive and negative pairs into 4:1 for training and test data, and we evaluated the classification accuracy using only the molecular information.
To obtain the graph of a drug molecule, we took as input the SMILES [Weininger(1988)] string encoding of the molecule from DrugBank and then converted it into the graph using RDKit [Landrum(2016)] as illustrated in Figure 2. For the atom features, we used randomly embedded vectors for each atoms (i.e., C, O, N, …). We also used 4 bond types: single, double, triple, or aromatic.
3.3 Training Settings
We employed mini-batch training using the Adam optimizer [Kingma and Ba(2015)]. We used L2 regularization to avoid over-fitting. We tuned the bias term for negative examples in the final softmax layer. For the hyper-parameters, please refer to the supplementary materials.
We employed pre-trained word embeddings trained by using the word2vec tool [Mikolov et al.(2013)Mikolov, Sutskever, Chen, Corrado, and Dean] on the 2014 MEDLINE/PubMed baseline distribution. The vocabulary size was 215,840. The embedding of the drugs, i.e., DRUG1 and DRUG2 were initialized with the pre-trained embedding of the word drug. The embeddings of training words that did not appear in the pre-trained embeddings were initialized with the average of all pre-trained word embeddings. Words that appeared only once in the training data were replaced with an UNK word during training, and the embedding of words in the test data set that did not appear in both training and pre-trained embeddings were set to the embedding of the UNK word. Word position embeddings are initialized with random values drawn from a uniform distribution.
We set the molecule-based vectors of unmatched entities to zero vectors.
|DDI Type||Mech.||Effect||Adv.||Int (%)|
Table 1 shows the performance of DDI extraction models. We show the performance without negative instance filtering or ensemble for the fair comparison. We observe the increase of recall and F-score by using molecular information, which results in the state-of-the-art performance with GGNN.
Both GCNs improvements were statistically significant ( for NFP and for GGNN) with randomized shuffled test.
Table 2 shows F-scores on individual DDI types. The molecular information improves F-scores especially on type Mechanism and Effect.
We also evaluated the accuracy of binary classification on DrugBank pairs by using only the molecular information in Table 3. The performance is high, although the accuracy is evaluated on automatically generated negative instances.
Finally, we applied the molecular-based DDI classification model trained on DrugBank to the DDIExtraction 2013 task data set. Since the DrugBank has no detailed labels, we mapped all four types of interactions to positive interactions and evaluated the classification performance. The results in Table 4 show that GCNs produce higher recall than precision and the overall performance is low considering the high performance on DrugBank pairs. This might be because the interactions of drugs are not always mentioned in texts even if the drugs can interact with each other and because hedged DDI mentions are annotated as DDIs in the text data set. We also trained the DDI extraction model only with molecular information by replacing with , but the F-scores were quite low ( 5%). These results show that we cannot predict textual relations only with molecular information.
5 Related Work
Various feature-based methods have been proposed during and after the DDIExtraction-2013 shared task [Segura Bedmar et al.(2013)Segura Bedmar, Martínez, and Herrero Zazo]. \newcitekim2015extracting proposed a two-phase SVM-based approach that employed a linear SVM with rich features that consist of word, word pair, dependency graph, parse tree, and noun phrase-based constrained coordination features. \newcitezheng2016graph proposed a context vector graph kernel to exploit various types of contexts. \newciteraihani2017rich also employed a two-phase SVM-based approach using non-linear kernels and they proposed five groups of features: word, drug, pair of drug, main verb and negative sentence features. Our model does not use any features or kernels.
Various neural DDI extraction models have been recently proposed using CNNs and Recurrent Neural Networks (RNNs). \newciteliu2016drug built a CNN-based model based on word and position embeddings. \newciteZheng2017 proposed a Bidirectional Long Short-Term Memory RNN (Bi-LSTM)-based model with an input attention mechanism, which obtained target drug-specific word representations before the Bi-LSTM. \newcitelim2018drug proposed Recursive neural network-based model with a subtree containment feature and an ensemble method. This model showed the state-of-the-art performance on the DDIExtraction 2013 shared task data set if systems do not use negative instance filtering. These approaches did not consider molecular information, and they can also be enhanced by the molecular information.
vilar2017detection focused on detecting DDIs from different sources such as pharmacovigilance sources, scientific biomedical literature and social media. They did not use deep neural networks and they did not consider molecular information.
Learning representations of graphs are widely studied in several tasks such as knowledge base completion, drug discovery, and material science [Wang et al.(2017)Wang, Mao, Wang, and Guo, Gilmer et al.(2017)Gilmer, Schoenholz, Riley, Vinyals, and Dahl]. Several graph convolutional neural networks have been proposed such as NFP [Duvenaud et al.(2015)Duvenaud, Maclaurin, Iparraguirre, Bombarell, Hirzel, Aspuru-Guzik, and Adams], GGNN [Li et al.(2016)Li, Tarlow, Brockschmidt, and Zemel], and Molecular Graph Convolutions [Kearnes et al.(2016)Kearnes, McCloskey, Berndl, Pande, and Riley], but they have not been applied to DDI extraction.
We proposed a novel neural method for DDI extraction using both textual and molecular information. The results show that DDIs can be predicted with high accuracy from molecular structure information and that the molecular information can improve DDI extraction from texts by 2.39 percept points in F-score on the data set of the DDIExtraction 2013 shared task.
As future work, we would like to seek the way to model the textual and molecular representations jointly with alleviating the differences in labels. We will also investigate the use of other information in DrugBank.
This work was supported by JSPS KAKENHI Grant Number 17K12741.
Appendix A Supplemental Material
a.1 Data Statistics
The statistics of the data set is shown in Table 5. This shows that the data is highly unbalanced and includes more negative pairs than positive pairs.
a.2 Hyper-parameter Settings
|Word embedding size||200|
|Word position embedding size||20|
|Convolution window size||[3, 5, 7]|
|Convolution filter size||100|
|Hidden layer size||500|
|Initial learning rate||0.001|
|L2 regularization parameter||0.0001|
|Molecular vector size||50|
|Number of steps||4|
|Hidden layer size||1,000|
|Initial learning rate||0.001|
|Hidden layer size of NFP||50|
|GRU unit size of GGNN||50|
- Y-Lan Boureau, Jean Ponce, and Yann LeCun. 2010. A theoretical analysis of feature pooling in visual recognition. In Proceedings of ICML. pages 111–118.
- David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alan Aspuru-Guzik, and Ryan P Adams. 2015. Convolutional networks on graphs for learning molecular fingerprints. In Proceedings of NIPS, pages 2224–2232.
- Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural message passing for quantum chemistry. In Proceedings of ICML. pages 1263–1272.
- María Herrero-Zazo, Isabel Segura-Bedmar, Paloma Martínez, and Thierry Declerck. 2013. The DDI corpus: An annotated corpus with pharmacological substances and drug–drug interactions. Journal of biomedical informatics 46(5):914–920.
- Steven Kearnes, Kevin McCloskey, Marc Berndl, Vijay Pande, and Patrick Riley. 2016. Molecular graph convolutions: moving beyond fingerprints. Journal of computer-aided molecular design 30(8):595–608.
- Sun Kim, Haibin Liu, Lana Yeganova, and W John Wilbur. 2015. Extracting drug–drug interactions from literature using a rich feature-based linear kernel approach. Journal of biomedical informatics 55:23–30.
- Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR.
- Greg Landrum. 2016. Rdkit: Open-source cheminformatics.
- Vivian Law, Craig Knox, Yannick Djoumbou, Tim Jewison, An Chi Guo, Yifeng Liu, Adam Maciejewski, David Arndt, Michael Wilson, Vanessa Neveu, et al. 2014. DrugBank 4.0: shedding new light on drug metabolism. Nucleic acids research 42(D1):D1091–D1097.
- Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2016. Gated graph sequence neural networks. In Proceedings of ICLR.
- Sangrak Lim, Kyubum Lee, and Jaewoo Kang. 2018. Drug drug interaction extraction from the literature using a recursive neural network. PloS one 13(1):e0190926.
- Shengyu Liu, Buzhou Tang, Qingcai Chen, and Xiaolong Wang. 2016. Drug-drug interaction extraction via convolutional neural networks. Computational and mathematical methods in medicine .
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS. pages 3111–3119.
- Anass Raihani and Nabil Laachfoubi. 2017. A rich feature-based kernel approach for drug-drug interaction extraction. International Journal of Advanced Computer Science and Applications 8(4):324–330.
- Sunil Kumar Sahu and Ashish Anand. 2017. Drug-drug interaction extraction from biomedical text using long short term memory network. arXiv preprint arXiv:1701.08303 .
- Isabel Segura Bedmar, Paloma Martínez, and María Herrero Zazo. 2013. SemEval-2013 Task 9: Extraction of drug-drug interactions from biomedical texts (DDIExtraction 2013). In Proceedings of the 7th International Workshop on Semantic Evaluations. pages 341–350.
- Caroline F Thorn, Teri E Klein, and Russ B Altman. 2013. PharmGKB: the pharmacogenomics knowledge base. Pharmacogenomics: Methods and Protocols pages 311–320.
- Yoshimasa Tsuruoka, Yuka Tateishi, Jin-Dong Kim, Tomoko Ohta, John McNaught, Sophia Ananiadou, and Junâichi Tsujii. 2005. Developing a robust part-of-speech tagger for biomedical text. In Panhellenic Conference on Informatics. pages 382–392.
- Santiago Vilar, Carol Friedman, and George Hripcsak. 2017. Detection of drug–drug interactions through data mining studies using clinical sources, scientific literature and social media. Briefings in bioinformatics .
- Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering 29(12):2724–2743.
- David Weininger. 1988. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences 28(1):31–36.
- Hong Yang, Chu Qin, Ying Hong Li, Lin Tao, Jin Zhou, Chun Yan Yu, Feng Xu, Zhe Chen, Feng Zhu, and Yu Zong Chen. 2016. Therapeutic target database update 2016: enriched resource for bench to clinical drug target and targeted pathway information. Nucleic acids research 44(D1):D1069–D1074.
- Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao, et al. 2014. Relation classification via convolutional deep neural network. In COLING. pages 2335–2344.
- Wei Zheng, Hongfei Lin, Ling Luo, Zhehuan Zhao, Zhengguang Li, Yijia Zhang, Zhihao Yang, and Jian Wang. 2017. An attention-based effective neural model for drug-drug interactions extraction. BMC Bioinformatics 18(1):445.
- Wei Zheng, Hongfei Lin, Zhehuan Zhao, Bo Xu, Yijia Zhang, Zhihao Yang, and Jian Wang. 2016. A graph kernel based on context vectors for extracting drug–drug interactions. Journal of biomedical informatics 61:34–43.