End-to-End Neural Entity Linking

End-to-End Neural Entity Linking

Nikolaos Kolitsas
ETH Zürich
nikos_kolitsas@hotmail.com
&Octavian-Eugen Ganea 11footnotemark: 1
ETH Zürich
octavian.ganea@inf.ethz.ch
\ANDThomas Hofmann
ETH Zürich
thomas.hofmann@inf.ethz.ch
  Equal contribution.
Abstract

Entity Linking (EL) is an essential task for semantic text understanding and information extraction. Popular methods separately address the Mention Detection (MD) and Entity Disambiguation (ED) stages of EL, without leveraging their mutual dependency. We here propose the first neural end-to-end EL system that jointly discovers and links entities in a text document. The main idea is to consider all possible spans as potential mentions and learn contextual similarity scores over their entity candidates that are useful for both MD and ED decisions. Key components are context-aware mention embeddings, entity embeddings and a probabilistic mention - entity map, without demanding other engineered features. Empirically, we show that our end-to-end method significantly outperforms popular systems on the Gerbil platform when enough training data is available. Conversely, if testing datasets follow different annotation conventions compared to the training set (e.g. queries/ tweets vs news documents), our ED model coupled with a traditional NER system offers the best or second best EL accuracy.

End-to-End Neural Entity Linking


Nikolaos Kolitsas thanks:   Equal contribution. ETH Zürich nikos_kolitsas@hotmail.com                        Octavian-Eugen Ganea 11footnotemark: 1 ETH Zürich octavian.ganea@inf.ethz.ch

Thomas Hofmann ETH Zürich thomas.hofmann@inf.ethz.ch

1 Introduction and Motivation

Towards the goal of automatic text understanding, machine learning models are expected to accurately extract potentially ambiguous mentions of entities from a textual document and link them to a knowledge base (KB), e.g. Wikipedia or Freebase. Known as entity linking, this problem is an essential building block for various Natural Language Processing tasks, e.g. automatic KB construction, question-answering, text summarization, or relation extraction.

1) MD may split a larger span into two mentions of
less informative entities:
B. Obama’s wife gave a speech […]
Federer’s coach […]
2) MD may split a larger span into two mentions of
incorrect entities:
Obama Castle was built in 1601 in Japan.
The Kennel Club is UK’s official kennel club.
A bird dog is a type of gun dog or hunting dog.
Romeo and Juliet by Shakespeare […]
Natural killer cells are a type of lymphocyte
Mary and Max, the 2009 movie […]
3) MD may choose a shorter span,
referring to an incorrect entity:
The Apple is played again in cinemas.
The New York Times is a popular newspaper.
4) MD may choose a longer span,
referring to an incorrect entity:
Babies Romeo and Juliet were born hours apart.
Table 1: Examples where MD may benefit from ED and viceversa. Each wrong MD decision (underlined) can be avoided by proper context understanding. The correct spans are shown in blue.

An EL system typically performs two tasks: i) Mention Detection (MD) or Named Entity Recognition (NER) when restricted to named entities – extracts entity references in a raw textual input, and ii) Entity Disambiguation (ED) – links these spans to their corresponding entities in a KB. Until recently, the common approach of popular systems (Ceccarelli et al., 2013; van Erp et al., 2013; Piccinno and Ferragina, 2014; Daiber et al., 2013; Hoffart et al., 2011; Steinmetz and Sack, 2013) was to solve these two sub-problems independently. However, the important dependency between the two steps is ignored and errors caused by MD/NER will propagate to ED without possibility of recovery (Sil and Yates, 2013; Luo et al., 2015). We here advocate for models that address the end-to-end EL task, informally arguing that humans understand and generate text in a similar joint manner, discussing about entities which are gradually introduced, referenced under multiple names and evolving during time (Ji et al., 2017). Further, we emphasize the importance of the mutual dependency between MD and ED. First, numerous and more informative linkable spans found by MD obviously offer more contextual cues for ED. Second, finding the true entities appearing in a specific context encourages better mention boundaries, especially for multi-word mentions. For example, in the first sentence of Table 1, understanding the presence of the entity Michelle Obama helps detecting its true mention ”B. Obama’s wife”, as opposed to separately linking B. Obama and wife to less informative concepts.

We propose a simple, yet competitive, model for end-to-end EL. Getting inspiration from the recent works of (Lee et al., 2017) and (Ganea and Hofmann, 2017), our model first generates all possible spans (mentions) that have at least one possible entity candidate. Then, each mention - candidate pair receives a context-aware compatibility score based on word and entity embeddings coupled with a neural attention and a global voting mechanisms. During training, we enforce the scores of gold entity - mention pairs to be higher than all possible scores of incorrect candidates or invalid mentions, thus jointly taking the ED and MD decisions.

Our contributions are:

  • We address the end-to-end EL task using a simple model that conditions the ”linkable” quality of a mention to the strongest context support of its best entity candidate. We do not require expensive manually annotated negative examples of non-linkable mentions. Moreover, we are able to train competitive models using little and only partially annotated documents (with named entities only such as the CoNLL-AIDA dataset).

  • We are among the first to show that, with one single exception, engineered features can be fully replaced by neural embeddings automatically learned for the joint MD & ED task.

  • On the Gerbil111http://gerbil.aksw.org/gerbil/ benchmarking platform, we empirically show significant gains for the end-to-end EL task when test and training data come from the same domain. Morever, when testing datasets follow different annotation schemes or exhibit different statistics, our method is still effective in achieving state-of-the-art or close performance, but needs to be coupled with a popular NER system.

2 Related Work

With few exceptions, MD/NER and ED are treated separately in the vast EL literature.

Traditional NER models usually view the problem as a word sequence labeling that is modeled using conditional random fields on top of engineered features (Finkel et al., 2005) or, more recently, using bi-LSTMs architectures (Lample et al., 2016; Chiu and Nichols, 2016; Liu et al., 2017) capable of learning complex lexical and syntactic features.

In the context of ED, recent neural methods (He et al., 2013; Sun et al., 2015; Yamada et al., 2016; Ganea and Hofmann, 2017; Le and Titov, 2018; Yang et al., 2018; Radhakrishnan et al., 2018) have established state-of-the-art results, outperforming engineered features based models. Context aware word, span and entity embeddings, together with neural similarity functions, are essential in these frameworks.

End-to-end EL is the realistic task and ultimate goal, but challenges in joint NER/MD and ED modeling arise from their different nature. Few previous methods tackle the joint task, where errors in one stage can be recovered by the next stage. One of the first attempts, (Sil and Yates, 2013) use a popular NER model to over-generate mentions and let the linking step to take the final decisions. However, their method is limited by the dependence on a good mention spotter and by the usage of hand-engineered features. It is also unclear how linking can improve their MD phase. Later, (Luo et al., 2015) presented one of the most competitive joint MD and ED models leveraging semi-Conditional Random Fields (semi-CRF). However, there are several weaknesses in this work. First, the mutual task dependency is weak, being captured only by type-category correlation features. The other engineered features used in their model are either NER or ED specific. Second, while their probabilistic graphical model allows for tractable learning and inference, it suffers from high computational complexity caused by the usage of the cartesian product of all possible document span segmentations, NER categories and entity assignments. Another approach is J-NERD (Nguyen et al., 2016) that addresses the end-to-end task using only engineered features and a probabilistic graphical model on top of sentence parse trees.

Figure 1: Our global model architecture shown for the mention The New York Times. The final score is used for both the mention linking and entity disambiguation decisions.

3 Neural Joint Mention Detection and Entity Disambiguation

We formally introduce the tasks of interest. For EL, the input is a text document (or a query or tweet) given as a sequence of words from a dictionary, . The output of an EL model is a list of mention - entity pairs , where each mention is a word subsequence of the input document, , and each entity is an entry in a knowledge base KB (e.g. Wikipedia), . For the ED task, the list of entity mentions that need to be disambiguated is additionally provided as input. The expected output is a list of corresponding annotations .

Note that, in this work, we only link mentions that have a valid gold KB entity, setting referred in (Röder et al., 2017) as InKB evaluation. Thus, we treat mentions referring to entities outside of the KB as ”non-linkable”. This is in line with few previous models, e.g. (Luo et al., 2015; Ganea and Hofmann, 2017; Yamada et al., 2016). We leave the interesting setting of discovering out-of-KB entities as future work.

We now describe the components of our neural end-to-end EL model, depicted in Figure 1. We aim for simplicity, but competitive accuracy.

Word and Char Embeddings.

We use pre-trained Word2Vec vectors (Mikolov et al., 2013). In addition, we train character embeddings that capture important word lexical information. Following (Lample et al., 2016), for each word independently, we use bidirectional LSTMs (Hochreiter and Schmidhuber, 1997) on top of learnable char embeddings. These character LSTMs do not extend beyond single word boundaries, but they share the same parameters. Formally, let be the character vectors of word . We use the forward and backward LSTMs formulations defined recursively as in (Lample et al., 2016):

(1)

Then, we form the character embedding of is from the hidden state of the forward LSTM corresponding to the last character concatenated with the hidden state of the backward LSTM corresponding to the first character. This is then concatenated with the pre-trained word embedding, forming the context-independent word-character embedding of . We denote the sequence of these vectors as and depict it as the first neural layer in Figure 1.

Mention Representation.

We find it crucial to make word embeddings aware of their local context, thus being informative for both mention boundary detection and entity disambiguation (leveraging contextual cues, e.g. ”newspaper”). We thus encode context information into words using a bi-LSTM layer on top of the word-character embeddings . The hidden states of forward and backward LSTMs corresponding to each word are then concatenated into context-aware word embeddings, whose sequence is denoted as .

Next, for each possible mention, we produce a fixed size representation inspired by (Lee et al., 2017). Given a mention , we first concatenate the embeddings of the first, last and the ”soft head” words of the mention:

(2)

The soft head embedding is built using an attention mechanism on top of the mention’s word embeddings, similar with (Lee et al., 2017):

(3)

However, we found the soft head embedding to only marginally improve results, probably due to the fact that most mentions are at most 2 words long. To learn non-linear interactions between the component word vectors, we project to a final mention representation with the same size as entity embeddings (see below) using a shallow feed-forward neural network FFNN (a simple projection layer):

(4)

Entity Embeddings.

We use fixed continuous entity representations, namely the pre-trained entity embeddings of (Ganea and Hofmann, 2017), due to their simplicity and compatibility with the pre-trained word vectors of (Mikolov et al., 2013). Briefly, these vectors are computed for each entity in isolation using the following exponential model that approximates the empirical conditional word-entity distribution obtained from co-occurrence counts.

(5)

Here, are fixed pre-trained word vectors and is the entity embedding to be trained. In practice, (Ganea and Hofmann, 2017) re-write this as a max-margin objective loss.

Candidate Selection.

For each span we select up to entity candidates that might be referred by this mention. These are top entities based on an empirical probabilistic entity - map built by (Ganea and Hofmann, 2017) from Wikipedia hyperlinks, Crosswikis (Spitkovsky and Chang, ) and YAGO (Hoffart et al., 2011) dictionaries. We denote by this candidate set and use it both at training and test time.

Final Local Score.

For each span that can possibly refer to an entity (i.e. ) and for each of its entity candidates , we compute a similarity score using embedding dot-product that supposedly should capture useful information for both MD and ED decisions. We then combine it with the log-prior probability using a shallow FFNN, giving the context-aware entity - mention score:

(6)

Long Range Context Attention.

In some cases, our model might be improved by explicitly capturing long context dependencies. To test this, we experimented with the attention model of (Ganea and Hofmann, 2017). This gives one context embedding per mention based on informative context words that are related to at least one of the candidate entities. We use this additional context embedding for computing dot-product similarity with any of the candidate entity embeddings. This value is fed as additional input of FFNN in Eq. 6. We refer to this model as long range context attention.

Training.

We assume a corpus with documents and gold entity - mention pairs is available. At training time, for each input document we collect the set of all (potentially overlapping) token spans for which . We then train the parameters of our model using the following minimization procedure:

(7)

where the violation term enforces the scores of gold pairs to be linearly separable from scores of negative pairs, i.e.

(8)

Note that, in the absence of annotated negative examples of ”non-linkable” mentions, we assume that all spans in and their candidates that do not appear in should not be linked. The model will be enforced to only output negative scores for all entity candidates of such mentions.

We call all spans training the above setting. Our method can also be used to perform ED only, in which case we train only on gold mentions, i.e. . This is referred as gold spans training.

Inference.

At test time, our method can be applied for both EL and ED only as follows. First, for each document in our validation or test sets, we select all possibly linkable token spans, i.e. for EL, or the input set of mentions for ED, respectively. Second, the best linking threshold is found on the validation set such that the micro F1 metric is maximized when only linking mention - entity pairs with score greater than . At test time, only entity - mention pairs with a score higher than are kept and sorted according to their scores; the final annotations are greedily produced based on this set such that only spans not overlapping with previously selected spans (of higher scores) are chosen.

Global Disambiguation

Our current model is ”local”, i.e. performs disambiguation of each candidate span independently. To enhance it, we add an extra layer to our neural network that will promote coherence among linked and disambiguated entities inside the same document, i.e. the global disambiguation layer. Specifically, we compute a ”global” mention-entity score based on which we produce the final annotations. We first define the set of mention-entity pairs that are allowed to participate in the global disambiguation voting, namely those that already have a high local score:

Since we initially consider all possible spans and for each span up to s candidate entities, this filtering step is important to avoid both undesired noise and exponential complexity for the EL task for which is typically much bigger than for ED. The final ”global” score for entity candidate of mention is given by the cosine similarity between the entity embedding and the normalized average of all other voting entities’ embeddings (of the other mentions ).

This is combined with the local score, yielding

The final loss function is now slightly modified. Specifically, we enforce the linear separability in two places: in (exactly as before), but also in , as follows

(9)

The inference procedure remains unchanged in this case, with the exception that it will only use the global score.

Coreference Resolution Heuristic.

In a few cases we found important to be able to solve simple coreference resolution cases (e.g. ”Alan” referring to ”Alan Shearer”). These cases are difficult to handle by our candidate selection strategy. We thus adopt the simple heuristic descried in (Ganea and Hofmann, 2017) and observed between 0.5% and 1% improvement on all datasets.

4 Experiments

F1@MA F1@MI

AIDA A

AIDA B

MSNBC

OKE-2015

OKE-2016

N3-Reuters-128

N3-RSS-500

Derczynski

KORE50

FREME 23.6 37.6 23.8 36.3 15.8 19.9 26.1 31.6 22.7 28.5 26.8 30.9 32.5 27.8 31.4 18.9 12.3 14.5
FOX 54.7 58.0 58.1 57.0 11.2 8.3 53.9 56.8 49.5 50.5 52.4 53.3 35.1 33.8 42.0 38.0 28.3 30.8
Babelfy 41.2 47.2 42.4 48.5 36.6 39.7 39.3 41.9 37.8 37.7 19.6 23.0 32.1 29.1 28.9 29.8 52.5 55.9
Entityclassifier.eu 43.0 44.7 42.9 45.0 41.4 42.2 29.2 29.5 33.8 32.5 24.7 27.9 23.1 22.7 16.3 16.9 25.2 28.0
Kea 36.8 40.4 39.0 42.3 30.6 30.9 44.6 46.2 46.3 46.4 17.5 18.1 22.7 20.5 31.3 26.5 41.0 46.8
DBpedia Spotlight 49.9 55.2 52.0 57.8 42.4 40.6 42.0 44.4 41.4 43.1 21.5 24.8 26.7 27.2 33.7 32.2 29.4 34.9
AIDA 68.8 72.4 71.9 72.8 62.7 65.1 58.7 63.1 0.0 0.0 42.6 46.4 42.6 42.4 40.6 32.6 49.6 55.4
WAT 69.2 72.8 70.8 73.0 62.6 64.5 53.2 56.4 51.8 53.9 45.0 49.2 45.3 42.3 44.4 38.0 37.3 49.6
Best baseline
69.2 72.8 71.9 73.0 62.7 65.1 58.7 63.1 51.8 53.9 52.4 53.3 45.3 42.4 44.4 38.0 52.5 55.9
base model 86.6 89.1 81.1 80.5 64.5 65.7 54.3 58.2 43.6 46.0 47.7 49.0 44.2 38.8 43.5 38.1 34.9 42.0
base model + att 86.5 88.9 81.9 82.3 69.4 69.5 56.6 60.7 49.2 51.6 48.3 51.1 46.0 40.5 47.9 42.3 36.0 42.2
base model + att + global 86.6 89.4 82.6 82.4 73.0 72.4 56.6 61.9 47.8 52.7 45.4 50.3 43.8 38.2 43.2 34.1 26.2 35.2
ED base model + att + global using
Stanford NER mentions
75.7 80.3 73.3 74.6 71.1 71.0 62.9 66.9 57.1 58.4 54.2 54.6 45.9 42.2 48.8 42.3 40.3 46.0
Table 2: EL strong matching results on the Gerbil platform. Micro and Macro F1 scores are shown. We highlight the best and second best models, respectively. Training was done on AIDA-train set.

Datasets and Metrics.

We used Wikipedia 2014 as our KB. We conducted experiments on the most important public EL datasets using the Gerbil platform (Röder et al., 2017). Datasets’ statistics are provided in Tables 5 and 6 (Appendix). This benchmarking framework offers reliable and trustable evaluation and comparison with state of the art EL/ED methods on most of the public datasets for this task. It also shows how well different systems generalize to datasets from very different domains and annotation schemes compared to their training sets. Moreover, it offers evaluation metrics for the end-to-end EL task, as opposed to some works that only evaluate NER and ED separately, e.g. (Luo et al., 2015).

As previously explained, we do not use the NIL mentions (without a valid KB entity) and only compare against other systems using the InKB metrics.

For training we used the biggest publicly available EL dataset, AIDA/CoNLL (Hoffart et al., 2011), consisting of a training set of 18,448 linked mentions in 946 documents, a validation set of 4,791 mentions in 216 documents, and a test set of 4,485 mentions in 231 documents.

We report micro and macro InKB F1 scores for both EL and ED. For EL, these metrics are computed both in the strong matching and weak matching settings. The former requires exactly predicting the gold mention boundaries and their entity annotations, whereas the latter gives a perfect score to spans that just overlap with the gold mentions and are linked to the correct gold entities.

solved%
matched gold entity%
position of ground
truth in
Number of
mentions

ED Global model

ED Base + Att model

ED Base model

EL Global model

EL Base + Att model

EL Base model

EL Global -log

1 3390 98.8 98.8 98.6 98.9 98.3 98.8 96.7 99.0 96.6 98.6 96.2 99.0 93.3 96.7
2 655 89.9 89.9 88.1 88.5 88.5 88.9 86.8 90.8 86.8 90.8 85.0 88.4 86.9 89.8
3 108 83.3 84.3 81.5 82.4 75.9 78.7 79.1 84.5 80.2 84.7 74.8 81.1 84.3 88.9
4-8 262 78.2 78.2 76.3 79.0 74.8 76.0 69.5 78.2 68.8 78.7 68.7 76.0 78.9 83.5
9+ 247 59.9 63.6 53.4 60.7 53.0 58.7 47.8 58.2 46.2 59.4 50.4 54.8 62.7 67.5
Table 3: AIDA A dataset: Gold mentions are split by the position they appear in the dictionary. In each cell, the upper value is the percentage of the gold mentions that were annotated with the correct entity (recall), whereas the lower value is the percentage of gold mentions for which our system’s highest scored entity is the ground truth entity, but that might not be annotated in the end because its score is below the threshold .

Baselines.

We compare with popular and state-of-the-art EL and ED public systems, e.g. WAT (Piccinno and Ferragina, 2014), Dbpedia Spotlight (Daiber et al., 2013), NERD-ML (van Erp et al., 2013), KEA (Steinmetz and Sack, 2013), AIDA (Hoffart et al., 2011), FREME, FOX (Speck and Ngomo, 2014), PBOH (Ganea et al., 2016) or Babelfy (Moro et al., 2014). These are integrated within the Gerbil platform. However, we did not compare with (Luo et al., 2015) and other models outside Gerbil that do not use end-to-end EL metrics.

Training Details and Model Hyper-Parameters.

We use the same model hyper-parameters for both EL and ED training. The only difference between the two settings is the span set : EL uses all spans training, whereas ED uses gold spans training settings as explained before.

Our pre-trained word and entity embeddings are 300 dimensional, while 50 dimensional trainable character vectors were used. The char LSTMs have also hidden dimension of 50. Thus, word-character embeddings are 400 dimensional. The contextual LSTMs have hidden size of 150, resulting in 300 dimensional context-aware word vectors. We apply dropout on the concatenated word-character embeddings, on the output of the bidirectional context LSTM and on the entity embeddings used in Eq. 6. The three FFNNs in our model are simple projections without hidden layers (no improvements were obtained with deeper layers). For the long range context attention we used a word window size of K = 200 and keep top R = 10 words after the hard attention layer (notations from (Ganea and Hofmann, 2017)). We use at most entity candidate per mention both at train and test time. is set to 0.2 without further investigations. is set to 0, but a value of 0.1 was giving similar results.

For the loss optimization we use Adam (Kingma and Ba, 2014) with a learning rate of 0.001. We perform early stopping by evaluating the model on the AIDA validation set each 10 minutes and stopping after 6 consecutive evaluations with no significant improvement in the macro F1 score.

1) Annotated document:
SOCCER - [SHEARER ] NAMED AS [1 [1ENGLAND] CAPTAIN]. [ LONDON ] 1996-08-30 The world ’s costliest footballer [ Alan Shearer ] was named as the new [England ] captain on Friday. The 26-year-old, who joined [Newcastle] for 15 million pounds sterling, takes over from [Tony Adams], who led the side during the [ European ] championship in June, and former captain [ David Platt ] . [2Adams] and [ Platt ] are both injured and will miss [England]’s opening [3[World Cup]] qualifier against [Moldova] on Sunday . [Shearer] takes the captaincy on a trial basis , but new coach [Glenn Hoddle] said he saw no reason why the former [Blackburn] and [Southampton] skipper should not make the post his own . ”I ’m sure [4Alan] is the man for the job , ” [Hoddle] said . […] I spoke to [5Alan] he was up for it […]. [Shearer] ’s [Euro 96] striking partner […].
Analysis:
[1 is considered a false negative and the ground truth is the entity England_national_football_team. Our annotation was for the span ”ENGLAND CAPTAIN” and wrongly linked to the England_cricket_team.
[3 The ground truth here is 1998_FIFA_World_Cup whereas our model links it to FIFA_World_Cup.
[2,4,5 are correctly solved due to our coreference resolution heuristic.
2) Annotated document:
[ N. Korea ] urges [ S. Korea ] to return war veteran . [ SEOUL ] 1996-08-31 [ North Korea ] demanded on Saturday that [ South Korea ] return a northern war veteran who has been in the [1 South ] since the 1950-53 war , [ Seoul ] ’s unification ministry said . ” …I request the immediate repatriation of Kim In-so to [ North Korea ] where his family is waiting , ” [1 North Korean ] Red Cross president Li Song-ho said in a telephone message to his southern couterpart , [2 Kang Young-hoon ] . Li said Kim had been critically ill with a cerebral haemorrhage . The message was distributed to the press by the [ South Korean ] unification ministry . Kim , an unrepentant communist , was captured during the [2 [3 Korean ] War ] and released after spending more than 30 years in a southern jail . He submitted a petition to the [ International Red Cross ] in 1993 asking for his repatriation . The domestic [ Yonhap ] news agency said the [ South Korean ] government would consider the northern demand only if the [3 North ] accepted [ Seoul ] ’s requests , which include regular reunions of families split by the [4 [4 Korean ] War ] . Government officials were not available to comment . [ South Korea ] in 1993 unconditionally repatriated Li In-mo , a nothern partisan seized by the [5 South ] during the war and jailed for more than three decades.
Analysis:
[1, [3, and [5 cases illustrate the main source of errors. These are false negatives in which our model has the correct ground truth entity pair as the highest scored one for that mention, but since it is not confident enough (score ) it decides not to annotate that mention. In this specific document these errors could probably be avoided easily with a better coreference resolution mechanism.
[3 and [4 cases illustrate that the gold standard can be problematic. Specifically, instead of annotating the whole span Korean War and linking it to the war of 1950, the gold annotation only include Korean and link it to the general entity of Korea_(country).
[2 is correctly annotated by our system but it is not included in the gold standard.
Table 4: Error analysis on a sample document. Green corresponds to true positive (correctly discovered and annotated mention), red to false negative (ground truth mention or entity that was not annotated) and orange to false positive (incorrect mention or entity annotation ).

Results and Discussion.

The following models are used.

i) Base model: only uses the mention local score and the log-prior. It does not use long range attention, nor global disambiguation. It does not use the head attention mechanism.

ii) Base model + att: the Base Model plus Long Range Context Attention.

ii) Base model + att + global: our Global Model (depicted in figure 1)

iv) ED base model + att + global Stanford NER: our ED Global model that runs on top of the detected mentions of the Stanford NER system (Finkel et al., 2005).

EL strong and weak matching results are presented in Tables 2 and  7 (Appendix).

We first note that our system outperforms all baselines on the end-to-end EL task on both AIDA-A (dev) and AIDA-B (test) datasets, which are the biggest EL datasets publicly available. Moreover, we surpass all competitors on both EL and ED by a large margin, at least 9%, showcasing the effectiveness of our method. We also outperform systems that optimize MD and ED separately, including our ED base model + att + global Stanford NER. This demonstrates the merit of joint MD + ED optimization.

In addition, one can observe that weak matching EL results are comparable with the strong matching results, showcasing that our method is very good at detecting mention boundaries.

At this point, our main goal was achieved: if enough training data is available with the same characteristics or annotation schemes as the test data, then our joint EL offers the best model. This is true not only when training on AIDA, but also for other types of datasets such as queries (Table 11) or tweets (Table 12). However, when testing data has different statistics or follows different conventions than the training data, our method is shown to work best in conjunction with a state-of-the-art NER system as it can be seen from the results of our ED base model + att + global Stanford NER for different datasets in Table 2. It is expected that such a NER system designed with a broader generalization scheme in mind would help in this case, which is confirmed by our results on different datasets.

While the main focus of this paper is the end-to-end EL and not the ED-only task, we do show ED results in Tables 8 and  4. We observe that our models are slightly behind recent top performing systems, but our unified EL - ED architecture has to deal with other challenges, e.g. being able to exchange global information between many more mentions at the EL stage, and is thus not suitable for expensive global ED strategies. We leave bridging this gap for ED as future work.

Additional results and insights are shown in the Appendix.

Ablation study

Table 3 shows an ablation study of our method. One can see that the prior is very helpful for correctly linking unambiguous mentions, but is introducing noise when gold entities are not frequent. For this category of rare entities, removing this prior completely will result in a significant improvement, but this is not a practical choice since the gold entity is unknown at test time.

Error Analysis.

We conducted a qualitative experiment shown in Table 4. We showcase correct annotations, as well as errors done by our system on the AIDA datasets. Inspecting the output annotations of our EL model, we discovered the remarkable property of not over-generating incorrect mentions, nor under-generating (missing) gold spans. We also observed that additional mentions generated by our model do correspond in the majority of time to actual KB entities, but are incorrectly forgotten from the gold annotations.

5 Conclusion

We presented the first neural end-to-end entity linking model and show the benefit of jointly optimizing entity recognition and linking. Leveraging key components, namely word, entity and mention embeddings, we prove that engineered features can be almost completely replaced by modern neural networks. Empirically, on the established Gerbil benchmarking platform, we exhibit state-of-the-art performance for EL on the biggest public dataset, AIDA/CoNLL, also showing good generalization ability on other datasets with very different characteristics when combining our model with the popular Stanford NER system.

Our code is publicly available222https://github.com/dalab/end2end_neural_el.

References

  • Ceccarelli et al. (2013) Diego Ceccarelli, Claudio Lucchese, Salvatore Orlando, Raffaele Perego, and Salvatore Trani. 2013. Dexter: an open source framework for entity linking. In Proceedings of the sixth international workshop on Exploiting semantic annotations in information retrieval, pages 17–20. ACM.
  • Chiu and Nichols (2016) Jason PC Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional lstm-cnns. Transactions of the Association for Computational Linguistics, 4:357–370.
  • Daiber et al. (2013) Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N Mendes. 2013. Improving efficiency and accuracy in multilingual entity extraction. In Proceedings of the 9th International Conference on Semantic Systems, pages 121–124. ACM.
  • van Erp et al. (2013) MGJ van Erp, G Rizzo, and R Troncy. 2013. Learning with the web: Spotting named entities on the intersection of nerd and machine learning. In CEUR workshop proceedings, pages 27–30.
  • Finkel et al. (2005) Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd annual meeting on association for computational linguistics, pages 363–370. Association for Computational Linguistics.
  • Ganea et al. (2016) Octavian-Eugen Ganea, Marina Ganea, Aurelien Lucchi, Carsten Eickhoff, and Thomas Hofmann. 2016. Probabilistic bag-of-hyperlinks model for entity linking. In Proceedings of the 25th International Conference on World Wide Web, pages 927–938. International World Wide Web Conferences Steering Committee.
  • Ganea and Hofmann (2017) Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2619–2629.
  • He et al. (2013) Zhengyan He, Shujie Liu, Mu Li, Ming Zhou, Longkai Zhang, and Houfeng Wang. 2013. Learning entity representation for entity disambiguation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 30–34.
  • Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780.
  • Hoffart et al. (2011) Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 782–792. Association for Computational Linguistics.
  • Ji et al. (2017) Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi, and Noah A Smith. 2017. Dynamic entity representations in neural language models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1830–1839.
  • Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  • Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL-HLT, pages 260–270.
  • Le and Titov (2018) Phong Le and Ivan Titov. 2018. Improving entity linking by modeling latent relations between mentions. arXiv preprint arXiv:1804.10637.
  • Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197.
  • Liu et al. (2017) Liyuan Liu, Jingbo Shang, Frank Xu, Xiang Ren, Huan Gui, Jian Peng, and Jiawei Han. 2017. Empower sequence labeling with task-aware neural language model. arXiv preprint arXiv:1709.04109.
  • Luo et al. (2015) Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Zaiqing Nie. 2015. Joint entity recognition and disambiguation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 879–888.
  • Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119.
  • Moro et al. (2014) Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity linking meets word sense disambiguation: a unified approach. Transactions of the Association for Computational Linguistics, 2:231–244.
  • Nguyen et al. (2016) Dat Ba Nguyen, Martin Theobald, and Gerhard Weikum. 2016. J-nerd: joint named entity recognition and disambiguation with rich linguistic features. Transactions of the Association for Computational Linguistics, 4:215–229.
  • Piccinno and Ferragina (2014) Francesco Piccinno and Paolo Ferragina. 2014. From tagme to wat: a new entity annotator. In Proceedings of the first international workshop on Entity recognition & disambiguation, pages 55–62. ACM.
  • Radhakrishnan et al. (2018) Priya Radhakrishnan, Partha Talukdar, and Vasudeva Varma. 2018. Elden: Improved entity linking using densified knowledge graphs. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 1844–1853.
  • Röder et al. (2017) Michael Röder, Ricardo Usbeck, and Axel-Cyrille Ngonga Ngomo. 2017. Gerbil–benchmarking named entity recognition and linking consistently. Semantic Web, (Preprint):1–21.
  • Sil and Yates (2013) Avirup Sil and Alexander Yates. 2013. Re-ranking for joint named-entity recognition and linking. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management, pages 2369–2374. ACM.
  • Speck and Ngomo (2014) René Speck and Axel-Cyrille Ngonga Ngomo. 2014. Ensemble learning for named entity recognition. In International semantic web conference, pages 519–534. Springer.
  • (26) Valentin I Spitkovsky and Angel X Chang. A cross-lingual dictionary for english wikipedia concepts.
  • Steinmetz and Sack (2013) Nadine Steinmetz and Harald Sack. 2013. Semantic multimedia information retrieval based on contextual descriptions. In Extended Semantic Web Conference, pages 382–396. Springer.
  • Sun et al. (2015) Yaming Sun, Lei Lin, Duyu Tang, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2015. Modeling mention, context and entity with neural networks for entity disambiguation. In Twenty-Fourth International Joint Conference on Artificial Intelligence.
  • Yamada et al. (2016) Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. Joint learning of the embedding of words and entities for named entity disambiguation. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 250–259.
  • Yang et al. (2018) Yi Yang, Ozan Irsoy, and Kazi Shefaet Rahman. 2018. Collective entity disambiguation with structured gradient tree boosting. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 777–786.
position of ground
truth in
AIDA Train AIDA A (dev) AIDA B (test)
1 13704 75.6% 3390 72.7% 3078 69.8%
2 2339 12.9% 655 14% 651 14.8%
3 565 3.1% 108 2.3% 184 4.2%
4-8 843 4.6% 262 5.6% 308 7%
9+ 686 3.8% 247 5.3% 187 4.2%
Total number of non-NIL mentions 18137 4662 4408
Unambiguous mentions, i.e.
for which ground truth is
the unique candidate
3575 19.7% 936 20.1% 833 18.9%
Table 5: Statistics on the number of mentions for various settings and for all AIDA datasets.

Dataset

Topic

Type of annotations

Exhaustive annotations

Num Docs

Avg. Words/Doc.

Num InKB Anns

Num of gold entities
not in our KB

Recall 30 (%)

Recall 10 (%)



AIDA
news NE 231 190 4483 5 98.3 95.7

AQUAINT
news NE&N 50 221 727 16 93.5 92.6

MSNBC
news NE 20 544 653 3 92.8 90.3

ACE2004
news NE 57 374 257 2 87.9 86.8

Derczynski
tweets NE 182 21 210 32 71.0 69.5

Microposts2016 dev
tweets NE 100 14 251 50 59.8 59.8

N3-Reuters-128
news NE 128 124 631 7 68.9 64.3

N3-RSS-500
news NE 500 31 519 11 81.3 79.4

DBpedia Spotlight
news NE&N 58 29 330 7 90.6 90.3

KORE 50
mixed NE 50 13 144 1 88.2 75.7

ERD2014
queries NE 91 3.5 59 1 72.9 72.9

Gerdaq
queries NE&N 250 3.6 408 3 70.8 68.6

OKE
wikipedia NE&roles 55 26.6 288 3 81.9 81.3

Table 6: Test datasets statistics. Topic has the following possible values: news articles, twitter messages, search queries, or sentences extracted from Wikipedia articles. Type of annotations can be NE (Named Entities i.e. persons, locations, organizations) and/or N (Nouns i.e. dataset annotates even simple nouns). Exhaustive annotations means that these datasets annotate all entities that appear in their text and that fall into the selected categories of Type of Annotations. Recall 30/10 shows the percentage of the gold mentions for which their ground truth entity is among the first 30/10 candidate entities returned by our dictionary.
F1@MA F1@MI

AIDA A

AIDA B

MSNBC

OKE-2015

OKE-2016

N3-Reuters-128

N3-RSS-500

Derczynski

KORE50

FREME 23.9 38.3 24.3 37.0 17.6 22.6 26.8 32.7 23.0 29.0 28.2 33.5 34.6 32.4 31.8 20.4 13.3 15.5
FOX 54.9 58.1 58.4 57.4 26.3 21.2 56.4 58.7 50.1 51.4 53.8 55.2 37.7 37.2 43.4 41.5 30.1 32.9
Babelfy 42.1 48.1 43.3 49.5 40.2 44.9 42.8 45.7 40.1 39.7 29.8 33.6 36.6 34.6 32.3 34.3 53.0 56.6
Entityclassifier.eu 44.9 47.0 45.3 47.6 47.2 48.9 32.7 33.0 35.7 33.6 31.8 35.3 24.5 24.1 18.1 19.3 27.1 30.1
Kea 38.7 42.9 42.1 44.7 35.0 36.3 53.9 54.4 54.9 55.2 22.1 22.4 25.9 24.7 33.2 29.2 46.5 50.5
DBpedia Spotlight 55.5 58.3 53.1 58.8 49.6 49.2 49.6 52.2 49.2 49.8 36.0 38.0 35.5 35.1 44.0 41.5 28.8 35.4
AIDA 69.0 72.6 72.3 73.5 66.0 68.1 61.7 65.8 43.3 47.2 44.7 48.5 47.3 47.5 41.5 34.8 52.9 58.6
WAT 69.7 73.3 71.6 73.6 66.7 68.9 58.4 60.7 55.2 56.6 54.4 58.3 47.6 46.5 47.1 43.6 39.9 52.2
Best baseline
69.7 73.3 72.3 73.6 66.7 68.9 61.7 65.8 55.2 56.6 54.4 58.3 47.6 47.5 47.1 43.6 53.0 58.6
base model 87.0 89.5 81.7 80.8 67.3 68.8 56.2 60.1 45.0 47.0 48.7 51.4 46.5 42.8 44.9 40.1 36.9 44.7
base model + att 87.1 89.3 82.4 82.6 72.6 72.6 59.0 63.0 49.2 51.6 51.0 54.7 48.4 44.6 49.3 44.4 38.0 45.1
base model + att + global 87.2 89.8 83.2 82.8 75.7 74.7 59.4 64.6 48.1 53.1 47.0 53.2 46.3 42.7 43.8 34.9 28.2 38.3
ED base model + att + global using
Stanford NER mentions
76.0 80.5 73.9 75.0 74.7 74.8 65.3 68.9 57.7 59.3 54.9 56.9 48.6 46.1 50.2 44.3 41.9 47.8
Table 7: EL weak matching results on the Gerbil platform. Micro and Macro F1 scores are shown. We highlight in red and blue the best and second best models, respectively. Training was done on AIDA-train set.
  • F1@MA F1@MI

    ACE2004

    AIDA A

    AIDA B

    AQUAINT

    MSNBC

    DBpediaSpotlight

    Derczynski

    ERD2014

    GERDAQ-Dev

    GERDAQ-Test

    KORE50

    N3-Reuters-128

    N3-RSS-500

    OKE-2015

    OKE-2016

    AGDISTIS 78.1 65.6 49.2 55.6 54.2 54.9 58.9 60.4 72.9 74.0 35.3 39.3 50.8 44.5 63.2 32.0 21.4 17.8 22.8 20.3 30.0 33.3 68.5 63.8 51.4 51.8 61.4 62.0 59.6 58.8
    FREME 73.5 60.3 25.3 33.0 25.7 40.3 43.7 57.6 17.9 24.3 37.9 52.5 48.3 38.2 71.3 51.8 37.6 44.2 33.4 39.6 13.8 17.4 30.2 31.7 45.5 43.9 25.0 33.3 27.4 35.0
    FOX 38.6 0.0 55.5 58.9 59.6 58.0 0.0 0.0 16.2 13.3 11.7 15.9 51.2 44.3 51.7 0.0 11.2 0.0 10.6 0.0 26.9 31.1 56.6 58.1 52.9 53.6 54.9 59.0 50.1 51.4
    Babelfy 76.6 61.9 66.4 71.9 70.0 75.8 70.3 70.4 72.7 76.2 51.0 51.8 64.4 63.2 69.6 45.0 30.6 34.3 31.5 35.0 70.5 74.1 54.6 55.4 62.8 64.1 65.1 65.9 60.2 59.8
    Entityclassifier.eu 72.6 61.9 53.6 53.7 54.3 53.9 39.9 43.1 54.1 54.6 19.9 25.5 43.2 32.6 51.7 0.0 11.2 0.0 10.6 0.0 25.9 29.9 40.3 44.4 37.6 44.9 35.3 37.6 40.3 41.4
    Kea 87.4 79.5 62.6 63.5 69.0 68.5 79.9 80.6 80.9 79.3 75.5 72.2 63.6 61.5 87.4 71.0 56.8 60.6 59.7 63.3 50.9 60.8 62.7 62.7 58.6 64.5 68.5 69.1 67.7 69.1
    DBpedia Spotlight 73.4 53.9 51.5 53.2 53.7 56.1 50.6 51.8 43.6 42.1 70.1 72.6 50.3 43.3 65.2 38.4 49.2 54.4 46.8 54.0 48.7 52.3 41.8 43.4 42.6 34.6 30.4 35.8 43.0 43.1
    AIDA 89.5 79.8 71.4 74.7 77.8 77.0 56.6 57.1 70.3 74.6 21.4 24.9 64.5 63.9 79.1 63.7 26.5 23.5 27.2 25.0 63.2 69.1 51.2 56.9 62.9 65.5 61.6 63.0 0.0 0.0
    WAT 86.9 79.6 75.6 78.5 79.8 80.5 75.6 75.4 79.7 78.8 68.8 67.1 70.4 69.5 86.5 76.0 61.2 63.9 57.4 60.7 52.4 62.2 59.2 63.1 62.8 63.9 62.2 64.9 0.0 0.0
    PBOH 88.0 83.2 77.3 80.1 79.0 80.4 83.3 84.1 85.5 86.1 80.5 80.1 63.6 64.8 91.0 80.7 67.0 68.8 64.4 67.8 58.6 63.2 72.1 69.0 47.6 55.2 67.4 67.6 0.0 0.0
    Best baseline
    89.5 83.2 77.3 80.1 79.8 80.5 83.3 84.1 85.5 86.1 80.5 80.1 70.4 69.5 91.0 80.7 67.0 68.8 64.4 67.8 70.5 74.1 72.1 69.0 62.9 65.5 68.5 69.1 67.7 69.1
    ed base model 89.9 83.5 86.5 88.8 82.7 82.7 78.4 79.7 82.4 82.5 71.1 71.0 63.5 62.7 81.0 65.3 54.2 58.4 52.7 57.6 45.1 50.8 62.9 66.4 64.0 65.8 69.7 70.0 71.9 73.0
    ed base model + att 89.5 84.1 86.0 88.9 83.6 83.1 81.2 82.2 88.4 86.4 67.8 72.8 64.3 65.0 79.9 65.2 41.8 49.2 46.6 54.7 48.3 55.5 62.5 67.3 67.2 68.4 73.7 74.5 75.1 76.1
    ed base model + att + global 92.0 85.5 86.6 89.0 83.8 83.0 82.3 83.2 88.5 86.2 74.4 78.8 65.3 65.4 76.9 61.4 38.6 46.3 43.2 53.7 52.4 60.8 63.4 67.3 66.6 68.6 73.2 74.0 76.7 78.1
Table 8: ED results on the Gerbil platform. Micro and Macro F1 scores are shown. We highlight in red and blue the best and second best models, respectively. Training was done on AIDA-train set.
F1@MA F1@MI

AIDA A

AIDA B

ACE2004

AQUAINT

MSNBC

ed base model 92.1 93.3 88.4 86.7 84.4 85.4 85.9 86.1 88.2 89.6
ed base model + att 91.7 93.5 89.0 87.2 87.3 87.0 89.1 89.4 93.2 92.3
ed base model + att + global 92.4 93.7 89.1 87.3 89.2 88.1 89.4 89.7 93.3 92.5
Table 9: ED results matched locally (not on Gerbil) using KB entity IDs. Micro and Macro F1 scores are shown. For the ED task there is a significant discrepancy between the local and the Gerbil scores which likely has two main causes: a different URL-based matching scheme used in Gerbil444https://github.com/dice-group/gerbil/issues/98and parsing or alignment errors.
Rec@MA Rec@MI

ACE2004

AQUAINT

FREME 44.8 30.8 23.7 23.7
FOX 0.0 0.0 0.0 0.0
Babelfy 10.0 17.8 34.9 35.8
Entityclassifier.eu 35.7 59.7 28.4 29.2
Kea 23.7 40.3 36.1 35.9
DBpedia Spotlight 39.1 60.5 45.1 45.2
AIDA 28.0 48.6 36.0 36.5
WAT 26.8 44.7 35.1 35.5
Best baseline
44.8 60.5 45.1 45.2
model+att+global 40.3 68.3 39.4 40.4
Table 10: El strong matching results on the Gerbil platform. Micro and Macro recall scores are shown for ACE2004 and AQUAINT datasets that only annotate each entity once, thus F1 scores not being useful. We highlight in red and blue the best and second best models, respectively. Our models were trained on AIDA as before.
F1@MA F1@MI

GERDAQ-Dev

GERDAQ-Test

DBpediaSpotlight

FREME 11.2 0.0 10.6 0.0 11.6 13.0
FOX 11.2 0.0 10.6 0.0 11.3 15.3
Babelfy 19.6 15.5 19.6 17.5 8.7 13.1
Entityclassifier.eu 11.2 0.0 10.6 0.0 18.3 23.1
Kea 40.2 40.2 40.2 43.7 42.0 42.6
DBpedia Spotlight 29.6 27.8 29.4 31.4 31.8 37.6
AIDA 11.2 0.0 10.6 0.0 14.4 19.1
WAT 12.9 3.3 12.6 6.0 14.5 17.0
Best baseline
40.2 40.2 40.2 43.7 42.0 42.6
model+att+global on gerdaq train 42.7 44.9 40.0 41.2 33.4 34.8
Table 11: El strong matching results on the Gerbil platform. Micro and Macro F1 scores are shown. Training was done on Gerdaq train dataset. We highlight in red and blue the best and second best models, respectively.
F1@MA F1@MI

MicropDev

MicropTest

FREME 3.9 3.8 77.9 3.7
FOX 6.1 7.5 71.9 3.7
Babelfy 6.2 9.5 52.5 2.8
Entityclassifier.eu 23.5 29.1 9.2 2.4
Kea 37.3 38.8 51.6 6.5
DBpedia Spotlight 29.1 34.0 13.6 5.2
AIDA 6.4 9.7 73.3 5.0
WAT 0.0 0.0 0.0 0.0
Best baseline
37.3 38.8 77.9 6.5
model on micropost train 57.3 63.7 17.4 10.0
Table 12: El strong matching results on the Gerbil platform when training on Micropost-train 2016. Micro and Macro F1 scores are shown. The majority of the errors are due to the newer KB version used in this dataset. We highlight in red and blue the best and second best models, respectively.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
264248
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description