Semantic Content Filtering with Wikipedia and Ontologies
The use of domain knowledge is generally found to improve query efficiency in content filtering applications. In particular, tangible benefits have been achieved when using knowledge-based approaches within more specialized fields, such as medical free texts or legal documents. However, the problem is that sources of domain knowledge are time-consuming to build and equally costly to maintain. As a potential remedy, recent studies on Wikipedia suggest that this large body of socially constructed knowledge can be effectively harnessed to provide not only facts but also accurate information about semantic concept-similarities. This paper describes a framework for document filtering, where Wikipedia’s concept-relatedness information is combined with a domain ontology to produce semantic content classifiers. The approach is evaluated using Reuters RCV1 corpus and TREC-11 filtering task definitions. In a comparative study, the approach shows robust performance and appears to outperform content classifiers based on Support Vector Machines (SVM) and C4.5 algorithm.
Recently, ontologies have become a broadly accepted solution for integrating semantic knowledge into document modeling tasks. By using ontologies as a source of background knowledge, the IR expert systems have achieved increased contextual understanding and ability to do accurate conceptual indexing. Yet, these advantages are not gained without time-consuming ontology engineering. To reduce the costs of managing complicated knowledge models, there is an ongoing quest for alternative approaches. Therefore, an emerging trend is to consider the use of socially developed sources of semantic information, such as Wikipedia, to complement expensive domain ontologies; see Medelyan et al. .
In this paper, we propose a new framework for document filtering, Wiki-SR
The evaluation of the Wiki-SR framework was carried out using Reuters RCV1 corpus. The data set was chosen due to the relevance judgements and topic definitions supplied by the assessors of TREC-11 filtering track. As benchmarks, we used the Support Vector Machines (SVM) and the decision-tree algorithm C4.5, which are well-known for their solid performance. Both algorithms were built using several different feature sets ranging from bag-of-words to Wikipedia- and ontology-based document models. As primary performance measures, we used F-score, precision, and recall. The overall result appeared very positive for the heuristic Wiki-SR model, which outperformed the benchmarks in terms of F-score by a fair margin.
The rest of this paper is organized as follows. Section 2 gives a short review of related work and summarizes the contributions of this paper. Section 3 provides an overview of the Wiki-SR framework. The components of the document model used by the semantic rules are introduced in Section 4. The notion of concept-relatedness measures and the definition of the Wiki-SR model are presented in Section 5. An experiment based on the algorithm is given in Section 6. We conclude in Section 7.
2Related work and contributions
Today, Wikipedia is increasingly recognized as a valuable source of semantic knowledge for various natural language processing tasks; see Medelyan et al.  for a comprehensive review. As pioneering research in this field, we acknowledge the work done by Milne et al. , Gabrilovich and Markovich , Medelyan et al. , Mihalcea and Csomai , and Strube and Ponzetto , who have examined different ways of using Wikipedia to compute semantic relatedness between concepts and perform automated cross-referencing of documents.
However, considering the large potential offered by Wikipedia, surprisingly little research has examined its use for document profiling, clustering and classification tasks. Perhaps, the best known papers, where Wikipedia has been used for information retrieval tasks, are the studies on query expansion by Gregorowicz and Kramer  and Milne et al. . Later, these have been followed by research on how pseudo-relevance feedback and explicit semantic analysis can be used to improve queries; see Li et al.  and Egozi et al. . Among the latest studies are also the papers by Wang et al.  where semantic kernels are derived from Wikipedia to be used in SVM classifiers and co-clustering methods.
In this paper, our main contribution to the existent literature is the introduction of Wikipedia-based semantic rules for document filtering. This technique capitalizes on the simplicity of ordinary boolean queries but improves it by performing an implicit expansion to take into account the actual semantic meanings of the concepts involved in the query. However, the idea in our Wiki-SR framework is quite different from what has become known as query expansion as considered by Milne et al. . Whereas explicit query expansion is commonly defined as addition of terms and phrases to the original query phrase to produce a more comprehensive and also more complex expression, we never add new terms to the original query. Instead, the synonyms and closely related concepts are taken into account implicitly through similarity measures in the first evaluation step of the semantic rule. Furthermore, although the Wiki-SR rules in certain sense build a semantic kernel to model concept-relatednesses, the system is closer to a semantic boolean query than a kernelised SVM-classifier.
The second contribution of this paper is concerned with the way of modelling document content. The model described in Section 4 combines three different approaches: a Wikipedia, an ontology, and the classical bag-of-words content models. Here, the Wikipedia-based content model is further divided into sub-models representing general concepts and named-entities (NE) by using a Conditional Random Fields (CRF) classifier. The benefit is that this separation allows us to take into account the inherent differences in the narrowness of concept definitions. In addition to Wikipedia, we also utilise a small business ontology (BTO) to account for specialized economic concepts which are not equally well captured by Wikipedia. The BTO ontology also provides a well-defined hierarchy, which has proven to be effective in defining Wiki-SR rules.
The Wiki-SR framework is an interactive content filtering system that combines the relevance statements supplied by the user with the concept-relatedness information in Wikipedia to produce semantic rules for identifying the documents that match the given topic. To summarize the steps involved in the filtering process, we split the overview of the framework into two parts: (1) the content modeling component; and (2) the Wikipedia-based semantic rule component.
The first component, content modeling, is shown in Figure 1. Once an incoming document has been preprocessed, the profile is constructed in three parts: a Wikipedia-content model (Section Section 4.1), an Ontology-content model (Section Section 4.2), and the classical Bag of Words (BOW) representation. Together, these constitute the document model (Section Section 4.3) used for filtering tasks. Although, there is overlap between the models, they tend to capture different aspects in the document, which makes them complementary. The resources used in profiling consists of the Wikipedia’s link-structure (Wiki DB), the business term ontology BTO (Ontology DB), and named-entity recognizer (NER). For further details on document model, see Section 4.
The second component, Wikipedia-based semantic rule, is described in Figure 2. The purpose is to represent the user’s information needs as compositions of standard boolean queries, which are expressed in terms of Wikipedia and ontology concepts. The resulting rule is referred to as a Wikipedia-based semantic rule (Wiki-SR rule), which represents the topic of user’s interest.
In order to build the rules (Wiki-SR builder in Figure 2), the user is expected to supply a topic statement (see Figure 3) defining the central concepts, and a small set of relevant/irrelevant example documents that can be used as a training data for learning the Wiki-SR rule that best describes the given topic. Each topic statement stands for a single topic by providing a short textual description of the concepts which are relevant or irrelevant. For implementation of the Wiki-SR builder, see discussion in Section 5.2.
Once the semantic rule has been learned, it is given to an evaluator (Wiki-SR evaluator in Figure 2) which checks whether the incoming documents match the given rule based on their profiles. This is the stage where the main benefit of constructing the rules in terms of Wikipedia’s concepts is realized. While checking the potential matches, the evaluator uses Wikipedia’s concept-relatedness information to judge whether the query’s concepts are present in the given document.
Research on ontology-based knowledge models has been largely motivated by their ability to provide unique definitions for concepts, their relationships and properties, which together create a unified description of a given domain. However, the use of ontologies has been limited by the large engineering costs, which has stimulated increasing research on socially or automatically constructed knowledge-resources. In this section, we describe a hybrid document model, where Wikipedia is used in conjunction with a small ontology-based document model and the classical bag-of-words representation.
4.1Wikipedia-based content model
Given a preprocessed document, we start building a document model by first detecting the Wikipedia-concepts
In its current form, Wikifier makes no difference between general concepts and named-entities. However, as we acknowledge in Section 5, there is a considerable difference in the specificity of a concept which is a named-entity, e.g. “Goldman Sachs”, and a general concept, e.g. “Investment banking”. For instance, to say that a certain document discusses Goldman Sachs practically requires that the bank’s name is explicitly mentioned. But, to say that a document is about investment banking is considerably more relaxed; it is sufficient to find a collection of investment banking related concepts rather than the exact concept name to identify the document as relevant. Clearly, this should be taken into account when specifying the sensitivity of semantic classifiers to different concept types. Therefore, we train a named-entity recognizer to complement the Wikifier.
Finally, having obtained both the set of Wikipedia concepts and the set of named-entities, we construct the Wiki-content model as a combination of the general Wikipedia-concepts and Wikipedia named-entities.
4.2Ontology-based content model
The ontology model considered in this paper is derived from the Business Term Ontology (BTO) proposed by Malo and Siitari . The primary purpose of the BTO ontology is to provide the system with a solid taxonomy of business domain concepts, and allow explicit expression of generality vs. specificity of concepts through subclassing relation.
The BTO-ontology model is built using the RDFS extension proposed by Suchanek et al. , where an ontology is defined as an injective mapping from a finite set of fact-identifiers to fact-triplets. This definition allows a very general description of an ontology as a graph, where the nodes may be either entities (e.g. concepts such as OptionContract, PutOption, CallOption), relations (e.g. subClassOf, hasWikiPage) or fact-identifiers. The basic element in the BTO model is thus an entity which may refer to any abstract or concrete thing. Throughout, we also assume that the entities are discernible and we can tell whether two entities are the same.
Following the notations introduced in the previous section, we can now define the BTO-content model as a simple set-valued mapping:
The full document model is then obtained as a combination of the Wikipedia and ontology based content models, which are augmented with the classical Bag of Words (BOW) representation.
This choice of document model leads to a standard vector-space representation, where the presence/absence of a Wikipedia-concept, BTO-concept or a word is indicated by ones and zeros. Although, the above form could be easily changed to support some weighting scheme, such as Tf-Idf, we have left them as a question for further research. For the purpose of the current experiment our primary interest is in the benefits obtained from the use of Wikipedia-based relatedness measures in detection of relevant documents.
5Semantic filtering with Wiki-SR
To illustrate the notion of semantic filtering in Wiki-SR, let’s consider the sample topic statement (see Figure 3), where the goal is to filter documents reporting telemarketing abuses in U.S. Now, by reading the statement, one could come up with a boolean query to represent the topic; e.g. (“U.S.” “telemarketing” (“fraudulent” “legislation” “regulation”)
In order to clarify more closely what Wiki-SR rules are, how they are constructed, and evaluated in practice, the section is divided into the following parts. In the first part (Section Section 5.1), we discuss how Wikipedia can be used to compute semantic relatedness between any pair of concepts. In particular, we consider how the existing measures can be adapted for usage in Wiki-SR rules. In the second part (Section Section 5.2), we present the formal definition of Wiki-SR model and discuss how it is constructed using the topic statement and the set of example documents supplied by the user. Finally, we describe the steps involved in evaluation of the Wiki-SR rules to determine whether a particular document matches the rule or not.
5.1Measuring semantic relatedness
Although approaches to measuring conceptual relatedness based on corpora or WordNet have been around already quite long, the use of Wikipedia as a source of background knowledge is a relatively new idea. The first step in this direction was taken by Strube and Ponzetto , who proposed their WikiRelate-technique that modified existing measures to better work with Wikipedia. This was soon followed by the paper of Gabrilovich and Markovitch , who suggested explicit semantic analysis (ESA) to define a highly accurate similarity measure using the full text of all Wikipedia articles. The most recent proposal is, however, the Wikipedia Link-based Measure (WLM) proposed by Milne et al. , where only the internal link structure of Wikipedia is used to define relatedness. The approach is known to be computationally very cheap and has still achieved relatively high correlation with humans, which is why we have adopted it as a basis for the document-concept similarity measure used in this paper. Below, we describe how semantic relatedness information of Wikipedia-links can be incorporated into filtering rules.
Commonly, a semantic relatedness measure is defined between two concepts. However, from our application’s perspective it is perhaps more interesting to ask: How strongly is the given concept related to the document at hands? Or how likely is it for the given concept to appear in the document? The idea of Milne et al.  was to construct a low-cost measure for semantic relatedness using only the hyperlink structure of Wikipedia rather than its category hierarchy or text content. The relatedness measure essentially corresponds to the Normalized Google Distance inspired by Cilibrasi and Vitanyi :
Because is defined only for uniquely identified Wikipedia-concepts, we need to extend the definition slightly to allow relatedness calculation for any pair of n-grams. That is, many words which are recognized as redirects or anchors are not counted into the set of Wikipedia-concepts . Therefore, we consider the following extension of from to .
Finally, recalling that we wanted a measure between a document and a concept, we can now use the above extension to introduce the following simple definition for Wikipedia-based relatedness measure:
The use of maximum, rather than sum-based operator such as average, in d-rel is a deliberate choice. Since this relatedness measure is intended to be used in filtering rules, we do not want to allow sum-operations to mask the presence of those concepts in a document which are not related to its central story.
Having introduced the semantic relatedness measure, d-rel, we can now provide a more detailed explanation to semantic filtering rules. Following our earlier discussion in Section 3, we decompose the definition of a semantic rule into two parts (see Figure 2): (1) the rule-builder which is responsible for learning the underlying query expression; and (2) the rule-evaluator which uses Wikipedia’s concept-relatedness information to perform an implicit expansion of the query to account for strongly related concepts.
Let be the set of available ontology and Wikipedia concepts, and let denote the space of all possible boolean query expressions that can be formulated using the concepts in and the boolean operators AND , OR , and NOT .
Now, assuming that the user has provided a topic statement and a small training collection of relevant/irrelevant document examples , the rule builder is defined as a mapping from the user-inputs to the query space, i.e.
where can contain only those concepts which appear in the topic statement. That is, if is the set of concepts included in , then the concepts must be such that .
The builder mapping is implemented by using the genetic programming (GP) technique proposed by Malo et al.  that extends the Inductive Query By Example (IQBE) paradigm of Smith and Smith  and Chen et al. . There, the idea is to use the relevance information collected from the user as fitness cases to find the query expression that best separates relevant from irrelevant document examples. The learning process is driven by the evolutionary pressure that guarantees that only the fittest individuals among all potential query candidates survive. In this paper, we used F-score as the fitness function to find a reasonable balance between precision and recall.
The reason, why IQBE-based query builders seem to be rarely used, is perhaps best explained by the tendency of GP to produce overfitted queries. The risk of overfitting is high, in particular, when the training sets are small and when the number of concepts (or literals/terminals in GP) is large. For these reasons, we had restricted the concept set to the ones that are detected from the topic statement. Thus, the version of GP used in this paper is a special case of the more advanced algorithm proposed by Malo et al. , where it is shown that the query learning can be generalized also to more realistic cases where predefined topic statements are not available. For further details on the use of GP-learning, see Koza . To find more information on the ways how GP can be modified for learning Wikipedia-based queries, see the forthcoming paper by Malo et al. .
The rule-evaluator in Wiki-SR provides a matching subsystem for determining whether a given document matches the currently active semantic rule. Now, assuming that the user’s topic definition has been transformed by the rule-builder component to a query , the evaluator is specified as a binary-valued mapping,
which operates in two steps: (i) concept-evaluation step; and (ii) expression-evaluation step.
To outline the procedure, let’s suppose that the query expression has form , where and each could be replaced by any of the boolean operators. Then the evaluation steps can be defined as follows:
(i) Concept-evaluation step: The purpose of the concept-evaluation step is to determine whether the query concepts are present in the active document - either directly or indirectly. The task is accomplished by a specific concept-evaluator function that is applied in turn to every concept in the query.
The concept-evaluator’s decision rule is carried out in three parts: (a) First, it tries to look whether the query concepts are directly featured in the document. (b) If no match is found, the rule then searches for related concepts or words, which would strongly predict the presence of the given concepts. If the d-rel based sensitivity threshold is exceeded, then the rule decides that the given concept is present in the document. (c) Finally, if these two steps fail, it is concluded that the given concept is not present.
To formalize this idea, we define the concept evaluator as function ,
where , and is a threshold function controlling the acceptance sensitivity by relatedness criteria. The threshold for d-rel depends on the type of concept, i.e. whether it is a named-entity, general Wikipedia-article, or BTO-concept,
Each sensitivity threshold is chosen based on training data. The purpose of the distinction between named-entities and general concepts is to allow stricter thresholds for named-entities which have by default narrower definitions than general concepts.
(ii) Expression-evaluation step: In Wiki-SR framework, it is the concept-evaluator function which does most of the work. Once the variables in the query have been evaluated, the expression-evalution step amounts to replacing the original query variables with the values given by the concept-evaluator , i.e. we obtain that
Thus, the value of this final expression can then be used to decide whether the given document is relevant or not. For instance, in the case of the previous example we would find that the particular document is not relevant, because was 0.
The evaluation of Wiki-SR framework is based on Reuters RCV1 corpus
The system used in the experiment was implemented using Java software on top of the GATE  platform, which provides tools for standard document preprocessing tasks. The Wikipedia-based content model was built using the WikipediaMiner published by Milne et al. , which was suitably modified and integrated into our framework. The named-entity recognition task was carried out using a Conditional Random Field (CRF) classifier proposed by Finkel et al. . For other classification tasks, we used Weka  through Java-ML  package. The manual ontology editing was done in Protégé
In the experiment, the performance of Wiki-SR framework is compared against Support Vector Machines (SVM) and the decision-tree classifier C4.5. The primary performance measures used for comparison are F-score, precision, recall, and accuracy.
The comparison was carried out as follows. The system started with the given collection of 100 topics and a set of training documents for each topic, where the documents had been pre-assigned as relevant or irrelevant by assessors of TREC-11. The task was then to train the classifiers using the information in the training-samples and the initial topic-statements. Here, each topic was considered separately and no cross-topic learning was allowed.
The construction of Wiki-SR classifiers was implemented in two steps. First, in order to obtain the boolean query statements expressed in terms of Wikipedia and ontology concepts, the topic-statements were profiled and the obtained concepts were used to build query rules. Then, the sensitivity thresholds required by the relatedness-based acceptance criteria were optimised using the training samples.
In similar fashion, the benchmark classifiers were optimised using only the training data. However, none of the benchmark classifiers used information in the original topic-statements. As feature sets, three different document models were considered: a bag-of-words profile (tokens), a Wikipedia profile (Wiki), and a profile where Wikipedia concepts are augmented with ontology concepts (BTO Wiki).
Table ? reports performance measures for models Wiki-SR, LibSVM and C4.5. In order to take the varying quality of the different topics into account, the results are further divided into three subtables based on the ratio of positive and negative examples in the training sample, i.e. . Panel A gives results for all topics, Panel B for topics with low ratio (), and Panel C for topics with high ratio ().
First of all, a general comparison of the models suggests that the Wiki-SR heuristic achieves consistently better results than the benchmark algorithms in terms of F-score. See Figure 4 for F-score and Recall boxplots computed using all 100 topics. When searching for causes, it appears that the performance differences are largely explained by the recall levels. Whereas the differences in accuracies and precisions are relatively small, as observed from Table ?, the heuristic Wiki-SR achieves considerably better results in terms of recall. Interestingly, when considering a division of topics based on the proportion of irrelevant and relevant documents in the training sample, we find that the heuristic has faired considerably better than its benchmarks on highly unbalanced topics; see Panels B and C in Table ?. This observation is possibly explained by that a rule-based model such as Wiki-SR is less sensitive to the quality of the training sample than for example SVM-classifiers. It is also known that imbalance between positive and negative examples in training sample can have an adverse effect on traditional classifiers.
Finally, to investigate the effect of document model given to the benchmark classifiers, both SVM and C4.5 models were built using three alternative profiles with different levels of concept information. A quick comparison reveals that wikification slightly improves results for all topics as measured by F-Score. Especially, when more unbalanced topics are considered. However, the case of BTO concepts shows mixed evidence.
In this paper, we have presented a new document filtering framework, Wiki-SR, where Wikipedia’s extensive domain-knowledge is utilised to produce effective semantic classification rules. An empirical experiment based on Reuters RCV1 corpus and TREC-11 topic statements revealed that the use of semantic concept-relatedness information along with a suitable document model have a considerable combined effect on classification performance. The results suggest that although there are some benefits already in the use of a concept-based representation of document’s contents, the profile is not truly effective unless there is also knowledge about relationships between different concepts. For this purpose, the use of Wikipedia as a source of domain knowledge is ideal due to its incredibly dense link-structure and broad scope.
In the future work, we investigate how machine-learning can be used to complement our Wikipedia-based approach to determining document-concept relatedness. In particular, we assume that the techniques used in multi-task learning could prove to be very beneficial in this respect. As another direction for further development, we are examining how the boolean rules used in Wiki-SR can be better extracted automatically from text in natural language form. Especially, we are interested in considering techniques, where the rule structures can be learned without the use of explicit topic definitions. One of such directions is examined in the forthcoming paper Malo et al. , where a modified GP-algorithm is developed to learn Wikipedia-based queries using only sample documents supplied by the user.
The authors would like to thank Emil Aaltonen Foundation and Finnish Cultural Foundation for their support.
- Wikipedia-based Semantic Rules
- The notion Wikipedia-concept is used interchangeably with Wikipedia-article, because each article in Wikipedia represents a single topic/concept with a short title. In effect, this amounts to considering Wikipedia as a very large thesaurus consisting of the terms derived from the titles of all articles.
- AND (), OR ()
- Reuters corpus volume 1, http://about.reuters.comresearchandstandardscorpus
- TREC 2002 Filtering Track Collections, http://trec.nist.govdata
- O. Medelyan, D. Milne, C. Legg, and I. Witten, “Mining meaning from Wikipedia,” International Journal of Human-Computer Studies, vol. 67, pp. 716–754, 2009.
- D. Milne, “Computing semantic relatedness using Wikipedia link structure,” in Proceedings of the New Zealand Computer Science Research Student Conference, 2007.
- D. Milne and I. Witten, “Learning to link with Wikipedia,” in Proc. CIKM, 2008.
- E. Gabrilovich and S. Markovitch, “Overcoming the brittleness bottleneck using Wikipedia,” in Proc. National Conference on Artificial Intelligence, Boston, MA, 2006.
- ——, “Computing semantic relatedness using Wikipedia-based explicit semantic analysis,” in Proc. IJCAI-07, 2007.
- O. Medelyan, I. Witten, and D. Milne, “Topic indexing with Wikipedia,” in Proceedings of the AAAI 2008 Workshop on Wikipedia and Artificial Intelligence (WIKIAI 2008), 2008.
- O. Medelyan and D. Milne, “Augmenting domain-specific thesauri with knowledge from Wikipedia,” in Proceedings of the New Zealand Computer Science Research Student Conference, 2008.
- R. Mihalcea and A. Csomai, “Wikify!: linking documents to encyclopedic knowledge,” in Proc. CIKM, 2007, pp. 233–242.
- M. Strube and S. Ponzetto, “WikiRelate! Computing semantic relatedness using Wikipedia,” in Proceedings of the 21nd AAAI conference on artificial intelligence, 2006.
- A. Gregorowicz and M. Kramer, “Mining a large-scale term-concept network from Wikipedia,” Mitre Corporation, Tech. Rep., 2006.
- D. Milne, I. Witten, and D. Nichols, “A knowledge-based search engine powered by Wikipedia.” in Proceedings of the 16th ACM Conference on Information and Knowledge Management CIKM’07, 2007, pp. 445–454.
- Y. Li, R. Luk, E. Ho, and K. Chung, “Improving weak ad-hoc queries using Wikipedia as external corpus,” in Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.1em plus 0.5em minus 0.4emACM Press, Amsterdam, 2007, pp. 797–798.
- O. Egozi, E. Gabrilovich, and S. Markovitch, “Concept-based feature generation and selection for information retrieval,” in Proceedings of the 23rd AAAI Conference on Artificial Intelligence (AAAI-08), 2008.
- P. Wang, C. Domeniconi, and J. Hu, “Cross-domain Text Classification using Wikipedia,” IEEE Intelligent Informatics Bulletin, vol. 9, pp. 5–17, 2008.
- P. Wang and C. Domeniconi, “Building Semantic Kernels for Text Classification using Wikipedia,” in KDD’08, 2008.
- J. Finkel, T. Grenader, and C. Manning, “Incorporating non-local information into information extraction systems by Gibbs sampling,” in Proceedings of the 43nd Annual Meeting of the Association for Computational Linguistics (ACL), 2005, pp. 363–370.
- P. Malo and P. Siitari, “A Context-aware Approach to User Profiling with Interactive Preference Learning,” Aalto University working paper W-482, 2010.
- F. Suchanek, G. Kasneci, and G. Weikum, “YAGO: A Large Ontology from Wikipedia and WordNet,” Elsevier Journal of Web Semantics, 2008.
- R. Cilibrasi and P. Vitanyi, “The Google similarity distance,” IEEE Transactions on Knowledge and Data Engineering, vol. 19, pp. 370–383, 2007.
- P. Malo, P. Siitari, and A. Sinha, “Automated Query Learning with Wikipedia and Genetic Programming,” Unpublished manuscript, 2010.
- M. Smith and M. Smith, “The use of genetic programming to build boolean queries for text retrieval through relevance feedback,” Journal of Information Science, vol. 23, no. 6, pp. 423–431, 1997.
- H. Chen, G. Shankaranarayanan, L. She, and A. Iyer, “A machine learning approach to inductive query by example: An experiment using relevance feedback, ID3, genetic algorithms, and simulated annealing,” Journal of the American Society for Information Science, vol. 49, no. 8, pp. 693–705, 1998.
- J. Koza, Genetic programming: On the programming of computers by means of natural selection.1em plus 0.5em minus 0.4emMIT Press, 1992.
- K. Bontcheva, V. Tablan, D. Maynard, and H. Cunningham, “Evolving GATE to Meet New Challenges in Language Engineering,” Natural Language Engineering, vol. 10, pp. 349—373, 2004.
- D. Milne and I. Witten, “An open-source toolkit for mining Wikipedia,” Unpublished manuscript, 2009.
- M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. Witten, “The WEKA Data Mining Software: An Update,” SIGKDD Explorations, vol. 11, pp. 10–18, 2009.
- T. Abeel, Y. de Peer, and Y. Saeys, “Java-ML: A Machine Learning Library,” Journal of Machine Learning Research, vol. 10, pp. 931–934, 2009.