Empirical Analysis of Foundational Distinctions in the Web of Data

Empirical Analysis of Foundational Distinctions in the Web of Data

   Luigi Asprino, Valerio Basile, Paolo Ciancarini, Valentina Presutti
STLab, ISTC-CNR (Rome, Italy)
University of Bologna (Italy)
Sapienza University of Rome (Italy)
luigi.asprino@unibo.it, basile@di.uniroma1.it, paolo.ciancarini@unibo.it, valentina.presutti@cnr.it
Abstract

A main difference between pre-Web artificial intelligence and the current one is that the Web and its Semantic extension (i.e. Web of Data) contain open global-scale knowledge and make it available to potentially intelligent machines that may want to benefit from it. Nevertheless, most of the Web of Data lacks ontological distinctions and has a sparse distribution of axiomatisations. For example, foundational distinctions such as whether an entity is inherently a class or an individual, or whether it is a physical object or not, are hardly expressed in the data, although they have been largely studied and formalised by foundational ontologies (e.g. DOLCE, SUMO). There is a gap between these ontologies, that often formalise or are inspired by pre-existing philosophical theories and algebras and are developed with a top-down approach, and the Web of Data that is mostly derived from existing databases or from crowd-based effort (e.g. DBpedia, Wikidata, Freebase). We investigate whether the Web provides an empirical foundation for characterising entities of the Web of Data according to foundational distinctions. We want to answer questions such as “is the DBpedia entity for dog a class or an instance?”. We report on a set of experiments based on machine learning and crowdsourcing that show promising results.

Empirical Analysis of Foundational Distinctions in the Web of Data


Luigi Asprino, Valerio Basile, Paolo Ciancarini, Valentina Presutti STLab, ISTC-CNR (Rome, Italy) University of Bologna (Italy) Sapienza University of Rome (Italy) luigi.asprino@unibo.it, basile@di.uniroma1.it, paolo.ciancarini@unibo.it, valentina.presutti@cnr.it

1 Commonsense and the Web of Data

In 1989,  [?] (re-)assessed the need for Artificial Intelligence (AI) of a “formalization of a sizable portion of common-sense knowledge about the everyday physical world” (cit.), which he says must have three main characteristics: uniformity, density, and breadth. After almost three decades, the Semantic Web effort has partly addressed his expressed requirements with the Web of Data (WoD): 150 billion linked facts111http://stats.lod2.eu/, formally (and uniformly) represented in RDF and OWL, and openly available on the Web. Nevertheless, there is still a long way to go before it can be said that the WoD formalises commonsense knowledge and addresses also density (high ratio of facts for concepts) and breadth (large coverage of physical phenomena). In fact, it is very rich in domains such as geography, linguistics, life sciences and scholarly publications, as well as in cross-domain knowledge, but mainly from an encyclopedic perspective. Our ultimate goal is to contribute to enrich the WoD with commonsense knowledge, going beyond the encyclopedic view. We have identified a first gap to be filled towards this goal: assessing foundational distinctions over the WoD. In this paper, we focus on two of them: whether an entity e.g. dbr:Rome222the prefix dbr: stands for http://dbpedia.org/resource/, (i) is inherently a class or an instance and (ii) whether a class is a kind of physical object or not. We think they can potentially boost a virtuous process towards richer axiomatisations of commonsense knowledge. For example, only physical objects can move in space or be the subject of axioms expressing their expected (naive) physical behavior (e.g. gravity). Similarly, learned facts on a certain entity have different reasoning consequences if the entity is to be treated as a class or an instance. These foundational distinctions have been theorised and modelled in foundational ontologies such as DOLCE [?] and SUMO [?] in a top-down approach, but populating and empirically validating them has been rarely addressed. We refer to the definition of Physical Object provided by DOLCE+DnS UltraLite (DUL) ontology333http://www.ontologydesignpatterns.org/ont/dul/DUL.owl: “Any Object that has a proper space region. The prototypical physical object has also an associated mass, but the nature of its mass can greatly vary based on the epistemological status of the object (scientifically measured, subjectively possible, imaginary)”. To the best of our knowledge, the most relevant works focused on foundational distinctions include: [?], which provide manual annotations of WordNet synsets444A WordNet synset is an equivalence class of word senses. according to the instance/class distinction, and  [?], which produced a dataset of DBpedia entities that are typed with DOLCE classes using an ontology learning approach. We are interested in exploring a different strategy, which may provide insights on the validity of the existing annotations and, at the same time, lead us to identify an automatic and scalable way of making such distinctions emerge empirically. In particular, we aim at answering the question whether the (Semantic) Web shows evidence for supporting certain foundational distinctions over WoD entities. We present and discuss a set of experiments, conducted on a sample of the WoD, involving manual inspection, ontology alignment, machine learning, and crowdsourcing. The experiments show promising results, and motivate us to extend this study to a much larger scale, in line with a recent inspiring talk“The Empirical Turn in Knowledge Representation”555https://goo.gl/BDSGY1 by van Harmelen, who suggests that the WoD is a unique opportunity to “observe how knowledge representations behave at very large scale”. After giving a brief account of relevant related literature in Section 2, we describe in Section 3 the methods developed and used in this study. Section 4 describes three reference datasets that we have build for evaluating and comparing the obtained results, presented in Section 5. Section 6 wraps up and sketches future development.

2 Related Work

Commonsense knowledge is ”knowledge about the everyday world that is possessed by all people”[?]. The availability of this knowledge plays a key role for enabling an intelligent behavior of machines.

Over the years, a number of commonsense knowledge bases have been proposed for supporting diverse tasks spanning from automated reasoning to natural language processing. DBpedia666http://dbpedia.org [?] is a very popular one as it is extracted from Wikipedia infoboxes. It is the de facto main hub of the Web of Data (WoD), with its 4.58 million entities. Most DBpedia knowledge is of encyclopedic nature. Nevertheless, due to its sheer size, DBpedia also contains a great deal of information on common concepts such as objects and locations, and was therefore chosen as the main data set for our experiments (cf. Section 5). ConceptNet777http://conceptnet5.media.mit.edu/ [?] is a large-scale semantic network that integrates commonsense facts from a number of existing resources. The knowledge it encodes is very diverse and with a sparse coverage distribution, which makes it difficult to isolate information related to foundational distinctions. Only a small portion is linked to the WoD, making it unsuitable for reuse in our current study. Another valuable resource is OpenCyC888http://www.opencyc.org/  [?], an ontology with an associated knowledge base of commonsense knowledge organised in modular theories, released as part of the long-standing CyC project. OpenCyc uses a proprietary representation language and there is a plan to make it available to the community as a linked data project999The authors were unable to get the resource so far but plan to investigate its reuse in the future.. YAGO [?] is a large knowledge base that combines data from Wikipedia, WordNet [?] and GeoNames101010http://www.geonames.org/. SUMO [?] and DOLCE [?] are two examples of foundational ontologies: they capture general concepts and their relations, such as event, time, object, participation, etc. as reference theories for a wide range of applications. Tipalo [?] provides typing axioms for DBpedia entities, based on DOLCE types (including Physical Object). Lexico-semantic resources such as WordNet [?] encode information about what can be expressed with language. WordNet organises English words according to their senses, in a semantic network of relations such as hypernymy, meronymy, antonymy. In [?], the authors have manually inspected a large number of WordNet synsets, and annotated them according to the instance/class distinction. The resulting annotations are indeed very useful for assessing the same distinction on DBpedia entities, by means of alignments (as e.g. provided by BabelNet [?]). We reuse most of these resources and studies in our experiments.

3 Automatic classification of foundational distinctions

We are interested in observing whether the Web of Data (WoD) and its related Web resources provide an empirical basis for making foundational distinctions over entities represented in the WoD. As anticipated in Section 1, we focus on two distinctions: between class and instance, and between physical and non-physical object. Considering that DBpedia is at the core of WoD, and most of the other datasets link to it, we use DBpedia in our study. DBpedia has an unclear distinction between classes and instances, and contains both of them in large number.

We approach this problem as a classification task, using two classification approaches: alignment-based (cf. Section 3.1) and machine learning-based (cf. Section 3.2).

3.1 Alignment-based classification

Alignment-based methods exploit the linking structure of WoD, in particular the alignments between DBpedia, foundational ontologies such as DUL, and lexical linked data (i.e. WoD datasets that encode lexical/linguistic knowledge). Examples of lexical linked data are BabelNet [?] and OntoWordNet [?]. The advantage of these methods is their inherent unsupervised nature. Their main disadvantages are the need of studying the data models for designing suitable queries, and the potential limited coverage and errors that may accompany the alignments. We have developed SENECA (Selecting Entities Exploiting Linguistic Alignments) which relies on existing alignments in WoD to make an automatic assessment on the defined foundational distinctions over DBpedia entities. A graphical description of SENECA is depicted in Figure 1.

Class vs instance. As far as this distinction is concerned, SENECA works based on the hypothesis that common nouns are mainly classes and they are expected to be found in dictionaries, while it is less the case for proper nouns, that denote mainly instances. The same hypothesis was taken by [?], who manually annotated instances in WordNet, information that SENECA reuses when available. A good quality alignment between some lexical resources and DBpedia is provided by BabelNet [?]. SENECA exploits these alignments and selects all the DBpedia entities that are linked to an entity in WordNet111111http://wordnet-rdf.princeton.edu/, Wiktionary121212https://www.wiktionary.org/ or OmegaWiki131313http://www.omegawiki.org/. With this approach 63,620 candidate classes have been identified, as opposed to WordNet annotations that only provide 38,701 classes. In order to further increase the potential coverage, SENECA leverages typing axioms of Tipalo [?] broadening it to 431,254 total candidate classes. All the other DBpedia entities are assumed to be candidate instances. SENECA criteria for selecting candidate classes among DBpedia entities is depicted in Figure 0(a).

(a) The alignment paths followed by SENECA for selecting candidate classes among DBpedia entities. It identifies as classes all DBpedia entities aligned via BabelNet to a WordNet synset, an OmegaWiki synset or a Wiktionary page, and all DBpedia entities typed as owl:Class in Tìpalo.
(b) The alignment paths used by SENECA for identifying candidate Physical Object subclasses among DBpedia entities. It navigates the YAGO taxonomy that via OntoWordNet links DBpedia entities to dul:PhysicalObject or Tìpalo that links DBpedia entities to dul:PhysicalObject.
Figure 1: SENECA approach for assessing whether a DBpedia entity is a class or an instance (Figure 0(a)) and whether it is a physical object or not (Figure 0(b)).

Physical object vs non-Physical object. It is worth noticing that almost 600,000 DBpedia entities are only typed as owl:Thing or not typed at all. However, every DBpedia entity belongs to at least one Wikipedia category. Wikipedia categories have been formalised as a taxonomy of classes (i.e. by means of rdfs:subClassOf) and aligned to WordNet synsets in YAGO [?]. WordNet synsets are in turn formalised as an OWL ontology in OntoWordNet [?]. OntoWordNet is based on DUL, hence it is possible to navigate the taxonomy up to the DUL class for Physical Object. SENECA looks up the Wikipedia category of a DBpedia entity and follows these alignments. Additionally, it uses Tipalo, which includes type axioms of DBpedia entities based on DUL classes. SENECA uses these paths of alignments and taxonomical relations, as well as the automatic inferences that they enable to assess whether a DBpedia entity is a Physical Object or not. With this approach, graphically summarised in Figure 0(b), 67,005 entities were selected as candidate physical object subclasses.

3.2 Machine learning-based classification

Within machine-learning, classification is the problem of predicting which category an entity belongs to, given a set of examples (i.e. a training set). The training set is processed by the algorithm in order to learn a predictive model based on the observation of a number of features, which can be categorical, ordinal, integer-valued or real-valued. We have designed our task as a hierarchy of binary classifications: at the top level there is the classification between classes and instances. Among classes, we are interested in distinguishing those that denote physical objects from the others. We experimented with eight binary classification algorithms: J48, Random Forest, REPTree, Naive Bayes, Multinomial Naive Bayes, Support Vector Machines, Logistic Regression, and K-nearest neighbors classifier. We used WEKA141414https://www.cs.waikato.ac.nz/ml/weka/ for the implementation of these algorithms.

Features. The classifiers were trained using four different features. {itemize*}

Abstract. We retrieve the abstract text associated with DBpedia entities and represent it as a 0-1 vector (i.e a Bag of Words) using the WEKA’s String to Word filter, with the default setting. This filter performs a naive tokenisation and builds a dictionary containing the 1000 most frequent tokens found in all the abstracts of dataset. The dictionary is case-sensitive since the filter does not normalise tokens. The resulting vector has a value 1 for each token mentioned in the abstract and 0 for the others. Intuitively, the words used in an entity’s abstract determine how to classify the entity. For example, most of classes definitions begin with “A” (“A knife is a tool…”).

Incoming and outgoing properties. An outgoing property of a DBpedia entity is the property of a triple having the entity as subject. On the contrary, an incoming property is the property of a triple having the entity as object. For example, considering the triple dbr:Rome :locatedIn dbr:Italy, the property :locatedIn is an outgoing property for dbr:Rome and and incoming property for dbr:Italy. For each DBpedia entity, we count and keep trace of its different incoming and outgoing properties. For example, properties such as dbo:birthPlace or dbo:birthDate are common outgoing properties of an individual person, hence suggesting that the entity is an individual.

URI. The most significant part of a URI (i.e. the rdf:ID) may be discriminating for the class vs. instance classification. The rdf:ID in DBpedia is the name of the entity (it is a common practice in order to make the URI more human-readable), and it always starts with an upper case letter. If the entity’s name is a compound term and the entity denotes an instance, each of its components starts with a capital letter. We noticed that DBpedia entity names are always mentioned at the beginning of their abstract and, for most of the instance entities, they have the same capitalisation pattern as the rdf:ID. Therefore, the entity’s URI has been tokenised and turned into a 0-1 vector using the WEKA’s String to Word filter and concatenated with other feature vectors.

Outcome of SENECA. The output of SENECA is used as a binomial feature (taking value “yes” or “no”) for the classifiers. This feature cannot be used with Multinomial Naive Bayes classifier.

The classifiers were trained and tested using several combinations of features, and by performing feature selection for each combination. Section 5 provides additional details together with the results of the experiments.

4 Reference datasets for classification experiments

In order to experiment with, and evaluate the results of the methods described in Section 3, we have built three reference datasets named Experts, Crowd and PO, that are available online151515https://github.com/fdistinctions/ijcai18. These datasets are samples extracted from DBpedia, a reference hub of the Web of Data. We also leverage NASARI[?], a resource of vector representations for BabelNet synsets and Wikipedia entities. Experts dataset: Class vs. Instance. In order to build a dataset with a balanced number of representatives for classes and instances, we manually selected 20 classes and 20 instances from DBpedia, and used them as seeds. For each entity, we retrieved its nearest neighbor entities in NASARI[?], using cosine similarity as a distance measure, following the intuition that we would collect similar type of entities (in terms of instances and classes). We limited the number of neighbors to 100. A random selection of resources in DBpedia would have caused a strong unbalance of the sample towards instances.

NASARI is based on a previous version of Wikipedia, hence we performed an alignment between the resulting entities and the current version of Wikipedia, discarding no longer available entities, entities without abstract, disambiguation pages, and redirects to entities already included in the dataset. Since physical locations are an ideal source of physical object types, the dataset was further enriched with DBpedia entities selected with two heuristics: {enumerate*}

The SUN database161616https://groups.csail.mit.edu/vision/SUN/is a computer vision oriented dataset containing a collection of annotated images, covering a large variety of environmental scenes, places and the objects within. We retrieved all DBpedia entities whose labels match a SUN location, and added them to our dataset.

all DBpedia entities having category dbc:Places, or one of its narrower categories, have been added to our dataset. A total number of 4502 entities were collected in the Experts dataset: a manual inspection showed an even distribution of instance and class representatives. Two authors of the paper manually annotated these entities by indicating whether they were instances or classes, using the DBpedia abstract as the entity reference description. The annotators gave their judgments separately, and they only disagreed on 286 entities (agreement: 93,6%). From a joint second inspection they agreed on additional 281 entities that were initially misclassified by one of the two. Examples of misclassified entities are: dbr:Select_Comfort (an U.S. manufacturer) that was annotated as class, but it is an instance; dbr:Catawba_Valley_Pottery (a kind of Pottery) annotated as instance instead of class. From the remaining entities, 5 are synecdoches, cases for which the entity and its description point to both type of referents. For example, the trademark Coke is often used to refer to any variety of cola. We decided to annotate these entities as classes. The resulting Experts annotated dataset contains 1983 classes and 2519 instances. Crowd dataset: Class vs. Instance. We also crowdsourced the annotation of the “Experts” dataset. The motivation was to validate the authors annotation, and to obtain a non-expert representation of judgements. Each worker of the crowd was asked to indicate whether an entity is a class or an instance by using the name of the entity, its abstract, and a link to the corresponding Wikipedia article. The task was executed on CrowdFlower171717https://crowdflower.com/ by English speakers with high trustworthiness. The quality of the contributors has been assessed with 51 test questions with a tolerance of only 20% of errors. We collected 22510 judgments from 117 contributors: each entity was annotated by at least 5 different workers. For each entity we computed the level of agreement on each class weighted by the trustworthiness scores

(1)

where is the sum of the trustworthiness scores of the workers that annotated entity with class ; and is the sum of the trustworthiness scores of all the workers that annotated the entity . Table 1 reports the results of the task indicating the size of the classes per level of agreement. The average agreement of the crowd’s annotations is 95.76% .

Agreement # Class # Instance Total

0.5
1934 2568 4502
0.6 1884 2495 4379
0.8 1631 2330 3961

Table 1: Class/Instance “Crowd” dataset. The table provide an insight of the dataset per level of agreement.

The experiments presented in Section 5 were performed on the entities annotated with an agreement score 80%. We compared the crowd’s annotations (with agreement greater 0.5) with experts’. The judgements of the crowd workers diverge from the experts’ only on 193 entities (agreement is 95,7%). Some of those entities (61) also caused a disagreement between experts, hence denoting ambiguous cases. Examples include synecdoches such as dbr:Zeke_the_Wonder_Dog or music genders (e.g. dbr:Ragga).

PO dataset: Physical Objects vs. Non Physical Objects. We also needed annotations that accounted for the distinction between physical object and non-physical object classes. We extracted all entities annotated as classes (1983) from the “Experts” dataset, and gave them to the crowd. The workers were asked to indicate whether the entity denotes physical objects or not, using the entity name, the abstract and Wikipedia page, for supporting their decision. The quality of the workers has been assessed with 21 test questions, used to exclude contributors that scored an accuracy lower than 70%181818Notice that we follow quality strategies recommended as best practices by the crowdsourcing platform team. We collected 7181 judgments from 99 workers. Each entity has been annotated by at least 3 different workers. The level of agreement for each entity and for each class is reported in Table 2. The average agreement of the crowd’s annotations is 92.52% . The experiments presented in Section 5 were performed on the entities annotated with an agreement score 80%.

Agreement # Physical Object # Non Physical Object Total

0.5
1434 549 1983
0.6 1385 502 1887
0.8 1211 416 1627

Table 2: PO dataset: crowd-based annotated dataset of physical objects. The table provides an insight of the dataset per level of agreement. Agreement values computed according to Formula  1

5 Experiments, Results and Discussion

We provide the results of our experiments in terms of precision, recall and F1 measure calculated separately on each classification and on each target class (class vs. instance and physical object vs. non physical object). The average F1 score is also provided. We compare the results of the different methods with the reference datasets Experts, Crowd and PO, described in Section 4 and built for this purpose.

5.1 Alignment-based methods: SENECA

Class vs. Instance. Table 3 compares SENECA’s results with the two reference datasets Experts and Crowd. SENECA shows very good performance with best avg F1 = .836 then compare with the Crowd dataset. Considering that SENECA is unsupervised, and is based on existing alignments in the WoD, this result suggests that the WoD better reflects commonsense of the generic user than of expert’s.

Dataset PC RC F1C PI RI F1I Avg F1
Experts .919 .693 .796 .753 .939 .836 .813
Crowd .935 .731 .818 .778 .945 .853 .836
Table 3: Results of SENECA on the Class vs.Instance classification compared against the reference datasets described in Section 4. P*, R* and F1* indicate precision, recall and F1 measure on Class (C) and Instance (I). Avg F1 is the average of the F1 measures.

Physical Object vs. Non-Physical Object. Table 4 shows the performance of SENECA on the physical object vs. non physical object classification task computed by comparing its results with the PO dataset (cf. Section 4). We observe a significant drop in the best average F1 score (.623) as compared to the class vs. instance classification task (.836). On one hand, this may suggest that the task is harder. On the other hand, the alignment paths followed in the two cases are different, since for classifying Physical Objects more alignment steps are required. In the first case, BabelNet directly provides the final alignment step (cf. Figure 0(a)), while in the second case, three more alignment steps are required: DBpedia Category YAGO WordNet (cf. Figure 0(b)). It is reasonable to think that this implies a higher rate of error propagation along the flow.

Method PPO RPO F1PO PNPO RNPO F1NPO Avg F1
SENECA .838 .376 .519 .597 .927 .727 .623
Table 4: Results of SENECA on Physical Object vs. Non Physical Object classification compared against the PO reference dataset described in Section 4. PPO, RPO and F1PO indicate precision, recall and F1 measure on the class Physical Object. PNPO, RNPO and F1NPO indicate precision, recall and F1 measure on the class Non Physical Object. Avg F1 is the average of F1 measure for the two classes.

5.2 Machine learning methods

In addition to evaluating the performance of SENECA, we carried out a number of experiments with eight classifiers: J48, Random Forest, REPTree, Naive Bayes, Multinomial Naive Bayes, Support Vector Machines, Logistic Regression, and K-nearest neighbors (cf. Section 3.2). We used a 10-fold cross validation strategy on the reference datasets. As for the Crowd dataset we only included the annotated entities with an agreement score 80%. Before training the classifiers, the datasets were adjusted in order to balance the dimension of the two sample classes. The Expert dataset was balanced by randomly removing a selection of annotations. The Crowd dataset was balanced by removing a number of annotations with lower agreement. With this approach we aimed at removing less strong examples for the classifiers. Each classifier was trained and tested with all four features (cf. Section 3.2) individually, and in all possible combinations, with and without performing feature selection. The two datasets allow for different configurations of the training set. As for the Experts dataset each sample can have a 1 or 0 value, representing that, according to the experts, the entity belongs to a certain class or not, respectively. The Crowd dataset provides richer information: each annotated sample has an associated agreement score agreement(e, c), computed according to Formula 1 (cf. Section 4). We used agreement(e, c) as a feature to weight each sample in the training set.

Due to space limit, we report the results of the best performing algorithms only (i.e. Support Vector Machine for class vs. instance, and Logistic Regression for Physical Object vs. Non-Physical Object) and show their performance with all combinations of features and best performing setting.

Class vs. Instance Table 5 shows the results of Support Vector Machine, trained and tested on the two reference datasets. The best performance is observed against the Crowd reference dataset by combining all features. In particular, the best precision and F1 on Class (C), is achieved by combining all features, while the best recall is achieved by only using Pr (i.e. incoming and outgoing properties). In this experiment we found that performing feature selection would make results worse. When the Experts dataset is used there is a slight drop in performance, although the quality of the classification remains high. This result suggests that the agreement-based weighing is an informative feature for the model.

A Uri Pr D PC RC F1C PI RI F1I Avg F1

Dataset: Experts

.939 .937 .938 .937 .939 .938 .938
.938 .935 .936 .935 .938 .937 .936
.944 .952 .948 .951 .944 .947 .948
.944 .942 .943 .942 .945 .943 .943
.945 .944 .944 .944 .945 .944 .944
.941 .945 .943 .945 .949 .943 .943
.943 .936 .939 .936 .943 .942 .939
.856 .972 .917 .967 .837 .897 .904
.867 .978 .919 .975 .857 .908 .914
.945 .955 .955 .955 .945 .950 .953


Dataset: Crowd

.962 .966 .964 .966 .961 .964 .964
.962 .966 .964 .966 .961 .964 .964
.979 .979 .975 .979 .977 .975 .975
.971 .982 .977 .982 .971 .976 .976
.972 .981 .977 .981 .972 .977 .977
.966 976 .968 .975 .966 .968 .968
.963 974 .965 .971 .960 .965 .965
.888 .983 .933 .981 .876 .925 .929
.885 .988 .934 .986 .872 .925 .929
.976 .983 .980 .983 .976 .979 .979


Table 5: Results of the Support Vector Machine classifier on Class vs. Instance classification task against the reference datasets described in Section 4. The first four columns indicate the features used by the classifier: A is the abstract, Uri is the URI, Pr are incoming and outgoing properties, D are the results of the alignment-based methods. P, R, F1 indicate precision, recall and F1 measure on Class (C) and Instance (I).

Physical Object vs. Non Physical Object. Table 6 shows the results of the Logistic Regression algorithm trained on the PO reference dataset. In line with the performance of SENECA, also with a statistical approach we observe a decline in the overall performance with respect to the Class vs. Instance classification. We also observe a different pattern in the impact of the single features. In fact, the best combination is the abstract text and the SENECA selection. Considering that SENECA is based also on the abstract text parsing, through Tìpalo, this may suggest that the accompanying text of entities in this case provide a more effective source of distinction as compared to incoming and outgoing properties. However, Pr also in this case shows a positive impact on most combinations.

A Uri Pr D PPO RPO F1PO PNPO RNPO F1NPO Avg F1


.876 .869 .872 .871 .877 .873 .873
.875 .838 .856 .844 .882 .862 .859
.894 867 .875 .865 .894 .879 .877
.872 .877 .874 .876 .871 .874 .874
.899 .869 .883 .873 .902 .887 .885
.917 .863 .889 .871 .922 .895 .892
900 .852 .875 .859 .905 .881 .878
.647 .676 .662 .661 .632 .646 .654
.702 .643 .671 .679 727 .697 .684
.863 .857 .859 .858 .867 .859 .859


Table 6: Results of the Logistic Regression classifier on Physical Object vs.Non Physical Object classification using the PO reference dataset described in Section 4. In this case Feature Selection showed to improve performance, hence it was executed before training the classifier. The first four columns indicate the features used by the classifier: A is the abstract, Uri is the URI, Pr are incoming and outgoing properties, D are the results of the alignment-based methods. P, R, F1 indicate precision, recall and F1 measure on the classes Physical Object (PO) and Non Physical Object (NPO).

5.3 Remarks and discussion

The experiments performed show promising results on the classification tasks under study. Statistical methods perform better than alignment-based methods. It is worth noting that DBpedia entities are accompanied by textual descriptions that follow a standardised style of writing. This may have an impact on the role of text-based features in the training set. In fact, one may think that the ability of text resources to support foundational distinctions may not be as effective if we extend the scope of the automatic classification to WoD datasets beyond DBpedia. Nevertheless, the observed behavior of features, especially for the class vs instance classification show that this may not be the case. In particular, we remark that Pr taken alone as a feature for Support Vector Machine with the Crowd reference dataset, shows an average F1 of .933, which is very high, and it positively contributes to the performance of the other combinations. We also remark that incoming and outgoing properties include links to and from datasets outside DBpedia, hence they are independent from infobox data, and in turn from the standardised content of Wikipedia. This raises the expectation on the capacity of the WoD to provide an empirical basis to make foundational distinctions emerge from data. Conversely, this claim looks less strongly supported for identifying physical object classes, although we can observe that this same feature positively contributes to increase the recall for physical object classes, in the best performing figure for recall (PO), and it shows its own distinctive capacity with a average F1 = .664. A more sophisticated use of assertions and axioms from the WoD can also be devised, this to say that there is potential room for improvement.

6 Conclusion and next development

This study reports on a number of experiments performed by adopting two different approaches, namely alignment-based methods and machine learning methods, for automatically assessing foundational distinctions over entities of the WoD. The ultimate goal of our research is to enrich the WoD with commonsense knowledge, and we think that asserting such foundational distinctions is a potential booster in that direction. With this study we aimed at investigating whether it is possible to use the WoD as an empirical basis for making such distinction emerge. The results of our experiments suggests a positive answer and motivates us to go a step further and broaden the experimental scope to a much larger scale, e.g. by using LOD Laundromat [?] and to investigate additional foundational distinctions and axiomatisations.

References

  • [Beek et al., 2016] Wouter Beek, Laurens Rietveld, Stefan Schlobach, and Frank van Harmelen. LOD laundromat: Why the semantic web needs centralization (even if we don’t like it). IEEE Internet Computing, 20(2):78–81, 2016. DOI:10.1109/MIC.2016.43.
  • [Bizer et al., 2009] Christian Bizer, Jens Lehmann, Georgi Kobilarov, Sören Auer, Christian Becker, Richard Cyganiak, and Sebastian Hellmann. DBpedia - A crystallization point for the Web of Data. International Journal of Web Semantics, 7(3):154–165, 2009. DOI:10.1016/j.websem.2009.07.002.
  • [Camacho-Collados et al., 2016] José Camacho-Collados, Mohammad Taher Pilehvar, and Roberto Navigli. Nasari: Integrating explicit knowledge and corpus statistics for a multilingual representation of concepts and entities. Artificial Intelligence, 240:36 – 64, 2016.
  • [Fellbaum, 1998] Christiane Fellbaum, editor. WordNet: an electronic lexical database. MIT Press, 1998.
  • [Gangemi et al., 2002] Aldo Gangemi, Nicola Guarino, Claudio Masolo, Alessandro Oltramari, and Luc Schneider. Sweetening ontologies with dolce. In Asunción Gómez-Pérez and V. Richard Benjamins, editors, Knowledge Engineering and Knowledge Management: Ontologies and the Semantic Web, pages 166–181, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg.
  • [Gangemi et al., 2003] Aldo Gangemi, Roberto Navigli, and Paola Velardi. The OntoWordNet Project: Extension and Axiomatization of Conceptual Relations in WordNet. In On The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE - OTM Confederated International Conferences, CoopIS, DOA, and ODBASE, Proceedings, volume 2888 of Lecture Notes in Computer Science, pages 3–7. Springer, 2003.
  • [Hayes, 1989] Patrick Hayes. The second naive physics manifesto. In Daniel S. Weld and Johan de Kleer, editors, Readings in qualitative reasoning about physical systems. Morgan Kaufmann, San Francisco, CA, USA, 1989.
  • [Lenat, 1995] Douglas B. Lenat. CYC: A large-scale investment in knowledge infrastructure. Communications of the ACM, 38(11):32–38, 1995.
  • [Liu and Singh, 2004] H. Liu and P. Singh. ConceptNet –a practical commonsense reasoning tool-kit. BT Technology Journal, 22(4):211–226, October 2004.
  • [Miller and Hristea, 2006] George A. Miller and Florentina Hristea. Wordnet nouns: Classes and instances. Computational Linguistics, 32(1):1–3, 2006. DOI:10.1162/coli.2006.32.1.1.
  • [Navigli and Ponzetto, 2012] Roberto Navigli and Simone Paolo Ponzetto. BabelNet: The Automatic Construction, Evaluation and Application of a Wide-Coverage Multilingual Semantic Network. Artificial Intelligence, 193:217–250, 2012.
  • [Nuzzolese. et al., 2012] Andrea G. Nuzzolese., Aldo Gangemi, Valentina Presutti, Paolo Ciancarini, and Alberto Musetti. Automatic Typing of DBpedia Entities. In Proceedings of the 11th International Semantic Web Conference (ISWC), Part I, volume 7649 of Lecture Notes in Computer Science, pages 68–81. Springer, 2012.
  • [Pease and Niles, 2002] Adam Pease and Ian Niles. IEEE standard upper ontology: a progress report. Knowledge Eng. Review, 17(1):65–70, 2002. DOI:10.1017/S0269888902000395.
  • [Suchanek et al., 2007] Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. Yago: A Core of Semantic Knowledge. In Proceedings of the 16th International Conference on World Wide Web (WWW), pages 697–706. ACM, 2007.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
133354
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description