A Commonsense Reasoning Framework for Explanatory Emotion Attribution, Generation and Re-classification
In this work we present an explainable system for emotion attribution and recommendation (called DEGARI and available here: http://di.unito.it/DEGARI) relying on a recently introduced commonsense reasoning framework (the logic) which is based on a human-like procedure for the automatic generation of novel concepts in a Description Logics knowledge base. Starting from an ontological formalization of emotions based on the Plutchik’s model (known as ArsEmotica and available at http://188.8.131.52/fuseki/ArsEmotica-core), the system exploits the logic to automatically generate novel commonsense semantic representations of compound emotions (e.g. Love as derived from the combination of Joy and Trust according to Plutchik’s model). The generated emotions corresponds to prototypes, i.e. commonsense representations of given concepts, and have been used to reclassify emotion-related contents in a variety of artistic domains, ranging from art datasets to the editorial content available in RaiPlay, the online multimedia platform of RAI Radiotelevisione Italiana (the Italian public broadcasting company). We have tested our system (i) by reclassifying the available contents in the tested dataset with respect to the new generated compound emotions (ii) with an evaluation, in the form of a controlled user study experiment, of the feasibility of using the obtained reclassifications as recommended emotional content. The obtained results are encouraging and pave the way to many possible further improvements and research directions.
keywords:Explainable AI, Commonsense reasoning, Knowledge Generation, Concept Combination
1 Introduction and Background
Emotions have been acknowledged as a key part of the aesthetic experience through all ages and cultures, as witnessed by terms such as “sublime” Russell (1964) and “pathos” Ross (1959), associated with the experience of art since the ancient times. The advent of computational tools and methods for investigating the way we respond to objects and situations has paved the way to a deeper understanding of the intricate relationship between emotions and artistic content. For example, Van Dongen et al. (2016) have studied how art affects emotional regulation by measuring the brain response through EEG: their research shows that, in comparison with photographs depicting real events, artworks determine stronger electro-physiological responses; in parallel, Leder et al. (2014) argue that the emotional response to art – measured through facial muscle movements – is attenuated in art critics, and stronger in non-expert, thus showing the universality and spontaneity of this response.
The association between art and emotions is even stronger when the artistic expression is conveyed by media, as in music and movies. For example, music has proven to be an effective tool for emotion regulation: as demonstrated by Thoma et al. (2012), music can induce specific emotional states in everyday situations, an effect which is sought for by the users and can be exploited to create effective affective recommender systems Andjelkovic et al. (2019). Finally, emotional engagement is of primary importance in narrative media, such as film and television, as extensively investigated by a line of research which draws from both film studies and emotion theories Smith (2003); Tan (2013).
As a consequence of the multi-faceted, complex role played by emotions in the experience of art and media, the investigation of this phenomenon with computational tools has relied on a variety of models and methodologies, ranging from dimensional models, better suited to investigate physiological, continuous correlate of emotions Russell (1980); Watson et al. (1988); Mehrabian (1996), to categorical models, which lend themselves to inspecting the conscious level of emotional experience Plutchik (1980); Ekman (1999); Bänziger and Scherer (2010). Dimensional models typically measure the emotional engagement along the arousal and hedonic axes, and are useful to study how the emotional response evolves over time. For example, Lopes et al. (2017) rely on crowdsourced annotations of tension, arousal and variance in audio pieces to realize sound-based affective interaction in games. Categorical models are useful to collect the audience experience as discrete emotional labels, and are easily mapped to textual descriptions of emotions across languages. As exemplified by Mohammad and Kiritchenko (2018), discrete emotional labels, merged from different categorical models (from Plutchik (1980) to Noy and Noy-Sharav (2013)), can shed light on the reception of art, letting correlations emerge between attributed emotions, liking and subjects.
In many cases, the emotional response of the audience is conveyed through language, non only in textual media, but also in relation to art and other media (consider, for example, tags and social media comments concerning artworks and exhibitions). Automatically detecting affective states and emotions from text has gained considerable attention over recent years, leading to the development of several resources - such as annotated corpora, ontologies and lexicons within the Computational Linguistics community Jurafsky and Martin (2019); Nissim and Patti (2017); Wang et al. (2020). Affective information expressed in texts is multi-faceted, and the wide variety of affective linguistic resources developed in the last years community, mainly for English, but also for other languages, basically reflects such richness. When we speak about affective states in the context of natural language communications, we mean to refer to several aspects, which vary in their degree of stability, such as: emotion, sentiment, personality, mood, attitudes or interpersonal stance. Given the wide variety of affective states, in recent years research has focused on a finer-grained investigation of the role of emotions, as well as on the importance of other affect dimensions such as emotion intensity Mohammad (2018) or activation. Depending on the specific research goals addressed, one could be interested in issuing a discrete label describing the affective state expressed (frustration, anger, joy, etc.) in accordance to different contexts of interaction and tasks. Both basic emotion theories, in the Plutchik-Ekman Ekman (1999) tradition, and dimensional models of emotions, provided a precious theoretical grounding for the development of lexical resources Strapparava and Mihalcea (2007); Mohammad and Turney (2013); Mohammad (2018); Cambria et al. (2020); Strapparava and Valitutti (2004) and computational models for emotion extraction. However, there is a general tendency to move towards richer, finer-grained models, possibly including complex emotions, especially in the context of data-driven and task-driven approaches, where restricting the automatic detection to a small set of basic emotions would fall short to achieve the objective. This is also our perspective.
From a computational perspective, the choice of the model of affect to be used in order to give psychological grounding to the resource or the corpus to be developed is driven from, and highly intertwined with, the specific sentiment analysis task to be addressed, which, in turn, usually depends on the application domain to be tackled and on the final purpose of mining affective contents in texts. In this sense, evaluating the emotional responses of an audience in front of an artwork, with the purpose of monitoring the emotional impact of a cultural heritage exhibition on visitors Bertola and Patti (2016), is different from monitoring political sentiment or or mining the levels of anger in comments threads of software developers Gachechiladze et al. (2017). There are still few works and resources specifically developed to address emotion detection in the art and media domain, including the work in Mohammad and Kiritchenko (2018), where authors described the WikiArt Emotions Dataset, that includes emotions annotations for thousands of pieces of art from the WikiArt.org’s collection, and the work in Bertola and Patti (2016); Patti et al. (2015) where the ArsEmotica framework is proposed, which relies on the combined use of NLP affect resources and an ontology of emotions to enable an emotion-driven exploration of online art collections.
The diversity of computational models implied by the analysis of the emotional response to art and media, and the applications that exploit this response to improve the user experience – from learning to entertainment – witnesses the complexity of the underlying processes (aesthetic, self-regulatory, social and cultural). This diversity, however, can be an obstacle to the development of models which work across domains and formats, preventing techniques from being transferred across similar tasks (e.g., emotion annotation and affective recommendation). In particular, the differences in emotion annotation between datasets can endanger the development and cross-validation of new techniques for analysing and exploiting emotions in art and media. In this sense, techniques for merging and extending emotional categories can be useful to overcome these limitations. A notable example of such a comprehensive system is SenticNet Cambria et al. (2020), which relies on the Hourglass model Cambria et al. (2012). The Hourglass model, recently revised and extended Susanto et al. (2020), is inspired by Plutchik’s model of emotions Plutchik (2001). Such model, formalized in the ArsEmotica ontology and described in detail in section 4, can be represented as a wheel of emotions and is formed by: basic or primary emotions; opposite emotions; similarity between the emotions; compound emotions (or complex emotions) generated by the primary ones.
Similarly to the SenticNet framework, our system also relies on Plutchik’s model. The choice of this model is based on the fact that it provides a recipe for the generation of compound emotions that is compliant with the commonsense reasoning framework of the logic. As such, we exploited the reasoning mechanisms of to generate the compound emotions according to the Plutchik’s theory. In this paper, we illustrate and validate this approach by means of the DEGARI system (Dynamic Emotion Generator And ReclassIfier) for emotion attribution and recommendation. In particular, we have exploited the generated compounds to automatically reclassify items in three datasets in the artistic and media domains. As a result of this reclassification process, an emotional enrichment is obtained and new emotional labels are associated with the items in the original datasets. Thanks to the properties of framework, the results of such reclassication - as will be shown in the paper - are entirely explainable.
The paper is organized as follows: after a brief overview of the rationale adopted by our commonsense reasoning framework (Section 2), we present in Section 3 - for the sake of self-containedness - a more detailed description of the logic (by referring to Lieto and Pozzato (2020) for a complete explanation). In Section 4 we present the ontological model ArsEmotica (enriched with an emotional lexicon) formalizing the Plutchik’s theory of emotions and used as a standard representation to leverage the reasoning capabilities of the within the system DEGARI. Sections 5 and 6 present the DEGARI system that, starting from the basic emotions represented in ArsEmotica (and according to the Plutchik’s theory), generates compound emotions and uses these novel emotional categories for artistic content reclassification. Finally, Section 7 shows both the outcome of the automatic and explainable reclassification obtained with DEGARI and the results of a user study on 44 people showing the feasibility of using the obtained reclassifications as recommended contents. Section 8 ends the paper.
2 Commonsense Concept Invention via Dynamic Knowledge Combination
The overall rationale assumed in the reasoning framework is that the process of automatic generation of novel concepts within a knowledge base (also known as knowledge invention) can be obtained, as happens in humans Lieto and Pozzato (2020), by exploiting a process of commonsense conceptual combination. This generative phenomenon highlights some crucial aspects of the knowledge processing capabilities in human cognition. Such ability, in fact, concerns high-level capacities associated to creative thinking and problem solving. Still, however, it represents an open challenge in the field of Artificial Intelligence (AI) Boden (1998).
Dealing with this problem, indeed, requires, from an AI perspective, the harmonization of two conflicting requirements that are hardly accommodated in symbolic systems Frixione and Lieto (2011): the need of a syntactic and semantic compositionality (typical of logical systems) and the one concerning the exhibition of typicality effects. According to a well-known argument Osherson and Smith (1981), in fact, prototypes (i.e. commonsense conceptual representations based on typical properties) are not compositional. The argument runs as follows: consider a concept like pet fish. It results from the composition of the concept pet and of the concept fish. However, the prototype of pet fish cannot result from the composition of the prototypes of a pet and a fish: e.g. a typical pet is furry and warm, a typical fish is grayish, but a typical pet fish is neither furry and warm nor grayish (typically, it is red). The pet fish phenomenon is a paradigmatic example of the difficulty to address when building formalisms and systems trying to imitate this combinatorial human ability.
In this paper, we exploit the recently introduced nonmonotonic extension of Description Logics (typicality-based compositional logic, introduced in Lieto and Pozzato (2020)), that is able to account for this type of human-like concept combination
In this logic, “typical” properties can be directly specified by means of a “typicality” operator enriching the underlying Description Logic (from now on, DL for short), and a TBox can contain inclusions of the form to represent that “typical s are also ”. As a difference with standard DLs, in the logic one can consistently express exceptions and reason about defeasible inheritance as well. Typicality inclusions are also equipped by a real number representing the probability/degree of belief in such a typical property: this allows us to define a semantics inspired to the DISPONTE semantics Riguzzi et al. (2015a) characterizing probabilistic extensions of DLs, which in turn is used in order to describe different scenarios where only some typicality properties are considered. Given a KB containing the description of two concepts and occurring in it, we then consider only some scenarios in order to define a revised knowledge base, enriched by typical properties of the combined concept by also implementing a heuristics coming from the cognitive semantics.
By relying on , this work introduce the system DEGARI which, first, automatically builds prototypes of existing compound emotions by extracting information about concepts or properties by relying on the ArsEmotica ontology enriched with the NRC Emotion Intensity Lexicon Mohammad (2018) (associating, in descending order of frequency, words to emotional concepts). In this setting, words with the highest frequencies of association to emotional concepts have been used as typical features of the basic emotions in the Plutchik model. Such prototypes of basic emotions have been formalized by means of a knowledge base, whose TBox contains both rigid inclusions of the form
in order to express essential desiderata but also constraints, for instance as well as prototypical properties of the form
representing typical concepts of a given emotion, where is a real number in the range , expressing the frequency of such a concept in items belonging to that emotion: for instance, is used to express that the typical feature of being surprised contains/refers to the emotional concept Delight with a frequency/probability/degree of belief of the .
Given the ArsEmotica knowledge base (see Section 4) with the prototypical descriptions of basic emotions, DEGARI exploits the reasoning capabilities of the logic in order to generate new derived emotions as the result of the creative combination of two (or even more) basic or derived ones. DEGARI also reclassifies the artistic and multimedia contents taking the new, derived emotions into account. Intuitively, an item of the tested dataset belongs to the new generated emotion if its metadata (name, description, title) contain all the rigid properties as well as at least the of the typical properties of such a derived emotion. In this respect, DEGARI can be seen as a “white box” recommender system, able to suggest to its users artistic contents belonging to new emotions by providing an explanation of such a recommendation.
We have tested DEGARI by performing two different kinds of evaluation that are reported and discussed in Section 7, namely an automatic evaluation, and an evaluation of the satisfaction of users, showing promising results. In the following, to make the paper self-contained we recall in more detail the main features of the above described logic.
3 The Description Logic for Concept Combination
The logic Lieto and Pozzato (2020), used by the system DEGARI as the basis for the generation of new compound emotions combines three main ingredients. The first one relies on the DL of typicality introduced in Giordano et al. (2015), which allows to describe the protoype of a concept. In this logic, “typical” properties can be directly specified by means of a “typicality” operator enriching the underlying DL, and a TBox can contain inclusions of the form to represent that “typical s are also ”. As a difference with standard DLs, in the logic one can consistently express exceptions and reason about defeasible inheritance as well. For instance, a knowledge base can consistently express that “normally, athletes are fit”, whereas “sumo wrestlers usually are not fit” by and , given that . The semantics of the operator is characterized by the properties of rational logic Lehmann and Magidor (1992), recognized as the core properties of nonmonotonic reasoning. is characterized by a minimal model semantics corresponding to an extension to DLs of a notion of rational closure as defined in Lehmann and Magidor (1992) for propositional logic: the idea is to adopt a preference relation among models, where intuitively a model is preferred to another one if it contains less exceptional elements, as well as a notion of minimal entailment restricted to models that are minimal with respect to such preference relation. As a consequence, inherits well-established properties like specificity and irrelevance: in the example, the logic allows us to infer (being bald is irrelevant with respect to being fit) and, if one knows that Hiroyuki is a typical sumo wrestler, to infer that he is not fit, giving preference to the most specific information.
As a second ingredient, we consider a distributed semantics similar to the one of probabilistic DLs known as DISPONTE Riguzzi et al. (2015b), allowing to label inclusions with a real number between 0.5 and 1, representing its degree of belief/probability, assuming that each axiom is independent from each others. Degrees of belief in typicality inclusions allow to define a probability distribution over scenarios: roughly speaking, a scenario is obtained by choosing, for each typicality inclusion, whether it is considered as true or false. In a slight extension of the above example, we could have the need of representing that both the typicality inclusions about athletes and sumo wrestlers have a degree of belief of , whereas we also believe that athletes are usually young with a higher degree of , with the following KB:
We consider eight different scenarios, representing all possible combinations of typicality inclusion: as an example, represents the scenario in which (2) and (4) hold, whereas (3) does not. Obviously, (1) holds in every scenario, since it represents a rigid property, not admitting exceptions. We equip each scenario with a probability depending on those of the involved inclusions: the scenario of the example has probability (since 2 and 4 are involved) (since 3 is not involved) . Such probabilities are then taken into account in order to choose the most adequate scenario describing the prototype of the combined concept.
As a third element of the proposed formalization we employ a method inspired by cognitive semantics Hampton (1987) for the identification of a dominance effect between the concepts to be combined: for every combination, we distinguish a HEAD, representing the stronger element of the combination, and a MODIFIER. The basic idea is: given a KB and two concepts (HEAD) and (MODIFIER) occurring in it, we consider only some scenarios in order to define a revised knowledge base, enriched by typical properties of the combined concept .
Let us now present the logic more precisely.
The language of extends the basic DL by typicality inclusions of the form equipped by a real number – observe that the extreme is not included – representing its degree of belief, whose meaning is that “we believe with degree/probability that, normally, s are also s”
Definition 3.1 (Language of )
We consider an alphabet of concept names , of role names , and of individual constants . Given and , we define:
We define a knowledge base where:
is a finite set of rigid properties of the form ;
is a finite set of typicality properties of the form
where is the degree of belief of the typicality inclusion;
is the ABox, i.e. a finite set of formulas of the form either or , where and .
A model in the logic extends standard models by a preference relation among domain elements as in the logic of typicality Giordano et al. (2015). In this respect, means that is “more normal” than , and that the typical members of a concept are the minimal elements of with respect to this relation
Definition 3.2 (Model of )
A model is any structure
is a non empty set of items called the domain;
is an irreflexive, transitive, well-founded and modular (for all in , if then either or ) relation over ;
is the extension function that maps each atomic concept to , and each role to , and is extended to complex concepts as follows:
A model can be equivalently defined by postulating the existence of a function , where assigns a finite rank to each domain element Giordano et al. (2015): the rank of is the length of the longest chain from to a minimal , i.e. such that there is no such that . The rank function and can be defined from each other by letting if and only if .
Definition 3.3 (Model satisfying a knowledge base in )
Let be a KB. Given a model , we assume that is extended to assign a domain element of to each individual constant of . We say that:
satisfies if, for all , we have ;
satisfies if, for all , we have that
4, i.e. ;
satisfies if, for each assertion , if then , otherwise if then .
Even if the typicality operator itself is nonmonotonic (i.e. does not imply ), what is inferred from a KB can still be inferred from any KB’ with KB KB’, i.e. the resulting logic is monotonic. As already mentioned, in order to perform useful nonmonotonic inferences, in Giordano et al. (2015) the authors have strengthened the above semantics by restricting entailment to a class of minimal models. Intuitively, the idea is to restrict entailment to models that minimize the atypical instances of a concept. The resulting logic corresponds to a notion of rational closure on top of . Such a notion is a natural extension of the rational closure construction provided in Lehmann and Magidor (1992) for the propositional logic. This nonmonotonic semantics relies on minimal rational models that minimize the rank of domain elements. Informally, given two models of KB, one in which a given domain element has rank 2 (because for instance , and another in which it has rank 1 (because only ), we prefer the latter, as in this model the element is assumed to be “more typical” than in the former. Query entailment is then restricted to minimal canonical models. The intuition is that a canonical model contains all the individuals that enjoy properties that are consistent with KB. This is needed when reasoning about the rank of the concepts: it is important to have them all represented.
Given a KB and given two concepts and occurring in , the logic allows defining a prototype of the combined concept as the combination of the HEAD and the MODIFIER , where the typical properties of the form (or, equivalently, ) to ascribe to the concept are obtained by considering blocks of scenarios with the same probability, in decreasing order starting from the highest one. We first discard all the inconsistent scenarios, then:
we discard those scenarios considered as trivial, consistently inheriting all the properties from the HEAD from the starting concepts to be combined. This choice is motivated by the challenges provided by task of commonsense conceptual combination itself: in order to generate plausible and creative compounds it is necessary to maintain a level of surprise in the combination. Thus both scenarios inheriting all the properties of the two concepts and all the properties of the HEAD are discarded since they prevent this surprise;
among the remaining ones, we discard those inheriting properties from the MODIFIER in conflict with properties that could be consistently inherited from the HEAD;
if the set of scenarios of the current block is empty, i.e. all the scenarios have been discarded either because trivial or because preferring the MODIFIER, we repeat the procedure by considering the block of scenarios, having the immediately lower probability.
Remaining scenarios are those selected by the logic . The ultimate output of our mechanism is a knowledge base in the logic whose set of typicality properties is enriched by those of the compound concept . Given a scenario satisfying the above properties, we define the properties of as the set of inclusions , for all that are entailed from in the logic . The probability is such that:
if is entailed from , that is to say is a property inherited either from the HEAD (or from both the HEAD and the MODIFIER), then corresponds to the degree of belief of such inclusion of the HEAD in the initial knowledge base, i.e. ;
otherwise, i.e. is entailed from , then corresponds to the degree of belief of such inclusion of a MODIFIER in the initial knowledge base, i.e. .
The knowledge base obtained as the result of combining concepts and into the compound concept is called -revised knowledge base, and it is defined as follows:
for all such that either is entailed in or is entailed in , and is defined as above.
As an example, consider the following version of the above mentioned Pet-Fish problem. Let KB contains the following inclusions:
representing that a typical fish is greyish , scaly and not affectionate , whereas a typical pet does not live in water , is loved by kids and is affectionate . Concerning rigid properties, we have that all fishes live in water . The logic combines the concepts and , by using the latter as the HEAD and the former as the MODIFIER. The prototypical Pet-Fish inherits from the prototypical fish the fact that it is scaly and not affectionate, the last one by giving preference to the HEAD since such a property conflicts with the opposite one in the modifier (a typical pet is affectionate). The scenarios in which all the three typical properties of a typical fish are inherited by the combined concept are considered as trivial and, therefore, discarded, as a consequence the property having the lowest degree ( with degree ) is not inherited. The prototypical Pet-Fish inherits from the prototypical pet only property , since conflicts with the rigid property , stating that all fishes (then, also pet fishes) live in water, whereas is blocked, as already mentioned, by the HEAD/MODIFIER heuristics. Formally, the -revised knowledge base contains, in addition to the above inclusions, the following ones:
In Lieto and Pozzato (2020) it has been also shown that reasoning in remains in the same complexity class of standard Description Logics.
For the purposes of this paper it is worth-noticing that, as mentioned, the reasoning framework presented in this section has been applied, via the DEGARI system, to the generation of new compound emotions by starting from the affective ontological knowledge base named ArsEmotica. Such ontological model is described in the next section.
4 The ArsEmotica Ontological Model enriched with the NRC Emotion Intensity Lexicon
The affective knowledge leveraged by the logic via the DEGARI system is encoded in an ontology of emotional categories based on Plutchik’s psychological circumplex model Plutchik (1980) called ArsEmotica
Basic or primary emotions: joy, trust, fear, surprise, sadness, disgust, anger, anticipation; in the color wheel this is represented by differently colored sectors.
Opposites: basic emotions can be conceptualized in terms of polar opposites: joy versus sadness, anger versus fear, trust versus disgust, surprise versus anticipation.
Intensity: each emotion can exist in varying degrees of intensity; in the wheel this is represented by the vertical dimension.
Similarity: emotions vary in their degree of similarity to one another; in the wheel this is represented by the radial dimension.
Complex emotions: complex emotions are a mixtures of the primary emotions; looking at the Plutchik’s wheel, the height emotions in the blank spaces are compositions of basic emotions called primary dyads.
We have chosen to encode the Plutchik’s model in the ontology for several reasons. First, it is well-grounded in psychology and general enough to guarantee a wide coverage of emotions. This is important for implementing successful strategies aimed at mapping tags to the emotional concepts of the ontology. Second, as already mentioned, the Plutchik’s “wheel of emotions” is perfectly compliant with the generative model underlying the logic. Finally, it encodes interesting notions, e.g. emotional polar opposites, which can be exploited for finding new relations among artworks.
Within the ArsEmotica ontology, the class Emotion is the root for all the emotional concepts. The Emotion’s hierarchy includes all the 32 emotional categories presented as distinguished labels in the model. In particular, the Emotion class has two disjoint subclasses: BasicEmotion and ComplexEmotion. Basic emotions of the Plutchik’s model are direct sub-classes of BasicEmotion. Each of them is specialized again into two subclasses representing the same emotion with weaker or the stronger intensity (e.g. the basic emotion Joy has Ecstasy and Serenity as sub-classes). Therefore, we have 24 emotional concepts subsumed by the BasicEmotion concept. Instead, the class CompositeEmotion has 8 subclasses, corresponding to the primary dyads. Other relations in the Plutchik’s model have been expressed in the ontology by means of object properties: the hasOpposite property encodes the notion of polar opposites; the hasSibling property encodes the notion of similarity and the isComposedOf property encodes the notion of composition of basic emotions. Moreover, a data type property hasScore was introduced to link each emotion with an intensity value i mapped into the above mentioned Hourglass model. Due to the need of modeling the link among words in a language and the emotions they refer to, the ArsEmotica Ontology is also integrated with the ontology framework LExicon Model for ONtologies (LEMON) McCrae et al. (2011). In particular, such integration allows to explicitly differentiate between the language level (lexicon based) and the conceptual one representing the emotional concepts Patti et al. (2015). Within this enriched framework, it is possible to associate a plethora of emotional words, with the encoding of language information, to the corresponding emotional concepts. Apart from the already available linking with the lexical resources such as WordNet-Affect, MultiWordNet, we have now equipped the ArsEmotica emotional concepts with the NRC Emotion Intensity Lexicon mentioned above Mohammad (2018). Such lexicon provides a list of English words, each with real-values representing intensity scores for the eight basic emotions of Plutchik’s theory. The lexicon includes close to 10,000 words including terms already known to be associated with emotions as well as terms that co-occur in Twitter posts that convey emotions. The intensity scores were obtained via crowdsourcing, using best-worst scaling annotation scheme. For our purposes, we considered the most frequent terms available in such lexicon (and associated to the basic emotions of the Plutchik’s wheel) as typical features of such emotions. In this way, once the prototypes of the basic emotional concepts were formed, the reasoning framework was used to generate the compound emotions.
5 DEGARI: GENERATING NOVEL EMOTIONS from ArsEmotica
In this section we describe DEGARI, the system exploiting the logic on the ArsEmotica knowledge base in order to generate and suggest novel emotion related contents and tested on the RaiPlay catalog
DEGARI’s prototypes generation proceeds in two steps: in the first one, it builds a prototypical description of basic emotions in the language of the logic , in order to describe their typical properties; as a second step, it exploits the above described reasoning mechanism of such a Description Logic in order to combine the prototypical descriptions of pairs of basic emotions, generating the prototypical description of compound emotions. As mentioned above, the obtained ontology is then tested by re-classifying the items belonging to RaiPlay, Wiki Art and ArsMeteo keeping the generated compound emotions into account: this allows us to describe a novel and completely explainable recommending system, which is able to suggest items belonging also to compound emotions.
Concerning the first step, DEGARI builds a knowledge base in the logic characterized by typicality inclusions of the form
where is one of the eight basic emotions of Plutchik’s theory: Joy, Trust, Fear, Surprise, Sadness, Disgust, Anger, and Anticipation. Typical properties are selected from the list of words characterizing each basic emotion in the NRC Emotion Intensity Lexicon where, as already mentioned, the probability represents the intensity scores for the emotion. In detail, for each basic emotion, we consider the six properties/words having the highest scores.
As an example, consider the basic emotion . The words having the highest scores are happiness (), bliss (), to celebrate (), jubilant (), ecstatic (), and euphoria (). Therefore, the knowledge base generated by DEGARI will contain, among others, the following inclusions:
DEGARI then computes novel compound emotions by combining existing ones (by using the same logical procedure of the pet-fish problem). As an example, let us consider the combination of the above basic emotion with , whose prototypical description is as follows:
In order to obtain a description of the compound emotion as the result of the combination of the two basic emotions () in the logic , DEGARI combines the two basic emotions by implementing a variant of CoCoS Lieto et al. (2018), a Python implementation of reasoning services for the logic in order to exploit efficient DLs reasoners for checking both the consistency of each generated scenario and the existence of conflicts among properties, following the line of the system DENOTER Chiodino et al. (2020). More in detail, DEGARI considers both the available choices for the HEAD and the MODIFIER, and it allows to restrict its concern to a given and fixed number of inherited properties. The combined emotion has the following description (concept ):
Obviously, rigid properties of basic emotions (if any) are inherited by the compound emotion (in the example, ), and this avoids the system to consider any inconsistent typical property even if it has the highest probability.
It is worth noticing that the properties of the derived emotion are still expressed in the language of the logic , therefore the combined emotion, in the example, can be used to be further combined with another emotion, in order to iterate the procedure.
6 Reclassification of Emotion-related Content based on DEGARI
By starting from the generated prototypes of the compound emotions in ArsEmotica, DEGARI is also able to perform an emotion-oriented reclassification of the items of the consideted datasets.
In particular, DEGARI employs two different strategies to extract metadata from the items to reclassify. In a first case (e.g. for the datasets of ArsMeteo and WikiArt) the metadata are either stored in the provided resource (e.g. in WikiArt) or are the result of a social tagging activty based on the artistic community. In the second case (e.g. in the case of the RaiPlay dataset) the metadata associated to every and each item (title, name of the program/episode, description of the program/series, description of the episode) are extracted from a crawler. Such metadata are then used to generate the typical description of the items via the computation of the most frequent terms retrieved in their textual description (the assumption is that the most frequently used terms to describe an item are also the ones that are more typically associated to them). The frequencies are computed as the proportion of each property with respect to the set of all properties characterizing the item, in order to compare them with the properties of the derived emotion. If the item contains all the rigid properties and at least the of the typical properties of the compound emotion under consideration, then the item is classified as belonging to it. Last, DEGARI suggests the set of classified contents, in a descending order of compatibility, where a rank of compatibility of a single item with respect to an emotion is intuitively obtained as the sum of the frequencies of “compatible” concepts, i.e. concepts belonging to both the item and the prototypical description of the genre. Formally:
Given an item , let be a compound emotion generated form the ArsEmotica mode as defined in Section 5 and let be the set of words occurring in . Given a knowledge base KB of compound emotions built by DEGARI, we say that is compatible with if the following conditions hold:
contains all rigid properties of , i.e.
contains at least the of typical properties of , i.e.
where is the set of typical properties of .
As another example, consider the derived emotion , which in Plutchik’s wheel corresponds to the combined emotion “delight”. The knowledge base in the logic describing such a compound emotion is as follows:
For instance, the multimedia item “È arrivata la felicità” (“Happiness has come”) (https://www.raiplay.it/programmi/earrivatalafelicita) from the RaiPlay dataset is reclassified in the novel, generated emotion , since:
all rigid properties of both basic emotions are satisfied, that is to say neither nor belong to the properties extracted for the item;
more than the of the typical properties of the compound emotion are satisfied; in particular, “È arrivata la felicità” has () and ().
This item will be then recommended by DEGARI as shown in Figure 2.
Figure 2 also shows how DEGARI can be considered as an explainable AI system: indeed, an explanation of the reasons why the multimedia item has been reclassified in the compound concept is provided, in order to let the user be aware of the procedure of the system. As a difference with “black box” approaches, DEGARI explicitly reports that the “instance’s description has the following word(s) in common with category prototype”, followed by the two above matching properties surprise and happiness. Moreover, the whole procedure is completely known and could be used to further expand the feedback provided by system: from the axiom system and the corresponding semantics of the Description Logic with typicality to the DISPONTE semantics adopted by the logic in order to compute the prototypical description of the compound emotion.
DEGARI evaluation has been carried out on three different datasets and evaluated in a twofold way. The datasets considered (described in detail below) are: ArsMeteo, the RaiPlay catalog and Wiki Art Emotion. The evaluation has concerned a first, completely automatic, test consisting in calculating the percentage of the reclassified items within the novel hybrid emotion classes generated by the system via . In this case, as a positive indicator has been considered the spread of the reclassified items along most of the concepts of the wheel of emotion. This aspect, indeed, shows how the created prototypes of the compound emotions are mostly meaningful and able to reclassify the artistic content available in three original datasets.
A second evaluation, aimed at measuring the satisfaction of the potential users of the system when exposed to the contents of the novel categories suggested by DEGARI, consisted in a user study
In the following we briefly describe the adopted datasets: two of them are art-related ones (ArsMeteo and WikiArt Emotion), while the RaiPlay dataset contains all the multimedia items (e.g. movies, tv series, tv shows, documentaries etc.) available on the online multimedia platform or RAI, Radiotelevisione Italiana.
7.1 ArsMeteo Dataset
ArsMeteo Acotto et al. (2009) is an art portal for sharing artworks and their emerging, connective meanings. Its development is leaded by a non-profit cultural organization called Associazione Culturale ArsMeteo (AMA), based in Turin, Italy. It enables the collection of digital (or digitalized) artworks and performances, belonging to a variety of artistic forms including poems, videos, pictures and music. Meanings are given by the tagging activity of the community. All contents are accessible as “digital commons”. Currently, the portal has collected over 350,000 visits and gathers a collection of over 9,000 artifacts produced by 307 artists; it has collected almost 38,000 tags.
7.2 RAIPLAY Dataset
The RaiPlay dataset is composed by 4,612 multimedia items extracted from RaiPlay https://www.raiplay.it/: the online platform of RAI. Such dataset contains different types of multimedia content grouped in six main narrative categories: Movies, Fiction, Kids, TV Series, Drama, Comedy.
As mentioned, each multimedia item/episode is equipped by some information, namely: title, name of the program/episode, description of the program/series, description of the episode. Such descriptions are used by DEGARI to extract the relevant information to associate to every item and to decide whether, given the extracted information, the item should be reclassified in one of the previously generated compound emotions.
7.3 WikiArt Emotions
WikiArt Emotions is a dataset of 4,105 artworks with annotations for the emotions evoked in the observer Mohammad and Kiritchenko (2018). The artworks were selected from the online visual art encyclopedia WikiArt.org. Each piece of art is annotated for one or more of 20 emotion categories (including neutral). Annotations were obtained via crowdsourcing, asking annotators all emotions evoked by the title of the artwork, the image of the artwork or the artwork as a whole. The annotators were also asked to point out if the artwork depicted a face, or a human body but not a face: these additional information is included in the dataset (if an artwork didn’t depict a face nor a body, it was marked as ”none”). In order to decide if an emotion applies or not to an artwork, the authors specified an aggregation threshold of : if at least of the responses indicated that a certain emotion applied, then the label was chosen. Other distributions of the dataset with different aggregation thresholds ( and ) are available, but we chose to use the threshold version, as recommended by the authors of the dataset Mohammad and Kiritchenko (2018).
7.4 Automatic Reclassification
The obtained results for what concerns the automatic evaluation are presented in the table below. Overall the figure shows that for two of the three datasets (ArsMeteo and RaiPlay) DEGARI is able to reclassify and spread the original items along the entire wheel of emotion assumed in the Plutchik’s model, thus allowing a more fine grained characterization.
In these two cases, the percentage of the reclassified items is of and respectively. On the other hand, the WkiArt Emotions dataset, contains orthogonal results since, in this case, 16 out of the 31 generated compound emotions are filled with reclassified items. In this case, however, a large part of the dataset () is involved in such a reclassification.
The main reason for these orthogonal results is in the kind of input considered by DEGARI. Indeed, while for ArsMeteo and RaiPlay the metadata associated to the items are either the result of a social tagging activity by a community of artists (like in ArsMeteo
While this fact, on one hand, creates - for the WikiArt Emotions dataset - more clean metadata and allows the reclassifications of most of the items available in the dataset, on the other hand forces the user to use a predefined vocabulary for annotation that, as such, inhibits more free association that could have led to a wider reclassification and redistribution of the items along the entire Plutchik’s wheel. In all the 3 cases, however, most of the compound emotions generated by DEGARI are filled with new items.
7.5 User Study
The goal of the user study was to assess the acceptance of the emotion categories suggested by DEGARI, with the ultimate goal of using the reclassifications produced by the system to improve the annotation of artworks and media, and, consequently, the applications which depend on it, such as personalization and recommendation.
Methods and material. The user study consisted in an online questionnaire (in Italian). The questionnaire contained items, each represented by an image, and for the multimedia items, by the film poster accompanied by the link to the online player for watching the content. For each item, the users received two questions: the first question (Question ) asked them to rate the association of the item with the emotional category provided by DEGARI on 10-point scale; the second question (Question ) asked them to associate the item to additional emotion categories, taken from Plutchik’s model. Users were divided in 3 groups, each corresponding to a different set of items. For each dataset, the selected items were the ones ranked higher by DEGARI for each generated compound emotional category.
Participants and procedure. The study involved users ( female, male). Concerning the age groups, users were below ; users were in the - age range; in the - range; in the - range; were older than . Users were randomly assigned to the questionnaires. The first questionnaire was filled out by users; the second questionnaire was filled out by users; the third questionnaire was assigned to users. As a result, ratings and emotion categories were collected.
Results and analysis. Concerning Question , the average rating assigned by the users to the emotion category proposed by Degari was , with only slight differences between the datasets (see Table 1). The average rating was for ArsMeteo, for RaiPlay, and for WikiArt. The standard deviation was ( for ArsMeteo, for WikiArt, and for RaiPlay), suggesting that the differences in ratings were limited. Also, the median rating is for all data sets, with only proposed emotion categories ( from Arsmeteo and from WikiArt) rated below .
Concerning Question (namely, the additional emotions attached by the users to the items), were attached to the items in ArsMeteo, to the items in WikiArt, and to the items in RaiPlay, yielding user emotion categories. The average number of emotion categories per users was . In order to investigate the overlapping between the set of emotion categories proposed by DEGARI for each item (apart for the top ranked category tested through Question ), we compared the emotion categories selected by the users with the ones proposed by DEGARI (Table 2). Data show that 22.05% of the emotion categories additionally proposed by DEGARI for each item matched those selected by the users, with a higher value for ArsMeteo (%), and a lower value for WikiArt (%) and RaiPlay (%). This datum is a positive one since is concerns the non-top ranked emotional categories suggested by the system (for which the degree of acceptability by the users was always above 5 out of a 10-point scale except for items, and with a median of 7 for every considered dataset).
To conclude, the collected data suggest that the emotion categories proposed by DEGARI as a result of the reclassification process are generally accepted by the users, with few exceptions that deserve further investigation. The acceptance is clear for the top ranked emotion, but an acceptable degree of acceptance can be inferred for the remaining suggested categories, for which an overlapping of % and more with the user tags has been found for all datasets.
8 Discussion and Conclusion
In this paper, we presented DEGARI: an explainable AI system relying on the Description Logics and on the ArsEmotica knowledge base to generate, according to the Plutchik’s theory of emotion, compound emotional concepts starting from the basic ones. Such newly created categories, characterized by lexicon-based typical features, are then used in DEGARI to reclassify, in an emotional settings, the items of three different datasets.
Overall, DEGARI propose a transparent approach for emotionally-driven content reclassification and its combinatorial reasoning engine could be useful for addressing the very well known filter bubble effect Parisier (2012) in recommender systems, by introducing seeds of serendipity in content discovery by users. One fundamental discussion about the applicability of DEGARI in practice is whether or not it represents a truly innovative technical solution for an emotion-based recommender system. According to Sohail et al. (2017) recommender systems “try to identify the need and preferences of users, filter the huge collection of data accordingly and present the best suited option before the users by using some well-defined mechanism”. Despite the huge amount of proposals, the main families of recommender systems can be identified as based on: i) collaborative filtering; ii) content-based filtering; iii) hybrid filtering. At their core of functioning, collaborative filtering exploits similarities of usage patterns among mutually affine users, while content-based filtering exploits content similarity. DEGARI by definition falls into the latter category since in its current form it uses content description (obtained in different ways) as the input. From the technical point of view, however, it differs from the current mainstream approaches that are mostly based on the comparison and matching of visual and perceptual features of the content Sun et al. (2018); Deldjoo et al. (2018). In practice, our approach adds a logic layer capable of mapping and representing - in a commonsense and cognitively compliant fashion - new emotional categories which can be used to affect user preferences and content consumption in a way that cannot be derived from the pure statistical analysis of content and/or the comparison of similar users. Moreover, the proposed approach has been applied to a well-known model, the Plutchik’s circumplex model of emotions Plutchik (1980), but could in principle applied to other models which organize emotions by similarity, opposition and composition, such as for example the extended version of the Hourglass model used in SenticNet Susanto et al. (2020). Being independent from the specific application model and type of expression, this approach can work effectively in different domains, as shown by its use on the datasets of artworks and media illustrated in this paper. In this sense, it can promote the interoperability of affective annotations and the cross-domain reuse of techniques and methods.
In the future work, we plan to extend the evaluation currently conducted in the form of a user study to a large scale one to further validate the effectiveness of the proposed approach. We also plan to extend the applications of this system to different domains. A first extension will be in the field of the emotional-oriented recommendation of artworks within Museums and cultural heritage sites (this is a work currently under development within the H2020 European SPICE project https://spice-h2020.eu/. In addition, also the field of music recommendation represents a current area of investigation. From a technical perspective, in future research, we aim at studying the application of optimization techniques in Alberti et al. (2017) in order to improve the efficiency of the DEGARI knowledge generation system. Secondly we aim at considering more accurate and multimodal descriptions of artistic and media items, by exploiting Automatic Speech Recognition data and semantic visual categories extracted from video and audio channels of the content. Finally, as mentioned, we plan to improve the provided recommendations by justifying the provided content reclassification (and recommendation) also referring to the probabilistic ranks assigned to the shared features between the generated emotion and the items to reclassify.
- Other works have already shown how such logic can be used to model complex cognitive phenomena Lieto and Pozzato (2020), creative problem solving Lieto et al. (2019b, a) and to build intelligent applications in the field of computational creativity Lieto and Pozzato (2019). Alternative approaches to the problem of commonsense conceptual combination have been recently discussed in Eppe et al. (2018), Lewis and Lawry (2016), Confalonieri et al. (2016). The main advantages of with respect to such approaches are detailed in Lieto and Pozzato (2020).
- The reason why we only allow typicality inclusions equipped with probabilities is due to our effort of integrating two different semantics: typicality based logic and DISPONTE. In particular, as detailed in Lieto and Pozzato (2020) this choice seems to be the only one compliant with both the formalisms. On the contrary, it would be misleading to also allow low degrees of belief for typicality inclusions, since typical knowledge is known to come with a low degree of uncertainty.
- It could be possible to consider an alternative semantics whose models are equipped with multiple preference relations. However the approach based on a single preference relation in Giordano et al. (2015) ensures good computational properties (reasoning in the resulting nonmonotonic logic has the same complexity of the standard ), whereas adopting multiple preference relations could lead to higher complexities.
- It is worth noticing that here the degree does not play any role. Indeed, a typicality inclusion holds in a model only if it satisfies the semantic condition of the underlying DL of typicality, i.e. minimal (typical) elements of are elements of . The degree of belief will have a crucial role in the application of the distributed semantics, allowing the definition of scenarios as well as the computation of their probabilities.
- The ArsEmotica ontology is available here: http://184.108.40.206/fuseki/ArsEmotica-core and queryable via SPARQL endpoint at: http://220.127.116.11/fuseki/dataset.html?tab=query&ds=/ArsEmotica-core
- This is one of the most commonly used methodology for the evaluation of recommender systems based on controlled small groups analysis, see Shani and Gunawardana (2011).
- The ArsMeteo dataset has the additional difficulty of being an heterogeneous dataset, touching different artistic genres (from poetry to literature to paintings).
- Arsmeteo: artworks and tags floating over the planet art. In Proceedings of the 20th ACM conference on Hypertext and hypermedia, pp. 331–332. Cited by: §5, §7.1.
- Cplint on SWISH: probabilistic logical inference with a web browser. Intelligenza Artificiale 11 (1), pp. 47–64. External Links: Cited by: §8.
- Moodplay: interactive music recommendation based on artists’ mood similarity. International Journal of Human-Computer Studies 121, pp. 142–159. Cited by: §1.
- Introducing the geneva multimodal emotion portrayal (gemep) corpus. Blueprint for affective computing: A sourcebook 2010, pp. 271–94. Cited by: §1.
- Ontology-based affective models to organize artworks in the social semantic web. Information Processing & Management 52 (1), pp. 139–162. External Links: Cited by: §1, §5.
- Creativity and artificial intelligence. Artificial Intelligence 103 (1-2), pp. 347–356. Cited by: §2.
- SenticNet 6: ensemble application of symbolic and subsymbolic ai for sentiment analysis. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 105–114. Cited by: §1, §1.
- The hourglass of emotions. In Cognitive behavioural systems, pp. 144–157. Cited by: §1, §4.
- A knowledge-based system for the dynamic generation and classification of novel contents in multimedia broadcasting. In ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 - September 8, 2020 - Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), G. D. Giacomo, A. Catalá, B. Dilkina, M. Milano, S. Barro, A. Bugarín and J. Lang (Eds.), Frontiers in Artificial Intelligence and Applications, Vol. 325, pp. 680–687. External Links: Cited by: §5.
- Conceptual blending in EL++. In Proceedings of the 29th International Workshop on Description Logics, Cape Town, South Africa, April 22-25, 2016., M. Lenzerini and R. Peñaloza (Eds.), CEUR Workshop Proceedings, Vol. 1577. External Links: Cited by: footnote 1.
- Audio-visual encoding of multimedia content for enhancing movie recommendations. In Proceedings of the 12th ACM Conference on Recommender Systems, pp. 455–459. Cited by: §8.
- Basic emotions. Handbook of cognition and emotion 98 (45-60), pp. 16. Cited by: §1, §1.
- A computational framework for conceptual blending. Artificial Intelligence 256, pp. 105–129. Cited by: footnote 1.
- Representing and reasoning on typicality in formal ontologies. In Proceedings of the 7th International Conference on Semantic Systems, pp. 119–125. Cited by: §2.
- Anger and its direction in collaborative software development. In Proceedings of the 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER ’17, Piscataway, NJ, USA, pp. 11–14. External Links: Cited by: §1.
- Semantic characterization of Rational Closure: from Propositional Logic to Description Logics. Artificial Intelligence 226, pp. 1–33. External Links: Cited by: §3, §3, §3, §3, footnote 3.
- Inheritance of attributes in natural concept conjunctions. Memory & Cognition 15 (1), pp. 55–71. Cited by: §3.
- Lexicons for sentiment, affect, and connotation. In Speech and language processing: an introduction to natural language processing, computational linguistics, and speech recognition, 3nd Edition, Prentice Hall series in artificial intelligence. Note: Draft, available online: https://web.stanford.edu/ jurafsky/slp3/21.pdf. Accessed on December 2020 Cited by: §1.
- What makes an art expert? emotion and evaluation in art appreciation. Cognition and Emotion 28 (6), pp. 1137–1147. Cited by: §1.
- What does a conditional knowledge base entail?. Artificial Intelligence 55 (1), pp. 1–60. External Links: Cited by: §3, §3.
- Hierarchical conceptual spaces for concept combination. Artificial Intelligence 237, pp. 204–227. Cited by: footnote 1.
- COCOS: a typicality based COncept COmbination System . In Proceedings of the 33rd Italian Conference on Computational Logic (CILC 2018), M. Montali and P. Felli (Eds.), CEUR Workshop Proceedings, Vol. , Bozen, Italy, pp. 55–59. External Links: Cited by: §5.
- Beyond subgoaling: a dynamic knowledge generation framework for creative problem solving in cognitive architectures. Cognitive Systems Research 58, pp. 305–316. Cited by: footnote 1.
- Knowledge capturing via conceptual reframing: a goal-oriented framework for knowledge invention. In Proceedings of the 10th ACM Conference on Knowledge Capture, K-CAP 2019, Marina del Rey, pp. 109–114. Cited by: footnote 1.
- Applying a description logic of typicality as a generative tool for concept combination in computational creativity. Intelligenza Artificiale 13 (1), pp. 93–106. Cited by: footnote 1.
- A description logic framework for commonsense conceptual combination integrating typicality, probabilities and cognitive heuristics. Journal of Experimental & Theoretical Artificial Intelligence 32 (5), pp. 769–804. Cited by: §1, §2, §3, §3, footnote 1, footnote 2.
- Modelling affect for horror soundscapes. IEEE Transactions on Affective Computing 10 (2), pp. 209–222. Cited by: §1.
- Linking lexical resources and ontologies on the semantic web with lemon. In Extended Semantic Web Conference, pp. 245–259. Cited by: §4.
- Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in temperament. Current Psychology 14 (4), pp. 261–292. Cited by: §1.
- WikiArt emotions: an annotated dataset of emotions evoked by art. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. External Links: Cited by: §1, §1, §7.3.
- Crowdsourcing a word-emotion association lexicon. Computational Intelligence 29 (3), pp. 436–465. Cited by: §1.
- Word affect intensities. Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC). Cited by: §1, §2, §4.
- Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 174–184. External Links: Cited by: §1.
- Chapter 3 - semantic aspects in sentiment analysis. In Sentiment Analysis in Social Networks, F. A. Pozzi, E. Fersini, E. Messina and B. Liu (Eds.), pp. 31 – 48. External Links: Cited by: §1.
- Art and emotions. International journal of applied psychoanalytic studies 10 (2), pp. 100–107. Cited by: §1.
- On the adequacy of prototype theory as a theory of concepts. Cognition 9 (1), pp. 35–58. Cited by: §2.
- The filter bubble: what the internet is hiding from you. Penguin Books Limited. Cited by: §8.
- Arsemotica for arsmeteo. org: emotion-driven exploration of online art collections. In The Twenty-Eighth International Florida Artificial Intelligence Research Society Conference (FLAIRS 2015), pp. 288–293. Cited by: §1, §4.
- A general psychoevolutionary theory of emotion. In Theories of emotion, pp. 3–33. Cited by: §1, §4, §8.
- The nature of emotions. American scientist 89 (4), pp. 344–350. Cited by: §1.
- Probabilistic description logics under the distribution semantics. Semantic Web 6 (5), pp. 477–501. External Links: Cited by: §2.
- Reasoning with probabilistic ontologies. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, Q. Yang and M. Wooldridge (Eds.), pp. 4310–4316. External Links: Cited by: §3.
- Ars rhetorica. aristotle. Clarendon Press, Oxford. Cited by: §1.
- On the sublime. Clarendon. Cited by: §1.
- A circumplex model of affect.. Journal of personality and social psychology 39 (6), pp. 1161. Cited by: §1.
- Evaluating recommendation systems. In Recommender systems handbook, pp. 257–297. Cited by: footnote 9.
- Film structure and the emotion system. Cambridge University Press. Cited by: §1.
- Classifications of recommender systems: a review. Engineering Science and Technology Review 10 (4), pp. 132–153. Cited by: §8.
- SemEval-2007 task 14: affective text. In Proceedings of the 4th International Workshop on Semantic Evaluations, SemEval ’07, Stroudsburg, PA, USA, pp. 70–74. Cited by: §1.
- WordNet affect: an affective extension of WordNet. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal. External Links: Cited by: §1.
- A multi-modality deep network for cold-start recommendation. Big Data and Cognitive Computing 2 (7). Cited by: §8.
- The hourglass model revisited. IEEE Intelligent Systems 35 (5), pp. 96–102. Cited by: §1, §8.
- Emotion and the structure of narrative film: film as an emotion machine. Routledge. Cited by: §1.
- Emotion regulation through listening to music in everyday situations. Cognition & emotion 26 (3), pp. 550–560. Cited by: §1.
- Implicit emotion regulation in the context of viewing artworks: erp evidence in response to pleasant and unpleasant pictures. Brain and Cognition 107, pp. 48–54. Cited by: §1.
- A review of emotion sensing: categorization models and algorithms. Multimedia Tools and Applications 79, pp. . External Links: Cited by: §1.
- Development and validation of brief measures of positive and negative affect: the panas scales.. Journal of personality and social psychology 54 (6), pp. 1063. Cited by: §1.