Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations
With the emergence of Web 2.0, tag recommenders have become important tools, which aim to support users in finding descriptive tags for their bookmarked resources. Although current algorithms provide good results in terms of tag prediction accuracy, they are often designed in a data-driven way and thus, lack a thorough understanding of the cognitive processes that play a role when people assign tags to resources. This thesis aims at modeling these cognitive dynamics in social tagging in order to improve tag recommendations and to better understand the underlying processes.
As a first attempt in this direction, we have implemented an interplay between individual micro-level (e.g., categorizing resources or temporal dynamics) and collective macro-level (e.g., imitating other users’ tags) processes in the form of a novel tag recommender algorithm. The preliminary results for datasets gathered from BibSonomy, CiteULike and Delicious show that our proposed approach can outperform current state-of-the-art algorithms, such as Collaborative Filtering, FolkRank or Pairwise Interaction Tensor Factorization. We conclude that recommender systems can be improved by incorporating related principles of human cognition.
Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations
|Supervised by: Prof. Stefanie Lindstaedt (Know-Center)|
|Know-Center, Graz University of Technology|
|Inffeldgasse 13, Graz, Austria|
Categories and Subject Descriptors H.2.8 [Database Management]: Database Applications—Data mining ; H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval—Information filtering
personalized tag recommendations, time-dependent recommender systems, human cognition, social tagging systems
Social tagging systems enable users to collaboratively assign freely chosen keywords, so-called tags, to resources. These tags can then be used for navigating, searching, organizing and finding content, and serendipitous browsing [?, ?]. Hence, tags have become an essential instrument of Web 2.0, the social Web, assisting users during these activities. While in social tagging systems users can freely choose keywords for their bookmarked resources, they have to create a set of descriptive tags on their own, which can be a demanding task [?].
As a solution, a variety of tag recommender algorithms, such as Collaborative Filtering, FolkRank or Pairwise Interaction Tensor Factorization, have been proposed. Tag recommenders suggest a set of tags for a given user-resource pair based on previously used and assigned tags and aim at helping not only the individual to find appropriate tags but also the collective to consolidate the shared tag vocabulary [?]. Furthermore, Dellschaft & Staab [?] have shown that personalized tag recommenders can increase the indexing quality of resources, making it easier for users to understand the information content of an indexed resource based on its assigned tags.
Although current state-of-the-art tag recommender approaches (see Section Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations) perform reasonably well in terms of recommender accuracy, most of them are designed purely data-driven. As a result, they are based on either simply counting tag frequencies or computationally expensive calculation steps (e.g., calculating user similarities or factorizing entities). Hence, these approaches typically ignore important insights originating from cognitive research of how people assign words or tags to resources, which is essential for the design of tag recommenders, that should attempt to mimic the user’s tagging behavior.
As a prominent example in this respect, the work proposed by Fu [?] discusses an interplay between individual micro-level (e.g., categorizing resources or temporal dynamics) and collective macro-level (e.g., imitating other users’ tags) processes in social tagging systems (see Section Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations). Based on that, we state the hypothesis, that a theory-driven approach that is build upon such insights can not only improve recommender accuracy in general but also can help to better understand the underlying cognitive processes.
This thesis aims to develop a novel tag recommender approach, which models the cognitive processes that play a role when people assign tags to resources. At the current stage, our contributions are as follows:
We propose a novel theory-driven approach for recommending tags, which models cognitive processes in social tagging in order to mimic the way humans assign tags to resources.
We conduct an extensive evaluation using dataset samples gathered from three real-world folksonomies (BibSonomy, CiteULike and Delicious) to show the effectiveness of our theory-driven approach.
We show that our approach can outperform several state-of-the-art tag recommender algorithms, such as FolkRank or Pairwise Interaction Tensor Factorization, in terms of recommender accuracy.
We introduce an open-source tag recommender benchmarking framework termed TagRec, which contains not only our proposed approach but also standardized baseline algorithms and evaluation methods.
To date, two types of tag recommenders have been established: folksonomy- and content-based approaches [?]. At the moment, in this work we focus on folksonomy-based algorithms. The most basic approach in this respect is the unpersonalized MostPopular (MP) algorithm that recommends for any user and any resource the same set of tags weighted by the frequency in all tag assignments [?]. A personalized extension of MP is the MostPopular (MP) algorithm that suggests the most frequent tags in the tag assignments of the user, MostPopular (MP), and the resource, MostPopular (MP) [?]. Another classic recommender approach is Collaborative Filtering (CF), which has been adapted for tag recommendations by Marinho et al. [?] to form the neighborhood of a user based on the tag assignments in the user profile. According to Gemmell et al. [?], the best results for CF in social tagging systems are obtained with a neighborhood size of 20 users.
Another well-known tag recommender approach is the FolkRank (FR) algorithm, which is an improvement of the Adapted PageRank (APR) approach [?]. FR and APR adapt the Google PageRank algorithm to rank the nodes within the graph structure of a folksonomy based on their importance in the network [?]. A different popular and recent tag recommender mechanism is Pairwise Interaction Tensor Factorization (PITF) proposed by Rendle & Schmidt-Thieme [?], which is an extension of Factorization Machines (FM) [?] and explicitly models the pairwise interactions between users, resources and tags in order to predict future tag assignments. As for algorithms that utilize topic models, to date mainly methods have been proposed based on Latent Dirichlet Allocation (LDA) (e.g., [?]).
With regard to time-dependent tag recommenders, there are two notable approaches. First, the GIRPTM algorithm presented by Zhang et al. [?], which considers the frequency and the temporal usage of a user’s tag assignments. GIRPTM models the temporal tag usage via an exponential distribution. Second, the BLL+C algorithm introduced in [?], which incorporates the activation equation from the cognitive model ACT-R proposed by Anderson et al. [?] and uses a power-law function to mimic the temporal decay of tag reuse. Both approaches imitate tagging by simply taking into account the most popular tags previously assigned to the resource (MP). Kowald et al. [?] have demonstrated that BLL+C outperforms GIRPTM and other well-established algorithms, such as CF, FR and PITF.
Our approach is based on an interplay between micro-level (i.e., the individual level) and macro-level (i.e., the collective level) processes in social tagging systems (e.g., [?]). Among others, micro-level processes (see Sections Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations and Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations) involve categorizing a resource (e.g., modeled as LDA topics) and turning the latent (i.e., non-observable) categorization into manifest (i.e., observable) words or tags [?]. Beyond that, temporal dynamics have turned out to influence the choice of words for a given resource, i.e., recently used tags have a higher probability of being reused than “older” ones [?]. The effect of macro-level structures (see Section Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations) is mediated by the user’s tendency to imitate other users’ tags [?, ?].
In this section we describe how we modeled and implemented these micro-macro dynamics and the corresponding cognitive processes in the form of a novel tag recommender approach. Needless to say, there are also other types of dynamics and processes (e.g., decision making or associative memory activations [?]) that play a role when people choose tags for resources but at the current stage of this thesis, we focus on the mentioned ones.
The basic version of our proposed approach is solely based on human categorization processes [?]. It is termed 3Layers (3L) and is schematically represented in Figure Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations. Similar to Kwantes [?], we apply a mechanism from MINERVA2, a computational theory of human categorization [?], to process the network constituted by the input, hidden and output layers shown in Figure Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations. First, the latent semantic topics of the resource to be tagged are represented in the input layer in the form of vector with features (i.e., the latent topics). The latent semantic topics of the resources have been calculated in advance using Latent Dirichlet Allocation (LDA) [?] based on the given tag distributions of the resources with a number of latent topics of 1000 [?].
Subsequently, is forwarded to the hidden layer, which represents the past posts of a user as a semantic matrix, ( posts latent topics matrix), and an interconnected lexical matrix, ( posts tags matrix). This way, each post of the user is represented by two associated vectors: a semantic vector of latent topics stored in and a verbatim vector of tags stored in . acts as a cue to activate each post () in depending on the cosine-similarity () between both vectors, i.e., and . To transform the resulting similarity values into activation values () and to reduce the influence of very low similarity values, is raised to the power of 3, i.e., (see [?]).
Finally, these activation values are propagated to to activate tags that are associated with highly activated posts in the semantic matrix (circled numbers 2 and 3 in Figure Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations). This step finalizes our basic 3L algorithm and is accomplished via the following equation that yields the value for each of the tags on the output layer:
To refine our approach drawing on temporal dynamics [?], we integrate a time (or recency) component to assign higher activation values to tags that have been used more recently. As shown by Anderson & Schooler [?], the temporal decay of the users’ word choices follows a power-law function. Thus, it can be modeled via the activation equation from the cognitive model ACT-R [?] (see also [?]).
We use a simplified version of this equation to calculate the time component : , where is the timestamp of the most recent post of the user and is the timestamp of the last occurrence of tag in the user’s posts. The exponent accounts for the power-law of temporal decay of the user’s tag choices and is typically set to [?]. Summed up, 3LT, our time-dependent extension of 3L, can be realized using the following equation:
Research on social tagging [?, ?] has shown that a substantial variance in a user’s tag choices can be explained by her tendency to imitate tags previously assigned by other users to a resource. Furthermore, modeling this imitation process allows recommending new tags, i.e., tags that were not used by the current user before. We realize tag imitation by taking into account the most popular tags in the tag assignments of the resource (i.e., MP [?]). This approach was also chosen by other researchers in the field (e.g., [?, ?]). Taken together, the list of recommended tags according to our proposed 3LT+MP approach for the current user and the current resource is given by:
where is the number of assignments of tag for and can be used to inversely weigh the two components. was set to .5 to assign equal weights to individual and collective processes. Furthermore, we normalized the values of and in order to combine them [?].
Our proposed approach presented in this section is fully implemented within our open-source and Java-based TagRec framework [?], which is freely available via our Github Repository111https://github.com/learning-layers/TagRec/. Among others, this framework also contains the baseline algorithms discussed in Section Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations and the evaluation method described in Section Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations.
In this section we describe the methodology to validate our novel approach, including the datasets and evaluation method used.
For reasons of reproducibility, we focused on three well-known folksonomy datasets, that are freely available for scientific purposes. Hence, we utilized datasets from the social tagging and publication sharing system BibSonomy222http://www.kde.cs.uni-kassel.de/bibsonomy/dumps/, the reference management system CiteULike333http://www.citeulike.org/faq/data.adp and the social bookmarking platform Delicious444https://www.uni-koblenz.de/FB4/Institutes/IFI/AGStaab/Research/DataSets/PINTSExperimentsDataSets/.
We excluded all automatically-generated tags from the datasets (e.g., no-tag or bibtex-import) and decapitalized all tags as suggested in related work (e.g., [?]). Furthermore, to reduce the computational effort, we randomly selected 10% of CiteULike and 3% of Delicious user profiles (see also [?]) but did not apply a -core pruning to avoid a biased evaluation (see [?]). The statistics of our used dataset samples are shown in Table Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations.
Dataset BibSonomy 400,983 5,488 346,444 103,503 1,479,970 CiteULike 379,068 8,322 352,343 138,091 1,751,347 Delicious 1,416,151 15,980 931,993 180,084 4,107,107 Table \thetable: Properties of the datasets, where is the number of posts, is the number of users, is the number of resources, is the number of tags and is the number of tag assignments. Figure \thefigure: Precision/Recall plots for BibSonomy, CiteULike and Delicious showing the performance of our approach (3LT+MP) compared to 3L, 3LT and state-of-the-art algorithms for = 1 - 10 recommended tags.
To evaluate our tag recommender approach, we followed a standard procedure in recommender research (e.g., [?]) and split the three datasets into training and test sets. In order to preserve the chronological order of the data, for each user we selected her most recent post (in time) and placed it into the test set. The remaining posts were then used to train the algorithms. This procedure is a promising simulation of a real-world environment, since it predicts the user’s future tagging behavior based on her tagging behavior in the past [?]. To ensure that a minimum amount of tagging “history” is available for training, we focused on users with at least 20 posts. We conducted this evaluation by applying a post-filtering method: while recommendations were calculated based on the entire folksonomy graph, accuracy estimates were computed only on the basis of the filtered user profiles. This resulted in 780 users in the case of BibSonomy, 1,757 in the case of CiteULike and 7,469 in the case of Delicious.
In order to finally quantify the recommender quality and to benchmark our approach against other tag recommendation algorithms mentioned in Section Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations, we compared the top- tags an algorithm suggested for a user-resource pair with the set of relevant tags in the corresponding post in the test set. Based on these comparisons, various evaluation metrics can be calculated, that have originated from information retrieval and recommender systems research (e.g., Precision, Recall, F1-score, MRR, MAP, nDCG) [?, ?]. At the moment, in this work we focus on Precision and Recall for = 1 - 10 recommended tags.
The preliminary results of our evaluation for BibSonomy, CiteULike and Delicious are shown in the three Precision/ Recall plots in Figure Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations. When looking at the results for the baseline algorithms (see Section Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations), it is apparent that, as expected, all personalized algorithms outperform the unpersonalized MP approach. Additionally, the highest accuracy estimates among these baselines are reached by the two time-dependent methods GIRPTM and BLL+C. This emphasizes the importance of taking temporal dynamics into account in tag recommender research. Moreover, BLL+C, which uses a power-law decay function, outperforms GIRPTM, which relies on an exponential decay function.
A comparison of the results for the basic version of our proposed approach 3L, which is solely based on human categorization processes, with 3LT, which further integrates temporal dynamics, shows that, as expected, 3LT provides higher Recall and Precision values than 3L. This further proves that temporal dynamics play an important role in tag prediction tasks. Additionally, the complete version of our approach 3LT+MP, which also utilizes macro-level processes in the form of imitating the most popular tags assigned to the target resource by other users (MP), provides better results than 3LT. This is especially true in the case of Delicious, where numerous tags from other users are available for imitation.
Apart from that, and even more important, our proposed approach 3LT+MP also outperforms the current state-of-the-art algorithms mentioned in Section Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations in all three settings. This includes the time-based GIRPTM and BLL+C approaches, that also utilize temporal dynamics and tag imitation but ignore categorization, which further reveals the importance of all three examined types of cognitive processes. In brief, the results of our experiments not only show that an interplay between individual micro-level and collective macro-level processes can be used to develop an effective tag recommender approach but also that such an approach can outperform current state-of-the-art algorithms.
This thesis aims at modeling the cognitive micro- and macro-level processes that play a role when people assign tags to resources. At the current stage of this thesis, this is achieved by the development and evaluation of a novel tag recommender termed 3LT+MP. The preliminary results of our evaluation for datasets gathered from BibSonomy, CiteULike and Delicious show that 3LT+MP can not only outperform current state-of-the-art algorithms but also provides higher accuracy estimates than 3L and 3LT that implement only some of the examined processes. These results corroborate our hypothesis from Section Modeling Cognitive Processes in Social Tagging to Improve Tag Recommendations, that our theory-driven tag recommender can not only improve recommender accuracy in general but also can help to better understand the underlying cognitive processes. Additionally, we introduce an open-source tag recommender benchmarking framework termed TagRec (see also [?]), which contains not only our proposed approach but also standardized baseline algorithms and evaluation methods.
At present, one limitation of this thesis is that we only focus on one cognitive model, namely MINERVA2, to account for the individual micro-level processes in social tagging systems. With that regard, in the future we would like to use and evaluate also another cognitive model (e.g., ACT-R) to realize these (and also additional) processes. We also plan to further improve our approach by investigating content data of the resources (e.g., title and description), which could highly expand the set of possible tags that can be recommended. Moreover, we will evaluate our proposed approach in terms of not only recommender accuracy but also computational costs in order to validate the efficiency of our theory-driven approach. We also plan to integrate our approach into a real online social tagging system, since only then it will be possible to examine our tag recommender’s performance with regard to user acceptance. Finally, we would like to adapt the idea of our approach also for other problems in the recommender systems domain, such as the recommendation of resources, topics and users.
Acknowledgments: The author would like to thank Elisabeth Lex, Tobias Ley, Paul Seitlinger and Christoph Trattner for valuable discussions related to this thesis. This work is funded by the Know-Center and the EU-funded project Learning Layers (Grant Agreement 318209). The Know-Center is funded within the Austrian COMET Program - Competence Centers for Excellent Technologies - under the auspices of the Austrian Ministry of Transport, Innovation and Technology, the Austrian Ministry of Economics and Labor and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency (FFG).
-  J. R. Anderson, M. D. Byrne, S. Douglass, C. Lebiere, and Y. Qin. An integrated theory of the mind. Psychological Review, 111(4):1036–1050, 2004.
-  J. R. Anderson and L. J. Schooler. Reflections of the environment in memory. Psychological Science, 2(6):396–408, 1991.
-  P. G. Campos, F. Díez, and I. Cantador. Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols. User Modeling and User-Adapted Interaction, pages 1–53, 2013.
-  K. Dellschaft and S. Staab. Measuring the influence of tag recommenders on the indexing quality in tagging systems. In Proc. of HT’12, pages 73–82, New York, NY, USA, 2012. ACM.
-  S. Doerfel and R. Jäschke. An analysis of tag-recommender evaluation procedures. In Proc. of Recsys’13, pages 343–346, New York, NY, USA, 2013. ACM.
-  F. Floeck, J. Putzke, S. Steinfels, K. Fischbach, and D. Schoder. Imitation and quality of tags in social bookmarking systems–collective intelligence leading to folksonomies. In On collective intelligence, pages 75–91. Springer, 2011.
-  W.-T. Fu. The microstructures of social tagging: a rational model. In Proc. of CSCW’08, pages 229–238. ACM, 2008.
-  J. Gemmell, T. Schimoler, M. Ramezani, L. Christiansen, and B. Mobasher. Improving folkrank with item-based collaborative filtering. Recommender Systems & the Social Web, 2009.
-  D. L. Hintzman. Minerva 2: A simulation model of human memory. Behavior Research Methods, Instruments, & Computers, 16(2):96–101, 1984.
-  A. Hotho, R. Jäschke, C. Schmitz, and G. Stumme. Information retrieval in folksonomies: Search and ranking. In The semantic web: research and applications, pages 411–426. Springer, 2006.
-  R. Jäschke, L. Marinho, A. Hotho, L. Schmidt-Thieme, and G. Stumme. Tag recommendations in folksonomies. In Proc. of PKDD’07, pages 506–514. Springer, 2007.
-  R. Jäschke, L. Marinho, A. Hotho, L. Schmidt-Thieme, and G. Stumme. Tag recommendations in social bookmarking systems. Ai Communications, 21(4):231–247, 2008.
-  W. Kintsch and P. Mangalath. The construction of meaning. Topics in Cognitive Science, 3(2):346–370, 2011.
-  C. Körner, D. Benz, A. Hotho, M. Strohmaier, and G. Stumme. Stop thinking, start tagging: tag semantics emerge from collaborative verbosity. In Proc. of WWW’10, WWW ’10, pages 521–530, New York, NY, USA, 2010. ACM.
-  D. Kowald, S. Kopeinik, P. Seitlinger, T. Ley, D. Albert, and C. Trattner. Refining frequency-based tag reuse predictions by means of time and semantic context. In Mining, Modeling, and Recommending Things’ in Social Media, pages 55–74. Springer, 2015.
-  D. Kowald, E. Lacic, and C. Trattner. Tagrec: Towards a standardized tag recommender benchmarking framework. In Proc. of HT’14, New York, NY, USA, 2014. ACM.
-  R. Krestel, P. Fankhauser, and W. Nejdl. Latent dirichlet allocation for tag recommendation. In Proc. of Recsys’09, pages 61–68. ACM, 2009.
-  P. J. Kwantes. Using context to build semantics. Psychonomic Bulletin & Review, 12(4):703–710, 2005.
-  M. Lipczak. Hybrid Tag Recommendation in Collaborative Tagging Systems. PhD thesis, Dalhousie University, 2012.
-  L. B. Marinho and L. Schmidt-Thieme. Collaborative tag recommendations. In Data Analysis, Machine Learning and Applications, pages 533–540. Springer, 2008.
-  S. M. Polyn, K. A. Norman, and M. J. Kahana. A context maintenance and retrieval model of organizational processes in free recall. Psychological review, 116(1):129, 2009.
-  S. Rendle. Factorization machines. In Proc. of ICDM’10, pages 995–1000. IEEE, 2010.
-  S. Rendle and L. Schmidt-Thieme. Pairwise interaction tensor factorization for personalized tag recommendation. In Proc. of WSDM’10, pages 81–90, New York, NY, USA, 2010. ACM.
-  P. Seitlinger, D. Kowald, C. Trattner, and T. Ley. Recommending tags with a model of human categorization. In Proc. of CIKM’13, pages 2381–2386, New York, NY, USA, 2013. ACM.
-  P. Seitlinger and T. Ley. Implicit imitation in social tagging: familiarity and semantic reconstruction. In Proc. of CHI’12, pages 1631–1640, New York, NY, USA, 2012. ACM.
-  L. Zhang, J. Tang, and M. Zhang. Integrating temporal usage pattern into personalized tag prediction. In Web Technologies and Applications, pages 354–365. Springer, 2012.