Dynamic Multi-Relational Chinese Restaurant Process
for Analyzing Influences on Users in Social Media
We study the problem of analyzing influence of various factors affecting individual messages posted in social media. The problem is challenging because of various types of influences propagating through the social media network that act simultaneously on any user. Additionally, the topic composition of the influencing factors and the susceptibility of users to these influences evolve over time. This problem has not studied before, and off-the-shelf models are unsuitable for this purpose. To capture the complex interplay of these various factors, we propose a new non-parametric model called the Dynamic Multi-Relational Chinese Restaurant Process. This accounts for the user network for data generation and also allows the parameters to evolve over time. Designing inference algorithms for this model suited for large scale social-media data is another challenge. To this end, we propose a scalable and multi-threaded inference algorithm based on online Gibbs Sampling. Extensive evaluations on large-scale Twitter and Facebook data show that the extracted topics when applied to authorship and commenting prediction outperform state-of-the-art baselines. More importantly, our model produces valuable insights on topic trends and user personality trends, beyond the capability of existing approaches.
Dynamic Multi-Relational Chinese Restaurant Process
for Analyzing Influences on Users in Social Media
|IBM Research - India|
|Indian Institute of Science|
|Indian Institute of Science|
Social networking sites, such as Twitter, Facebook, MySpace etc, have proven to be extremely popular platforms for users for sharing views and opinions using short posts111Our work focuses on such micro-blogging sites.. Understanding and analyzing topics in social media has become immensely important for a variety of stakeholders, such as companies advertising products and identifying customer segments, social scientists and national security agencies, leading to a surge in research interest [?, ?, ?, ?, ?, ?, ?]. There are two major distinguishing features of social media data. First, users are influenced by a variety of factors when posting messages. The four major factors have been identified to be personal preferences of the users, their immediate network of friends on the network, geographic or regional issues and events and world-wide happenings [?]. While all factors typically affect all users, different users have different ‘personalities’, in that they are influenced by these factors in different degrees. Secondly, social media data is inherently dynamic. Topics follow different ‘trends’; individual interests of influential users, or issues starting off within a small network of friends, sometimes lead to global upheavals, while other enjoying global popularity are slowly relegated to individual favorites. Similarly, user personalities also evolve and show trends over a geography or sub-network.
Owing to these multitude of factors, and the intrinsic interplay between them, analysis of social media data has been a major challenge. Most existing approaches fall short of addressing the problem in its entirety, and only model isolated factors and their interactions [?, ?, ?]. A major hurdle for sophisticated models is the scale of the data; the associated inference algorithms need to be scalable and efficient.
In this paper, we propose a non-parametric probabilistic approach for analyzing social media data. Specifically, we first propose an augmentation of the Chinese Restaurant Process[?], called the Multi-Relational Chinese Restaurant Process (MRelCRP), that accommodates users and multiple relationships over them, for assigning topics to posts. By using relationships, the MRelCRP defines a new and different family of distributions compared to the traditional non-parametric processes such as the Dirichlet Process [?], and its hierarchical versions [?]. We further propose a dynamic version of the MRelCRP (D-MRelCRP) that allows temporal evolution of the model parameters to capture topic and personality trends. The rich interactions of various parameters in the model are able to capture the various interplays in social media data. Crucially, we propose an efficient and multi-threaded algorithm, based on online collapsed Gibbs sampling, for performing learning and inference for Dynamic MRelCRPs.
We evaluate the proposed model on two large scale datasets. The first dataset consists of 360 million posts from Twitter. The second dataset consists of 300K posts from Facebook. We demonstrate both qualitatively and quantitatively the goodness of the topics discovered by our model. When employed for predicting authorship and user activity, models using these topics significantly outperform state-of-the-art baselines. More importantly, our model is able to discover interesting and insightful topic and personality trends. For example, our analysis shows that users posts are mostly influenced by personal preferences, rather than global, regional or social-network factors, except in times of major world events, when users become swayed by global influences at the cost of personal preferences. We are not aware of any existing model that can perform such a wide array of analyses effectively on social media data.
The rest of the paper is structured as follows. We discuss related work on social media analysis and topic models in Section Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media. We describe our proposed model in Section Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media and the associated inference algorithm in Section Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media. Experimental results are presented in Section Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media and we conclude in Section Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media.
Here, we discuss our contributions in the light of related work in non-parametric probabilistic modeling and social media analysis.
Non-parametric models: The Dirichlet Process (DP) [?] is a prior over a countably infinite set of atoms, and is popularly used as a prior for mixture models (DP Mixture Model) in applications, where the number of clusters is difficult to provide as a parameter. The Chinese Restaurant Process [?] provides a generative description for the Dirichlet Process, and is useful for designing sampling algorithms for DP mixture models. The distributions defined by these models are exchangeable, in that different permutations of the data are equally probable.
CRPs have been extended to handle distances and relations. The distance dependent CRP (DD-CRP) [?] takes into account a distance matrix over the input data points. Unlike the DP-HDP family, this results in a distribution that is not exchangeable, which is a feature of many applications. In comparison, the RelCRP uses an additional non-unique label for each data point, and a general graph defined over them. The resultant distribution is exchangeable. As such, the DD-CRP and the RelCRP define different families of distributions, and one cannot be represented by the other. There is different body of work that use CRPs for modeling relations [?] and their dynamic evolution [?]. These are unsuitable for our current application, where we do topical analysis of the data points, based on relations between their (user) labels.
Many applications require multiple coupled Dirichlet Processes. The Hierarchical Dirichlet Process (HDP) [?] is one way to introduce coupling using a two level structure. The HDP can be useful, for example, for extending the popular Latent Dirichlet Allocation (LDA) model [?], for countably infinite number of topics [?]. The HDP can be equivalently represented by an extension of the CRP called the Chinese Restaurant Franchise (CRF) [?]. Just as the CRF introduces coupling between CRPs, the MultiRelCRP introduces coupling between RelCRPs. However, the nature of the coupling in the MultiRelCRP can be much richer, depending on the relationships, as we explain in Section Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media.
Temporal evolution has been addressed in the context of non-parametric models [?, ?, ?], where the parameters of the the corresponding static model become functions of time. Some of the approaches are amenable to scalable inference, while others are not. For the Dynamic MRelCRP, we use the dynamic evolution of the parameters proposed in the context of Recurrent CRF [?, ?], because of the scalability of the associated inference problem. Note, however, that the similarity between the Recurrent CRF and the Dynamic-MRelCRP is only in the temporal evolution of model parameters. The static model is a HDP/CRF, as compared to the RelCRP in our case.
Social Media Analysis: There has been a surge of literature on problems involving social media content. Work has mostly been focused around (a) Content analysis on microblogs, (b) Inferring user interests and (c) Mining patterns of variation on social media, as we discuss below.
(a) Most content analysis papers [?] use standard topic models such as LDA [?] or basic metrics like tf-idf. Focusing on the specific content of miroblogs, Ramage et. al. [?] proposed an LDA variant that accounts for hashtags in content analysis. One problem with this approach is that hastags are not general features of social media data, and are often unreliable. There is little modeling work that takes into account the rich features of social media such as network, geography, etc.
(b) In the context of microblogging sites, content recommendation approaches [?, ?, ?] assessing user interests based on their activities. Recently, Wen et. al [?] have proposed an approach which studies the influence of the network on users. Ahmed et. al. [?] model the dynamics of user interest and also the account generic popularity of a particular item, but do not consider the influence of various external factors like network of users and geography. Thus most of related work either deals with the influence of a single factor or user preferences.
(c) Yang et. al. [?] made one of the first attempts at understanding the temporal evolution of patterns on social networking sites like Twitter. Apart from temporal dynamics, study of such patterns with respect to geography and other factors has not been explored for content on social networking sites.
In this section, we describe our Dynamic Multi-Relational Chinese Restaurant Process model, which we employ to study the interplay of world-wide, geographic, network and user specific factors, and their dynamics, in social media. We build up our model in steps, first describing the static Relational Chinese Restaurant Process, then incorporating multiple relations, and finally adding temporal evolution to it. In our application, the basic task is to associate topics with individual posts or tweets. The topics correspond to concepts such as ‘movies’, ‘sports’, ‘politics’ etc. Unlike topic models such as LDA [?], which associate a distribution over topics with each document, we assume that each post, considering its shortness, corresponds to exactly one topic. This makes the model simpler and the associated inference algorithm more efficient and scalable.
The Dirichlet Process [?] has become a popular non-parametric prior in clustering applications, where the number of clusters is not needed to be specified apriori, but instead grows with the data size. The Chinese Restaurant Process (CRP) [?] provides a fanciful description of the Dirichlet Process, by imagining data points as customers being seated at tables, which represent clusters, as they enter the restaurant. Let denote the data point, or post in our case, and denote the cluster (or table assignment) for the post. Then, given the assignments of the first customers to tables, the conditional distribution for the table assignment of the customer is given by the CRP as follows:
where is the number of customers already assigned to table . Note that this a ‘rich gets richer’ model, where tables with more customers have a higher probability of getting new customers, but new tables also have a non-zero probability () of getting customers.
When each table is associated with a (topic) distribution, with parameters drawn iid from an appropriate base distribution , the CRP can be used as a prior for mixture distributions. Once the customer is seated at a table , the corresponding data item can be drawn independently from the distribution associated with the table. For a generative model for posts, each topic distribution can be a multinomial over the post vocabulary, so that each word of the post is generated independently from that topic, and the base distribution can be chosen to be a Dirichlet , since it is conjugate to the multinomial.
Though defined as a sequential process, the CRP mixture model can be easily shown to be exchangeable, which means that all permutations of observed data have the same probability under the model. The Chinese Restaurant Process has been widely used in generative models for different applications [?, ?, ?]. However, it is unsuitable for social media data, since it ignores a fundamental aspect — the social network over users who generate the content. Specifically, each post has associated with it a user variable , that takes values from a finite set of users . These users are further connected by a network of relationships. To accommodate this, we augment the Chinese Restaurant Process to handle such relationships.
In the Relational Chinese Restaurant Process (RelCRP), each customer (data point) is associated with a label . In the context of social media data, we will refer to each element in as a user, and say that each data point or post has a user label. In addition, we have a relationship , such that each element is a subset of . We can imagine as defining a set of hyper-edges over elements in . Note that we do not fix the candinality of the elements in . We will see the need for this shortly. Using , we can define the neighbors of an element as all other elements that share a relation with : .
Given the additional labels and the relationship , the conditional distribution of the table assigned to the customer is defined in RelCRP as follows:
where is the number of neighbors of in already assigned to table .
Let us now look at some example uses of the RelCRP in the context of social media data. We start from the trivial case, where the RelCRP reduces to the CRP, and then move on to more interesting ones.
Influence of World-wide Factors: Very commonly users are influenced by globally popular events or entities when choosing a post topic. For example, users who are not fans of Michael Jackson tweeted on this topic in the event of his unexpected death. This can be captured in the RelCRP by associating a unique label with each data point, along with a ‘complete’ relationship , that contains a single relation (hyper-edge) over all . In this case, Equation (2) reduces to:
where is the number posts by all users (which is the neighbor set of ) already assigned to table (topic) . Note that this is the same as Equation (1). Thus the RelCRP is able to recover the traditional CRP, using unqiue data labels and a ‘complete relationship’.
User’s Personal Preferences: One of the most significant factors influencing the content of a post is the preference of the associated user. A specific user may be more interested in ‘movies’ that in ‘sports’ or ‘politics’. Evidence of this can be found in the topics of this user’s earlier posts — a user is more likely to post on a topic that she has used more frequently. To capture this in the RelCRP, we set to be user identifier, and simply construct an empty relation over . Given , Equation (2) reduces to:
where is the number of posts by user (who is her only neighbor) already been assigned to table . Note that even the case of the empty relation cannot be captured by the traditional CRP.
Influence of Friend Network: A user is often influenced by the post topics of her friends. To capture this, as before, we set the label of the post to be the user id, and construct based on the friendship network: for each follower or friendship relation between users and , we add a tuple to . Note that in this case all elements of have cardinality . Given , has the same form as Equation (2), where is now the number of times followees or friends of user have posted on topic .
Influence of Geography: As a final example, a user’s posts may also be influenced by geographic trends. For instance, an national election draws a lot of attention from citizens of that country. This can be captured by the RelCRP, by again associating labels with user id’s, and constructing to capture geographic locations: adding a hyper-edge in over all users in a specific country. Typically, the geographic location can be known from the profile of the user, and we assume such a construction to be possible. Note that in this case every edge has a different cardinality, and most will be extremely large. Interestingly, the RelCRP does not require maintaining the explicit relations, but only simple statistics over them. Given , again takes the form of Equation (2), where is now the number of times users in the same geography as user have posted on topic .
Thus, the RelCRP can be used to capture the different posting patterns in social media within a single framework, in a way that the traditional CRP cannot. Just like the CRP, however, the RelCRP can be used to define a mixture model by associating a topic with each table. It can be shown that the resultant distribution remains exchangeable.
We have seen that the RelCRP is able to model the individual effect of the world-wide factors, user preferences, friend network and geographic factors when the labels and relationships are appropriately defined. However, in reality, all of these influences act simultaneously on any user, and their interplay determines the content of each of her posts. Further, this aggregate influence pattern is user-specific. For example, different users are affected differently by the same combination of world and geographic events. We now present the Multi-Relational Chinese Restaurant Process (MRelCRP) that captures such aggregate influences using multiple relations defined over the same user labels.
The MRelCRP is characterized by a set of labels , along with relations defined over . With the data point (post), we associate an additional variable , which takes values from , indicating the relationship that influenced this data point. This depends on the associated label (user) . For each label , there is a -dimensional multinomial distribution . Each is assumed to be generated iid from a Dirichlet . We interpret as the probability of label being influenced by the relationship , i.e. . We may imagine as reflecting the ‘personality’ of user . Given these parameters, and the assignment of the first posts to topics, the MRelCRP defines the conditional distribution of the topic assignment of the post with label as follows:
which is a mixture of individual RelCRP distributions, defined according to Equation (2). This can be interpreted as first selecting a particular RelCRP from a prior distribution specific to the label , and then selecting a table using the selected RelCRP.
The aggregated influences in the post generation process can now be captured by the MRelCRP framework, by considering the set of relationships . A -dimensional influence factor is sampled for each user from . This can be imagined to reflect the personality of the user. Then, for each post, a topic is selected for it, in two steps, using Equation (5). Finally, the individual words in the post are sampled iid from this selected topic. This is described in Table Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media.
|1. For each topic|
|2. For each user|
|3. For each post|
|c. For the each word of post|
Couplings in the MRelCRP: It is important to observe the coupling that the MRelCRP creates between different RelCRPs, that helps capture the interplay of various factors for social media. (a) First, we analyze the dependencies for a single relationship . Observe that there are RelCRPs, one for each user (label). However, all of these RelCRPs need not be distinct. This depends on the nature of the relationship. For example, in the setting above, is a ‘complete’ relationship. As a consequence, the neighbor sets are the same for all users, and the world-wide RelCRP is identical for all users. For the geographic relationship , since the individual relations do not overlap, the geographic RelCRP is identical for all users from the same country. This creates one type of dependence across users. In contrast, for the friend relationship , in general distinct users have different sets of friends, and their RelCRPs are distinct. However, they are still coupled, since the underlying topics are the same, and a post by user on topic increases the count for all friends of . Thus, for all of these three relationships, evidence can flow between users over hyper-edges in the relationship. Finally, for the user preference relationship , the RelCRP for each user is distinct, and there are no dependencies. (b) Now, we analyze the new dependencies that are created when multiple relationships are coupled in the MRelCRP. Observe that for relationships, there is a total of RelCRPs, for each user, but all of these need not be distinct, as above. The distinct RelCRPs for each user are now coupled; evidence can flow between relationships through the users. In the context of social media, this leads to interplay between world-wide, geographic, network factors and personal preferences.
The two key distinguishing aspects of social media data are the network structure, and the dynamic nature of the topics and user influence patterns or personalities. The MRelCRP captures the network aspect, but falls short on the second count. Before extending our model, we first enumerate the different aspects of the data that evolve with time. (a) The number of topics changes as old topics die out and new topics are born. (b) Popularity of topics change, world-wide, in specific geographies, sub-networks and in the preferences of individual users. We call these topic trends. (c) User personalities change, and they become more or less susceptible to being influenced by world-wide, geographic, network and individual preferences. (d) Existing topics also evolve as new words enter the vocabulary and existing words go out of fashion. We now propose the Dynamic Multi-relational Chinese Restaurant Process (D-MRelCRP) that accounts for all of these temporal evolutions. In reality, the number of users also change over time and the network grows or shrinks, but we do not consider this aspect in our current model.
We assume that the data has been segmented into epochs, or in other words, each data element is labeled with a time-stamp that takes values from . In practice, epochs may be appropriately defined (eg. hour, day, week, etc) depending on the application. The Dynamic MRelCRP consists of one MRelCRP for each epoch. We introduce dependencies between the parameters of the MRelCRPs across epochs to capture the different aspects of temporal evolution, as we describe next. We use additional subscripts on parameters to indicate epochs.
Note that individual RelCRP’s naturally allow the number of topics to change. We do not need to address this separately in the D-MRelCRP.
Different topics have different trends, in that some start out being popular in certain geographies, to being global hits. Some others may start as preferences of influential individual users and evolve to regional or world favorites. To capture this, topic popularities in our model need to change over epochs. Since we have modeled popularity of topics using counts, to make this approach dynamic, topic counts of specific epochs are made dependent on those of earlier epochs, following the approach of [?]. We extend the basic RelCRP conditional distribution (Equation (2)) with epoch indices as follows:
where is the number of neighbors of in already assigned to table in the epoch, while captures the historical counts in recent epochs, with exponentially decaying weights, as follows:
where is the decay factor. The MRelCRP for epoch is now defined using a mixture of such RelCRP conditionals as in Equation (5).
It is natural for user personalities to be time dependent as well. A user may become more susceptible to the influence of her friends and deviate from her earlier personal preferences. In the MRelCRP framework, this corresponds to the mixture distribution for each user being a function of the epoch. Recall that each is sampled iid from a Dirichlet prior . We introduce a temporal dependence by adding a dynamic component to the prior parameter, in the spirit of [?], as follows:
for , and being the number of times user was influenced by relationship in epoch .
The topic-word distributions also evolve with time. To capture this, we again introduce a temporal dependence in the prior distribution. Specifically, each topic distribution is now sampled from . The element of dynamic component depends on how frequently the word in the vocabulary has been historically observed under topic until epoch :
where corresponds to the number of times word is associated with the topic in epoch .
These three dynamic dependences introduced between the parameters of the MRelCRPs corresponding to different epochs, defines our complete D-MRelCRP.
In this section, we discuss the key challenges in performing inference for the proposed D-MRelCRP model, and present our inference algorithm addressing these challenges. The inference problem involves determining the posterior distribution over the two latent variables variables, the topic label and the influence variable , for all posts in all epochs . The parameter estimation problem involves finding the posterior distribution of the model parameters, the topic distributions and the personalities of the users. The two problems are coupled, and solving them exactly is intractable [?]. We resort to approximate techniques based on collapsed Gibbs sampling. However, the traditional approach [?], where the topic and influence labels of each post are repeatedly sampled until convergence from the conditional distributions given all other labels, is infeasible for us given the size of the data. Even Sequential Monte Carlo methods [?], that rejuvenate a few older labels, are infeasible. We adopt the online algorithm [?], which was proposed for parametric models, and modify it appropriately for our model. In this approach, earlier labels are not revisited. This allows the algorithm to scale, at the expense of sub-optimal estimates at the beginning, and is also concordant with the online nature of social media data [?]. Before describing the details, we first describe the conditional distributions that are required by the algorithm.
Conditional Distributions: In the online setting, the distribution for the influence factor for the post is conditioned on the topic and influence labels of all earlier posts, their user labels and the content of the current post. For the Dynamic MRelCRP, this looks as follows:
where, , and the counts are as defined in Section Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media.
The conditional distribution for topic label , additionally conditioned on influence factor , is given by:
where, corresponds to the word of the vocabulary, corresponds to the number of times word in the vocabulary is associated with topic during epoch . Note that online inference, the counts in the equations above correspond to the data instances (posts) which have arrived before the instance. Also, the conditional distributions for the static model (MRelCRP) can be obtained by removing the historical counts in the above expression, specifically by setting in Equation (Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media), and in Equation (Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media). Similarly, the conditional distribution for RelCRP, which has a single relationship can be obtained as a special case of the MRelCRP, by taking counts with respect to .
Parallel Inference Algorithm: A straight-forward online algorithm, that makes a single sequential pass over the data, is infeasible considering the scale of social media data. This necessitates a parallelized inference algorithm. Sampling based parallel inference algorithms for hierarchical bayesian models has received attention in the literature [?, ?]. These approaches split data across threads or processors, execute Gibbs iteration on them independently, and finally, consolidates labels across threads asynchronously at the end of each iteration. In contrast, parallelization of our algorithm results in independent, online updates in each thread. Additionally, D-MRelCRP being a non-parametric model, new topics are created by each thread, and in the absence of repeated Gibbs iterations, are not sufficiently consolidated. As a result, we require a synchronous architecture, where all new topics are explicitly consolidated by a master thread at the end of each iteration. Our multi-threaded inference algorithm is described in Table Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media.
|1. Read first N posts|
|2. Iterate t times|
|3. For each post , sample|
|4. Update joint counts|
|5. Iterate until no new post|
|6. Read next N posts|
|7. For child thread j=1 to K|
|8. Send posts j(N/K) + (j+1)N/K, joint counts|
|9. Receive labels for N/K posts|
|10. Wait until child threads complete|
|11. Iterate t times|
|12. For each post with new label , sample|
|13. Update joint counts|
|1. Sleep until invoked by Master Thread|
|2. Receive N/K posts, joint counts|
|3. Iterate t times|
|4. For each post , sample|
|5. Return N/K labels to Master Thread|
After an initial batch phase (master thread: steps 1-4), the algorithm iterates over three phases: data access (master thread: step 6), computation (child threads: steps 2-5) and synchronization (master thread: steps 8-9, 11-13). The initial batch phase is necessary to prevent creation of too many new topics at the beginning by different child threads. Note that the computation phase happens in parallel across the child threads. Each child thread creates multiple new topics, whose counts are maintained locally. These counts are passed back to the master thread, along with other counts at the end of the computation phase. After receiving back labels from all child threads, the master thread re-samples labels for all posts assigned new topics by child threads. This helps in the consolidation of new topics, many of which may otherwise be quite similar. The iterations continue until all posts have been processed.
In this section we discuss in detail the experiments that we carried out using the proposed D-MRelCRP model on multiple large real social media datasets.
We evaluate the following aspects of the model:
(a) Model goodness: Ability to explain unseen data
(b) Topics and topic labels: Our inference and learning algorithms assign a topic to each post, and also finds a distribution over words for each topic. We evaluate both aspects qualitatively and quantitatively.
(c) User personalities and their trends: The major distinctive feature of our model is the influence label associated with each post. Using this label, we are able to estimate the user personalities, or the susceptibility of the user to various influencing factors, and their dynamics. We discuss various insights that we were able to find from personality trends.
(d) Scalability: One of the main strengths of our inference algorithm is the ability to scale to hundreds of millions of data samples. We evaluate how the running time of our multi-threaded implementation scales with data size.
(e) Relative importance of factors: The MRelCRP and D-MRelCRP models are able to combine together various influence factors and their dynamics. We analyze the usefulness of the different factors for social media analysis.
We would like to point out that no other single model is able to perform such a wide array of tasks in social media analysis. Wherever possible, we make use of available ground truth or surrogates of it for quantitative evaluation and compare against best available baselines. However, as regards our main contribution — discovering user personality trends — there does not exist any existing algorithm that can perform this.
Datasets: We carried out all our experiments on two different datasets: (1) Twitter: a collection of 360 million tweets crawled between June and December 2009, and (2) Facebook: a collection of about 300,000 posts obtained by extracting feeds from publicly available profiles over a span of three months.
Default Parameter Settings: The hyper-parameters of our online Gibbs Sampler were initialized as : = 0.1/K+1 and = 0.1.
Baselines: We compare the performance of our models against the following state of the art models that have been shown to be effective for analyzing microblogs. (a) Latent Dirichlet Allocation (LDA) [?] (b) Labeled LDA [?], and (c) Timeline [?]. Labeled LDA is not very generally applicable since it makes use of hashtags assigned to the posts to identify topic labels. While this meta-information is available on Twitter, Facebook does not support it. The Timeline model is the closest to our model in that it is a non-parametric topic model that also captures topic dynamics.
Goodness of a model is evaluated by how well it is able to fit previously unseen data.
Perplexity is a commonly used to measure generalization ability of topic models [?].
It is defined as
the inverse of the geometric mean per-word likelihood:
,where is the number of words in the post in the held-out test set , and is its log-likelihood. Lower values of perplexity indicate better generalization ability.
We consider two different datasets for this experiment. For each model under consideration, we first train it on Twitter data, and then consider as test set a sample of 8 million tweets from the last one month in our dataset. Similarly, each model is trained on Facebook data, and evaluated on a sample of 40K posts from the last month’s activity. Perplexities of various models are recorded in Table Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media. Note that unlike our model, LDA requires the number of topics to be specified. We set it to the average number of topics discovered by our model across epochs. Labeled-LDA cannot be applied for Facebook data, since it requires hashtags. It can be seen that D-MRelCRP has the least perplexity in both the cases.Among baselines, Timeline has the best perplexity. This demonstrates that capturing both temporal evolution and relationships is important for explaining future data.
|Perplexity||Clustering Acc. (Twitter)|
The D-MRelCRP model assigns a topic label to each post, indicating its category, and also finds a distribution over vocabulary words for each unique topic label, indicating the semantics of the topic. Our hypothesis is that by modeling the different influences on the users, D-MRelCRP is able to better identify topics. To evaluate this, we check topic quality in different ways. We directly evaluate the topic labels of posts by comparing against a reasonable gold-standard. Then, we indirectly evaluate the topic labels of posts by using them as features in two prediction tasks. Finally, we identify significant topic trends and compare them qualitatively with world knowledge. We provide more details on these three evaluations next.
In our proposed models, there is a single topic label associated with each post. This results in a hard clustering of the posts according to topics. Therefore, one way to evaluate the topic assignment quality is to evaluate the clustering accuracy. Gold standard clusters of posts is typically hard to obtain. As an alternative, in the case of Twitter data, we consider hastags as cluster indicators. Since it is well known that hashtags are often poor indicators of post clusters, we consider only a few authoritative hasgtags as follows. We collected posts with hashtags corresponding to highly specific topics, such as #NIPS2009, #ICML2009, #bollywood, #hollywood, #www2009 etc. We consider this as the test set with gold-standard labels for evaluation.
We use three standard metrics to evaluate clustering accuracy - Normalized mutual information, Rand index and F-measure. In Table Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media, we record the performance of our D-MRelCRP model, and those of the three baselines on the Twitter dataset. Not surprisingly, labeled LDA correctly identifies the clusters all the time, by virtue of taking hashtags as inputs. DMRelCRP comes close, in spite not using knowledge of hashtags at all, and performs better than all other models across all the three evaluation metrics. Further, on closer inspection, we found that the Labeled LDA clustering is not as good as the numbers indicate, and the two proposed models are often better. For example, DMRelCRP splits the 3K posts corresponding to the #movies hashtag into two topics, and separates out posts originating from India. Comparison using KL-divergence shows this topic to be very similar to the #bollywood hashtag. The #sports hashtag shows a similar split. Such fine-grained distinction is not possible for Labeled LDA, or Timeline, which do not capture geographic or other influencing factors.
Since the cluster gold-standards for posts are unreliable for Twitter, and unavailable for Facebook, we additionally perform the following indirect evaluation of topic assignment to posts. Topic labels are commonly used as reliable low-dimensional features for learning classifiers [?]. We use the topic labels for posts for two representative prediction tasks in social media with reliable gold-standards: predicting post authorship and predicting commenting activity.
|Authorship Prediction||Commenting Prediction|
Predicting Authorship: Given a post and user , this task is to predict if user is the author of post . We construct a Twitter test set having 20M tweets from the last 15 days, and a Facebook test set having 40K posts from the last one month. For each user, we create training sets for Twitter and Facebook by including as positives all posts authored by that user, and as negatives a equal-sized random sample from posts authored by other users in the recent past. As features, we use the topic label of the post inferred by a specific algorithm, and the time-stamp. We use k-nearest neighbor classifier (k=5), where we consider minimum distance between the post posts authored by , with KL-divergence as topic distance and number of in-between days as the time difference.
Predicting Commenting Activity: Given a post by some user , the task is to predict if user comments on the post. We similarly construct test and training sets from Twitter and Facebook. As an additional feature, we consider the number of past interactions between users and . We again use a classifier (k=5) for prediction.
The accuracies for both prediction tasks for different algorithms are recorded in Table Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media. It can be seen that DMRelCRP performs significantly better on both datasets. The standard topic model baselines, and also Timeline, do not perform very well on this task. This shows the usefulness of topics inferred by considering both relationships and temporal evolution. Labeled-LDA performs better than LDA, but in spite of using hash-tags, is significantly outperformed by our proposed approaches.
In summary, the topics inferred using our model are significantly more useful for prediction tasks involving users and posts compared to state of the art topic models.
The inferred topic label for each post, in conjunction with the user label, can be used to identify various topic trends.
From the joint counts of the number of posts by user at time epoch on topic , the probability of user posting on topic at epoch can be estimated by normalization.
By subsequent aggregation over subsets of users, popularities of different topics across different user segments can be plotted against epochs.
When a particular topic dominates over all others in an epoch, we flag that topics as a major event, and analyze it using the dominant words in the topic distribution .
We were able to identify several break-out events using DMRelCRP topics labels, as we describe below.
World-wide Events: World wide popularity of a topic at epoch is estimated by aggregating over all users . The major world-wide events discovered by D-MRelCRP include the demise of Michael Jackson (Jun 30) (top words: mj, michael, dead, singer, jackson, pop), the Fifa World Cup (Sep 15-30) (football, soccer, fifa, worldcup) and the launch of Google Wave (Dec 1-15) (wave, invite, google, launching)
Geographic Events: The popularity of a topic in a specific geography at epoch is estimated by aggregating only over all users in that geography. Jeff Goldblum’s demise (Jul 1-15) (death, jeff, actor, goldblum, dies, end, era) was detected as major event in Australia and the UK.
We were able to verify these world-wide and geographic major events using Google Insights222http://www.google.com/insights/search/. The words from the specific topics appeared in the top searches during these specific intervals, world-wide or in the specific geographies. We were similarly able to find major events for small networks of users (e.g. official page for Microsoft on Twitter @MSFTNews and its followed pages) and for important individual users (such as @ICML2009). In summary, DMRelCRP enables us to discover interesting topic trends and major events at different levels of granularity.
The distinctive aspect of our model is the label indicating the influencing factor behind the post. It is difficult to evaluate the accuracy of these inferred factors directly. Instead, we focus on aggregate analysis that can performed using this label, and the rich insights that we were able to unearth using this.
Using the joint counts of topics (), users () and influence factors () in each epoch (), we can estimate the probability of a specific user posting on a topic after being influenced by a factor in epoch , by normalizing appropriately. On aggregating over topics , the distribution (corresponding to model parameter ) over factors indicates the personality of the user at epoch . Plotting these distributions over epochs shows the personality trend for user . Since trends over individual anonymous users are not insightful, below we plot aggregate trends over different interesting user subsets. For this, we use heat-maps, where the matrix rows correspond to influence factors, columns to epochs and hotter colors indicate higher probability values.
World-wide Personality Trends: First, we aggregate over all users to estimate the world-wide susceptibility of users to specific factors at a specific epoch (15 day period). This trend is shown in the heat-map of Figure Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media (best viewed in color). The positions of the hotter colors and the color gradients are of interest. We can observe that the world-wide factor has the largest variance, followed by personal preferences, while the other trends are largely flat. Also, we can see that surges in world-wide influence happens mostly as the expense of personal preference. The largest such surge happens around Jun 30. This is when the news of Michael Jackson’s death broke on Twitter, and we can see that users discarded their personal preferences and posted about this event. The strength of world-wide influence then subsides gradually, and users return to their personal preferences. We can see that world-wide influence rises again around Sep 15 and Dec 1, again at the expense of personal preference. The most popular topics at these times were FIFA World Cup and Google Wave. In summary, users are usually influenced mostly by their personal preferences and friend network, apart from times of significant world-events.
Personality trends in specific geographies: Next, we aggregate over users in specific geographies to estimate susceptibility of users to specific factors in different parts of the world. Personality trends for 5 different geographies, USA, UK, Australia, China, India, are shown using heat-maps in Figure Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media. We can see many interesting patterns. The personality trends in USA, UK and Australia are largely similar, apart from the geographic influences which are high at different epochs. For USA, one such high occurs around Sep 15, when US Open is a dominating topic. For UK, we can see at high around Aug 15 (Football, Premiere League) and for Australia around Jul 1 (Jeff Goldblum’s demise). For India, the relative strengths of world-wide and geographic influences are somewhat weaker. For China, the pattern looks different. The strengths of the various influences stay relatively stable, and geographic influence is much stronger than the other 4 cases.
Topic Character Trends: As a final example of the variety of analysis that D-MRelCRP can perform, we look at trends in topic characters. By aggregating over all users and then using Bayes rule, we can find the posterior distribution over different influence factors for each topic at epoch . By plotting this over epochs, we can see how a topic changes its ‘character’, and moves from a ‘geographic’ topic to a ‘world-wide’ topic, for example. We illustrate this in Figure Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media (best viewed in color), using 3 topics. Japan Earthquake evolved from a geographic topic to a world-wide topic, Google Wave from a personal preference topic to a world-wide topic, and Tiger Woods from a personal preference topic, to a geographic topic, and finally a world-wide topic.
In summary, DMelCRP enables a wide variety of analysis of influences, leading to many interesting insights, beyond the capability of existing models.
In our experiments, we have employed a Java-based multi-threaded framework over an 8-core, 32 GB RAM machine. We employed child threads, read posts in a mini-batch, and used Gibbs iterations per batch. In Figure Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media, we plot the time taken (in micro-secs) to process one post by the multi-threaded version and a sequential version, after having processed posts. This time increases as the number of living topics increases with . The two plots clearly demonstrate that superior scalability of our multi-threaded inference algorithm.
Finally, in Table Dynamic Multi-Relational Chinese Restaurant Process for Analyzing Influences on Users in Social Media, we analyze the contributions of the different aspects to DMRelCRP’s final performance. We can see that the model improves (both in terms of perplexity and prediction accuracy) through the addition of more relationships and the interplay between them, which is the main strength of the model.
In this paper, we have made a first attempt at studying the important problem of analyzing user influences in generation of social media data. We have proposed a new non-parametric model called Dynamic Multi-Relational CRP that incorporates the aggregated influence of multiple relationships into the data generation process as well as dynamic evolution of model parameters to capture the essence of social network data. Our multi-threaded online inference algorithm allowed us to analyze a collection of 360 million tweets. Through extensive evaluations, we demonstrated that the topic trends discovered by our model are superior to those from state-of-the-art baselines. More importantly, we found insightful patterns of influence on social network users, beyond the capability of existing models.
-  A. Ahmed, Y. Low, M. Aly, and V. Josifovski. Scalable distributed inference of dynamic user interests for behavioral targeting. In SIGKDD, 2011.
-  A. Ahmed and E. Xing. Dynamic non-parametric mixture models and the recurrent chinese restaurant process: with applications to evolutionary clustering. In SDM, 2008.
-  A. Ahmed and E. Xing. Timeline: A dynamic hierarchical dirichlet process model for recovering birth/death and evolution of topics in text stream. In UAI, 2010.
-  C. Antoniak. Mixtures of Dirichlet Processes with applications to Bayesian nonparametric problems. Ann. Statist., 2(6):1152–1174, 1974.
-  A. Asuncion, P. Smyth, and M. Welling. Asynchronous distributed learning of topic models. In NIPS, 2008.
-  A. Banerjee and S. Basu. Topic models over text streams: a study of batch and online unsupervised learning. In SDM, 2007.
-  D. Blei and P. Frazier. Distance dependent chinese restaurant processes. In ICML, 2010.
-  D. Blei, A. Ng, and M. Jordan. Latent dirichlet allocation. JMLR, 3, 2003.
-  K. Canini, L. Shi, and T. Griffiths. Online inference of topics with latent dirichlet allocation. In International Conference on Artificial Intelligence and Statistics, 2009.
-  L. Hong and B. Davidson. Empirical study of topic modeling in twitter. In KDD Workshop on Social Media Analytics, 2010.
-  K. Ishiguro, T. Iwata, N. Ueda, and J. Tenenbaum. Dynamic infinite relational model for time-varying relational data analysis. In NIPS, 2010.
-  J.Chen, R.Nairn, L.Nelson, M.Bernstein, and E.Chi. Short and tweet: experiments on recommending content from information streams. In International Conference on Human Factors in Computing Systems, 2010.
-  C. Kemp, J. Tenenbaum, T. Griffiths, T. Yamada, and N. Ueda. Learning systems of concepts with an infinite relational model. In AAAI, 2006.
-  H. Lakkaraju, A. Rai, and S. Merugu. Smart news feeds using joint latent factor models. In World Wide Web, 2011.
-  O. Phelan, K.McCarthy, and B.Smyth. Using twitter to recommend real-time topical news. In ACM Conference on Recommender Systems, 2010.
-  J. Pitman. Combinatorial stochastic processes. Lecture Notes for St. Flour Summer School, 2002.
-  D. Ramage, S. Dumais, and D. Liebling. Characterizing microblogs with topic models. In International AAAI Conference on Weblogs and Social Media, 2010.
-  A. Smola and S. Narayanamurthy. An architecture for parallel topic models. In VLDB, 2010.
-  E. Sudderth, A. Torralba, W. Freeman, and A. Willsky. Tdescribing visual scenes using transformed dirichlet processes. In NIPS, 2005.
-  Y. Teh, M. Jordan, M. Beal, and D. Blei. Hierarchical dirichlet processes. JASA, 2006.
-  Z. Wen and C. Lin. On the quality of inferring interests from social neighbors. In SIGKDD, pages 373–382, 2010.
-  E. Xing, M. Jordan, and R. Sharan. Bayesian haplotype inference via the dirichlet process. Journal of Computational Biology, 14(3), 2007.
-  J. Yang and J. Leskovec. Patterns of temporal variation in online social media. In WSDM, 2011.