Analysis and Modeling of Behavioral Changes in a News Service
Information is transmitted through websites, and immediate reactions to various kinds of information are required. Hence, efforts by users to select information themselves have increased, which is fueling further improvements in recommendation services that can reduce such burdens. On the other hand, filter bubbles that only provide biased information to users are generated due to redundant recommendations. In this research, we analyzed behavioral changes prior to recommendation by clustering, and we found that user attributes and cluster contents are different among users with different behavioral changes. The proportion of users under forty and women was relatively large in the diversity-increasing group.
We also proposed an article selection model to clarify the influence of recommendation systems on behavioral changes. We compared our proposed model with the target data, verified it, and evaluated the effect of recommendation systems on user behavior. Our simulation results showed that diversity usually decreases, but collaborative filtering can suppress the diversity decrease more effectively than non-recommendations. We also found that the category that users are interested in the most is easily strengthened and is one factor that leads to less diversity, and a recommendation method that can suppress the strengthening of the category that users are interested in the most will be effective for developing a recommendation system that can suppress diversity decreasing.
Information is transmitted through websites and newspapers, and immediate reactions to various kinds of information are required. Hence, we believe that the efforts by users to select information themselves are increasing, which is leading to refinement of recommendation services that can reduce such burdens in many fields, especially for e-commerce and news services. Many companies, including Amazon, Netflix, and Apple, have also improved their recommendation systems to increase their profits .
On the other hand, filter bubbles, which only provide biased information to users, are generated due to excessive recommendations . People actively read specific information sources using the internet, and this practice may decrease selection diversity. Recommendation systems for news articles might also change user selections  . A recommendation system must be developed that can provide opportunities to come in contact with various opinions and prevent filter bubbles.
In this research, we analyzed behavioral changes prior to developing recommendations by clustering and showed that behavior changes during a certain period.
In Section 3, to analyze user behavior, we propose a method based on a user’s browsing history to classify articles from online news services provided by Nikkei Inc., one of the largest newspaper companies in Japan. In Section 4, we analyze the process of behavioral changes using information on categorized articles. In Section 5, we show the characteristics of users whose browsing behavior changed.
2 Related works
Along with the development of online media, research on recommendation systems is increasing. Schafer et al.ã, Sarwar et al. , and Linden et al.  analyzed the use of them in e-commerce from the viewpoint of marketing viewpoint and scalability.
Schein et al.  considered a solution to the cold start problem, which is the difficulty of recommending items that were not evaluated sufficiently when introducing recommendation systems. Senecal et al.  showed that online recommendation systems are more influential than recommendations from traditional human experts or other consumers. In addition, Lee et al.  argued that appropriate recommendation systems depend on target products.
Pariser  identified a filter bubble that only provides user-biased information by excessive recommendations. Bakshy et al.  analyzed news articles on Facebook and argued that cross-cutting news from information sources from the opposite political spectrum is not shared. Because of recommendation systems, articles are selectively presented in the field of political news encountered in social media.
On the other hand, contrary to Pariser’s conclusion, Linden  argued that since people have difficulty obtaining exposure to ideas and articles of which they are unaware, recommendation systems increase serendipity.
Fleder et al.  discussed the impact of recommendation systems on sales diversity and argued that some recommendations lead to a net decrease in such diversity, but personal diversity increases. However, this model is limited to two items, and the process of adding new items like news articles was ignored.
Nguyen et al.  showed that diversity of both recommended and evaluated items decreases after a certain period of time in MovieLens111http://www.movielens.org/, however the diversity of items, whose users who are affected by the recommendation system using collaborative filtering, decreases only a little. However, analysis is insufficient of the biases of users who are subjected to filtering and those who are beyond its affects, and only collaborative filtering is handled as a recommendation system.
In addition, Cosley et al.  concluded that the interface of recommendation systems influences the acceptance of recommendations. Ahmed et al.  evaluated the performance of real-time searches on twitter by simulations.
Although these previous studies present interesting knowledge about filter bubbles, behavioral changes in online media, and the development of recommendation systems and their influences, discussion remains insufficient about media that are relatively impartial like most newspaper companies. Comparisons among recommendation systems are also inadequate. Therefore, in this research, we analyze behavioral changes in online news media and compare content-based recommendation systems and collaborative filtering based on user’s similarity and show how recommendation systems work in online media.
3 Network clustering method
In this research, we use data from online news services provided by Nikkei Inc. during a three-month period from May 21 to August 20, 2017. About 600,000 articles were read during this time, and the number of subscribers was about 2 million.
We excluded articles that were read by fewer than 100 users for two reasons. First, in the development of future recommendation systems, a system should recommend articles that are as new as possible because such systems are designed for news services that demand immediacy and periodic updates to reduce the amount of data from the viewpoint of calculation time. Second, to prevent filter bubbles in recommendations, a system should not recommend esoteric article that will only appeal to few users. After conducting these processes, about 70,000 articles remained for analysis.
3.2 Article network construction
In this section, we explain a method that classifies the articles to grasp the change of user’s interest. We classified articles using the clustering method proposed by Baba et al.  . Although the method was proposed to classify twitter posts, we applied it to classify the articles. With it, we can extract the topics of articles based on a user’s browsing history without language information. We summarize this network clustering method as follows. We classified articles from Nikkei news based on the similarity of the users who read them by calculating the similarity between a pair of articles. This similarity is based on overlapping users who read both articles, and we constructed an article network using the similarity. By classifying the articles, we can evaluate s user’s behavioral changes by clusters.
When two or more users read the same two articles, the two articles have common interests. In other words, the similarity of the articles can be calculated from the degree of overlap of the users who read them. Therefore, we can construct an article network by linking articles with high similarity. The degree of the similarity of two articles, , , is calculated using Simpson coefficients:
In this formula, and represent the user groups of articles and . We linked articles with a similarity of 0.62 or more and constructed a weighted network, which has modularity Q = 0.936, 38002 links, and 17,233 nodes.
Other indices measure similarity as well as the Simpson coefficient, e.g., the Jaccard coefficient and the Dice coefficient. Baba et al. used the Jaccard coefficient for similarity, but we applied the Simpson coefficient because it appropriately expresses the degree of relationships based on co-occurrence .
3.3 Network clustering
Next, we classify the above article network and acquire a set of similar articles. For community detection, we use the Louvain method  based on modularity, which represents the degree of connectivity among a set of clusters. Highly connected clusters can be detected by maximizing modularity.
The number of articles included in one of the clusters is presented in Fig. 1. The horizontal axis is the posted date of the articles, and the vertical axis is the number of articles. The amount of articles fluctuates depending on the day of the week, weekends, and holidays . This figure includes many articles published during the target period. On the other hand, the number of articles in the clustering result decreased in the later stage, since the number of days after publication is small and the amount of browsing is not properly accumulated. In this research, we compared the first and middle periods of the clustering target period.
4 Analysis of behavioral changes
4.1 Evaluation of diversity of behavior
We defined the behaviors of the users as the class of articles consumed by them and evaluated their behavioral changes after a certain period of time. In this research, we analyzed the change of the diversity of an article’s cluster by evaluating the user’s behavior by the diversity of the cluster to which the article being read belongs.
We evaluated the browsing articles of each user based on the cluster to which the article belongs and the diversity of the browsing behavior based on the degree of the cluster concentration of read articles, calculated using information entropy:
In this formula, represents the existence probability of each user of each cluster .
A user with high cluster entropy is reading articles from various clusters, and a user with low cluster entropy is intensively concentrating on articles that just belong to a specific cluster.
We evaluated the diversity of browsing behaviors during the period and analyzed its changes. We also analyzed the characteristics of users whose diversity changed. Generally, with filter bubbles, recommendation systems isolate users from information that does not match their viewpoints, and the information is limited to a range of interest to the user. We believe that influence of the recommendation systems can be measured by the cluster concentration degree of the articles being read. For example, if filter bubbles limit the contact of users to information that matches the interest espoused by the filter bubbles, the articles will be concentrated on a specific cluster, reducing diversity and decreasing cluster entropy.
In our analysis, we compared the behaviors of June 1 to 10 (period 1) and July 6 to 15 (period 2) because the number of articles included in these clusters decreased in the later stage of the clustering target period (Fig. 1).
In this period, we selected users who joined on May and read 10 to 100 articles and extracted 1037 users who read at least one article that was included in the cluster. For these users, we calculated cluster entropy .
4.3 Results and discussions
Figure 2 shows the cluster entropy, which is an indicator of the degree of the concentration of clusters based on a user’s interest and measures the range of the interest of articles read by users. In other words, a decrease in cluster entropy means that interest has narrowed.
The average cluster entropy in period 1 is 0.718, and it is 0.619 in period 2. The cluster entropy decreases significantly at a significance level of 1%. Therefore, from this analysis, even in existing display systems, interest will be biased as the period continues. We believe the introduction of recommendation ranking systems is affecting this aspect.
5 Analysis of features leading to behavioral change
5.1 Features and target users
Next we analyzed the change of each user’s cluster entropy of periods 1 and 2. As mentioned in the Section 4, cluster entropy decreased as a whole; that is, diversity decreased. On the other hand, since there were users whose cluster entropy and diversity increased, we compared a group whose diversity increased with a group whose diversity decreased.
The diversity-increasing group has 355 users out of 1037 users. For the diversity-decreasing group, we selected the same number of top 355 users as the increasing group from the diversity-decreasing group. We compared such user attributes as age and gender as well as characteristic clusters of the diversity-increasing and diversity-decreasing groups.
The target data included information about prefecture, occupation, age, and gender as user attributes, each of which was divided into ten or more categories. Since insufficient users were included to confirm significant differences in each category, our analysis focused on age and gender.
Table 1 shows the results of each indicator. We found more women in the diversity-increasing group. When we divide users into 2 groups by ages are over 40 and ages are between 10 and 39, the number of users whose ages are between 10 and 39 of diversity-increasing group is bigger compared to the ones in diversity-decreasing group.
Looking at each generation, users in their 60s are more frequent in the diversity-decreasing group.
|gender (male/female)||(268, 87)||(298, 57)||P0.01|
|(over 40, under 39)||(223, 132)||(268, 87)||P0.05|
|age under 30||72||54||P0.10|
|age in 50s||91||98||NS|
|age in 60s||32||54||P0.05|
|age over 70||19||13||NS|
5.3 Characteristic clusters
Next we analyzed the clusters that are often included in one of the diversity-increasing or diversity-decreasing groups under the assumption that the behavior of users who consume articles in a specific cluster is changed. We examined how many users of the diversity-increasing and decreasing-groups are reading in each cluster, extracted the differences in descending order, and confirmed the contents with human annotators.
Table 2 shows the main contents of the extracted clusters.
The topics of clusters, which are often included in the diversity-increasing group, include international politics, economic information, and market information. On the other hand, clusters that have more in common in diversity-decreasing groups contain concentrated information on individual industries and companies and such specific topics as shogi. This result suggests a positive correlation between interest in international politics and economic information and a widening range of interests.
| Mostly browsed by DI DI DD Main contents of cluster 39.4 26.5 Politics 34.4 19.4 US market & preparations for one’s death 33.2 18.9 North Korea issues 31.5 18.3 Stock market information/updates 38.9 18.3 Business acquisition and withdrawal|
| Mostly browsed by DD DI DD Main cluster contents 24.8 33.2 Market information about tech companies 15.5 26.2 Asian monetary policy 17.2 24.5 Yahoo! 14.1 22.0 The latest technology on automobiles 7.0 14.4 Famous shogi player|
Although we identified few diversity-increasing groups, they contain relatively young users and more women, unlike the main users of Nikkei news who are men in their 40s and 60s. This observation suggests a positive correlation between age and a narrower range of interest, which is also consistent with a report in brain science  that suggests that cognitive flexibility declines with age.
In addition, there are clusters with different browsing ratios in the diversity increasing group and the diversity decreasing group. Users belonging to the diversity decreasing group read a lot articles that are of specific topics such as specific companies and industries and trendy topics such as shogi. On the other hand, for users belonging to the diversity increase group, the clusters contain a wide range of information such as international politics, economic information, and market information.
Perhaps users in the diversity-decreasing group read interesting articles because they are very interested in a specific topic and much less interested in other topics. On the other hand, users in the diversity-increasing group are interested in such wide perspectives as financial markets and international politics, and their interests may be more easily targeted to such a broader range of topics.
6.1 Modeling of Interest
Next, we propose an article selection model to clarify the influence of recommendation systems on behavioral changes.
In this research, we describe the behavior of general users in online media by a multi-agent model that expresses the behavior of all aspects of media. Our model assumes that online media are news sites to which articles are added and updated every day. We observed the behavioral changes of users.
In our proposed model, for simplicity, we defined the unit time that corresponds to one day in the real world for updating the articles and user’s interests and multiple articles selected by the users. In this model, the displayed articles based on browsing history and user’s interests are updated at most once a day.
The topics of the articles and the interests of users are represented by multiple categories. Categories, which are assumed to have different properties, correspond to articles and users, and the topics of the user’s interests and articles are determined by the distribution of categories. Article continues to have an immutable property defined by the following vector constituted by category :
User evaluates article based on the interest defined by the following vector and decides whether to browse it:
The evaluation of article of user is calculated by the following formula:
Depending on the articles browsed by the user, the interests of user are updated:
In the formula, is a weight expressing how much of previous day’s interest was stored, and is the value of category of article . is a weight given to each category based on previous day’s interest . The weight expresses that the influence obtained from browsing is different based on the previous interests of each category. For example, categories with low interest are hardly affected by browsing, and interesting areas are more likely to be strengthened by browsing interesting categories. In the experiment, we divided by the average value of categories, , for the average and above, and for below average.
6.2 Presentation and selection of articles
The top articles are presented to all users and individual articles are presented individually to each user. A fixed number of top articles are randomly selected. Individual articles are selected by each recommendation system. The users select multiple articles a day from a set of top articles and individual articles and update interests.
6.2.1 Selection of articles
The user selects the articles to be browsed by elite and roulette selections based on the article’s evaluation value calculated by formula 5. If the user selects an article whose evaluation value is less than threshold , the user selects again, and if the user cannot find an article that exceeds threshold over a certain number of times, the user ends that turn’s selection.
- Elite selection
: Choose a certain number from the top of the evaluation value among the presented articles.
- Roulette selection
: Choose a certain number of articles by roulette selection from articles that were not chosen by elite selection.
Probability that is selected is defined by the following formula:
In this formula, is the evaluation value of article calculated by formula 5.
6.2.2 Presentation of individual articles
Individual articles are a set of articles selected by a recommendation system and randomly selected articles. We compared two types of recommendation systems: a content-based recommendation based on the similarity of past browsing history and users, and a collaborative filtering recommendation that suggests items viewed by other similar users.
6.2.3 Content-based recommendation
In content-based recommendations, profile is calculated based on the browsing history of user using the following formula, and articles are selected in descending order of similarity between them and the profile:
Similar to 5, the similarity of profile and article is calculated:
At this time, instead of using interest vector of the users, we use profile . Because identifying interest vector of the users from the recommendation system is relatively difficult, we have to predict interest vector of the users from the viewing history.
6.2.4 Collaborative filtering
With collaborative filtering, we recommend articles from users that have high similarity with other users as well as articles that have not been read yet. The similarity between the users is calculated based on the overlapping rate of the browsed articles. From their browsing history, two users, ï¼, and the overlapping rate of their browsed articles, ï¼, is obtained by the Simpson coefficient:
First, we calculated the overlapping rate of browsed articles of all users, and select the users that have high similarity. Then, the user chooses the articles that have not been read yet but have been mostly read by the similar users.
6.2.5 Presentation without recommendations
Without a recommendation system, actual users browse the articles displayed at the top of a website and those summarized as their favorites. Without recommendations, the day’s most recent article is displayed and such highlighted articles depend on the browsing times of the users during the day. Considering the difference between their favorite types and daily browsing times, 100 articles were chosen by elite selection from the newest 1500 articles updated on that day, and 50 articles were randomly selected from 100. To verify the simulation, we consider a method that presents all 5000 presentation article candidates for that day.
6.3 Simulation procedure
As an initial setting, the categories of the interests of the articles and users are generated as uniform random numbers. At each turn of the simulation, steps 1-4 are updated sequentially. For each turn, we evaluated the degree of the concentration of the categories of the interests of the users and observed the changes in them.
Update new articles.
Update the presented articles:
Update the top articles.
Update the recommended articles.
Calculate the user similarities and select similar users (collaborative filtering).
Calculate users and article similarities (content-based recommendation).
Select recommended articles.
Update individual random articles.
Select articles by users.
Update the user’s interest.
7 Verification of Model
7.1 Simulation settings
Next we compared our proposed model and the Nikkei news data that were analyzed in Section 5. In this verification experiment, we compared the changes in the cluster entropy of the Nikkei news and in the entropy of the simulation’s interest categories. If these changes are similar, they are an appropriate article selection model of online media.
We selected articles depending on the proposed model and compared the change of diversity with Nikkei’s electronic version. As an indicator of diversity, like in Section 4, we evaluated the diversity of the browsing behavior based on the degree of concentration of the categories of the articles that were read. The degree of category concentration was calculated by the following formula as information entropy:
In this formula, represents each user’s existence probability for each category .
A user with high category entropy is reading articles composed of various categories, and a user with low category entropy is intensively reading articles composed of a specific category. We also compared whether the maximum category for each user changed before and after the period to analyze changes in users’ interests.
In this simulation, we changed and compared the recommendation system. Table 3 shows the detailed settings of the simulation. For a breakdown of the daily presentation articles, the top article is fixed at 50 articles, and the remaining 50 articles were suggested by each recommendation system. We set the following four scenarios.
ContentBase : 50 articles selected by content-based recommendation.
Collaborative : 50 articles selected by collaborative filtering recommendation
NonRecommendation : 50 articles selected randomly from 100 articles by elite from the latest 1500 articles.
All : All 5000 presented articles for that day, and the only scenario: All is different from the other scenarios respect to the number of articles presented per day.
This simulation period lasted for 45 turns (45 days) from June 1, 2017 to July 15, 2017. We did this simulation 20 times under identical conditions. The results described below are averages.
|Number of simulations||20|
|Number of users||1000|
|Number of categories||20|
|Number of presented candidate articles per day||5000|
|Number of articles updated per day||1500|
|Number of articles presented per day||100|
|Number of articles read per day||10|
|Elite selected articles per day||3|
|Number of high similarity users for collaborative filtering||20|
|Threshold value of evaluation value||0.055|
|Weight of interest of previous day||5|
Table 4 shows the simulation results when the recommendation system was changed. Among the results of each scenario, the average category entropy had a significant difference at a significance level of 1%. For the proportion of users with the largest category change, All-NonRecommendation, NonRecommendation-ContentBase had a significant difference at a significance level of 5%, and ContentBase-Collaborative, Collaborative-All, NonRecommendation-Collaborative had a significant difference at a significance level of 1%. There was no significant difference between All and ContentBase.
7.2.1 Change in entropy
Figure 3 shows the change in the interest category entropy at the start and end of the simulation in each setting.
The interested category entropy is identical to that shown in Fig. 2 during the period. As the distribution moves to the left, entropy tends to decrease. We confirmed that since the simulation is done through the period, the diversity of the browsed articles tended to decrease. From the above result, the proposed model is valid because it can present the same article selection as Nikkei.
We are also aware that compared to a recommendation system, recommendation based on contents (ContentBase) reduces the diversity the most. We notice that the diversity of recommendations based on collaborative filtering (Collaborative) has the smallest decrease in diversity, and the presentation method (NonRecommendation and All), which assumes a case without recommendations, is located in the middle.
Between NonRecommendation and All, since only NonRecommendation gives a favorite genre to a candidate, diversity decreases more. When all the articles are presented (All), diversity decreases. When the user updates its interested category entropy, the interest presented in the past browsed articles tends to weigh more than the actual achieving individualization in Nikkei, which reduces the diversity. Hence our proposed model shows its advantage at this time.
Since Nikkei currently has no recommendation system, it corresponds to the middle of NonRecommendation and All, which is a presentation method that assumes a case without recommendations. This is because the presented articles correspond to determining whether to browse based on article titles in the real world. Although interesting articles spill over genres, articles that generally belong to a single genre (uninteresting) are not recognized, and not even their titles are confirmed. In other words, some choices are made at the stage of the recognition of articles, even in the present situation where individualization is not substantially done. Therefore, in reality, the reduction of diversity is likely to occur without an individualized recommendation system.
Next, concerning the influence of such recommendation systems as content-based recommendations and collaborative filtering, the former only presents articles that were valued from browsing history, and since positive feedback is strongly applied, the diversity fell the most. Recommending articles with high similarity based on contents is not preferable for suppressing the decrease of diversity. In addition, collaborative filtering, which decreases diversity less than NonRecommendation, is appropriate for suppressing filter bubbles. Although diversity decreases in any case after a certain period of time, collaborative filtering decreases diversity less than NonRecommendation, and this result is also consistent with the MovieLens analysis by Nguyen et al. .
From this, a method that recommends articles with high similarity based on contents is not desired for avoiding a decrease of diversity. Collaborative filtering, which suppresses a decrease in diversity more than without recommendations, is appropriate for suppressing filter bubbles.
7.2.2 Change in interest category
We identified a slightly different tendency from the changes in interest category entropy. However, the collaborative filtering rate changed the most and seems to indicate not only a smaller diversity decrease but also that user interest is likely to change. Fig. 4 shows the changes of interest category 1 for each user in each scenario. The horizontal axis represents the number of simulation steps, and the vertical axis represents the proportion of interest category 1 in the interest category of each user. At the beginning, the category bias is small and distributed between . At the end of the period, although bias is occurring in every case, the bias in the case of recommendations based on content (ContentBase) is remarkable and the convergence is also fast. NonRecommendation also has a large bias. Even at the end of the simulation by collaborative filtering, the proportion of category 1 of the group with a larger category 1 is , which indicates that the deviation is weaker than the others.
From these results, the maximum category is easily enhanced, and at the simulation’s end many users account for more than half of the interest category. Such a maximum category is likely to be strengthened, reducing diversity. In other words, a recommendation method that can suppress the strengthening of the maximum category is effective for developing a recommendation system that can suppress a decrease in diversity.
| ContentBase  Collaborative|
| All  NonRecommendation|
We analyzed behavioral changes prior to recommendations by network clustering and found that user attributes and cluster contents are different among users with different behavioral changes. The proportion of users younger than 40 and women was relatively large in the diversity-increasing group.
We also proposed an article selection model and clarified the influence of recommendation systems on behavioral changes. Simulation results showed that diversity generally decreases, but collaborative filtering can suppress its decrease more than without recommendations. In addition, we found that the maximum category is easily strengthened, which is one factor that causes less diversity.
Future research will continue to analyze the factors that lead to behavioral changes for developing effective recommendation systems. In addition, we will develop recommendation systems based on these results and improve our models. In the development and experiments of recommendation systems, we will use our findings and compare recommendation systems and behavioral changes.
-  L. Ahmed and A. Abhari. Agent-based simulation of twitter for building effective recommender system. In Proceedings of the 17th Communications & Networking Simulation Symposium, page 5. Society for Computer Simulation International, 2014.
-  S. Baba, F. Toriumi, T. Sakaki, K. Shinoda, S. Kurihara, K. Kazama, and I. Noda. Classification method for shared information on twitter without text data. In Proceedings of the 24th International Conference on World Wide Web, pages 1173–1178. ACM, 2015.
-  E. Bakshy, S. Messing, and L. A. Adamic. Exposure to ideologically diverse news and opinion on facebook. Science, 348(6239):1130–1132, 2015.
-  A. S. Berry, V. D. Shah, S. L. Baker, J. W. Vogel, J. P. O’Neil, M. Janabi, H. D. Schwimmer, S. M. Marks, and W. J. Jagust. Aging affects dopaminergic neural mechanisms of cognitive flexibility. Journal of Neuroscience, 36(50):12559–12569, 2016.
-  V. D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008.
-  D. Cosley, S. K. Lam, I. Albert, J. A. Konstan, and J. Riedl. Is seeing believing?: how recommender system interfaces affect users’ opinions. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 585–592. ACM, 2003.
-  D. Fleder and K. Hosanagar. Blockbuster culture’s next rise or fall: The impact of recommender systems on sales diversity. Management science, 55(5):697–712, 2009.
-  S. Kiousis. Public trust or mistrust? perceptions of media credibility in the information age. Mass Communication & Society, 4(4):381–403, 2001.
-  W.-P. Lee, C.-H. Liu, and C.-C. Lu. Intelligent agent-based systems for personalized recommendations in internet commerce. Expert Systems with Applications, 22(4):275–284, 2002.
-  G. Linden. Eli pariser is wrong. available at: http://glinden.blogspot.com/2011/05/eli-pariser-is-wrong.html, 2011. Accessed: May 3 2018.
-  G. Linden, B. Smith, and J. York. Amazon. com recommendations: Item-to-item collaborative filtering. IEEE Internet computing, 7(1):76–80, 2003.
-  Y. Matsuo, H. Tomobe, H. Nakashima, M. Ishizuka, et al. Social network extraction from the web information. pages 46–56, 2005.
-  R. J. Mooney and L. Roy. Content-based book recommending using learning for text categorization. In Proceedings of the fifth ACM conference on Digital libraries, pages 195–204. ACM, 2000.
-  T. T. Nguyen, P.-M. Hui, F. M. Harper, L. Terveen, and J. A. Konstan. Exploring the filter bubble: the effect of using recommender systems on content diversity. In Proceedings of the 23rd international conference on World wide web, pages 677–686. ACM, 2014.
-  E. Pariser. The filter bubble: What the Internet is hiding from you. Penguin UK, 2011.
-  B. Sarwar, G. Karypis, J. Konstan, and J. Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web, pages 285–295. ACM, 2001.
-  J. B. Schafer, J. Konstan, and J. Riedl. Recommender systems in e-commerce. In Proceedings of the 1st ACM conference on Electronic commerce, pages 158–166. ACM, 1999.
-  A. I. Schein, A. Popescul, L. H. Ungar, and D. M. Pennock. Methods and metrics for cold-start recommendations. In Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, pages 253–260. ACM, 2002.
-  S. Senecal and J. Nantel. The influence of online product recommendations on consumersâ online choices. Journal of retailing, 80(2):159–169, 2004.
-  K. Uchida, F. Toriumi, and T. Sakaki. Evaluation of retweet clustering method classification method using retweets on twitter without text data. In Proceedings of the International Conference on Web Intelligence, pages 187–194. ACM, 2017.