Personalised Query Suggestion for Intranet Search with Temporal User Profiling

Personalised Query Suggestion for Intranet Search with Temporal User Profiling

Abstract

Recent research has shown the usefulness of using collective user interaction data (e.g., query logs) to recommend query modification suggestions for Intranet search. However, most of the query suggestion approaches for Intranet search follow an “one size fits all” strategy, whereby different users who submit an identical query would get the same query suggestion list. This is problematic, as even with the same query, different users may have different topics of interest, which may change over time in response to the user’s interaction with the system.

We address the problem by proposing a personalised query suggestion framework for Intranet search. For each search session, we construct two temporal user profiles: a click user profile using the user’s clicked documents and a query user profile using the user’s submitted queries. We then use the two profiles to re-rank the non-personalised query suggestion list returned by a state-of-the-art query suggestion method for Intranet search. Experimental results on a large-scale query logs collection show that our personalised framework significantly improves the quality of suggested queries.

Personalised Query Suggestion for Intranet Search with Temporal User Profiling


Thanh Vu, Alistair Willis, Udo Kruschwitz, and Dawei Song
The Open University, Milton Keynes, United Kingdom
University of Essex, Essex, United Kingdom
Tianjin University, Tianjin, P.R.China {thanh.vu, alistair.willis, dawei.song}@open.ac.uk, udo@essex.ac.uk


\@float

copyrightbox[b]

\end@float

Categories and Subject Descriptors: H.3.3 [Information Systems Applications]: Information Search and Retrieval

Keywords: Interactive IR, Intranet Search; Personalised Query Suggestion; Temporal User Profiles; Learning to Rank;

Query suggestion is an important feature in web search engines (e.g., Bing, Google) as well as in domain-specific search engines (e.g., Intranet search) [?]. Query suggestions help users quickly refine the input query to better meet the user’s information need by recommending possible terms to modify the original input query.

In this paper, we focus on query suggestion for Intranet search, which is different from web search [?]. Specifically, Intranet search (e.g., university intranets) is domain-specific and built to satisfy the user’s information need related to a specific domain (e.g., the university’s document corpus). Moreover, the Intranet may not be fully indexed and accessible by web search engines. For example, web search engines cannot access those Intranet documents which require authorised logins. The searcher, therefore, may need to use an Intranet search engine to locate relevant documents.

Using collective user interaction data (e.g., query logs) for query suggestions has been shown useful for Intranet search [?, ?]. Existing Intranet search approaches appear to follow a “one size fits all” strategy. That is different users who submit the same query receive the same query suggestion list. However, different users may have different topics of interest. Consequently, the users who have submitted the same query may have different search intentions. For example, a sociology student submitting the query “lecture notes” is likely to be more interested in sociology classes than maths classes. Moreover, users’ interests and search intentions may be dynamically evolving depending on their interactions with the system (e.g., clicks on documents), and when the interactions are made, during a search session [?].

To address these issues, we propose a unified framework to personalise query suggestions for Intranet search. Specifically, we use the interaction data of each user during a search session to build user profiles, which represent the user’s topics of interests and may change over time in response to the user’s interaction with the system.

It is worth noting that search personalisation (e.g., search result re-ranking, query suggestion, query auto-completion, etc.) has been studied extensively in the context of web search engines [?, ?, ?, ?, ?, ?, ?]. However, little attention has been paid to the same task for Intranet search. Moreover, personalisation methods on web search engines typically construct users’ profiles using their click information [?, ?, ?, ?], but less account has been taken of how users modify their queries for building the profiles.

In our proposed framework, we construct two temporal topic-based user profiles for each search session. The first is a click user profile based on the clicked documents. The second is a query user profile based on the user’s query modification history within the search session. We then use the two profiles within a learning-to-rank framework to re-rank suggested queries generated by a non-personalised method for query suggestion on Intranet search [?]. Experimental results show that our approach helps to significantly improve the query suggestion performance.

Each search session contains two types of event (i.e., queries and clicks). Given a search session, we propose to build two temporal profiles for the specific user. These are a click user profile (denoted as ), built using the user’s clicked documents, and a query user profile (denoted as ), built using the user’s submitted queries within the session. Click profile have been extensively used in other search personalisation methods [?]; we expect that the query profile will enrich the representation of the user’s search interests.

Since a user’s interests and search intentions may change over time, the more recently clicked documents and submitted queries could better represent the user’s current interests. In this paper, we propose to use a decay function to capture this characteristic as in [?, ?].

We consider that click information (e.g., clicked documents) is a good indicator of those documents’ relevance to the user’s interests [?]. To build the user profile, we use the topics discussed in the documents. We first extract clicked documents from the Intranet search’s query logs. After that, we employ Latent Dirichlet Allocation (LDA) [?] to automatically extract latent topics (denoted as ) from the clicked documents (denoted as ). After training an LDA model using the clicked documents, we apply the model to extract topics for the remaining documents in the collection. Finally, each document is described as a multinomial distribution over the topics (denoted as ), in which each topic is represented as a multinomial distribution over the entire vocabulary.

We represent the temporal click user profile as a multinomial distribution over the topics as in [?]. Specifically, the user set is denoted as . Let be an instance of . Let be the set of clicked documents of the user in the current search session, we define the click user profile of the user (given the clicked document set ) as a distribution over topics (denoted as ). The probability indicates how much the user is interested in topic . is defined as a mixture of probabilities of given as follows

(1)

is the exponential decay function of , which is the order of the document clicked by the user in the search session. indicates that is the most recently clicked document; is the normalisation factor. is the decay parameter ().

Let be the submitted query set of in the search session. Because the number of Intranet documents is smaller and can be assumed to change less frequently than web search engines’, we make the simplifying assumption of describing each query by the set of documents that contain all the query words, denoted as . Then, each search query (given the document set ) is modelled as a distribution over topics (denoted as ). The probability of a topic given (i.e., ) is defined as a mixture of probabilities of given a document as follows

(2)

is the size of the document set .

We then model the query user profile of the user (given the query set ) as a distribution over topics (denoted as ). The probability of a topic given (i.e., ) is defined as a mixture of probabilities of given query as follows

(3)

is defined in Equation Personalised Query Suggestion for Intranet Search with Temporal User Profiling. Similar to the click user profile, is the exponential decay function of , which is the order of the query submitted by the user in the search session. indicates that is the most recent query; is the normalisation factor. is the decay parameter ().

We use the two user profiles in a learning-to-rank mechanism to re-rank the query suggestion list returned by a non-personalised query suggestion method proposed by Adeyanju et al. [?], denoted as Adeyanju’s. Specifically, Adeyanju’s first constructs a domain knowledge structure in the form of a concept subsumption hierarchy using both the Intranet document collection and collective users’ query logs. Next, the suggestion list is generated using the top terms most relevant to the query in the hierarchy.

For each input query, our re-ranking method is detailed as follows

(1) We generate the top ranked suggested queries using Adeyanju’s method. We denote a suggested query as .

(2) We then compute similarity scores between and , and between and . Both the suggested query and a user profile (denoted as which is either or ) are modelled as distributions over topics (Section Personalised Query Suggestion for Intranet Search with Temporal User Profiling). To measure the similarity between and the user profile , we use Jensen-Shannon divergence (), which is a popular method of measuring the divergence (similarity) between two distributions, to measure the similarity between and

(4)

Here, and are distributions over topics of and , respectively. is the Kullback-Leibler divergence and . We consider the scores as the personalised features. We also extract other non-personalised features of the input query and the suggested query . Table Personalised Query Suggestion for Intranet Search with Temporal User Profiling shows the features extracted for re-ranking the suggestion list.

Feature Description
Personalised Features
ClickPersonalisedScore The similarity score between the suggested query and the user click profile
QueryPersonalisedScore The similarity score between the suggested query and the user query profile
Non-personalised Features
QueryRank Rank of the suggested query on the original list
QuerySim The cosine similarity score between the current query and the previous query
QueryNo Total number of queries that have been submitted to the Search Engine
SuggestedQueryCosine The cosine similarity score between the current query and the suggested query
SuggestedQueryJaccard The Jaccard distance score between the current query and the suggested query
SuggestedQueryEdit The edit distance between the current query and the suggested query
SuggestedQueryLevenshtein The Levenshtein distance between the current query and the suggested query
SuggestedQueryPreUsed Whether the suggested query was used by the user in the same search session?
Table \thetable: The personalised query suggestion features

(3) After extracting the query features, to re-rank the top suggested queries, we employ LambdaMART [?] to train ranking models. Among many learning-to-rank algorithms, LambdaMART is regarded as one of the best-performing algorithms and has been chosen as the base learning algorithm in various recent approaches to search personalisation [?].

Evaluation methodology For evaluation, we use AutoEval, an automated evaluation framework, which measures the performance of query suggestions automatically based on the actual query logs of an Intranet search [?]. For each query suggestion list, we assign a positive label for a suggestion if it is an actual refinement, which is the next submitted query in the search session, and there is at least one user click on retrieved results after the refinement. In other words, we interpret the user click after a reformulation as the criterion of a relevant suggestion. The remainder of the suggestion list is assigned negative (irrelevant) labels. We use the rank positions of the positively labelled queries as an approximation of the ground truth to evaluate the performance of query suggestions before and after re-ranking.

We also follow the experimental methodology in [?], that is, the model is evaluated continuously at periodic intervals. Specifically, we use the logs in week for training the re-ranking model and the following week for testing the trained model; where, in our experiments, , the number of weeks in the test period.

Dataset The dataset used in our experiments contains large-scale query logs collected from the search engine installed at the Web site of the University of Essex during the two years covering 1 January 2012 - 31 December 2013. Each log sample contains a session identifier, the event type (i.e., a query or a click), an auto-increment id, the event content (i.e., query text, click URL), and the event time-stamp.

We apply a simple pre-processing step to remove single event search sessions because, with those sessions, it is not possible to determine whether the user found the required information. We also remove those queries whose positive label set is empty from the dataset. We then analyse the remainder of the query logs. Table Personalised Query Suggestion for Intranet Search with Temporal User Profiling shows basic statistics.

Item 2012 2013 Total
#search sessions 397,461 338,391 735,804
#events 1,263,179 1,083,992 2,347,171
#events/session 3.22 3.25 3.23
#queries 757,645 659,284 1,416,929
#query/session 1.91 1.95 1.93
#clicked url 505,534 424,708 930,242
#clicks/session 1.27 1.26 1.26
Table \thetable: Basic statistics of the evaluation search logs

Our personalisation method and baselines We name our proposed re-ranking model as Ours. We choose two baselines to compare our work against. The first baseline is Adeyanju’s method [?], which we reimplemented to generate the original suggestion list for re-ranking.

We use the session-based approach proposed by Bennett et al. [?] as the second baseline. Specifically, in the baseline, we use only the click user profile (i.e., ) together with the non-personalised features detailed in Table Personalised Query Suggestion for Intranet Search with Temporal User Profiling to re-rank the suggestion list. We name the baseline as Click.

It is worth noting that Adeyanju’s is a non-personalised approach and achieved a good performance of query suggestion on Intranet search [?]. Click is a personalised approach and achieved good performances in web search personalisation [?]. Moreover, instead of the session-based approach by Bennett et al. [?] as our personalised baseline we could alternatively have used Shokouhi et al. [?] or Shokouhi [?].

LDA & LambdaMART We train an LDA model on the clicked documents extracted from the query logs, as detailed in Section Personalised Query Suggestion for Intranet Search with Temporal User Profiling. The number of topics (i.e., 300 in our experiments) is decided by using a held-out validation set which consists of 10% of all clicked documents. The selected number of topics is the one that gives the lowest perplexity value. The decay parameter for the two user profiles is set to as in [?]. The ranking function is learned using LambdaMART [?]. We used the default setting for LambdaMART’s prior parameters111Number of leaves = 10, minimum documents per leaf = 200, number of trees = 100 and learning rate = 0.15.

Evaluation metrics The evaluation is based on the comparison between our personalised approach and the baselines. We use four evaluation metrics: Mean Average Precision (MAP), Precision (P@k), Mean Reciprocal Rank (MMR), and Normalized Discounted Cumulative Gain (nDCG@k).

Table Personalised Query Suggestion for Intranet Search with Temporal User Profiling shows promising results when user profiles are used to personalise the query suggestion list. We can see that, even using only the click user profile, the Click method has led to an improvement of 7.22% on MAP over Adeyanju’s method. The combination of query and click profiles (i.e., Ours) achieves the highest improvement of 10.97% over Adeyanju’s in terms of MAP score. The improvements indicate that personalisation helps improve the query suggestion performance. The improvements over Adeyanju’s are all significant (paired t-test, p < 0.01).

Model MAP P@1 P@5 MRR@10 nDCG@5 nDCG@10
Adeyanju’s 0.5440 0.4113 0.1823 0.5447 0.5714 0.6000
Click 0.5833 0.4271 0.1981 0.5839 0.6193 0.6583
%rel +7.22% +3.84% +8.67% +7.22% +8.38% +9.73%
Ours 0.6037 0.4526 0.2026 0.6043 0.6413 0.678
%rel +10.97% +10.04% +11.14% +10.94% +12.23% +13.02%
Table \thetable: Overall performance of the methods. %rel denotes the relative improvement over Adeyanju’s.

In the comparison between personalisation methods (i.e., Ours and Click), Table Personalised Query Suggestion for Intranet Search with Temporal User Profiling shows that using both the click and query user profiles (i.e., Ours) significantly improves the suggestion quality over the Click baseline (). Interestingly, our method produces a significantly better quality of the first query in the suggestion list with the improvement of 5.97% on P@1 over Click. The improvements of Ours over Click also indicate that the query user profile is important in the query suggestion task, especially for the quality of the first suggested query.

With more submitted queries and clicked documents, we are able to build richer user profiles. In this experiment, we aim to study whether the position of a query in a search session has any effect on the performance of personalised query suggestion. For each search session, we label queries by their positions during the session. Because there are few sessions containing more than three queries (i.e., 7.65% of sessions in the query logs), we label the first three queries from one to three according to the order of submission in the search session; the remaining queries are labelled as .

Figure \thefigure: Relative MAP improvements over Adeyanju’s with different query positions.

It is worth noting that for the first query, we cannot build the click user profile because there is no previously clicked document. However, we can still build the query user profile for the first query. We show the improvement in performance of the personalised methods over Adeyanju’s in term of MAP metric with different query positions in Figure Personalised Query Suggestion for Intranet Search with Temporal User Profiling. Here the statistical significance is verified by t-test (p < 0.01). For the first query within a search session, our method, which can use only the query user profile, significantly improves the query suggestion performance over Adeyanju’s. It again confirms the effectiveness of the query information on personalised query suggestion for Intranet search.

From the second query, we can build both the query and click user profiles. One can see that the higher position of a query is, the larger improvement in performance the personalised query suggestion can be achieved. Specifically, from the query with high positions (i.e., ), the improvements of Click and Ours are 11.37% and 18.22%, respectively. Figure Personalised Query Suggestion for Intranet Search with Temporal User Profiling also shows that Ours outperforms Click significantly with the improvements of at least 3.45% (p < 0.01). It indicates that richer user profiles (by observing more clicked documents and submitted queries during the search session) help achieve better query suggestion performances. The findings offer future research directions that use user profiles which go beyond single sessions

The query length is defined by the number of words in the query (e.g., the query “University webmail” has the length of 2). The length of a query might give an indication as to how specific the information need of an individual user is (i.e., a longer query can typically be assumed to reflect a more specific information need). In this experiment, we aim to show the impact of personalisation on query suggestion with different query lengths. We label each query by its length, which is the number of words in the query (i.e., from one to three and words because there are few queries containing more than three words (i.e., 5.7% of queries in the query logs)).

Figure Personalised Query Suggestion for Intranet Search with Temporal User Profiling shows the improvement in performance of Click and Ours over Adeyanju’s in term of MAP with different query lengths. We see that personalisation methods achieve significantly better performances than the non-personalised method does (p < 0.01). Even for short queries (length 1 and 2) which tend to be more generic, the Click and Ours methods outperform Adeyanju’s method with the improvement of more than 6.11% and 9.09%, respectively. We see that the longer a query is, the higher improvement personalised methods can achieve. Specifically, with a longer query (i.e., with length ), the Click and Ours methods yield the highest improvements, i.e., 29.82% and 53.85%, respectively. This indicates that a longer query would also get more benefit from personalisation.

Figure \thefigure: Relative MAP improvements over Adeyanju’s with different query lengths.

Figure Personalised Query Suggestion for Intranet Search with Temporal User Profiling also indicates that by combining the query user profile with the click user profile, Ours significantly improves the query suggestion performance over Click (p < 0.01). Moreover, the improvements are larger with the longer queries (i.e., with length ). In particular, the improvements of Ours over Click are 6.7% and 18.2% on the query length 3 and the query length , respectively.

In this paper, we proposed a personalised query suggestion framework and showed how it performed on Intranet search. We built two session-specific temporal user profiles, a query user profile using the submitted queries, and a click user profile using the clicked documents. We then extracted the personalised features using the two profiles and combined them with non-personalised features to learn a ranking model using LamdaMART. Finally, we used the ranking model to re-rank the query suggestion list returned by a state-of-the-art query suggestion approach for Intranet search.

Experimental results on a large-scale query log dataset collected from a university intranet search engine show that personalisation significantly improved the query suggestion performance. Using both the click user profile and query user profile achieved the highest performance indicating that personalised query suggestion for Intranet search should take into account both click and query information. Moreover, the positive impact of personalised query suggestions is more pronounced with longer queries and queries submitted later within a session.

  • [1] I. A. Adeyanju et al. Adaptation of the concept hierarchy model with search logs for query recommendation on intranets. In SIGIR’12, 2012.
  • [2] M.-D. Albakour et al. Autoeval: An evaluation methodology for evaluating query suggestions using query logs. In ECIR’11, 2011.
  • [3] P. N. Bennett et al. Modeling the impact of short- and long-term behavior on search personalization. In SIGIR’12, 2012.
  • [4] D. M. Blei et al. Latent dirichlet allocation. J. Mach. Learn. Res., 2003.
  • [5] C. J. C. Burges et al. Learning to rank with non-smooth cost functions. In NIPS’06, 2006.
  • [6] D. Hawking. Enterprise search. In Modern Information Retrieval, 2nd Ed. 2010.
  • [7] T. Joachims et al. Accurately interpreting clickthrough data as implicit feedback. In SIGIR’05, 2005.
  • [8] M. Shokouhi. Learning to personalize query auto-completion. In SIGIR’13, 2013.
  • [9] M. Shokouhi et al. Fighting search engine amnesia: Reranking repeated results. In SIGIR’13, 2013.
  • [10] T. Vu et al. Improving search personalisation with dynamic group formation. In SIGIR’14, 2014.
  • [11] T. Vu et al. Modelling time-aware search tasks for search personalisation. In WWW’15 Companion, 2015.
  • [12] T. Vu et al. Temporal latent topic user profiles for search personalisation. In ECIR’15, 2015.
  • [13] T. Vu et al. Search personalization with embeddings. In ECIR’17, 2017.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
13791
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description