Programming Bots by Synthesizing Natural Language Expressions into API Invocations

# Programming Bots by Synthesizing Natural Language Expressions into API Invocations

## Abstract

At present, bots are still in their preliminary stages of development. Many are relatively simple, or developed ad-hoc for a very specific use-case. For this reason, they are typically programmed manually, or utilize machine-learning classifiers to interpret a fixed set of user utterances. In reality, real world conversations with humans require support for dynamically capturing users expressions. Moreover, bots will derive immeasurable value by programming them to invoke APIs for their results. Today, within the Web and Mobile development community, complex applications are being stringed together with a few lines of code – all made possible by APIs. Yet, developers today are not as empowered to program bots in much the same way. To overcome this, we introduce BotBase, a bot programming platform that dynamically synthesizes natural language user expressions into API invocations. Our solution is two faceted: Firstly, we construct an API knowledge graph to encode and evolve APIs; secondly, leveraging the above we apply techniques in NLP, ML and Entity Recognition to perform the required synthesis from natural language user expressions into API calls.

## 1Introduction

The API economy is in full swing. Applications are rapidly developed by composing APIs. Companies increasingly structure their development and internal systems in terms of APIs, and business strategies for many companies revolve around APIs. This novel economy facilitates the creation of new business models and opens many revenue opportunities [1]. Technological advances, the cloud, awareness at the CxO level (starting from the famous memo by Jef Bezos [2]), and the increased need for business agility are making applications based on composing APIs mainstream. The “Internet of Things” is pushing this trend even more by providing API access to all sort of devices, so that we can use APIs not only to place an order but also to make coffee.

This paper leverages the API economy in combination with advances in bots technology to facilitate the development of intuitive computing solutions that connect user needs, expressed in natural language, with invocations of the APIs that can fulfill these needs. Intuitive computing envision a collaboration between users and “an intelligent digital assistant that models user intent, and that suggests or carries out actions likely to satisfy that intent” [3]. Developing applications that enable such kind of collaboration today is facilitated by the many bot builders frameworks and systems provided by nearly every big IT player such as Facebook [4], Google [5], Microsoft [6], Amazon [7], IBM Watson [8] and more.

Messaging bots could in fact, be well set as the viable alternative or even successor of mobile apps [9]. Text messaging itself are already embedded in many apps and used by over 3 billion people, with this number is increasing rapidly [9]. Messaging bots look and feel not much different to chatting with a friend via instant messaging. There is no need to learn, understand or navigate disparate interfaces or languages – yet they carry the potential of being programmed to automate conversations, transactions or workflows [10].

However, at present, bot building systems while useful in detecting the user’s intent, still require significant development and configuration work for each usage scenario, and they hardcode the dependency with the APIs to be invoked. In other words, today, for each class of functionality we want to enable through a bot, and for each API, we need to hardcode the logic that interacts with the user to identify their intent and obtain the desired values for the specific API parameters. Bot building framework have no knowledge of these APIs: they support the identification of specific intents and parameters and then they can be configured to make a REST call. APIs are therefore external to the bot builder and is completely outside of the process, only appearing at the end in an hardcoded fashion.

Our vision consists of making APIs first class citizens of bot builders. We aim at synthesizing natural language expression, and at dynamically determining which API to invoke based on our understanding of the users’ intent and on the knowledge over an API knowledge graph that describes what the methods do and how they can be invoked. The vision we set forth in this paper is that of users being able to talk with assistants (as some of us do every day with Siri, Alexa or Cortana) and, with the help of a knowledge of APIs modeled via a knowledge graph and built incrementally, dynamically identify intents, APIs fulfilling the identified intent, and collect from the user the value of the required parameters for invocation. If successful, this can enable a new approach to the development of cognitive services where the “program” is built on the fly based on users’ requests and available services exposed through APIs.

More specifically, we devise a technique for synthesizing natural language user expressions into concrete API calls by leveraging an API Knowledge Graph (KG) to achieve this. The API KG contains information about APIs, their declarations, expressions, parameters and possible values.

## 2Preliminaries

The following are accepted terminology as related to bot-technology [4][5][8]:

Expression.

This refers to a natural language user utterance whilst conversing with a bot (e.g. ’what is the weather forecast for tomorrow?’). Based on the users’ expression, the bot recognizes a category such as ’weather_condition’, ’find_restaurant’. This refers to the users’ intention.

Intent.

The notion of intent is widely used in conversation systems, and refers to the users’ purpose in an expression. For the example above, the underlying intent may be summarized as ’weather_condition’. In conversation systems, a fundamental step to answer a users’ query is to recognize intention.

Entity.

A common notion amongst NLP systems, an entity represents a concept specific to domain or context. In the previous example, an expression such as ’what is the weather like tomorrow?’ identified a keyword such as ’tomorrow’. When reshaped as an entity, we link this to a type such as time/date. Formulating entities often requires analyzing the keyword in conjunction with the overall intent.

Conversation.

In order to execute an action by the bot (e.g. calling an API), all required parameters would need to filled in prior to execution. However, in the event that something is missing, the bot will typically prompt users with more questions to obtain any missing information. This back and forth question-answering between user and bot (and vice versa) is referred to as conversation.

Action.

This ties a high-level user expression into a concrete execution plan, in order to obtain the required results to satisfy the user’s query. For example, we may have an action such as FindRestaurant. In many ways, actions are akin to functions in traditional programming languages. Whereby, just like functions, actions also often requires input parameters. For example, more specifically the action above could likely be described as FindRestaurant(type,place,location), such as: FindRestaurant(‘Italian’,‘cafe’,‘Sydney’).

Our Approach. BotBase builds upon all the above concepts, plus in our work we propose the notion of Declaration, which refers to the expressivity of an API. Moreover, the notion of API and its constituent declarations, in our context are synonymous with the traditional notion of intent.

Figure 1 illustrates the relationship between expression, APIs, Declarations, Entities and API Calls.

## 3API Knowledge Graph

### 3.1Knowledge Model

We propose a lightweight model to capture useful information about APIs (and their related information). Such knowledge, which may be created incrementally and collectively, will significantly assist bot programming both during development and execution. During development, it is the onus of the developer to properly specify set of expressions (or patterns thereof), possible keywords and the required actions for each. Clearly, specifying this each time is not only tedious but likely to lack full inclusion (e.g. there may be some valid expressions that the developer did not think about). Thus, the value of a knowledge graph in this context would alleviate these challenges. More so, in our proposed work the synthesis technique performs in real-time (with the help of the knowledge-graph), thus also acting during execution time to assist in the bot’s processing of the result.

The following are the main entities of our model: API is the root and contains a name, description, a set of descriptive tags and a base_URI (e.g. api.yelp.com). An API may have one or more Declarations that are supported for this particular API. It contains a type (e.g. GET or POST) and a path (e.g. /search?term=[term]&location=[location]). Declarations are linked to a set of textual Sample Expressions. Declarations may also have one or more Parameters (e.g. term, location); which may be linked to possible values Para_Values (e.g. food, San Francisco). Parameters also has a required field to flag if the parameter is optional or not.

Finally, we define an entity called Value_WEValue that links concrete values with semantically similar words that are derived from our deep learning model. (Note, WE is an acronym for Word Embedding.)

### 3.2Knowledge Acquisition

We employ a combination of two techniques for acquisition and enrichment of new knowledge: Firstly, we harness crowdsourcing, and this paradigm enables an incremental and collective approach. Crowdsourced workers such as (i) developers enrich the KG by entering sample expressions at bot training time, (ii) end-users enrich the KG whilst talking with bots. Secondly, we apply deep learning, and this paradigm introduces a layer of intelligence and automation.

Word Embedding – Deep Learning Model. We use a deep-learning model called Word Embedding[11] to generate semantically similar keywords, that relate to existing parameter values. To more the dense, the easier it makes it to more successfully link arbitrary user expressions with API declaration. This is because we would be able to better recognize a match to the API’s declaration and parameters, and find an API that would useful to process the expression’s intent.

More specifically, the model we use is Word2vec [11]. This model takes as input a corpus and produces a vector-space where each word in the corpus is represented by a vector. Words that are semantically similar are close to each other in the vector space and such distance is typically measured with the help of the cosine similarity metric [12]. While our model can be trained on any text corpus, in this paper we use a dataset of Wikipedia1 because it is general and it covers a variety of topics. This corpus contains more than 2 billion words. The neural network uses 150 hidden layers; sliding window size of five; skip gram with negative sampling[11]; sets the minimum occurrence of a words to fifty (i.e. the word must appear at least fifty times to be inside the training set); and ignores stopwords by using the NLTK[13] library for preprocessing the corpus. Our trained model contains both unigrams (e.g. car, restaurant, computer) and biagrams2(e.g. Opera House, New York, ice cream).

We use our trained model (i) in API KG enrichment process: for a given parameter value (e.g. “Paris”), we can add its neighbours (from the word embedding model) such as “Sydney”, “New York”, “London” as another possible values for that parameter (e.g. “location”), (ii) in the synthesis process: for an extracted value (e.g. “Melbourne”) from user expression, we can link it to the most relevant parameter (e.g. “location”) by computing similarity ratio between the extracted value and values already linked to that parameter inside the API KG.

## 4Synthesis of Expression to API

BotBase is actualized as a set of components, that when placed in orchestration serves to fulfill the fundamental goal of mapping a natural language user expression into a concrete API call. We refer to this overall process as “synthesis”. Underlying this process, the components critically relies on the intelligence of the knowledge graph and word embedding model that we proposed in the previous section. This process (or even individual components) could potentially be used at every stage of a bots lifecycle. For example, during bot development, a bot developer usually have to select the API and declaration from a pool of APIs. Especially when there are a large number of APIs, this could be much simplified using the KG to help guide the developer select the matching API/s. The developer could input a set of sample user expressions – and then feed them into the process to discover the API that the synthesis process deems most relevant. Similarly, during bot training, developers may enter sample expressions, to allow the bot to learn possible values for the various parameters of the selected API declaration. Finally, during execution, of course it serves the fundamental purpose of processing user expressions into API calls.

Figure 2 illustrates the functional architecture of these components in orchestration. We now describe in more detail, the role and individual functionality of each of these components:

Entity Extractor. This component takes a user expression and decomposes into a set of entities; which are then classified into nouns, adjectives and proper nouns with the help of Stanford POS tagger [14]. This filters all words that are irrelevant (e.g., stop words like the and at). Secondly, this component derives bigrams (e.g., ice cream and New York) with the help of our word embedding model (Figure ?).

API Selector. Upon performing entity extraction from the inputted user expression, this provides the first step towards understanding the basic intent. Accordingly, we can use this information to select APIs that could potentially be a match in satisfying the user’s request. We can match with APIs based on semantic similarity: this means, using the extracted entity information (i.e. nouns, verbs, as well as complex analysis derived from bigrams, see Figure ?), we can match with APIs in the KG by comparing with their associated tags, parameter names and values. It is important to note finding the best API match will require additional layers of analysis, namely examining the declarations of APIs and checking if their parameter signatures are covered. This will therefore be dealt with at subsequent steps (as we will describe in more detail below). However, the purpose of this stage is to perform an early filtration and produce a set of viable APIs.

API Declaration Selector. Now that we have a viable set of APIs, the next step is identifying the best matched declarations. We may recall, declaration curated in the KG are linked with “sample expressions”. Accordingly, we can perform semantic similarity analysis between the inputted user expression and the set of sample expressions stored for each declaration. To implement this, we again leverage our word embedding model to compute semantic similarity. We compute the vector representation of both the inputted user expression and each of the sample expressions as follows:

where and are the vector representation of the user expression and a sample expression from the API Knowledge Graph respectively; and are the vector representation of a single word within each expression. is the vector space defined by the Word Embedding model.

The actual semantic similarity of and is computed using the cosine similarity metric [12]. Using this metric, the API Declaration Selector chooses the declaration that has the highest score for , which is equated as follows:

where is the sample expression from the API Knowledge Graph that has the highest similarity with the user expression , and is the declaration that is associated to the sample expression. The declaration is obtained through the function above that takes as input an expression and returns an API declaration.

Entity-Parameter Mapper(EPM). We are now ready to map extracted entities to the parameters of the chosen declaration. Let’s return to our earlier example for illustration. Consider we have a user expression that states, “Is there any Chinese restaurant near Sydney Opera House”, and the selected API declaration is api.yelp.com/search?term=[term]&location= [location]. The job of the Entity-Parameter Mapper is to recognize that Chinese restaurant should be mapped to the parameter term, and that Sydney Opera House to location.

In order to determine which entity should be mapped to which parameter, we once again apply the Word Embedding model. We compute for each parameter of the selected API declaration, the semantic similarity with stored parameter values for that declaration. (We may recall these stored values are acquired based on the methods discussed at Section 3.2). Therefore, more specifically in this example, if we compare the input entity Chinese, with all stored parameter values of the selected API declaration, we could conclude the keyword Chinese is probably a value of the term parameter because of its semantic similarity with e.g. ’French’, which is also a value of the term parameter as stored in the Knowledge Graph.

The output of the EPM component is paraValue matrix. It contains parameter-value pairs (i.e. mapped entities to parameters), together with confidence ratio for each. We also use a component named KG updater to store new values into the knowledge graph as users are conversing with the bot. This component only store new values provided the confidence ratio above a certain threshold. At this stage, we set it to . However, this may considered an initial threshold for experimentation purposes. We believe, as increasingly more knowledge is gained, it will become clearer to understand the right balance, and this figure will be more precise.

Coverage Checker. This component is responsible for performing two main tasks: (i) Firstly, to choose the best declaration that has the maximum number of fulfilled required parameters amongst candidate API declarations; (ii) Secondly, to make sure that all the required parameters of the selected declaration are fulfilled before calling the API. Accordingly, the Coverage Checker component takes two inputs: the candidate API declarations; and secondly, the paraValue matrix that was produced by the previous component.

We determine coverage for each viable declaration by computing the following:

where represents the ratio of parameters of the declaration with a value assigned to it, indicates whether or not a parameter has been mapped to a value, and is the total count of required parameters for the API declaration. When this component is performed during execution, we require that the should be equal to . This implies, all required parameters of the API declaration are fulfilled. When this happens, a call to the API can be performed and the results returned as a JSON object for processing of the final results and display to end-user. If is not equal to , then Coverage Checker notifies the end-user, and tries to obtain the necessary feedback about parameters for which no suitable value could be found.

In some cases, we may want to choose one declaration amongst a set (e.g., many declarations appear too similar, or if used during development mode, the bot developer may simply wants help to compare various declarations). Accordingly, the best declaration is picked based on equation (Equation 4):

where represents the declaration that scored the highest metric, as defined in equation (Equation 3). As a result of this, the Coverage Checker outputs the best API declaration.

## 5Related Work and Discussions

There is a considerable body of research conducted on spoken dialog systems. Some works used crowds in areas like writing and editing [15], image description and interpretation [16]. Some others worked on taking advantage of the crowd to empower the dialog system abilities (answer to questions in several domains with various styles) by creating API calls [17]. BotBase takes inspiration from the latter work for building API calls dynamically, albeit by using machine learning and knowledge graph.

Besides, there are some works focused on API recommendation techniques which are relevant to our work. RACK [18] translates a natural language query into a list of relevant APIs by mapping keywords to APIs, Portfolio [19] recommends relevant API methods for a given natural language query by employing indexing and graph based algorithms, proposed technique in [20] recommends a graph of API methods from precise textual phrase matching.

Furthermore, improvements in natural language processing techniques as well as bot development approaches have led to a renewal in bots. Most recent efforts in natural language processing have focused on translating user expressions into program codes [21] [22]. For bot development side, rule-based approaches such as AIML, RiveScript, ChatScript, Pandorabots and Superscript provide scripting-based languages for describing service-user conversations as a rule set (e.g., if-then statements). Flow-based approaches such as Motion.ai, ChatFuel, Manychat, FlowXO and Sequel provide platforms to describe end-user conversations as a flowchart (e.g. sequence of options and actions). Hybrid approaches like Wit.ai, API.ai, Microsoft LUIS, Meya, Pullstring, Gupshup, Microsoft BotFramework, IBM Conversation Service and Amazon Lex provide platforms to describe service-user conversations as intents and natural language expressions. However, these approaches are forcing developers to handle almost all the parts (e.g. training bot, writing pieces of codes for each user intents, interacting with underlying backend services such as APIs, code invocation commands and programming libraries) to generate user responses. BotBase benefits from techniques in hybrid platforms to learn possible values for API parameters and to enrich its API Knowledge Graph.

The idea of building an API Knowledge graph for BotBase comes from Augur [23] which is a knowledge graph of human actions, activities and their relation with objects; KnowBot [24] a question-answer system that builds a knowledge graph of concepts (in science) and their relations through conversational dialogs between user and system; Commandspace [25] a knowledge graph of user expressions and Adobe Photoshop application commands; and QF-Graph [26] a mapping between user’s vocabulary and system features. We use the same idea to construct API knowledge graph contains API, API declarations, parameters and values. Furthermore, BotBase leverages concepts in previous effort on synthesizing Java language codes for a given input text by using PCFG model and mapping words and API declarations [27]. BotBase constructs mappings between API declarations to sample expressions and parameters to values in its knowledge graph.

## 6Limitations & Future Work

In this paper, we have unleashed the first generation of our vision towards an era of intelligent bots that learns from complex data sets and mimics the way of the human brain. At this stage, we make certain assumptions, we elaborate upon these limitations below together with possible solutions for each.

Conversational Bots. As BotBase is only within its early stages, it does not yet support conversational bots. At this stage we assume a stateless environment, where each question from the user has an answer from the bot, and the bot does not remember anything from the previous chat messages. Future works therefore requires a stateful conversation mechanism in BotBase.

Process Workflows. One of the most significant limitations of our current work is the assumption that an action can be performed by one API declaration alone. While it is true, there are currently a plethora of APIs and large amounts of user requests could in fact be fulfilled by a single declaration. Nevertheless, in future work we intend to support executing dynamic process workflows (i.e. calling upon a series or combination of APIs) to fulfill a user request.

Advanced Knowledge Mining. The success of this proposed approach depends greatly on the quantity and accuracy of the knowledge graph. At present, we have acquired a sizable quantity of data using bootstrapping, crowdsourcing and general purpose Wikipedia corpus to train our word embedding model. This truly does supply an enormous amount of data, however in future we would benefit from gathering from other sources of data, no only formal datasets or corpuses.

## 7Conclusion

In this paper, we provide a solution for dynamically synthesizing natural language user expressions into API invocations. To achieve this we have proposed a lightweight API Knowledge Graph that contain the required information about APIs. We enrich the KG with new APIs, as well as appending supplementary knowledge about existing APIs. Enrichment is achieved with the intelligence of deep-learning (Word Embedding model trained by Wikipedia corpus) in conjunction with crowdsourcing approaches. In the synthesis process, BotBase benefits from the API KG and the word embedding model to determine which API to invoke based on user expressions.

## Acknowledgment

We Acknowledge Data to Decisions CRC for funding scholarship on Query Answering and Predictive Techniques for Analyst Tasks in End-User Big Data Analytic.

### References

1. =2plus 43minus 4 L. Columbus. 2017 is quickly becoming the year of the api economy. [Online]. Available: https://www.forbes.com/sites/louiscolumbus/2017/01/29/2017-is-quickly-becoming-the-year-of-the-api-economy =0pt
2. J. Hernandez, “Jeff bezos’ mandate: Amazon and web services,” Psychology, Management and Leadership, 2012.
3. =2plus 43minus 4 U. of Rochester Computer Science (URCS). Intuitive computing. [Online]. Available: http://www.cs.rochester.edu/research/intuitive/ =0pt
4. =2plus 43minus 4. (2017) Natural Language for Developers. [Online]. Available: https://wit.ai/ =0pt
5. =2plus 43minus 4. (2017) Conversational User Experience Platform. Now we’re talking. [Online]. Available: https://api.ai/ =0pt
6. =2plus 43minus 4. (2016) Build a great conversationalist. [Online]. Available: https://dev.botframework.com/ =0pt
7. =2plus 43minus 4. (2016) Amazon Lex: Conversational interfaces for your applications. [Online]. Available: https://aws.amazon.com/lex/ =0pt
8. =2plus 43minus 4. (2016) Conversation Service. [Online]. Available: https://www.ibm.com/watson/developercloud/conversation.html =0pt
9. =2plus 43minus 4 H. Francis. Rise of the chatbots: bigger than apps? [Online]. Available: http://www.smh.com.au/technology/technology-news/rise-of-the-chatbots-bigger-than-apps-20160407-go1ca3.html =0pt
10. =2plus 43minus 4 B. Sheth. Forget apps, now the bots take over. [Online]. Available: https://techcrunch.com/2015/09/29/forget-apps-now-the-bots-take-over/ =0pt
11. =2plus 43minus 4 T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Proceedings of the 26th International Conference on Neural Information Processing Systems, ser. NIPS’13.1em plus 0.5em minus 0.4em USA: Curran Associates Inc., 2013, pp. 3111–3119. [Online]. Available: http://dl.acm.org/citation.cfm?id=2999792.2999959 =0pt
12. D. Jurafsky and J. H. Martin, Speech and Language Processing (2Nd Edition).1em plus 0.5em minus 0.4emUpper Saddle River, NJ, USA: Prentice-Hall, Inc., 2009, ch. Computational Lexical Semantics.
13. M. Goossens, F. Mittelbach, and A. Samarin, Natural language processing with Python.1em plus 0.5em minus 0.4emO’Reilly Media, Inc., 2009.
14. =2plus 43minus 4 C. D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J. Bethard, and D. McClosky, “The Stanford CoreNLP natural language processing toolkit,” in Association for Computational Linguistics (ACL) System Demonstrations, 2014, pp. 55–60. [Online]. Available: http://www.aclweb.org/anthology/P/P14/P14-5010 =0pt
15. =2plus 43minus 4 M. S. Bernstein, J. Brandt, R. C. Miller, and D. R. Karger, “Crowds in two seconds: Enabling realtime crowd-powered interfaces,” in Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’11.1em plus 0.5em minus 0.4emNew York, NY, USA: ACM, 2011, pp. 33–42. [Online]. Available: http://doi.acm.org/10.1145/2047196.2047201 =0pt
16. =2plus 43minus 4 J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller, R. Miller, A. Tatarowicz, B. White, S. White, and T. Yeh, “Vizwiz: Nearly real-time answers to visual questions,” in Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’10. 1em plus 0.5em minus 0.4emNew York, NY, USA: ACM, 2010, pp. 333–342. [Online]. Available: http://doi.acm.org/10.1145/1866029.1866080 =0pt
17. T.-H. K. Huang, W. S. Lasecki, and J. P. Bigham, “Guardian: A crowd-powered spoken dialog system for web apis,” in Third AAAI Conference on Human Computation and Crowdsourcing, 2015.
18. M. M. Rahman, C. K. Roy, and D. Lo, “Rack: Automatic api recommendation using crowdsourced knowledge,” in Software Analysis, Evolution, and Reengineering (SANER), 2016 IEEE 23rd International Conference on, vol. 1.1em plus 0.5em minus 0.4emIEEE, 2016, pp. 349–359.
19. =2plus 43minus 4 C. McMillan, M. Grechanik, D. Poshyvanyk, Q. Xie, and C. Fu, “Portfolio: Finding relevant functions and their usage,” in Proceedings of the 33rd International Conference on Software Engineering, ser. ICSE ’11. 1em plus 0.5em minus 0.4emNew York, NY, USA: ACM, 2011, pp. 111–120. [Online]. Available: http://doi.acm.org/10.1145/1985793.1985809 =0pt
20. W.-K. Chan, H. Cheng, and D. Lo, “Searching connected api subgraph via text phrases,” in Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering.1em plus 0.5em minus 0.4emACM, 2012, p. 10.
21. =2plus 43minus 4 A. Desai, S. Gulwani, V. Hingorani, N. Jain, A. Karkare, M. Marron, S. R, and S. Roy, “Program synthesis using natural language,” in Proceedings of the 38th International Conference on Software Engineering, ser. ICSE ’16.1em plus 0.5em minus 0.4emNew York, NY, USA: ACM, 2016, pp. 345–356. [Online]. Available: http://doi.acm.org/10.1145/2884781.2884786 =0pt
22. =2plus 43minus 4 S. Gulwani and M. Marron, “Nlyze: Interactive programming by natural language for spreadsheet data analysis and manipulation,” in Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, ser. SIGMOD ’14.1em plus 0.5em minus 0.4emNew York, NY, USA: ACM, 2014, pp. 803–814. [Online]. Available: http://doi.acm.org/10.1145/2588555.2612177 =0pt
23. E. Fast, W. McGrath, P. Rajpurkar, and M. S. Bernstein, “Augur: Mining human behaviors from fiction to power interactive systems,” in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems.1em plus 0.5em minus 0.4emACM, 2016, pp. 237–247.
24. B. Hixon, P. Clark, and H. Hajishirzi, “Learning knowledge graphs for question answering through conversational dialog,” in Proceedings of the the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, 2015.
25. =2plus 43minus 4 E. Adar, M. Dontcheva, and G. Laput, “Commandspace: Modeling the relationships between tasks, descriptions and features,” in Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’14.1em plus 0.5em minus 0.4emNew York, NY, USA: ACM, 2014, pp. 167–176. [Online]. Available: http://doi.acm.org/10.1145/2642918.2647395 =0pt
26. =2plus 43minus 4 A. Fourney, R. Mann, and M. Terry, “Query-feature graphs: Bridging user vocabulary and system functionality,” in Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’11.1em plus 0.5em minus 0.4emNew York, NY, USA: ACM, 2011, pp. 207–216. [Online]. Available: http://doi.acm.org/10.1145/2047196.2047224 =0pt
27. =2plus 43minus 4 T. Gvero and V. Kuncak, “Synthesizing java expressions from free-form queries,” in Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, ser. OOPSLA 2015.1em plus 0.5em minus 0.4emNew York, NY, USA: ACM, 2015, pp. 416–432. [Online]. Available: http://doi.acm.org/10.1145/2814270.2814295 =0pt
You are adding the first comment!
How to quickly get a good reply:
• Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
• Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
• Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters