Studying Mutual Phonetic Influence with a Web-Based Spoken Dialogue System
This paper presents a study on mutual speech variation influences in a human-computer setting. The study highlights behavioral patterns in data collected as part of a shadowing experiment, and is performed using a novel end-to-end platform for studying phonetic variation in dialogue. It includes a spoken dialogue system capable of detecting and tracking the state of phonetic features in the user’s speech and adapting accordingly. It provides visual and numeric representations of the changes in real time, offering a high degree of customization, and can be used for simulating or reproducing speech variation scenarios. The replicated experiment presented in this paper along with the analysis of the relationship between the human and non-human interlocutors lays the groundwork for a spoken dialogue system with personalized speaking style, which we expect will improve the naturalness and efficiency of human-computer interaction.
Keywords:Spoken dialogue systems Phonetic convergence Human-computer interfaces.
asp short=ASP, long=additional speech processing \DeclareAcronymasr short=ASR, long=automatic speech recognition, long-plural=s, short-plural=s \DeclareAcronymcall short=CALL, long=computer-assisted language learning, long-plural=s, short-plural=s \DeclareAcronymcapt short=CAPT, long=computer-assisted pronunciation training, long-plural=s, short-plural=s \DeclareAcronymcat short=CAT, long=communication accommodation theory, long-plural=s, short-plural=s \DeclareAcronymdfg short=DFG, long=German Research Foundation \DeclareAcronymdm short=DM, long=dialogue manager, long-plural=s, short-plural=s \DeclareAcronymgui short=GUI, long=graphical user interface, long-plural=s, short-plural=s \DeclareAcronymhci short=HCI, long=human-computer interaction \DeclareAcronymhhi short=HHI, long=human-human interaction, long-plural=s, short-plural=s \DeclareAcronymhmm short=HMM, long=hidden markov model, long-plural=s, short-plural=s \DeclareAcronymhts short=HTS, long=\achmm based Speech Synthesis System \DeclareAcronymits short=ITS, long=intelligent tutoring system, long-plural=s, short-plural=s \DeclareAcronymnlg short=NLG, long=natural language generation, long-plural=s, short-plural=s \DeclareAcronymnlp short=NLP, long=natural language processing, long-plural=s, short-plural=s \DeclareAcronymnlu short=NLU, long=natural language understanding, long-plural=s, short-plural=s \DeclareAcronymrmse short=RMSE, long=root-mean-square error \DeclareAcronymsds short=SDS, long=spoken dialogue system, long-plural=s, short-plural=s \DeclareAcronymsmo short=SMO, long=sequential minimization optimization, long-plural=s, short-plural=s \DeclareAcronymsvm short=SVM, long=support vector machine, long-plural=s, short-plural=s \DeclareAcronymtts short=TTS, long=text-to-speech
With expanding research on, and growing use of, \acpsds, a main challenge in the development of \achci systems of this kind is making them as close as possible to \achhi in terms of naturalness, fluency, and efficiency. One aspect of such \acphhi is the relationship of mutual influences between the interlocutors. Influence here means changes in one interlocutor’s conversational behavior triggered by the behavior of the other interlocutor. We refer to changes that make the interlocutors’ behaviors more similar as convergence. Convergence can occur in different modalities and with respect to various aspects of the conversation, like eye gaze, gestures, lexical choices, body language, and more. In this paper, we concentrate on phonetic-level influences, i.e., phonetic convergence. More specifically, we examine pronunciation variations over the course of \acphci. As speech is the principal modality used for interacting with \acpsds, we believe it is an especially important modality to study in the field of \achci. Simulating and triggering convergence on the phonetic level, as found in \achhi, may contribute a lot to the naturalness of dialogues of humans with computers. \Acpsds with such personalized speech style are expected to offer more natural and efficient interactions, and move one more step away from the interface metaphor  toward the human metaphor .
The novel system introduced in Section 3 tracks the states of segment-level phonetic features during the dialogue. All of the analyses are automated and run in real time. This not only saves time and manual work typically needed in convergence studies, but also makes the system more suitable for integration into other applications. In Section 4, we use this newly introduced system with recordings collected as part of a shadowing experiment to examine the relationship of mutual influences between a (simulated) user and the system. Using these signals, the system provides both visual and numerical evidence of the mutual influences between the interlocutors over the course of the interaction. The system itself will be made freely available under an open-source license.
2 Background and Related Work
Integrating support for changes in the speech signal into computer systems may enhance \achci and provide improved tools for studying convergence in \achci.  discusses the advantages of systems that dynamically adapt their speech output to that of the user, and the challenges involved in developing and using these systems.
2.1 Phonetic Convergence
According to , phonetic convergence is defined as an increase in segmental and suprasegmental similarity between two interlocutors (e.g., ). In contrast to entrainment, we use the term convergence to describe dynamic, mutual, and non-imposing changes. Phonetic convergence has been found to various extent in conversational settings . There is evidence for phonetic convergence being both an internal mechanism  and socially motivated . Previous studies of phonetic convergence in spontaneous dyadic conversations have focused on speech rate , timing-related phenomena , pitch , intensity , and perceived attractiveness . Phonetic convergence is often examined in the scope of shadowing experiments, in which the participants are asked to produce certain utterances after hearing them produced in some stimuli (e.g., ). This is typically done with single target words embedded in a carrier sentence. The experiment showcasing our system in Section 4 uses whole sentences as stimuli, in which the target features are embedded, making it a semi-conversational \achci setting.
2.2 Adaptive Spoken Dialogue Systems
Various studies have investigated entrainment and priming in \acpsds, aiming to better understand \achci dynamics and improve task-completion performance. , for example, focused on dynamic entrainment and adaptation on the lexical level. Others, like , concentrated on word frequency.  examined changes in both lexical choice and word frequency. While these studies addressed the changes in experimental, scripted scenarios, the theoretical foundations for studying these changes in spontaneous dialogue exist as well .  provide examples of online adaptation for dialogue policies and belief tracking.
It is important to note that while all of the studies mentioned above examine various aspects of dialogues, none of those are related to speech – the primary modality used to interact with \acpsds. Studying convergence of speech in an \achci context is made possible with more natural synthesis technology, which gives fine-grained control over parameters of the system’s spoken output. Many systems that deal with adaptation of speech-related features focus on prosodic characteristics like intonation or speech rate.  sheds light on acoustic-prosodic entrainment in both \achhi and \achci via the use of interactive avatars.  found that users’ speech rate can be manipulated using a simulated \acsds. Similar results were found when intensity changes in children’s interaction with synthesized \actts output were examined .
All of the above provide solid ground for further investigation of phonetic convergence in \achci using \acpsds.
The system introduced here is an end-to-end, web-based \acsds with a focus on phonetic convergence and its analysis over the course of the interaction. Besides placing convergence in the spotlight, it is designed to be flexible and to meet the researcher’s needs by offering a wide range of customizations (see Section 3.2). Its online access via a web browser makes it scalable and simple for the end-user to operate. The system’s architecture and functionality are described in Section 3.1, its \acgui and operation in Section 3.3, and an example of its utilization is demonstrated in Section 4. Ultimately, it offers an experimentation platform for studying phonetic convergence, with emphasis on the following:
- Temporal analysis
offering real-time visualization of the interlocutors’ relations with respect to selected phonetic features over the course of the interaction.
allowing the user to experiment with different scenarios by configuring parameters and definitions in many of the system’s components.
- Online scalability
connecting multiple web clients to a server, allowing users to use it anywhere without preceding installation and configurations, and helping experimenters to collect and replay acquired data.
As the system aims to offer a customizable playground for experimenting and studying phonetic convergence in \achci, a key aspect of its architecture is the separation between client-side, server-side, and external resources (see Figure 1). All of the resources and configuration files needed for designing the interaction are located on the server. Running the client and server on different machines allows users to interact with the system using a web browser alone.
As shown in Figure 2, the dialogue system component consists of typical \acsds modules such as \acnlu and a \acdm, but also contains an \acfasp module . This module is responsible for processing the audio and extracts the features required by the convergence model. While the \acnlu component uses merely the transcription provided by the \acasr, the \acasp module analyzes the speech signal itself. More specifically, it tracks occurrences of the defined features and passes their measured values to the convergence model, which, in turn, forwards the tracked feature parameters to the \actts synthesis component.
3.2 Models and Customizations
The computational model for phonetic convergence used in the system is described in . Different phonetic convergence behavioral patterns that were observed in \achhi and \achci experiments can be simulated by combinations of the model’s parameters presented in Table 1. All of the parameters can be modified in the system’s configuration file.
The entire convergence process is based on the the tracked phonetic features that are considered \enquoteconvergeable, i.e., prone to variation, and is triggered whenever the \acasr component detects a segment containing a phoneme associated with one or more of these features. Each feature is defined by a key-value map, in which the parameters from Table 1 are configured. A classifier can be associated with each feature to provide real-time predictions for both the user’s and the system’s realizations of that feature, as demonstrated in Figure 3. With this information available, more meaningful insights can be gained into the dynamics of phonetic changes in the dialogue.
The dialogue domain is specified in an XML-based file. More details on the domain file can be found in . The format of the domain file makes it easy to define new scenarios for the system, such as a task-specific dialogue, general-purpose chat, or an experimental setup.
Speech processing is a central aspect of the system. Different models can be used, e.g., for improving performance or changing the language or the \acasr module or the output voice of the \actts module.
3.3 Graphical User Interface
The system’s \acgui consists of three main areas:
In the chat area, the interaction between the user and the system is shown in a chat-like representation. Each turn’s utterance appears inside a chat bubble with different colors and orientations for the user and the system. The turns are also numbered, to better track the dialogue progress and analysis shown by the plots in the graph area. It is also possible to replay the utterance of a turn by clicking the \enquotePlay button in its corresponding bubble.
In the interaction area, the user can interact with the system with written or spoken input. Text-based interactions progress through the dialogue (if applicable) and trigger any subsequent domain model, but will not affect convergence-related models, since there is no audio input to process. Spoken input can be provided either by speaking into the microphone or via audio files with pre-recorded speech. The latter option is especially useful for simulating specific user input, or for reproducing a previous experiment, as done in Section 4.
In the graph area, each of the tracked features is visualized in a separate plot, and new data points are added whenever a new instance of the feature is detected. Hovering over a data point in a graph reveals additional information, such as the turn in which it was added, or the realized variant of the feature produced in that turn as predicted by its classifier. These dynamic, interactive plots make it possible to shed light on how the interlocutors influence each other, whether or not they are aware of it, throughout their exchanges. Figure 3 shows such a graph with several accumulated data points.
4 Showcase: Examining Convergence Behaviors
For demonstrating a possible use of the system, we simulated the shadowing experiment detailed in  using the system and its analyses to look into types of participant convergence behavior with respect to the features examined in the experiment (see Table 2). This experiment is designed to trigger phonetic convergence by confronting the participants with stimuli in which certain phonetic features are realized in a manner different from their own realizations. The simulation was carried out by building a domain file with the experimental procedure, including the transition between the experiment’s phases, as well as the flow within each phase. This automates the procedure and adapts it to the participant’s pace. Participants were simulated by using their recorded speech from the original experiment in the same order. The use of the system for this purpose results in an automated, reproducible execution, with additional insights like classification of feature realizations and dynamic visualizations in the \acgui. The classifiers were trained offline on the data points acquired from analyzing the stimuli. However, the system also supports incremental, online re-training whenever requested by the user, for example after every time the convergence model is updated. For the demonstration presented here, a \acsmo  implementation of the \acsvm classifier was used for training. Each turn’s number and prediction are added as an interactive annotation to the dynamic graph of the relevant features, as shown in Figure 3. Finally, using the system, the experiment is transformed into an automated dialogue scenario, which enhances its \achci nature.
|War||das||Gerät||sehr||teuer?||\textipa[E:] vs. \textipa[e:] in word-medial ä|
|Ich||bin||süchtig||nach||Schokolade.||\textipa[Iç] vs. \textipa[Ik] in word-final -ig|
|Wir||besuchen||euch||bald||wieder.||\textipa[\sn] vs. \textipa[@n] in word-final -en|
4.1 Finding Behavioral Patterns
In this section, we focus on the validation for the feature \textipa[E:] vs. \textipa[e:] as a representative example for the phonetic adaptation capability of the system. Although the classified realization is binary (\textipa[E:] or \textipa[e:]), the underlying representation used by the model is gradual. Both of these views on the feature can be seen in the graph area, as shown in Figure 3.
The degree of convergence was examined per utterance in the shadowing phase of the experiment. Three main groups emerged, each with a different behavior: one group of participants showing little to no tendency to converge (changes in of their utterances), the second, with varying degrees of convergence ( to ), and a third group of participants who were very sensitive to the stimuli’s variation (). We refer to these groups as Low, Mid, and High, respectively. The feature’s classifier was determined on the fly, so that the prediction for each utterance was made based on the type of the stimulus to which the participant was listening. As Table 3 shows, the Low and High groups are both of significant size, indicating that these two distinct behaviors exist in the data and can be spotted by the system.
In addition, we validated the separation between these behaviors.
To this end, we regarded the shadowing phase as an annotation task, where the annotators are the predictors of the user and the system.
Note that similarity would mean complete convergence to every stimulus, which cannot be reasonably expected (cf. ).
The Cohen’s kappa () values
5 Conclusion and Future Work
We have introduced a system with an integrated \acfsds, which can track and analyze mutual influence on the phonetic level during the interaction based on an internal convergence model. This combines work done in the fields of phonetic convergence and adaptive \acpsds, and contributes to the understanding of power relations between a human and a computer interlocutors. Many aspects of the system are customizable, which makes it flexible in terms of possible supported scenarios. The system can also run on a separate server, which makes it easier to scale its online use.
To showcase its capabilities, we simulated a replication of a shadowing experiment, which examined phonetic convergence regarding certain segment-level phonetic features. Three main user behaviors were found with respect to their tendency to change their pronunciation based on the system’s stimulus input. This sheds light on possible relations and dynamics between a user and a system in \achci. Running the experiment in this way not only saved time by automating the annotation and phonetic analysis, but also offered additional insight such as visualization and on-the-fly classification. We believe that this shows that phonetic convergence can be studied using our \acsds, and that this is one step forward toward personalized, phonetically aware \acpsds, which will enable more natural and efficient interaction.
|Similarity ()||Agreement ()||Size ()|
Future work will pursue two independent directions. Regarding phonetic convergence, supporting more features will make the system more comprehensive and useful for studying a wider range of phenomena. Specifically, adding support for supra-segmental features will enable replication of experiments similar to e.g.,  in the same manner as in Section 4. As for user acceptance, it would be interesting to examine whether users show any preference toward an \acsds that converges to their speech on the phonetic level, and whether they would change their speaking style based on the system’s output, forming an interaction with mutual and dynamic convergence similar to \achhi. The first research question can be tested by comparing user interaction with a baseline system and one with convergence capabilities, and evaluating the users’ performance and satisfaction. The second research question can be investigated by comparing the users’ speech when interacting with either system configuration. Additionally, to test the system’s influence on users’ speech, the users can train with an intelligent \acfcall, such as a \acfcapt system, which will change its learner model based on their input. Metrics such as task completion rate, performance accuracy, and completion time can be used to evaluate how helpful the system is.
Funded by the \acdfg under grants STE 2363/1 and MO 597/6.
- email: email@example.com
- email: firstname.lastname@example.org
- email: email@example.com
- as calculated by the kappa2 command of the irr R package (v0.84), https://cran.r-project.org/package=irr
- Bell, L., Gustafson, J., Heldner, M.: Prosodic adaptation in human-computer interaction. In: 15th International Congress of Phonetic Sciences (ICPhS). pp. 2453–2456. Barcelona (2003), https://www.internationalphoneticassociation.org/icphs-proceedings/ICPhS2003/p15_2453.html
- Brennan, S.E.: Lexical entrainment in spontaneous dialog. In: International Symposium on Spoken Dialogue (ISSD). pp. 41–44. Philadelphia, PA, USA (1996)
- Carlson, R., Edlund, J., Heldner, M., Hjalmarsson, A., House, D., Skantze, G.: Towards human-like behaviour in spoken dialog systems. In: Swedish Language Technology Conference (SLTC). Gothenburg, Sweden (2006)
- Coulston, R., Oviatt, S., Darves, C.: Amplitude convergence in children’s conversational speech with animated personas. In: Interspeech. pp. 2689–2692. Denver, CO, USA (2002), http://www.isca-speech.org/archive/icslp_2002/i02_2689.html
- Edlund, J., Heldner, M., Gustafson, J.: Two faces of spoken dialogue systems. In: Workshop Dialogue on Dialogues: Multidisciplinary Evaluation of Advanced Speech-based Interactive Systems. Pittsburgh, PA (2006)
- Gašić, M., Breslin, C., Henderson, M., Kim, D., Szummer, M., Thomson, B., Tsiakoulis, P., Young, S.: On-line policy optimisation of Bayesian spoken dialogue systems via human interaction. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 8367–8371. Vancouver, BC, Canada (2013). https://doi.org/10.1109/ICASSP.2013.6639297
- Gessinger, I., Raveh, E., Le Maguer, S., Möbius, B., Steiner, I.: Shadowing synthesized speech – segmental analysis of phonetic convergence. In: Interspeech. pp. 3797–3801. Stockholm, Sweden (2017). https://doi.org/10.21437/Interspeech.2017-1433
- Gessinger, I., Schweitzer, A., Andreeva, B., Raveh, E., Möbius, B., Steiner, I.: Convergence of pitch accents in a shadowing task. In: Speech Prosody. pp. 225–229. Poznań, Poland (2018). https://doi.org/10.21437/SpeechProsody.2018-46
- Kim, M., Horton, W.S., Bradlow, A.R.: Phonetic convergence in spontaneous conversations as a function of interlocutor language distance. Laboratory Phonology 2(1), 125–156 (2011). https://doi.org/10.1515/labphon.2011.004
- Levitan, R.: Acoustic-prosodic Entrainment in Human-human and Human-computer Dialogue. Ph.D. thesis, Columbia University, New York, NY, USA (2014). https://doi.org/10.7916/D8GT5KCH
- Levitan, R., Beňuš, Š., Gálvez, R.H., Gravano, A., Savoretti, F., Trnka, M., Weise, A., Hirschberg, J.: Implementing acoustic-prosodic entrainment in a conversational avatar. In: Interspeech. pp. 1166–1170. San Francisco, CA, USA (2016). https://doi.org/10.21437/Interspeech.2016-985
- Levitan, R., Hirschberg, J.: Measuring acoustic-prosodic entrainment with respect to multiple levels and dimensions. In: Interspeech. pp. 3081–3084. Florence, Italy (2011), http://www.isca-speech.org/archive/interspeech_2011/i11_3081.html
- Lewandowski, N.: Talent in Nonnative Phonetic Convergence. Ph.D. thesis, University of Stuttgart, Stuttgart, Germany (2012). https://doi.org/10.18419/opus-2858
- Lison, P., Kennington, C.: Developing spoken dialogue systems with the OpenDial toolkit. In: Workshop on the Semantics and Pragmatics of Dialogue (SemDial). pp. 194–195. Gothenburg, Sweden (2015)
- Lopes, J., Eskenazi, M., Trancoso, I.: Automated two-way entrainment to improve spoken dialog system performance. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 8372–8376. Vancouver, BC, Canada (2013). https://doi.org/10.1109/ICASSP.2013.6639298
- Michalsky, J., Schoormann, H.: Pitch convergence as an effect of perceived attractiveness and likability. In: Interspeech. pp. 2253–2256. Stockholm, Sweden (2017). https://doi.org/10.21437/Interspeech.2017-1520
- Nenkova, A., Gravano, A., Hirschberg, J.: High frequency word entrainment in spoken dialogue. In: ACL Human Language Technologies (HLT). pp. 169–172. Columbus, OH, USA (2008), http://aclweb.org/anthology/P08-2043
- Oviatt, S., Darves, C., Coulston, R.: Toward adaptive conversational interfaces: Modeling speech convergence with animated personas. ACM Transactions on Computer-Human Interaction 11(3), 300–328 (2004). https://doi.org/10.1145/1017494.1017498
- Pardo, J.S.: On phonetic convergence during conversational interaction. Journal of the Acoustical Society of America 119(4), 2382–2393 (2006). https://doi.org/10.1121/1.2178720
- Parent, G., Eskenazi, M.: Lexical entrainment of real users in the Let’s Go spoken dialog system. In: Interspeech. pp. 3018–3021. Makuhari, Chiba, Japan (2010), http://www.isca-speech.org/archive/interspeech_2010/i10_3018.html
- Pickering, M.J., Garrod, S.: Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences 27(2), 169–190 (2004). https://doi.org/10.1017/S0140525X04000056
- Platt, J.C.: Fast training of support vector machines using sequential minimal optimization. In: Burges, C.J.C., Schölkopf, B., Smola, A.J. (eds.) Advances in Kernel Methods, pp. 185–208. MIT Press (1999)
- Putman, W.B., Street, R.L.: The conception and perception of noncontent speech performance: Implications for speech-accommodation theory. International Journal of the Sociology of Language 1984(46), 97–114 (1984). https://doi.org/10.1515/ijsl.1984.46.97
- Raveh, E., Steiner, I.: A phonetic adaptation module for spoken dialogue systems. In: Workshop on the Semantics and Pragmatics of Dialogue (SemDial). pp. 162–163. Saarbrücken, Germany (2017)
- Raveh, E., Steiner, I., Möbius, B.: A computational model for phonetically responsive spoken dialogue systems. In: Interspeech. pp. 884–888. Stockholm, Sweden (2017). https://doi.org/10.21437/Interspeech.2017-1042
- Schweitzer, A., Walsh, M.: Exemplar dynamics in phonetic convergence of speech rate. In: Interspeech. pp. 2100–2104. San Francisco, CA, USA (2016). https://doi.org/10.21437/Interspeech.2016-373
- Walker, A., Campbell-Kibler, K.: Repeat what after whom? Exploring variable selectivity in a cross-dialectal shadowing task. Frontiers in Psychology 6(546), 1–18 (2015). https://doi.org/10.3389/fpsyg.2015.00546