From Bilingual to Multilingual Neural Machine Translation by Incremental Training

From Bilingual to Multilingual Neural Machine Translation by Incremental Training

Carlos Escolano,  Marta R. Costa-jussà,  José A. R. Fonollosa,
TALP Research Center
Universitat Politècnica de Catalunya, Barcelona

Multilingual Neural Machine Translation approaches are based on the use of task-specific models and the addition of one more language can only be done by retraining the whole system. In this work, we propose a new training schedule that allows the system to scale to more languages without modification of the previous components based on joint training and language-independent encoder/decoder modules allowing for zero-shot translation. This work in progress shows close results to the state-of-the-art in the WMT task.

From Bilingual to Multilingual Neural Machine Translation by Incremental Training

Carlos Escolano,  Marta R. Costa-jussà,  José A. R. Fonollosa, {carlos.escolano,marta.ruiz,jose.fonollosa} TALP Research Center Universitat Politècnica de Catalunya, Barcelona

1 Introduction

In recent years, neural machine translation (NMT) has had an important improvement in performance. Among the different neural architectures, most approaches are based in an encoder-decoder structure and the use of attention-based mechanisms (Cho et al., 2014; Bahdanau et al., 2014; Vaswani et al., 2017). The main objective is computing a representation of the source sentence that is weighted with attention-based mechanisms to compute the conditional probability of the tokens of the target sentence and the previously decoded target tokens. Same principles have been successfully applied to multilingual NMT, where the system is able to translate to and from several different languages.

Two main approaches have been proposed for this task, language independent or shared encoder-decoders. Language independent architectures(Firat et al., 2016a, b; Schwenk and Douze, 2017) in which each language has its own encoder and some additional mechanism is added to produce shared representations, as averaging of the context vectors or sharing the attention mechanism. These architectures have the flexibility that each language can be trained with its own vocabulary all languages are trained in parallel. Recent work (Lu et al., 2018) show how to perform many to many translations with independent encoders and decoders just by sharing additional language-specific layers that transformed the language-specific representations into a shared one without the need of a pivot language,

On the other hand, architectures that share parameters between all languages (Johnson et al., 2017) by using a single encoder and decoder trained to be able to translate from and to any of the languages of the system. This approach presents the advantage that no further mechanisms are required to produced shared representation of the languages as they all share the same vocabulary and parameters, and by training all languages without distinction they allow low resources languages to take benefit of other languages in the system improving their performance. Even though by sharing vocabulary between all languages the number of required tokens grows as more languages are included in the system, especially when languages employ different scripts in the system, such as Chinese or Russian. Recent work proposes a new approach to add new languages to a system by adapting the vocabulary (Lakew et al., 2018), relying on the shared tokens between the languages to share model parameters, showing that the amount of shared tokens between the languages had an impact in the model performance. This could limit the capability of the system to adapt to languages with a different script.

These approaches can be further explored into unsupervised machine translation where the system learns to translate between languages without parallel data just by enforcing the generation and representation of the tokens to be similar (Artetxe et al., 2017; Lample et al., 2018).

Also related to our method, recent work has explored transfer learning for NMT (Zoph et al., 2016; Kim et al., 2019) to improve the performance of new translation directions by taking benefit of the information of a previous model. These approaches are particularly useful in low resources scenarios when a previous model trained with orders of magnitude more examples is available.

This paper proposes a proof of concept of a new multilingual NMT approach. The current approach is based on joint training without parameter or vocabulary sharing by enforcing a compatible representation between the jointly trained languages and using multitask learning (Dong et al., 2015). This approach is shown to offer a scalable strategy to new languages without retraining any of the previous languages in the system and enabling zero-shot translation. Also it sets up a flexible framework to future work on the usage of pretrained compatible modules for different tasks.

2 Definitions

Before explaining our proposed model we introduce the annotation and background that will be assumed through the paper. Languages will be referred as capital letters while sentences will be referred in lower case given that , and .

We consider as an encoder () the layers of the network that given an input sentence produce a sentence representation () in a space. Analogously, a decoder () is the layers of the network that given the sentence representation of the source sentence is able to produce the tokens of the target sentence. Encoders and decoders will be always considered as independent modules that can be arranged and combined individually as no parameter is shared between them. Each language and module has its own weights independent from all the others present in the system.

3 Joint Training

In this section, we are going to describe the training schedule of our language independent decoder-encoder system. The motivation to choose this architecture is the flexibility to add new languages to the system without modification of shared components and the possibility to add new modalities in the future as the only requirement of the architecture is that encodings are projected in the same space. Sharing network parameters may seem a more efficient approach to the task, but it would not support modality specific modules while

Given two languages and , our objective is to train independent encoders and decoders for each language, and that produce compatible sentence representations . For instance, given a sentence in language , we can obtain a representation from that the encoder that can be used to either generate a sentence reconstruction using decoder or a translation using decoder . With this objective in mind, we propose a training schedule that combines two tasks (auto-encoding and translation) and the two translation directions simultaneously by optimizing the following loss:


where and correspond to the reconstruction losses of both language and (defined as the cross-entropy of the generated tokens and the source sentence for each language); and correspond to the translation terms of the loss measuring token generation of each decoder given a sentence representation generated by the other language encoder (using the cross-entropy between the generated tokens and the translation reference); and corresponds to the distance metric between the representation computed by the encoders. This last term forces the representations to be similar without sharing parameters while providing a measure of similarity between the generated spaces. We have tested different distance metrics such as L1, L2 or the discriminator addition (that tried to predict from which language the representation was generated). For all these alternatives, we experienced a space collapse in which all sentences tend to be located in the same spatial region. This closeness between the sentences of the same languages makes them non-informative for decoding. As a consequence, the decoder performs as a language model, producing an output only based on the information provided by the previously decoded tokens. Weighting the distance loss term in the loss did not improve the performance due to the fact that for the small values required to prevent the collapse the architecture did not learn a useful representation of both languages to work with both decoders. To prevent this collapse, we propose a less restrictive measure based on correlation distance (Chandar et al., 2016) computed as in equations 2 and 3. The rationale behind this loss is maximizing the correlation between the representations produced by each language while not enforcing the distance over the individual values of the representations.


where and correspond to the data sources we are trying to represent; and correspond to the intermediate representations learned by the network for a given observation; and and are, for a given batch, the intermediate representation mean of and , respectively.

4 Incremental training

Given the jointly trained model between languages and , the following step is to add new languages in order to use our architecture as a multilingual system. Since parameters are not shared between the independent encoders and decoders, our architecture enables to add new languages without the need to retrain the current languages in the system. Let’s say we want to add language . To do so, we require to have parallel data between and any language in the system. So, assuming that we have trained and , we need to have either or parallel data. For illustration, let’s fix that we have parallel data. Then, we can set up a new bilingual system with language as source and language as target. To ensure that the representation produced by this new pair is compatible with the previously jointly trained system, we use the previous decoder () as the decoder of the new system and we freeze it. During training, we optimize the cross-entropy between the generated tokens and the language reference data but only updating the layers belonging to the language encoder (). Doing this, we train not only to produce good quality translations but also to produce similar representations to the already trained languages. No additional distance is added during this step. The language sentence representation is only enforced by the loss of the translation to work with the already trained module as it would be trained in a bilingual NMT system.

Figure 1: Language addition and zero shoot training scheme

Our training schedule enforces the generation of a compatible representation, which means that the newly trained encoder can be used as input of the decoder from the jointly trained system to produce zero-shot to translations. See Figure 1 for illustration.

The fact that the system enables zero-shot translation shows that the representations produced by our training schedule contain useful information and that this can be preserved and shared to new languages just by enforcing the new modules to train with the previous one, without any modification of the architecture. Another important aspect is that no pivot language is required to perform the translation, once the added modules are trained the zero-shot translation is performed without generating the language used for training as the sentence representations in the shared space are compatible with all the modules in the system.

A current limitation is the need to use the same vocabulary for the shared language () in both training steps. The use of subwords (Sennrich et al., 2015) mitigates the impact of this constraint.

5 Data and Implementation

Experiments are conducted using data extracted from the UN (Ziemski et al., 2016) and EPPS datasets (Koehn, 2005) that provide 15 million parallel sentences between English and Spanish, German and French. newstest2012 and newstest2013 were used as validation and test sets, respectively. These sets provide parallel data between the four languages that allow for zero-shot evaluation. Preprocessing consisted of a pipeline of punctuation normalization, tokenization, corpus filtering of longer sentences than 80 words and true-casing. These steps were performed using the scripts available from Moses (Koehn et al., 2007). Preprocessed data is later tokenized into BPE subwords (Sennrich et al., 2015) with a vocabulary size of 32000 tokens. We ensure that the vocabularies are independent and reusable when new languages were added by creating vocabularies monolingually, i.e. without having access to other languages during the code generation.

Baseline 32.60 32.90 31.81 28.96
Joint 29.70 30.74 - -
Added lang - - 30.93 27.63
Table 1: Experiment results measured in BLEU score. All blank positions are not tested or not viable combinations with our data.
System FR-ES DE-ES
Pivot 29.09 21.74
Zero-shot 19.10 10.92
Table 2: Zero-shot results measured in BLEU score
Figure 2: Plot A shows the source sentence representation of each of the encoder modules(ES,EN,DE,FR). Plots B and C show the representation of the target sentence generated by English(B) and Spanish(C) decoders given the sentence encodings of parallel sentences generated for all four language encoder modules.

6 Experiments

Our first experiment consists in comparing the performance of the jointly trained system to the standard Transformer. As explained in previous sections, this joint model is trained to perform two different tasks, auto-encoding and translation in both directions. In our experiments, these directions are Spanish-English and English-Spanish. In auto-encoding, both languages provide good results at 98.21 and 97.44 BLEU points for English and Spanish, respectively. In translation, we observe a decrease in performance. Table 1 shows that for both directions the new training performs more than 2 BLEU points below the baseline system. This difference suggests that even though the encoders and decoders of the system are compatible they still present some differences in the internal representation.

Note that the languages chosen for the joint training seem relevant to the final system performance because they are used to define the representations of additional languages. Further experimentation is required to understand such impact.

Our second experiment consists of incrementally adding different languages to the system, in this case, German and French. Note that, since we freeze the weights while adding the new language, the order in which we add new languages does not have any impact on performance. Table 1 shows that French-English performs 0.9 BLEU points below the baseline and German-English performs 1.33 points below the baseline. French-English is closer to the baseline performance and this may be due to its similarity to Spanish, one of the languages of the initial system languages.

The added languages have better performance than the jointly trained languages (Spanish-English from the previous section). This may be to the fact that the auto-encoding task may have a negative impact on the translation task.

Finally, another relevant aspect of the proposed architecture is enabling zero-shot translation. To evaluate it, we compare the performance of each of the added languages compared to a pivot system based on cascade. Such a system consists of translating from French (German) to English and from English to Spanish with the standard Transformer. Results show that the zero shot translation provides a consistent decrease in performance for both cases of zero-shot translation.

7 Visualization

Our training schedule is based on training modules to produce compatible representations, in this section we want to analyze this similarity at the last attention block of encoders, where we are forcing the similarity. In order to graphically show the presentation a UMAP (McInnes et al., 2018) model was trained to combine the representations of all languages. Figures show 130 sentences extracted from the test set. These sentences have been selected to have a similar length to minimize the amount of padding required.

Figure 2 (A) shows the representations of all languages created by their encoders. Languages are represented in clusters and no overlapping between languages occurs, similarly to what (Lu et al., 2018) reported in their multilingual approach, the language dependent features of the sentences have a great impact in their representations.

However, since our encoder/decoders are compatible and produce competitive translations, we decided to explore the representations generated at the last attention block of the English decoder, and are shown in Figure 2 (B). We can observe much more similarity between English, French, and German, (except for a small German cluster) and separated clusters for Spanish. The reason behind these different behaviors may be due to the fact that French and German have directly been trained with the frozen English decoder and being adjusted to produce representations for this decoder. Finally, figure 2 (C) shows the representations of the Spanish decoder. Some sentences have the same representation for all languages, whereas others no. Looking at the specific sentences that are plotted, we found that close representations do not correlate with better translations or better BLEU. Sentence examples are shown in the appendix. More research is required to analyze which layer in the decoder is responsible for approaching languages in a common space. This information could be used in the future to train encoders of new languages by wisely sharing parameters with the decoder as in previous works (He et al., 2018).

8 Conclusions

This work proposes a proof of concept of a bilingual system NMT which can be extended to a multilingual NMT system by incremental training. We have analyzed how the model performs for different languages. Even though the model does not outperform current bilingual systems, we show first steps towards achieving competitive translations with a flexible architecture that enables scaling to new languages (achieving multilingual and zero-shot translation) without retraining languages in the system.


This work is supported in part by a Google Faculty Research Award. This work is also supported in part by the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund and the Agencia Estatal de Investigación, through the postdoctoral senior grant Ramón y Cajal, contract TEC2015-69266-P (MINECO/FEDER,EU) and contract PCIN-2017-079 (AEI/MINECO).


  • Artetxe et al. (2017) Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017. Unsupervised neural machine translation. arXiv preprint arXiv:1710.11041.
  • Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  • Chandar et al. (2016) Sarath Chandar, Mitesh M Khapra, Hugo Larochelle, and Balaraman Ravindran. 2016. Correlational neural networks. Neural computation, 28(2):257–285.
  • Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.
  • Dong et al. (2015) Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 1723–1732.
  • Firat et al. (2016a) Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016a. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 866–875, San Diego, California. Association for Computational Linguistics.
  • Firat et al. (2016b) Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. 2016b. Zero-resource translation with multi-lingual neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 268–277, Austin, Texas. Association for Computational Linguistics.
  • He et al. (2018) Tianyu He, Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, and Tie-Yan Liu. 2018. Layer-wise coordination between encoder and decoder for neural machine translation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 7955–7965. Curran Associates, Inc.
  • Johnson et al. (2017) Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339–351.
  • Kim et al. (2019) Yunsu Kim, Yingbo Gao, and Hermann Ney. 2019. Effective cross-lingual transfer of neural machine translation models without shared vocabularies. arXiv preprint arXiv:1905.05475.
  • Koehn (2005) Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit, volume 5, pages 79–86.
  • Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions, pages 177–180.
  • Lakew et al. (2018) Surafel M. Lakew, Aliia Erofeeva, Matteo Negri, Marcello Federico, and Marco Turchi. 2018. Transfer learning in multilingual neural machine translation with dynamic vocabulary. In Proceedings of the 15th International Workshop on Spoken Language Translation, pages 54–61, Belgium, Bruges.
  • Lample et al. (2018) Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations.
  • Lu et al. (2018) Yichao Lu, Phillip Keung, Faisal Ladhak, Vikas Bhardwaj, Shaonan Zhang, and Jason Sun. 2018. A neural interlingua for multilingual machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 84–92, Belgium, Brussels. Association for Computational Linguistics.
  • McInnes et al. (2018) Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. 2018. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861.
  • Schwenk and Douze (2017) Holger Schwenk and Matthijs Douze. 2017. Learning joint multilingual sentence representations with neural machine translation. arXiv preprint arXiv:1704.04154.
  • Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
  • Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
  • Ziemski et al. (2016) Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1. 0. In Lrec.
  • Zoph et al. (2016) Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575, Austin, Texas. Association for Computational Linguistics.

Appendix A Examples

This appendix shows some examples of sentences visualized in Figure 2 in order to further analyse the visualization. Table 1 reports outputs produced by the Spanish decoder given encoding representations produced by the Spanish, English, French and German encoder. The first two sentences have similar representations between the languages in Figure 2 (right) (in the Spanish decoder visualization). While the first one keeps the meaning of the sentence, the second one produces meaningless translations. The third sentence produces disjoint representations but the meaning is preserved in the translations. Therefore, since close representations may imply different translation performance, further research is required to understand the correlation between representations and translation quality.

System Sentence
Reference ponemos todo nuestro empeño en participar en este proyecto .
ES ponemos todo nuestro empeño en participar en este proyecto .
EN participamos con esfuerzo en estos proyctos .
FR nos esfuerzos por lograr que los participantes intensivamente en estos proyectos.
DE nuestro objetivo es incorporar estas personas de manera intensiva en nuestro proyecto.
Reference ¡Caja Libre!
ES ¡Caja Libre—
EN Free chash points!
FR librecorrespondinte.
DE cisiguinente
Reference ¿Cómo aplica esta definición en su vida cotidiana y en las redes sociales?
ES ¿Cómo aplica esta definición en su vida cotidiana y en las redes sociales?
EN ¿Cómo se aplica esta definición a su vida diaria?
FR ¿Cómo aplicar esta definición en la vida diaria y sobre los red sociales?
DE ¿Qué es aplicar este definición a su dadadato y las redes sociales?
Table 3: Outputs produced by the Spanish decoder given encoding representations produced by the Spanish, English, French and German encoder.
System Sentence
Reference it was a terrific season.
ES we had a strong season .
EN it was a terrific season.
FR we made a very big season .
DE we have finished the season with a very strong performance.
Reference in London and Madrid it is completely natural for people with serious handicaps to be independently out in public, and they can use the toilets, go to the museum, or wherever …
ES in London and Madrid , it is very normal for people with severe disability to be left to the public and be able to serve , to the museum , where …
EN in London and Madrid it is completely natural for people with serious handicaps to be independently out in public, and they can use the toilets, go to the museum, or wherever …
FR in London and Madrid, it is quite common for people with a heavy disability to travel on their own in public spaces; they can go to the toilets, to the museum, anywhere …
DE in London and Madrid, it is absolutely common for people with severe disabilities to be able to move freely in public spaces, go to the museum, use lets, etc.
Reference from the Czech viewpoint, it seems they tend to put me to the left.
ES from a Czech point of view, I have the impression that people see me more than on the left.
EN from the Czech viewpoint, it seems they tend to put me to the left.
FR from a Czech point of view , I have the impression that people are putting me on the left .
DE from a Czech point of view, it seems to me that people see me rather on the left.
Table 4: Outputs produced by the English decoder given encoding representations produced by the Spanish, English, French and German encoder.

Table 2 shows outputs produced by the English decoder given encoding representations produced by the Spanish, English, French and German encoder. All examples appear to be close in Figure 2 (center) between German, French and English. We see that the German and French outputs preserve the general meaning of the sentence. Also and differently from previous Table 1, the outputs do not present errors in the attention, repeating several times tokens or non unintelligible translations. There are no sentences from French that appear distant in the visualization, so again, we need further exploration to understand the information of this representation.

Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description