Quaternion Convolutional Neural Networks for EndtoEnd Automatic Speech Recognition
Abstract
Recently, the connectionist temporal classification (CTC) model coupled with recurrent (RNN) or convolutional neural networks (CNN), made it easier to train speech recognition systems in an endtoend fashion. However in realvalued models, time frame components such as melfilterbank energies and the cepstral coefficients obtained from them, together with their first and second order derivatives, are processed as individual elements, while a natural alternative is to process such components as composed entities. We propose to group such elements in the form of quaternions and to process these quaternions using the established quaternion algebra. Quaternion numbers and quaternion neural networks have shown their efficiency to process multidimensional inputs as entities, to encode internal dependencies, and to solve many tasks with less learning parameters than realvalued models. This paper proposes to integrate multiple feature views in quaternionvalued convolutional neural network (QCNN), to be used for sequencetosequence mapping with the CTC model. Promising results are reported using simple QCNNs in phoneme recognition experiments with the TIMIT corpus. More precisely, QCNNs obtain a lower phoneme error rate (PER) with less learning parameters than a competing model based on realvalued CNNs.
Titouan Parcollet, Ying Zhang , Mohamed Morchid, Chiheb Trabelsi, Georges Linarès,
Renato De Mori and Yoshua Bengio
Université de Montréal, MILA, Canada
McGill University, Montréal, Canada
Orkis, Aix en provence, France
Element AI, Montréal, Canada
\emailtitouan.parcollet@alumni.univavignon.fr, ying.zhlisa@gmail.com, mohamed.morchid@univavignon.fr, chiheb.trabelsi@polymtl.ca, georges.linares@univavignon.fr, rdemori@cs.mcgill.ca
Index Terms: quaternion convolutional neural networks, automatic speech recognition, deep learning
1 Introduction
Recurrent (RNN) and convolutional (CNN) neural networks have improved the performance over hidden Markov models (HMM) combined with gaussian mixtures models (GMMs) in automatic speech recognition (ASR) systems [1, 2, 3, 4, 5] during the last decade.
More recently, endtoend approaches received a growing interest due to the promising results obtained with connectionist temporal classification (CTC) [6] combined with RNNs [1] or CNNs [7].
However, despite such evolution of models and paradigms, the acoustic features remain almost the same. The main motivation is that filters spaced linearly at low frequencies and logarithmically at high frequencies make it possible to capture phonetically important acoustic correlates. Early evidence was provided in [8] showing that mel frequency scaled cepstral coefficients (MFCCs) are effective in capturing the acoustic information required to recognize syllables in continuous speech.
Motivated by these analysis, a small number of MFCCs (usually ) with their first and second timederivatives, as proposed in [9], have been found suited for statistical and neural ASR systems. In most systems, a time frame of the speech signal is represented by a vector with realvalued elements that express sequences of MFCCs, or filter energies, and their temporal context features. A concern addressed in this paper, is the fact that the relations between different views of the features associated with a frequency are not explicitly represented in the feature vectors used so far.
Therefore, this paper proposes to:

Introduce a new quaternion representation (Section 2) to encode multiple views of a timeframe frequency in which different views are encoded as values of imaginary parts of a hypercomplex number. Thus, vectors of quaternions are embedded using operations defined by a specific quaternion algebra to preserve a distinction between features of each frequency representation.

Merge a quaternion convolutional neural network (QCNN, Section 3) with the CTC in a unified and easily reusable framework
^{2} .
There are advantages which could derive from bundling groups of numbers into a quaternion. Like capsule networks [11], quaternion networks create a tighter association between small groups of numbers rather than having one homogeneous representation. In addition, this kind of structure reduces the number of required parameters considerably, because only one weight is necessary between two quaternion units, instead of 4. The hypothesis tested here is whether these advantages lead to better generalization. The conducted experiments on the TIMIT dataset yielded a phoneme error rate (PER) of % for QCNNs which is significantly lower than the PER obtained with realvalued CNNs (%), with the same input features. Moreover, from a practical point of view, the resulting networks have a considerably smaller memory footprint due to a smaller set of parameters.
2 Quaternion algebra
The quaternions algebra defines operations between quaternion numbers. A quaternion Q is an extension of a complex number defined in a four dimensional space. , with, r, x, y, and z four real numbers, and , i, j, and k are the quaternion unit basis. Such a definition can be used for describing spatial rotations that can also be represented by the following matrix of real numbers:
(1) 
In a quaternion, is the real part while is the imaginary part () or the vector part. Basic quaternion definitions are

all products of ,k are: ,

conjugate of is: ,

unit quaternion ,

the Hamilton product between and is defined as follows:
The Hamilton product is used in QCNNs to perform transformations of vectors representing quaternions, as well as scaling and interpolation between two rotations following a geodesic over a sphere in the space as shown in [12].
3 Quaternion convolutional neural networks
This section defines the internal quaternion representation (Section 3.1), the quaternion convolution (Section 3.2), a proper parameter initialization (Section 3.3), and the connectionist temporal classification (Section 3.4).
3.1 Quaternion internal representation
The QCNN is a quaternion extension of wellknown realvalued and complexvalued deep convolutional networks (CNN) [13, 14]. The quaternion algebra is ensured by manipulating matrices of real numbers. Consequently, a traditional convolutional layer, with a kernel that contains feature maps, is split into 4 parts: the first part equal to , the second one to , the third one to and the last one to of a quaternion . Nonetheless, an important condition to perform backpropagation in either real, complex or quaternion neural networks is to have cost and activation functions that are differentiable with respect to each part of the real, complex or quaternion number. Many activation functions for quaternion have been investigated [15] and a quaternion backpropagation algorithm have been proposed in [16]. Consequently, the split activation [17, 18] function is applied to every layer and is defined as follows:
(2) 
with corresponding to any standard activation function.
3.2 Quaternionvalued convolution
Following a recent proposition for convolution of complex numbers[14] and quaternions [19], this paper presents basic neural networks convolution operations using quaternion algebra. The convolution process is defined in the realvalued space by convolving a filter matrix with a vector. In a QCNN, the convolution of a quaternion filter matrix with a quaternion vector is performed. For this computation, the Hamilton product is computed using the realvalued matrices representation of quaternions. Let be a quaternion weight filter matrix, and the quaternion input vector. The quaternion convolution w.r.t the Hamilton product is defined as follows:
(3) 
and can thus be expressed in a matrix form:
(4) 
An illustration of such operation is depicted in Figure 1.
3.3 Weight initialization
Weight initialization is crucial to efficiently train neural networks. An appropriate initialization improves training speed and reduces the risk of exploding or vanishing gradient. A quaternion initialization is composed of two steps. First, for each weight to be initialized, a purely imaginary quaternion is generated following an uniform distribution in the interval . The imaginary unit is then normalized to obtain following the quaternion normalization equation. The later is used alongside to other well known initializing criterion such as [20] or [21] to complete the initialization process of a given quaternion weight named . Moreover, the generated weight has a polar form defined by :
(5) 
with
(6) 
Therefore, is generated as follows:

,

,

,

.
However, represents a randomly generated variable with respect to the variance of the quaternion weight and the selected initialization criterion. The initialization process follows [20] and [21] to derive the variance of the quaternionvalued weight parameters. Therefore, the variance of W has to be investigated:
(7) 
is equals to since the weight distribution is symmetric around . Nonetheless, the value of is not trivial in the case of quaternionvalued matrices. Indeed, follows a Chidistributed with four degrees of freedom (DOFs) and is expressed and computed as follows:
(8) 
Therefore, in order to respect the He Criterion [21], the variance would be equal to:
(9) 
3.4 Connectionist Temporal Classification
In the acoustic modeling part of ASR systems, the task of sequencetosequence mapping from an input acoustic signal to a sequence of symbols is complex due to:

and could be in arbitrary length.

The alignment between and is unknown in most cases.
Specially, is usually shorter than in terms of phoneme symbols.
To alleviate these problems, connectionist temporal classification (CTC) has been proposed [6]. First, a softmax is applied at each timestep, or frame, providing a probability of emitting each symbol at that timestep. This probability results in a symbol sequences representation , with in the latent space . A blank symbol is introduced as an extra label to allow the classifier to deal with the unknown alignment. Then, is transformed to the final output sequence with a manytoone function defined as follows:
(10) 
Consequently, the output sequence is a summation over the probability of all possible alignments between and after applying the function . Accordingly to [6] the parameters of the models are learned based on the cross entropy loss function:
(11) 
During the inference, a best path decoding algorithm is performed. Therefore, the latent sequence with the highest probability is obtained by performing argmax of the softmax output at each timestep. The final sequence is obtained by applying the function to the latent sequence.
4 Experiments
The performance and efficiency of the proposed QCNNs is evaluated on a phoneme recognition task. This section provides details on the dataset and the quaternion features representation (Section 4.1), the models configurations (Section 4.2), and finally a discussion of the observed results (Section 4.3).
4.1 TIMIT dataset and acoustic features of quaternions
The TIMIT [10] dataset is composed of a standard 462speaker training dataset, a 50speakers development dataset and a core test dataset of sentences. During the experiments, the SA records of the training set are removed and the development set is used for early stopping. The raw audio is transformed into dimensional log melfilterbank coefficients with deltas, deltadeltas, and energy terms, resulting in a one dimensional vector of length . An acoustic quaternion associated with a frequency and a time frame is defined as follows:
(12) 
It represents multiple views of a frequency at time frame , consisting of the energy in the filter band corresponding to , its first time derivative describing a slope view, and its second time derivative describing a concavity view. Finally, a unique quaternion is composed with the three corresponding energy terms. Thus, the quaternion input vector length is ().
4.2 Models architectures
The architectures of both CNN and QCNN models are inspired by [7]. A first D convolutional layer is followed by a maxpooling layer along the frequency axis. Then, D convolutional layers are included, together with dense layers of sizes and respectively for real and quaternionvalued models (with ). Indeed, the output of a dense quaternionvalued layer has nodes and is times larger than the number of units. The filter size is rectangular , and a padding is applied to keep the sequence and signal sizes unaltered. The number of feature maps varies from to for the realvalued models and from to for quaternionvalued models. Indeed, the number of output feature maps is times larger in the QCNN due to the quaternion convolution, meaning quaternionvalued feature maps correspond to realvalued ones. The PReLU activation function is employed for both models [21]. A dropout of and a regularization of are used across all the layers, except the input and output ones. CNNs and QCNNs are trained with the Adam learning rate optimizer and vanilla hyperparameters [22] during epochs. Then, a finetuning process of epochs is performed with a standard and a learning rate of . Finally, the standard CTC loss function defined in [6] and implemented in [23] is applied. Experiments are performed on Tesla P100 and Geforce Titan X GPUs.
4.3 Results and discussion
Results on the phoneme recognition task of the TIMIT dataset are reported in Table 1. It is worth noticing the important difference in terms of the number of learning parameters between real and quaternion valued CNNs. It is easily explained by the quaternion algebra. In the case of a dense layer with input values and hidden units, a realvalued model will have M parameters, while to maintain equal input and output nodes () the quaternion equivalent has quaternions inputs and quaternionvalued hidden units. Therefore the number of parameters for the quaternion model is M. Such a complexity reduction turns out to produce better results and may have other advantages such as a smallest memory footprint while saving NN models. Moreover, the reduction of the number of parameters does not result in poor performance in the QCNN. Indeed, the best PER reported is % from a QCNN with feature maps and layers, compared to a PER of % for a realvalued CNN with feature maps and layers. It is worth underlying that both model accuracies are increasing with the size and the depth of the neural network. However, bigger realvalued feature maps leads to overfitting. In fact, as shown in Table 1, the best PER for a realvalued model is reached with () feature maps and decreasing at (%) and (). The QCNN does not suffer from such weaknesses due to the smaller density of the neural network and achieved a constant PER improvement alongside with the increasing number of feature maps. Furthermore, QCNNs always performed better than CNNs independently of the model topologies.
Models  Dev PER %  Test PER %  Params 

CNN6L32FM  22.18  23.54  3.3M 
QCNN6L32FM  22.16  23.20  0.87M 
CNN10L32FM  21.77  23.43  3.4M 
QCNN10L32FM  22.25  23.23  0.9M 
CNN6L64FM  21.19  22.12  4.8M 
QCNN6L64FM  21.44  21.99  1.2M 
CNN10L64FM  19.53  20,57  5.4M 
QCNN10L64FM  19.78  20.44  1.4M 
CNN6L128FM  20.33  22.14  9M 
QCNN6L128FM  20.12  21.33  2.3M 
CNN10L128FM  19.37  20.62  11.5M 
QCNN10L128FM  19.02  19.87  2.9M 
CNN6L256FM  20.43  22.25  22.3M 
QCNN6L256FM  19.94  20.54  5.6M 
CNN10L256FM  18.89  21.23  32.1M 
QCNN10L256FM  18.33  19.64  8.1M 
With much fewer learning parameters for a given architecture, the QCNN performs always better than the realvalued one on the reported task. In terms of PER, an average relative gain of % (w.r.t CNNs result) is obtained on the testing set. It is also worth recalling that the best PER of % is obtained with just a QCNN without HMMs, RNNs, attention mechanisms, batch normalization, phoneme language model, acoustic data normalization or adaptation. Further improvements can be obtained with exactly the same QCNN by just introducing a new acoustic feature in the real part of the quaternions.
5 Related work
Early attempts to perform phoneme and phonetic feature recognition with multilayer perceptrons (MLP) were proposed in [24, 25, 26]. A PER of % is reported in [25] using RNNs. More recently, in [27] a MeanCovariance Restricted Boltzmann Machine (RBM) is used for recognizing phonemes in the TIMIT corpus using RBM for feature extraction. Along this line of research, in [6] an approach called the Connectionist Temporal Classification (CTC) has been developed and can be used without an explicit inputoutput alignment. Bidirectional RNNs (BRNNs) are used in [28] for processing input data in both directions with two separate hidden layers, which are then composed in an output layer. With standard mel frequency energies, first and second time derivatives a PER of % was obtained. Other recent results with realvalued vectors of similar features are reported in [29, 4, 30, 31]. Other types of quaternion valued neural networks (QNNs) were introduced for encoding RGB color relations in image pixels [32, 33, 34], and for classifying human/human conversation topics [35, 36, 18]. A quaternion deep convolutional and residual neural network proposed in [19] have shown impressive results on the CIFAR images classification task. However, a specific quaternion is used for each RGB color value as in [14] rather than integrating pixel multiple views as in [37], and suggested in this paper for an ASR task.
6 Conclusions
Summary.
This paper proposes to integrate multiple acoustic feature views with quaternion hyper complex numbers, and to process these features with a convolutional neural network of quaternions. The phoneme recognition experiments have shown that: 1) Given an equivalent architecture, QCNNs always outperform CNNs with significantly less parameters; 2) QCNNs obtain better results than CNNs with a similar number of learning parameters; 3) The best result obtained with QCNNs is better than the one observed with the realvalued counterpart. This demonstrates the initial intuition that the capability of the Hamilton product to learn internal latent relations helps quaternionsvalued neural networks to achieve better results .
Limitations and Future Work.
So far, traditional acoustic features, such as mel filter bank energies, first and second derivatives have shown that significantly good results can be obtained with a relative small set of input features for a speech time frame. Nevertheless, speech science has shown that other multiview contextdependent acoustic relations characterize signals of phonemes in context.
Future work will attempt to characterize those multiview features that mostly contribute to reduce ambiguities in representing phoneme events. Furthermore, quaternionsvalued RNNs will also be investigated to see if they can contribute to the improvement of recently achieved top of the line results with real number RNNs.
7 Acknowledgements
The experiments were conducted using Keras [23]. The authors would like to acknowledge the computing support of Compute Canada and the founding support of Orkis, NSERC, Samsung, IBM and CHISTERA/FRQ. The authors would like to thank Kyle Kastner and Mirco Ravanelli for their helpful comments.
Footnotes
 thanks: CIFAR Senior Fellow
 The full code is available at https://git.io/vx8so
References
 H. Sak, A. Senior, and F. Beaufays, “Long shortterm memory recurrent neural network architectures for large scale acoustic modeling,” in Fifteenth annual conference of the international speech communication association, 2014.
 G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath et al., “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” Signal Processing Magazine, IEEE, vol. 29, no. 6, pp. 82–97, 2012.
 O. AbdelHamid, A.r. Mohamed, H. Jiang, and G. Penn, “Applying convolutional neural networks concepts to hybrid nnhmm model for speech recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on. IEEE, 2012, pp. 4277–4280.
 M. Ravanelli, P. Brakel, M. Omologo, and Y. Bengio, “Improving speech recognition by revising gated recurrent units,” Proc. Interspeech 2017, 2017.
 K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, and J. Schmidhuber, “Lstm: A search space odyssey,” IEEE transactions on neural networks and learning systems, vol. 28, no. 10, pp. 2222–2232, 2017.
 A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd international conference on Machine learning. ACM, 2006, pp. 369–376.
 Y. Zhang, M. Pezeshki, P. Brakel, S. Zhang, C. L. Y. Bengio, and A. Courville, “Towards endtoend speech recognition with deep convolutional neural networks,” arXiv preprint arXiv:1701.02720, 2017.
 S. B. Davis and P. Mermelstein, “Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences,” in Readings in speech recognition. Elsevier, 1990, pp. 65–74.
 S. Furui, “Speakerindependent isolated word recognition based on emphasized spectral dynamics,” in Acoustics, Speech, and Signal Processing, IEEE International Conference on ICASSP’86., vol. 11. IEEE, 1986, pp. 1991–1994.
 J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, and D. S. Pallett, “Darpa timit acousticphonetic continous speech corpus cdrom. nist speech disc 11.1,” NASA STI/Recon technical report n, vol. 93, 1993.
 S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing between capsules,” arXiv preprint arXiv:1710.09829v2, 2017.
 T. Minemoto, T. Isokawa, H. Nishimura, and N. Matsui, “Feed forward neural network with random quaternionic neurons,” Signal Processing, vol. 136, pp. 59–68, 2017.
 K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
 C. Trabelsi, O. Bilaniuk, D. Serdyuk, S. Subramanian, J. F. Santos, S. Mehri, N. Rostamzadeh, Y. Bengio, and C. J. Pal, “Deep complex networks,” arXiv preprint arXiv:1705.09792, 2017.
 D. Xu, L. Zhang, and H. Zhang, “Learning algorithms in quaternion neural networks using ghr calculus,” Neural Network World, vol. 27, no. 3, p. 271, 2017.
 T. Nitta, “A quaternary version of the backpropagation algorithm,” in Neural Networks, 1995. Proceedings., IEEE International Conference on, vol. 5. IEEE, 1995, pp. 2753–2756.
 P. Arena, L. Fortuna, L. Occhipinti, and M. G. Xibilia, “Neural networks for quaternionvalued function approximation,” in Circuits and Systems, 1994. ISCAS’94., 1994 IEEE International Symposium on, vol. 6. IEEE, 1994, pp. 307–310.
 T. Parcollet, M. Morchid, P.M. Bousquet, R. Dufour, G. Linarès, and R. De Mori, “Quaternion neural networks for spoken language understanding,” in Spoken Language Technology Workshop (SLT), 2016 IEEE. IEEE, 2016, pp. 362–368.
 A. M. Chase Gaudet, “Deep quaternion networks,” arXiv preprint arXiv:1712.04604v2, 2017.
 X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in International conference on artificial intelligence and statistics, 2010, pp. 249–256.
 K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1026–1034.
 D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
 F. Chollet et al., “Keras,” https://github.com/kerasteam/keras, 2015.
 H. Bourlard and S. Dupont, “A mew asr approach based on independent processing and recombination of partial frequency bands,” in Spoken Language, 1996. ICSLP 96. Proceedings., Fourth International Conference on, vol. 1. IEEE, 1996, pp. 426–429.
 A. J. Robinson, “An application of recurrent nets to phone probability estimation,” IEEE transactions on Neural Networks, vol. 5, no. 2, pp. 298–305, 1994.
 Y. Bengio, R. De Mori, G. Flammia, and R. Kompe, “Global optimization of a neural networkhidden markov model hybrid,” IEEE transactions on Neural Networks, vol. 3, no. 2, pp. 252–259, 1992.
 G. Dahl, A.r. Mohamed, G. E. Hinton et al., “Phone recognition with the meancovariance restricted boltzmann machine,” in Advances in neural information processing systems, 2010, pp. 469–477.
 A. Graves, A.r. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in Acoustics, speech and signal processing (icassp), 2013 ieee international conference on. IEEE, 2013, pp. 6645–6649.
 J. Chorowski, D. Bahdanau, K. Cho, and Y. Bengio, “Endtoend continuous speech recognition using attentionbased recurrent nn: First results,” arXiv preprint arXiv:1412.1602, 2014.
 L. Lu, L. Kong, C. Dyer, N. A. Smith, and S. Renals, “Segmental recurrent neural networks for endtoend speech recognition,” arXiv preprint arXiv:1603.00223, 2016.
 L. Lu, L. Kong, C. Dyer, and N. A. Smith, “Multitask learning with ctc and segmental crf for speech recognition,” arXiv preprint arXiv:1702.06378, 2017.
 Y.Z. Hsiao and S.C. Pei, “Edge detection, color quantization, segmentation, texture removal, and noise reduction of color image using quaternion iterative filtering,” Journal of Electronic Imaging, vol. 23, no. 4, p. 043001, 2014.
 B. Chen, H. Shu, G. Coatrieux, G. Chen, X. Sun, and J. L. Coatrieux, “Color image analysis by quaterniontype moments,” Journal of mathematical imaging and vision, vol. 51, no. 1, pp. 124–144, 2015.
 M. A. Garg and M. S. Goyal, “Vector sparse representation of color image using quaternion matrix analysis based on genetic algorithm,” Imperial Journal of Interdisciplinary Research, vol. 3, no. 7, 2017.
 T. Parcollet, M. Morchid, and G. Linares, “Deep quaternion neural networks for spoken language understanding,” in Automatic Speech Recognition and Understanding Workshop (ASRU), 2017 IEEE. IEEE, 2017, pp. 504–511.
 P. Titouan, M. Morchid, and G. Linares, “Quaternion denoising encoderdecoder for theme identification of telephone conversations,” Proc. Interspeech 2017, pp. 3325–3328, 2017.
 H. Kusamichi, T. Isokawa, N. Matsui, Y. Ogawa, and K. Maeda, “A new scheme for color night vision by quaternion neural network,” in Proceedings of the 2nd International Conference on Autonomous Robots and Agents, vol. 1315, 2004.