EEG-based Communication with a Predictive Text Algorithm
Several changes occur in the brain in response to voluntary and involuntary activities performed by a person. The ability to retrieve data from the brain within a time space provides basis for in-depth analyses that offer insight on what changes occur in the brain during its decision making processes. In this work, we present the technical description and software implementation of an electroencephalographic (EEG) based communication system. We use EEG dry sensors to read brain waves data in real-time with which we compute the likelihood that a voluntary eye blink has been made by a person and use the decision to trigger buttons on a user interface in order to produce text. Relevant texts are suggested using a modification of the T9 algorithm. Our results indicate that EEG-based technology can be effectively applied in facilitating speech for people with severe speech and muscular disabilities, providing a foundation for future work in the area.
Communication is a vital aspect of human life and it is as old as man, it is an indispensable tool for human co-existence. This has necessitated digital communication platforms such as chat applications, messaging applications, and social networks. duplaga identified a digital divide among the physically challenged people, this is obvious as many existing communication software require typing on the keyboard (for text messaging) or speaking to a microphone (for voice messaging) for transmission to a second party. The need for alternative means of communication that adequately includes those who suffer from severe speech and muscular disabilities is therefore paramount. roy on EEG data analysis showed that there is a link between eye blinks and the brain. We leverage on this create a prototype of a reliable and easy to use EEG based communication system for people who suffer from severe speech and muscular disabilities employing eye blinks as input modality. This has a great advantage over eye tracking cameras as cameras will always have to focus on the eye to track its gestures.
2 Related Work
Over the past decade, a wide range of Brain-Computer Interface (BCI) applications have emerged. According to lebedev , the classic goals of brain-computer interfaces are:
to unveil and utilize principles of operation and plastic properties of the distributed and dynamic circuits of the brain and,
to create new therapies to restore mobility and sensations to severely disabled patients.
Recent BCI applications considerably expanded the stated goals. The second goal stated above provides the basis for our work. nagels implemented a yes/no communication paradigm relying on mental imagery (mental drawing) and portable functional near-infrared spectroscopy. In their study, participants either performed mental drawing, to encode a ’yes’, or did not change their mental state, to encode a ’no’. Decoding accuracy reached 70% in almost half of the participants.
According to mackenzie , in a survey distributed to people with motor-neuron disease in New South Wales, 15%-21% had difficulty with speaking, writing and/or using a keyboard. Majority indicated a preference for portable solutions like laptops, tablets, mobile phones. A third of respondents used Skype or its equivalent, but few used this to interact with health professionals.
In usability study carried out by holz , a BCI prototype was evaluated in terms of effectiveness (accuracy), efficiency (Information Transfer Rate (ITR) and subjective workload) and users’ satisfaction by four severely motor restricted end-users; two were in the locked-in state and had unreliable eye-movement, sources for dissatisfaction were low effectiveness, time-consuming adjustment and difficulty of use. Although, all 4 users accepted the BCI application well, rated satisfaction medium to high and indicated ease of use as being one of the most important aspect of BCI.
3.1 EEG Sensors
Neurosky Mindwave headset also used in katona is used to capture brain signals from the human brain, a similar approach was discussed by joyce . The headset filters out extraneous noise and electrical interference and converts the signal to digital power which is continuously transmitted via Bluetooth network to the EEG application in the PC. The headset contains two dry EEG sensors, one is placed on the forehead and another is clipped to the right ear lobe.
3.2 Data transfer
In order to establish communication between the EEG application and the headset, a communication driver which takes COM Port and baud rate as arguments are used to established a communication link. Upon the selection of the appropriate COM Port and baud rate, the application is able to continuously fetch EEG parameters eeg which include important parameters like delta, alpha, beta, gamma, and blink strength.
3.3 Data processing
Blink strength parameter is continuously monitored for a deflection above a set threshold. We conclude a voluntary eye blink is made each moment the blink strength moves above the set threshold. The application contains two main features: text generation through key strokes and text generation by selecting from existing texts or predefined messages. A first blink is to select between the customized message panel and the compose text panel. In the customized message panel, message categories like home, office, hospital and frequently used texts are present and each category gains focus based on the specified selector speed. A further blink transfers focus to a specific category and each message in the category gains focus based on the specified speed, a further blink selects the message and the selected text is displayed in the message panel and read out as speech (using .NET speech synthesis library). The message is copied automatically to clipped board and the program switches automatically to Skype application where the message is automatically pasted and sent to a recipient.
3.4 Text generation:
A modification of the T9 algorithm (a predictive text algorithm) t9 which we call T6 algorithm is employed to help word suggestions. We call it T6 because text are generated using 6 keys as against 9 in order to ease the process. T6 uses the trie data structure for its implementation. Trie is tree data structure that associates each node with a character and the words that terminate at that node. It allows O(n) traversal to find words matching a T6 input (the depth of the tree corresponds with the length of the word). The T6 algorithm suggest words. Words suggestions in this implementation is the display of N number of words that have a T6 prefix matching the current input. For example, the T6 prefix for "apple" is 14431 (see figure 2c). Figure 1 is a diagram illustrating trie. In figure 1, traversing from root, words like THERE, THEIR, ANSWER, ANY, BYE can be generated.
Step 1: Build a trie (add all words from dictionary to it). Step 2:ΨInitially a current node is a root of the trie. Step 3:ΨWhen a new character is generated, simply go to the next node from the current node by the edge that corresponds to this character (or report an error if there is nowhere to go). Step 4:ΨGet all(or k first) possible words with a given prefix by traversing the trie in breadth first search order starting from the current node(if only k first words are needed, stop search when k words are found). Step 5:ΨWhen the entire word is typed in and a new word is started, move to the root again and repeat steps 3 - 5 for the next word.
Note: All nodes that correspond to a word (not a prefix of a word, but an entire word) is marked when the trie is built so that it is easy to understand whether a new word is found or not when traversing the trie at step 4.
In building the trie in step 1, a list of all words represented by a certain number are mapped to that number. This is done for few possible combination of numbers (and up to whatever length desired), such that when the user types in a number, the program simply retrieves the list associated with that number or key. A file containing every possibility is compiled before the start of the program. This makes searches fast and almost in constant time.
Breadth-First Search (BFS) is a tree/graph traversal algorithm implemented via a queue. BFS visits all children nodes of a parent node before traversing in depth. This property is useful in autocomplete because it allows same length words to be discovered before longer words. BFS algorithm is still widely used today bfs1 bfs2 bfs3 bfs4 .
In step 4 of the T6 algorithm above, an adaptation of BFS was used to find certain number of words.
Will return if it has explored maxSearch nodes or found numWords of words. T6BFSfunction(node): while(wordsList.length < numWords and queue.length > 0 and maxSearch > 0): Ψcurrent = queue.dequeue() foreach(current, function(el, key): queue.push(el)) if current.words: wordsList = wordsList.concat(current.words) end if ΨΨ maxSearch = maxSearch - 1 end while return wordsList end
A prototype of the system was designed which has a provision for COM port selection, baud rate selection in order to establish connection to the headset. It also made provision for selector/focus speed, and blink threshold value setting, the selector speed determines how long each control gains focus. The system was tested on two different users, the users were able to select a sentence from the frequently used panel and successfully sent it to a Skype contact withing a time space of 10 seconds. Both users were also able to generate the text "I will be late for work" using the simulated keypad and sent it to a Skype user within 31 seconds. See user interfaces below:
In this paper, a desktop based windows presentation foundation (WPF) application was developed to leverage on EEG signals to enhance communication for people who suffer from severe speech and muscular disabilities using eye blinks as input modality. The prototype was tested and showed evidences of enhancing communication for this category of people. Future work involves integrating in app chat functionality, reducing text generation time by seeking a better algorithm, and applying machine learning in predicting words during text generation.
- (1) Duplaga, Mariusz. "Digital divide among people with disabilities: analysis of data from a nationwide study for determinants of Internet use and activities performed online." PloS one 12, no. 6 (2017): e0179825.
- (2) Roy, Raphaëlle N., Sylvie Charbonnier, and Stephane Bonnet. "Eye blink characterization from frontal EEG electrodes using source separation and pattern recognition algorithms." Biomedical Signal Processing and Control 14 (2014): 256-264.
- (3) Lebedev, Mikhail A., and Miguel AL Nicolelis. "Brain-machine interfaces: From basic science to neuroprostheses and neurorehabilitation." Physiological reviews 97, no. 2 (2017): 767-837.
- (4) Nagels-Coune, L. M. J., D. Kurban, N. Reuter, A. Benitez, L. K. Gossé, L. Riecke, R. Goebel, and B. Sorger. "YES OR NO?–BINARY BRAIN-BASED COMMUNICATION UTILIZING MOTOR IMAGERY AND FNIRS." In Proceedings of the 7th Graz Brain-Computer Interface Conference. 2017.
- (5) Mackenzie, Lynette, Prarthna Bhuta, Kim Rusten, Janet Devine, Anna Love, and Penny Waterson. "Communications technology and motor neuron disease: an Australian survey of people with motor neuron disease." JMIR rehabilitation and assistive technologies 3, no. 1 (2016).
- (6) Holz, Elisa Mira, Johannes Höhne, Pit Staiger-Sälzer, Michael Tangermann, and Andrea Kübler. "Brain–computer interface controlled gaming: Evaluation of usability by severely motor restricted end-users." Artificial intelligence in medicine 59, no. 2 (2013): 111-120.
- (7) Joyce, Carrie A., Irina F. Gorodnitsky, and Marta Kutas. "Automatic removal of eye movement and blink artifacts from EEG data using blind component separation." Psychophysiology 41.2 (2004): 313-325.
- (8) Katona, Jozsef, et al. "Speed control of Festo Robotino mobile robot using NeuroSky MindWave EEG headset based brain-computer interface." Cognitive Infocommunications (CogInfoCom), 2016 7th IEEE International Conference on. IEEE, 2016.
- (9) Subha, D. Puthankattil, et al. "EEG signal analysis: a survey." Journal of medical systems 34.2 (2010): 195-212.
- (10) Silfverberg, Miika, I. Scott MacKenzie, and Panu Korhonen. "Predicting text entry speed on mobile phones." Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 2000.
- (11) Mitsuishi, Takuji, et al. "Breadth first search on cost-efficient multi-gpu systems." ACM SIGARCH Computer Architecture News 43.4 (2016): 58-63.
- (12) Zhang, Feng, et al. "An adaptive breadth-first search algorithm on integrated architectures." The Journal of Supercomputing (2018): 1-21.
- (13) Shi, Xuanhua, et al. "Graph processing on GPUs: A survey." ACM Computing Surveys (CSUR) 50.6 (2018): 81.
- (14) Shatnawi, Anas, et al. "A Static Program Slicing Approach for Output Stream Objects in JEE Applications." arXiv preprint arXiv:1803.05260 (2018).