BrainNet: A Multi-Person Brain-to-Brain Interface for Direct Collaboration Between Brains

BrainNet: A Multi-Person Brain-to-Brain Interface for Direct Collaboration Between Brains

Linxing Jiang Andrea Stocco Darby M. Losey Justin A. Abernethy Chantel S. Prat Rajesh P. N. Rao
Abstract

We present BrainNet which, to our knowledge, is the first multi-person non-invasive direct brain-to-brain interface for collaborative problem solving. The interface combines electroencephalography (EEG) to record brain signals and transcranial magnetic stimulation (TMS) to deliver information noninvasively to the brain. The interface allows three human subjects to collaborate and solve a task using direct brain-to-brain communication. Two of the three subjects are designated as “Senders” whose brain signals are decoded using real-time EEG data analysis. The decoding process extracts each Sender’s decision about whether to rotate a block in a Tetris-like game before it is dropped to fill a line. The Senders’ decisions are transmitted via the Internet to the brain of a third subject, the “Receiver,” who cannot see the game screen. The Senders’ decisions are delivered to the Receiver’s brain via magnetic stimulation of the occipital cortex. The Receiver integrates the information received from the two Senders and makes a decision using an EEG interface about either turning the block or keeping it in the same position. A second round of the game provides an additional chance for the Senders to evaluate the Receiver’s decision and send feedback to the Receiver’s brain, and for the Receiver to rectify a possible incorrect decision made in the first round. We evaluated the performance of BrainNet in terms of (1) Group-level performance during the game; (2) True/False positive rates of subjects’ decisions; (3) Mutual information between subjects. Five groups, each with three human subjects, successfully used BrainNet to perform the Tetris task, with an average accuracy of 81.25%. Furthermore, by varying the information reliability of the Senders by artificially injecting noise into one Sender’s signal, we investigated how the Receiver learns to integrate noisy signals in order to make a correct decision. We found that Receivers are able to learn which Sender is more reliable based solely on the information transmitted to their brains. Our results raise the possibility of future brain-to-brain interfaces that enable cooperative problem solving by humans using a “social network” of connected brains.

Introduction

Direct brain-to-brain interfaces (BBIs) in humans[1, 2, 3, 4] are interfaces which combine neuroimaging and neurostimulation methods to extract and deliver information between brains, allowing direct brain-to-brain communication. A BBI extracts specific content from the neural signals of a “Sender” brain, digitizes it, and delivers it to a “Receiver” brain. Because of ethical and safety considerations, existing human BBIs rely on non-invasive technologies, typically electroencephalography (EEG), to record neural activity and transcranial magnetic stimulation (TMS) to deliver information to the brain. For example, the first human BBI by Rao and colleagues[1] decoded motor intention signals using EEG in the Sender and conveyed the intention via TMS directly to the motor cortex of the Receiver to complete a visual-motor task. Stocco and colleagues[4] extended these results by showing that a Sender and a Receiver can iteratively exchange information using a BBI to identify an unknown object from a list, using a question-and-answer paradigm akin to “20 Questions” (see also [3] for a related but offline non-iterative BBI).

Early interest in human BBIs came from the potential for expanding human communication and social interaction capabilities [5, 6, 7, 8]. However, previous BBIs have lacked several key features of real-world human communication. First, the degree of interactivity has been minimal; for example, in the case of the “20 Questions” BBI [4], the Sender only responds to the question the Receiver chooses, and the Receiver’s performance does not affect the Sender’s decision. Second, their interface required physical action: the Receiver touched the screen to select a question. Thus, the communication loop was completed via a motor output channel rather than a brain interface. Third, all past human BBIs have only allowed two subjects. Human communication, on the other hand, has become increasingly dominated by means (such as social media) that allow multiple parties to interact in a network. The potential for BBIs allowing interactions between multiple subjects has previously been theorized[2, 9] but not demonstrated.

Here, we present BrainNet (Figure 1), a next-generation BBI that addresses many of the limitations of past BBIs. First, BrainNet is designed to be a BBI for more than two human subjects; its current implementation allows two Senders and one Receiver to communicate, but it can be readily scaled up to include multiple Senders. The two Senders have the same role in observing the current state of the task and conveying their decisions to the Receiver. The Receiver has the role of integrating these two independent decisions and deciding on a course of action. Second, BrainNet’s design incorporates a second round of interactions between the Senders and the Receiver, so that the action of the Receiver in the first round can be perceived by the Senders, giving them a second chance to convey (potentially corrective) decisions to the Receiver. Third, the Receiver is equipped with both TMS (to receive Senders’ decisions) and EEG (to perform an action in the task), thereby completely eliminating the need to use any physical movements to convey information. We report results from five groups, each with three human subjects (henceforth, “triad”), who successfully used BrainNet to perform a collaborative task based on a Tetris-like game.

An important feature of communication in social networks is deciding which source of information to prioritize [10]. To investigate whether BrainNet allows such a capability, we additionally explored whether the Receiver can learn the reliability of each Sender over the course of their brain-to-brain interactions. We varied the reliability of the signal from one Sender compared to the other by injecting noise into the signals from one randomly chosen Sender. Our results show that like conventional social networks, BrainNet allows a Receiver to learn to trust the Sender who is more reliable, i.e., whose signal quality is not affected by our manipulation.

Results

To measure the direct brain-to-brain communication capabilities of BrainNet, we asked each triad of participants to perform 16 trials of an iterative Tetris-like game. In each trial, one participant, designated as the Receiver, is in charge of deciding whether or not to rotate a block before it drops to fill a gap in a line at the bottom of the screen. Critically, the Receiver is prevented from seeing the bottom part of the screen and must rely on the counsel of the other two participants, designated as the Senders, who can see the screen in its entirety. These Senders are tasked with making the correct decision (rotate or not) based on the shape of the current block and the gap at the bottom, and informing the Receiver of the decision via the brain-to-brain interface. All members of the triad communicate their decisions through an EEG-based interface using steady state visually evoked potentials (SSVEPs) (see Methods). The Senders’ decisions are delivered to the Receiver through sequential TMS pulses to the occipital cortex, indicating a "yes" or "no" rotation decision (see Methods). Each trial is composed of two rounds: the first round is as described above; after the first round, the Senders are given the opportunity to examine the Receiver’s decision (shown on their screen as the block, now potentially rotated, mid-way through its fall) and have another chance to make new (possibly corrective) suggestions. A successful completion of a trial thus requires accurate communication between the Receiver and the Senders across these two rounds (See Figure 2). Further, to examine the issue of reliability of Senders, our software randomly choses one Sender to be less reliable by making the decision sent to the Receiver from that Sender incorrect ten out of sixteen trials. The order of trials requiring the block to be rotated and trials not requiring rotation was randomized, with the constraint that each half of the session contained 4 rotation and 4 non-rotation trials. Trials 8-12 for the first triad were excluded from all analysis due to a problem with the timestamp routine. We analyzed both the EEG and behavioral data from the subjects in the rest trials.

Overall Performance

The simplest measure of overall performance of the interface is the proportion of correct block rotations (or, equivalently, the proportion of number of lines cleared, or the proportion of the maximum theoretical total score, i.e. 16 points, achieved) for each of the five triads of participants. Figure 3 shows the results. The mean accuracy across all triads was 0.8125, which is significantly higher than the performance expected by chance (one-sample -test, (4) = 7.91, = .001).

Another important metric is the mean performance of participants in the SSVEP task since all three participants in each triad had to use this method to share information. In the task, subjects focused their attention on a 17 Hz flashing LED to indicate a "rotate" decision and a 15 Hz flashing LED for a "do not rotate" decision. Figure 4 shows that before and after the SSVEP task, the 17 Hz and 15 Hz average power values overlap, whereas during the task, the average power of the frequency corresponding to the correct answer in the trial is significantly larger than that of the frequency corresponding to the wrong answer (two-sided t-test; (15) = 9.709, < 0.0001 for "Rotate" signal; = 10.725, < 0.0001 for "Do Not Rotate" signal). Since our SSVEP classifier compares the magnitude of power values to decode a Sender’s decision, the large difference in power values implies good performance for our EEG-based brain interfaces.

As noted in previous studies from our group [1, 4, 11], accuracy can be an inadequate measure of performance because it does not differentiate the kind of mistakes being made, i.e., whether they are misses or false positives. A better measure of performance can be obtained by calculating each triad’s Receiver Operating Characteristic (ROC) curve[12], which plots the True Positive Rate versus the False Positive rate, and calculating the area under this curve (AUC) (Figure 5). Responding uniformly at random yields an AUC of 0.5 while the AUC of an ideal observer is 1.0. The mean AUC across all triads of participants was 0.83, significantly higher than the performance expected by chance (one-sample -test, (4) = 9.00, <.001).

As Figure 5 shows, the overall AUC value for each network of brains (i.e., each triad) is affected by the bad Sender’s performance, but not by much. The overall AUC values are significantly different from the AUC values of both good Senders (two-sided -test, , = 0.02) and bad Senders (two-sided -test, , < 0.0001). The overall network AUC values are also significantly different from that for chance performance (one-sample -test, (4) = 9.00, < .00081)

Mutual Information between Subjects

An important measure of a brain-to-brain interface is the mutual information (MI)[13] transmitted between subjects, which is defined as:

where represents a decision made by the Receiver (0 or 1 corresponding to "do not rotate" or "rotate"), represents a decision made by one of the Senders, represents the probability of the Receiver making a decision , represents the probability of one of the Senders making a decision , and represents the joint probability of the Receiver making a decision and a Sender making a decision . Note that chance performance corresponds to while perfect communication corresponds to .

Due to our experimental design, we expect significantly higher values (that is, larger amounts of information being transferred) between a good Sender and the Receiver than between a bad Sender and the Receiver. This is corroborated by our results (Figure 6).

For both types of Senders, the information transmitted was significantly greater than the MI for chance performance (Good Senders: = 0.336, (4) = 5.374, = .005; Bad Senders: = , (4) = 3.543, =.023). The difference between good and bad Senders was also statistically significant (two-sided t-test, (8) = 4.44, = .002), with the good Senders transmitting, on average, more information than the bad Senders.

Learning of Sender Reliability by Receiver

The differences in accuracy and mutual information between the "good" and "bad" Senders in the previous section suggest that the Receiver successfully learned which of the two Senders is a more reliable source of information. Confirming that this is indeed the case would bring BrainNet a step closer to conventional social networks where users utilize differential weighting for different sources of information. To further investigate this issue, we divided each experimental session into four consecutive blocks of four trials each. We quantified the time course of the Receiver’s learning process using two measures, (1) block-by-block estimates of the linear regression weights () for the Receiver’s decisions versus each Sender’s decisions; and (2) the block-by-block correlation of decisions made by the Receiver and by each Sender [1]. Because of the small number of trials () in each block, the decision vectors for Senders and Receivers were created by concatenating the decisions of participants belonging to the same role (Receiver, good Sender, or bad Sender) across the five triads; this procedure captures group-level behavior and is less sensitive to outliers. Thus, if is a decision vector for the five Receivers in a four-trials block (each decision is encoded as a 0 or 1), and is a decision vector for one type of Senders ("good" or "bad"), the linear regression weights can be estimated using the standard pseudoinverse method[14] as: . The same concatenated vectors and were also used to estimate the Pearson correlation coefficients for each four-trials block.

As shown in Figure 7, the time course of both the beta weights and correlation coefficients show a steep ascending trend for the good Sender, but not for the bad Sender. To test the difference between these trends, we estimated two simple linear trend models of the relationship between each measure and the block number, one for the good Sender and one for the bad Sender. The difference between the linear trend model’s slope coefficients and for the good and bad Senders respectively was then tested for statistical significance using the exact formula derived by Paternoster and colleagues [15]:

where and are the variances of and , respectively. The difference in linear trends was statistically significant for both measures (beta weight measure: Z = 5.87, < 0.001; correlation coefficient measure: Z = 7.31, < 0.001). These results strongly suggest that Receivers were able to learn which Sender was more reliable during the course of their brain-to-brain interactions with the two Senders.

Discussion

This paper presents, to our knowledge, the first successful demonstration of multi-person non-invasive direct brain-to-brain interaction for solving a task. We believe our brain-to-brain interface, which we call BrainNet, improves upon previous human brain-to-brain interfaces (BBIs) on three fronts: (1) BrainNet expands the scale of BBIs to multiple human subjects working collaboratively to solve a task; (2) BrianNet is the first BBI to combine brain recording (EEG) and brain stimulation (TMS) in a single human subject. With sufficient hardware, our system can be scaled to the case where every subject can both send and receive information using the brain interface; (3) Using only the information delivered by BrainNet, Receivers are able to learn to differentiate the reliability of information conveyed to their brains by other subjects and choose the more credible sender. This makes the information exchange mediated by BrainNet similar to real-life social communication, bringing us a step closer to a "social network of brains."

Our results on combining information from multiple users builds on previous work in the field of brain-computer interfaces (BCIs) linking the individual contributions of more than two brains to control a computer. In humans, researchers have studied “collaborative BCIs” (rather than BBIs) that pool information from multiple human brains to improve performance in a Go-NoGo decision making task[16]; no brain stimulation was used in this study to convey information directly to subjects’ brains. Similarly, a recent study [17] demonstrated the possibility of three non-human primates jointly controlling a 3D arm model using brain signals recorded using invasive implanted electrodes in the motor cortex; again, the goal was distributing a single task across multiple individuals linked to a common BCI without encoding any neural information for feedback via stimulation. A recent study[18] used implanted electrodes to both decode information from and transmit information to the sensory cortex of multiple rodents to demonstrated the possibility of distributing computations across multiple brains. The brains of the rats were linked to solve a multiple-constraint non-linear problem (specifically, weather prediction given weather parameters collected from local airport). However, the subjects were unaware of the actual task being solved and their collaboration with others, unlike BrainNet where the subjects are acutely aware of collaborating in a “social network” of brains.

Although our results are encouraging, BrainNet could be improved in the several ways: (1) From the first human BBI[1] to BrainNet, the level of information complexity has remained binary, i.e., only a bit of information is transmitted during each iteration of communication. To address this limitation, we are currently exploring the use of functional Magnetic Resonance Imaging (fMRI) to increase the bandwidth of human BBIs. Other approaches worth exploring include combining EEG and fMRI to achieve both high spatial and temporal resolution[19] for decoding, and using TMS to stimulate higher-order cortical areas to deliver more complex information such as semantic concepts. (2) We purposefully introduced a “bad” sender in BrainNet design to study whether the Receiver can learn which Sender is more reliable. It would be interesting to investigate whether the Receiver can learn the reliability of Senders in more natural scenarios where the unreliability originates from the noisy nature of a Sender’s brain recordings, or from a Sender’s lack of knowledge, diminished attention, or even malicious intent; (3) From an implementation standpoint, BrainNet uses a typical server-client TCP protocol to transmit information between computers. However, the server is solely designed for BrainNet’s experimental task and is not a general purpose server. A cloud-based BBI server could direct information transmission between any set of devices on the BBI network and make it globally operable through the Internet, thereby allowing cloud-based interactions between brains on a global scale. The pursuit of such BBIs has the potential to not only open new frontiers in human communication and collaboration but also provide us with a deeper understanding of the human brain.

Methods

Participants

Fifteen healthy participants (aged 18–35 yrs, eight female), took part in a controlled laboratory experiment. All participants were recruited through word of mouth, were fully informed about the experimental procedure and its potential risks and benefits, and gave written consent prior to the beginning of the experiment according to the guidelines put forth by the University of Washington. They were divided into five groups, with each group being a triad of one participant playing the role of the “Receiver” and two playing the role of the “Sender.” To maintain their decision to participate free of any external influence, all participants received monetary compensation that was independent of their role and proportional to the total amount of time devoted to the study.

Experimental Task

During each session, a triad of three participants collaborated to play a simplified Tetris game. The game consisted of independent trials, each of which involved deciding whether or not to rotate a single Tetris-like piece 180 degrees. At the bottom of the screen, there was a partially filled line whose gaps could be filled by either the top or bottom part of the Tetris-like piece at the top of screen. The goal of the game was to achieve the highest possible score by making the correct decision to rotate or not rotate the current piece so that when dropped at the end of the trial, it would fill the missing parts of the line at the bottom. We designed the task such that the actual player of the game, namely the Receiver, could only see the piece at the top of screen and not the bottom line. The other two subjects, namely the Senders, could see both the piece at top and the line at bottom (See Figure 2). Thus, the only way for the Receiver to achieve a high score was by integrating the decisions transmitted by both Senders and make his/her own decision for the game.

Each session was made of sixteen independent trials; in half of them the falling piece had to be rotated and in the other half, it had to be left in the original orientation. The order of rotation and non-rotation trials was randomized, with the constraint that each half of the session had to contain 4 rotation and 4 non-rotation trials.

Each trial to be comprised of two rounds of interactions between the Senders and the Receiver. Each round offered a chance to rotate the piece. After the first round, the piece was rotated or remained in the same orientation based on Receiver’s decision. The piece then dropped halfway and the screens shown to all three subjects were updated to show the (possibly rotated) piece at the halfway location (See Figure 2). Note that one decision is sufficient to complete the task of filling the bottom line but because of our two-step design, the Senders receive feedback on the Receiver’s action in the first round and can send the Receiver new suggestons, allowing the Receiver to potentially correct a mistake made in the first round and still successfully complete a trial.

The three participants in a triad were located in different rooms in the same building on University of Washington campus and could only communicate with each other through the brain-to-brain interface.

BrainNet: Multi-Person Brain-to-Brain Interface

Figure 1 depicts the architecture of BrainNet, our multi-person brain-to-brain interface. BrainNet relies on two well-known technologies: Electroencephalography (EEG)[20] for non-invasively recording brain signals from scalp and transcranial magnetic stimulation (TMS)[21] for non-invasively stimulating the visual cortex. The Senders convey their decisions of "rotate" or "do not rotate" by controlling a horizontally moving cursor (Figure 8) using steady-state visually-evoked potentials (SSVEPs) [22]: to convey a "rotate" decision, Senders focused their attention on a "Yes" LED light flashing at 17 Hz placed on one side of their computer screen; to convey a "do not rotate" decision, they focused on the “No” LED light flashing at 15 Hz placed on the other side. These LEDs are depicted as circles attached to the screens in Figure 1. The cursor position provided real-time visual feedback to the Senders. The direction of movement of the cursor was determined by comparing the EEG power at 17 Hz versus 15 Hz, with a higher power at 17 Hz over that at 15 Hz moving the cursor towards the side near the “Yes” light, and vice versa for the "No" light. A "rotate" ("do not rotate") decision was made when the cursor hit the "YES" ("NO") side of the screen in Figure 8. In trials where the cursor did not reach either side of the screen due to trial time elapsing, the decision closest to the last location of the cursor was chosen as the subject’s decision.

The decisions of the two Senders were sent to the Receiver’s computer through a TCP/IP network and were further translated into single pulses of transcranial magentic stimulation (TMS) delivered to the occipital cortex of the Receiver. The intensity of the stimulation was set above or below the threshold at which the Receiver will perceive a flash of light known as a phosphene: a “Yes” response was translated to an intensity above the threshold, and “No” was translated to an intensity below the threshold. The Receiver made his/her decision based on whether a phosphene was perceived and this decision was conveyed to the game by the Receiver using the same SSVEP procedure used by both Senders. After the game state was updated, the trial moved into the second round and the above process was repeated. At the end of each trial, all three subjects received feedback on this result of the trial (Figure 2, last row).

Differential Reliability of Senders

When the decisions from the two Senders do not agree with each other, the Receiver must decide which Sender to trust. To investigate whether the Receiver can learn the reliability of each Sender and choose the more reliable Sender for making decisions, we designed the system to deliberately make one of the Senders less accurate than the other. Specifically, for each session, one Sender was randomly chosen as the "bad" Sender and in ten out of sixteen trials in that session, this Sender’s decision when delivered to the Receiver was always incorrect, both in the first and second round of the trial.

EEG Procedure for Senders

Each Sender performed the tasks in a dedicated room in front of a 21" LCD monitor, with two Arduino-controlled LED lights attached to the left and right outer frames of the monitor for eliciting SSVEPs. EEG signals were recorded through an 8-channel Cyton system (OpenBCI: Brooklyn, NY) with a sampling rate of 250Hz at a resolution of 16 bits. Signals were acquired from gold-plated electrodes and a layer of electro-conductive paste was applied between the electrode and the participant’s scalp. For the experimental session, three electrodes were set up along the midline in a custom montage with the signal recorded from one occipital electrode (location Oz in the 10–10 placement system), and two frontal electrodes (locations AFz and FCz in the 10–10 system) used as the ground and reference, respectively.

Incoming EEG data was passed through a 4th-order butterworth filter [23] between 0 and 30 Hz to remove signal drifting and line noise . The time-series EEG data was then divided into 1-second epochs and transformed to the frequency domain through Welch’s method[24]. The intention to rotate the falling piece or not was decoded by comparing the relative power at 17 Hz and 15 Hz, respectively. The final decision was made by tallying up the number of epochs in which the greatest power was recorded at either 17 Hz or 15 Hz over a 10-second period. Signal processing and data storage were managed through a custom software library developed by two of the authors (LJ and DL).

There is no prior training required for controlling the cursor using SSVEP. During the experiment, the Sender’s monitor displays either the cursor-control interface or a gray background with a text prompt indicating that the Receiver is making a decision.

TMS Procedure for the Receiver

Participants playing the role of the Receiver came in for two consecutive sessions. During the first session, as part of informed consent, they were asked to complete a TMS safety screening questionnaire, aimed at identifying potential conditions (such as family history of seizures or frequent migraines) that might represent potential risk factors for adverse side effects of TMS. No participant was rejected for failing the safety questionnaire. In addition to the safety screening, all Receivers underwent a procedure to determine their absolute phosphene threshold, that is, the minimum amount of stimulation necessary to elicit the perception of an induced phosphene 100% of the time. The absolute threshold was assessed using the PEST method[25]. During both the testing session and the experimental session, TMS was delivered through a 70-mm Figure-8 Alpha coil (Magstim, UK) positioned over the left occipital lobe, in a location corresponding to site O1 in the 10-20 system. The coil was positioned flushed to the head, with the handle parallel to the ground and extending towards the left. The coil was attached to a SuperRapid2 magnetic stimulator (Magstim, UK). The maximum intensity of the electric field for our TMS equipment is 530 V/m, and with our coil, the maximum intensity of the induced magnetic field is 2.0 T.

EEG Procedure for the Receiver

The EEG procedure for the Receiver was identical to that of the Senders, except that the signal was acquired from a BrainAmp system (BrainVision, Berlin, Germany) with a sampling rate of 5000 Hz and a resolution of 20 bits. The system was equipped with a DC voltage amplifier to reduce signal distortions due to the TMS pulses. Participants wore a standard 32-channel headcap, using AFz and FCz as the ground and reference, respectively. As in the case of the Senders, only data from the Oz channel was recorded. After downsampling to 500Hz, the incoming data underwent the same preprocessing steps described above for the Senders.

References

  • 1. Rao, R. P. N. et al. A direct brain-to-brain interface in humans. PLOSONE 9, 1–12, DOI: 10.1371/journal.pone.0111332 (2014).
  • 2. Min, B.-K., J Marzelli, M. & Yoo, S.-S. Neuroimaging-based approaches in brain-computer interface. Trends biotechnology 28, 552–60 (2010).
  • 3. Grau, C. et al. Conscious brain-to-brain communication in humans using non-invasive technologies. PLOSONE 9, 1–6, DOI: 10.1371/journal.pone.0105225 (2014).
  • 4. Stocco, A. et al. Playing 20 questions with the mind: Collaborative problem solving by humans using a brain-to-brain interface. PLOSONE 10, 1–15, DOI: 10.1371/journal.pone.0137303 (2015).
  • 5. Rao, R. P. & Stocco, A. When two brains connect. Scientific AmericanMind 25, 36–39 (2014).
  • 6. Dingemanse, M. Brain-to-brain interfaces and the role of language in distributing agency. DistributedAgency 59 (2017).
  • 7. Kyriazis, M. Systems neuroscience in focus: from the human brain to the global brain? Frontiers systemsneuroscience 9, 7 (2015).
  • 8. Hongladarom, S. Brain-brain integration in 2035: metaphysical and ethical implications. J. Information, Communication Ethics Society 13, 205–217 (2015).
  • 9. Nicolelis, M. A. L. Beyond Boundaries (Macmillan, 2011).
  • 10. Bakshy, E., Rosenn, I., Marlow, C. & Adamic, L. The role of social networks in information diffusion. In Proceedings of the 21st international conference on World Wide Web, 519–528 (ACM, 2012).
  • 11. Losey, D. M., Stocco, A., Abernethy, J. A. & Rao, R. P. Navigating a 2D virtual world using direct brain stimulation. Frontiers Robotics AI 3, 72 (2016).
  • 12. Fawcett, T. An introduction to ROC analysis. Pattern Recogn.Lett. 27, 861–874, DOI: 10.1016/j.patrec.2005.10.010 (2006).
  • 13. Cover, T. M. & Thomas, J. A. Elements of Information Theory (Wiley, 2006).
  • 14. Kenney, J. & Keeping, E. Mathematics of statistics (Part I) (Van Nostrand, 1947).
  • 15. Paternoster, R., Brame, R., Mazerolle, P. & Piquero, A. Using the correct statistical test for the equality of regression coefficients. Criminology 36, 859–866 (1998).
  • 16. Wang, Y. & Jung, T.-P. A collaborative brain-computer interface for improving human performance. PLOSONE 6, 1–11, DOI: 10.1371/journal.pone.0020422 (2011).
  • 17. Ifft, P. J., Shokur, S., Li, Z., Lebedev, M. A. & Nicolelis, M. A. L. A brain-machine interface enables bimanual arm movements in monkeys. Science TranslationalMedicine 5, 210ra154–210ra154, DOI: 10.1126/scitranslmed.3006159 (2013). http://stm.sciencemag.org/content/5/210/210ra154.full.pdf.
  • 18. Pais-Vieira, M., Chiuffa, G., Lebedev, M., Yadav, A. & Nicolelis, M. A. L. Building an organic computing device with multiple interconnected brains. ScientificReports 5, 11869 EP – (2015). Article.
  • 19. Huster, R. J., Debener, S., Eichele, T. & Herrmann, C. S. Methods for simultaneous EEG-fMRI: An introductory review. J. Neuroscience 32, 6053–6060, DOI: 10.1523/jneurosci.0447-12.2012 (2012).
  • 20. Nunez, P. L. & Srinivasan, R. Electric Fields of the Brain (Oxford University Press, 2005).
  • 21. O’Shea, J. & Walsh, V. Transcranial magnetic stimulation. CurrentBiology 17, R196–R199, DOI: 10.1016/j.cub.2007.01.030 (2007).
  • 22. Vialatte, F.-B., Maurice, M., Dauwels, J. & Cichocki, A. Steady-state visually evoked potentials: Focus on essential paradigms and future perspectives. Prog. Neurobiology 90, 418–438, DOI: 10.1016/j.pneurobio.2009.11.005 (2010).
  • 23. Sorrentino, R. & Bianchi, G. Electronic Filter Simulation & amp; Design (McGraw-Hill Education, US, 2007). https://mhebooklibrary.com/doi/pdf/10.1036/9780071712620.
  • 24. Welch, P. The use of fast Fourier transform for the estimation of power spectra: A method based on time averaging over short, modified periodograms. IEEE Transactions on Audio Electroacoustics 15, 70–73, DOI: 10.1109/tau.1967.1161901 (1967).
  • 25. Taylor, M. M. & Creelman, C. D. PEST: Efficient estimates on probability functions. The J. Acoustical Society America 41, 782–787, DOI: 10.1121/1.1910407 (1967).

Acknowledgements

This work is made possible by a W. M. Keck Foundation Award to AS, CP, and RPNR, and a Levinson Emerging Scholars Award to LJ. RPNR was also supported by NSF grant no. EEC-1028725. We thank Nolan Strait for software testing.

Author Contributions Statement

RPNR, AS, CP conceived the experiment, LJ, JA conducted the experiment, LJ, DL implemented the software, LJ and AS analyzed the results. All authors were involved in writing and reviewing the manuscript.

Competing Interests

The authors declare no competing interests.

Figure 1: Architecture of BrainNet. Two participants ("Sender 1" and "Sender 2") each use a Brain-Computer Interface (BCI) based on EEG to convey information about a collaborative task (here, a Tetris-like game) directly to the brain of the third participant ("Receiver"). Information from each Sender is transmitted over the internet to the Receiver’s brain via a Computer-Brain Interface (CBI) based on TMS. After consciously processing the two inputs from the Senders, the Receiver uses a BCI based on EEG to execute an action in the task. The Senders see the result of this action on the task and have another opportunity to convey to the Receiver’s brain new information to potentially rectify an incorrect choice in the first round. While our experiment only used two rounds, BrainNet allows an arbitrary number of interactions between the Senders and the Receiver as they collaborate to solve a task.
Figure 2: Examples of Screens seen by the Receiver and the Senders across Two Rounds. While the Receiver see the three example screens on the left side, the Senders see the screens on the right side. (First Row) Screens at the beginning of the trial. Note that the Receiver does not see the bottom line with the gap but the Senders do. The Receiver must rely on the Senders to decide whether or not the Tetris-like piece must be rotated to fill the gap and clear the line. (Middle Row) After the Receiver makes a decision in the first round (in this case, “Rotate”), the game state is updated to show the rotated piece. (Bottom Row) After the second round, all participants see the results of the Receiver’s action and whether the line was cleared. In this example, the Receiver executed a corrective action to rotate the piece again, thereby filling the gap with the bottom part of the piece and clearing the line.
Figure 3: Performance of Triads of Participants using BrainNet. The plot shows accuracy achieved by each of the five triads of participants. Accuracy was defined as the proportion of correct block rotations achieved by the triad. The dashed line shows chance accuracy (0.5).
Figure 4: Average Power Spectra of EEG Signals across Subjects during the SSVEP Task. Power values were averaged across one-second epochs and across subjects. The plot shows the average power values during the SSVEP task (between dashed lines) and for comparison, the power values three seconds before and three seconds after the task. Note that before and after the task, the power values overlap for the two frequencies, whereas during the task, the power of the frequency corresponding to the correct answer is significantly larger.
Figure 5: ROC Curves for the Five Triads of Participants. The plot shows the overall performance of each triad (blue dots) as well as the performances of the two types of Senders in each triad (green and red dots). The superscript on each dot denotes the triad number. Shaded areas represent the area under the curve (AUC) for each triad’s ROC curve. The dashed line denotes chance performance.
Figure 6: Mutual Information transmitted between the Senders and the Receiver. Across all five triads of BrainNet participants, the mutual information transmitted between the Receiver and the good Sender is significantly higher than that between the Receiver and the bad Sender. See text for details on the experimental design used to create a “good” versus “bad” Sender.
Figure 7: Quantification of Learning of Sender’s Reliability by Receiver. (Top Panel) Evolution over time of linear regression weights (beta) for the Receivers’ decision vector and decision vector of each type of Senders for each 4-trial block (see text for details). (Bottom Panel) Pearson Correlation Coefficient between the decisions of Receivers and Senders of each type. Both plots exhibit ascending trends for the good Sender but not the bad Sender, suggesting that Receivers learned which Sender was more reliable during the course of their brain-to-brain interactions with their two Senders.
Figure 8: SSVEP-Based EEG Brain-Computer Interface used by the Senders and the Receiver. Participants conveyed their decisions regarding whether or not to rotate the current piece by controlling a cursor (white filled circle) using steady state visually evoked potentials (SSVEPs) based on EEG. Participants focused on a flashing LED to the left of the screen (depicted as a circle attached to the screen in Figure 1) to move the cursor leftwards towards the “Yes” side. Focusing on the LED to the right of the screen (flashing at a different frequency) caused the cursor to move rightwards towards the "No" side. If the cursor reached the green "Yes" bar, the interface interpreted the participant’s decision to be rotation of the piece (by 180 degrees). If the cursor reached the “No” bar, the interface took the participant’s decision to be to keep the piece’s current orientation.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
283425
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description