aBRI
In telemanipulation scenarios it is of interest to have knowledge about the operator's mental state. Our approach is adaptive Brain Reading (aBR). BR is the external observation and online analysis of brain activity by means of electroencephalogram (EEG) measurements with the goal to detect changes that are related to mental states or certain activity of the subject. Note that aBR takes place without the participation of the subject and by thus does not restrict the operator's full attention and work performance. In the subproject aBRI we focus on the machine learning based analysis of EEG to investigate two types of clinically established event-related potentials (ERPs): the P300 potential for detection of successful information processing and the lateralized readiness potential (LRP) for the prediction of impending movements (Figure 1).
Alert Recognition:
Consciously recognised stimuli evoke an event-related potential called P300. Based on the detection of these potentials in the EEG, the aBRI system should decide whether a presented piece of information was acknowledged by the operator or not. By obtaining this information the system can then react adequately to the situation, e.g. by presenting the information again, in case the operator did not become consciously aware of it.
Movement Prediction:
The aBRI system will also be used to detect the LRP, which signals preparation of movements prior to their actual onset and provides valuable real-time information to the system that controls the force-feedback exoskeleton the operator is wearing.
Test Bed:
To investigate the P300 in an application setting, we set up the test scenario "Labyrinth Oddball" that requires elevated level of concentration, fine motor control as well as response to presented information. Within the test scenario a subject plays the game BRIO® Labyrinth and, concurrently, has to react to important information that are displayed on a monitor by pressing a buzzer (Figure 2 and 3).
This test bed also allows to record EEG data during preparation of movements. While playing the game subjects will continuously execute finetuned motor movements of the hand. Before pressing the buzzer, a movement of the whole arm has to be planned. Besides playing the real labyrinth game, subjects are able to play a simulation of the game, which is controlled directly by sensor data of the real game. This setup enables investigation of EEG signals recorded in a virtual immersion situation (Figure 4 and 5).
Single-trial Processing of EEG within the aBRI:
A modular software (aBRI-DP) for single-trial processing and classification of EEG data has been developed, which allows to combine different signal processing and machine learning methods in a flexible manner. Furthermore, aBRI-DP was recently extended to support real-time processing and visualization of EEG data (figure 6).
An additional framework (Benchmarking on Rails; BOR) allows to evaluate and analyze different combinations of preprocessing and machine learning methods with different parameter settings in an automated way in a distributed computing environment.
Videos
aBri: Movement prediction for exoskeleton control
The video shows how EEG based movement prediction safely improves exoskeleton control.
aBri: Recognition of warnings during teleoperation
The video shows how to improve operator support by predicting his success in information processing by online EEG analysis.
