Michael T. Knierim

Earables are all the rage. With startups receiving large funding sums (e.g., the enophones), brain-sensing is increasingly merging with everday wearables. Join such R&D efforts with this new open-source alternative. This project combines the OpenBCI biosignal acquisition platform (amplifier & AgCl-electrodes) with the open-source 3D-printed headphones from print.plus (headphones), and the open-source headphone electronics from the Homebrewphones project. Altogether this system features up to 10 recording electrodes with 19 variable recording sites around the ear and in the headband. What's great about it? All the hardware and software used are available open-source - so you can prototype whatever you like! Initial tests already show that classical phenomena like visual stimulation (posterior Alpha power changes) and mental workload changes (Theta band power) can be captured well.

Main Tools/Software Used:

• Autodesk Fusion 360 (Modeling)
• Ultimaker Cura (Slicing)
• FDM 3D-Printing (PLA, Anycubic 4.0 Max Pro 2)
• OpenBCIGUI (EEG Recording)
• Python (EEG Processing & Analysis)

By integrating cEEGrid around-the-ear EEG electrode arrays into the OpenBCI platform, comfortable and wearable long-time EEG recordings are enabled for a much more affordable price than previously possible. I created various designs for the integration of the components and re-designed many components myself for a better use with the OpenBCI system. The combination enables the fast and easy prototyping of new electrode arrangements for future wearables that could, for example, help to detect flow experiences, stress, mental workload, awake bruxism, and eating behaviors.

Main Tools/Software Used:

• Autodesk Fusion 360 (Modeling)
• Ultimaker Cura (Slicing)
• FDM 3D-Printing (PLA, Anycubic i3 Mega S)
• Fritzing (PCB Design)
• OpenBCIGUI (EEG Recording)
• Matlab/EEGLab (EEG Processing)
• RStudio (Data Processing & Analysis)

Further Reading:

• Evaluation Study (Technical Feasibility, ECG, Mental Workload, Flow)
• Study on Awake Bruxism Detection
• Study on Face Activity/Expression Detection
• Development Documentation

Arrange your loose EEG cap cables with these 3D-printable components. You'd better do it to minimize mechanical recording artefacts. I made stations with multiple pipes and cable spools that can be inserted in the already available holes in your EEG cap. The spools can be used to shorten cables if they all have the same length (e.g. for electrodes close to the amplifier).

Main Tools/Software Used:

• Autodesk Fusion 360 (Modeling)
• Ultimaker Cura (Slicing)
• FDM 3D-Printing (TPU, Anycubic i3 Mega S)

Initially, I set out to learn and improve the understanding of what goes on in the peripheral nervous system during the flow experience. From this, a vivid discussion emerged on how observations of heart rate variability can inform on the interplay of the two autonomous nervous system branches during flow. Presently, there are a few contesting theories. In principle, researchers appear to agree that at least some level of increased physiological activation is required to experience flow. Beyond that, there is a discussion on whether or not flow is a state of high activation (low HRV and therefore a state of physiological strain), moderate activation ("optimal" activation due to optimal demands), or both high activation accompanied by increased relaxation (a rare configuration of the ANS called non-reciprocal co-activation of the ANS branches). In my experiments, the proposition of moderate activation is supported best, but extended analyses are currently under way to elaborate on this discussion further, for example through the analysis of non-linear relationships.

Main Tools/Software Used:

• Biosignalsplux Signal Hub (Lead II ECG Recording)
• Java (Implementation of Experiment and Sensor Interface)
• R (HRV Analysis, Inferential Statistics, Plotting)

Further Reading:

• Knierim et al. (Forthcoming) - Predicting In-Field Flow Experiences Over Two Weeks From ECG Data: A Case Study
• Knierim (2020) - Detecting Flow Experiences in Cognitive Tasks - A Neurophysiological Approach - Awarded Summa Cum Laude
• Knierim et al. (2018) - Exploring Flow Psychophysiology in Knowledge Work
• Knierim et al. (2017) - The Psychophysiology of Flow: A Systematic Review of Peripheral Nervous System Features

Afterwards, I became increasingly intrigued with the brains configuration during the flow experience. In the literature two main theories revolve around flow as a state of transient hypofrontality or a state of synchronization of attention and reward networks - two propositions the EEG can provide evidence for. In my research, I mainly investigated how wearable EEG can be used to detect or infer the occurence of flow experiences through time-frequency power analyses. To do so, I conducted various experiments using different experimental paradigms to induce more intense flow experiences, specifically manipulations of task difficulty, complexity, autonomy, and social interaction. Presently, my results most strongly point to flow as a state of elevated and stable task attention, neutral motivational-affective tone, and moderate mental workload.

Main Tools/Software Used:

• Emotiv Epoc+ & Flex (EEG Recording)
• Java (Implementation of Experiment and Sensor Interface)
• R (Data Preparation, Inferential Statistics, Plotting)
• Matlab (EEG Processing - EEGLab)
• RStudio (Shiny Web-App Environment)

Further Reading:

• Knierim (2020) - Detecting Flow Experiences in Cognitive Tasks - A Neurophysiological Approach - Awarded Summa Cum Laude
• Knierim et al. (2019) - Flow and Optimal Difficulty in the Portable EEG: On the Potentiality of using Personalized Frequency Ranges for State Detection
• Knierim et al. (2018) - Flow Neurophysiology in Knowledge Work: Electroencephalographic Observations from Two Cognitive Tasks - Nominated for the Best Conference Paper Award
• Shiny Web-App for a Literature Review on EEG Findings on the Flow Experience

To enable the development of adaptive systems that are able to foster flow experiences, we have started to explore the possibilities of classifying the state occurence using machine learning methods. So far, Random Forest classifiers provide the best results using ECG and EEG data. Two-class models (lower/higher flow) have shown accuracies of ~65% (laboratory) to ~70% (field) using heart rate variability metrics alone. Lab-based classifiers could be further improved by adding information on the task performance. Recently, we have also identified accuracies of up to 70% for high flow classification in a three-class model (no/slight/intense flow) using EEG frequency band features from over 150 participants that took part in a laboratory experiment.

Main Tools/Software Used:

• R (Data Preparation, Inferential Statistics, Plotting)
• Matlab (EEG Processing - EEGLab)
• Python (Random Forest Classifier)

Further Reading:

• Knierim et al. (Forthcoming) - Predicting In-Field Flow Experiences Over Two Weeks From ECG Data: A Case Study
• Rissler et al. (2020) - To Be or Not to Be in Flow at Work: Physiological Classification of Flow using Machine Learning
• Knierim et al. (2019) - Flow in Knowledge Work Groups – Autonomy as a Driver or Digitally Mediated Communication as a Limiting Factor?
• Rissler et al. (2018) - Got Flow?: Using Machine Learning on Physiological Data to Classify Flow

In this project, I implemented various types of sensor and biofeedback instances, to provide information on self-regulation of physiological activation and stress (as indicated of heart rate variability changes) during work. The version shown here is one that uses an ECG chest belt and the Philips Hue color-changeable lamps. However, the system is more generic and allows for the use of gelled ECG chest electrodes or an EDA ring too on the sensor side. To enable the communication with these wearable sensors, the BLED112 dongle was programmed in Java. On the feedback side, the lamps can be switched to various desktop based feedback interfaces (from prominent ones that change display brightness - e.g. darker screens with higher stress - to more subtle ones that show a warning icon in the system tray).

Main Tools/Software Used:

• Biosignalsplux & Bitalino Signal Hubs (Lead II ECG Recording)
• Zephyr HxN (Chest Belt ECG Recording)
• Moodmetric EDA Ring (Finger EDA Recording)
• BLED112 Dongle (Programmable BLE Interface)
• Philips Hue Java API (Lamp-Based Biofeedback)
• Java (Implementation of Sensor Interface, Online Feature Extraction, and Biofeedback GUIs)

Further Reading:

• Lux et al. (2018) - Live Biofeedback as a User Interface Design Element: A Review of the Literature
• Blog Post

In this project, I adapted the popular card game Werewolf (Witches of Salem for some) to be used with Live Heart Rate Biofeedback from all players. By observing the HR traces of all players after a "night" of the werewolf going around town one can ask the question: Who is exposed by the excitement of keeping a secret? The system features the collection and processing of up to eight ECG signals (gelled chest electrodes in Lead II configuration) at the same time in a single biosignal acquisition hub. The idea for the game arose when Karlsruhe City and the Karlsruhe Institute of Technology were looking for public demonstrations during the so-called "Night of Science". Since the conception of this event, Bio-Wolf has been a steady event for three years and has attracted increasing numbers of interested viewers and Werewolf enthusiasts (>50 participants in the last year). Would you like to play a round too? Let me know - we even adapted it for academic circles (then called: Dr. Biokyll and Mr. Hyde).

Main Tools/Software Used:

• Biosignalsplux & Bitalino Signal Hubs (Lead II ECG Recording)
• Java (Implementation of Game Logic, Sensor Interface, Online Feature Extraction, and Biofeedback GUI)

Further Reading:

• Lux et al. (2018) - Live Biofeedback as a User Interface Design Element: A Review of the Literature
• Blog Post

Recent research states that there is something like a team IQ - and apparently it is linked more strongly to team members abilities of recognizing others' emotions than it is to individual IQ. Therefore, we considered whether or not team interaction could not be improved by enriching digital interactions with internal state information (here: changes in team members' heart rates). These heart rates were collected using chest ECG (gelled electrodes in Lead II configuration), processed online and interpreted as changes in physiological activation (relative HR increases from baseline rest recordings). The possible influences on intra- and interpersonal emotion recognition and regulation were explored in pilot studies. The results showed both promising effects (more awareness of inner state changes), but also showed limitations in terms of information interpretation and resulting actions. We even briefly prototyped a version of this team biofeedback with the Microsoft Band 2 - which was unfortunately discontinued in the middle of our implementation. It could still be an application scenario though. The learnings and key challenges from this project were subsequently summarised in a conceptual article on furthering the development of such systems.

Main Tools/Software Used:

• Biosignalsplux & Bitalino Signal Hubs (Lead II ECG Recording)
• Microsoft Band 2 (PPG Measurement and Feedback)
• Java (Implementation of Experiment, Sensor Interface, Online Feature Extraction, and various Biofeedback GUIs)

Further Reading:

• Lux et al. (2018) - Live Biofeedback as a User Interface Design Element: A Review of the Literature
• Knierim et al. (2017) - Emotion Feedback in Small Group Collaboration: A Research Agenda for Group Emotion Management Support Systems - Nominated for Best Conference Article
• Knierim et al. (2017) - Designing Live Biofeedback for Groups to Support Emotion Management in Digital Collaboration - Awarded Best Conference Prototype

Do you like brains and cookies? Well, I do! To unite both passions I made these puzzle-like brain cookie cutters. It's like alphabet soup, just with brains and cookies. For this project I first had to assemble neuroanatomy materials for making at least grossly correct re-creations of macroscopic encephalon regions. To spread the love, these figures were then transformed into stickers and 3D-printed cookie cutters out of food-grade and dishwasher-safe material. Finally, I created a simple Etsy web-shop to distribute these items. In a sense you could say, that I started my first own business that way (at least after my photography detour).

Main Tools/Software Used:

• Autodesk Fusion 360 (Modeling)
• Ultimaker Cura (Slicing)
• FDM 3D-Printing (PLA, Anycubic i3 Mega S)
• Bohemian Coding Sketch (Graphics)
• Etsy (Marketplace)

Further Reading:

• Etsy Webshop
• Facebook Page

I learned in a TED Talk that we apparently get the formula wrong: You are much more likely to be successful in what you do when you take care of your wellbeing or happiness first, not the other way around. By following the old logic (success first, then happiness), we often tend to neglect perspectives, activities or behaviors that foster our well-being. Luckily, there are simple and proven strategies available that can help us to transform this pattern. For example, by focusing on things that are good in your life and that you are grateful for on a daily basis, you train your brain to notice more naturally what is good in your life. You have to at least stick with it for some time though (e.g., 30 days). To facilitate this process, I created a simple three-graces logging web-app.

Main Tools/Software Used:

• RStudio (Shiny Web-App Environment)
• Shinyapps.io (Hosting Service)
• Bootstrap (HTML, CSS, and JS Library)

Further Reading:

• Shiny Webapp
• Related TED Talk

Ok, here i just wanted to learn some Python and GUI design. As I was acquiring an interest for tea at the time, I decided to program a simple, but customisable and nice-looking tea timer app. I used Python 3 and the PyQt5 bindings for the Qt toolkit for graphical user interface creation. The app features a color-changing background that mimics the teas color change (becoming a darker green as brewing time ends) and a smooth pulsing notion of the leaves to indicate that the tea is ready for consumption. The app also allows to configure two types of tea with customization of up to three durations for different infusion cycles (as you can, for example, use the same green tea leaves multiple times but have to increase the infusion duration accordingly with each re-use).

Main Tools/Software Used:

• Python 3 (Logic)
• PyQt 5 (Bindings for Qt 5)
• Qt 5 (Graphical User Interface)
• Bohemian Coding Sketch & Adobe Photoshop (Graphics)