Paweł Dzikiewicz, Tomasz Koralewski, Jakub Ner and Adam Pawłowski from the „Neuron” Neuroinformatics Research Club won this year's edition of the CreatiWITy Competition. Their team won with the Mentalizer project. It is a toolkit that enables the integration of emotion recognition models into applications.

Defining motions

Students from the Neuron Neuroinformatics Research Club wondered if it was possible to adapt applications to the feelings of the user. In order to verify this idea, they developed a framework that allows the use of a brain-computer interface in the form of a special cap.

– The cap measures the electrical activity of our brain. Electrical impulses jump between neurons and we are able to collect this signal from the surface of the brain using electrodes located in the cap. We process it accordingly and use it to train models, as well as to later recognize the condition of the user – explains Paweł Dzikiewicz, president of Neuron and one of the developers behind the Mentalizer.

One of the biggest challenges was to precisely define emotions. Owing to scientific analysis, the students managed to define boredom – insufficient stimulus, immersion (flow) – optimal stimulus, and frustration – overwhelming/too large stimulus.

– You simply plug in the device and you can play the game. If the user gets bored, more opponents appear. The emotions recognised by Mentalizer depend on how the classifier is trained. We trained two classifiers based on two different experiments. One classified boredom and flow, the second being the state when the user is engaged in the gameplay – says Jakub Ner, a team member.

Combining technology with psychology

According to the students, Mentalizer is an innovative tool that combines technology with psychology to better understand and respond to users' emotions. They observe that – it enhances interaction with every application, providing more personalised software, and by recording brainwaves, it dynamically responds to mood changes. As a result, the user has more satisfaction.

– Our main intention is to keep the player in this state of engagement and detect deviations into two other states, namely boredom or frustration. When the user performed an activity that was too easy for them, we increased the difficulty level to give them a bit of an adrenaline boost, so they would simply become interested in what was happening on the screen – explains Tomasz Koralewski, one of the students.

It also works in the opposite direction. If the gameplay was too challenging and the player became frustrated, the difficulty level was then lowered for them.

Game developers, aiming to offer players the most engaging gameplay, test the games before they release them to the market.

– However, this is part of the production and testing stage. We, on the other hand, focused on ensuring that the game adjusts in real-time. We have developed not just a game that changes difficulty levels, but an entire set of tools that can be implemented into other applications – notes Paweł Dzikiewicz.

Security is priority

Games with dynamic difficulty levels are one example of how Mentalizer can be used. The other is a simulator of the work of a baggage controller at the airport.

– His task is to identify dangerous objects by looking into the x-ray device. Classic systems at airports often perform a trick and sometimes display something that isn't actually in the suitcase. For example, a gun, so as to identify the controller's reaction to it – explains Adam Pawłowski, one of the four developers of the Mentalizer.

The students from the Neuron Club thought they could connect their boredom detector to this system so as to display dangerous objects only when the controller actually falls into a routine. They believe that in effect the system could work more effectively.

Exceptional tool

The developers of the tool were also finalists in the "Innovation" category at the Young Talents 2024 gala organised by the Lower Silesian Capital Club. Their latest success is a victory in the team project category in the CreatiWITy competition organised by our Faculty.

– Winning in the CreatiWITy competition is an opportunity for public recognition. It's a means of demonstrating our worth and receiving recognition for our efforts and input. It can also serve as inspiration for others to undertake such projects – believes Paweł Dzikiewicz.

The team is not slowing down. Future plans include adding a module that allows fine-tuning AI models, publishing a scientific paper on the impact of the classifier on learning abilities, collaborating with application developers in order to make Mentalizer used in education, gaming, or other fields.

– Our project is open-source. We do not want to phase it out. At some point a person from outside our project contributed to the code and supported us. We hope it will happen again – adds Adam Pawłowski.