Alex Fletcher

Alex Fletcher

Assessing the design and potentials of an Adaptive and Interactive Music System for Biosensor-Based Serious Games

The serious (applied) games market is forecast to raise considerably by 2025 [1], with its popularity within educational, medical, training, simulation and advertisement applications [2], [3]. Serious games are created to fulfil an learning objective, rather than for pure entertainment [4]. Furthermore, Ravyse et al (2017) define them to have an intended impact on cognition, behavior or motor skills and reviewed what are the success factors that enhance learning [5]

 

The adoption of neuroscience, clinical psychology and meditation research within serious games has created a new genre known as ‘neurofeedback’. This has created a market for personal EEG and biosensor devices, as an affordable, high-resolution alternative to professional EEG equipment, along with applications to be used both in the home and medical environment [6], [7], [8], [9].

 

The designs of these applications are starting to incorporate machine learning algorithms, that have been proved by EEG research studies to have successfully identified arousal and emotion at a relatively high-resolution [10], [11], [12]. Research in combining EEG with physio sensors to enhance the resolution and to investigate other neuro-signatures is still in the early stages [13], [14].  Predominantly, neurofeedback research is neuroscience based and is commonly aimed towards clinical psychology practitioners in non-pharmacological treatment for ADHD and other psychiatric disorders [15]. To date, only a small set of computer science studies tackle neurofeedback game design concepts for both entertainment and applied games [16], [17]. Equally limited is audio specific research, with only one reviewing the design of an adaptive audio system for a EEG-based horror game, another in the design of an audio-biofeedback system for patients with parkinson’s disease and a meditation mobile application designed around audio-feedback for the Interaxon Muse Headset [18], [19], [20]. Moreover, there are no neurofeedback games that explore music creation and expression to create an immersive musical experience during gameplay, other than by individual artist installations [21], [22], [23]. However, extensive research has proved the potential of game audio for wellbeing, audio as gameplay and interactive music video games for musical development [24], [25], [26], [27], [28].

 

Predominantly, the design of serious games use pre-existing popular games such as minecraft and real-world situations [29], [30]. Currently, there seems to be a lack of studies identifying the effects of applied games and a lack of novel designs within the consumer serious game market. However, M.Nazry and Romano (2017) created two navigation-based serious games and successfully proved that the relationship of enhanced mood and improved learning efficiency existed, with the findings showing the games did enhance mood post-experience, there was an overall trend of improved learning in-game and time spent was longer and more engaged when happier [31].  Another area to explore is the role empathy plays in developing engagement. There have been extensive case studies and VR experiences that have been used as an effective tool to shift individuals into pro-social emotional states [32], [33]. In particular, it has become popular amongst applied games designed to teach primary/secondary school children [34].      

 

From this apparent gap between the relationships of game audio, EEG-based game research and the effects of serious games, I propose to examine the potentials of using adaptive audio as a form of musical expression for biosensor-based serious games to aid in improving people’s mood, wellbeing and creative performance.






References

[1]“Global Serious Game Market to Reach 9840 Million US by 2025, from 2770 Million US in 2018,” 16-Jan-2019. [Online]. Available: http://www.wboc.com/story/39804156/global-serious-game-market-to-reach-9840-million-us-by-2025-from-2770-million-us-in-2018. [Accessed: 25-Jan-2019].

[2]G. Baptista and T. Oliveira, “Gamification and serious games: A literature meta-analysis and integrative model,” Comput. Human Behav., vol. 92, pp. 306–315, Mar. 2019.

[3]D. Tack, “Serious Games And The Future Of Education,” Forbes Magazine, Forbes, 12-Sep-2013.

[4]T. Susi, M. Johannesson, and P. Backlund, “Serious Games : An Overview,” 2007.

[5]W. S. Ravyse, A. Seugnet Blignaut, V. Leendertz, and A. Woolner, “Success factors for serious games to enhance learning: a systematic review,” Virtual Real., vol. 21, no. 1, pp. 31–58, Mar. 2017.

[6]“EEG (Electroencephalography) Systems Market | Industry Report, 2025.” [Online]. Available: https://www.grandviewresearch.com/industry-analysis/electroencephalography-eeg-systems-devices-market. [Accessed: 25-Jan-2019].

[7]P. Frye, “Global Consumer EEG Device Market Report 2018: Neurosky & More,” Positive Source24, 07-Sep-2018. [Online]. Available: https://positivesource24.com/global-consumer-eeg-device-market/. [Accessed: 25-Jan-2019].

[8]S. N. Abdulkader, A. Atia, and M.-S. M. Mostafa, “Brain computer interfacing: Applications and challenges,” Egyptian Informatics Journal, vol. 16, no. 2, pp. 213–230, Jul. 2015.

[9]D. Wortley, “The Future of Immersive Technologies and Serious Games,” in Subconscious Learning via Games and Social Media, O. Sourina, D. Wortley, and S. Kim, Eds. Singapore: Springer Singapore, 2015, pp. 45–56.

[10]P. Bashivan, I. Rish, and S. Heisig, “Mental State Recognition via Wearable EEG,” arXiv [cs.CV], 02-Feb-2016.

[11]Y. Lin et al., “EEG-Based Emotion Recognition in Music Listening,” IEEE Transactions on Biomedical Engineering, vol. 57, no. 7, pp. 1798–1806, Jul. 2010.

[12]Q. Wang, O. Sourina, and M. K. Nguyen, “Fractal dimension based neurofeedback in serious games,” Vis. Comput., vol. 27, no. 4, pp. 299–309, Apr. 2011.

[13]M. A. Hogervorst, A.-M. Brouwer, and J. B. F. van Erp, “Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload,” Front. Neurosci., vol. 8, p. 322, Oct. 2014.

[14]K. Katahira, Y. Yamazaki, C. Yamaoka, H. Ozaki, S. Nakagawa, and N. Nagata, “EEG Correlates of the Flow State: A Combination of Increased Frontal Theta and Moderate Frontocentral Alpha Rhythm in the Mental Arithmetic Task,” Front. Psychol., vol. 9, p. 300, Mar. 2018.

[15]Q. Wang, O. Sourina, and M. K. Nguyen, “EEG-Based ‘Serious’ Games Design for Medical Applications,” in 2010 International Conference on Cyberworlds, 2010, pp. 270–276.

[16]O. Sourina and Y. Liu, “EEG-Based Serious Games,” in Subconscious Learning via Games and Social Media, O. Sourina, D. Wortley, and S. Kim, Eds. Singapore: Springer Singapore, 2015, pp. 135–152.

[17]X. Y. Lee, E. Koukouna, C. G. Lim, C. Guan, T. S. Lee, and D. S. S. Fung, “Can We Play with ADHD? An Alternative Game-Based Treatment for Inattentive Symptoms in Attention-Deficit/Hyperactivity Disorder,” in Subconscious Learning via Games and Social Media, O. Sourina, D. Wortley, and S. Kim, Eds. Singapore: Springer Singapore, 2015, pp. 69–86.

[18]J. Craig, “Adaptive Audio Engine for EEG-Based Horror Game.” Unpublished, 2016.

[19]A. Mirelman et al., “Audio-biofeedback training for posture and balance in patients with Parkinson’s disease,” J. Neuroeng. Rehabil., vol. 8, p. 35, Jun. 2011.

[20]“Guided Meditation App - Get the Muse Mindfulness & Biofeedback App,” Muse. [Online]. Available: https://choosemuse.com/muse-app/. [Accessed: 25-Jan-2019].

[21]“Eunoia II — LISA PARK,” LISA PARK. [Online]. Available: http://www.thelisapark.com/eunoia-ii/. [Accessed: 25-Jan-2019].

[22]“On Your Wavelength, Marcus Lyall in collaboration with Rob Thomas and Alex Anpilogov,” Merge Festival. [Online]. Available: http://mergefestival.co.uk/merge-events-2015/2015/8/18/on-your-wavelength-marcus-lyall-in-collaboration-with-rob-thomas. [Accessed: 25-Jan-2019].

[23]S. Whitmarsh and P. Huttner, “The EEGsynth,” The EEGsynth. [Online]. Available: http://www.eegsynth.org/. [Accessed: 25-Jan-2019].

[24]G. F. Tondello, R. Orji, K. Vella, D. Johnson, M. M. M. van Dooren, and L. E. Nacke, “Positive Gaming: Workshop on Gamification and Games for Wellbeing,” in Extended Abstracts Publication of the Annual Symposium on Computer-Human Interaction in Play, 2017, pp. 657–660.

[25]K. Rogers and L. Nacke, “Exploring the Potential of Game Audio for Wellbeing,” in Positive Gaming: Workshop on Gamification and Games for Wellbeing, 2017.

[26]“Game Studies - Sonic Mechanics: Audio as Gameplay.” [Online]. Available: http://gamestudies.org/1301/articles/oldenburg_sonic_mechanics. [Accessed: 25-Jan-2019].

[27]J. R. Parker and J. Heerema, “Audio Interaction in Computer Mediated Games,” International Journal of Computer Games Technology, vol. 2008, Nov. 2007.

[28]L. Gower and J. McDowall, “Interactive music video games and children’s musical development,” British Journal of Music Education, vol. 29, no. 1, pp. 91–105, Mar. 2012.

[29]“Impact | Minecraft: Education Edition,” Minecraft: Education Edition. [Online]. Available: https://education.minecraft.net/impact/. [Accessed: 30-Jan-2019].

[30]“Learning Games - 26 Serious Games to Change the World,” Mission to Learn - Lifelong Learning Blog, 18-Apr-2008. [Online]. Available: https://www.missiontolearn.com/learning-games-for-change/. [Accessed: 30-Jan-2019].

[31]N. N. M. Nazry and D. M. Romano, “Mood and learning in navigation-based serious games,” Comput. Human Behav., vol. 73, pp. 596–604, Aug. 2017.

[32]S. Hill, “Your Brain on VR Empathy – Virtual Reality Pop,” Virtual Reality Pop, 05-Jan-2017. [Online]. Available: https://virtualrealitypop.com/your-brain-on-vr-empathy-fc7a3743b2e5. [Accessed: 30-Jan-2019].

[33]“Project Empathy,” Project Empathy. [Online]. Available: https://www.projectempathyvr.com/. [Accessed: 30-Jan-2019].

[34]“Top Games That Teach Empathy | Common Sense Education,” Common Sense Education, 24-Sep-2013. [Online]. Available: https://www.commonsense.org/education/top-picks/top-games-that-teach-empathy. [Accessed: 30-Jan-2019].

Home institution: Queen Mary

Supervisor: Dr Laurissa Tokarchuk

Ready to apply?

Once you have identified your potential supervisor, we would encourage you to contact them to discuss your research proposal.

Learn More