Department of Computer Science

How can multisensory stimuli be used to elicit a complex response in VR?

Unterwasser HMD

Thursday 14 September 2023

As part of the DAAD project "Program of Project-Related Exchange of Persons Japan JSP 2023-2025, there is the international project between Hochschule Bonn-Rhein-Sieg (H-BRS) and Nara Institute of Science and Technology (NAIST) in Japan called "Multisensory Cues to Evoke Emotional Responses and Awe in Virtual Reality". The study has now been published in Virtual Reality Society of Japan (VRSJ) 2023.
Projekt Unterwasser-VR - Kooperation mit Japan
f.l.: Prof. Dr. André Hinkenjann, Dr. Felix Dollack and Assistant Professor Monica Perusquía-Hernández (both from NAIST-Japan) with Dr. Alexander Marquardt at H-BRS

In cooperation with the Institute of Visual Computing (IVC) and the Cybernetics and Reality Engineering Laboratory (CARE Lab), headed by Prof. Kiyoshi Kiyokawa, we will use multisensory stimuli in an attempt to trigger complex emotional responses such as the feeling of awe in VR. Awe is a positive, self-transcending emotion that makes people feel like they are part of something bigger than themselves. It occurs when people experience something grand or sublime, such as natural wonders or impressive architecture. Since awe contributes to greater life satisfaction, the research team believes that effectively experiencing awe in VR could increase well-being. The use of multisensory stimuli, e.g., audiovisual stimuli of impressive environments complemented by various tactile stimuli such as vibration or heat stimuli, or stimuli related to the sense of balance, will be used to specifically evoke the feeling of awe in virtual environments.

Projekt: Unterwasser VR
Dr. Alexander Marquardt with underwater HMD - Nettebad Osnabrück

In April/May 2023, a small delegation consisting of Assistant Professor Monica Perusquía-Hernández and Dr. Felix Dollack from Japan came to the H-BRS for the first time to work on preparations for the first underwater test. These tests, supported by Volker Heemann (Stadtwerke Osnabrück) were performed together in the Nettebad of the Stadtwerke Osnabrück. Here, commercial underwater HMDs with different VR scenarios are in use, which were tested in use and function as part of the research.

Prof. Dr.Ernst Kruijff (l) und Prof. Dr. Andre Hinkenjann (re) testen VR
NAIST- Prof. Dr. Ernst Kruijff (li) unf Prof. Dr. André Hinkenjann tauchen mit einem Head-Mounted Disply (HMD) durch eine echte Tür in eine VR-Welt ein.

After the first successful tests and findings, the H-BRS research team, consisting of Prof. Dr. André Hinkenjann, Prof. Dr. Ernst Kruijff (project leader), Dr. Alexander Marquardt (deputy project leader), student assistant Melissa Meilinger and student assistant Marvin Lehnort, visited NAIST in June 2023. Beforehand, Hinkenjann and Kruijff were able to attend a Shonan workshop on "Augmented Perception" organized by Yuta Itoh (The University of Tokyo, Japan), Kai Kunz (Keio University, Japan) and Jason Orlosky (Augusta University, USA & Osaka University, Japan).

Unterwasser HMD

One specific VR scenario tested in the pool in conjunction with the underwater HMD (head-mounted display) was the view from space to Earth - the so-called overview effect - which is a famous example of a sense of awe, as the feeling of "infinite space" is said to be particularly palpable here. Multisensory technology plays a role here in that underwater stimuli exert an influence on the sense of balance to simulate a feeling of weightlessness, which of course is particularly congruent with the visual stimuli of the VR scene.

The special thing about the cooperation with NAIST is that it operates its own small pool facility for research purposes. Marquardt used the pool to optimize the underwater prototypes developed in this project based on the findings from the visit to Osnabrück and to test them in the controlled pool environment. In Virtual Reality Society of Japan (VRSJ) 2023, his study has now been published and can be read here:

Prototypen VR  - Unterwasser VR
The developed prototypes of the underwater HMD