A Survey on Usage of Multimedia Databases for Emotion Elicitation: A Quantitative Report on How Content Diversity Can Improve Performance
Marko Horvat, University of Zagreb, Faculty of Electrical Engineering and Computing, Department of Applied Computing, Zagreb, Croatia
Petar Jerčić, Graz University of Technology, Institute of Interactive Systems and Data Science, Graz, Austria

Summary and motivation:

The survey examined how researchers in psychology, neurology, cognitive science, and related fields use different multimedia stimuli datasets developed for the stimulation of emotional responses.

Affective picture databases provide a standardized set of images to elicit controlled and consistent emotional responses in research participants. They are a valuable tool for studying various emotion-related phenomena across several research domains. These domains include emotion perception, emotion regulation, and the neural basis of emotion. However, affective picture databases have diverse schemas, structures, and content that make them difficult to use.

Traditionally, creating personalized sequences targeted at specific semantic cues and emotional responses is time-consuming and professionally challenging. As a result, this work is almost always accomplished manually and requires high expertise and professional knowledge.

This survey aimed to identify common patterns in using multimedia databases for emotion elicitation (i.e., multimedia stimuli databases).

This survey followed up on similar research undertaken precisely ten years ago and published in the Proceedings of 36th International Convention on Information and Communication Technology, Electronics and Microelectronics MIPRO 2013: https://www.bib.irb.hr/622474, https://www.researchgate.net/publication/261424676_Multimedia_stimuli_databases_usage_patterns_A_survey_report.

Survey questions:

The survey questions were:
Q1. What best describes your research topic?
Q2. Please rate the image retrieval process from multimedia stimuli databases such as IAPS, NAPS, GAPED, OASIS, KDEF, POFA, JAFFE, and IADS?
Q3. How satisfied are you with the above-mentioned level of difficulty in the image retrieval process from the database?
Q4. How much time was necessary to effectively search the database and construct one picture sequence that was used in your research?
Q5. On a scale of 1 to 5, with one being very inadequate and 5 being very adequate, please rate the quality of the documentation (guide instructions and support)?
Q6. Have you at any time felt that the picture set you were using is missing images with a particular emotion that would be useful for your stimuli sequence?
Q7. Have you at any time felt that the picture set you were using is missing images with semantic content that would be useful for your stimuli sequence?
Q8. Have you felt that the picture set you were using addressed the cultural values in your target group?
Q9. On a scale of 1 to 5, with one being very insufficient and 5 being very sufficient, please rate the diversity of semantic and emotional content in the picture sets you used.
Q10. On a scale of 1 to 5, with one being very inadequate and 5 being very adequate, please rate how inadequate (insufficient) did you find the predefined semantic descriptors (e.g., keywords or tags) of the images you used?
Q11. On a scale 1 to 5, with one being extremely useless and 5 being extremely useful, please rate how helpful would a user-friendly software tool for intelligent retrieval of emotionally annotated images be to your research?
Q12. Have you at any time during your research wanted to find the most appropriate emotionally-annotated images faster and more efficiently?
Q13. Can you give a rough indication of the duration in minutes of one of your test sessions (the period in which one participant is tested continuously)?
Q14. Did you construct the sequence manually or with a help of a software tool?
Q15. Which software tool did you use (if any)?
Q16. Did your group actually develop the tool used in your experiment or acquired it elsewhere?
Q17. How useful or useless a stimuli database with realistic and immersive Virtual Reality (VR) images instead of just still ones, would be to your work?
Q18. Please list the names of multimedia stimuli databases you have used in your research (e.g., IAPS, NAPS, GAPED, OASIS, KDEF, POFA, JAFFE, IADS).
Q19. Do you wish anything to add that was not covered in the survey?

Key findings:

The results clearly highlight the widespread dissatisfaction with the diversity of stimuli and their cultural bias, especially in the context of emotional and semantic interpretation. Moreover, we observed remarkable stability in user ratings, which were mostly around the median, regardless of whether the stimulus sequence was created manually or by automated software. This stability was evident across the different variables examined.

Interestingly, the quality of the semantic descriptors proved to be a significant factor in user satisfaction with the automated retrieval software. High-quality semantic descriptors were associated with higher score ratings, while low-quality semantic descriptors resulted in significantly lower scores - even lower than those for manually created sequences. This suggests that in the absence of good semantic descriptors, automatic software may become practically useless for creating satisfactory sequences, a task that users with manual pipelines can still accomplish under adverse conditions in multimedia databases.

The survey identified a need for novel data observatory software. This finding motivates the authors' intention to develop and validate such software platform that relies on AI. Such a platform would describe better, retrieve, and integrate various semi-structured affective multimedia datasets. The results prominently indicate the overwhelming dissatisfaction regarding stimuli content diversity and cultural bias, specifically regarding emotional and semantic context. The main driver of satisfaction from users of existing automated retrieval software is the quality of semantic descriptors available, which points to the direction AI should take in novel data observatory software.

Notably, regarding user satisfaction with these databases, we found that semantic descriptors have the most significant influence on predicting user satisfaction and significantly affect rating scores. Furthermore, the effect was strong for users with previous experience with automated retrieval software, while it was weaker for those who retrieved manually. This effect was significant even after controlling for the variance from other confounding factors in the survey.

In addition, the survey results point to the need to develop new stimuli databases with innovative multimedia content. Exploring generative AI graphics and VR delivery as a research direction could offer numerous advantages over acted or real-world visualizations. However, the expansion of these databases highlights the need for intelligent software that can efficiently manage the increasing complexity of affective multimedia content.

Particularly, in comments on various topics (Q19), participants called for more consistency in documentation. They also pointed out that more racial diversity of images and more emotion types would be helpful (e.g., bored, disgusted). Some asked for neutral stimuli that are as semantically related as the emotional stimuli and positive arousing stimuli and emotional stimuli that do not occur in distant, exotic locations. The number of available images, as is the ability to assign images based on various features other than valence, is essential. It was also pointed out that there have yet to study children. Some were concerned about the quality of the ground truth data and how the dataset authors collected it. Some researchers found it easy to construct sequences based on published normative ratings but found it challenging to work with data dimensions beyond those covered by the normative ratings (i.e., beyond valence, arousal, and basic emotions).

Although the survey did not include a large number of participants or complex questions, it yielded statistically significant results that should not be ignored.

Citation:

Please cite this paper as:

Horvat, M., & Jerčić, P. (2023, May). A Survey on Usage of Multimedia Databases for Emotion Elicitation: A Quantitative Report on How Content Diversity Can Improve Performance. In 2023 46th MIPRO ICT and Electronics Convention (MIPRO) (pp. 1148-1154). IEEE.

Aviability:

The aggregated survey results in this repository will remain permanently publicly available to the scientific community for research purposes.

Acknowledgement:

The authors would like to express their sincere gratitude to all the experts who participated in this survey. Their invaluable insights have laid the foundation for the development of a more effective method for managing and integrating semi-structured stimuli datasets.

PDF: