We present a dataset called Toadstool that aims to contribute to the field of reinforcement learning, multimodal data fusion, and the possibility of exploring emotionally aware machine learning algorithms. Furthermore, the dataset can also be useful to researchers interested in facial expressions, biometric sensors, sentiment analysis, and game studies. The dataset consists of video, sensor, and demographic data collected from ten participants playing a Super Mario Bros. The sensor data is collected through an Empatica E4 wristband, which provides high-quality measurements and is graded as a medical device. In addition to the dataset, we also present a set of baseline experiments which show that sensory input can be used to train fully autonomous agents, which, in this case, play a video game. We think that the presented dataset can be interesting for a manifold of researchers to explore different exciting questions.
Update 13-05-2022
We have now added true emotion labels and synchronization to the dataset. This expanded vesrion can be dowlodaded below (toadstool2.zip).
Download
| File | Description | Size | Download |
|---|---|---|---|
| toadstool.zip | The toadstool.zip dataset in one zip file. | 7.5GB | |
| toadstool2.zip | The toadstool2.zip dataset in one zip file. | 24.7GB |
Cite
@inproceedings{10.1145/3339825.3394939,
title = {Toadstool: A Dataset for Training Emotional Intelligent Machines Playing Super Mario Bros},
author = {
Svoren, Henrik and Thambawita, Vajira and Halvorsen, P\r{a}l and
Jakobsen, Petter and Garcia-Ceja, Enrique and Noori, Farzan Majeed and
Hammer, Hugo L. and Lux, Mathias and Riegler, Michael Alexander and
Hicks, Steven Alexander
},
year = {2020},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
doi = {10.1145/3339825.3394939},
booktitle = {Proceedings of the 11th ACM Multimedia Systems Conference},
pages = {309–314},
numpages = {6},
location = {Istanbul, Turkey},
series = {MMSys '20}
}
Terms of use
The data is released fully open for research and educational purposes. The use of the dataset for purposes such as competitions and commercial purposes needs prior written permission. In all documents and papers that use or refer to the dataset or report experimental results based on Toadstool, a reference to the related article needs to be added: https://dl.acm.org/doi/10.1145/3339825.3394939.
Ethics approval
In this study, we used fully anonymized data approved by Privacy Data Protection Authority. Furthermore, we confirm that all experiments were performed in accordance with the relevant guidelines and regulations of the Regional Committee for Medical and Health Research Ethics - South East Norway, and the GDPR.
Contact
Email steven (at) simula (dot) no if you have any questions about the dataset and our research activities. We always welcome collaboration and joint research!