The Toadstool Dataset

A Dataset for Training Emotional Intellgient Machines Playing Super Mario Bros.

The (size 7.5GB) archive contains video, sensor, and facial data of ten different participants playing Super Mario Bros.


@misc{Svoren2020, title={Toadstool: A Dataset for Training Emotional IntellgientMachines Playing Super Mario Bros}, url={}, DOI={10.31219/}, publisher={OSF Preprints}, author={Svoren, Henrik and Thambawita, Vajira and Halvorsen, Pål and Jakobsen, Petter and Garcia-Cejae, Enrique and Noori, Farzan Majeed and Hammer, Hugo and Riegler, Michael and Hicks, Steven}, year={2020}, month={Feb} }


We present a dataset called Toadstool that aims to contribute to the field of reinforcement learning, multimodal data fusion, and the possibility of exploring emotionally aware machine learning algorithms. Furthermore, the dataset can also be useful to researchers interested in facial expressions, biometric sensors, sentiment analysis, and game studies. The dataset consists of video, sensor, and demographic data collected from ten participants playing a Super Mario Bros. The sensor data is collected through an Empatica E4 wristband, which provides high-quality measurements and is graded as a medical device. In addition to the dataset, we also present a set of baseline experiments which show that sensory input can be used to train fully autonomous agents, which, in this case, play a video game. We think that the presented dataset can be interesting for a manifold of researchers to explore different exciting questions.

Terms of use

The data is released fully open for research and educational purposes. The use of the dataset for purposes such as competitions and commercial purposes needs prior written permission. In all documents and papers that use or refer to the dataset or report experimental results based on Toadstool, a reference to the related article needs to be added: PREPRINT:

Ethics approval

In this study, we used fully anonymized data approved by Privacy Data Protection Authority. Furthermore, we confirm that all experiments were performed in accordance with the relevant guidelines and regulations of the Regional Committee for Medical and Health Research Ethics - South East Norway, and the GDPR.


Email steven (_at_) simula (_dot_) no if you have any questions about the dataset and our research activities. We always welcome collaboration and joint research!