The CABB dataset: A multimodal corpus of communicative interactions for behavioural and neural analyses
Eijk,Lotte Rasenberg,Marlou Arnese,Flavia Blokpoel,Mark Dingemanse,Mark Doeller,Christian F. Ernestus,Mirjam Holler,Judith Milivojevic,Branka Özyürek,Asli
Eijk,Lotte
Rasenberg,Marlou
Arnese,Flavia
Blokpoel,Mark
Dingemanse,Mark
Doeller,Christian F.
Ernestus,Mirjam
Holler,Judith
Milivojevic,Branka
Özyürek,Asli
Abstract
We present a dataset of behavioural and fMRI observations acquired in the context of humans involved in multimodal referential communication. The dataset contains audio/video and motion-tracking recordings of face-to-face, task-based communicative interactions in Dutch, as well as behavioural and neural correlates of participants’ representations of dialogue referents. Seventy-one pairs of unacquainted participants performed two interleaved interactional tasks in which they described and located 16 novel geometrical objects (i.e., Fribbles) yielding spontaneous interactions of about one hour. We share high-quality video (from three cameras), audio (from head-mounted microphones), and motion-tracking (Kinect) data, as well as speech transcripts of the interactions. Before and after engaging in the face-to-face communicative interactions, participants’ individual representations of the 16 Fribbles were estimated. Behaviourally, participants provided a written description (one to three words) for each Fribble and positioned them along 29 independent conceptual dimensions (e.g., rounded, human, audible). Neurally, fMRI signal evoked by each Fribble was measured during a one-back working-memory task. To enable functional hyperalignment across participants, the dataset also includes fMRI measurements obtained during visual presentation of eight animated movies (35 min total). We present analyses for the various types of data demonstrating their quality and consistency with earlier research. Besides high-resolution multimodal interactional data, this dataset includes different correlates of communicative referents, obtained before and after face-to-face dialogue, allowing for novel investigations into the relation between communicative behaviours and the representational space shared by communicators. This unique combination of data can be used for research in neuroscience, psychology, linguistics, and beyond.
Description
Funding Information: This work was funded by the Netherlands Organisation for Scientific research, through a gravitation grant (024.001.006) to the Language in Interaction Consortium. Publisher Copyright: © 2022
Date
2022-12-01
Journal Title
Journal ISSN
Volume Title
Publisher
Research Projects
Organizational Units
Journal Issue
Keywords
Conceptual alignment, Face-to-face interaction, fMRI, Motion tracking, Multimodal data, Referential communication
Citation
Eijk, L, Rasenberg, M, Arnese, F, Blokpoel, M, Dingemanse, M, Doeller, C F, Ernestus, M, Holler, J, Milivojevic, B, Özyürek, A, Pouw, W, van Rooij, I, Schriefers, H, Toni, I, Trujillo, J & Bögels, S 2022, 'The CABB dataset : A multimodal corpus of communicative interactions for behavioural and neural analyses', Neuroimage, vol. 264, 119734. https://doi.org/10.1016/j.neuroimage.2022.119734