Research

This page lists peer-reviewed papers by the Behavioral Signals team on recognizing emotion and behavior in speech, on methods, and on applied work. Each entry links to an open copy of the paper where one is available, and to the publisher’s page.

Emotion and behavior recognition

Data Augmentation Using GANs for Speech Emotion Recognition

Aggelina Chatziagapi, Georgios Paraskevopoulos, Dimitris Sgouropoulos, Georgios Pantazopoulos, Malvina Nikandrou, Theodoros Giannakopoulos, Athanasios Katsamanis, Alexandros Potamianos, and Shrikanth Narayanan. Interspeech 2019, pp. 171–175.

This paper uses generative adversarial networks to create training examples for underrepresented emotions, and reports a 10% relative performance improvement on IEMOCAP and 5% on FEEL-25k when those minority classes are augmented.

Open PDF (ISCA Archive) · DOI

Audio and ASR-based Filled Pause Detection

Aggelina Chatziagapi, Dimitris Sgouropoulos, Constantinos Karouzos, Thomas Melistas, Theodoros Giannakopoulos, Athanasios Katsamanis, and Shrikanth Narayanan. ACII 2022, IEEE, pp. 1–7.

This paper detects filled pauses — hesitation sounds such as um, uh, and er — from the audio and from automatic speech recognition, including cases where a transcript is imperfect or unavailable.

Open PDF (SAIL) · DOI

Designing and Evaluating Speech Emotion Recognition Systems: A Reality Check Case Study with IEMOCAP

Nikolaos Antoniou, Athanasios Katsamanis, Theodoros Giannakopoulos, and Shrikanth Narayanan. ICASSP 2023, IEEE, pp. 1–5.

This paper reviews how speech emotion recognition systems are tested on the IEMOCAP corpus, sets out evaluation guidelines, and checks whether published open-source models can be reproduced.

Open PDF (arXiv) · DOI

Methods

RobuSER: A Robustness Benchmark for Speech Emotion Recognition

Antonia Petrogianni, Lefteris Kapelonis, Nikolaos Antoniou, Sofia Eleftheriou, Petros Mitseas, Dimitris Sgouropoulos, Athanasios Katsamanis, Theodoros Giannakopoulos, and Shrikanth Narayanan. ACII 2024, IEEE, pp. 219–227.

This paper introduces an open-source benchmark for testing how speech emotion recognition models hold up when audio is corrupted by background noise or signal distortion, and reports accuracy drops of up to 22.77% unweighted and 20.32% weighted on corrupted IEMOCAP for the baseline model.

Publisher page (DOI) · Code on GitHub

Applied work

Cross-Lingual Features for Alzheimer’s Dementia Detection from Speech

Thomas Melistas, Lefteris Kapelonis, Nikos Antoniou, Petros Mitseas, Dimitris Sgouropoulos, Theodoros Giannakopoulos, Athanasios Katsamanis, and Shrikanth Narayanan. Interspeech 2023, pp. 3008–3012.

This paper tests whether speech features trained in one language can help detect signs of Alzheimer’s dementia in another, including a zero-shot setup, and reports that an ensemble trained on English and evaluated on Greek outperforms a baseline by 4.4%.

Open PDF (ISCA Archive) · DOI

Using Oliver API for emotion-aware movie content characterization

Theodoros Giannakopoulos, Spiros Dimopoulos, Georgios Pantazopoulos, Aggelina Chatziagapi, Dimitris Sgouropoulos, Athanasios Katsamanis, Alexandros Potamianos, and Shrikanth Narayanan. CBMI 2019, IEEE, pp. 1–4.

This paper uses the company’s speech emotion API, then called Oliver, to describe the emotional tone of 60 film soundtracks and to tell the work of eight directors apart from those descriptors.

Open PDF (NTUA preprint) · DOI

Emotion-Aware Speech Popularity Prediction: A Use-Case on TED Talks

Dimitrios Sgouropoulos, Petros Mitseas, Sofia Eleftheriou, Theodoros Giannakopoulos, Antonia Petrogianni, Lefteris Kapelonis, Nikolaos Antoniou, Athanasios Katsamanis, and Shrikanth Narayanan. ACII 2024, IEEE, pp. 300–308. Nominated for best paper.

This paper asks whether emotion in a speaker’s voice, used together with the words, helps predict how widely a TED talk is watched, using 2,462 talks and four popularity metrics.

Publisher page (DOI) · Company article (not the paper)

Sign up for the portal and read the docs.