CAFAS: teaching AI to hear emotion in the voice — and doing it lightweight
The voice reveals emotion even without words: tone, tempo, a tremor — all of these are signals a human hears intuitively. Our task is to teach a neural network the same thing, and to do it so that running the model does not require renting an expensive server GPU but can be deployed on an ordinary device.
Our team has presented a new architecture for speech emotion recognition.
Modern SER models achieve good accuracy but pay for it with an enormous number of parameters, becoming heavy and slow. Most of them also rely either on acoustic features (tone, frequency) or on semantic embeddings from self-supervised models, and rarely combine the two types of signal effectively.
Our CAFAS architecture combines two streams of information through a cross-attention mechanism:
the acoustic branch — MFCC features pass through a dedicated encoder (LSTM + transformer) that teaches them to "talk" to each other;
the semantic branch — Wav2vec 2.0 extracts contextual representations of speech;
cross-attention fusion — queries are taken from the acoustic features while keys and values come from the semantic ones, allowing the model to selectively highlight relevant details.
An auxiliary classifier was added for more stable training.
The numbers. On the IEMOCAP dataset (12 hours of dialogue with annotated emotions) the model achieved:
74.6% weighted accuracy, making it competitive with modern counterparts;
the best unweighted accuracy among all compared models;
only 112M parameters — almost three times fewer than some alternatives (a comparable model with 317M parameters shows similar accuracy);
24 ms latency on a GPU, which is effectively instantaneous for real applications.
The model reaches 91.87% of the accuracy of the best baseline while using only 49.1% of its parameters. Separately interesting: analysis of the attention maps showed that the model really does "notice" key emotionally loaded words and ignores pauses, which makes its decisions interpretable.
The applications are numerous: from voice assistants to mental health monitoring — in other words, anywhere it matters not only what a person said but how. If you work in speech technology, audio processing or HCI, we are open to collaboration.