How to test a speech emotion API on your own recordings

How to evaluate a speech emotion API on your own recordings, and why a score from acted speech is not a buying decision.

Illustration of three voice waveforms compared through one window, each feeding a row of a checklist.

How to test a speech emotion API on your own recordings

How to evaluate a speech emotion API on your own recordings, and why a score from acted speech is not a buying decision.

Start with the checklist for choosing a speech emotion API, then use this guide to go deeper. These APIs measure vocal signals and behavior, not inner feelings. A score from acted speech, or from data like the vendor’s training set, is a best case, not a prediction of performance on your audio. Run a fair test on your own recordings.

When you are ready to try an API, create a Behavioral Signals account for 30 free minutes, usable on any job, with no credit card required.

Voice versus words

A speech emotion API should estimate vocal signals and behavior from the audio itself, not from a transcript, and not claim to read what a person feels. A transcript tells you what was said, not that the speaker sped up or sounded withdrawn, and it fails when the words are wrong.

Some vendors score sentiment from the transcript text rather than from the voice. That is a different product from an acoustic model. If you already have speech-to-text, emotion should sit next to that stack, not replace it.

Lisa Feldman Barrett puts the limit plainly. In The Verge, she said of emotion-recognition tools: “They can detect a scowl, but that’s not the same thing as detecting anger.” (The Verge, 2019) In Scientific American, writing also about voice and other signals, she added: “Emotion AI systems, therefore, do not detect emotions. They detect physical signals… not the psychological meaning of those signals.” (Scientific American, 2022)

Labels, calibration and test data

Ask which labels come back, and whether each class includes a probability. Prefer the same named measures every time, such as emotion, strength and positivity, plus hesitation, engagement and speaking rate. Some APIs instead answer prompts you write, which can change from one request to the next and makes a fair comparison harder. If you need open-ended answers to prompts, a named-measure API is not the obvious fit.

Label sets are not interchangeable. Your application, not the vendor, should set the threshold at which something happens. A missing or failed estimate is “no signal”, never “neutral”, and never a zero filled in later.

Ask, too, whether a published score was measured on real conversations or on acted speech.

Why acted-speech benchmarks do not predict accuracy on real calls

Many published speech emotion scores come from recordings in which people were asked to portray an emotion. That is a different task from recognizing emotion in ordinary conversation.

A 2025 study of customer-service audio reports “noticeable performance degradation of models trained on acted data to the naturalistic data,” with a particular focus on anger. (Kawaoto, Gupta, Yu and Dakota, RANLP 2025)

A 2025 paper on naturalistic conversational speech makes the dataset problem explicit: “Most training and evaluation datasets contain acted or pseudo-acted speech (e.g., podcast speech) in which emotion expressions may be exaggerated or otherwise intentionally modified.” On labeled Switchboard conversations, the authors found “especially poor generalization for anger.” In their results table, the highest unweighted-average F1 among the models they tested was 0.391. (Romana et al., arXiv:2510.13906)

Treat a headline percentage, especially from an acted or in-distribution test, as a best case. The test that matters is the one you run on your own calls.

Language and accent

Start by asking about the model family, not how many languages are listed. Ask whether the API reads how something is said, or depends on a transcript. Language-agnostic, paralinguistic models are not tied to one language. A published list shows where performance is strongest, not that the model works only there.

Ask whether accuracy was checked across languages and accents. Coverage in research data is uneven: KuralHub’s catalog of speech emotion datasets for the 70 most widely spoken languages found at least one dataset for 45 of them, and none for the other 25. (KuralHub) Bias can travel with the models. Slaughter et al. tested 16 English speech models and found that 14 or more associated more positive, pleasant-sounding affect with U.S. accents than with non-U.S. accents, and with other historically favored groups. In 69% of their tests, that association carried into speech emotion recognition. (Slaughter et al., arXiv:2310.18877)

If a vendor quotes an accuracy figure, ask which language and accents it was measured on.

Batch, streaming and transport

Use batch for recorded files, and streaming when you need estimates during a call. Many teams need both. Do not assume that a field available in a file job is also available on the live stream.

Ask how long batch jobs take on files of the size you actually have. Ask for minimum and maximum file or session lengths, and which transports each mode uses. Batch is commonly REST. Live streams may use gRPC, WebSocket, or another long-lived connection.

WebSocket can be difficult on enterprise networks. The proxy or DMZ often terminates TLS and re-encrypts the traffic, which is where the HTTP upgrade is stripped or the long connection times out. Ask whether the API can stream over a transport your network already allows.

Behavioral Signals has been deployed and tested in production environments at major banks and telcos. Enterprise networks route traffic through internal proxies, TLS inspection and security review, which is where WebSocket-only streaming stalls. Behavioral Signals streams over gRPC on port 443.

If you are wiring estimates into a live agent, continue on the voice-agent emotion API page.

Transcription and diarization

Ask whether the transcript and speaker labels come back with the emotion result, in which mode, and whether they are billed separately. Some APIs include speech recognition and diarization only in batch. Do not assume a live stream also returns words or speaker labels.

Behavioral Signals is not the obvious choice if your main need is a live transcript from the same API: streaming returns emotion and behavior results, not a transcript. If you need live transcripts, keep your own speech-to-text path.

Latency

Do not buy on a published latency figure alone. Ask how often results are emitted, then measure end-to-end time on your own network, including your proxy. A vendor’s result interval is not the time until your application can act. For batch, measure time-to-result on your own files.

For a live agent, plan to adjust from the next turn onward, not mid-sentence. If an estimate is late or missing, continue with “no signal” for that turn, never with “neutral”.

Deployment, security and the rules that apply

Ask whether the same models can run as a cloud API and at the edge. Ask whether the vendor holds a SOC 2 report, and request the current report rather than relying on a marketing line. Where your jurisdiction and use case require it, callers should be told their audio is analyzed.

Rules on emotion inference differ by context and region. Before you deploy, check the rules that apply to your use case and jurisdiction, and take advice from counsel.

Deepfake detection

Include deepfake detection on the same checklist. Ask for audio detection that returns a probability, not only a yes-or-no flag. Ask how much audio the vendor needs before it flags synthetic speech, because that amount affects whether you can flag a live call in time. Then ask whether video is supported, in which mode, and whether the API can name the generator that most likely produced a flagged clip.

Emotion estimation and deepfake detection should be separate jobs, even when they share an account. Until a dedicated API page is live, more detail is on deepfake speech detection for contact centers and in the API documentation.

Pricing and a free trial

Ask what unit you are billed in — per minute, per hour, per second, tokens, or a subscription — and whether you can estimate an hour of your own audio up front. Confirm what is free to start, whether a card is required, whether transcription, diarization or deepfake detection are billed separately, and whether a free allowance covers the jobs you need to test.

Current Behavioral Signals terms are on behavioralsignals.com/portal/. Every account starts with 30 free minutes, usable on any job including deepfake detection, with no credit card required.

Documentation, integrations and continuity

Before you build a proof of concept, read the docs for both batch and streaming. Look for a published request and response shape, current field names, authentication, error behavior, and an official SDK or a proto file. Check that batch and streaming are documented as different contracts, especially for transcripts.

Ask whether the integration is a stable release, and whether the vendor has recently retired APIs, and how much notice it gave. Thin docs, an unstable integration, or a short notice period can force a migration you did not plan.

How to run your own evaluation

Use this method on every vendor you shortlist. Work from the checklist so the questions stay the same. Label sets differ, so a single percentage is not comparable across APIs without a shared test set.

  1. Ask how any published test was run. Was it acted speech or real conversations? Was the model trained on the same kind of data it was scored on? Treat an acted or in-distribution score as a best case.
  2. Build a small, consented set of your own recordings. Use the audio you will run in production, with the same languages, channels, accents and devices. Annotate it yourself, with the labels your application will use. If the decision that matters is “should the agent slow down?”, annotate for that.
  3. Hold the set out. Do not tune thresholds on the same files you use for the final comparison, and do not send the evaluation set to a vendor as training help if you need an independent read.
  4. Compare like for like. Use the same files, the same task and the same scoring rules for every API. If the label sets differ, do not invent a conversion. Score each API on the decision your application actually makes.
  5. Measure more than a headline accuracy. Record misses, false positives, calibration, failed requests, end-to-end streaming time, and how long batch jobs take on your files. Do not fill missing scores with zero or “neutral”.
  6. Start with a free allowance before you buy. Create a Behavioral Signals account to get 30 free minutes, usable on any job including deepfake detection, with no credit card required. The current request shape is in the API documentation.

Run the test, then decide. A short worked example of how Behavioral Signals answers each row is on the how to choose a speech emotion API page.


All Behavioral Signals outputs are probabilistic estimates of vocal signals and behavior expressed in speech, not determinations of what a person feels, intends or believes.

Where your jurisdiction and use case require it, callers should be told their audio is analyzed.

← All posts

← All posts