Focus API · Voice agents · Behavioral Signals
Emotion detection API for voice agents: respond to a caller’s expressed emotion in real time
To make an AI voice agent respond to a caller’s emotion in real time, stream the caller’s audio to a speech emotion API. Read the per-utterance estimate it returns: the expressed emotion and its probability, plus strength, positivity, hesitation and engagement. Stress and confidence signals, built with Focus, are added to your API output on request. Then add that estimate to your LLM’s prompt for the next turn, or use it to trigger a rule such as slowing down, confirming, or offering a human.
Behavioral Signals is a drop-in replacement for the Hume API. Send it the caller’s audio, and it returns the expressed emotion plus strength, positivity, hesitation and engagement, each with probabilities. Your own logic decides what the agent does next.
30 free minutes, no credit card required. Behavioral: $0.10/min batch, $0.15/min streaming.
forked in two
All outputs are probabilistic estimates of signals expressed in speech. They are not determinations of what a person feels or intends.
01 · The direct answer
What does a speech emotion API add to a voice agent?
A transcript tells your agent what the caller said. It doesn’t tell it that the caller is speaking fast, pausing, or sounds withdrawn. Some voice platforms ask the LLM to guess the caller’s emotion from the transcript. An acoustic API reads tone, pace and hesitation from the audio itself. Behavioral Signals returns acoustic emotion and behavior estimates per utterance — voice emotion analysis your agent can act on.
What a transcript misses
A transcript tells your agent what the caller said. It doesn’t tell it that the caller is speaking fast, pausing, or sounds withdrawn.
Acoustic, alongside the transcript
An acoustic API reads tone, pace and hesitation from the audio itself. Behavioral Signals returns acoustic emotion and behavior estimates per utterance.
02 · Moving off Hume EVI?
Moving off Hume EVI? Keep emotion in your voice agent
Hume’s 2 October 2026 changelog notice says access to its EVI and TTS APIs ends on November 13, 2026 at 12:01 a.m. EST, and that account data is permanently deleted after that date (Forasoft; Burki).
A Forasoft guide to audio emotion detection advises teams on EVI to “move the voice loop to another stack and the emotion layer to a self-hosted wav2vec2, HuBERT or emotion2vec+ head, or to a remaining API such as Behavioral Signals.”Forasoft
Behavioral Signals is a drop-in replacement for the Hume API. Send it the caller’s audio, and it returns the expressed emotion plus strength, positivity, hesitation and engagement, each with probabilities. Stress and confidence signals, built with Focus, are added to your API output on request. For EVI agents, it replaces the Hume API’s emotion layer.
Replace Hume EVI’s emotion layer: get an API key and run your own recorded calls before you switch.
Start now
Every account starts with 30 free minutes, with no credit card required (pricing). Run your own recorded calls before you switch.
Get an API keyMoving off Hume Expression Measurement instead? See the Hume AI alternative.
03 · Signals
Which emotion signals can a voice agent read from live audio?
The streaming API returns results per fixed two-second segment, per complete utterance, or both. In the Python SDK this is set with the level option (segment, utterance or all). Behavioral tasks and their classes, as listed in the API docs:
emotionhappy, angry, neutral, sad
positivitynegative, neutral, positive
strengthweak, neutral, strong
hesitationyes / no
engagementengaged, neutral, withdrawn
speaking_rateslow, normal, fast
intensitya continuous score between 0 and 1, with no label
genderfemale, male (estimated from the voice)
age18–22, 23–30, 31–45, 46–65 (estimated from the voice)
With Focus
Also available: stress and confidence signals, built with Focus and added to your API output on request. About Focus.
In batch, the same API also returns the transcript, speakers and language per utterance. Speaker diarization and ASR are included at no extra charge.
Probabilities, not verdicts
Each classified task returns a posterior (probability) for every class plus a finalLabel, the API’s single answer for that task. Your application sets the threshold at which the agent acts.
Segment results arrive at a fixed interval, which suits immediate feedback. Utterance results analyze the whole utterance after voice-activity detection, and the docs describe them as more accurate.
Languages
Our emotion and behavior models are paralinguistic: they read how something is said, not the words, so they aren’t tied to one language. Performance is strongest in the 11 languages they’re trained on: English, Arabic, Russian, Chinese, French, German, Spanish, Portuguese, Japanese, Hindi and Tamil.
What a result looks like
The field names below are the documented result shape. The values are placeholders, not real output.
{
"message_id": 12,
"results": [
{
"id": "4",
"startTime": "<seconds>",
"endTime": "<seconds>",
"task": "hesitation",
"prediction": [
{"label": "yes", "posterior": "<probability>"},
{"label": "no", "posterior": "<probability>"}
],
"finalLabel": "yes",
"level": "utterance"
}
]
}
04 · Your stack
Where does emotion detection sit in a voice-agent stack?
It sits next to your stack, not in place of it. In the streaming setup, fork the caller’s audio. One copy goes to your speech-to-text and LLM as it does today. The other goes to the Behavioral Signals streaming API for emotion, strength and positivity per utterance.
Streaming · live call
Behavioral Signals replaces the Hume API’s emotion layer next to your LLM, TTS and turn-taking. It connects through your own code over the Python SDK or a gRPC client generated from the published api.proto for streaming (REST for batch). That means it can feed any LLM, agent framework or voice platform.
The same models run as a cloud API, in customer-hosted deployments and at the edge.
Keep the emotion path optional. If the emotion stream drops or lags, the call should carry on with transcription and the LLM as normal. The agent then simply has no emotion context for that turn. Treat a missing estimate as “no signal”, never as “neutral”.
05 · Integration
How do I wire it in? (illustrative integration flow)
Illustrative only. Check the streaming docs for current method signatures.
Get credentials
Create a self-serve account and issue an API key at https://behavioralsignals.com/portal/. The console shows your client ID. New accounts include 30 free minutes, with no credit card required.
Send the caller’s audio
From your telephony or WebRTC layer, send the inbound caller channel as 16-bit linear PCM, mono. The models run at 16 kHz, and audio at other sample rates is resampled to 16 kHz. The docs recommend chunks of 100–500 ms.
Open a stream
Install the SDK with
pip install behavioralsignals(Python 3.10+), or generate a client from the publishedapi.proto.Read estimates
Per segment or utterance. Keep a short rolling window per call so one segment doesn’t swing the agent.
Pass them to the LLM by two paths
Per turn: annotate the caller’s message with a compact line, e.g.
[voice: hesitation=yes p=<posterior>, speaking_rate=fast, engagement=withdrawn]. Per session: a standing instruction, e.g. “If voice signals suggest hesitation, slow down and confirm before moving on.”Or fire rules your code owns
Such as changing TTS pacing, confirming instead of upselling, or offering a human.
# Illustrative sketch: SDK calls from the docs, agent logic is pseudo-code
from behavioralsignals import Client, StreamingOptions
client = Client() # reads BEHAVIORALSIGNALS_CID / BEHAVIORALSIGNALS_API_KEY
options = StreamingOptions(sample_rate=16000, encoding="LINEAR_PCM", level="utterance")
for msg in client.behavioral.stream_audio(audio_stream=caller_pcm_chunks, options=options):
for r in msg.results:
call_state.update(r.task, r.finalLabel, r.prediction) # pseudo
if call_state.p("hesitation", "yes") > THRESHOLD: # your threshold
agent.add_context(call_state.summary()) # pseudo
agent.slow_down_and_confirm() # pseudo
Example signal-to-behavior rules
Starting points you tune on your own calls. They are not product outputs.
| If the estimate shows… | the agent might… |
|---|---|
| hesitation = yes, above your threshold | pause, restate the options, confirm before moving on |
| speaking_rate = fast | shorten replies, get to the answer first |
| engagement = withdrawn over several utterances | check in, offer a simpler path or a human |
| strength = strong for several utterances in a row | slow TTS pacing, keep replies brief |
Escalation, satisfaction and intent aren’t API outputs. They are decisions your agent makes from these estimates.
Tell callers
Let callers know that their audio is analyzed, as your jurisdiction and use case require.
06 · Timing
When do results arrive during a live call?
Results stream back during the call. Segment results arrive at fixed two-second intervals, and utterance results arrive at the end of each utterance. Plan for the agent to adjust from the next turn onward, not mid-sentence.
07 · Deepfake detection
Is the caller’s voice real? Adding deepfake detection alongside the call
Behavioral Signals also detects synthetic speech. Deepfake detection is speaker-agnostic. It returns bonafide or spoofed with a posterior for each, per utterance or segment. It is a detection result about the audio, independent of the emotion outputs, and not a verdict about a person.
For streaming it is a separate gRPC method (DeepfakeDetection, alongside StreamAudio in the published api.proto). In the SDK it is called as client.deepfakes.stream_audio. Validate on your own audio before launch.
Pricing
Start with 30 free minutes.
Every account starts with 30 free minutes, with no credit card required. Then pay per minute from one prepaid balance.
$0.15 per minute streaming
$0.50 per minute streaming
Video deepfake detection is experimental.
08 · FAQ
Questions
Which API can I use to add emotion detection to a voice agent?
Behavioral Signals has a streaming speech emotion API you can use. You send live audio, and for each two-second segment or full utterance it returns estimates of expressed emotion, positivity, strength, hesitation, engagement and speaking rate, with a probability for every class. Your agent reads them through your own code.
Does Behavioral Signals replace the Hume API in a voice agent?
Yes. Behavioral Signals replaces the Hume API’s emotion layer, next to your own LLM, text-to-speech and turn-taking. Per utterance it returns the expressed emotion plus strength, positivity, hesitation and engagement, each with probabilities. Stress and confidence signals, built with Focus, are added to your API output on request. In batch, the transcript, speakers and language come back in the same result.
What audio format does streaming need?
16-bit linear PCM, mono. The models run at 16 kHz, and audio at other sample rates is resampled to 16 kHz. Recommended chunks are 100–500 ms.
Which languages does it support?
The emotion and behavior models are paralinguistic: they read how something is said, not the words, so they aren’t tied to one language. Performance is strongest in the 11 languages they’re trained on: English, Arabic, Russian, Chinese, French, German, Spanish, Portuguese, Japanese, Hindi and Tamil.
What does it cost?
Free start: every account starts with 30 free minutes, with no credit card required. Behavioral: $0.10 per minute batch, $0.15 per minute streaming. Deepfake audio: $0.25 per minute batch, $0.50 per minute streaming. Deepfake video: $0.50 per minute, batch. Video deepfake detection is experimental. Volume bonus: a prepaid top-up of $50 or more adds 5% credit, $100 or more adds 10%, and $200 or more adds 20%. Tiers don’t stack; the highest tier you clear applies. ASR: speaker diarization and ASR are included in batch processing at no extra charge. See pricing.
Can I run it on-premises or at the edge?
Yes. The same models run as a cloud API, in customer-hosted deployments and at the edge. On-premises deployment isn’t listed with a per-minute price, so contact us for details.
Get an API key (30 free minutes) · Read the streaming docs
Last reviewed: 10 November 2026