Focus API · Voice agents · Behavioral Signals

Emotion detection API for voice agents: respond to a caller’s expressed emotion in real time

To make an AI voice agent respond to a caller’s emotion in real time, stream the caller’s audio to a speech emotion API. Read the per-utterance estimate it returns: the expressed emotion and its probability, plus strength, positivity, hesitation and engagement. Stress and confidence signals, built with Focus, are added to your API output on request. Then add that estimate to your LLM’s prompt for the next turn, or use it to trigger a rule such as slowing down, confirming, or offering a human.

Behavioral Signals is a drop-in replacement for the Hume API. Send it the caller’s audio, and it returns the expressed emotion plus strength, positivity, hesitation and engagement, each with probabilities. Your own logic decides what the agent does next.

30 free minutes, no credit card required. Behavioral: $0.10/min batch, $0.15/min streaming.

Caller audio
forked in two
Your stack, as today
Speech-to-textwhat the caller said
LLM / agentdecides the reply
Text-to-speechspeaks the reply
In parallel
Behavioral Signalsstreaming API
Per-utterance estimates
hesitationengagementspeaking_rate
Your logicsets the threshold
Estimate added to the LLM prompt for the next turn, or used to trigger a rule
Illustration only. Not real API output.

All outputs are probabilistic estimates of signals expressed in speech. They are not determinations of what a person feels or intends.

01 · The direct answer

What does a speech emotion API add to a voice agent?

A transcript tells your agent what the caller said. It doesn’t tell it that the caller is speaking fast, pausing, or sounds withdrawn. Some voice platforms ask the LLM to guess the caller’s emotion from the transcript. An acoustic API reads tone, pace and hesitation from the audio itself. Behavioral Signals returns acoustic emotion and behavior estimates per utterance — voice emotion analysis your agent can act on.

What a transcript misses

A transcript tells your agent what the caller said. It doesn’t tell it that the caller is speaking fast, pausing, or sounds withdrawn.

Acoustic, alongside the transcript

An acoustic API reads tone, pace and hesitation from the audio itself. Behavioral Signals returns acoustic emotion and behavior estimates per utterance.

02 · Moving off Hume EVI?

Moving off Hume EVI? Keep emotion in your voice agent

Hume’s 2 October 2026 changelog notice says access to its EVI and TTS APIs ends on November 13, 2026 at 12:01 a.m. EST, and that account data is permanently deleted after that date (Forasoft; Burki).

A Forasoft guide to audio emotion detection advises teams on EVI to “move the voice loop to another stack and the emotion layer to a self-hosted wav2vec2, HuBERT or emotion2vec+ head, or to a remaining API such as Behavioral Signals.”Forasoft

Behavioral Signals is a drop-in replacement for the Hume API. Send it the caller’s audio, and it returns the expressed emotion plus strength, positivity, hesitation and engagement, each with probabilities. Stress and confidence signals, built with Focus, are added to your API output on request. For EVI agents, it replaces the Hume API’s emotion layer.

Replace Hume EVI’s emotion layer: get an API key and run your own recorded calls before you switch.

Start now

Every account starts with 30 free minutes, with no credit card required (pricing). Run your own recorded calls before you switch.

Get an API key

Moving off Hume Expression Measurement instead? See the Hume AI alternative.

03 · Signals

Which emotion signals can a voice agent read from live audio?

The streaming API returns results per fixed two-second segment, per complete utterance, or both. In the Python SDK this is set with the level option (segment, utterance or all). Behavioral tasks and their classes, as listed in the API docs:

emotion

happy, angry, neutral, sad

positivity

negative, neutral, positive

strength

weak, neutral, strong

hesitation

yes / no

engagement

engaged, neutral, withdrawn

speaking_rate

slow, normal, fast

intensity

a continuous score between 0 and 1, with no label

gender

female, male (estimated from the voice)

age

18–22, 23–30, 31–45, 46–65 (estimated from the voice)

With Focus

Also available: stress and confidence signals, built with Focus and added to your API output on request. About Focus.

In batch, the same API also returns the transcript, speakers and language per utterance. Speaker diarization and ASR are included at no extra charge.

Probabilities, not verdicts

Each classified task returns a posterior (probability) for every class plus a finalLabel, the API’s single answer for that task. Your application sets the threshold at which the agent acts.

Segment results arrive at a fixed interval, which suits immediate feedback. Utterance results analyze the whole utterance after voice-activity detection, and the docs describe them as more accurate.

Languages

Our emotion and behavior models are paralinguistic: they read how something is said, not the words, so they aren’t tied to one language. Performance is strongest in the 11 languages they’re trained on: English, Arabic, Russian, Chinese, French, German, Spanish, Portuguese, Japanese, Hindi and Tamil.

What a result looks like

The field names below are the documented result shape. The values are placeholders, not real output.

{
  "message_id": 12,
  "results": [
    {
      "id": "4",
      "startTime": "<seconds>",
      "endTime": "<seconds>",
      "task": "hesitation",
      "prediction": [
        {"label": "yes", "posterior": "<probability>"},
        {"label": "no",  "posterior": "<probability>"}
      ],
      "finalLabel": "yes",
      "level": "utterance"
    }
  ]
}

04 · Your stack

Where does emotion detection sit in a voice-agent stack?

It sits next to your stack, not in place of it. In the streaming setup, fork the caller’s audio. One copy goes to your speech-to-text and LLM as it does today. The other goes to the Behavioral Signals streaming API for emotion, strength and positivity per utterance.

Streaming · live call

Streaming live-call architecture Caller audio goes to the voice agent. An audio fork sends one branch to the customer speech-to-text and LLM, and the other to the Behavioral Signals streaming API, which returns emotion, strength and positivity per utterance to the agent logic. The response goes out through the customer text-to-speech. Caller audio Voice agent agent logic FORK Speech-to-text + LLM (yours) Behavioral Signals streaming API emotion · strength · positivity emotion signals TTS yours Streaming live-call architecture Caller audio goes to the voice agent. An audio fork sends one branch to the customer speech-to-text and LLM, and the other to the Behavioral Signals streaming API, which returns emotion, strength and positivity per utterance to the agent logic. The response goes out through the customer text-to-speech. Caller audio Voice agent agent logic FORK Speech-to-text + LLM (yours) Behavioral Signals streaming API emotion · strength · positivity emotion signals TTS yours
Streaming architecture (live call) Caller audio goes to the voice agent. An audio fork sends one branch to your speech-to-text and LLM, and the other to the Behavioral Signals streaming API, which returns emotion, strength and positivity per utterance to the agent logic (your code reads them on the next turn or fires a rule). The response goes out via your TTS.

Behavioral Signals replaces the Hume API’s emotion layer next to your LLM, TTS and turn-taking. It connects through your own code over the Python SDK or a gRPC client generated from the published api.proto for streaming (REST for batch). That means it can feed any LLM, agent framework or voice platform.

The same models run as a cloud API, in customer-hosted deployments and at the edge.

Keep the emotion path optional. If the emotion stream drops or lags, the call should carry on with transcription and the LLM as normal. The agent then simply has no emotion context for that turn. Treat a missing estimate as “no signal”, never as “neutral”.

05 · Integration

How do I wire it in? (illustrative integration flow)

Illustrative only. Check the streaming docs for current method signatures.

  1. Get credentials

    Create a self-serve account and issue an API key at https://behavioralsignals.com/portal/. The console shows your client ID. New accounts include 30 free minutes, with no credit card required.

  2. Send the caller’s audio

    From your telephony or WebRTC layer, send the inbound caller channel as 16-bit linear PCM, mono. The models run at 16 kHz, and audio at other sample rates is resampled to 16 kHz. The docs recommend chunks of 100–500 ms.

  3. Open a stream

    Install the SDK with pip install behavioralsignals (Python 3.10+), or generate a client from the published api.proto.

  4. Read estimates

    Per segment or utterance. Keep a short rolling window per call so one segment doesn’t swing the agent.

  5. Pass them to the LLM by two paths

    Per turn: annotate the caller’s message with a compact line, e.g. [voice: hesitation=yes p=<posterior>, speaking_rate=fast, engagement=withdrawn]. Per session: a standing instruction, e.g. “If voice signals suggest hesitation, slow down and confirm before moving on.”

  6. Or fire rules your code owns

    Such as changing TTS pacing, confirming instead of upselling, or offering a human.

# Illustrative sketch: SDK calls from the docs, agent logic is pseudo-code
from behavioralsignals import Client, StreamingOptions

client = Client()  # reads BEHAVIORALSIGNALS_CID / BEHAVIORALSIGNALS_API_KEY
options = StreamingOptions(sample_rate=16000, encoding="LINEAR_PCM", level="utterance")

for msg in client.behavioral.stream_audio(audio_stream=caller_pcm_chunks, options=options):
    for r in msg.results:
        call_state.update(r.task, r.finalLabel, r.prediction)    # pseudo
    if call_state.p("hesitation", "yes") > THRESHOLD:             # your threshold
        agent.add_context(call_state.summary())                    # pseudo
        agent.slow_down_and_confirm()                              # pseudo

Example signal-to-behavior rules

Starting points you tune on your own calls. They are not product outputs.

If the estimate shows…the agent might…
hesitation = yes, above your thresholdpause, restate the options, confirm before moving on
speaking_rate = fastshorten replies, get to the answer first
engagement = withdrawn over several utterancescheck in, offer a simpler path or a human
strength = strong for several utterances in a rowslow TTS pacing, keep replies brief

Escalation, satisfaction and intent aren’t API outputs. They are decisions your agent makes from these estimates.

Tell callers

Let callers know that their audio is analyzed, as your jurisdiction and use case require.

06 · Timing

When do results arrive during a live call?

Results stream back during the call. Segment results arrive at fixed two-second intervals, and utterance results arrive at the end of each utterance. Plan for the agent to adjust from the next turn onward, not mid-sentence.

07 · Deepfake detection

Is the caller’s voice real? Adding deepfake detection alongside the call

Behavioral Signals also detects synthetic speech. Deepfake detection is speaker-agnostic. It returns bonafide or spoofed with a posterior for each, per utterance or segment. It is a detection result about the audio, independent of the emotion outputs, and not a verdict about a person.

For streaming it is a separate gRPC method (DeepfakeDetection, alongside StreamAudio in the published api.proto). In the SDK it is called as client.deepfakes.stream_audio. Validate on your own audio before launch.

Pricing

Start with 30 free minutes.

Every account starts with 30 free minutes, with no credit card required. Then pay per minute from one prepaid balance.

Behavioral$0.10 per minute batch
$0.15 per minute streaming
Deepfake audio$0.25 per minute batch
$0.50 per minute streaming
Deepfake video$0.50 per minute, batch.
Video deepfake detection is experimental.
Volume bonusPrepaid top-up of $50+ adds 5%, $100+ adds 10%, $200+ adds 20%. Tiers don’t stack.
Get an API key See pricing Focus

08 · FAQ

Questions

Which API can I use to add emotion detection to a voice agent?

Behavioral Signals has a streaming speech emotion API you can use. You send live audio, and for each two-second segment or full utterance it returns estimates of expressed emotion, positivity, strength, hesitation, engagement and speaking rate, with a probability for every class. Your agent reads them through your own code.

Does Behavioral Signals replace the Hume API in a voice agent?

Yes. Behavioral Signals replaces the Hume API’s emotion layer, next to your own LLM, text-to-speech and turn-taking. Per utterance it returns the expressed emotion plus strength, positivity, hesitation and engagement, each with probabilities. Stress and confidence signals, built with Focus, are added to your API output on request. In batch, the transcript, speakers and language come back in the same result.

What audio format does streaming need?

16-bit linear PCM, mono. The models run at 16 kHz, and audio at other sample rates is resampled to 16 kHz. Recommended chunks are 100–500 ms.

Which languages does it support?

The emotion and behavior models are paralinguistic: they read how something is said, not the words, so they aren’t tied to one language. Performance is strongest in the 11 languages they’re trained on: English, Arabic, Russian, Chinese, French, German, Spanish, Portuguese, Japanese, Hindi and Tamil.

What does it cost?

Free start: every account starts with 30 free minutes, with no credit card required. Behavioral: $0.10 per minute batch, $0.15 per minute streaming. Deepfake audio: $0.25 per minute batch, $0.50 per minute streaming. Deepfake video: $0.50 per minute, batch. Video deepfake detection is experimental. Volume bonus: a prepaid top-up of $50 or more adds 5% credit, $100 or more adds 10%, and $200 or more adds 20%. Tiers don’t stack; the highest tier you clear applies. ASR: speaker diarization and ASR are included in batch processing at no extra charge. See pricing.

Can I run it on-premises or at the edge?

Yes. The same models run as a cloud API, in customer-hosted deployments and at the edge. On-premises deployment isn’t listed with a per-minute price, so contact us for details.

Get an API key (30 free minutes) · Read the streaming docs

Get an API key

Last reviewed: 10 November 2026