Request a demo Sign in
Sign in Request a demo
4–5 min read

Tell it what matters: Focus is now in the Behavioral Signals API

Focus lets you list the labels you care about and pick a sensitivity. The API flags moments worth a look next to the results you already get, in batch or streaming.

Illustration of Focus in the Behavioral Signals API: an audio waveform with three stretches flagged as moments worth a look, above a results table in which the flag row has the same id, start time and end time as its utterance. Illustration only, not real API output.

You wired up an emotion API, opened the JSON and wondered which numbers to use. Every utterance comes back with a probability for every label. That is a lot of data, and picking what matters to your application is hard.

Focus is our answer. It is now part of the Behavioral Signals API, for recorded audio and live streams. You list the labels you care about and pick a sensitivity. The API flags the moments worth a look, right next to the results you already get.

Also available: stress and confidence signals, built with Focus and added to your API output on request.

The signal was already in your results

For every utterance, the API returns a probability for each class label of each behavioral task, and finalLabel is the label with the largest probability. A top label keeps only the winner. So a moment that is mostly neutral, with a strong second reading, can slip by.

Focus reads the full set of probabilities for you. Where a label you chose is likely but isn’t already the top result, it adds a flag.

How Focus works

  1. List the labels you want flagged. Choose the labels that matter to your app, such as angry, happy or negative tone. Stress and confidence signals are also available through Focus.
  2. Pick a sensitivity. Low, default or high. High flags more moments; low flags fewer.
  3. Read the flag rows. Each flag comes back as one extra row, with the same id, start time and end time as the utterance or segment it belongs to.
  4. Ask for nothing and nothing changes. If you don’t request Focus, your results are exactly what you get today.

Focus is a parameter on the results request you already make, and a focus block in your stream config. Your existing integration keeps working, and there is nothing to migrate.

One utterance, without and with Focus

Here is one utterance from a roadside-assistance call. Without Focus, emotion and positivity both come back neutral. Read the same results with focus=negative_tone&sensitivity=low and everything else in the response stays the same. The response gets one extra row:

{
  "id": "0",
  "startTime": "0.031",
  "endTime": "7.473",
  "task": "focus",
  "prediction": [
    {"label": "negative_tone", "posterior": "0.6072"}
  ],
  "finalLabel": "suspected_negative_tone",
  "level": "utterance"
}

The full before and after is on the Focus page.

Recorded audio: re-tune, don’t re-upload

Upload your audio as usual. Then add focus and, if you like, sensitivity to your results request:

GET …/v5/clients/{client-id}/processes/{pid}/results?focus=angry,negative_tone&sensitivity=high

Focus is worked out when you read results, so you can read the same job again with other labels or another sensitivity, with no new upload. Past jobs work too.

With the Python SDK, pass the same choices when you read results:

result = client.behavioral.wait_for_result(
    pid=process.pid,
    focus=["angry", "negative_tone"],
    sensitivity="high",
)
for r in result.results:
    if r.task == "focus":
        print(r.st, r.et, r.finalLabel)

Live streams: same labels, same sensitivity

For a live gRPC stream, add a focus block, with your labels and an optional sensitivity, to the first config message. Flag rows arrive next to your other results, at segment level, utterance level or both. The labels and sensitivity settings are the same in batch and streaming, so one choice of what matters carries across both.

You don’t need to be a voice expert

Stress and confidence signals are built with Focus. Focus combines several voice signals for you, so your code reads a flag, not a formula. No thresholds to hand-tune, no formulas combining voice signals, and no re-uploads to try a new setting.

What a flag means

A flag marks a moment worth a look, based on how the speech sounds. It isn’t a statement about what the speaker feels, and what your app does with it is your call.

What else comes with the API

Speaker diarization and ASR are included at no extra charge. Our emotion and behavior models are paralinguistic: they read how something is said, not the words, so they aren’t tied to one language. Performance is strongest in the 11 languages they’re trained on.

Focus covers the behavioral API. Voice deepfake detection runs through its own Deepfakes API.

Focus is new, but the science under it isn’t

Behavioral Signals spun out of USC’s SAIL in 2016, the lab behind pioneering work in machine detection of human emotion. Our team has kept publishing with the lab since, including:

Related articles

Start with 30 free minutes

Sign up on the portal and try Focus on your own audio. The free minutes cover both batch and streaming, and no credit card is needed. After that, it’s $0.10 per minute for batch and $0.15 per minute for streaming. You can also read the Focus docs first.

Start with 30 free minutes

Author: Theodoros Giannakopoulos is Director of ML at Behavioral Signals.

← All posts