pipecat-behavioralsignals 0.2.0 is out. It’s an open-source perception producer that adds emotion and behavior readings from the voice to most user turns your LLM sees.
By Dimitris Sgouropoulos, Technical Product Lead, Behavioral Signals
Speech-to-text turns every user turn into words. It doesn’t capture how those words were said: the clipped “fine” from a frustrated customer, the hesitation before a yes, the energy that drains out of a conversation. Your LLM is left to guess from the text alone.
pipecat-behavioralsignals fixes that for agents built on Pipecat. It’s an open-source package that streams the user’s voice to the Behavioral Signals API and adds what it reads to most user turns, so the LLM gets the reading and the words together.
The integration is listed in Pipecat’s Supported Services catalog, under the Perception category. Pipecat lists it as a community-maintained integration. It sits next to Pipecat’s own Tone and Voice Reader. Setup and usage are in the Pipecat docs. There’s also an overview on our Pipecat page.
What your LLM sees
The package is a perception producer for Pipecat’s Real-Time Context Standard (RCS) and expressive conversation. With it in the pipeline, most user messages reach the LLM with a reading in front of them, like this:
([voice_emotion angry=0.71, neutral=0.20, sad=0.06, happy=0.03],[positivity negative=0.72, neutral=0.20, positive=0.08],[strength strong=0.64, neutral=0.30, weak=0.06], …) I already told you, I was charged twice.
Every label comes with a probability, sorted from most to least likely, so your prompt or your rules decide how much weight to give each reading.
| Signal | What it reads |
|---|---|
voice_emotion | The emotion in the voice: happy, angry, sad or neutral |
strength | How much energy is in the voice: strong, weak or neutral |
positivity | How positive the tone of voice is: positive, negative or neutral |
speaking_rate | Fast, slow or normal, compared to speakers in general |
hesitation | Whether the user sounds hesitant |
engagement | Whether the user sounds engaged, withdrawn or neutral |
intensity | How intense the emotion is, from 0 to 1 |
Each reading reflects how the user sounded in that turn, so your agent can respond to the person, not just the words. Calm words said in a tense voice can now get a different reply from the same words said in a relaxed one.
Easy to add, easy to tune
uv add pipecat-behavioralsignals
Or pip install pipecat-behavioralsignals.
Create an account and API key in the Behavioral Signals portal. Then put BehavioralSignalsPerceptionService right after your STT service and turn on expressive conversation:
from pipecat_behavioralsignals import BehavioralSignalsPerceptionService
perception = BehavioralSignalsPerceptionService()
pipeline = Pipeline([
transport.input(),
stt,
perception,
user_aggregator,
llm,
tts,
transport.output(),
assistant_aggregator,
])
worker = PipelineWorker(pipeline, params=PipelineParams(enable_expressive_conversation=True))
That’s the whole integration. You keep your STT, LLM and TTS. The package never speaks; it adds readings to the user’s transcriptions and passes every other frame through unchanged. It also works alongside Pipecat’s own readers, such as PipecatToneReader, which reads the words: it adds its reading to the one a transcription already carries rather than replacing it.
Version 0.2.0, released 12 November 2026, supports Pipecat 1.16 and Python 3.12 or later. It’s BSD-2-Clause licensed, and the code, changelog and issues are on GitHub. The package is on PyPI.
See it work: this 48-second demo runs two recorded calls through Deepgram STT and the package. One is an angry roadside-assistance call and the other is an excited call with good news. Each user turn is shown exactly as the LLM receives it. Readings are from a live run of 0.2.0, shown unedited.
What you can do with it
- De-escalate earlier. When
voice_emotionleans angry andpositivityturns negative, shorten replies, apologize, or offer a human. - Pace the conversation. When
speaking_rateis fast, get to the answer sooner. Whenhesitationshows up before a “yes”, confirm before you act. - Catch drop-off. When
engagementturns withdrawn over several turns, check in or change approach.
More from the same API
The package draws on the Behavioral Signals API, which also offers:
- Stress and confidence signals, built with Focus, added to your API output on request.
- ASR and speaker diarization in batch processing, at no extra charge.
- Language-agnostic paralinguistic models. They read how something is said, not the words, so they aren’t tied to one language. They’re strongest in the 11 languages they’re trained on: English, Arabic, Russian, Chinese, French, German, Spanish, Portuguese, Japanese, Hindi and Tamil.
Get started
New accounts start with 30 free minutes, with no credit card needed. Get your key in the portal, install the package, and read the Behavioral Signals docs or visit the Pipecat page. Questions and ideas are welcome on GitHub issues or on our Discord.
Behavioral Signals outputs are probabilistic estimates of signals expressed in speech, not determinations of what a person feels, intends or believes.
About Behavioral Signals Behavioral Signals builds AI that measures emotion and behavior in speech. Its Behavioral API returns emotion and behavioral signals per utterance, in batch and streaming, and its Deepfakes API detects synthetic voices. The same models run as a cloud API and at the edge. The company spun out of USC’s Signal Analysis and Interpretation Laboratory (SAIL) in 2016. The missing link: bringing EQ to AI IQ. behavioralsignals.com
