Behavioral Signal Processing Pipeline: How it all works

BSP engineering pipelines can take raw signal data and transform them into actionable insights on behavior and emotion. The engineering components are quite specific to the actual application, but there are general attributes of any such project. There are three primary building blocks of BSP:

• Data collection: all further steps in the BSP pipeline are affected by this stage. Collected data should be as complete and “clean” as possible. Complete data contains all of the relevant information needed to make decisions; for our use cases, all of the audio/text from all of the participants in a call/chat as well as any relevant metadata.
• Signal analysis: descriptors or features that inform or indicate aspects of “who, what, when, how, where, and why” are computed from the data.
• Modeling: mapping from behavioral signal descriptors to behavioral constructs (e.g., anger, happiness, call success).

An example engineering component of an end-to-end pipeline is that of speaker diarization, the process of determining who spoke when, for how long. Following up with speech-to-text can also deliver what they spoke about (this process is illustrated below). This component of the pipeline informs who is doing what, but it still not sufficient for measuring the higher level behaviors such as emotion and attributes of interaction success.

Fig.4 Converting a raw speech signal into a spectrogram (energies in various frequencies, over time), detecting voice, determining who is speaking, and transcribing the speech; then using the how and the what to track emotions, behaviors, and KPIs.