The Challenges of Quantifying Behavior from Signals

Of course, quantifying such nuanced behavior is not straightforward and comes with a unique set of challenges. The expression and experience of human behavior is complex, and each person has idiosyncrasies in their behavior that make them who they are as an individual. For example, males have an average pitch of roughly 100 Hz, while females have a higher average around 200 Hz. A pitch of 150 Hz for a male may indicate excitement, while it would indicate a calm or depressed state for a female! Of course, behavior is actually much more complex than this, encoded in multiple modalities/channels and changing with context (for example, a person yelling would typically indicate excited anger or joy, but at a train station it may simply be a practical means of communicating). Think about how many times it has been difficult to communicate through text, Tweet, or email; sure, emoticons help somewhat, but hearing the person’s voice is much better: the emotion, the pausing, the ability to “feel-out” the intent of the person that you’re interacting with.
Measuring emotion with machine learning is no different; if we look at what a person says in conjunction with how they say it, we can more accurately recognize their emotional state. In terms of usability, the systems must be able to process data at scale, as efficiently as possible. And the systems must work across multiple languages, and thus must detect when a different language is being spoken (known as code-switching); this is not a straightforward computational task but can be robustly solved with well-formulated mathematical models. Only with carefully engineered systems can we robustly, efficiently create valuable KPIs.

These are precisely the challenges that the Behavioral Signal team has spent the past two decades investigating. We understand both the power and the limitations of this technology, and we have the vision for how best to advance it into our increasingly technological future.