Behavioral Signals? Say what? No Wikipedia lemma (as of May 2018). Oops…

Do not despair. Fortunately there is a lemma for “behavior” and “signal”. So let’s try to put 2 and 2 together, assuming that meaning is compositional, of course!

So what is a behavior?

Wikipedia says: “Behavior… is the range of actions and mannerisms made by individuals, organisms, systems, or artificial entities in conjunction with themselves or their environment, … It is the response of the system or organism to various stimuli or inputs, whether internal or external, conscious or subconscious, overt or covert, and voluntary or involuntary”.

Casting a pretty wide net there …

… and a signal?

Wikipedia enlightens: “… In the physical world, any quantity exhibiting variation in time or variation in space … is potentially a signal that might provide information on the status of a physical system, or convey a message between observers, … The IEEE Transactions on Signal Processing states that the term “signal” includes audio, video, speech, image, communication, geophysical, sonar, radar, medical and musical signals.”

I would augment this somewhat outdated list to also include written documents, emails, texting etc., as well as expand on medical signals to include the wide variety of biosensors available today measuring brain activity, skin conductance, stress etc.

So behavioral signals are aural, visual, textual and sensory sources of information, about actions and mannerisms, of potentially groups of humans and machines (… and living organisms and the environment — but let’s focus only on humans and machines here).

 

 

Simply put: behavioral signals are the observable part of a behavior, i.e., how this behavior is expressed, perceived and captured multimodally by humans (and possibly machines). Behavioral signals are slices of information — including those characterized as thin slices in the literature, typically shorter in duration than the behavior— that need to be combined and composed to jointly describe a behavioral event, action or state. For example, laughter is a behavioral event captured on a series of frames in a video where your face muscles (mainly around your mouth region) contract and expand, and your voice is recorded as a high-energy, semi-periodic audio signal. Politeness is a behavioral state expressed by a warm, mildly energetic voice, a bunch of lexical constructs like “thank you”, “please” and possibly a smile 🙂

Psychologists and cognitive scientists have refrained from defining a general taxonomy of behaviors and behavioral signals, although taxonomies do exist for specific contexts, e.g., a taxonomy of leadership behavior can be found in the Journal of Leadership & Organizational Studies. Lexicographers are a bit more bold and tend to associate behaviors with social and emotional context, as well as break down the subset of behaviors, that can be valenced, into “good” and “bad”. Engineers are a lot bolder. In the social signal processing literature, an intellectual offspring of the book about Honest Signals by Sandy Petland, there has been an effort to classify behaviors into emotion, personality, status, dominance, persuasion, rapport, and regulation related.

The contradiction here is that human behaviors cannot be categorized without understanding the low- and high-level cognitive processes that generate them. These processes are complex, intertwined and not fully understood.

Roughly speaking our behaviors are driven or conditioned by who we are(personality, morality), how we feel (emotion, mood), how we are perceived and think of ourselves socially (status, dominance, rapport), what are our goals and how we plan to achieve them (persuasion), what we believe in, and — to make things even more complex — how self-aware we are about all these things (awareness, regulation, attention). If you don’t believe this last part, read How Emotions Inform Judgement and Regulate Thought, by Gerald L. Clore and Jeffrey R. Huntsinger; in paragraph 6 they explain how the weather affects our mood, our judgements about our own happiness, and how once we are made aware about the weather — and thus implicitly about our mood — the weather mood bias about our state of happiness disappears. Human Beings are strange animals indeed!

So, what is the difference between emotion and behaviors?

A simple way to answer this question is that emotions are “behavioral motivators”, they are part of the “decision-making” processes that produce behaviors. Sometimes a behavior might have a wholly emotional root-cause, although rarely is our brain that simple. So crying can be a behavior that expresses a range of emotional states from sadness to fear to despair, even happiness! But most behaviors don’t have a purely emotional interpretation. For example, a customer service representative might be polite because he is in a good mood or because politeness is being tracked by behavioral analytics software and is associated with a bonus at the end of each month. The relationship between emotions and behaviors has fascinated both, scientists and laymen. In Science Magazine, an interesting article on Emotion, Cognition, and Behavior by R.J. Dolan, discusses “how emotion interacts with and influences other domains of cognition, in particular, attention, memory, and reasoning”.

Now to make things harder, we often add an emotional valence to observed behaviors saying this is a “good” or a “bad” behavior. This emotional rating of behaviors is associated with the perceived or anticipated impact of a behavior to other individuals or society as a whole — rather than the process that generated this behavior. For example, a toxic tweet can be generated by a blogger that feels miserable, be commissioned by a politician’s campaign, or even be generated automatically by a bot. Yet another catch-22!

To summarize by paraphrasing Elkman — the inspiration behind the TV series Lie to me — “emotions [and thoughts] determine the quality of our life” while behaviors and actions determine the outcomes in our life.

In conclusion, I’ll leave you with some further reading on the health applications of behavioral signal processing — the discipline that analyzes quantitatively behavioral signals, written by leaders in the field, some of whom are colleagues in our startup… appropriately named — Behavioral Signals 🙂

Human behavior offers a window into the mind. When we observe someone’s actions, we are constantly inferring his or her mental states — their beliefs, intents, and knowledge — a concept known as the theory of mind.

For example:

• Is a person growing impatient?
• Are they feeling down?
• Are they being truthful?
• Will they buy my product?”

For more details about how Behavioral Signals Processing and behavioral AI can be used to analyze audio recordings among humans or between humans and machines keep on reading or check out my interview with WSJ’s Market Watch, where I explain how emotion recognition and behavioral prediction is used in different use cases.

 

This article also appeared on Medium. The editor is Alexandros Potamianos, CEO and co-founder of Behavioral Signals. Alex is a well-regarded innovator in the field of speech and natural language processing, interactive voice response systems and behavioral informatics. He has over 20 years of leadership experience both in the corporate and entrepreneurial sides of business. His background includes working at AT&T Labs-Research, Bell Labs and Lucent Technologies. His academic achievements have gone hand in hand with his extensive research work, receiving his M.Sc. and Ph.D. degrees in Engineering Sciences from Harvard University and later his MBA from the Stern School of Business, NYU. He’s a Professor at NTUA and currently living in Los Angeles where he enjoys kayaking and sailing.

Follow Alex on Twitter or read his previous article Does Google duplex scare you?