II.4.1 Measuring Emotions from Body Signals

In our work we have recognized emotions based on body signals collected with sociometric badges and smartwatches. We started with custom-built sociometric badges developed by the team of Sandy Pentland at the MIT Media Lab in 2006. The badges were then further developed by a startup founded by a group of graduate students of Sandy. The badges were worn around the neck, and captured body movement through an accelerometer, physical proximity through triangulating the wearer’s position with a Bluetooth sensor, and tracked who was facing whom with an infrared sensor, and who was speaking with a microphone. 

Figure 38. Sociometric badges used in some of our projects

We used these badges (see Figure 38) to measure personality characteristics, and to study interaction in hospitals among nurses in a PACU (post anesthesia care unit) and surgeons, teamwork among employees in a bank, creative project work in a graduate student seminar, flow among Jazz musicians, and among teams of software developers, to name a few use cases. We found that the way how nurses moved around in the PACU predicted the speed of recovery of their patients, it also predicted the quality of the output of the bank employees, and the grades of the graduate students. For instance, surprisingly, the more time nurses spent close to a recovering patient, the faster was the recovery. This is quite unexpected, as being close to a patient could just be an indicator of the surgery patient’s need for special care. However, it seems patients get well faster simply by having more direct interpersonal interaction with the PACU nurses.

More recently, when smartwatches became widely available, we built the happimeter (see section III.2.3), a body signal collecting apps where the data was collected from the sensors of the smartwatch, and then processed on a server in the cloud using predictive machine learning models, with the results being visualized on the smartphone. The system constructed a generic model pretrained with hundreds of users that made predictions about the emotions of the wearer of the smartwatch based on accelerometer, heartrate, light sensor, temperature, time of the day, speech energy, and GPS location (Figure 39). The emotion prediction can be corrected by the wearer of the smartwatch, if the prediction is wrong, the system learns from its mistakes to create a personalized emotion tracking system. This is an extension of experience-based sampling, pioneered by Mihaly Csikszentmihalyi, who had participants in his study wear a device that would beep at random times, whereupon they had to write down their emotion. Our system extends his concept, as the smartwatch can not only remind the participant at random times, but even present an automatic prediction, and allow the candidate to correct it right on the smartwatch if it is wrong.

On a side note, when measuring happiness, we were quite surprised to find that about fifty percent of the happiness of a person is based on the current weather. We integrated this into our model by including the current weather forecast based on the GPS coordinate of the smartwatch.

Figure 39. Sensor data input for smartwatch emotion tracking

We used the smartwatch emotion tracking system to predict happiness and stress, training the machine learning system with hundreds of students wearing the watches in different courses, and entering their happiness repeatedly. The system was then used among others to measure emotions of actors and audience in a theater play, to measure the emotions of jazz musicians during rehearsals, to track the performance of surgeons in training, to predict the sales success of bank employees when interacting with customers, to analyze the satisfaction of visitors in a museum, to measure mood and performance of logistics workers in a warehouse, and to measure happiness of employees in the innovation center of a bank.

When teaching a happimeter workshop with twenty to forty participants, I found that just making the participants jumping and dancing around at the beginning of the workshop increased their happiness. 

Figure 40 illustrates evolution of happiness at the executive committee meeting of an IT company, where the twelve participants were wearing the smartwatch during the meeting. We can clearly see that lunch was not good, after lunch in the first presentation by an external speaker happiness was rising, in the second presentation in the afternoon right after 14:00 happiness went down. 

 

Figure 40. Evolution of Happiness at an Executive Meeting with twelve participants

In another project, in a two-day creativity workshop with fifteen participants, we found that going running in the morning increased creativity later in the day. At another even, where small teams of participants had to develop business plans and then present them in the plenary, which voted on the best business plans, we found that the single most predictive predictor of a highly rated presentation was participative presentation. In other words, if the entire team went on stage and presented its result together, the collective energy of the presenters assembled on the stage together, got the audience to rate the quality of the presentation significantly higher.