This website uses cookies primarily for visitor analytics. Certain pages will ask you to fill in contact details to receive additional information. On these pages you have the option of having the site log your details for future visits. Indicating you want the site to remember your details will place a cookie on your device. To view our full cookie policy, please click here. You can also view it at any time by going to our Contact Us page.

First AI-powered smartphone app detects depression from facial cues

29 February 2024

Researchers launched MoodCapture, a smartphone app that uses artificial intelligence and facial-image processing technology to detect early signs of depression from subtle facial cues, before the user even knows what's wrong.

Image: Shutterstock
Image: Shutterstock

Called MoodCapture, the app uses a phone's front camera to capture a person's facial expressions and surroundings during regular use, then evaluates the images for clinical cues associated with depression.

In a study of 177 people diagnosed with major depressive disorder, the app correctly identified early symptoms of depression with 75 percent accuracy.

These results suggest the technology could be publicly available within the next five years with further development, said the researchers, who are based in Dartmouth's Department of Computer Science and Geisel School of Medicine. 

"This is the first time that natural 'in-the-wild' images have been used to predict depression," said Andrew Campbell, the paper's corresponding author and Dartmouth's Albert Bradley 1915 Third Century Professor of Computer Science. 

"There's been a movement for digital mental-health technology to ultimately come up with a tool that can predict mood in people diagnosed with major depression in a reliable and non-intrusive way."

"People use facial recognition software to unlock their phones hundreds of times a day," said Campbell, whose phone recently showed he had done so more than 800 times in one week.

"MoodCapture uses a similar technology pipeline of facial recognition technology with deep learning and AI hardware, so there is terrific potential to scale up this technology without any additional input or burden on the user," he said. 

"A person just unlocks their phone and MoodCapture knows their depression dynamics and can suggest they seek help."

For the study, the application captured 125,000 images of participants over 90 days. People in the study consented to having their photos taken via their phone's front camera but did not know when it was happening.

The first group of participants was used to program MoodCapture to recognize depression. They were photographed in random bursts using the phone’s front-facing camera as they answered the question, "I have felt down, depressed, or hopeless." The question is from the eight-point Patient Health Questionnaire or PHQ-8, which is used by clinicians to detect and monitor major depression.

The researchers used image-analysis AI on these photos so that MoodCapture's predictive model could learn to correlate self-reports of feeling depressed with specific facial expressions – such as gaze, eye movement, positioning of the head, and muscle rigidity – and environmental features such as dominant colours, lighting, photo locations, and the number of people in the image.

The concept is that every time a user unlocks their phone, MoodCapture analyses a sequence of images in real time. The AI model draws connections between expressions and background details found to be important in predicting the severity of depression, such as eye gaze, changes in facial expression, and a person's surroundings. 

Over time, MoodCapture identifies image features specific to the user. For example, if someone consistently appears with a flat expression in a dimly lit room for an extended period, the AI model might infer that the person is experiencing the onset of depression.

The researchers tested the predictive model by having a separate group of participants answer the same PHQ-8 question while MoodCapture photographed them and analysed their photos for indicators of depression based on the data collected from the first group. It is this second group that the MoodCapture AI correctly determined were depressed or not with 75 percent accuracy. 

"This demonstrates a path toward a powerful tool for evaluating a person's mood in a passive way and using the data as a basis for therapeutic intervention," said Campbell, noting that an accuracy of 90 percent would be the threshold of a viable sensor. 

"My feeling is that technology such as this could be available to the public within five years. We've shown that this is doable."

MoodCapture meets major depression on the irregular timescale on which it occurs, said Nicholas Jacobson, a study co-author and Assistant Professor of biomedical data science and psychiatry at Dartmouth's Center for Technology and Behavioral Health.

"Many of our therapeutic interventions for depression are centred around longer stretches of time, but these folks experience ebbs and flows in their condition. Traditional assessments miss most of what depression is," said Jacobson, who directs the AI and Mental Health: Innovation in Technology Guided Healthcare (AIM HIGH) Laboratory.

"Our goal is to capture the changes in symptoms that people with depression experience in their daily lives," Jacobson said. 

"If we can use this to predict and understand the rapid changes in depression symptoms, we can ultimately head them off and treat them. The more in the moment we can be, the less profound the impact of depression will be." 

Jacobson anticipates that technologies such as MoodCapture could help close the significant gap between when people with depression need intervention and the access they have to mental health resources. 

On average, less than one percent of a person's life is spent with a clinician, such as a psychiatrist, he said. "The goal of these technologies is to provide more real-time support without adding additional pressure on the care system," Jacobson said.

An AI application like MoodCapture would ideally suggest preventive measures such as going outside or checking in with a friend instead of explicitly informing a person they may be entering a state of depression, Jacobson said.

"Telling someone something bad is going on with them has the potential to make things worse," he said. "We think that MoodCapture opens the door to assessment tools that would help detect depression in the moments before it gets worse.
“These applications should be paired with interventions that actively try to disrupt depression before it expands and evolves. A little over a decade ago, this type of work would have been unimaginable."

The study builds on a 2012 study led by Campbell's lab that collected passive and automatic data from the phones of participants at Dartmouth to assess their mental health.

The advancement of smartphone cameras since then allowed the researchers clearly to capture the kind of "passive" photos that would be taken during normal phone usage, Campbell said. 

The new study shows that passive photos are key to successful mobile-based therapeutic tools, Campbell said. They capture mood more accurately and frequently than user-generated photographs – or selfies – and do not deter users by requiring active engagement. 

These neutral photos are very much like seeing someone in the moment when they're not putting on a veneer, which enhanced the performance of our facial-expression predictive model," Campbell said. 

the next steps for MoodCapture include training the AI on a greater diversity of participants, improving its diagnostic ability, and reinforcing privacy measures.

The researchers envision an iteration of MoodCapture for which photos never leave a person's phone, Nepal said. Pictures would instead be processed on a user's device to extract facial expressions associated with depression and convert them into code for the AI model. 

"Even if the data ever does leave the device, there would be no way to put it back together into an image that identifies the user," he said.

Meanwhile, the application's accuracy could be enhanced on the consumer end if the AI is designed to expand its knowledge based on the facial expressions of the specific person using it, Nepal said. 

"You wouldn't need to start from scratch – we know the general model is 75 percent accurate, so a specific person's data could be used to fine-tune the model. Devices within the next few years should easily be able to handle this," Nepal said. 

"We know that facial expressions are indicative of emotional state. Our study is a proof of concept that when it comes to using technology to evaluate mental health, they're one of the most important signals we can get."

Print this page | E-mail this page