Researchers from MIT and the Qatar Computing Research Institute have developed a novel new facility in the current rush of interest towards computer vision – an algorithm that can identify overweight individuals based on their social media photos.

The paper Face-to-BMI: Using Computer Vision to Infer Body Mass Index on Social Media outlines how the team, led by Enes Kocabey, used a Reddit-derived image dataset developed by the Visual BMI Project to teach a computer how to understand facial topology that seems to indicated above-average Body Mass Index (BMI).

visual-bmi-projectThe dataset (consisting of 4206 faces in relatively arbitrary angles to camera) derives from a series of ‘before’ and ‘after’ pictures of people who have undertaken weight-loss regimes, isolating just the facial elements from the resulting photos, in order to establish some idea of baselines beyond which increased BMI might be indicated in the average person.

The researchers acknowledge the risk of perpetuating existing stereotypes when approaching this kind of qualitative assessment model, observing that ‘as African Americans have higher obesity rates in the US population, an automated system might learn a prior probability that increases the likelihood of a person to be labeled as obese simply based on their race.’

Nonetheless the tools developed are anticipated as more useful for studying blocks of populations rather than individuals – not only because of the ethical considerations that the scientists acknowledge, but because of the controversy around using BMI at a very granular level to assess individuals.


Some research projects centred around evaluating facial characteristics have garnered criticism or controversy in recent times, such as one that claims to be able to predict a disposition towards criminality, and another from Microsoft which, though launched in a light-hearted manner, claims to be able to distinguish age from similar photos as the MIT/Qatar study over its Azure cloud framework.

The paper itself addresses some of the very concerns that the research might bring up for users:

‘Together with a person’s gender, age and race, their weight status is a publicly visible signal that can have profound influence on many aspects of their life. Most obviously, it can affect their health as having a larger BMI is linked to an increased risk of both cardio-vascular diseases and diabetes, though not necessarily in a straight-forward manner…

‘However, other aspects of the burden imposed by obesity come in the form of “fat shaming” and other forms of “sizeism”. For example, obesity is related to a lower income1 and part of the reason seems to be due to weight-based discrimination…Even among health professionals “sizeism” is so prevalent that it has become a health hazard as, when faced with overweight patients, care providers stop to look for alternative explanations for a medical condition.’

The team is making the project’s algorithms and scripts available to other institutions.