Overcoming AI’s diversity problem when creating images

An AI-generated image of a Black female doctor. Image by atlascompany on Freepik

Artificial intelligence-based text-to-image generators like DALL-E 2 have grown in popularity — with more than 15 billion AI images created annually. Many of these often free platforms allow users to describe the image they want and generate one based on the text prompt.

A recent study in JAMA Open Network is the latest of several publications that have found that when generating images of physicians, the platforms often depict people who are white and male.

It’s unclear how many freelance journalists or newsrooms are using AI-generated images, but in today’s era of shrinking news staffs and smaller budgets for photography or art departments, it’s an important issue for reporters and editors to note. By adding better descriptors into the search parameters (see tips below), journalists using these programs can generate more diverse, inclusive images of health care providers.

More about the JAMA study

AI generated doctor — *AI-generated image of a physician by freepik*

Sang Won Lee of Harvard Medical School and colleagues generated images of physicians using current versions of five popular text-to-image platforms: DALL-E 2, Imagine AI Art Generator, Jasper Art: AI Art Generator; Midjourney Beta; and Text-to-Image.

They asked each platform to create 50 images using search terms like “face of a doctor in the United States,” or “photo of a physician in the United States.” Then, they compared the distribution of race, ethnicity and gender within each platform and against U.S. physician demographics from the Association of American Medical Colleges.

Overall, the AI-generated images of physicians were more frequently white (82%) and more frequently male (93%) than the actual U.S. physician population, which is 63% white and 62% male. Three platforms produced no images of Latino physicians, two platforms produced no images of Asian physicians and one platform produced no images of female physicians.

The biases depicted in the AI-generated images have “the potential to reinforce stereotypes and undermine DEI (diversity, equity and inclusion) initiatives within health care,” the authors wrote. “Although strides toward a more representative health care workforce are being made as trainees from increasingly diverse backgrounds enter the workforce, this representation remains lacking within generative AI, highlighting a critical need for improvement.”

Future work should focus on enhancing training dataset diversity and creating algorithms capable of generating more representative images, while educating AI developers and users about the importance of diversity and inclusivity in AI output, they added.

Other recent studies had similar findings:

A report in Clinical Ophthalmology found that when searching DALL-E 2 for images of “American ophthalmologist, portrait photo,” the majority of ophthalmologists were portrayed as white (75%) and male (77.5%) among 1,560 images generated. White males also were most often generated in searches for subcategories like retina specialist. Young, inexperienced ophthalmologists were, however, perceived to have greater non-white racial diversity (27.5%) and female representation (28.3%).

A study in JAMA Surgery that asked three AI text-to-image generators for photos of various types of surgeons found that Midjourney and Stable Diffusion, two popular programs, depicted more than 98% of surgeons as white males. DALL-E 2 came closer to reflecting actual demographics of U.S. surgeons, with 15.9% depicted as female and 22.6% as non-white, the Boston Globe reported. However, it portrayed 15.9% of trainees as female compared to the actual share of 35.8%. The study analyzed a total of 2,400 images.

A commentary in the Lancet Global Health described how authors asked Midjourney to show images of Black African doctors providing care to white suffering children. The program was able to show either Black African doctors or white suffering children, but couldn’t seem to merge the two, instead showing Black children as the recipients of care or sometimes wildlife instead of doctors. When they asked for an image of a traditional African healer with a white child, it superimposed African clothing and body art on the child, which could be considered culturally insensitive.

Tips for generating diverse images

Biases in imagery likely result from the data these models are trained on, said Jigyasa Grover, a generative AI lead, in an interview. Image generators “rely on massive datasets scraped from the internet,” she said. “Unfortunately, the content out there tends to overrepresent certain demographics — white males in the case of physicians. This reflects historical and ongoing biases in society and the media.”

To generate more diverse images, she offered these tips:

Be specific in your prompts. Instead of just typing “physician,” try adding descriptors like “female doctor,” “Black male physician,” or “Asian woman in a lab coat.” The more specific you are, the more likely you’ll get the diversity you’re aiming for. In the Clinical Ophthalmology study, when authors queried DALL-E 2 for a specific demographic of ophthalmologist such as “American Black ophthalmologist, portrait photo” or “American male ophthalmologist, portrait photo,” 40 of 40 images did represent the specific race or gender queried.
Use contextual keywords. Sometimes, just adding terms related to diversity can help steer the generator in the right direction. Words like “diverse,” “inclusive,” or “multicultural” can nudge the AI model to consider a broader range of images.
Adjust and reiterate. If the first result isn’t quite right, tweak your prompt and try again.
Give feedback. Many image generators allow you to provide feedback on their results. If you notice a consistent bias, reporting it can help developers fine-tune the model for future use.
Utilize post-processing tools. It may be necessary to take generated images and then use other tools or platforms to further diversify or adjust them. Combining or editing images also can help create more accurate representation.