Artificial intelligence is revolutionizing medical image analysis

One in two Australians regularly use artificial intelligence (AI), with that number expected to grow. AI is showing up in our lives more prominently than ever, with the arrival of ChatGPT and other chatbots.

Researchers at CSIRO’s Australian e-Health Research Center (AEHRC) are exploring how AI—including the systems that underpin chatbots—can be leveraged for a more altruistic endeavor: to revolutionize health care.

Earlier versions of ChatGPT were built on an AI system called a large language model (LLM) and were entirely text-based. You would “talk” to it by entering text.

The latest version of ChatGPT, for instance, incorporates visual-language models (VLM) which add visual understanding on top of the LLM’s language skills. This allows it to “see,” describe what it “sees” and connect it to language.

AEHRC researchers are now using VLMs to help interpret medical images such as X-rays.

It’s complicated technology, but the aim is straightforward: to support radiologists and reduce the burden on them.

Visual language models are transforming X-ray analysis

Dr. Aaron Nicolson, Research Scientist at AEHRC, is one of the researchers working on the project.

He said any kind of image can be used with VLMs, but his team is focusing on chest X-rays.

Chest X-rays are used for many important reasons, including to diagnose heart and respiratory conditions, screen for lung cancers and to check the positioning of medical devices such as pacemakers.

Typically, trained specialists—radiologists—are required to interpret the complex images and produce a diagnostic report.

But in Australia, radiologists are overburdened.

“There are too few radiologists for the mountain of work that needs to be completed,” Nicolson said.

The problem will likely get worse with the number of patients and chest X-rays taken set to keep increasing, especially as the population ages.

That’s why Nicolson is developing a model that uses a VLM to produce radiology reports from chest X-rays.

“The goal is to create technology that can integrate into radiologists’ workflow and provide assistance,” he said.

Practice makes (almost) perfect

Training the VLM involves lots of data. The more information a model has, the better it can make predictions.

The VLM is given the same information that a radiologist would receive—X-ray images and the patient’s referral, Nicolson explained.

“Then we give the model the matching radiology report written by a radiologist. The model learns to produce a report based on the images and information it is given,” he said.

Like humans, AI models improve by practicing.

“We train the model using hundreds and thousands of X-rays. As the model trains on more data, it can produce more accurate reports,” said Nicolson.

At this stage of his research, Nicolson was looking to improve the accuracy of the reports even further—so he decided to try something new.

“We gave model the patient’s records from the emergency department as well,” he said.

“That means information like the patient’s chief complaint when triaged, their vital signs over the course of the stay, the medications they usually take and the medications administered during the patient’s stay.”

Just as he had hoped, giving the model this extra information improved the accuracy of the radiology reports.

“We are trying to get the technology to a point where it can be considered for prospective trials. This is a big step in that direction,” he said.

Ethical and applicable AI

As well as generating diagnostic reports from chest X-ray images, AEHRC is exploring other applications of VLMs.

Dr. Arvin Zhuang, at post-doc at AEHRC is using VLMs to retrieve information from images of medical documents. Processing the documents as an image rather than text enables the information to be retrieved more efficiently.

It’s an exciting time for Nicolson and Zhuang, but ethical and safety considerations are always at the front of their minds.

“We want to make sure that the model is effective for all populations. To do that, we have to consider and manage issues like demographic biases in the data we train our models on,” Nicolson said.

He also notes that the technology is not designed to replace medical specialists.

“The technology will not be making clinical decisions by itself. There will always be a radiologist in the loop,” Nicolson said.

He and his team are currently conducting a trial of the technology in collaboration with the Princess Alexandra Hospital in Brisbane, assessing how the AI-generated reports compare with those produced by human radiologists.

They are also actively seeking additional clinical sites to participate in further trials, to evaluate the technology’s effectiveness across a broader range of settings.