AI in Medical Imaging

From Theory to Implementation

Image credit: Medium via Emerj AI Research

A lot has been said about the potential of AI in transforming the medical sector, in particular in relation to diagnostic processes; yet, cases of successful implementation are relatively rare. This holds true especially for the field of medical imaging; while machine learning techniques have been implemented in logistical processes, the hype around AI in diagnostics seems to stem more from findings in academia than from practical adoption. Why is it that despite countless papers demonstrating proof-of-concept and high performance on validation sets for different deep learning techniques, adoption by healthcare businesses and institutions is relatively scarce?

This article is made up of two parts: First, I will address the problems that the medical imaging sector is facing and argue why AI presents a viable solution to alleviating these challenges. I will discuss the value of deep learning techniques on all levels of the medical imaging workflow, from image restoration to content-based image retrieval, using MRI as an exemplary case. Secondly, some key obstacles hindering practical implementation of these promising techniques will be analyzed, together with a potential solution that would enable small- to middle-scale businesses and organizations to profit from the transformative power of deep learning systems in an efficient and realizable way.

Recap: Definition of AI, ML, and DL

Before we proceed to a discussion of medical imaging in particular, it is important to establish what we mean by Artificial Intelligence (AI), Machine Learning (ML), and Deep Learning (DL) — three terms that are often used interchangeably, when actually in each case the latter is contained in the former.

AI presents the most controversial out of these three terms, as the commonly used definitions are either too ambiguous, or tautological (e.g. “the ability of a computer to perform intelligent tasks”; “the simulation of human intelligence”; “intelligence demonstrated by machines”). It has even been argued that artificial intelligence does not exist, however, the term generally is used to describe systems that can perform tasks that previously required human intelligence. One subset within the field of AI is Machine Learning. Conceptually, ML techniques allow a computer to internalize concepts found in data to form predictions for new situations. From a technical perspective, a model is tuned on a set of training data to form predictions on new data. This is done through an optimization algorithm from the gradient descent family, which is applied on a cost function to minimize “loss”, i.e. the distance between the model’s current understanding of the world and the reality. Following this stepwise optimization or “training”, the model’s ability to generalize is tested on a validation set. Within Machine Learning, Deep Learning describes a particular class of algorithmic architectures where the model learns relevant representations and features automatically from raw data. While conventional ML techniques require structured data (i.e., in an ordered format like Excel or SQL), DL enables analysis of unstructured data like text or images.

Data in Healthcare

Healthcare providers generate and capture enormous amounts of unstructured data, at a pace that is surpassing the capacities of trained medical staff on the one hand, and going beyond the limits of what conventional methods of analysis can process on the other. Due to the ability of DL techniques to uncover patterns and recognize structures based on autonomously extracted features, the first application that comes to mind is intelligent image analysis for diagnosis, e.g., detection of malignant cancer structures in radiology. I will discuss the value AI brings to medical imaging on multiple levels, from applications very close to the physics of MRI, to content-based image retrieval. DL techniques have the potential of pervading every part of the medical imaging workflow. However, rather than replacing a doctor’s role, DL has the potential of enhancing it through intelligent assistance, allowing medical professionals to focus on more value-adding tasks.

What are the problems in healthcare?

One major trend in the medical sector is increased usage of imaging techniques, leading to large amounts of complex data in the form of, e.g., X-rays, CAT scans, and MRIs. While imaging as a technique in medical practice is increasing, and consequently the workload associated with the analysis of this data, the number of trained radiologists stays more or less constant. The amount of data is surpassing processing powers, with overworked radiologists as a result. A large-scale study from the Mayo Clinic found that within a decade, radiologists have gone from reading three MRI images per minute to 12. With CT and MRI utilization significantly increasing in recent years, more work must be done by roughly the same number of radiologists; the authors of the paper mention the risk of increased errors due to overworked medical professionals. Furthermore, it is important to remember that a radiologist’s profession extends beyond image analysis, and with the workload within one task increasing, other responsibilities become harder to dedicate time to. Within radiology, ongoing challenges include decreased reimbursement for individual radiology reports while the needed quantity is increasing, resulting in burnout becoming more prevalent within the profession. Interestingly, a number of articles describe emerging Deep Learning techniques as another threat to radiologists. I will proceed to argue why it is just the opposite. The problems described above can in fact be alleviated by these new techniques, allowing radiologists to perform more value-adding tasks, i.e., where medical experience and expert knowledge, based on research and informed by judgment, come into play to form a holistic diagnosis.

Why is AI the solution?

AI is valuable in supporting processes that are repetitive but based on complex data in varying conditions. DL algorithms excel at automatically recognizing complex patterns in unstructured data. For medical imaging, deep learning is therefore particularly interesting.

To show how far the value of this technology extends beyond one application, we will start by looking at how DL can be applied to the entire MRI processing chain in more detail by means of three examples.

Accelerating MRI through deep learning image restoration

Penetration of DL techniques into lower levels of MRI includes applications in MR signal processing, denoising and super-resolution, or image synthesis. In 2016, a deep learning approach for accelerating MRI was developed by a group at the Paul C. Lauterbur Research Center for Biomedical Imaging. One issue of MRI applications is the long scanning time, which is linked to elevated motion artifact perturbing image quality, as well as higher medical cost. One approach attempting to accelerate MRI scans is the development of signal processing-based methods. This describes efforts to explore and use prior information on MR images, to reconstruct undersampled k-space measurements (k-space = an array of numbers representing spatial frequencies in the MR image). Previous attempts were only able to use prior information from the image to be reconstructed (a very limited source of information) or from very few reference images, that information would then be manually integrated. Due to the anatomical similarities between individuals, and the large amounts of existing MR images, this research group had the idea of training a neural network on an extensive set of reference images, in order to reconstruct a new image in more detail. The algorithmic architecture of interest is the convolutional neural network, or CNN. One advantage of using CNNs is their strong ability to capture image structures due to their loosely inspired biological structure. The CNN was trained to learn an end-to-end mapping between zero-filled and fully sampled MR images. By training the deep neural network on high-quality MR images, the group developed a model capable of restoring details and fine structures.

content-based image retrieval as diagnostic aid

Example of content-based image retrieval as diagnostic aid (Source: Müller et al, 2005). A new image is presented on the left, and relevant results are shown on the right with their diagnoses and a link to the complete case description.

Deep learning in disease detection and prediction

A higher-level application of deep learning in the MR imaging workflow is centered around diagnosis and prediction through image segmentation or pattern detection. Segmentation is the process of partitioning an image into multiple regions that have certain relevant attributes in common. In line with the common narrative of deep-learning assisted solutions, recent research supports the claim that segmentation is performed more efficiently by computers than it is by humans. Automatizing this task improves disease detection workflows, one example being a recent project led by Google AI Healthcare, on the classification of prostate cancer. This particular type of cancer is often non-aggressive. Accurately differentiating between the malignant and harmless form is critical due to the severe risk on the one hand, and the drastic treatment options (e.g., radiotherapy or prostate removal) on the other. The central biomarker to distinguish one type from the other is the appearance of cancer cells; the classification system for risk stratification is the Gleason grade. Currently, Gleason grading is performed manually by medical professionals. Yet, due to its complexity, it is to an extent subjective and varies significantly between individuals. The paper shows a proof-of-concept for a deep learning system performing image segmentation to predict Gleason grade group with 9% higher accuracy than a board of certified pathologists.

Gleason's patterns used for grading of prostate cancer

Left: Examples of Gleason patterns used for grading of prostate cancers. Lower numbers correspond to more normal, well-differentiated tissue. Right: Comparison of performance accuracy of the Deep Learning system with certified pathologists. (Source: Google AI Healthcare)

Deep learning for content-based image retrieval

While the prospect of using deep learning for computer-aided detection (CAD), or at some point in the future for fully automated diagnosis, seems promising at first, the adoption in clinical practice has met some resistance. Radiologists are tasked to detect certain patterns, however, they are trained to interpret and contextualize the visual data based on their broad medical knowledge. Traditional CAD classifiers represent a “black box” approach, as deep neural nets tend to lack interpretability, and rather deliver a quantitative output, e.g., the likelihood of malignancy. A higher-level approach is represented by content-based image retrieval (CBIR). CBIR, in its essence, is an intelligent reverse image search engine, which can be used to inform the formation of a diagnosis, by presenting past judgments in similar cases of relevance. Reverse image search, in general, is an established technology, however, deep learning comes into play due to the additional layer of complexity that variations in medical imaging add. Due to, e.g., anatomical differences and exposure time, differences, in contrast, may be circumstantial, rather than of clinical significance. For a CBIR system to be useful, it must be able to assess the clinical significance of individual features, which conventional technologies were unable to do in the past.

Content-based image retrieval s diagnostic aid.

Example of content-based image retrieval as diagnostic aid (Source: Müller et al, 2005). A new image is presented on the left, and relevant results are shown on the right with their diagnoses and a link to the complete case description.

What are the hurdles AI implementation faces in medical imaging?

All the examples described above have been proven to work conceptually. The technology is available, in most cases showing high accuracy on validation sets. Nevertheless, when it comes to medical imaging, implementation of DL-applications is scarce. What is holding people back from making use of this valuable toolset?

For a problem as diverse and complex as medical imaging, deep learning requires large data sets in order to reach the required levels of accuracy. In combination with long training durations, this presents a major drawback for any small to medium-scale institution or enterprise. Privacy is an issue that attention has been shifting towards in almost all areas, however, medical data is one of the most sensitive forms of data. A recent study by UC Berkeley showed that a large unidentified dataset could, to a major extent, be re-identified by combining the anonymized data with publicly available information; therefore, data sharing is a complex issue, especially in the medical sector. Explainability presents another major problem in healthcare. When acting upon a decision where a lot is at stake, and the consequences directly impact someone else, responsibility and accountability play a central role. Deep learning models can make predictions with near-infallible accuracy, however, as the model’s internal logic is hard to uncover and interpret, the reasoning as to why this decision is correct often remains elusive. As medical decisions can carry incredible weight, the prospect of full automation is regarded critically by many.

How to overcome these obstacles?

With the emergence of cloud services, the issue of long training durations might gradually become a limitation of the past. To some of these issues, however, e.g., lacking interpretability of deep neural networks, or the large amounts of protected medical data required, no clear solution is in sight, and everyone working in the field feels their limitations. To achieve wider adoption of DL, which to an extent has been restricted to academia or large tech giants like Google, amongst small to medium-scale businesses, a new practical approach is needed. One possible solution is the use of a hybrid approach, which combines human and artificial intelligence to overcome the limitations described above. Instead of aiming for full automation, AI needs to be incorporated into radiology workflows in a way that effectively decreases workload, while being able to access expert knowledge in critical cases. Through an interface, the trained model can query the human expert in uncertain situations; this feedback is then stored and used for re-training purposes to improve performance over time. This concept could very well be applied to the diagnosis workflow in healthcare: a hybrid approach allows for doctors and medical experts to be consulted in ambiguous cases, where model accuracy is deemed to be insufficient to deliver reliable diagnoses. This frees up capacity among medical staff, as the most well-known and common instances are automatically diagnosed through the use of deep-learning software. In effect, medical treatments become faster, more reliable and more cost-efficient, benefiting medical staff and patients alike.

About the Author

Arianna Dorschel

Arianna is currently completing her BSc in Neuroscience at the University of St Andrews, with a focus on Computational Neuroscience. Previously, she has done research at Tuebingen University, St Andrews Perception Lab, and Harvard Medical School. Her primary interest lies in computer vision.


Supercharge your processes

Reach out

Follow us on:

LinkedIn loFacebook LogoMedium LogoMedium Logo
Copyright © 2021 Luminovo GmbH · Made in Munich · Imprint & Privacy Statement