What Makes Imagenet Good For Transfer Learning

Our consulting visitor, Large Vision, has a long history of solving challenging computer vision and AI problems in diverse fields ranging from certificate analysis, security, manufacturing, real estate, dazzler and mode, automotive, and medical diagnostics, to name a few.

The spectacular growth of AI too means that the knowledge nosotros acquired merely a yr back is now outdated. And so, we continuously acquire, and before embarking on a new trouble, we exercise an extensive survey of land of the fine art in the industry.

We recently started working on a new medical diagnostic problem that utilizes X-ray photographs. This blog post distills the cognition we refreshed about transfer Learning practical to medical data.

Earlier we dive into the details, permit's quickly review the steps we use for training a new Deep learning model.

Steps for Training a Deep Learning Model
Should you use ImageNet architectures for solving medical problems?
Which ImageNet architectures are popular in the medical domain?
Should you use ImageNet pre-trained weights for medical data?
ImageNet pre-trained weights accept produced Human-Level accuracy.
ImageNet pretrained weights produce better results in chest Ten-Ray data.
How to solve a medical image classification problem: A Prescription
Feedback
Acknowledgment

1. Steps for Training a Deep Learning Model

Using Deep Learning, AI has fabricated massive progress in solving computer vision bug similar paradigm nomenclature, object detection, image partitioning, pose estimation, etc.

When nosotros meet a new problem, the steps for obtaining a reasonably proficient model are well-established.

Gear up data: Annotate Data, carve up information technology into grooming, validation, and test sets.
Choose a standard model trained on ImageNet large enough to overfit the training set.
Fine-Tuning / Transfer Learning: Modify the network to fit your trouble, and use transfer learning/fine-tuning to train the final few layers. Typically yous start with ImageNet pre-trained weights.
Bank check dataset: Analyze errors. Check if the information is noisy or inconsistent. If needed, get dorsum to step 1 and iterate a few times.
Hyper-parameter optimization / Model architecture: To squeeze out more accurateness, perform hyper-parameter optimization or even play with many different architectures.

This post volition share our thoughts on steps two and 3 for medical data.

2. Should you use ImageNet architectures for solving medical issues?

Absolutely.

Creating a new model architecture is an expensive and time-consuming black art that we should get out to researchers in academia and industrial research labs.

Should you decide to become down that path, the gains in accuracy y'all see volition be minuscule (if any). That is precisely why nearly nobody starts with a new model compages.

You tin detect many examples in this paper. Subsequently in this post, nosotros will share several more than instances where human-level performance was achieved using standard ImageNet architectures.

At that place are always exceptions to this rule. For example, CoroDet: A deep learning based classification for COVID-19 detection using chest Ten-ray images uses a custom model.

three. Which ImageNet architectures are pop in the medical domain?

The post-obit paper presents a big-calibration review of transfer learning in medical image analysis.

A scoping review of transfer learning research on medical paradigm analysis using ImageNet

The newspaper as well mentions the widespread success of transfer learning using ImageNet pretrained weights for medical paradigm analysis.

The table below shows popular model families used for various anatomical regions.

Anatomical Region	Model Architecture
Breast	Inception
Eyes	VGGNet
Skin	VGGNet
Tooth	VGGNet
Brain	AlexNet
Lungs	DenseNet

The table below shows popular model families used for various imaging modalities.

Imaging Modality	Model Architecture
Ultrasound	Inception
Endoscopy	Inception
Skeletal system 10-rays	Inception
Fundus	VGGNet
Optical Coherence Tomography (Oct)	VGGNet
Brain MRI	AlexNet
Breast Ten-Rays	AlexNet

4. Should you use ImageNet pre-trained weights for medical data?

Anyone who has worked with medical data feels uneasy nearly using ImageNet pre-trained weights as a starting point because of two reasons –

Medical data looks unlike than ImageNet data. For example, in the case of an x-ray epitome, each pixel represents the density of the material. The x-ray prototype is, therefore, grayscale. On the other paw, ImageNet consists of natural colour images where each pixel measures the reflectance of some surface in the existent globe.
Medical datasets are modest: Medical datasets are usually small – sometimes a few hundred images or a few thousand if y'all are lucky. Nosotros accept to freeze nearly network layers to preclude overfitting and train just the last few layers. In other words, nosotros are relying on pre-trained weights very heavily even when we know medical data does non wait like images in ImageNet.

So, on the surface, it looks like using ImageNet pre-trained weights for medical information may not be a good idea.

5. ImageNet pre-trained weights have produced Human-Level accuracy.

It is time to be surprised!

Before nosotros ditch the idea of using ImageNet pre-trained weights, it makes sense to dig deeper to see what practitioners are doing in the field.

Let'due south look at a few examples.

CheXNet achieved radiologist-level pneumonia detection on breast x-rays in 2017. They used a network based on DenseNet and used ImageNet pre-trained weights.
Dermatologist-level classification of skin cancer was achieved by fine-tuning Inception V3 with ImageNet pre-trained weights.
Expert level detection of diabetic retinopathy in retinal fundus photographs was achieved by fine-tuning Inception V3 on ImageNet pre-trained weights.
Homo expert level diagnosis of choroidal neovascularization, diabetic macular edema, and drusen using Optical Coherence Tomography (OCT) images was achieved past fine-tuning Inception V3 with ImageNet pre-trained weights.
CoroDet: A deep learning based classification for COVID-xix detection using breast 10-ray images pretrains a custom model on ImageNet.

Some of these papers are collaborations between acme-notch medical teams and leaders in Deep Learning and AI. For example, the CheXNet paper has experts from Harvard Medical Schoolhouse and Dr. Andrew Ng – a leading expert in AI. Similarly, some authors of the skin cancer paper are from the Departments of Dermatology and Pathology, and others are from Electrical Engineering and Computer science.

These experts in the field make conscientious choices, and so we tin can safely conclude that using ImageNet pretrained models is not a bad idea at all. Nosotros will meet concrete evidence to support this claim in the following department.

In the futurity, we may have an extensive medical dataset rivaling ImageNet in size, and when that happens, we should probably switch to using weight trained on that large dataset.

6. ImageNet pretrained weights produce improve results in chest X-Ray data.

The post-obit paper presents the well-nigh comprehensive analysis of transfer learning using popular ImageNet architectures and ImageNet pretrained weights on breast X-ray dataset –

CheXtransfer: Performance and Parameter Efficiency of ImageNet Models for Breast Ten-Ray Interpretation

They studied 16 different architecture – DenseNet (121, 169, 201), ResNet (18, 34, 50, 101), Inception (V3, V4), MNASNet, EfficientNet (B0, B1, B2, B3), and MobileNet (V2, V3). They used a big chest x-ray dataset chosen CheXPert for their analysis.

Here is a summary of the conclusions in the paper.

6.one ImageNet operation does non correlate with CheXpert performance.

Architectures that perform better on ImageNet do not necessarily perform better on CheXNet regardless of whether ImageNet pretrained weights were used or not.

Specifically, "newer architectures generated through search (EfficientNet, MobileNet, MNASNet) underperform older architectures (DenseNet, ResNet) on CheXpert."

In other words, the newer architectures may be overfitting to ImageNet, which probably explains the popularity of older architectures in the medical domain.

I wonder if how these new architectures perform on other transfer learning tasks.

half dozen.2 Option of Model family > Model Size

The choice of a model family unit (say DenseNet vs. MobileNet) has a more significant impact on performance than the model'due south size within the same family (due east.g., DenseNet 121 vs. DenseNet169).

6.3 Employ ImageNet Pretraining

ImageNet pretraining yields a statistically significant heave in operation across architectures. Smaller architectures benefit a lot more than than larger architectures.

It is important to notation that a previous paper titled Transfusion: Understanding Transfer Learning for Medical Imaging had concluded that pretraining using ImageNet weights produces negligible performance improvement. They had studied ResNet50 and InceptionV3.

Fortunately, there is no disharmonize betwixt the two studies. CheXTransfer newspaper finds that "pretraining does non boost operation for ResNet50, InceptionV3, InceptionV4, and MNASNet only does boost functioning for the remaining 12 architectures."

vi.4 Truncated Architectures for All-time of Both Worlds

The ChexTransfer paper and the Transfusion paper mentioned above conclude that many ImageNet architectures are needlessly big.

An easy manner to reduce the model's size while preserving the benefits of pretrained weights is to truncate the terminal blocks of pretrained models. This way, you lot tin reduce the model'due south size by 3.5x without affecting the performance.

Truncated architectures are DeepCakes, as in you can accept your cake and eat it too!

7. How to solve a medical image classification problem: A Prescription

We have covered a lot in this post. So, let me summarize how to solve a medical image classification problem in like shooting fish in a barrel prescriptive steps.

Data is the Male monarch: It would be best if yous planned on spending more than 70% of the time gathering data, getting it annotated from multiple experts, and fixing noise and inconsistencies in the data. We will encompass these aspects in a future post, merely this video should get you started on data-centric AI problem-solving.
Utilise a standard ImageNet architecture: Check the most pop architectures in your domain. For 10-Rays, DenseNet121 would be a good choice. We take besides had a good experience with Inception V3. Older architectures like AlexNet and VGG are all the same prevalent in some domains. Earlier using these ancient architectures, delight check if people go good results with slightly newer architectures similar DenseNet, Inception, etc. Similarly, I'd stay away from EfficientNet, MobileNet, and MNASNet.
Fine-Tuning / Transfer Learning using ImageNet pre-trained weights: Always use ImageNet pretrained weights to initialize the model. Depending on the corporeality of data, fine-tune the model. Ofttimes this is achieved by freezing all but the last one or ii layers of the network. If the dataset is small (<yard samples), you should probably freeze all but the final layer. Use standard hyper-parameters.
Mistake assay: Medical datasets are oftentimes small. To empathise noise and ambiguity in your dataset, it is crucial to manually cheque every prototype where the model is making a mistake. You may detect that two experts looking at the same picture differ in their diagnoses. You have to go back and make the information consistent in such cases. Remember, spend most of your time looking at the information.
Optimize model: Once you lot have obtained a skillful baseline model using a standard ImageNet architecture, y'all may want to experiment with hyperparameter optimization, truncating the model to reduce its size, etc. Remember, this step will give y'all just half to one percentage betoken increment in performance. Analyzing and fixing your data may provide you with an gild of magnitude college return on time investment.
Tell your friends near this post: This pace is optional, but it ensures a steady supply of high-quality articles!

8. Feedback

The literature on medical image assay is vast, and we have simply touched the surface. If I take missed an of import point, please feel free to point it out in the comments section. If possible provide a reference paper I can look at. Our goal is to present the best information to the readers.

9. Acquittance

We thank Pranav Rajpurkar for valuable pointers.
Characteristic Visual Credits : Public dataset from Eye for Artificial Intelligence in Medicine & Imaging, Stanford University

If you liked this article, please subscribe to our newsletter. You will besides receive a complimentary Computer Vision Resource guide. In our newsletter, we share Computer Vision, Machine Learning and AI tutorials written in Python and C++ using OpenCV, Dlib, Keras, Tensorflow, CoreML, and Caffe.