Prospective and Deep Learning Based Retrospective Motion Correction for Magnetic Resonance Imaging

What's the problem?

Motion!

The Enemy of Photographers and MRI Users!

• Please ignore the unfortunate subject in the photos!

• Whether you're taking a selfie with your latest top smartphone (top row) or an old Nokia 3310, motion can always degrade image quality, regardless of the resolution you can achieve.

Magnetic Resonance Imaging

• A non-invasive imaging technique
• Uses strong magnetic fields and radio waves
• Produces detailed images of organs and tissues

Image Formation

1. An MRI image is created using strong magnetic fields and radio waves (pulse sequence) to excite hydrogen atoms in the body.
2. The excited hydrogen atoms emit signals that are captured during imaging.
3. These signals are acquired in k-space, where each point represents different frequency and phase information.
4. The k-space data is transformed to produce detailed images of internal structures.

Top image: left magnitude k-space, right reconstructed image.
Bottom image: left magnitude k-space, center phase k-space, right reconstructed image.

K-space and Motion Artefacts

1. Movement of the subject during imaging causes inconsistencies in phase and amplitude.
2. These inconsistencies lead to blurring and ghosting in the images.
3. The nature of the motion artefact depends on the timing of the motion relative to the acquisition.

From left: Reconstructed Image, Magnitude (k-space) and Phase (k-space)

Motion comes in "many forms" ...

• Motion artefacts in MRI are distortions or errors in the images caused by patient movement during the scanning process.

• These artefacts can be caused by:

  • Patient Movement
  • Physiological Motion

• Types of Motion Artefacts:

  • Ghosting Artefacts
  • Blurring Artefacts

Possible solutions: (1) Prospective Motion Correction (PMC)

• Real-Time Monitoring: Uses motion tracking to detect head movements during scans.

• Dynamic Adjustments: Adjusts imaging parameters in real-time to correct detected motion.

• Integration with Sequences: Can be used with various imaging sequences.

[1] Maclaren et al. Magn Reson Med. 2013 Mar 1;69(3):621-36. doi: 10.1002/mrm.24314. Epub 2012 May 8. PMID: 22570274.

Possible solutions: (2) Retrospective Motion Correction (RMC)

How RMC Works:

1. Data Acquisition
2. Motion Detection: post-processing algorithms identify motion in the data.
3. Image Reconstruction: the algorithm adjusts the data to compensate for motion.
4. Final Image: corrected images are reconstructed for clearer anatomical representation.

Goal of this thesis

Investigate and address the challenge of motion correction in MRI on both ends of the motion correction spectrum:

• Quantitative evaluation of PMC for small involuntary motion in ultra-high resolution imaging

• Deep Learning-Based Retrospective Motion Correction (RMC) for gross motion in clinical (1.5-3.0 Tesla) imaging
  

Study: Evaluating PMC in High-Resolution MRI @7T

• Novelty & Differences:
  • NO INTENTIONAL MOTION to mimic a routine research setting,
  • MULTIPLE WEIGHTINGS & RESOLUTIONS (, , , ), 252 scans in total,
  • LARGER COHORT OF SUBJECTS.

Subjective Image Quality Assessment

• Expert Reviewers:

  • Four experts with over five years of MR image-quality assessment expertise.

• Blinded Comparison:

  • Raters performed side-by-side comparisons of images with and without PMC.
  • Image presentations were randomised to avoid bias.

• Scoring System:

  • Image quality rated from 1 (worst) to 10 (best) based on motion artefacts.

Objective Image Quality Assessment

Evaluation Metrics

• Average Edge Strength (AES): Measures the average strength of edges in an image, indicating the clarity and sharpness of structural details. (higher AES -> higher image quality)
• Gradient Entropy (GE): Evaluates the randomness and complexity of the gradient distribution, reflecting the overall texture and contrast of the image.
  (lower GE -> higher image quality)

• High-Resolution MRI:
  • PMC significantly improves image quality in high-resolution scans.
  • Only three scenarios showed statistically significant improvement, but overall subjective and objective measures indicate better quality with PMC.

• Applicability of PMC:
  • PMC is beneficial for high-resolution scans even in the absence of deliberate motion.
  • Effective for healthy, compliant participants.

Image Quality Assessment (IQA)

• Limits for Subjective IQA: Variability Among Observers, Bias, Limited Reproducibility, Time-Consuming, Lack of Objectivity, ...

• Current no-reference IQA metrics are not always robust and reliable for MR images

Study: SSIM Prediction for Detection and Quantification of Motion Artefacts in Brain MR Images

[Structural Similarity Index Measures: SSIM]

• Collection of motion artefact-free MR scans (from 1.5 to 7.0T)
• With/Without Data Augmentation (brightness/contrast adjustment)
• Artificial corruption of MR images
• SSIM calculation between corrupted and motion-free images
• Training ResNet models (ResNet-18 and ResNet-101) to predict SSIM using only corrupted images as input and the calculated SSIM value
• Testing models on both artificially corrupted and clinical data

SSIM Predictions and Model Evaluation

Results:

• Best model: ResNet-18 trained with data augmentation
• Worst model: ResNet-101 trained without data augmentation
• The analysis of clinical data samples revealed that the highest agreement rate between subjective scores and SSIM predictions was 77.7% for ResNet-101 trained with data augmentation.

This work is under review:

IEEE Access, "Automated SSIM Regression for Detection and Quantification of Motion Artefacts in Brain MR Images"
Preprint: arXiv:2206.06725

What does it happen when the extra hardware required by PMC is not available?

(Like for example in a clinical setting.)

Solution: We can use Deep Learning based Retrospective Motion Correction!

Study: Retrospective Motion Correction of MR Images using Prior-Assisted Deep Learning

• Collection of 100 motion artefact-free MR scans from IXI public dataset
• Artificial corruption of MR images
• Image Priors (Similar Slices):
  • 10 similar slices (same position, same contrast) from different subjects.
  • using T1 and PD-weighted images from the same subject.
  • motion correction only for T2-weighted images.
• Network Architectures
  • Baselines: ReconResNet* and U-Net**.
  • Prior Supply Techniques:
    • Multi-Channel Network: Concatenated motion-corrupted image with priors.
    • Dual-Branch Network: Main branch for corrupted image, auxiliary branch for priors.

*Chatterjee, Soumick, et al. "Reconresnet: Regularised residual learning for mr image reconstruction of undersampled cartesian and radial data." Computers in biology and medicine 143 (2022): 105321.
**Ronneberger, Olaf, et al. "U-net: Convolutional networks for biomedical image segmentation." MICCAI 2015

Some result

Conclusion

• Key Findings:
  • Additional contrast images from the same subject are more beneficial than similar slices from different subjects.
  • Multi-channel and dual-branch approaches showed improvements, especially for ReconResNet.

Medical Imaging Workshop of 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada

Study: Generalised Retrospective Motion Correction (RMC) using Deep Learning and Contrast Augmentation

• Collection of 600 motion artefact-free MR scans from IXI public dataset, in-house scans @3.0 and 7.0T
• Data Augmentation (brightness/contrast adjustment)
• Artificial corruption of MR images
• No image priors required
• Training of ReconResNet model*

*Chatterjee, Soumick, et al. "Reconresnet: Regularised residual learning for mr image reconstruction of undersampled cartesian and radial data." Computers in biology and medicine 143 (2022): 105321.

Results

• Average SSIM improvement from 0.7 to 0.9.
• Consistent performance across different experimental runs.
• Qualitative results show significant enhancement in image quality.

Conclusion

• Proposed method demonstrates broad generalisation across various MRI contrasts and artefacts.
• Notable improvements in image quality and robustness.

Manuscript under preparation

Some test on clinical data!

Conclusions

• Demonstrated the effectiveness of PMC
• Presented a novel method for motion artefacts evaluation
• Novel Deep Learning methods for RMC
  • Using image priors
  • Using generalised correction models

Future work

• Amalgamation of PMC and RMC with DL
  • Motion model approach
  • PMC, then DL-based RMC

A big thank you to:

Prof. Oliver Speck, Kathrin Schulze, Frank Godenschweger, Daniel Stucht, Hendrik Mattern, Astrid Wollrab, Yi-Hang Tung, Mahsa Fatahi, Peter Schulze, Sebastian Hupfeld, Renat Yakupov, Oleg Posnansky, Urte Kägebein, Yan Arnold, Nicolas Huch, Michel Pohl, Uten Yarach, Weiqiang Dou, Shan Yang, Myung-Ho In, Dominik Kolmann, Tobias Leutritz, Denis Kokorin, Tino-Johannes Lüttge, Falk Lüsebrink-Rindsland
Prof. Steffen Oeltze-Jafra, Yannic Waerzeggers, Max Dünnwald, Juliane Müller
Special thanks to Soumick Chatterjee and Max Dünnwald

Funding information

Initial Training Network, funded by the FP7 Marie Curie Actions of the European Commission (FP7- PEOPLE- 2012- ITN- 316716) and the National Institutes of Health (1R01-DA021146), the DFG (DFG- MA 9235/1-1), and the federal state of Saxony-Anhalt (“I 88”)