Converting Medical Scans to 3D Models and Augmented Reality for Medical Education

Dr. Leanne Lin and her colleagues at the University of Michigan have developed a novel method to visualize and interact with 3D images using scans from common medical imaging systems. The team’s approach also includes augmented reality, where a 3D model of a specific feature can be superimposed over a live video feed of a patient’s body, allowing trainees to contextualize the imaged features.

Unlike previous methods, this new approach uses open-source software that is readily available. Viewing the 3D models does not require specialized software, and can even be performed using a smartphone.

So, how does the system work? The first step is to export the individual medical scans to a computer, where they can be incorporated into 3D images. The files are in a standard format called Digital Imaging and Communications in Medicine – or ‘DICOM’ – and can come from a variety of imaging modalities, including MRI and CT.

Each image represents a cross-sectional ‘slice’ of the body that encompasses a certain amount of tissue. For the best quality of 3D reconstruction, the slice thickness should be 1 millimeter or less. The images are then segmented, meaning that many features are mapped using software. Features of interest, such as individual blood vessels or bones, can be colored to make them stand out, and the images are then stitched together to form a 3D model.

The resulting models can be viewed on virtually any computer or mobile device, with no need for specialized software. Augmented reality overlays of anatomical features can even be viewed using a smartphone’s ‘selfie’ camera.

The low expense and simplicity of the approach will make it accessible to institutions with low resources, enhancing medical training for all. This technology can even be used as a visual aid for patient education during the consultation and consenting process, allowing patients to feel more ownership over their disease processes.