(195i) Development of Multimodal Cancer Diagnosis Method Based on Medical Image Analysis and Raman Spectroscopy

Digital pathology (DP) [1] is playing an ever-increasing role in clinical decision making. Technological advancements have made efficient data recording, processing, and analyzing possible. This enables the transformation of cancer diagnosis such that diagnosis can now be routinely assisted by algorithms in, for instance, computational image analysis [2], not least a wide variety of artificial neural networks (ANNs) developed for computer vision. Meanwhile, effective (and perhaps more intuitive) cancer identification and classification can also be realized based on appropriately selected morphological features of the biopsy samples, e.g. object contour, color, and texture [1]. It is therefore desirable to combine the advantages of both approaches so that a reliable and easy-to-interpret diagnosis can be achieved. Moreover, Raman spectroscopic analysis of specimens is gaining popularity in the biomedical research community [3], as molecular composition and microenvironment information of the cells unavailable in medical images can be highly sensitively captured in the Raman spectroscopy. Thus, Raman spectral profiles and their spatial distribution can be used to assist in characterizing cell/tissue samples. In this work, we construct a multimodal cancer diagnosis model based on the information provided by both the medical image analysis and Raman spectroscopy. Furthermore, we demonstrate the effectiveness and efficiency of the approach by running it on a wide variety of diagnostic samples.

[1] R. Bhargava and A. Madabhushi, Annu. Rev. Biomed. Eng. 18, 387–412 (2016).

[2] D. Shen, G. Wu, and H.-I. Suk, Annu. Rev. Biomed. Eng. 19, 221–248 (2017).

[3] H. J. Butler et al., Nat Protoc. 11, 664-687 (2016).