Clinical Artificial Intelligence

Our group has long-standing expertise in computer vision, specifically in digital histopathology and radiology image analysis. We employ weakly supervised prediction methods to directly extract clinically actionable insights from raw image data, bypassing manual intermediary steps. Utilizing self-supervised learning on extensive datasets, we develop foundation models to identify subtle patterns in images, which are subsequently fine-tuned on clinically relevant downstream tasks. Our approach, predominantly using vision transformers, has successfully predicted genetic alterations, responses to immunotherapy, and patient prognosis directly from routine histopathology slides or from radiology imaging data.

Our team also integrates large language models to process unstructured clinical text data. By embedding these models into the oncology research workflow, we extract structured data from radiology, pathology, and clinical reports. This approach not only unlocks the value of unstructured data but also minimizes inaccuracies, providing a reliable tool for interpreting clinical text records or oncology knowledge stored in clinical guidelines.

Looking ahead, we are focusing on multimodal data analysis, combining various data types like immunohistochemistry images, genomic data, and clinical text. This emerging research area aims to develop holistic clinical decision support tools that transcend single data types and clinical scenarios, enhancing real-world applicability in diverse medical settings.

Jiang X, Hoffmeister M, Brenner H, Muti HS, Yuan T, Foersch S, West NP, Brobeil A, Jonnagaddala J, Hawkins N, Ward RL, Brinker TJ, Saldanha OL, Ke J, Müller W, Grabsch HI, Quirke P, Truhn D, Kather JN. End-to-end prognostication in colorectal cancer by deep learning: a retrospective, multicentre study. Lancet Digit Health, 2024. doi: 10.1016/S2589-7500(23)00208-X.

Wagner SJ, Reisenbüchler D, West NP, Niehues JM, Zhu J, Foersch S, Veldhuizen GP, Quirke P, Grabsch HI, van den Brandt PA, Hutchins GGA, Richman SD, Yuan T, Langer R, Jenniskens JCA, Offermans K, Mueller W, Gray R, Gruber SB, Greenson JK, Rennert G, Bonner JD, Schmolze D, Jonnagaddala J, Hawkins NJ, Ward RL, Morton D, Seymour M, Magill L, Nowak M, Hay J, Koelzer VH, Church DN; TransSCOT consortium; Matek C, Geppert C, Peng C, Zhi C, Ouyang X, James JA, Loughrey MB, Salto-Tellez M, Brenner H, Hoffmeister M, Truhn D, Schnabel JA, Boxberg M, Peng T, Kather JN. Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study. Cancer Cell, 2023 doi: 10.1016/j.ccell.2023.08.002.

Truhn D, Reis-Filho JS, Kather JN. Large language models should be used as scientific reasoning engines, not knowledge databases. Nature Medicine, 2023. doi: 10.1038/s41591-023-02594-z.

Jiang X, Zhao H, Saldanha OL, Nebelung S, Kuhl C, Amygdalos I, Lang SA, Wu X, Meng X, Truhn D, Kather JN, Ke J. An MRI Deep Learning Model Predicts Outcome in Rectal Cancer. Radiology, 2023. doi: 10.1148/radiol.222223.

Saldanha, O., Quirke, P., West, N., James, J., Loughrey, M., Grabsch, H., Salto-Tellez, M., Alwers, E., Cifci, D., Laleh, N., Seibel, T., Gray, R., Hutchins, G., Brenner, H., Yuan, T., Brinker, T., Chang-Claude, J., Khader, F., Schuppert, A., Luedde, T., Foersch, S., Muti, H., Trautwein, C., Hoffmeister, M., Truhn, D. and Kather, JN. Swarm learning for decentralized artificial intelligence in cancer histopathology. Nature Medicine, 2022 doi: 10.1038/s41591-022-01768-5

Shmatko A, Ghaffari Laleh N, Gerstung M, Kather JN. Artificial intelligence in histopathology: enhancing cancer research and clinical oncology. Nature Cancer, 2022 doi: 10.1038/s43018-022-00436-4.

Muti HS, Heij LR, Keller G, Kohlruss M, Langer R, Dislich B, Cheong JH, Kim YW, Kim H, Kook MC, Cunningham D, Allum WH, Langley RE, Nankivell MG, Quirke P, Hayden JD, West NP, Irvine AJ, Yoshikawa T, Oshima T, Huss R, Grosser B, Roviello F, d’Ignazio A, Quaas A, Alakus H, Tan X, Pearson AT, Luedde T, Ebert MP, Jäger D, Trautwein C, Gaisa NT, Grabsch HI, Kather JN. Development and validation of deep learning classifiers to detect Epstein-Barr virus and microsatellite instability status in gastric cancer: a retrospective multicentre cohort study. The Lancet Digital Health, 2021, doi: 10.1016/S2589-7500(21)00133-3.

Kather JN, Heij LR , Grabsch HI , Loeffler C , Echle A , Muti HS, Krause J, Niehues JM, Sommer KA, Bankhead P, Kooreman LFS, Schulte J, Cipriani NA , Bülow RD, Boor P, Ortiz Bruechle N, Hanby AM, Speirs V, Kochanny, Patnaik A, Srisuwananukorn A, Brenner H, Hoffmeister M, van den Brandt PA, Jaeger D, Trautwein C, Pearson AT , Luedde T. Pan-cancer image-based detection of clinically actionable genetic alterations. Nature Cancer, 2020, doi: 10.1038/s43018-020-0087-6

Kather JN, Pearson AT, Halama N, Jaeger D, Krause J, Loosen SH, Marx A, Boor P, Tacke F, Neumann UP, Grabsch HI, Yoshikawa T, Brenner H, Chang-Claude J, Hoffmeister M, Trautwein C, Luedde T. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nature Medicine, 2019, doi: 10.1038/s41591-019-0462-y