Vacancies & Student Research Projects

Foundation models, such as BiomedParse, have demonstrated exceptional generalization capabilities across various tasks using multimodal healthcare data. In this project, we want to finetune such foundation models for predicting surgical outcomes with limited available data. We also want to investigate techniques for fusing domain knowledge during the finetuning process, and measure the impact of domain knowledge on model performance and interpretability. (contact)

In potential navigation systems for minimally invasive surgeries, models created by CT and MRT scans are rendered and overlaid transparently to be shown on the endoscopic video stream. This might lead to sorting errors, because occlusions that usually work as visual cues for depth perception are ambiguous due to the transparent rendering. Explore and implement different rendering methods that could help users in the correct interpretation of the augmented reality overlay. (contact)

Accurate 3D reconstruction of surgical environments is crucial for robotic-assisted surgery, providing precise spatial awareness of anatomical structures and surgical instruments. Traditional 3D representations—such as explicit meshes, volumetric grids, and neural implicit models—have their own strengths and weaknesses.
This project aims to develop a hybrid 3D reconstruction approach that combines the strengths of different 3D representations to improve reconstruction accuracy, efficiency, and adaptability in surgical settings.
Related skills: Computer vision, 3D reconstruction, PyTorch.(contact)

Accurate 3D reconstruction of surgical environments is crucial for robotic-assisted surgery, providing real-time spatial awareness of anatomical structures and surgical instruments. However, reconstructing deformable soft tissues poses challenges due to dynamic changes, occlusions, and lighting variations.
This project explores 4D Gaussian Splatting, an advanced technique that models dynamic scenes efficiently by representing them as time-aware Gaussian primitives. By leveraging a foundation model, the goal is to enhance reconstruction quality, improve generalization across different surgical scenarios, and enable real-time rendering.
Related skills: Computer vision, 3D reconstruction, PyTorch. (contact)

SLAM (Simultaneous Localization and Mapping) plays a crucial role in robotic-assisted surgery by enhancing 3D scene understanding, precise navigation, and spatial awareness of surgical instruments. However, traditional SLAM methods struggle in surgical environments due to soft tissue deformation, making accurate localization challenging.
This project aims to tackle this problem by utilizing implicit 3D reconstruction for camera localization in surgical scenes. We seek to improve localization accuracy, even in dynamically changing anatomy, leading to safer and more precise minimally invasive procedures.
Related skills: Computer vision, SLAM, neural implicit representations (e.g., NeRF, Gaussian Splatting), PyTorch. (contact)

In biomechanical simulations, the accuracy of soft tissue deformation highly depends on the quality of the mesh. To ensure high precision, remeshing is required after each simulation step to maintain element quality and improve numerical stability. Traditional remeshing techniques can be computationally expensive and challenging to optimize.
This project aims to explore reinforcement learning (RL) as an adaptive remeshing strategy, where an intelligent agent learns to optimize the mesh dynamically. By integrating RL, we hope to develop an efficient and automated remeshing framework that enhances simulation accuracy while minimizing computational costs.
Related skills: Reinforcement learning, fundamental knowledge about finite element methods (FEM), PyTorch. (contact)

Spatial Transcriptomics (ST) is a cutting-edge technique that maps gene expression across tissue sections while preserving spatial context. By aligning molecular data with cell morphology in Hematoxylin and Eosin (H&E) stained histology images, ST enables a deep understanding of cancer biology at the molecular level. However, ST experiments are costly and time-intensive.
This thesis explores the use of generative models to predict spatial transcriptomics directly from H&E images—eliminating the need for expensive lab procedures. Instead of relying on traditional regression approaches, we will combine powerful tools such as pathology foundation models, graph neural networks, and diffusion models within a GraphVAE framework to tackle this challenge.
What you’ll work on:
•    Multimodal graph-based modeling of histology and transcriptomics data
•    Deep generative learning using Graph Variational Autoencoders (GraphVAE) and diffusion models
•    Leveraging recent advances in computational pathology and spatial omics
Requirements:
•    Solid experience with Python and deep learning frameworks (e.g., PyTorch)
•    Familiarity with graph neural networks is a plus
•    No prior knowledge of biology is required!(contact)

Excess pulling force on the small intestine in surgery could lead to life-threatening bleeding conditions. It is therefore critical that surgeons finely control their movements and the force asserted to the organ by their instruments.

This group has previously procured recordings of silicone small intestine phantom being pulled by surgical graspers. Force estimation has been attempted with ResNet and Video Swin Transformer. We would like to further explore LLM models in fulfilling this computer vision task with simple modifications to their architecture (e.g. changing output layer).Required technical skills: Mastery in Python, very good literature analysis skills and implementation skills. Course requirements:
Must have completed 2 computer science-related course (contact)

Develop and evaluate tracking methods or optical flow techniques to propagate full-scene segmentations from a single video frame onto subsequent frames. 
The project aims to generate additional annotations by leveraging temporal consistency in video data, reducing the need for extensive manual labeling. 
The topic includes challenges such as handling occlusions, dynamic object movements, and ensuring accurate alignment across frames. This work will 
contribute to improving annotation efficiency in large-scale video datasets. (contact)

Visual Question Answering (VQA) in surgical contexts is a challenging task due to the domains inherent data scarcity and the complexity of surgical scenes. While specialized Vision-Language-Models (VLMs) have been proposed, this project focuses on evaluating how effectively general-purpose VLMs can be adapted to the surgical domain. Help us benchmark pre-trained VLMs on surgical VQA tasks. Fine-tune state-of-the-art VLMs on existing datasets and evaluate question-answer biases to check for text-prior contamination. (contact)

In order to take the next steps in automating surgical tasks in laparascopy, robot assistance systems have to gain a deeper understanding of the surgical procedure. In this project we aim to solve the problem of detecting and predicting fine grained surgical actions. Make use of supervised deep learning methods and annotated video/sensor data. Compare different model architectures to investigate the relevance of temporal information for the learning task. (contact)