Digital twins for the win (Johns Hopkins Malone Center for Engineering in Healthcare)
Published in Numerical Simulations, Organisations, Personalized medicine.
When developing technologies to support surgeons in the operating room, it’s vital to rigorously evaluate new AI models, workflow optimizations, and imaging techniques in digital simulations and laboratory studies before testing them in the real world.
Led by John C. Malone Associate Professor of Computer Science Mathias Unberath, researchers from the Advanced Robotics and Computationally AugmenteD Environments (ARCADE) Lab are advancing the science of surgery by using digital twins—virtual replicas of a patient or operating room that are updated continuously with real-time data—to refine new technologies without risking patient safety.
They have constructed digital twins of operating rooms to support immersive space planning applications and digital twins of patients as a means to minimize human exposure to harmful X-rays.
Now, at the 28th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), they’ve demonstrated how OR digital twins can help machine learning models analyze surgical videos and identify workflow bottlenecks.
For an ML model to automatically identify potential cost savings, monitor safety protocol compliance, and evaluate team coordination in an OR, it must use reasoning segmentation, or RS, to interpret high-level user commands without explicit step-by-step guidance.
Traditional RS methods rely on large language model (LLM) fine-tuning, but even state-of-the-art LLMs struggle with complex medical terminology, data variability across hospitals and health care institutions, and identifying semantic and spatial relationships when analyzing OR videos.
“Constant fine-tuning requires frequent updates to maintain compatibility with evolving models, potentially disrupting continuous monitoring and increasing implementation costs,” the researchers write.
To address this issue, Unberath and his team—graduate students Yiqing Shen, Chenjia Li, and Bohan Liu; alumnus Cheng-Yi “Charlie” Li, Engr ’25 (MS); and Tito Porras, a resident in the Department of Neurology and Neurosurgery at the School of Medicine—propose a new approach that preserves the semantic and spatial relationships observed in OR videos while restructuring the traditional RS approach into a “reason-retrieve-synthesize” paradigm, enabling adaptive RS without fine-tuning.