Project Portugal 2030

Summary

The ExpRT project is dedicated to transforming laparoscopic exploration, and by extension, all laparoscopic interventions that incorporate it as the initial step, by leveraging AI-powered vision-language models (VLMs) to improve decision-making, ensure consistency in exploration, and improve the quality and accuracy of surgical reporting. To achieve this ambitious goal, our interdisciplinary team has outlined three intertwined research paths, all supported by a comprehensive annotated database of laparoscopic videos (Task 1): Aim 1: Standardization of laparoscopic exploration reporting (Task 2): The first aim addresses the challenge of existing variability and inconsistencies in surgical reports, arising from time constraints and the varied expertise across surgeons and surgical specialties. Through a Delphi study involving surgeons from general surgery, urology, gynecology, and pediatric surgery, we intend to create a consensus-driven surgical report template. This template will standardize annotations for laparoscopic data (Task 1) and provide a foundation for further software developments (Tasks 4 & 5). Aim 2: Support tools for enhanced laparoscopic exploration and reporting (Task 3): The second aim focuses on augmenting surgical reporting’s accuracy and efficiency through AI-based support tools that complement the VLM report generator. This includes: Surgical phase recognition (Task 3.1): Automating the identification of the phases of laparoscopic exploration to facilitate video feed truncation and support additional tools. Smart data representation (Task 3.2): Developing an innovative method based on surgical camera-based tracking and mosaicking technology to generate wide-angle panoramic views of the abdominal cavity, aiding in completeness assessment and report generation. Exploration completeness assessment (Task 3.3): Creating a video classifier informed by a Delphi-derived inspection checklist to identify instances of incomplete explorations, offering timely feedback to surgeons. Aim 3: Autonomous reporting tool with built-in explainability (Tasks 4 & 5). The third aim is to develop an autonomous reporting tool that incorporates explainability within a specialized VLM, thereby building trust in AI-driven decision-making and reporting processes. This development entails: Fine-tuning a VLM (Task 4): Adapting a foundational VLM to generate accurate surgical reports using domain-specific surgical video encoders, parameter-efficient fine-tuning, advanced prompt engineering, and text-based augmentation techniques. This process is geared towards creating a model that understands complex laparoscopic videos and translates this understanding into coherent, standardized reports. Integrating explainability (Task 5): Enhancing the VLM to provide explanatory visual feedback alongside textual reports and introducing dual-format reporting options (prose and bullet points). Furthermore, it includes the creation of an intuitive interface for report interaction, simplifying the review process and strengthening the surgeons’ confidence in the AI-generated insights. In summary, by empowering surgeons with advanced, user-centered tools for intraoperative assistance and autonomous reporting, the ExpRT project aspires to improve surgical decision-making and report quality in laparoscopic explorations, ultimately driving improvements in patient outcomes.