Original ArticleAugmented Reality during Open Liver Surgery Using a Markerless Non-rigid Registration System
Introduction
For primary and secondary hepatobiliary malignancies, hepatic resection represents one of the few potentially curative treatments. Despite recent advances,1 liver surgery still faces several challenges, as a delicate intervention within an anatomically complex organ.2
A proper resection indeed implies the complete removal of the tumor(s) (with oncological margins) while preserving the surrounding tissues, vessels, and biliary tree, in order to keep enough remnant parenchyma3 and vascularization (inflow/outflow) for a functional tissue. An accurate design of the area to be resected thus needs to be performed based on preoperative imaging and 3-D reconstructions and then faithfully transposed into the operative field thanks to intraoperative navigation. To guide surgeons, the main tool used for decades still remains ultrasonography.4 It is, however, an expensive imaging system that requires radiological skills and mental effort to understand the 3-D anatomy from the provided images. Because of these limitations, more than half of the patients receive a non-optimal resection (i.e., with insufficient surgical margins), even in tertiary centers.5 In case of missing metastases (disappeared lesions after chemotherapy while residual tumor tissue remaining6,7), there is no visible scar so that ultrasonography is useless to guide resection or ablation and surgeons are often in delicate situations with the impossibility of resecting the scar. Conversely, augmented reality (AR) can be based on pre-chemotherapy images and could guide the curative procedure.
Other navigation modalities (cone beam CT,8 US-CT/MR fusion imaging,9 fusion-fluorescence imaging10) are investigated worldwide because they do not require any elastic registration (direct matching between imaging and operative view), but their cost and their level of practical expertise are also drawbacks that prevent their widespread routine use. Hence, it is becoming increasingly clear that surgeons need a minimally invasive, non-irradiating, and practical device to replace or complement ultrasonographic examination (difficult/deep resection only, probably useless for peripheral/subcapsular lesions). Such an ideal tool has to prove its accuracy in many different surgical situations.
It can be pointed out that for complex, multiple, central (perihilar), or large lesions deforming the biliary and vascular tracts, open surgery remains the gold-standard approach. In this setting, it appears essential to develop AR. In fact, planning the ideal resection area and applying it intraoperatively (without visible targets) thanks to AR may help surgeons to optimize the surgical plan and to minimize the recurrence risk. However, unlike neurosurgery or orthopedic surgeries in which organs do not suffer from large deformation, real-time intraoperative AR is still not a routine procedure during hepatic surgery, and no commercial devices are actually available in this setting. Indeed, open liver surgery faces important technical issues that actually prevent using AR with sufficient accuracy for clinical use: a large deformation due to the mobilization of the liver (soft tissue) and stabilization by packing support material, as well as a limited access to the organ surface area (anterior surface hidden by costal arches, posterior face not seen).
In this work, we propose a clinical and experimental assessment of an original solution for intraoperative AR during open liver surgery. We aimed to demonstrate the feasibility of AR navigation in the operating room (in vivo qualitative evaluation) and to clearly assess its spatial accuracy with ex vivo experiments (quantitative evaluation). This preliminary study appeared as a necessary step before further works that will quantify the clinical impact of such navigation tools.
Section snippets
Study Population and Surgical Procedures
Patients were operated between February and March 2019 at Paul Brousse University Hospital. We included five patients: three patients receiving partial hepatectomy for malignancy (in vivo experiments, Patients [P] 1 to 3) and two recipients of whole graft liver transplantation (P4 for the in vivo experiment, P5 for the ex vivo experiment). All of the patients provided their informed and written consent to participate to this study.
In the in vivo group, patients were informed that their
Results
In vivo experiments on four patients succeeded in displaying the preoperative 3-D meshes on external screen, with real-time deformation according to the position of the tracked livers. The models visually correctly fitted with organs’ outer edges, and the complete pipeline execution appeared as compatible with surgical environment (space work and time constraints). In terms of computation time, an update frequency above 15 Hz was obtained for all patients, thus ensuring a smooth visualization
Statement of Principal Findings
This study is the first report of a single depth camera-based system used for open liver surgery AR in humans. The in vivo experiment qualitatively validated the relevance and performance of this novel method, while the ex vivo tests showed acceptable accuracy during extreme deformations of the liver (internal precision < 1 cm). This markerless and non-irradiating setup is a promising technique that could be helpful in hepatobiliary centers to facilitate complex liver surgeries.
Strengths and Weaknesses of the Study
In the specific
Conclusion
Our non-rigid registration system demonstrated its feasibility and convenience in clinical conditions (in vivo) with satisfactory anatomical accuracy in a standardized (ex vivo) setting, even after very large deformations occurred. This proof-of-concept study lays the foundations for further tests and improvements. Surgical impact of such an innovative tool must now be assessed in surgical practice on a large cohort in order to confirm the expected gain on intraoperative features (surgical
Authors’ Contribution
N Golse and A Petit: wrote manuscript, performed experiments. M Lewin: CT scan acquisition, images analysis. S Cotin and E Vibert: research supervision, manuscript rewriting, scientific advices.
Conflict of Interest
The authors declare that they have no conflict of interest.
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References (32)
- et al.
Remnant Liver Volume to Body Weight Ratio ≥0.5%: A New Cut-Off to Estimate Postoperative Risks after Extended Resection in Noncirrhotic Liver
J Am Coll Surg
(2007) - et al.
Colorectal liver metastases: disappearing lesions in the era of Eovist hepatobiliary magnetic resonance imaging
HPB (Oxford).
(2016) - et al.
Augmented reality in laparoscopic surgical oncology
Surg Oncol
(2011) - et al.
Surgery of Colorectal Liver Metastases: Pushing the Limits
Liver Cancer.
(2016) - et al.
Anatomy of the liver: an outline with three levels of complexity--a further step towards tailored territorial liver resections
J Hepatol
(2014) - et al.
Peroperative ultrasonography in the surgery of hepatic tumours
Br J Surg
(1988) - Viganò L, Torzilli G, Cimino M, Imai K, Vibert E, Donadon M, et al. Drop-out between the two liver resections of...
- Ntourakis D, Memeo R, Soler L, Marescaux J, Mutter D, Pessaux P. Augmented Reality Guidance for the Resection of...
- et al.
Real-time image guidance in laparoscopic liver surgery: first clinical experience with a guidance system based on intraoperative CT imaging
Surg Endosc.
(2014) - et al.
Real-time US-CT/MR fusion imaging for percutaneous radiofrequency ablation of hepatocellular carcinoma
J Hepatol.
(2017)
Applications of fusion-fluorescence imaging using indocyanine green in laparoscopic hepatectomy
Surg Endosc.
A unified formulation of small-strain corotational finite elements: I. Theory. Comput Methods Appl Mech Engrg
North-Holland
Cited by (45)
Augmented Reality Guided Laparoscopic Liver Resection: A Phantom Study With Intraparenchymal Tumors
2024, Journal of Surgical ResearchA method of three-dimensional non-rigid localization of liver tumors based on structured light
2024, Optics and Lasers in EngineeringInterpretation and Use of Applied/Operational Machine Learning and Artificial Intelligence in Surgery
2023, Surgical Clinics of North AmericaAugmented reality in liver surgery
2023, Journal of Visceral SurgeryNavigated liver surgery: State of the art and future perspectives
2022, Hepatobiliary and Pancreatic Diseases International