Original Article
Augmented Reality during Open Liver Surgery Using a Markerless Non-rigid Registration System

https://doi.org/10.1007/s11605-020-04519-4Get rights and content

Abstract

Introduction

Intraoperative navigation during liver resection remains difficult and requires high radiologic skills because liver anatomy is complex and patient-specific. Augmented reality (AR) during open liver surgery could be helpful to guide hepatectomies and optimize resection margins but faces many challenges when large parenchymal deformations take place. We aimed to experiment a new vision-based AR to assess its clinical feasibility and anatomical accuracy.

Patients and Methods

Based on preoperative CT scan 3-D segmentations, we applied a non-rigid registration method, integrating a physics-based elastic model of the liver, computed in real time using an efficient finite element method. To fit the actual deformations, the model was driven by data provided by a single RGB-D camera. Five livers were considered in this experiment. In vivo AR was performed during hepatectomy (n = 4), with manual handling of the livers resulting in large realistic deformations. Ex vivo experiment (n = 1) consisted in repeated CT scans of explanted whole organ carrying internal metallic landmarks, in fixed deformations, and allowed us to analyze our estimated deformations and quantify spatial errors.

Results

In vivo AR tests were successfully achieved in all patients with a fast and agile setup installation (< 10 min) and real-time overlay of the virtual anatomy onto the surgical field displayed on an external screen. In addition, an ex vivo quantification demonstrated a 7.9 mm root mean square error for the registration of internal landmarks.

Conclusion

These first experiments of a markerless AR provided promising results, requiring very little equipment and setup time, yet providing real-time AR with satisfactory 3D accuracy. These results must be confirmed in a larger prospective study to definitively assess the impact of such minimally invasive technology on pathological margins and oncological outcomes.

Introduction

For primary and secondary hepatobiliary malignancies, hepatic resection represents one of the few potentially curative treatments. Despite recent advances,1 liver surgery still faces several challenges, as a delicate intervention within an anatomically complex organ.2

A proper resection indeed implies the complete removal of the tumor(s) (with oncological margins) while preserving the surrounding tissues, vessels, and biliary tree, in order to keep enough remnant parenchyma3 and vascularization (inflow/outflow) for a functional tissue. An accurate design of the area to be resected thus needs to be performed based on preoperative imaging and 3-D reconstructions and then faithfully transposed into the operative field thanks to intraoperative navigation. To guide surgeons, the main tool used for decades still remains ultrasonography.4 It is, however, an expensive imaging system that requires radiological skills and mental effort to understand the 3-D anatomy from the provided images. Because of these limitations, more than half of the patients receive a non-optimal resection (i.e., with insufficient surgical margins), even in tertiary centers.5 In case of missing metastases (disappeared lesions after chemotherapy while residual tumor tissue remaining6,7), there is no visible scar so that ultrasonography is useless to guide resection or ablation and surgeons are often in delicate situations with the impossibility of resecting the scar. Conversely, augmented reality (AR) can be based on pre-chemotherapy images and could guide the curative procedure.

Other navigation modalities (cone beam CT,8 US-CT/MR fusion imaging,9 fusion-fluorescence imaging10) are investigated worldwide because they do not require any elastic registration (direct matching between imaging and operative view), but their cost and their level of practical expertise are also drawbacks that prevent their widespread routine use. Hence, it is becoming increasingly clear that surgeons need a minimally invasive, non-irradiating, and practical device to replace or complement ultrasonographic examination (difficult/deep resection only, probably useless for peripheral/subcapsular lesions). Such an ideal tool has to prove its accuracy in many different surgical situations.

It can be pointed out that for complex, multiple, central (perihilar), or large lesions deforming the biliary and vascular tracts, open surgery remains the gold-standard approach. In this setting, it appears essential to develop AR. In fact, planning the ideal resection area and applying it intraoperatively (without visible targets) thanks to AR may help surgeons to optimize the surgical plan and to minimize the recurrence risk. However, unlike neurosurgery or orthopedic surgeries in which organs do not suffer from large deformation, real-time intraoperative AR is still not a routine procedure during hepatic surgery, and no commercial devices are actually available in this setting. Indeed, open liver surgery faces important technical issues that actually prevent using AR with sufficient accuracy for clinical use: a large deformation due to the mobilization of the liver (soft tissue) and stabilization by packing support material, as well as a limited access to the organ surface area (anterior surface hidden by costal arches, posterior face not seen).

In this work, we propose a clinical and experimental assessment of an original solution for intraoperative AR during open liver surgery. We aimed to demonstrate the feasibility of AR navigation in the operating room (in vivo qualitative evaluation) and to clearly assess its spatial accuracy with ex vivo experiments (quantitative evaluation). This preliminary study appeared as a necessary step before further works that will quantify the clinical impact of such navigation tools.

Section snippets

Study Population and Surgical Procedures

Patients were operated between February and March 2019 at Paul Brousse University Hospital. We included five patients: three patients receiving partial hepatectomy for malignancy (in vivo experiments, Patients [P] 1 to 3) and two recipients of whole graft liver transplantation (P4 for the in vivo experiment, P5 for the ex vivo experiment). All of the patients provided their informed and written consent to participate to this study.

In the in vivo group, patients were informed that their

Results

In vivo experiments on four patients succeeded in displaying the preoperative 3-D meshes on external screen, with real-time deformation according to the position of the tracked livers. The models visually correctly fitted with organs’ outer edges, and the complete pipeline execution appeared as compatible with surgical environment (space work and time constraints). In terms of computation time, an update frequency above 15 Hz was obtained for all patients, thus ensuring a smooth visualization

Statement of Principal Findings

This study is the first report of a single depth camera-based system used for open liver surgery AR in humans. The in vivo experiment qualitatively validated the relevance and performance of this novel method, while the ex vivo tests showed acceptable accuracy during extreme deformations of the liver (internal precision < 1 cm). This markerless and non-irradiating setup is a promising technique that could be helpful in hepatobiliary centers to facilitate complex liver surgeries.

Strengths and Weaknesses of the Study

In the specific

Conclusion

Our non-rigid registration system demonstrated its feasibility and convenience in clinical conditions (in vivo) with satisfactory anatomical accuracy in a standardized (ex vivo) setting, even after very large deformations occurred. This proof-of-concept study lays the foundations for further tests and improvements. Surgical impact of such an innovative tool must now be assessed in surgical practice on a large cohort in order to confirm the expected gain on intraoperative features (surgical

Authors’ Contribution

N Golse and A Petit: wrote manuscript, performed experiments. M Lewin: CT scan acquisition, images analysis. S Cotin and E Vibert: research supervision, manuscript rewriting, scientific advices.

Conflict of Interest

The authors declare that they have no conflict of interest.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References (32)

  • M. Terasawa et al.

    Applications of fusion-fluorescence imaging using indocyanine green in laparoscopic hepatectomy

    Surg Endosc.

    (2017)
  • Faure F, Duriez C, Delingette H, Allard J, Gilles B, Marchesseau S, et al. SOFA: A Multi-Model Framework for...
  • Petit A, Lippiello V, Fontanelli GA, Siciliano B. Tracking elastic deformable objects with an RGB-D sensor for a pizza...
  • C.A. Felippa et al.

    A unified formulation of small-strain corotational finite elements: I. Theory. Comput Methods Appl Mech Engrg

    North-Holland

    (2005)
  • Cook RD. Finite element modeling for stress analysis. Wiley; 1994...
  • Petit, A., Cotin, S., Lippiello, V., Siciliano, B.: Capturing deformations of interacting non- rigid objects using...
  • Cited by (45)

    • Augmented reality in liver surgery

      2023, Journal of Visceral Surgery
    • Navigated liver surgery: State of the art and future perspectives

      2022, Hepatobiliary and Pancreatic Diseases International
    View all citing articles on Scopus
    View full text