Preliminary Study on the Clinical Application of Augmented Reality Neuronavigation

Abstract

Objective To develop an augmented reality (AR) neuronavigation system with Web cameras and examine its clinical utility.

Methods The utility of the system was evaluated in three patients with brain tumors. One patient had a glioblastoma and two patients had convexity meningiomas. Our navigation system comprised the open-source software 3D Slicer (Brigham and Women's Hospital, Boston, Massachusetts, USA), the infrared optical tracking sensor Polaris (Northern Digital Inc., Waterloo, Canada), and Web cameras. We prepared two different types of Web cameras: a handheld type and a headband type. Optical markers were attached to each Web camera. We used this system for skin incision planning before the operation, during craniotomy, and after dural incision.

Results We were able to overlay these images in all cases. In Case 1, accuracy could not be evaluated because the tumor was not on the surface, though it was generally suitable for the outline of the external ear and the skin. In Cases 2 and 3, the augmented reality error was ∼2 to 3 mm.

Conclusion AR technology was examined with Web cameras in neurosurgical operations. Our results suggest that this technology is clinically useful in neurosurgical procedures, particularly for brain tumors close to the brain surface.

Keywords

neuronavigation - augmented reality - camera devices - superimposition - tractography

Introduction

Image-guided neurosurgeries, so-called neuronavigation systems, were introduced into neurosurgery in the late 1980s and early 1990s, earlier than they were introduced into other surgical areas. These systems are becoming routinely used in many neurosurgical procedures and have proven to be important neurosurgical tools.[1] [2] The most popular type of neuronavigation system is the optical system (reflection of infrared light).[3] One of the shortcomings of this optical system is that neurosurgeons must look away from the surgical field to see the navigation display because the navigation monitors are located far from the lens tubes of the microscope and monitors of the neuroendoscope. A navigation system that can be used without movement of the surgeon's eyes away from the surgical field would be a vast improvement.

Augmented reality (AR) is a recently developed technology that has the potential to overcome the need to look away from the surgical field. It adds information to a real environment with computers, and its usefulness has become gradually recognized in the field. For application in the medical field, AR overlays a virtual image provided by three-dimensional (3D) reconstruction of a computed tomography (CT) or magnetic resonance imaging (MRI) image onto an actual video or image.[4] [5] [6]

Some neurosurgical research groups have published reports on AR navigation, but there have been only a few papers on AR neuronavigation in clinical neurosurgical cases.[1] [7] [8] [9] The system described by Kockro et al. required a special handheld probe with an integrated lipstick-shaped camera.[1] The system described by King et al. required bone-implanted markers and a locking acrylic dental stent.[7] The system described by Kawamata et al. required reference markers mounted on a goggle-type frame.[8] The system described by Paul et al. required a surgical microscope, optical localizer by means of light-emitting diodes, and dynamic reference frame attached to the patient's head.[9] However, our AR neuronavigation system does not require these types of special equipment and uses simple, commercially available camera devices. Unlike the above-mentioned AR neuronavigation systems, our AR neuronavigation system can superimpose not only tumors and vessels but can also perform tractography. The purpose of this study was to evaluate the feasibility and effectiveness of an AR neuronavigation system based on Web camera images.

Patients and Methods

The proposed system was applied in three patients (one with a glioblastoma and two with convexity meningiomas). All three patients underwent operations with this navigation system. We developed and validated the utility of this AR navigation system by superimposing tumors and vessels that had been segmented in advance onto a Web camera image. Preoperative segmentation and calibration of Web cameras was required.

Navigation System

Segmentation

The open-source software 3D Slicer (Brigham and Women's Hospital, Boston, Massachusetts, USA) was used as the software platform. Thin-slice sagittal cerebral Gd-enhanced T1-weighted magnetic resonance (MR) images of patients with attached fiducial markers were acquired in Digital Imaging and Communications in Medicine (DICOM) format 1 day before the operation. The MR images were acquired using a 1.5-tesla MRI scanner, and 200 axial cerebral T1-weighted sequences of 1.2-mm thickness were acquired in DICOM format.

DICOM data were taken using 3D Slicer, and 3D models of tumors and vessels were made and stored in Visualization Toolkit (VTK) file format as 3D surface models. For corticospinal tractography, diffusion tensor imaging was acquired by a 3.0-tesla MRI scanner. We created diffusion tensor imaging (DTI) tractography using the labelmap seeding function of 3D Slicer. For labelmap seeding, we made tract fibers from a cerebral peduncle and set up the posterior limb of the internal capsule as the region of interest.

Web Camera and Its Calibration

We used two different Web cameras in this study. One was a Web camera with 2 million pixels (Qcam Pro 9000 QCAM-200S; Logicool Co., Tokyo, Japan), and the other was a Web camera with 300,000 pixels (Qcam Connect; Logicool Co.). Both cameras' refresh rates were 30 frames per second. We used the former by hand intraoperatively (handheld type in Cases 1 and 3) ([Fig. 1A]) and mounted the latter on the assistant's head (headband type in Case 2) ([Fig. 1B]). Optical markers were attached to each Web camera. The open-source OpenCV library (Willow Garage, Menlo Park, California, USA) was used to calibrate the Web cameras. The first step of calibration was to extract the camera's intrinsic parameters using a snapshot of a chessboard in 10 different positions ([Fig. 2]). The extrinsic parameters that showed a relationship between the 3D world coordinate and camera coordinate were then calculated using a chessboard with optical markers. We used the method of Zhang et al. for this camera calibration.[10]

Fig. 1 The two types of Web cameras used. Optical markers were attached to each Web camera. (A) Handheld. (B) Headband.

Fig. 2 Chessboard for calibration. We extracted each Web camera's intrinsic parameters using a snapshot of the chessboard in different positions. The extrinsic parameters that showed a relationship between the three-dimensional (3D) world coordinate and camera coordinate were then calculated using the chessboard with optical markers.

Registration

Our navigation system comprised 3D Slicer software, the infrared optical tracking sensor Polaris (Northern Digital Inc., Waterloo, Canada), and Web cameras. A surgical navigation system, StealthStation Treon plus (Medtronic, Coal Creek, Colorado, USA), was used in addition to 3D Slicer. In the operating room, Web cameras with optical markers were connected to the navigation system. Polaris was used for the position sensor to detect and track the Web cameras. We made a point-based registration in 3D Slicer with fiducial markers[11] and a Medtronic reference table. We used a Medtronic reference frame connected with the head fixation holder without a new frame. 3D Slicer displayed navigational information on a 20-inch monitor divided into two windows. During the skin incision and craniotomy, overlaid images were displayed with Web cameras. After being informed of the surgeon's intention, the image-guided surgery team in our institute controlled this system.

Results

We were able to overlay these images in all cases ([Figs. 3], [4]). Before performing AR, the registration error was computed. The fiducial registration errors were 1.79, 1.67, and 1.65 mm. Figure 2 shows the overlaid image of the tumor and skin during the operation. Accuracy could not be evaluated because the tumor was not on the surface, though it was roughly suitable in the outline of the external ear and the skin in Case 1. In Cases 2 and 3, a tumor was present on the brain surface, and the gap between the outline of the actual tumor and the outline of the created tumor was visible. AR accuracy, measured with a paper ruler in the plane of the operative fields, was ∼2 to 3 mm. This indicates that the system is suitable for clinical use. Convexity meningiomas were present in two patients who were discharged without new neurological deficits after total removal of the tumors. One patient with a glioblastoma had no new neurological deficits, but tumor remained at the corpus callosum and inside the resection cavity. This patient was discharged in good general condition after radiochemotherapy (temozolomide 75 mg/m² for 42 days + total 60 Gy).

Fig. 3 (A) Augmented reality navigation monitors using a handheld Web camera in Cases 1 and 2. Tumor (green) and skin (ochre) superimposed onto the patient before disinfection in Cases 1 and 2. (B) Augmented reality navigation monitor using a headband type Web camera in Case 2. Left: the virtual 3D graphical image. Right: the superimposed image.

Fig. 4 (A) Augmented reality navigation monitor using a handheld Web camera in Case 3. A tumor (red) and motor tractography (green) were superimposed onto the patient's head before disinfection and after dural incision. (B) Upper: dual three-dimensional (3D) layout display in 3D Slicer. Lower: Distance between the bipolar tip and motor tractography was measured.

Illustrative Case

This case involved a left parietal convexity meningioma in a 64-year-old right-handed woman. She had undergone an operation for breast cancer 2 years previously. At admission, she had no neurological deficits. She was positioned supine and underwent an operation by left parietal craniotomy. The skin incision was marked directly in relation to the expected subcortical tumor position. After the craniotomy, the tumor and corticospinal tract were superimposed on the monitor. During tumor resection, the navigation monitor displayed a dual 3D layout; that is, 3D was indicated from two different directions in real time. The monitor indicated the distance between the tip of the bipolar forceps and the corticospinal tract ([Fig. 4B]). The tumor was very close to the left corticospinal tract. During the operation, we manually measured the distance from the tractography to the bipolar tip using the measurement function of 3D Slicer, as shown in [Fig. 4B]. Using subcortical motor evoked potentials, the tumor was removed with only a small amount remaining. The patient was discharged with no new neurological deficits.

Discussion

This is a clinical report on a new AR neuronavigation system for brain tumors adjacent to the brain surface. A main advantage of this navigation system is that it consists of the open-source software 3D Slicer and Web cameras: thus, any facility could easily set up this system. 3D Slicer is superior to commercial systems in terms of expressing 3D images, facilitating a more intuitive understanding of 3D spatial relationships. This 3D display function allows for real-time navigation while watching 3D displays from two different angles, which is termed a 3D dual-layout display. In addition, we can easily measure the distance between points of objects and surgical instrument tips.

We superimposed segmented objects onto Web camera images on the monitor by connecting a Web camera to 3D Slicer. We were able to not only overlay tumors and vessels but also perform motor tractography, which differs from past reports on camera AR navigation.[1] [12] Visualization of the corticospinal tract on AR display proved to be a useful tool for the surgeon to avoid inadvertently damaging the tract. Moreover, the positions of tumors, vessels, and tracts can be readily identified during preoperative planning of the skin incision.

We used two types of Web cameras. We were able to move the handheld camera, but there was some limitation in the range of movement because of the position of optical markers attached to the camera. We were able to ascertain the depth of the lesion on the lateral side.

However, the operation must be interrupted when the handheld camera is moved. The headband type of camera can continuously track through the assistant's line of sight, but the assistant's position is limited because of the position of the optical markers and the fact that he or she cannot see the monitor. In addition, it is difficult to move Web cameras without shaking them and taking pictures at the same eye level of the operator.

3D Slicer has two features that differ from those of commercially available navigation systems. One is related to the 3D images. 3D Slicer has notable advantages in displaying objects in 3D space with very intuitive and customizable models. It can display arbitrary cross-sectional planes in 3D space according to the position of surgical instruments with customizable offsets. In addition, the 3D graphic objects such as tumors or vessels are visualized by optimal volume- and surface- rendering techniques, the parameters of which can be finely adjusted according to the surgeon's intention. The other feature is related to the superimposition function of medical camera devices. Our system can utilize not only Web cameras but also rigid neuroendoscopes and microscopes with optical markers. The proposed system can also be applied to the navigation surgeries based on neuroendoscopes and microscopes with high levels of camera-calibration techniques.[13] The depth perception problem can also be overcome by the dual-layout display of 3D Slicer.

Certainly, commercially available navigation systems such as BrainLab and Medtronic can insert superimposed two-dimensional (2D) images into one optical channel of the microscope. However, 3D spatial relationships are unclear on superimposed 2D images. 3D Slicer can provide virtual 3D images and identical superimposed images with various view angles simultaneously ([Fig. 3B]). Virtual 3D images can allow surgeons to intuitively perceive the depth of lesions, which is difficult to grasp on conventional superimposed images. In addition, target lesions can be preoperatively evaluated by AR navigation with Web camera in a simpler, less expensive manner.

It is very difficult to evaluate AR error in three dimensions, and no such evaluation method has yet been established. We performed several steps to evaluate AR error in three dimensions. The first step is to measure the phantom superimposed images at different angles. The 3D error can be estimated from each angle's error.[14] The second step is to check the AR error on the patient's scalp image before beginning the operations. In this study, we confirmed that the previously segmented scalp image was not so far from the patient's scalp image at various angles. The final step is to measure the targets in the operative fields. In this step, we measured the distance between actual brain tumors and previously segmented tumors in two dimensions with a paper ruler.

The AR navigation system still has some problems. First, it is difficult to accurately judge the depth of tumors from 2D displays. Second, it is not suitable for deep tumors because of the performance of Web cameras. Third, there is no established method for precise measurement of AR error; although we can measure it in one plane, it is difficult to accurately measure error in 3D. Fourth, the AR navigation monitor must be shown to operators. The monitor position and timing of presenting AR images to neurosurgeons must be considered so that eye movement is minimized.

However, the AR neuronavigation system has considerable potential in neurological surgery. We have herein described our clinical experience using this system in the operating room. In the future, we plan to continue the evaluation of its clinical utility by using it in operations involving neuroendoscopes and operating microscopes.

Conclusion

AR technology was examined with Web cameras in neurosurgical operations. The proposed navigation system may help surgeons to perform safe surgical procedures and confirm their decisions. The results of this study suggested that this technology was useful in clinical neurosurgical procedures, particularly for brain tumors close to the brain surface.

Conflict of Interest

None

• References

• 1 Kockro RA, Tsai YT, Ng I , et al. Dex-ray: augmented reality neurosurgical navigation with a handheld video probe. Neurosurgery 2009; 65: 795-807 , discussion 807–808
• 2 Grunert P, Darabi K, Espinosa J, Filippi R. Computer-aided navigation in neurosurgery. Neurosurg Rev 2003; 26: 73-99 , discussion 100–101
• 3 Enchev Y. Neuronavigation: geneology, reality, and prospects. Neurosurg Focus 2009; 27: E11
• 4 Masutani Y, Dohi T, Yamane F, Iseki H, Takakura K. Augmented reality visualization system for intravascular neurosurgery. Comput Aided Surg 1998; 3: 239-247
• 5 Sielhorst T, Feuerstein M, Navab N. Advanced medical displays: a literature review of augmented reality. J Disp Technol 2008; 4: 451-467
• 6 Shuhaiber JH. Augmented reality in surgery. Arch Surg 2004; 139: 170-174
• 7 King AP, Edwards PJ, Maurer Jr CR , et al. A system for microscope-assisted guided interventions. Stereotact Funct Neurosurg 1999; 72: 107-111
• 8 Kawamata T, Iseki H, Shibasaki T, Hori T. Endoscopic augmented reality navigation system for endonasal transsphenoidal surgery to treat pituitary tumors: technical note. Neurosurgery 2002; 50: 1393-1397
• 9 Paul P, Fleig O, Jannin P. Augmented virtuality based on stereoscopic reconstruction in multimodal image-guided neurosurgery: methods and performance evaluation. IEEE Trans Med Imaging 2005; 24: 1500-1511
• 10 Zhang Z. Flexible camera calibration by viewing a plane from unknown orientations. International Conference on Computer Vision 1999; 1: 666-673
• 11 Hong J, Hashizume M. An effective point-based registration tool for surgical navigation. Surg Endosc 2010; 24: 944-948
• 12 Lovo EE, Quintana JC, Puebla MC , et al. A novel, inexpensive method of image coregistration for applications in image-guided surgery using augmented reality. Neurosurgery 2007; 60: 366-371 , discussion 371–372
• 13 Cho B, Oka M, Matsumoto N , et al. Augmented reality of surgical microscope for otologic surgery. In: Proceedings of CARS, Berlin, Germany, 2011; 6: S245-S246
• 14 Kim S, Hong J, Joung S , et al. Dual surgical navigation using augmented and virtual environment techniques. Int J Optomechatronics 2011; 5: 155-169