Parameter estimation and registration of dynamically deformable surfaces and volumes in medical applications

Assistive technologies play an increasingly important role in modern surgery procedures. A number of concepts for surgery assistance have been implemented in recent years. One of these is the concept of "Virtual Fixtures" [1]. A system designed according to those principles is shown in Figure 1 below.

While the system described in that work can provide essential help for surgeons in difficult tasks in static environments, the method has some drawbacks when it comes to non-static environments, which are quite frequently encountered in most surgery processes. There are many sources for disturbance, e.g., breathe, heartbeat, muscle contractions etc.

Within this project, we want to address this problem by finding a suitable model for the environment, approximating physical processes that lead to deformations. The modeling allows us to adapt the fixtures to the deformations and movements and do a limited prediction of movements.

Figure 1: Schematic overview of a medical assistance system. The system generates 3D information which is provided as visual feedback to the surgeon. At the same time, it receives control signals from the surgeon and enforces compliance with some pre-defined constraints.

Involved Areas

There are several scientific areas involved in designing systems such as the ones discussed above:

Computer Vision: Processing raw video data, performing tracking and 3D reconstruction
Sensor Fusion: Estimate and predict the system state, process input from different sensors
Surface modelling: Which mathematical model is chosen to model a surface? B-Splines, implicit surfaces etc.
Physical modelling: How do we approximate the physical process describing the environment?

In the following, we will discuss some of the research that has taken place in each to the areas.

Physical modelling

There are quite a number of possibilities for simulating physical processes on computer systems. Some of the options we have looked into are:

Finite Elements: A very sophisticated method for approximating solutions of differential equations. This is probably the state-of-the-art method when it comes to physical simulations. It is very accurate and is, e.g., also used for simulation of car crashes in car design. It can also be applied to our problem, and would be the most accurate of all the options introduced here. Drawbacks are that it is also computationally very expensive and rather complicated to implement.
Mass-Spring-Models: Instead of properly modelling the physical process, its behaviour is approximated by assuming that a surface acts like a network of springs. Figure 2 below shows the concept of a mass-spring-model. Advantages as compared to the Finite Elements method are its simplicity and the better computational performance. The drawback is decreased accuracy. The behaviour of a mass-spring-model can easily be approximated using a finite differences approach.
As-rigid-as-possible-deformations [2]: A relatively new approach that is being used successfully in computer graphics. It is not clear if it would be a reasonable model for medical applications, but seems very promising.
Chain-Mail [3]: Another simplified approach for simulation soft tissue.

For now, we have concentrated our efforts on mass-spring-models.

Figure 2: Schematic drawing of a mass-spring-model. The black dots represent mass points, the jagged lines between the dots are springs.

Sensor fusion, parameter estimation and state prediction

In any task where we obtain measurements from the real world by means of sensors for the purpose of determining the state of a physical system, we have to deal with the following problems:

Sensor measurements are always only accurate up to a certain precision, and thus are uncertain values.
The state of the observed system can also only be determined up to a certain accuracy, and thus is also uncertain.
Physical processes can only be modeled with limited accuracy, so state changes in the observed systems will introduce even more uncertainty.

The usual approach for addressing these issues is employment of sensor fusion schemes. This means treating the system state and sensor measurements as random variables whose behaviour is described by a certain random distribution. Then we can simply compute the system state that has the highest probability and will obtain a pretty good estimate of the actual system state. There are several alternatives for sensor fusion methods, and all have their benefits and drawbacks:

Extended Kalman Filter: This is the method that is usually applied when the system transition (physical processes) are nonlinear functions. Uncertainties are modeled as Gaussian distributions.
Unscented Kalman Filter: A more sophisticated filter. Has a better approximation of the true random distribution, but is computationally more intensive.
DD1, DD2 Filters [4]: Other alternatives to the Extended Kalman Filter.
Particle Filters: A Monte-Carlo-Method, most general, but also quite computationally intensive.

Until now, we have mainly focused on employment of the Extended Kalman Filter. Benefits of employing other methods will be determined through experiments.

Surface Tracking

Ramey et al. [5] show how real-time capable tracking in medical applications can be achieved. However, their tracking approach is based only on the disparity map of the image. We have generalized the approach so that it can perform intensity-based tracking of a surface that is modeled as a B-Spline patch. Instead of only modeling the disparity map, we are using a full 3D model of the surface we are interested in. Our method simultaneously refines the surface parameters and determines the camera position. Figure 3 shows an example result of the tracking process.

Figure 3: Result of the surface tracking algorithm on a synthetic sequence. The green lines indicate matchings of some reference points from the template image (left) to the current image (right). Even though the number of reference points is very low and the comparison is only intensity-based, the result is quite accurate.

Media

There is a video here that shows the tracking method in action on a test sequence.

Planned Work

A lot of work remains to be done in order to achieve the goals stated above for this project. The plans for the near future concerning the surface tracking algorithm look like this:

Until now, there is no initialization algorithm implemented for the surface tracking system. A sensible initial guess of surface parameters and camera position have to be provided. There are several options here for performing the initialization automatically. Bundle adjustment could be used, which would yield a point cloud of the surface, and then a B-Spline model can be fitted to that point cloud.
The computations performed for the surface tracking algorithm can be improved both with respect to computation speed and with respect to accuracy.
Tracking only works on static 3D surfaces for now. This should be extended some time so it works with deformable surfaces as well.

Once the surface tracking and modeling algorithm is implemented, we can continue on to the problem of physical modelling. The open problems in that area are:

Try out and determine which physical modeling approaches are suitable (both when it comes to accuracy and computational cost) for our applications.
Implement parameter estimation and sensor fusion methods for that model.
Predict deformation movements and camera movements to avoid collisions, guide the surgeon etc.

People

References

[1] Darius Burschka, Jason J. Corso, Maneesh Dewan, William Lau, Ming Li, Henry Lin, Panadda Marayong, Nicholas Ramey, Gregory D. Hager, Brian Hoffman, David Larkin, Christopher Hasser, Navigating inner space: 3-D assistance for minimally invasive surgery, Robotics and Autonomous Systems, Volume 52, Issue 1, Advances in Robot Vision, 31 July 2005, Pages 5-26, ISSN 0921-8890.

[2] Olga Sorkine and Marc Alexa, As-Rigid-As-Possible Surface Modeling, Proceedings of Eurographics/ACM SIGGRAPH Symposium on Geometry Processing , p.109-116, 2007

[3] Sarah F. Gibson, 3D chainmail: a fast algorithm for deforming volumetric objects, Proceedings of the 1997 symposium on Interactive 3D graphics, p.149-ff., April 27-30, 1997, Providence, Rhode Island, United States.

[4] M. Nørgaard, N.K. Poulsen, O. Ravn: Easy and Accurate State Estimation for Nonlinear Systems, 14th IFAC World Congress, Beijing, China, July 5-9, 1999, Vol. J, pp. 343-348.

[5] Nicholas A. Ramey and Jason J. Corso and William W. Lau and Darius Burschka and Gregory D. Hager, Real Time 3D Surface Tracking and Its Applications , Proceedings of Workshop on Real-time 3D Sensors and Their Use (at CVPR 2004), 2004.

Machine Vision and Perception Group (MVP@TUM)