IMU-Based Tool Tracking During Visual Occlusion

2022 COE Engineering Design Project (RP05)

---

Faculty Lab Coordinator

Robnier Reyes Perez

Topic Category

Software Systems

Preamble

Visual based tracking of tools is becoming essential across many industries, where precision is of utmost importance. Within the medical field specifically, surgical tools are visually tracked via stereoscopic cameras in order to give the surgeon accurate information regarding the current position and orientation of the tool in relation to the patient. Various systems such as the Medtronic Mazor X, 7D Surgical Flash, etc… help guide the surgeon during a procedure. These stereoscopic cameras are either proprietary or off the shelf (NDI Polaris), with the NDI being able to have sub-millimeter accuracy. However, these cameras are subject to occlusion and will not track if either the vision is blocked or the tool is outside the tracking environment. This results in loss of accuracy and could potentially be dangerous in certain situations (surgery). These problems can be alleviated if one has a secondary method of tracking that does not rely on visuals.

Objective

The goal of this project is to use an Inertial Motion Unit (IMU) and visuals (ArUco) to track a surgical tool. Position and orientation of the tool must be measured. Once visual tracking is either stopped, due to occlusion or being out of bounds, the IMU must supplement the tracking.

Partial Specifications

- Must be developed with off the shelf components.
- A mock surgical tool must be made for the purpose of tracking.
- A graphical user interface (GUI) showing the current tool pose must be implemented.
- Milimetric tracking accuracy is expected to be achieved with the IMU, however if the drift is too high and the measurement is much too inaccurate, a warning must appear on the GUI. The GUI interface must be present to show the position and orientation of the tool in a 3D space.
- Filtering techniques should be used in order to reduce IMU drift. Students should report on how long they can maintain millimetric tracking with the IMU.

Suggested Approach

- An off the shelf IMU with self calibration is preferred. Otherwise, the accelerometer, gyroscope and magnetometer must be calibrated individually.
- The mock surgical tool should be designed in a CAD software (Fusion 360) and 3D printed.
- The IMU should be placed within the tool with a microprocessor and battery to perform tracking. The microprocessor should communicate wirelessly with the main computer displaying the GUI.
- ArUco tracking should be used for visual tracking, as this is simple and does not require stereoscopic cameras. A high resolution camera/webcam is preferred.

Group Responsibilities

- The group is responsible for performing literature review on the limitations and advantages of IMU’s. They are also responsible for understanding techniques which will help reduce accelerometer and gyroscope drift.
- Must create a block level design of the entire system in order to demonstrate/understand how the system will work as a whole. Every member of the group must partake in this task, as they are responsible for understanding the entire workflow and the choice of components.

Student A Responsibilities

This student is in charge of creating the CAD for the tool, making sure that there is sufficient space inside to fit all of the components. This means that they must coordinate with the other members of the group in order to create the design. It is expected that this student will 3D print the tool in order to get an accurate and robust physical rendition of the tool. This student is expected to work with student B to create the GUI. Additional responsibilities as assigned by the FLC.

Student B Responsibilities

This student is responsible for creating the GUI and the bluetooth framework which will combine the ArUco tracking and the IMU tracking. The student must show the tool in 3D space in regards to orientation and position. Create an algorithm that will switch from ArUco tracking to IMU tracking at the correct time and display an error if the tracking accuracy is too low. Student B will work in conjunction with student A and C to create the GUI. Additional responsibilities as assigned by the FLC.

Student C Responsibilities

This student will be responsible for setting up visual tracking (ArUco) and calibrating it. They are also responsible for helping student D with transmitting information from the IMU via bluetooth with the microcontroller to the main computer running the GUI. Additional responsibilities as assigned by the FLC.

Student D Responsibilities

This student is responsible for setting up the IMU, applying filters and measuring the accuracy of the IMU. This student must report on how long in respect to time the IMU can maintain millimetric accuracy. This student is also responsible for transmitting the IMU data via bluetooth with the microcontroller. Additional responsibilities as assigned by the FLC.

Course Co-requisites

COE318, COE328, COE428, COE528, COE538

To ALL EDP Students

Due to COVID-19 pandemic, in the event University is not open for in-class/in-lab activities during the Winter term, your EDP topic specifications, requirements, implementations, and assessment methods will be adjusted by your FLCs at their discretion.