Intelligent Personal Trainer / Dance Instructor

2022 COE Engineering Design Project (NMK03)

---

Faculty Lab Coordinator

Naimul Mefraz Khan

Topic Category

Software Systems

Preamble

Recent advancements in machine learning has provided us with the ability to extract semantic information from images and videos. A particular development in this area is pose estimation, where a skeleton and a consequent pose of a person can be easily extracted from a video. This opens up the opportunity to create an intelligent system that can automatically analyze a trainee's performance while following the performance of a trainer on a video. The analysis result can show where the trainee is lagging behind for suggestions to improve performance. This has potential application in exercise, dance, and home-based physical rehabilitation.

Objective

The objective of this project is to create an intelligent system, where a trainee can follow along a performance/exercise routine by a trainer on a video, and receive feedback on the quality of performance as a score.

Partial Specifications

1. Through capturing the trainee's performance on a webcam, the algorithm should be able to automatically extract the skeleton and pose of both the trainee and the trainer.

2. The two skeletons should be compared to create a score metric, and provide feedback to the trainer on the performance quality in real-time/at the end of the performance through the developed score metric.

Suggested Approach

1. Capture performance of trainer through a webcam.
2. Pre-analyze trainer video and save pose/skeleton.
3. Analyze trainee performance in real-time through extraction of pose/skeleton and comparison with the trainers' performance through an algorithm such as Dynamic Time Warping.
4. Generate a score metric, and provide the trainee the score as a feedback. The feedback should show specific frames where the difference was the highest so that the trainee can assess and fix their stance/pose.

Group Responsibilities

1. Study literature on existing approaches for skeleton/pose estimation.
2. Identify best approach for real-time pose estimation (ideally with deep learning)
2. Study required SDKs to develop the system.
3. end-to-end development for a demonstrable product.
4. Test the final prototype for robustness (sensitivity to angles, illumination, speed of performance, accuracy).

Student A Responsibilities

Webcam image capture/optimize speed of capture

Student B Responsibilities

Identifying appropriate skeleton/pose estimation approach

Student C Responsibilities

Identifying accurate scoring approach

Student D Responsibilities

end-to-end integration of different modules

Course Co-requisites

To ALL EDP Students

Due to COVID-19 pandemic, in the event University is not open for in-class/in-lab activities during the Winter term, your EDP topic specifications, requirements, implementations, and assessment methods will be adjusted by your FLCs at their discretion.