Project: Neural network-based object detection of anatomical structures and medical artifacts in Virtual Reality

Projects Radiology
, ,
#1

Several medical procedures in surgery, or interventional radiology are recorded as videos that are used for review, training and quality monitoring. These videos have at least 3 interesting artifacts - (1.) anatomical structures such as organs, tumors, tissues etc. (2.) medical equipment and (3.) medical information overlayed or described about the patient. It will be immensely helpful for review and search purposes if these can be identified and automatically labeled in the videos.

Parellely, there is a need to scale the apprenticeship model of being in a procedure room. Virtual reality and live video streams seem to have picked up steam in the recent years as being able to provide immersive experience to participate in such procedures. Thus, this project will use deep learning approaches to scale the apprenticeship model of training future providers by doing object detection and then automatic labeling the artifacts of interest.

The following will mean successful completion of the project:

  1. Train a model that can do object detection on Kvasir dataset
  2. Convert the Kvasir video to an immersive experience on VR headset like Google cardboard (or another mobile VR) or Oculus
  3. Implement inference of the object detection model from Step #1 in the VR experience.

Mentors: @judywawira @pri2si17 Skills required: Python or ML.Net and C# programming for Unity SDK

1 Like
#2

Intro tasks for this project:

  1. Build a cross-platform object detection model on the cityscapes dataset that can run on mobile phones
  2. Implement this as an APK (Android mobile app) built through the Gitlab CI
#3
#4

Hello Everyone, I am Shivam Agarwal currently in my 3rd year at BITS Pilani, pursuing integrated course in CS and Economics. LibreHealth is an organization of one of its kind. I hope to contribute and learn from the organization in Gsoc 2021. I have one doubt, are the intro tasks given to gauge the ability of the contributor? (Because the model trained on cityscapes dataset cant be used in transfer learning for the actual project on Kvasir Dataset.) Is this the only intension or am I missing something here?

#5

yes, it is to identify the ability of the contributor, like prereq. The cityscapes is a good example for the kind of architecture that needs to be selected for this project. and being able to deploy it to an app (even better a VR app) is another such prereq.