# Project: Neural network-based object detection of anatomical structures and medical artifacts in Virtual Reality **URL:** \ **Category:** Radiology\ **Tags:** gsoc2021-project, radiology, gsoc2021\ **Created:** [May 17, 2021, 9:35pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159 "2021-05-17T21:35:52Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![r0bby](https://forums.librehealth.io/user_avatar/forums.librehealth.io/r0bby/32/1781_2.png) [@r0bby](https://forums.librehealth.io/u/r0bby)\ **Post date:** [May 17, 2021, 9:35pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/1 "2021-05-17T21:35:52Z")
This is the thread for [Project: Neural network-based object detection of anatomical structures and medical artifacts in Virtual Reality](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/3975) @sshivaditya this is where you will communicate – you may also use our chat. I am considering this Jitsi meet room public if you need video conferencing. ---
**Author:** ![r0bby](https://forums.librehealth.io/user_avatar/forums.librehealth.io/r0bby/32/1781_2.png) [@r0bby](https://forums.librehealth.io/u/r0bby)\ **Post date:** [May 17, 2021, 11:36pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/4 "2021-05-17T23:36:51Z")
---
**Author:** ![sshivaditya](https://forums.librehealth.io/user_avatar/forums.librehealth.io/sshivaditya/32/1630_2.png) [@sshivaditya](https://forums.librehealth.io/u/sshivaditya)\ **Post date:** [May 22, 2021, 7:12am UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/5 "2021-05-22T07:12:04Z")
Thank you for providing this opportunity to work on this project. @judywawira @pri2si17 I have few question regarding the implementation of the model. 1. Since Kvasir Dataset provides images with just class labels is a classification model enough ? 2. Live Inference of model requires a lot of computational power so is it fine if the model runs on a video of GI Endoscopy and the class labels are displayed at the corner of the frame. Is this approach fine ? ---
**Author:** ![pri2si17](https://forums.librehealth.io/user_avatar/forums.librehealth.io/pri2si17/32/1320_2.png) [@pri2si17](https://forums.librehealth.io/u/pri2si17)\ **Post date:** [May 22, 2021, 7:02pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/6 "2021-05-22T19:02:11Z")
Hi @sshivaditya Please fork from below repository and send us your code as MR. [LibreHealth / LibreHealth Radiology / LH Radiology NN VR Detection · GitLab](https://gitlab.com/librehealth/radiology/lh-radiology-nn-vr-detection) ---
**Author:** ![pri2si17](https://forums.librehealth.io/user_avatar/forums.librehealth.io/pri2si17/32/1320_2.png) [@pri2si17](https://forums.librehealth.io/u/pri2si17)\ **Post date:** [May 25, 2021, 5:30pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/7 "2021-05-25T17:30:43Z")
Hi @sshivaditya Please update here the docs as discussed last Saturday. ---
**Author:** ![sshivaditya](https://forums.librehealth.io/user_avatar/forums.librehealth.io/sshivaditya/32/1630_2.png) [@sshivaditya](https://forums.librehealth.io/u/sshivaditya)\ **Post date:** [May 25, 2021, 6:04pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/8 "2021-05-25T18:04:14Z")
**Observations:** _Game Engine_ Both Unity and Unreal Engine support VR Development. VR applications for Oculus, Steam, Google Cardboard can be designed with both. But Unreal provides support for Python so using external libraries tensorflow can be made to work on the Unreal engine. Unity does not support tensorflow natively as tensorflow C# has been phased out. So libraries like barracuda have to be used which can not be well optimised. _Platform to Develop for_ Developing for Oculus and Steam VR would be better than Google Cardboard because of two reasons. One, Google cardboard API support is being phased out in both unity and unreal and daydream is no longer supported. Secondly, Models get compressed during android build process and the inference doesn’t work properly in unity ---
**Author:** ![sshivaditya](https://forums.librehealth.io/user_avatar/forums.librehealth.io/sshivaditya/32/1630_2.png) [@sshivaditya](https://forums.librehealth.io/u/sshivaditya)\ **Post date:** [June 8, 2021, 3:42am UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/9 "2021-06-08T03:42:50Z")
_ **Model Development** _ _Dataset_ The dataset has labelled-images, unlabelled-images, labelled videos and segmented images. For the classification process I am going to use the labelled-images. They have 22 classes labelled in them. _Model_ I am training two model, out of which one model will be a custom model and other one will be pre-trained one which is adapted to Kvasir Dataset. A size of 224x224 will be used as the input shape for the model. For data logging purpose i have used tensorboard. _Results_ Both the models were trained. The pre-trained model provides low generalisation error in-comparison to the Custom-Model. CustomModel: loss: 2.4398 - accuracy: 0.7283 - f1\_m: 1.1505 - precision\_m: 142484.9571 - recall\_m: 1.1362 - val\_loss: 3.0672 - val\_accuracy: 0.0990 - val\_f1\_m: 0.9958 - val\_precision\_m: 0.9958 - val\_recall\_m: 0.9958 PreTrained Model: loss: 0.9046 - acc: 0.7174 - f1\_m: 0.2450 - precision\_m: 301254.4277 - recall\_m: 0.2275 - val\_loss: 1.0795 - val\_acc: 0.6811 - val\_f1\_m: 0.2366 - val\_precision\_m: 248592.8750 - val\_recall\_m: 0.2270 _ **Virtual Reality** _ For handling virtual reality portion of the project I am going to use WebXR. This doe not require installation of application on the phone. It supports various platforms like oculus and android _ **References** _ [MobileNetV2](https://ai.googleblog.com/2018/04/mobilenetv2-next-generation-of-on.html) [WebXR](https://www.w3.org/TR/webxr/) [CustomModel](https://tensorboard.dev/experiment/MjCRqAZZRwmfZI81TGMHmA/#scalars&run=train) [PreTrained](https://tensorboard.dev/experiment/Ye7eljU0TGCVIPsvmpFoiw/#scalars&run=train) ---
**Author:** ![sshivaditya](https://forums.librehealth.io/user_avatar/forums.librehealth.io/sshivaditya/32/1630_2.png) [@sshivaditya](https://forums.librehealth.io/u/sshivaditya)\ **Post date:** [June 8, 2021, 3:43am UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/10 "2021-06-08T03:43:34Z")
Project Diagram: ![Blank diagram](https://forums.librehealth.io/uploads/default/original/2X/2/249ffa16b0621158be41544aa4764981d7b994d5.png) ---
**Author:** ![pri2si17](https://forums.librehealth.io/user_avatar/forums.librehealth.io/pri2si17/32/1320_2.png) [@pri2si17](https://forums.librehealth.io/u/pri2si17)\ **Post date:** [June 8, 2021, 6:47pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/11 "2021-06-08T18:47:36Z")
@sshivaditya Can you please put the F1 Scores for each class? ---
**Author:** ![sshivaditya](https://forums.librehealth.io/user_avatar/forums.librehealth.io/sshivaditya/32/1630_2.png) [@sshivaditya](https://forums.librehealth.io/u/sshivaditya)\ **Post date:** [June 9, 2021, 6:38pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/12 "2021-06-09T18:38:53Z")
``` precision recall f1-score support 0 0.50 0.10 0.17 10 1 0.00 0.00 0.00 13 2 0.98 0.52 0.68 126 3 0.76 0.99 0.86 222 4 0.87 0.96 0.91 206 5 0.73 0.24 0.37 201 6 0.55 0.91 0.68 215 7 0.29 0.35 0.32 77 8 0.41 0.22 0.29 55 9 0.00 0.00 0.00 1 10 0.89 0.95 0.92 186 11 0.89 0.53 0.67 30 12 0.75 0.87 0.80 202 13 0.91 0.54 0.68 89 14 0.92 0.81 0.86 149 15 0.00 0.00 0.00 6 16 0.29 0.13 0.18 46 17 0.00 0.00 0.00 4 18 0.00 0.00 0.00 90 19 0.02 0.25 0.03 4 20 0.23 0.88 0.36 25 21 0.62 0.68 0.65 175 ``` Net Accuracy Obtained 68% . Each Number represents a class. These score are obtained on the validation set. @pri2si17 ---
**Author:** ![sshivaditya](https://forums.librehealth.io/user_avatar/forums.librehealth.io/sshivaditya/32/1630_2.png) [@sshivaditya](https://forums.librehealth.io/u/sshivaditya)\ **Post date:** [June 10, 2021, 6:23pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/13 "2021-06-10T18:23:56Z")
This week’s [blog post link](https://blog.shivadityas.com/google-summer-of-code-week-1/) @pri2si17 I have sent a MR ---
**Author:** ![pri2si17](https://forums.librehealth.io/user_avatar/forums.librehealth.io/pri2si17/32/1320_2.png) [@pri2si17](https://forums.librehealth.io/u/pri2si17)\ **Post date:** [June 12, 2021, 11:38am UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/14 "2021-06-12T11:38:28Z")
@sshivaditya Can you please put test dataset metrics as that is what matters. Also is this for baseline model or pretrained mobilenet? ---
**Author:** ![sshivaditya](https://forums.librehealth.io/user_avatar/forums.librehealth.io/sshivaditya/32/1630_2.png) [@sshivaditya](https://forums.librehealth.io/u/sshivaditya)\ **Post date:** [June 12, 2021, 2:11pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/15 "2021-06-12T14:11:30Z")
The metrics are of test dataset on which model was not trained. This metric is for pre-trained model. ---
**Author:** ![pri2si17](https://forums.librehealth.io/user_avatar/forums.librehealth.io/pri2si17/32/1320_2.png) [@pri2si17](https://forums.librehealth.io/u/pri2si17)\ **Post date:** [June 13, 2021, 2:31pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/16 "2021-06-13T14:31:38Z")
Well you have written validation set. The model is not good then. 😑 You need to improve it. ---
**Author:** ![sshivaditya](https://forums.librehealth.io/user_avatar/forums.librehealth.io/sshivaditya/32/1630_2.png) [@sshivaditya](https://forums.librehealth.io/u/sshivaditya)\ **Post date:** [June 16, 2021, 4:23pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/17 "2021-06-16T16:23:31Z")
@pri2si17 I trained a new model . I have a BiT pre-trained model. loss: 0.6331 - acc: 0.8733 - f1\_m: 0.8899 - precision\_m: 195757.4552 - recall\_m: 0.9773 - val\_loss: 1.2562 - val\_acc: 0.7955 - val\_f1\_m: 0.9079 - val\_precision\_m: 126641.6953 - val\_recall\_m: 1.0619 ``` precision recall f1-score support 0 0.10 0.80 0.17 10 1 0.00 0.00 0.00 13 2 0.98 0.96 0.97 126 3 0.97 0.97 0.97 222 4 0.98 0.98 0.98 206 5 0.71 0.95 0.81 201 6 0.92 0.71 0.80 215 7 0.37 0.40 0.39 77 8 0.65 0.44 0.52 55 9 0.00 0.00 0.00 1 10 0.78 0.95 0.86 186 11 0.81 0.83 0.82 30 12 0.98 0.99 0.99 202 13 1.00 0.40 0.58 89 14 0.95 0.99 0.97 149 15 0.00 0.00 0.00 6 16 0.40 0.43 0.42 46 17 0.00 0.00 0.00 4 18 0.64 0.30 0.41 90 19 0.00 0.00 0.00 4 20 0.33 0.92 0.48 25 21 0.82 0.56 0.66 175 ``` ```auto accuracy 0.80 2132 macro avg 0.56 0.57 0.54 2132 weighted avg 0.83 0.80 0.80 2132 ``` These scored were obtained when the model was tested on testing set. Testing set was kept seperate and model was not trained on this dataset. Accuracy Obtained 80% [BiT Model](https://tfhub.dev/google/bit/s-r50x1/1) ---
**Author:** ![pri2si17](https://forums.librehealth.io/user_avatar/forums.librehealth.io/pri2si17/32/1320_2.png) [@pri2si17](https://forums.librehealth.io/u/pri2si17)\ **Post date:** [June 17, 2021, 6:17pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/18 "2021-06-17T18:17:31Z")
Hi @sshivaditya well the F1 score doesn’t seems to be good. We can’t rely on accuracy only, you need to look for it. Also, can you try ResNext ( [ResNeXt | Papers With Code](https://paperswithcode.com/lib/torchvision/resnext#:~:text=A%20ResNeXt%20repeats%20a%20building,dimensions%20of%20depth%20and%20width.))? Also any updates on development of VR prototype and segmentation/detection for polyps class? Like with the current model itself (though the performance won’t be good, but it will be helpful for the prototype.) ---
**Author:** ![sshivaditya](https://forums.librehealth.io/user_avatar/forums.librehealth.io/sshivaditya/32/1630_2.png) [@sshivaditya](https://forums.librehealth.io/u/sshivaditya)\ **Post date:** [June 18, 2021, 5:22pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/19 "2021-06-18T17:22:15Z")
@pri2si17 Sure will try with ResNext For implementing the tensorflowJS pipeline, it took some time as some of the layers used are not supported in tensorflowJS. For the VR portion , the basic WebXR setup is ready. This week’s[blog link](https://blog.shivadityas.com/google-summer-of-code-week-1-2/) ---
**Author:** ![sunbiz](https://forums.librehealth.io/user_avatar/forums.librehealth.io/sunbiz/32/395_2.png) [@sunbiz](https://forums.librehealth.io/u/sunbiz)\ **Post date:** [June 18, 2021, 8:33pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/20 "2021-06-18T20:33:12Z")
> [@pri2si17](#): > > We can’t rely on accuracy only, you need to look for it. Also, can you try ResNext ( [ResNeXt | Papers With Code](https://paperswithcode.com/lib/torchvision/resnext#:~:text=A%20ResNeXt%20repeats%20a%20building,dimensions%20of%20depth%20and%20width.))? I agree with this. Even if I am not mentoring this project, my suggestion would be to get the end-to-end working first. i.e. get the Unreal engine-based VR working with any model, even if it doesn’t have good accuracy/f1 scores. The WebXR support is still flaky and slow on most devices, so I’d like to see the program work on Oculus or another VR headset. ---
**Author:** ![r0bby](https://forums.librehealth.io/user_avatar/forums.librehealth.io/r0bby/32/1781_2.png) [@r0bby](https://forums.librehealth.io/u/r0bby)\ **Post date:** [June 19, 2021, 2:38am UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/21 "2021-06-19T02:38:58Z")
I echo both @sunbiz and @pri2si17, get something basic working. Doesn’t even have to be perfect. We don’t expect production-ready code from GSoC projects since we know you are still learning. We just need you to have something that works and does the bare minimum, the [Minimum viable product (MVP)](https://en.wikipedia.org/wiki/Minimum_viable_product). ---
**Author:** ![pri2si17](https://forums.librehealth.io/user_avatar/forums.librehealth.io/pri2si17/32/1320_2.png) [@pri2si17](https://forums.librehealth.io/u/pri2si17)\ **Post date:** [June 20, 2021, 2:32pm UTC](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159/22 "2021-06-20T14:32:03Z")
@sshivaditya Please follow what @sunbiz suggested. We need a POC ready with your current model. Model can be improved later. [Next page](https://forums.librehealth.io/t/project-neural-network-based-object-detection-of-anatomical-structures-and-medical-artifacts-in-virtual-reality/4159.md?page=2)