Project: Machine Learning models for image classification, object detection and segmentation in Virtual Reality


Congratulations on being selected for GSoC 2022! Use this topic for all communications this summer. No private messages should happen. Use the Jitsi link below if you need to have real-time discussions with your mentors.


So all communication with my mentors can be conducted in this thread?

@sunbiz hi, I would like to discuss my project: get feedback, discuss my work plan and any required fixes.

1 Like

One of the things you could do is look over your proposal and link it here. The timeline part becomes your project plan. None of that needs to remain secret. The only changes may be that the UI/UX things won’t need to be addressed and effectively just backend code.

@sunbiz and @pri2si17 probably have more and they’re the leads on this.

1 Like

Then here is my proposal. Any recommendations regarding the VR-domain adaptation are more than velcome

Hi @DaniilOr please let me know what time works for you so that we can schedule a meeting to discuss next steps.

Any time from UTC 12 to UTC 16 are fine for me today and tomorrow works fine for me

Hi @DaniilOr How about every Saturday 5:00 PM EST? If this is fine with you please let me know your email id so that I can send the calendar invite.

1 Like

It works great for me. My email is

@pri2si17 can you tag me here 1 hour before the call (just in case Google Calendar does not notify)?

1 Like

@DaniilOr sent the calendar invite.

@DaniilOr You are joining meeting today?

Sure, I will join it, however, Google Calendar did not match it as “going” (no idea why).

Please, sorry for this inconvenience. But since the date was changed, see you next week, hope now my “going” is visible

1 Like

Hi @DaniilOr can we meet 30 mins later? It will be 9:30 PM UTC. Please let me know if that is possible.

Yes, it is fine. I will join at 9:30 UTC

My tasks:

  1. Search for open dataset for VR medical imaging
  2. Search for ML applications that use VR medical imaging and try to find out what can be learned from these applications (approach? data? common issues?)
  3. Come up with ideas on VR modelling (probably, advanced ones)

Main blockers:

  1. Availability of the data. VR medical imaging is a narrow sphere, so the could be few/no free datasets
  2. Modeling for Segmentation / detection is not that easy, since no only the source, but the target should be ‘adjusted’

My expectations / what help I expect to receive:

  1. Reviewing of my sources
  2. Suggestions of sources

@pri2si17 Hi! I was thinking about what you told during the meeting regarding the distortion. I guess that using it for segmentation is not an issue as well, since here is albumentations.augmentations.functional.optical_distortion, which can be easily applied to the mask as well. I will research if applying it to bboxes is an option

1 Like

OpenCV for re-distortion of X,Y points : Re-distorting a set of points after camera calibration - OpenCV Q&A Forum

And a paper which uses different kinds of bounding boxes:

I believe that using an ellipse is fine (we can suggest that most of the detection cases are in the center of the image, so it will not be significantly distorted)

My tasks:

  1. Find a Medical Images Segmentation Dataset
  2. Build a baseline model (no distortion) and evaluate it
  3. Add distortion and train another model
  4. Compare the performance of models (if the degradation is insignificant, we can assume that the model will work fine).

Main blockers:

  1. We do not know if using distortion to the image and mask is ‘efficient’, so there are no guarantees that anything will work fine

My expectations / what help I expect to receive:

  1. I expect the model to work fine, without any significant degradation

@DaniilOr, where is your repo? I’d like to see some updates there and provide comments on the baseline model training process.