Interest in Project 8-Refining Zero-Shot Object Segmentation by Combining Vision Foundation Models

[darksapien23151]

Dear Daan, Klaas, and Samet,

I hope you're doing well. My name is Harshitha Manne, and I’m currently a junior at IIIT Kottayam with a strong interest in computer vision and deep learning. I came across the project Refining Zero-Shot Object Segmentation by Combining Vision Foundation Models, and it immediately caught my attention, mainly as my skill set aligns with the project requirements.

I have experience working with PyTorch, deep learning, and computer vision frameworks. Also, I've raised, pulled and closed the following PRs in the repo:

#28594
#28740
#29225
#29229
#28776

Issues raised:
#28689

While I have a good grasp of feature extraction, embeddings, and segmentation models, I am relatively new to DINOv2 and SAM. I’d like to understand how the refinement process is being approached in this project, whether it primarily involves heuristic-based filtering or if there’s potential for an ML-based refinement approach. Also, I’m curious about the benchmarking framework you plan to use.

I’d be thrilled to contribute to this project and would appreciate any guidance in reviewing my prototype and proposal.
Looking forward to hearing your thoughts the earliest!

@adrianboguszewski , can you please help me connect with the mentors

Best,
Harshitha Manne

[adrianboguszewski]

@Daankrol @samet-akcay

[Daankrol]

The idea is to follow the interface/implementation of Visual Prompting in https://github.com/openvinotoolkit/model_api/tree/master
You are free to choose a method for mask refinement. It could be heuristic based, ML based or something new.
You could check out repos such as SAM, DINO, PerSAM or Matcher.
We currently have no benchmarking framework in mind, so you are free to choose a suitable approach.