How to extract the gaussians of a specific query object?

[RoyAmoyal]

Hey,

I’d like to use your method with my own custom scene.
How can I run object segmentation on my custom scene and extract the Gaussians of a specific query object?

Thanks for your great work!

[JulietteMarrie]

Hi,

Thank you for your interest in our work! What would be the input for segmentation—visual input (e.g., point prompt, scribbles, mask) or a text query?

• If you already have 2D segmentation masks, you can call ludvig_uplift.py using a configuration file similar to configs/demo_rgb.yaml. Your masks will be loaded and uplifted.
• If you have a visual 2D input (e.g., scribbles), you can predict and uplift SAM masks using ludvig_uplift.py (as in script/seg.sh) with a custom configuration file using the SAM predictor (as in configs/sam_NVOS.yaml). The predictor uplifts your scribbles into 3D, reprojects them into each view, extracts point prompts, and predicts 2D SAM masks.
• If you have textual prompts, you can uplift CLIP and DINOv2 features into 3D (see script/lerf_uplift.sh). You can then take inspiration from script/lerf_eval.sh to segment based on these uplifted features.

Once you obtain the 3D scores, you can define how to process them using your own evaluation protocol (which should subclass evaluation.base.EvaluationBase). You can select the 3D Gaussians by thresholding with e.g., automatic methods such as Otsu or Li, as done in LUDVIG. If you want to remove Gaussians from the scene (e.g., remove all Gaussians belonging to the mask), you can use self.gaussian.prune_points_noopt and self.gaussian.recover_points, which are implemented in our version of the GaussianModel.

I hope this helps! Please feel free to reach out if you need more details on adapting our code to your specific use case.

[RoyAmoyal]

Thanks for your comprehensive response!

I would really appreciate it if you could provide an example of deleting or extracting 3D objects from a custom scene. The most useful approach would likely involve using CLIP with a textual prompt. Ideally, with a clean Gaussian splatting scene like in your demo.

It would take me some time to implement this myself, but I’m guessing you could do it really fast, and it would be super beneficial for a lot of people!

Anyway, great work!

[RoyAmoyal]

Hi,

Thank you for your interest in our work! What would be the input for segmentation—visual input (e.g., point prompt, scribbles, mask) or a text query?

• If you already have 2D segmentation masks, you can call ludvig_uplift.py using a configuration file similar to configs/demo_rgb.yaml. Your masks will be loaded and uplifted.
• If you have a visual 2D input (e.g., scribbles), you can predict and uplift SAM masks using ludvig_uplift.py (as in script/seg.sh) with a custom configuration file using the SAM predictor (as in configs/sam_NVOS.yaml). The predictor uplifts your scribbles into 3D, reprojects them into each view, extracts point prompts, and predicts 2D SAM masks.
• If you have textual prompts, you can uplift CLIP and DINOv2 features into 3D (see script/lerf_uplift.sh). You can then take inspiration from script/lerf_eval.sh to segment based on these uplifted features.

Once you obtain the 3D scores, you can define how to process them using your own evaluation protocol (which should subclass evaluation.base.EvaluationBase). You can select the 3D Gaussians by thresholding with e.g., automatic methods such as Otsu or Li, as done in LUDVIG. If you want to remove Gaussians from the scene (e.g., remove all Gaussians belonging to the mask), you can use self.gaussian.prune_points_noopt and self.gaussian.recover_points, which are implemented in our version of the GaussianModel.

I hope this helps! Please feel free to reach out if you need more details on adapting our code to your specific use case.

Also, where can I get the 3D scores in the code? And use it to prune the gaussians? Thanks