Guide to the Segment Anything Model (SAM)

Computer vision is evolving quickly, and industries such as healthcare, insurance, and logistics are starting to use it to improve everyday workflows. But a major obstacle remains: most industries lack tailored datasets and computer vision models. Many open-source datasets exist, but they’re often too broad to meet the specific needs of real-world applications.

This gap creates a significant challenge—developing high-quality, custom AI/ML datasets and models from scratch. Fortunately, recent advances in computer vision have brought foundational models—pre-trained models that can handle a wide range of tasks without retraining. These out-of-the-box solutions dramatically simplify the process of building, deploying, and scaling AI and CV systems.

One of the most important examples is Meta AI’s Segment Anything Model (SAM), an open-source model built for both research and practical use. This guide explains what SAM is, why it matters, and how to use it for fast, scalable image annotation with Unitlab Annotate.

What is SAM?

SAM, short for Segment Anything Model, is a foundational image segmentation model that can identify specific objects—or all objects—within an image. Unlike traditional models that need retraining or fine-tuning for new tasks, the model is promptable: it can generate accurate masks from minimal input such as clicks, bounding boxes, or reference masks.

Developed by Meta AI, this segmentation model was trained on the SA-1B dataset, which contains 11 million images and 1.1 billion masks—about 100 masks per image. This large-scale training allows SAM to generalize to new, unseen data without extra training or domain-specific tuning.

It is a zero-shot transfer model, meaing it can handle new tasks without prior exposure. Architecturally, SAM consists of an image encoder based on the Vision Transformer (ViT), paired with a lightweight decoder. This makes it faster than most traditional segmentation models and often better than both machines and humans in labeling speed and quality.

The Segment Anything Model represents a new paradigm in image segmentation: interactive, flexible, and ready to use in practical, real-world scenarios.

Why SAM?

In any AI/ML workflow, dataset preparation—and particularly data annotation—is the most resource-intensive phase. Among all annotation tasks, pixel-perfect segmentation is the most time-consuming and complex.

While this foundation model can be used for tasks such as synthetic data generation and real-time AR/VR, its most direct and impactful application is image segmentation

Here’s why Meta AI's model and its training dataset, SA-1B are now considered the gold standard:

SAM is fast

Its efficiency stems from both its lightweight design and zero-shot capability. Once prompted, SAM can generate accurate masks within seconds, requiring little adjustment from human annotators.

For example, an image that might take five minutes to segment manually can now be done in about 30 seconds, as shown in the video.

SAM is both accurate and flexible

SA-1B contains 1.1 billion masks across 11 million images, making it six times larger than the next biggest open-source dataset, OpenImages v5. This scale helps the model generalize well across most domains

But it's not just the number of images and masks in the dataset—it's the diversity of the dataset. The dataset spans a wide range of domains, objects, people, scenarios, and environments.

Additionally, researchers focused on the diversity of cultural, national, and even continental contexts to address unconcious bias towards less represented groups. The carefully curated SA-1B dataset resulted in the model that works well for global use across domains, not just for high-income Western countries across selected industries.

As noted above, this foundational model supports different types of prompts: point clicks, bounding boxes, freeform masks, and even text inputs when linked with generative AI tools like ChatGPT. This capability makes SAM adapt easily to different tools, workflows, and user preferences.

No wonder that almost all data annotation platforms some form of support for SAM-based data annotation.

SAM is scalable

SAM’s zero-shot accuracy, flexible input system, and consistency make it highly scalable. WWhether you have 100 images or 100,000, SAM lets you annotate faster, with higher quality and fewer errors.

On supported platforms, you can even perform segmentation in batches. For instance, Unitlab Annotate offers Batch Auto-Annotation, which lets you label many images simultaneously with SAM:

No configuration needed. You just click—and it labels the batch for you.

Demo Project

To see SAM in action, we have created a demo project on Unitlab Annotate. First, create a free account. SAM-powered annotation (called Magic Touch) is free forever, and supports both Image Segmentation and Instance Segmentation.

Download sample fashion images here.

Step 1: Project Setup

After logging in, click Add a Project, set the data type to Image, and choose Image Segmentation as the labeling type:

Upload the downloaded sample images as a folder:

Then assign the annotation task to an annotator (yourself):

Next, enable Automation Workflow by going to Automation > Create Automation:

For this example, choose Fashion Segmentation. You may also add other foundational models (e.g., Person Detection, OCR) if needed in other projects. The default settings for this foundational model are sufficient for this demo:

Name your workflow, start the run, and note that you can create multiple workflows within a project:

Step 2: SAM-powered Labeling

Magic Touch lets you annotate by simply clicking an object. One click is enough for SAM to return an accurate segmentation mask, for which you specify the class:

Alternatively, draw a box around the area of interest—SAM will segment everything inside it and assign the right classes.

To speed up work, you can annotate large volumes of data with Batch Auto-Annotation. This often boosts labeling speed by 10x. After that, you can review and fine-tune the results if needed:

Step 3: Dataset Release

Once all 22 images are annotated, go to Release Datasets > Release Dataset to create the first version. Choose the dataset format you need. In real-world projects, your dataset will likely grow over time, with new images and annotation classes added constantly:

Note: under the free plan, all datasets are public by default and accessible by anyone. Private datasets require an active or pro account. You can view the released dataset here:

Conclusion

The Segment Anything Model, trained on one of the largest and most diverse datasets in computer vision, is reshaping image annotation. Its speed, prompt-based interaction, and accuracy make it a key model for the future of AI-assisted labeling.

In this guide, we introduced SAM, explained how it works, and showed why it's a breakthrough for annotation tasks. With the hands-on demo in Unitlab Annotate, you’re ready to start using SAM to accelerate your own image labeling pipeline.

Frequently Asked Questions (FAQ)

• Is SAM free in Unitlab Annotate? Yes. SAM, branded as Magic Touch in Unitlab Annotate, is completely free for all users—forever. You just need an account to get started.
• Can I use SAM for Polygons? Yes. Magic Touch currently supports SAM-powered auto-annotation for both Image Polygons and Image Segmentation tasks.

Explore More

Interested in more resources for automating your annotation workflow?

1. 3-step Hybrid Way for Superior Image Labeling
2. AI Model Management in Unitlab Annotate
3. 5 Tips for Auto Labeling

References

1. Nikolaj Buhl (Dec 10, 2024). Meta AI's Segment Anything Model (SAM) Explained: The Ultimate Guide. Encord Blog: Link
2. Shohrukh Bekmirzaev (Jan 4, 2024). Data Annotation with Segment Anything Model (SAM). Unitlab Blog: Link