Guide to the Segment Anything Model (SAM)

Computer vision is evolving rapidly, and industries like healthcare, insurance, and logistics are beginning to harness its potential to transform everyday workflows. But a major obstacle remains: most industries lack tailored datasets and computer vision models. While many open-source datasets are available, they are often too generalized to meet the standards and specific needs of real-world applications.

This gap creates a significant challenge—developing high-quality, custom AI/ML datasets and models from scratch. Fortunately, recent advances in computer vision have introduced foundational models—pretrained models capable of solving a broad range of tasks without retraining. These out-of-the-box solutions dramatically simplify the process of building, deploying, and scaling AI and CV systems.

Among the most important of these is Meta AI’s Segment Anything Model (SAM), an open-source model developed for research and practical use. In this guide, we explain what SAM is, why it matters, and how to use it for fast and scalable image annotation with Unitlab Annotate—a fully automated data labeling platform.

What is SAM?

SAM, short for Segment Anything Model, is a foundational image segmentation model that identifies specific objects—or all objects—within an image. Unlike traditional models that require retraining or fine-tuning for new tasks, SAM is promptable, meaning you can generate accurate masks simply by providing minimal input such as clicks, bounding boxes, or reference masks:

Developed by Meta AI, SAM was trained on the SA-1B dataset, a massive corpus of 11 million images and 1.1 billion masks—averaging 100 masks per image. This extensive training enables SAM to generalize to new and unseen data out of the box, with no additional training or domain-specific tuning required.

The model is designed for zero-shot transfer, meaning it can handle new tasks without prior exposure. Architecturally, SAM consists of an image encoder based on the Vision Transformer (ViT), paired with a lightweight decoder. This makes it much faster than most traditional segmentation models—often outperforming both machines and humans in terms of labeling speed.

SAM represents a new paradigm in image segmentation: interactive, flexible, and ready to use in practical, real-world scenarios.

Why SAM?

In any AI/ML workflow, dataset preparation—and particularly data annotation—is the most resource-intensive phase. Among all annotation tasks, pixel-perfect segmentation is the most time-consuming and complex.

While SAM is useful for various tasks like synthetic data generation for AI/ML datasets and real-time AR/VR applications, its most immediate and impactful use case is image segmentation.

Here’s why SAM and its training dataset, SA-1B are now considered the gold standard:

SAM is fast

SAM’s efficiency stems from both its lightweight design and zero-shot capability. Once prompted, it can produce accurate masks in seconds, which a human data annotator does not have to modify frequently.

For example, segmenting an image that might take five minutes manually can now be done in just 30 seconds with SAM, as shown in the video:

SAM is both accurate and flexible

With 1.1 billion masks across 11 million images, SA-1B is 6x larger than the next biggest open-source dataset, OpenImages v5. So many images and segmentations can generalize across most domains quite well.

But it's not just the number of images and masks in the dataset—it's the diversity of the dataset. The dataset spans a wide range of domains, objects, people, scenarios, and environments. Additionally, researchers focused on the diversity of cultural, national, and even continental contexts in order to address unconcious bias towards less represented groups. The SA-1B dataset was carefully curated to ensure SAM works well for global use across domains, not just for high-income Western countries across a few number of industries.

Moreover, as mentioned above, SAM accepts several types of prompts: point clicks, bounding boxes, freeform masks, and even text inputs if connected to generative AI, such as ChatGPT. This capability makes SAM adapt easily to different tools, workflows, and user preferences.

No wonder that almost all data annotation platforms some form of support for SAM-based data annotation.

SAM is scalable

SAM’s zero-shot accuracy, flexible input system, and consistency make it highly scalable. Whether you're working with 100 images or 100,000, SAM allows you to annotate faster, with better quality and fewer errors.

On supported platforms, segmentation can even be performed in batches. For instance, Unitlab Annotate offers Batch Auto-Annotation, which lets you label many images simultaneously with SAM:

There’s no setup or configuration needed. You just click—and SAM labels the batch for you.

Demo Project

To see SAM in action, we have created a demo project on Unitlab Annotate. First, create a free account. SAM-powered annotation (called Magic Touch) is free forever, and supports both Image Segmentation and Instance Segmentation.

Download sample fashion images here.

Step 1: Project Setup

After logging in, click Add a Project, select Image as the data type, and choose Image Segmentation as the labeling type:

Upload the downloaded sample images as a folder:

Then assign the annotation task to an annotator (yourself):

Next, enable Automation Workflow by going to Automation > Create Automation:

For this example, choose Fashion Segmentation. You may also add other foundational models (e.g., Person Detection, OCR) if needed in other projects. The default settings for this foundational model are sufficient for this demo:

Name your workflow, start the run, and note that you can create multiple workflows within a project:

Step 2: SAM-powered Labeling

Magic Touch lets you annotate by simply clicking an object. One click is enough for SAM to return an accurate segmentation mask, for which you specify the class:

Alternatively, draw a rectangle to define the area of interest—SAM will segment everything inside it and set the relevant classes:

To speed up work, you can annotate large volumes of data with Batch Auto-Annotation. This often boosts labeling speed by 10x. After that, you can review and fine-tune the results if needed:

Step 3: Dataset Release

Once all 22 images are annotated, go to Release Datasets > Release Dataset to create the first version. Choose the dataset format you need. In real-world projects, your dataset will likely grow over time, with new images and annotation classes added continuously:

Note: under the free plan, all datasets are public by default and accessible by anyone. Private datasets require an active or pro account. You can view the released dataset here:

Conclusion

The Segment Anything Model, trained on one of the most diverse and extensive datasets in computer vision, is redefining what’s possible in image annotation. Its combination of speed, prompt-based interaction, and accuracy makes it a foundational model for the future of AI-assisted labeling.

In this guide, we introduced SAM, explained how it works, and showed why it’s a major breakthrough for annotation tasks. With the hands-on demo in Unitlab Annotate, you’re now equipped to start using SAM to supercharge your own image labeling pipeline.

Frequently Asked Questions (FAQ)

• Is SAM free in Unitlab Annotate? Yes. SAM, branded as Magic Touch in Unitlab Annotate, is completely free for all users—forever. All you need is an account to get started.
• Can I use SAM for Polygons? Yes. Magic Touch currently supports SAM-powered auto-annotation for both Image Polygons and Image Segmentation tasks.

Explore More

Interested in more resources for automating your annotation workflow?

1. 3-step Hybrid Way for Superior Image Labeling
2. AI Model Management in Unitlab Annotate
3. 5 Tips for Auto Labeling

References

1. Akruti Acharya (May 3, 2023). How to use SAM to Automate Data Labeling in Encord. Encord Blog: Link
2. Nikolaj Buhl (Dec 10, 2024). Meta AI's Segment Anything Model (SAM) Explained: The Ultimate Guide. Encord Blog: Link
3. Shohrukh Bekmirzaev (Jan 4, 2024). Data Annotation with Segment Anything Model (SAM). Unitlab Blog: Link