Foundation-Model-Assisted Dataset Bootstrapping from Uncurated Visual Data (Auto-Annotation)

Working MVP

ZAP-IT 0.1.0 is available as an unpublished local research release candidate with a tested batch workflow and bounded single-image HTTP service.

View ZAP-IT on GitHub

Overview

ZAP-IT (Zero-shot Anything Pipeline for Image Tasks) is a working YAML-driven MVP that converts uncurated images and video into segmentation masks, open-vocabulary labels, optional visual-language verification, visualisations, and YOLO annotations. It combines SAM2, CLIP, and an optional BLIP3 verifier. Collections and videos can be processed through the batch CLI, while a deliberately simple single-image HTTP API supports easy onboarding and sequential processing from any client. The MVP was qualified on an 11 GB RTX 2080 Ti using memory-aware model residency, demonstrating that it can operate on a modest dedicated GPU host.

Outcome

Auto-annotated training datasets derived from existing recordings, including YOLO labels, segmentation masks, class assignments and scores, per-object metadata, exact mask RLE, identity-mask PNGs, QC overlays, warnings, timings, and configuration provenance. The batch exporter creates standard YOLO dataset metadata and configurable training/validation splits. Users can then train YOLO or another lightweight model in their preferred training environment. ZAP-IT produces the training dataset; it does not perform the downstream model training.

Service Card

Category	Infrastructure and data-related services
Domain	Computer Vision, Dataset Engineering
Vertical	Cross-domain
Target users	Industry/SMEs with Uncurated Visual Data, Agritech and Environmental Monitoring, Applied AI Teams, R&D and Innovation Units, Public-Sector Inspection/Monitoring
Service type	Pipeline / Local Service
Outputs	YOLO annotations; deterministic identity-mask PNGs; exact per-object mask RLE; class labels and confidence evidence; per-object metadata; QC overlays and bounded debug artifacts; JSON or ZIP API responses; batch image-directory and video outputs.
Requirements	Access rights to the input data and a YAML task specification defining classes, prompts, thresholds, and optional regions of interest. The current local service requires an operator-managed Python and GPU environment with pinned model snapshots, uses one worker and one active inference request, and is intended for trusted local use.
Limitations	The current 0.1.0 MVP is an unpublished local research release candidate, not a production SLA or accuracy guarantee. The HTTP API accepts one image per request; collection and video processing are available through the batch CLI or sequential API calls. Public exposure, multi-tenancy, geometry, and panoptic service outputs are not part of the current candidate.
Availability	Working MVP available now
Licensing	ZAP-IT software is MIT licensed; foundation-model terms apply separately. The current research-only BLIP3 checkpoint will be replaced by an Apache-2.0 BLIP3/XGen-MM variant before broader deployment, followed by renewed technical and licence qualification.
Visual content
Visual caption

Why this matters to SMEs

Many SMEs already possess the raw material for a useful computer-vision system: years of production photographs, inspection footage, field images, drone recordings, laboratory images, or fixed-camera video. What they usually lack is not data but a sufficiently large and consistent set of labels.

Manual annotation is expensive, slow, and difficult to justify before the organisation knows whether a model will create business value. This creates a cold-start problem: training requires labels, but obtaining a budget for labels often requires a convincing model demonstration.

The practical promise

ZAP-IT uses foundation vision models to perform the expensive first annotation pass. Human experts can then focus on quality control, ambiguous cases, and the examples that matter most instead of drawing every box and mask from scratch.

ZAP-IT processes existing recordings; it does not generate synthetic images. The result is an inspectable training dataset that can be used to train YOLO or another smaller task-specific model in the user's preferred environment.

flowchart TD
    subgraph LEVEL1["Level 1 - Zero shot configuration bootstrapping"]
        SAMPLES["Several representative images"] --> VLM["Frontier vision LLM"]
        GOAL["Business goal and class definitions"] --> VLM
        TEMPLATE["ZAP-IT YAML template and allowed schema"] --> VLM
        VLM --> DRAFT["Draft prompts hard negatives thresholds and rules"]
        DRAFT --> HUMAN["Domain expert review and schema validation"]
        HUMAN --> YAML["Versioned ZAP-IT YAML"]
    end
    subgraph LEVEL2["Level 2 - Zero shot dataset bootstrapping"]
        ARCHIVE["Complete uncurated image and video archive"] --> ZAP["ZAP-IT foundation model pipeline"]
        ZAP --> QC["Masks labels scores metadata and QC overlays"]
        QC --> DATASET["YOLO training and validation dataset"]
    end
    subgraph LEVEL3["Level 3 - Task specific model creation"]
        TRAIN["Train and evaluate YOLO or another lightweight model"] --> MODEL["Small fast deployable domain model"]
    end
    YAML --> ZAP
    DATASET --> TRAIN

From an archive to a training dataset

The working process is deliberately straightforward:

Select the recordings. Use an image directory, a video, or an application that sends images to the HTTP endpoint.
Describe the task in YAML. Define classes and prompts, optional regions of interest, mask-generation settings, filtering thresholds, and optional verification rules.
Run the foundation-model pipeline. SAM2 proposes regions, filtering removes unsuitable candidates, CLIP assigns labels, and the optional visual-language verifier can check selected cases.
Inspect the evidence. Review labels, scores, masks, object metadata, warnings, and overlays. Adjust the configuration and repeat when needed.
Export the dataset. The batch exporter writes YOLO annotations, dataset metadata, and a configurable training/validation split.
Train the deployment model. Use the resulting dataset to train YOLO or another lightweight model. Downstream training is intentionally separate from ZAP-IT.

This separation is useful for SMEs. Large foundation models provide broad visual knowledge during dataset creation, while the eventual operational model can be smaller, faster, cheaper, and specialised for one production task.

Two ways to use the MVP

Interface	Best suited for	Current behaviour
Batch CLI	Existing image archives, videos, repeatable dataset production	Processes directories and videos, writes configured visual outputs, and exports YOLO datasets with training/validation splits.
Local HTTP API	Easy onboarding, application integration, notebooks, and custom client workflows	Accepts one image and one API-safe YAML document per request and returns JSON or ZIP. Collections are handled through ordinary sequential requests.

Batch processing

After installing the documented GPU environment and model snapshots:

.venv-gpu/bin/python zap-it-batch.py \
  --config configs/tomato.yaml \
  --input-image-dir /path/to/images \
  --verbose full

The same configuration system supports video input. When YOLO export is enabled, the exporter produces labels together with dataset.yaml and data.yaml, using the configured trainsplit percentage for the training and validation subsets.

Simple HTTP integration

The local service exposes a bounded ZAP-IT-specific endpoint:

POST /v1/completions
multipart: one image + one YAML configuration

An example request is intentionally ordinary:

curl --fail-with-body \
  -H "Authorization: Bearer ${SLAIF_ZAP_IT_API_KEY}" \
  -F image=@sample.jpg \
  -F config=@task.yaml \
  -F verbosity=2 \
  -F response_format=json \
  http://127.0.0.1:${SLAIF_ZAP_IT_PORT}/v1/completions

A dataset client can simply iterate over its files and submit one request at a time. The one-image contract keeps onboarding, failure handling, and integration simple; it is not a restriction on the number of images that can be processed.

A reproducible task description

YAML makes the annotation policy explicit and versionable. A task can specify preprocessing, regions of interest, SAM2 quality thresholds, post-filtering, CLIP labels and prompts, optional verification rules, visualisations, and batch outputs.

The following shortened, valid example is inspired by the real goats2.yaml field configuration. Its prompt ensemble describes the target from several viewpoints, while the negative class records the signs, vegetation, wires, shadows, and shapes that the camera commonly confuses with an animal.

preprocessing:
  roi: "3,1644,5565,1210"
  resize: 1.0

mask_generator:
  points_per_side: 64
  pred_iou_thresh: 0.70
  stability_score_thresh: 0.70
  min_mask_region_area: 30
  crop_n_layers: 2

postsam2processing:
  maxsize: 60000
  max_w: 300
  max_h: 200

clip:
  padding: 40
  labels:
    goat: |
      a white Boer goat with a brown head,
      a goat partially hidden by tall grass,
      a grazing goat seen from a distance,
      a horned goat near an electric fence
    negative: |
      a numbered white field sign,
      tall grass and flowering weeds,
      a fence wire or an unlabelled post,
      a shadow or bright patch shaped like an animal

blip3:
  goat:
    question: "Is there an animal in the image? Yes or no!"
    trueresult: "Yes"
    falseresult: "No"
    newcategory: goat
  "any,0.20":
    question: "Is there an animal in the image? Yes or no!"
    trueresult: "Yes"
    falseresult: "No"
    newcategory: goat

visualization:
  labels: ["goat"]
  sam2:
    - id: sam2-review
      renderer: annotated
  clip:
    - id: clip-review
      renderer: annotated

export_yolo_det:
  labels: "goat"
  trainsplit: 80
  sample_roi: true

Configurations can be stored beside the resulting dataset, reviewed like code, rerun on new recordings, and compared across iterations. The HTTP boundary accepts only algorithmic fields and rejects paths, URLs, model identifiers, devices, commands, credentials, and other host-level controls.

The configuration can also be created zero-shot

A frontier multimodal LLM with vision capabilities can inspect a small representative sample and draft this YAML before any task-specific model or labelled dataset exists. Given the business goal and example images, it can propose:

candidate classes and multiple positive descriptions for each class;
scene-specific hard negatives and likely sources of false positives;
verifier questions for low-confidence candidates;
an initial region of interest, size filters, and quality thresholds;
QC visualisations and the requested YOLO export policy.

The real goats2.yaml illustrates why this matters. Its most valuable content is not the generic SAM2 parameters but the extensive description of what a goat looks like in that particular field and, even more importantly, what the camera repeatedly sees that is not a goat. Domain knowledge becomes an executable and versioned annotation policy.

This is what makes the initial workflow genuinely zero-shot: it does not require a previously trained task-specific detector or an existing labelled corpus. Human review remains important because the domain expert owns class meaning, acceptable errors, privacy, and the final annotation policy. Sensitive images should be shown only to an approved local or external vision model under the organisation's data policy.

Outputs designed for training and review

ZAP-IT produces standard artifacts rather than locking users into a proprietary annotation platform:

YOLO labels and dataset metadata;
configurable training/validation splits in the batch exporter;
segmentation masks and deterministic 16-bit identity-mask PNGs;
exact per-object uncompressed mask RLE;
class assignments, CLIP scores, and available verification answers;
bounding boxes, areas, centroids, and model quality fields;
QC overlays and bounded debug artifacts;
warnings, stage status, timings, and configuration provenance;
structured JSON or ZIP responses from the HTTP API.

The HTTP response levels are monotonic:

Level	Information returned
0	YOLO detections and minimum response metadata
1	Level 0 plus the identity-mask PNG
2	Level 1 plus per-object labels, scores, geometry, and available verification evidence
3	Level 2 plus bounded overlays, exact mask RLE, timings, warnings, and provenance

Human review remains valuable

Foundation-model output is not automatically ground truth. Image domain, camera position, lighting, object size, prompts, and thresholds all affect annotation quality.

ZAP-IT therefore exposes evidence rather than hiding uncertainty. Obvious examples can be accepted in bulk, while low-confidence or disputed examples can be routed to a human reviewer. QC overlays make systematic errors visible, and a revised YAML configuration can be applied reproducibly to the complete collection.

No accuracy shortcut

The MVP reduces the cost of reaching a credible first dataset. High-consequence applications still require representative evaluation and human validation appropriate to their risk.

Where SMEs can apply it

Industrial inspection

Bootstrap datasets for defects, missing components, surface damage, assembly states, or product variants.
Agriculture and environmental monitoring

Label crops, weeds, fruit maturity, pests, vegetation, wildlife, or objects captured by fixed cameras and drones.
Laboratory and research imagery

Prepare candidate masks and labels for repeated visual structures while keeping sensitive material on local infrastructure.
Infrastructure and field inspection

Turn archived inspection images or footage into datasets for damage, object, and condition detection.

The strongest starting points have repeatable visual conditions, classes that domain experts can describe clearly, and enough expertise for targeted spot checks.

Evidence behind the current MVP

ZAP-IT 0.1.0 is packaged as an unpublished local research release candidate. The repository contains the typed in-memory pipeline, batch compatibility layer, FastAPI service, operator scripts, documentation, release-artifact checks, and automated tests.

At the reviewed candidate state:

374 CPU/fake-engine tests passed, with one live GPU test explicitly opt-in;
total measured test coverage was approximately 78%;
GitHub CI covers Python 3.10, 3.11, and 3.12, packaging, installed-wheel smoke tests, documentation integrity, and release-artifact inspection;
CodeQL analysis passed on the current main branch;
the real model path was qualified on a dedicated 11 GB RTX 2080 Ti, including memory-aware sequential model residency.

These results demonstrate a functioning MVP, not an accuracy benchmark or production SLA.

Current operating boundary

The HTTP candidate currently runs on a trusted local GPU host with one worker and one active inference request. Request images, configurations, and results remain in memory; the service does not persist them. The batch interface deliberately writes the dataset outputs selected by the operator.

Public exposure, multi-tenancy, geometry, and panoptic service outputs are outside the current candidate. ZAP-IT software is MIT licensed, while model terms apply separately. The current research-only BLIP3 checkpoint will be replaced with an Apache-2.0 XGen-MM variant and requalified before broader deployment.

Explore and reproduce

The public repository contains installation instructions, example configurations, API documentation, algorithm descriptions, security boundaries, testing instructions, and measured runtime evidence.

Open the ZAP-IT repository Read the API contract Read the installation guide