• Project organisation

• S3 storage design

• Team access control

• Quality enforcement

CVAT is an open-source annotation platform that ML teams use to label images and videos for

• Object detection

• Segmentation

• Tracking

• Classification models

AWS CVAT runs on EC2 for compute, S3 for storage, and IAM for access control across annotation teams. Yobitel CVAT AMI on AWS Marketplace delivers this environment pre-configured, with Docker, Python, and all core dependencies installed. So, teams skip the setup entirely and go straight to annotation work.

Defining label schemas per task causes class names to drift across batches, making exports incompatible with each other. A large task prevents quality tracking because it’s impossible to isolate the failure. Storing raw data on EC2 volumes puts the entire dataset at risk when an instance fails. It allows annotators to modify label schemas and task configurations that they shouldn't touch. Every one of these problems has a direct fix, and every fix is applied before annotation begins.

The guide describes the exact configurations, structures, instance sizing, and access rules. This keeps a large CVAT deployment on AWS running smoothly from the first job to the final export.

1. Build the project hierarchy before you need it

CVAT operates on a 3-tier hierarchy, with each level having a specific role. Using them correctly is the foundation of a scalable annotation pipeline.

• Projects

CVAT projects are more than folders. The task has authority over every label name, annotation type, and attribute used across all tasks. Projects should define label schemas, not tasks at the task level.

When label schemas are defined per task, label drift is inevitable. One task uses "car", another uses "vehicle", and a third uses "Car" with a capital C. Each looks like a minor naming variation. Collectively, they produce 3 incompatible label sets that cannot be merged into a single training dataset without manually correcting hundreds or thousands of annotations.

Before creating the first task, define every class name and attribute in the Project settings. Use lowercase, underscore-separated names consistently. "traffic_light" not "TrafficLight". Add placeholder labels for known edge cases now. Adding a label later is simple. Re-annotating tasks because a label was missing is not.

• Tasks

Each Task should contain a fixed, manageable number of images. The recommended range for static images is 500 to 1,500 per task. Image sequences work best at 200 to 500. Video tasks should contain one continuous sequence per task.

Keeping tasks within these limits determines how granularly you can track progress, isolate quality problems, and reassign work. A task of 10,000 images gives you one progress data point and zero ability to identify which 2,000 images caused a quality failure. Ten tasks of 1,000 images each give you ten data points and full visibility into exactly where problems occurred.

Name every task with the same format: {project-code}-{domain}-{batch-number}-{date}. For example: AVD-urban-night-B04-20250601.

This makes S3 export paths predictable, backup archives self-describing, and progress dashboards readable at a glance.

• Jobs

When a Task is created, CVAT automatically divides it into Jobs based on a segment size you define. A task of 1,000 images with 200 segments results in 5 jobs. A single annotator oversees each and tracks their quality.

Video tasks require a frame overlap of 10 to 20 frames between jobs. Without overlap, objects crossing a job boundary produce broken tracks in the export. Errors in tracking continuity are not visible frame by frame. They only appear when the full track is reviewed across job boundaries. This is also why honeypots are not supported for video tasks. For video, ground truth jobs are the only automated QA mechanism available.

2. Select the EC2 instance for peak workload

The right EC2 instance is sized for your peak annotation workload. Undersizing does not produce an immediate error. It produces latency that accumulates across annotator sessions until jobs time out, exports stall, and productivity drops in ways that are hard to diagnose.

CVAT runs PostgreSQL, Redis, and multiple Docker containers simultaneously. Every active annotator session adds load to the same compute. The compute profile shifts significantly depending on annotation types in use. Standard bounding box annotation places minimal load on the instance. AI-assisted polygon segmentation on video frames requires significantly more computing power. It saturates an undersized instance within hours of active annotation work.

A team of 12 annotators running polygon segmentation on a medical imaging dataset on a t3.large. It will see response times degrade past 2 seconds per interaction within days. Moving to a t3.xlarge resolves the latency immediately. Upgrading to a larger instance costs less than losing one week of productivity on an undersized one.

AI-assisted annotation runs as Nuclio serverless functions and requires dedicated compute headroom. Sharing one instance for both AI inference and annotation degrades CPU performance and slows down every annotator on the team. Segment Anything Model or DEXTR using GPU inference is the right setup.

Attach a dedicated EBS volume for CVAT's PostgreSQL data and Docker volumes, separate from the root volume. Keep the root at 20 to 30 GB and allocate at least 50 GB to the data volume. For a 200,000-image dataset, size the data volume at 200 GB from the start. Expanding mid-project requires a maintenance window that interrupts active annotation work. Keeping volumes separate also means an instance failure or replacement never touches annotation state.

3. Use S3 as the only location for raw data

Every image in your annotation pipeline must be stored in S3, not on the EC2 volume, not on a local machine, and not split across multiple locations. S3 serves as the single data store for all raw annotation data. CVAT connects to it as the annotation layer, reading images directly from S3, writing annotation metadata to its own database, and exporting labelled datasets on demand.

Storing raw images on the EC2 volume ties data to compute. If the instance fails, is replaced, or is resized, the data is at risk. Rebuilding the instance is straightforward. Recovering annotation data from a failed volume is not. Keeping images in S3 means the compute lifecycle and the data lifecycle are completely independent. CVAT reads from S3, writes annotation metadata to its own database, and exports labelled datasets on demand.

Use IAM role authentication

CVAT supports native S3 integration with IAM role authentication. The setup is:

• Create an IAM role with S3 read and write permissions scoped to your specific annotation bucket

• Attach the role to the EC2 instance profile so the instance authenticates automatically with no keys stored anywhere in the application

• Register the bucket in CVAT under Cloud Storage

• Create all tasks by referencing the S3 path directly, so images are never copied to the EC2 volume

The IAM policy for the EC2 instance role:

json

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Action": [

"s3:GetObject",

"s3:PutObject",

"s3:ListBucket",

"s3:DeleteObject"

],

"Resource": [

"arn:aws:s3:::your-annotation-bucket",

"arn:aws:s3:::your-annotation-bucket/*"

]

}

]

}

Static access keys expire, rotate, and leak. The IAM instance profile issues short-lived credentials automatically. The application never handles a key at all.

Organise your S3 bucket to match the pipeline:

• raw/ - organised by project and batch, e.g. raw/autonomous-driving/urban-day-batch-01/

• exports/ - organised by format, e.g. exports/coco-json/ and exports/yolo-txt/

• backups/ - task archives for completed work

The training script points directly at the exports folder when the task export runs. There is no intermediate copying step, no staging locally, and no format conversion between annotation completion and pipeline ingestion.

Automate exports via the CVAT rest API

Do not rely on manual exports in a production annotation operation:

bash #!/bin/bash TASK_ID=$1 CVAT_HOST="http://:8080" TOKEN="" FORMAT="YOLO 1.1"

curl -X GET "\({CVAT_HOST}/api/tasks/\){TASK_ID}/annotations?format=\({FORMAT// /%20}&action=download" -H "Authorisation: Token \){TOKEN}" --output "task_${TASK_ID}_export.zip"

aws s3 cp "task_\({TASK_ID}export.zip" s3://your-annotation-bucket/exports/yolo-txt/task\){TASK_ID}_export.zip

echo "Task ${TASK_ID} exported and uploaded."

Schedule this via AWS EventBridge on a nightly cadence, or trigger it from your MLOps pipeline on task completion events.

Export format for your training framework

CVAT exports completed annotations in over 20 formats. The format you choose determines which training frameworks can consume the data directly, how well it handles complex geometries, and how the files behave at scale. The three formats most teams use, plus one recommended for long-term archiving, are covered below.

For teams running multiple model types on the same dataset, export to Datumaro as the master archive first. Datumaro preserves every annotation geometry, metadata layer, and data provenance attribute without loss. YOLO or COCO outputs are then generated programmatically during the training pipeline preprocessing phase, eliminating any risk of data loss from repeated format conversions.

4. Configure quality control before annotation starts

Manually reviewing every job at production scale creates a bottleneck at the reviewer, delays feedback to annotators, and still misses systematic errors that only appear statistically across a large sample.

Bad annotations lead to model rejection, wasted computation cycles, and annotation work that needs to be repeated. Catching quality problems during annotation costs a fraction of what it costs to catch them after training. CVAT provides 3 automated QA mechanisms. All 3 must be configured before the first annotator assignment.

Ground truth jobs

A Ground Truth job is a separately annotated reference set that CVAT uses to benchmark the quality of regular annotation work. A curated sample of 5 to 10 per cent of task images is sufficient to estimate quality across the full dataset. Annotate this job carefully before releasing regular jobs to annotators. CVAT then compares each completed regular job against the GT job and generates per-annotator quality scores automatically. For video tasks, ground truth jobs are the only automated QA option available.

Honeypots

Honeypots embed GT frames invisibly into regular jobs. CVAT randomly inserts reference frames into each annotator's job, and the annotator cannot tell which frames are being evaluated. After submission, CVAT scores the job by comparing the annotator's work on those hidden frames against the reference labels. This produces an accurate measure of real annotation behaviour. 2 constraints apply: honeypots are only supported for image tasks, not video, and the GT frame set is fixed at task creation. Configure honeypots before the task is published.

Immediate annotator feedback

Once a job is scored, CVAT surfaces the result to the annotator immediately. Set Max Validations per Job to 3 under Quality Control Settings. Annotators correct their own work rather than waiting for a reviewer to return it. Reviewer time is spent on genuinely difficult cases, and annotation accuracy improves faster across the team.

5. Set role boundaries at the organisation level

Set roles at the Organisation level, not per project or task. Annotators see only the jobs assigned to them and have no visibility into other projects or tasks. Quality Analysts access quality analytics and assignment controls without needing CTO-level permissions.

Contractors and external annotators receive the Annotator role without exception. A contractor with Operations Manager access can modify label schemas, change task configurations, and break completed work across the entire project. There are no exceptions to this rule.

6. Mistakes that compound at scale

All of these are common to large annotation teams. Their impact grows with dataset size.

Starting with a production-ready foundation like the Yobitel CVAT AMI on AWS Marketplace removes every infrastructure decision that stands between your team and productive annotation work. The only thing left to do is log in and start labelling.

The Number That Matters Most Is Not 21 Billion

GPT-OSS-20B has 21 billion total parameters. The number that actually governs its inference cost is 3.6 billion, the active parameters per forward pass.

The model uses a Mixture-of-Experts (MoE) architecture with 32 experts and Top-4 routing. For every token, a learned router scores all 32 experts and sends the token to the 4 highest-scoring ones. The other 28 do nothing. You get the knowledge capacity of a 21B model at the compute cost of roughly a 3.6B dense model.

This is why a single 16 GB GPU can run it. A standard dense 20B model in full precision requires 40 to 48 GB. The MoE design, combined with MXFP4 quantisation on the expert weights, collapses the memory footprint to a fraction of what you would expect.

The architecture also explains a counterintuitive finding from the independent peer-reviewed evaluation published on arXiv: gpt-oss-20B consistently outperforms gpt-oss-120B on several benchmarks, including HumanEval and MMLU. At the 20B scale, the routing mechanism achieves better load balancing and more coherent expert specialisation. A higher active-to-total parameter ratio (17.3% vs roughly 4.4% for the 120B) means each active parameter contributes proportionally more signal.

What the Benchmarks Actually Show

Released on August 5, 2025, and trained using reinforcement learning techniques informed by OpenAI's frontier systems, including o3, the model's benchmark results reflect training quality more than raw scale.

On the AIME 2024 and 2025 competition mathematics, it scores 96% and 98.7% respectively, outperforming not just o3-mini but in some cases the proprietary o3 model and its own larger sibling. On MMLU, it reaches approximately 85%, on par with o3-mini. On GPQA (PhD-level science), it scores 71.5%, where the limits of a smaller knowledge store begin to show.

On code generation, HumanEval results place it ahead of the 120B variant. On SWE-Bench Verified (real-world software engineering), the 120B model scores 62.4%, slightly behind DeepSeek R1 in agentic mode at approximately 65.8% and GLM-4.5 at 64.2%.

The model exposes three reasoning modes via the system prompt: Reasoning: low, Reasoning: medium, and Reasoning: high. The benchmark scores above are at high reasoning. In production, medium is the sensible default, trading roughly 5 to 10% accuracy for a meaningful throughput increase.

Production Throughput on Real Hardware

On matched H100 clusters, end-to-end throughput measured 178 tokens per second for the 20B model versus 128 for the 120B. Time to first token is lower, and steady-state throughput under moderate concurrency is better. Per active billion parameters, the 20B model delivers approximately 8.66 tokens per second per billion compared to 0.74 to 0.77 for dense baselines, roughly 11 to 12 times higher efficiency.

Throughput degrades as context grows. At 4K context it is approximately 40% below baseline decode speed, so if your workload makes heavy use of the 128K context window, factor this into your instance sizing.

For production serving, vLLM with continuous batching is the right stack for most teams. Ollama works for local testing but does not support continuous batching and queues concurrent requests, making it unsuitable for more than two or three simultaneous users.

Where It Stands Against the Competition

Against DeepSeek R1, GPT-OSS-20B has a cleaner licence (Apache 2.0, no additional restrictions), better throughput on equivalent hardware, and comparable reasoning performance. DeepSeek edges ahead on complex agentic coding tasks.

Against Qwen3, GPT-OSS-20B is competitive on English-language STEM and reasoning. Qwen3 is the stronger choice for multilingual workloads across 119 languages.

Against Llama 4 Scout, the main gap is multimodal support. Llama has vision capability that GPT-OSS-20B does not. For text-only workloads requiring deep reasoning, GPT-OSS-20B holds its ground.

The honest summary: GPT-OSS-20B occupies a mid-tier position in the broader open-weight landscape, with particular strength in code generation and mathematical reasoning. It is not the highest-performing model available. It is one of the most efficiently deployable, with a clean licence, mature tooling across vLLM, TensorRT-LLM, Ollama, and Hugging Face Transformers, and a production-grade fine-tuning path via LoRA and QAT.

Running It on AWS

Setting up a production vLLM endpoint for GPT-OSS-20B from scratch involves matching CUDA versions, building vLLM with MXFP4 kernel support from a custom wheel index, configuring the harmony response format, and tuning GPU memory utilisation for KV cache headroom. It is not a weekend project.

The team at Yobitel built a pre-configured AMI on AWS Marketplace that handles all of this at the image layer. The REST endpoint on port 8000 is live approximately 200 seconds after the instance reaches Running state, with MXFP4 kernels active and harmony format configured. No pip installs, no CUDA configuration, no manual model loading scripts.

The recommended instance is the g5.12xlarge (four NVIDIA A10G GPUs, 96 GB combined GPU memory). At MXFP4 precision, the model weights occupy approximately 16 GB, leaving substantial headroom for KV cache under concurrent load.

If your use case requires data residency, predictable infrastructure costs, or fine-tuning on proprietary data, it is worth checking out the Yobitel GPT-OSS-20B LLM Inference Server on AWS Marketplace. A trial period is included so you can validate the setup before committing. For architecture advice, compliance configurations, or fine-tuning support, the team at Yobitel can help.