HumanSignal / HumanSignal/label-studio

COCO export does not distinguish intentionally empty annotations from unfinished/unlabeled tasks

Open
#9,774 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
28.3k
Forks
3.7k
Avg merge
14h
Merged PRs (30d)
15

Description

**Describe the bug**

COCO export from a partially labelled project includes tasks/images without annotations, but the exported dataset does not distinguish between:

1. tasks that were reviewed and intentionally completed with empty annotations, for example valid negative samples with no objects;
2. tasks that have not been labeled yet and are still waiting for annotation.

This creates a data quality issue. In COCO output, both cases appear as images with no annotations. Downstream training/evaluation code cannot determine whether an image without annotations is a valid negative sample or an unfinished task that should have been excluded.

During export, Label Studio only logs warnings like:

```text
[2026-06-15 08:05:45,394] [label_studio_sdk.converter.converter::convert_to_coco::629] [WARNING] No annotations found for item #4621
[2026-06-15 08:05:45,410] [label_studio_sdk.converter.converter::convert_to_coco::629] [WARNING] No annotations found for item #4624
[2026-06-15 08:05:45,445] [label_studio_sdk.converter.converter::convert_to_coco::629] [WARNING] No annotations found for item #4631
```

The export succeeds, but the resulting COCO dataset may silently contain unfinished/unlabeled images.

**To Reproduce**

Steps to reproduce the behavior:

1. Create a Label Studio project for image object detection / image annotation.
2. Import multiple image tasks.
3. Annotate only part of the project, for example 30–50% of the tasks.
4. Leave the remaining tasks unannotated / waiting for labeling.
5. Run COCO export from CLI:

```bash
label-studio export coco --export-path=./
```

6. Observe warnings like:

```text
[WARNING] No annotations found for item #4621
```

7. Inspect the exported COCO dataset.
8. Images/tasks without annotations are included, but there is no clear indication whether they are intentionally empty completed tasks or unfinished unlabeled tasks.

**Expected behavior**

COCO export should not silently mix unfinished/unlabeled tasks with valid empty-annotation tasks.

Expected behavior should be one of the following:

* Export only completed/annotated/reviewed tasks by default; or
* Provide a CLI option such as `--only-annotated`, `--only-completed`, or `--exclude-unlabeled`; or
* Include metadata/manifest information that distinguishes:

* completed task with empty annotation result;
* skipped/cancelled task;
* unfinished/unlabeled task; or
* Fail the export or return a clear error when unfinished tasks are included in COCO export.

At minimum, the CLI documentation should clearly explain how COCO export handles unfinished tasks and intentionally empty annotations.

Example CLI warning:

```text
[2026-06-15 08:05:45,394] [label_studio_sdk.converter.converter::convert_to_coco::629] [WARNING] No annotations found for item #4621
[2026-06-15 08:05:45,410] [label_studio_sdk.converter.converter::convert_to_coco::629] [WARNING] No annotations found for item #4624
[2026-06-15 08:05:45,445] [label_studio_sdk.converter.converter::convert_to_coco::629] [WARNING] No annotations found for item #4631
[2026-06-15 08:05:45,460] [label_studio_sdk.converter.converter::convert_to_coco::629] [WARNING] No annotations found for item #4634
```

**Environment**

* OS: Linux
* Label Studio Version: `1.13.1`
* Export format: COCO
* Export method: CLI
* Command used:

```bash
label-studio export coco --export-path=./
```

CLI help does not show an obvious option to exclude unlabeled/unfinished tasks from COCO export:

```text
usage: label-studio export ... project_id export_format

optional arguments:
--export-path EXPORT_PATH
--export-serializer-context EXPORT_SERIALIZER_CONTEXT
```

**Additional context**

This is dangerous for ML dataset generation.

In object detection datasets, images with no annotations can be valid negative samples. However, in a partially labeled Label Studio project, images with no annotations can also mean “not labeled yet”.

After COCO export, these two cases are indistinguishable. This can silently pollute training or evaluation data with unfinished tasks.

The warning `No annotations found for item #...` is not enough because the export still succeeds and produces a dataset that looks usable. There should be a safe way to export only finished tasks or to preserve task status in the export output.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.