ABACUS

A Benchmark for Image Editing Models to Assess Object Category and Counting Under Scene Conservation

Aymen Lassoued, Angelique Loesch, Romaric Audigier
Université Paris-Saclay, CEA, List, F-91120 Palaiseau, France
WACV 2027, Evaluations and Datasets track

_{Asked to add 4 persons and add 3 boats, a leading editor modifies or removes existing objects of the target category (red boxes) instead of only adding new ones. VLM judges and agentic evaluators still mark these edits as successful. ABACUS identifies both as failures.}

Overview

Instruction-based image editors can now add, remove, or change objects from a short command. Producing the exact number of objects requested, while leaving the rest of the scene untouched, is much harder, and the failures grow sharply with scene complexity and requested count. Existing benchmarks miss these failures: similarity metrics are blind to counts, and VLM judges are themselves poor counters.

ABACUS is a benchmark, a judge-free evaluation protocol and a scoring metric for numerical image editing.

• 10,944 instructions over 3,845 COCO images and all 80 categories, covering add, remove and change with target counts from 1 to 15 and easy, medium and hard tiers.
• Verified ground truth. Counts come from COCONut panoptic masks confirmed by two independent detectors, and every sample is checked by two annotators.
• Judge-free protocol. Each object instance receives a verdict (kept, added, removed or transformed) from segmentation, drift-robust matching and two-detector agreement. No VLM counts objects or judges an edit.
• One unforgiving score. Count fidelity, scene consistency and original-category conservation are multiplied, so an edit scores well only when all three hold.
• Validated against humans. 0.82 exact-count agreement with human counts, against 0.55 for a top VLM counter and 0.28 for a single open-vocabulary detector.

Contents

• Benchmark
• Evaluation protocol
• Leaderboard
• Release status
• Evaluation setup
• Code
• Citation
• License

Benchmark

Every instruction is paired with a known target count and an instance-level reference:

1. Ground-truth counts from COCO images with COCONut panoptic annotations.
2. Detector-verified audit. RF-DETR and Co-DETR must both confirm the count, and both must see zero instances after a non-generative inpainting removes the category.
3. Addition plausibility check by a VLM, which never sets a count or scores an edit.
4. Addable capacity, an upper bound on how many new instances would look natural.
5. Instruction generation (for example add K boats) followed by human verification of feasibility and difficulty tier.

Tier	Add	Remove	Change	Total
Easy	1,159	1,146	1,000	3,305
Medium	1,363	1,340	1,128	3,831
Hard	1,374	1,308	1,126	3,808
Total	3,896	3,794	3,254	10,944
Mean K	3.20	2.32	2.30
Max K	15	10	10

Difficulty combines three rank-normalized axes (foreground coverage, target capacity and target-object scale), split at the terciles of their sum.

Evaluation protocol

Addition and removal share one pipeline of four stages; change couples a removal branch and an addition branch.

Stage	What it does
Novelty discovery	Presence-gated SAM 3 segments the category in both images. A geometric matcher with Hungarian assignment separates preserved objects from genuinely new ones and repairs missing panoptic labels.
Clean-image construction	Preserved objects are erased with classical, non-generative inpainting, so only additions remain.
Dual-detector confirmation	RF-DETR and Co-DETR vote independently on every candidate. An instance counts only when both agree.
Ambiguity resolution	The rare unconfirmed crop (6.4% of instances) is super-resolved and checked with a yes or no category question.

A change is credited only in place, where a removed source instance and an added target instance coincide, which rejects delete-here, paint-there edits.

Each edit receives

$$\mathrm{Score} = \mathrm{CF}\cdot\mathrm{SC}\cdot\mathrm{OC}, \qquad \mathrm{CF} = \frac{2\min(d,K)}{d+K}, \qquad \mathrm{OC} = 1-\frac{N_{\mathrm{rem}}}{N_{\mathrm{GT}}},$$

where $d$ is the delivered count, $K$ the requested count, $\mathrm{SC}$ the harmonic mean of a RAFT optical-flow term and MS-SSIM on the background, and $N_{\mathrm{rem}}$ the number of pre-existing instances destroyed by the edit.

Leaderboard

Overall factors are averaged over all 10,944 edits and the Score is their product. Higher is better.

Model	Type	Count fidelity	Scene consistency	Conservation	Score
Seedream 4.0	Proprietary	0.60	0.79	0.66	0.31
Gemini 2.5 Flash Image	Proprietary	0.59	0.77	0.64	0.29
GPT-Image-1	Proprietary	0.56	0.75	0.62	0.26
Qwen-Image-Edit	Open	0.42	0.69	0.55	0.16
FLUX.1 Kontext	Open	0.39	0.67	0.53	0.14
Step1X-Edit	Open	0.34	0.64	0.50	0.11
BAGEL	Open	0.30	0.62	0.47	0.09

Score by operation and difficulty tier, shown as Easy / Medium / Hard:

Model	Add	Remove	Change
Seedream 4.0	0.55 / 0.42 / 0.28	0.46 / 0.31 / 0.20	0.38 / 0.25 / 0.14
Gemini 2.5 Flash Image	0.52 / 0.39 / 0.26	0.45 / 0.30 / 0.18	0.31 / 0.22 / 0.12
GPT-Image-1	0.51 / 0.36 / 0.25	0.38 / 0.24 / 0.17	0.31 / 0.18 / 0.11
Qwen-Image-Edit	0.35 / 0.23 / 0.13	0.26 / 0.18 / 0.08	0.18 / 0.11 / 0.05
FLUX.1 Kontext	0.30 / 0.20 / 0.12	0.24 / 0.14 / 0.08	0.14 / 0.09 / 0.05
Step1X-Edit	0.24 / 0.16 / 0.10	0.18 / 0.12 / 0.07	0.11 / 0.06 / 0.03
BAGEL	0.20 / 0.13 / 0.08	0.14 / 0.08 / 0.05	0.10 / 0.06 / 0.03

The three proprietary models are statistically tied (overlapping 95% bootstrap intervals). Even the strongest editor delivers the exact requested count on fewer than half of the edits.

Agreement with human counts (600 human-annotated edits):

Evaluator	Easy	Medium	Hard	Overall
Grounding-DINO (single detector)	0.37	0.26	0.23	0.28
Qwen3.6-27B, thinking (solo VLM)	0.71	0.54	0.45	0.55
ABACUS (ours)	0.89	0.82	0.76	0.82

Release status

• Project page
• Benchmark construction and evaluation code
• Benchmark dataset on the Hugging Face Hub
• Paper on arXiv

Evaluation setup

Models used by the protocol

Component	Role	Checkpoint
SAM 3	presence-gated segmentation	facebook/sam3
RF-DETR 2X-Large	detector (consensus)	RFDETR2XLarge (rfdetr package)
Co-DETR ViT-L	detector (consensus)	co_dino_5scale_vit_large_coco
Qwen3.6-27B	yes or no category check on rare crops	Qwen/Qwen3.6-27B
Real-ESRGAN	crop super-resolution	RealESRGAN_x4plus
RAFT	optical flow for scene consistency	raft_large

Editing prompt

Every editor receives the benchmark instruction followed by a fixed scene-preservation suffix, so that all models are compared under the same request. For addition:

Add K {category} to the scene. Keep everything else exactly as-is: same background, lighting, colors, composition, and camera angle. Do not alter, move, or remove existing objects. Only add the new element in a realistic, naturally-placed position.

Removal and change use the analogous instruction with a matching preservation suffix.

Data

The benchmark is available on the Hugging Face Hub at Aymen-Lassoued/ABACUS_Bench, with one configuration per operation (add, remove, change) and one split per difficulty tier (easy, medium, hard).

from datasets import load_dataset

bench = load_dataset("Aymen-Lassoued/ABACUS_Bench", "add", split="hard")
sample = bench[0]
print(sample["prompt"], sample["count"], sample["category"])

Each sample provides the source image, its COCO image_id and coco_split, the editing prompt and the requested count. add and remove samples name the target category; change samples name the source_category and target_category.

ABACUS builds on COCO images and COCONut panoptic annotations.

Code

Layout

benchmark/      benchmark construction, from the COCONut audit to the Hub export
  review/       manual review page, its local server and the list of rejected pairs
editing/        editor runners and the conversion of edits to the evaluator layout
evaluation/     judge-free evaluators for add, remove and change, and the SC metric
baselines/      solo VLM counter, VLM judge and EdiVal counting
human_eval/     600-edit human counting set and its annotation page
slurm/          example SLURM array launchers

Setup

pip install -r requirements.txt

All commands below are run from the repository root.

• Co-DETR runs in its own environment (Python 3.7, mmdet 2.x). Clone Co-DETR, install it there and download the co_dino_5scale_vit_large_coco checkpoint. The evaluators call it through CODETR_PYTHON, CODETR_ROOT and CODETR_WEIGHTS.
• Non-local patch-based inpainting (Newson et al.) is the IPOL C++ code. Build it and set IPOL_INPAINT_BIN to the inpaint_image binary.
• Checkpoints go to checkpoints/ (or CHECKPOINTS): rf-detr-xxlarge.pth, RealESRGAN_x4plus.pth and the Co-DETR checkpoint. SAM 3 (SAM3_MODEL) and Qwen3.6-27B (VLM_MODEL) are loaded from the Hugging Face Hub.
• Data goes to data/ (or DATA_ROOT): COCO 2017 images and annotations under data/coco/, and the COCONut panoptic masks.

huggingface-cli download xdeng77/coconut_s --repo-type dataset --local-dir data/coconut_s
huggingface-cli download xdeng77/relabeled_coco_val --repo-type dataset --local-dir data/relabeled_coco_val

Benchmark construction

The released benchmark is on the Hub, so these steps are only needed to rebuild it.

export COCO_DIR=data/coco/images/train2017
python benchmark/audit_coconut.py all              # detector-verified counts
python benchmark/vlm_feasibility.py --input data/coconut_train_audit/pass_train.csv \
       --output data/vlm_feasibility.csv --model-path Qwen/Qwen3.6-27B --rank 0 --world-size 1
python benchmark/merge_ranks.py --output data/vlm_feasibility.csv --world-size 1
python benchmark/vlm_capacity.py --input data/vlm_feasibility.csv \
       --output data/vlm_count_full.csv --model-path Qwen/Qwen3.6-27B --rank 0 --world-size 1
python benchmark/merge_ranks.py --output data/vlm_count_full.csv --world-size 1
python benchmark/make_prompts.py                   # add instructions
python benchmark/compute_features.py               # difficulty score
python benchmark/split_tiers.py                    # tiers and sampling
python benchmark/download_images.py
python benchmark/review/make_viewer.py && python benchmark/review/server.py   # manual review
python benchmark/review/apply_review.py            # -> data/benchmark_add.csv
python benchmark/build_variants.py                 # remove and change
python benchmark/export_hf.py                      # Hub export (dry run)

Editing and evaluation

Each evaluator reads one folder per edit, with before.jpg, after.jpg and a prompt.txt holding the instruction, the category and the requested count.

# 1. Edit (BAGEL shown here; Gemini, Seedream and FLUX.1 Kontext runners are in editing/)
BENCH_CSV=data/benchmark_add.csv BENCH_OUT=data/bagel/add python editing/bagel.py

# 2. Arrange the edits for the evaluator
OP=add BENCH_CSV=data/benchmark_add.csv EDITS=data/bagel/add OUT=data/bagel_add \
    python editing/prepare_eval_input.py

# 3. Evaluate (CF, SC, OC and Score per edit in data/eval/bagel_add/summary.csv)
SAMPLE_SUBDIR=bagel_add python evaluation/evaluate_add.py

Remove and change work the same way with data/benchmark_delete.csv, data/benchmark_change.csv, OP=remove|change and evaluation/evaluate_remove.py or evaluation/evaluate_change.py. On a cluster, slurm/evaluate.slurm runs one shard per array task (SHARD, NSHARD).

Baselines and human evaluation

# Solo VLM counter and VLM judge on the same edits (input written by prepare_eval_input.py)
python baselines/vlm_count.py --input data/vlm_count_input.csv --output data/vlm_solo.csv \
       --model-path Qwen/Qwen3.6-27B
python baselines/vlm_judge.py --input data/vlm_count_input.csv --output data/vlm_judge.csv \
       --model-path Qwen/Qwen3.6-27B
python baselines/merge_results.py --eval-glob 'data/eval/bagel_add*/summary.csv' \
       --vlm-glob 'data/vlm_solo*.csv' --out data/abacus_bagel_eval.csv

# 600-edit human counting set and its annotation page
python human_eval/build_set.py
python human_eval/server.py            # http://localhost:8010/human_eval/human_eval.html

# EdiVal counting on the same 600 edits (edival_count.py runs from a clone of EdiVal)
python baselines/edival_input.py

Citation

If you find ABACUS useful in your research, please cite:

@inproceedings{lassoued2027abacus,
  title         = {{ABACUS}: A Benchmark for Image Editing Models to Assess
                   Object Category and Counting Under Scene Conservation},
  author        = {Lassoued, Aymen and Loesch, Angelique and Audigier, Romaric},
  booktitle     = {Proceedings of the IEEE/CVF Winter Conference on
                   Applications of Computer Vision (WACV)},
  year          = {2027},
  eprint        = {2610.28316},
  archivePrefix = {arXiv}
}

License

The ABACUS annotations, instructions and evaluation code are released under CC BY 4.0. Source images and panoptic labels remain under the licenses of COCO and COCONut. Edited images produced by third-party editors are subject to each provider's terms.

Acknowledgements

Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union nor the European Commission. Neither the European Union nor the granting authority can be held responsible for them. This work was supported under the Horizon Europe Project CustomAI (grant number 101226029) and the FactoryIA supercomputer (funded by the Ile-de-France Regional Council).

ABACUS builds on COCO, COCONut, SAM 3, RF-DETR, Co-DETR, RAFT and Real-ESRGAN. We thank their authors for making them available.