form-field-vlm · commercial

Detect and understand interactive form fields from a page image. The Nutrient system returns each field's bounding box, type, text label, and radio-group identity as constrained JSON. The production system can combine a document-specialized VLM with FF-DETR. The Nutrient VLM performs the semantic work: field detection, fine field typing, text labeling, and radio-group linking. FF-DETR is an optional localization component that supplies object boxes only; it does not label fields or link radio buttons.

The runnable weights are commercial and are not downloadable from this repository. This public page is the product specification and scorecard.

• 🎯 Try it: form-field-vlm-demo
• 🏆 Leaderboard: form-field-vlm-leaderboard
• 📊 Benchmark: form-field-vlm-benchmark

Headline result

Models are ranked by Typed F1 at IoU 0.5. A prediction must overlap the gold widget by at least 0.5 IoU and return the correct fine field type. This strict threshold rewards usable, tightly grounded fields; IoU 0.2 is also reported to expose models that find the right area but draw loose boxes.

System	Typed F1 .5 / .2	Box F1 .5 / .2	Box Recall .5 / .2	sec/page
Nutrient Hybrid	0.477 / 0.516	0.594 / 0.671	0.790 / 0.892	3.24
Nutrient VLM	0.406 / 0.478	0.514 / 0.646	0.551 / 0.692	3.04
GPT-5.6 Sol	0.357 / 0.521	0.403 / 0.580	0.364 / 0.524	—

The evaluation contains 100 held-out clean form pages and 701 annotated fields. The Nutrient hybrid achieves 0.477 F1 at IoU 0.5. The Nutrient VLM alone reaches 0.406, ahead of the strongest cloud VLM in this evaluation, GPT-5.6 Sol (0.357). The hybrid also narrows the strict-to-loose Typed F1 gap from 0.072 for the VLM alone to 0.039, evidence that FF-DETR localization converts more approximately correct detections into strict IoU 0.5 matches.

Cells show IoU 0.5 / IoU 0.2; ranking uses only the first Typed F1 value. Typed F1 requires the correct fine field type. Box F1 measures localization while penalizing missed and extra boxes. Box Recall measures field coverage without penalizing extra boxes. IoU 0.2 is diagnostic for approximately correct but loose localization.

Detailed result

Each detail cell is Typed F1 @ IoU 0.5 / Typed F1 @ IoU 0.2. Seconds per page are shown only for models measured in the same local batch environment; provider batch APIs do not provide comparable request latency.

The benchmark is intentionally small for the first release. Treat it as a transparent, reproducible comparison set, not a complete estimate of every form distribution. The leaderboard exposes additional precision, recall, density, return-count, and Box Recall breakdowns.

Cloud systems are labeled with the model generation returned by the evaluated batch: GPT-5.6; Gemini 3.1 Flash-Lite, Gemini 3.5 Flash, and Gemini 3.1 Pro Preview; and Claude 4.5/4.8/5 as applicable. Leaderboard details retain both the resolved Gemini model_version and the submitted gemini-*-latest alias, so future alias changes cannot alter this benchmark snapshot.

Output

[
  {
    "box": [82, 164, 418, 205],
    "type": "text",
    "label": "Account number",
    "group_id": null
  }
]

Coordinates are [x0, y0, x1, y1] on a 0–1000 page grid. Supported types are text, choice_checkbox, choice_radio, choice_select, and signature.

Intended use and limits

• Empty digital forms are the supported release target.
• Multilingual forms and field labels are supported.
• Very dense pages can require adaptive tiling and longer generation budgets.

License and attribution

The Nutrient VLM weights are offered under a commercial Nutrient license. The evaluation set is public and reproducible through nutrientdocs/form-field-vlm-benchmark.

The hybrid uses FF-DETR / CommonForms by jbarrow, Apache-2.0, for object detection and box localization. Benchmark pages derive from CommonForms, CC-BY-4.0.

📩 Get access

form-field-vlm is commercial and its weights are not downloadable here. To evaluate or deploy it on-prem, with documents remaining in your infrastructure, contact Nutrient: nutrient.io/contact-sales.

About the author

This project is maintained and funded by Nutrient - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks.