A benchmark for graph-assisted mathematics that tests whether models can construct useful plots and ground their answers in visualization

Amirhossein Dabiriaghdam1, Shayan Vassef*2, Mohammadreza Bakhtiari*3, Yasamin Medghalchi*1, Ilker Hacihaliloglu1, Mesrob Ohannessian2, Lele Wang1, Giuseppe Carenini1

1University of British Columbia    2University of Illinois Chicago    3Stony Brook University

*Equal contribution

1,168 QA pairs
18 Models
3 Regimes
English Persian Languages
Paper Code Dataset
VAMPS positioning relative to prior math benchmarks: agentic tool loop, self-generated plots, auditable traces
Figure 1. VAMPS is the first bilingual Persian–English benchmark with an agentic Desmos tool loop and auditable traces, evaluating whether models can construct useful graphs rather than only read fixed visuals.

Abstract

We introduce VAMPS (Visual-Assisted Mathematical Problem Solving), a benchmark for graph-assisted mathematics. It contains 1,168 multimodal, bilingual multiple-choice pairs from the Iranian University Entrance Exam (Konkour) and human-reviewed synthetic variants, curated so that plotting reveals intersections, extrema, asymptotes, and related structure.

VAMPS tests whether models can construct a useful graph via Desmos and ground their answer in the resulting visualization. Across a diverse set of models, direct analytical solving surprisingly outperforms tool-enabled visual solving, even on problems where plotting is a natural strategy.

Benchmark

218 seed questions from the Iranian University Entrance Exam (Konkour, 2016–2023) are manually verified in Persian and English, then expanded with human-reviewed synthetic variants. Each item is a 4-option MCQ where graphing is a natural solution strategy.

Konkour · English 218 Konkour · Persian 218 Synthetic · English 366 Synthetic · Persian 366

Synthetic variants were generated with Claude Opus 4.7, GPT-5.4, and Gemini 3.1 Pro, then human-reviewed for mathematical validity. Konkour English translations use GPT-5.4.

VAMPS dataset construction pipeline and graph-mediated task families
Figure 2. Konkour seed collection, bilingual verification, synthetic expansion, and representative task families (intersections, inverses, asymptotes, extrema).

Question Samples

Illustrative R2 trajectories from the paper (Claude Opus 4.7 on Konkour English items).

1 / 2

Solving Regimes

Three complementary regimes isolate analytical reasoning, tool-mediated plot construction, and visual interpretation.

R1 Analytical — solve from text/options only; no external tools
R2 Tool-enabled — Desmos desmos_plot with auditable screenshots
R3 Provided-visualization — fixed layered plots; visual evidence only
End-to-end R2 trajectory from prompt to Desmos calls to answer extraction
Figure 3. R2 tool-enabled trajectory: iterative Desmos calls, screenshot inspection, and strict option extraction.

Results

On the Konkour subset, R1 analytical solving consistently beats R2 tool-enabled solving. Claude Opus 4.7 leads across all regimes; GPT-5.4 drops 23+ points from R1 to R2 despite strong symbolic ability.

Performance-cost trade-off under R2 tool-enabled visual solving
Figure 4. R2 accuracy vs. token cost. Higher token usage does not guarantee better tool-enabled performance.

Tools can hurt

GPT-5.4: 89.0% R1 → 66.5% R2 (English). Qwen3-VL 8B: 92.2% → 50.0%.

R3 < R1 for strong models

Gemma4 31B: 97.2% analytical vs. 88.0% with provided visualizations.

Reasoning-to-perception gap

Failures span malformed tool calls, auto-label over-trust, and misread plot windows.

R3 analytical vs provided-visualization probe and visualization level distribution
Figure 5. R3 diagnostic probe: even curated visualizations often fall below the analytical baseline.

Leaderboard

Konkour subset accuracy for all 18 evaluated models. Judge = VLM-as-judge filtered R2 accuracy.

Profile: R1 · R2 · Judge for selected language
# Model Regime Profile R1 Analytical (English)

Citation

@misc{dabiriaghdam2026vamps,
  title={VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark},
  author={Amirhossein Dabiriaghdam and Shayan Vassef and Mohammadreza Bakhtiari and Yasamin Medghalchi and Ilker Hacihaliloglu and Mesrob Ohannessian and Lele Wang and Giuseppe Carenini},
  year={2026},
  eprint={2606.04244},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2606.04244}
}