Aditya Kuniyil Kattil
The Archive5 min read
researchmlretrievalevaluation

Retrieval Adaptation Pipeline

Config-driven ML evaluation infrastructure for procedural adaptation under disruption.

Status
Research · Purdue CS490
Role
Research project
Stack
Python · NumPy · PyTorch · Transformers · Hugging Face · vLLM
Exhibit A — architecturefig. 01
Retrieval and visual-grounding evaluation architectureProcedural examples pass through retrieval, visual context, and reranking conditions before evaluation across vision-language model families.YouCook2 · final reportsource · procedural datafull benchmark · 6,044 examplesvisual missing-ingredient test · 390 examplestext retrievaldense · cosinevision pipelineframe · contextrerankingBGE text rerankerVLM evaluation · Qwen · Gemma familiesper-method scoring · 2B → 7B+ parameter scalefig. 03 — retrieval augmentation · reranking · VLM evaluation · persisted traces

01Problem

Standard VLM evaluations test what a model knows, not how it adapts when inputs are disrupted or incomplete. Procedural adaptation — substituting for a missing ingredient, working around an unavailable tool step — requires finding and applying analogous procedural knowledge rather than recalling a memorised answer.

The linked study asks how visual context, retrieval of analogous procedural cases, and reranking affect adaptation to missing-ingredient disruptions. Text-only conditions use the same vision-language model without image input, so they serve as controls for visual conditioning.

02Study scope

The source pipeline supports a broader configurable disruption taxonomy. The experiments described in the linked final report focus on synthetic missing-ingredient disruptions in YouCook2 cooking procedures with a current-state image.

The reported comparison varies text or vision input, retrieval, and reranking. It does not establish results across every disruption type supported by the benchmark builder.

03Benchmark construction

The final report identifies YouCook2 as its source dataset. The repository also contains WikiHow ingestion support, but that capability does not establish WikiHow participation in the reported benchmark.

The report lists 6,044 examples: 4,286 training, 893 development, and 865 test. Its visual missing-ingredient subset contains 1,950 training, 418 development, and 390 held-out test examples. The reported visual-study results use the 390-example test subset, not the full test split.

Splits are grouped by source procedure with seed 42. The report describes rule-based substitute templates as the reference-adaptation source. The repository provides a manual gold-subset workflow, but no reviewed dataset or completed curation artifacts are tracked in the inspected snapshot.

04Retrieval pipeline

The inspected implementation stores dense embeddings in a NumPy index and ranks candidates by cosine similarity. The reported visual-study configuration uses BGE embeddings and an optional BGE text cross-encoder reranker; image input is supplied to the generator in vision-conditioned runs.

Retrieved contexts are injected into the VLM prompt as augmentation. The retrieval pipeline logs retrieved candidates and scores to a persisted trace file, enabling post-hoc analysis of retrieval quality independently from generation quality.

05Experiment orchestration

Experiments are defined as YAML manifests specifying dataset split, retrieval method, model, and evaluation metric. The pipeline supports resume — interrupted experiments pick up from the last completed batch using the persisted trace.

Config-driven design means adding a new model or retrieval method requires only a manifest change, not a code change. This was important for comparing methods without diverging codebases across runs.

06Evaluation pipeline

The source pipeline records heuristic checks for constraint violations, feasibility, adaptation quality, helpfulness, and target similarity, with optional LLM-judge evaluation. The final report uses binary LLM-judge success with missing-ingredient guardrails and also discusses heuristic rescoring.

Both modes operate on persisted generation outputs — the evaluation step is decoupled from inference. Generation can be run once and evaluated multiple ways without re-running the model.

07Reproducibility

The source runner saves configuration snapshots, manifests, retrieval traces, generation outputs, and evaluation records, and can resume using existing per-example records. The split builder groups examples by source procedure with a fixed seed.

Generated benchmark rows, split files, retrieval indexes, and run outputs are excluded from the inspected public repository. The dataset counts here are attributed to the linked report; reproducing those counts requires the corresponding input data and generated artifacts.

08Tradeoffs

The benchmark assumes procedural knowledge transfer is the right frame — that adaptation improves by finding relevant procedural analogies. This may not hold for tasks where the adaptation requires reasoning rather than retrieval.

The reported study is limited to YouCook2 cooking procedures and missing-ingredient disruptions. Its findings do not establish performance on other procedural domains or disruption types.

09Limitations

The final report includes a completed six-condition Qwen2-VL-2B comparison and larger-model follow-ups. Some larger-model conditions were limited by memory or serving constraints; the completed configurations do not constitute a complete model-scale comparison.

The LLM judge evaluation introduces its own consistency concerns: judge model scores can vary across runs and are sensitive to prompt wording. Rule-based metrics are more reproducible but have lower coverage of adaptation quality.

10What I learned

Key finding

Benchmark construction is design work — the framing determines what capabilities are visible and what are invisible. The disruption taxonomy is a design choice, not a neutral observation about the task.

Persisting retrieval traces as a first-class output separates retrieval quality from generation quality. Without traces, a generation failure is ambiguous — it could be the model or the retrieval. With traces, both are independently inspectable.