InstructFollow-15

InstructFollow-15 is an official BenchLocal Bench Pack for constraint-heavy instruction following. It measures whether a model can satisfy practical instructions while respecting exact constraints, exclusions, formats, and edge-case requirements.

A Bench Pack is an installable benchmark package that runs inside the BenchLocal desktop app. BenchLocal provides the shared app experience for provider setup, model selection, sampling controls, run histories, and side-by-side comparison across benchmark packs.

This repository contains the benchmark source: instruction-following scenarios, scoring logic, methodology, a BenchLocal adapter, and a CLI runner for local development.

Run With BenchLocal

1. Download BenchLocal from the latest BenchLocal release.
2. Open BenchLocal and install InstructFollow-15 from the official Bench Pack registry.
3. Add one or more models, select InstructFollow-15, and start a run.

Bench Pack Structure

lib/                    Benchmark core, scoring, and model runtime
benchlocal/             Thin BenchLocal SDK adapter
cli/                    Non-UI runner
benchlocal.pack.json  Static install/discovery manifest
METHODOLOGY.md          Published benchmark methodology

BenchLocal Adapter

• benchlocal/index.ts is the only place that imports @benchlocal/sdk.
• lib/ stays framework-agnostic and is shared by the CLI and BenchLocal.
• benchlocal.pack.json is the canonical Bench Pack metadata manifest used for install, inspection, and runtime metadata.
• Per-pack default sampling belongs on the manifest, not on global host settings.

Development

• BenchLocal build: npm run build:benchlocal
• CLI runner: npm run cli
• Methodology: METHODOLOGY.md

Validation

npm run typecheck
npm run build:benchlocal