Testing methodology
Every model we run gets the same battery so cloud and local land in one table.
The harness is tests/refusal_eval.py; it talks to the LiteLLM gateway on port 14000
so any route the gateway exposes is testable by name.
Axes
| Axis | Prompt set | Metric | Why |
|---|---|---|---|
| Over-refusal | tests/prompts/benign_borderline.jsonl (XSTest-style: benign requests with scary surface forms) |
refusal rate | The number that drops to ~0 under ablation; the cleanest fingerprint |
| Capability retention | tests/prompts/capability.jsonl (short reasoning, code, factual items with checkable answers) |
accuracy | Measures the capability tax |
| Sycophancy | tests/prompts/sycophancy.jsonl (false-premise questions) |
agreement rate | Rises under ablation; matters for agent judgment |
| Tool calling | sparkagent scripts/bench-tools.sh |
4-case score | Agent fitness |
| Throughput | harness timing | tok/s, latency | Deployment fit |
| Harmful-request compliance (ASR) | not shipped in this repo | rate only | Use HarmBench or a public set when a study needs it; record the rate, store nothing else. Off by default |
Refusal classification
Two-stage: keyword heuristic ("I can't", "I'm not able to", "I won't", etc.) then an
optional judge call to a stock model (--judge spark-qwen38) that labels each response
REFUSE / COMPLY / PARTIAL. Judge disagreements with the heuristic are flagged.
Storage policy
tests/results/*.jsonl: per-prompt verdicts plus a 200-character excerpt. Gitignored.models/tested/<model>-<date>.md: summary table, committed.- No full outputs for anything beyond the benign sets.
Comparing a model to its base
Always run the stock base through the same battery (--compare <route>). The delta,
not the absolute number, is the finding.
Known blind spots
- One prompt set is a weak discriminator; two models scoring equal means the test failed to separate them, not that they are equivalent.
- Thinking models can refuse inside the think block and comply in the answer, or the reverse. Run with thinking on and off.
- Quantisation (AWQ, NVFP4, GGUF) changes refusal behaviour on its own. Record the quant.