Autoreason: Iterative Refinement Methodology
Complete reference for the autoreason iterative refinement method, derived from experimental results across subjective writing tasks, competitive programming, and four model tiers. Use this when any output (paper draft, experiment script, analysis, task definition) needs iterative improvement.
Source: NousResearch/autoreason — "Autoreason: When Iterative LLM Refinement Works and Why It Fails"
Strategy Selection Guide
Decision Tree
`
Is the task objectively verifiable (code, math, factual)?
├── YES → Does the model solve it on the first attempt?
│ ├── YES → Use single pass (no refinement needed)
│ └── NO → Use autoreason (structured analysis → reason-informed revision)
│
└── NO (subjective) → What model tier are you using?
├── Weak (Llama 8B, small models)
│ → Single pass. Model too weak for refinement to help.
│ Invest in generation quality, not iteration.
│
├── Mid-tier (Haiku 3.5, Gemini Flash)
│ → Autoreason with stronger judges. This is the sweet spot.
│ Self-refinement DESTROYS weak model outputs — autoreason prevents this.
│
├── Strong (Sonnet 4)
│ → Autoreason for open-ended tasks. Wins 3/5.
│ Critique-and-revise for concrete technical tasks (2/5).
│
└── Frontier (Sonnet 4.6, Opus)
├── Constrained scope? → Autoreason. Wins 2/3 constrained tasks.
└── Unconstrained? → Critique-and-revise or single pass.
Autoreason FAILS on unconstrained frontier tasks (comes last).
`
Strategy Comparison Table
| Strategy | Best For | Avoid When | Compute (per iteration) | ||
| ---------- | ---------- | ------------ | ------------------------ | ||
| Single pass | Frontier models, template tasks, tight budgets | Mid-tier models where quality ceiling is low | 1 call | ||
| Critique-and-revise | Concrete technical requirements (system design, specifications) | Weak models (degrades output), unconstrained subjective tasks | 2 calls | ||
| Autoreason | Mid-tier models, constrained scope, tasks with genuine tradeoffs | Weak models (Llama 8B), frontier + unconstrained | ~6 calls | ||
| Best-of-N | Almost never recommended | Weak models especially — worse than single pass | N calls | ||
| Strategy | Failure Mode | Mechanism | |||
| ---------- | ------------- | ----------- | |||
| Single pass | Quality ceiling | No mechanism to improve beyond first attempt | |||
| Critique-and-revise | Progressive degradation | Model hallucinates problems (sycophancy), scope creeps each pass, never declines to change | |||
| Best-of-N | Random selection | Without good ranking signal, more samples = more mediocre options | |||
| Autoreason (unconstrained) | Synthesis drift | Stronger models produce syntheses so consistently preferred that incumbent never stabilizes | |||
| Role | Input | Output | Key Rule | ||
| ------ | ------- | -------- | ---------- | ||
| Critic | Task + Incumbent A | List of problems | Find problems ONLY. No fixes. No suggestions. | ||
| Author B | Task + A + Critique | Revised version B | Address each criticism. State which problem each change fixes. | ||
| Synthesizer | Task + X + Y (randomized labels) | Synthesis AB | Take strongest elements of each. Not a compromise. | ||
| Judge Panel | Task + A, AB, B (randomized labels + order) | Ranking | Rank best to worst. No authorship stake. | ||
| Parameter | Value | Rationale | |||
| ----------- | ------- | ----------- | |||
| Convergence k | 2 | k=1 premature (94% displaced later). k=2 converges 100%, quality plateaus. k=3 fails 24%, 2x cost, no quality gain. | |||
| Author temperature | 0.7-0.8 | Encourages diverse revisions | |||
| Judge temperature | 0.3 | Encourages consistent evaluation | |||
| In-loop judges | 3 | Balance per-pass cost vs evaluation stability | |||
| Final evaluation judges | 7 | Higher statistical power for final comparison | |||
| Max tokens | 4096 | Standard; 8192 for long-form (papers) | |||
| Judge type | Chain-of-thought | 3x faster convergence on some tasks. Always use. | |||
| Tiebreak | Conservative (incumbent wins) | Prevents false positives — A must be genuinely beaten | |||
| Max passes | 25 (constrained), 50 (remedy) | Safety cap; most converge by pass 10-15 | |||
| Baseline | Prompt | ||||
| ---------- | -------- | ||||
| Conservative | "Make minimal improvements while preserving what works. Do not add new sections or significantly expand scope." | ||||
| Improve this | "Improve this document." (no further guidance) | ||||
| Harsh critic | "Critically evaluate and rewrite, fixing all weaknesses you identify." | ||||
| Critique & revise | Step 1: "Produce a structured critique. List specific weaknesses." Step 2: "Revise to address each criticism." | ||||
| Rank | Points (3 candidates) | ||||
| ------ | ---------------------- | ||||
| 1st | 3 | ||||
| 2nd | 2 | ||||
| 3rd | 1 | ||||
| Model | Autoreason Wins | Autoreason Avg Borda | Best Baseline | Margin | Recommendation |
| ------- | ---------------- | --------------------- | --------------- | -------- | ---------------- |
| Llama 3.1 8B | 1/3 | 23.7 | 25.0 (single) | -1.3 | Skip autoreason. Model too weak for diverse candidates. |
| Gemini 2.0 Flash | 2/3 | 25.0 | 20.0 (single) | +5.0 | Good candidate. Moderate gains. |
| Haiku 3.5 | 3/3 | 42.0 | 33.7 (single) | +8.3 | Best candidate. Perfect scores. Baselines actively destroy quality. |
| Sonnet 4 | 3/5 | 27.8 | 22.4 (C&R) | +5.4 | Good candidate for open tasks. C&R better for technical tasks. |
| Sonnet 4.6 (unconstrained) | 0/1 | 7.0 | 31.0 (C&R) | -24.0 | Do NOT use autoreason without constraints. |
| Sonnet 4.6 (constrained) | 2/3 | 29.0 | 27.0 (improve) | +2.0 | Use only with scope constraints. |
| Author Model | Recommended Judge | Rationale | |||
| ------------- | ------------------ | ----------- | |||
| Llama 8B | Don't use autoreason | Model too weak | |||
| Gemini Flash | Sonnet 4 | Cross-model evaluation works | |||
| Haiku 3.5 | Sonnet 4 | Strong external eval is the mechanism | |||
| Haiku 3.5 | Haiku 3.5 (same) | Still works — tournament structure provides value even without strong judges (20.7 vs 18.3 avg Borda) | |||
| Sonnet 4 | Sonnet 4 (same) | Same-model judges work at this tier | |||
| Sonnet 4.6 | Sonnet 4.6 (same) | Only with scope constraints | |||
| Constraint Type | Example | Why It Works | |||
| ---------------- | --------- | ------------- | |||
| Fixed facts | "Use only these 8 data points, add nothing else" | Bounds information space | |||
| Fixed deliverable | "500-word startup pitch" (not "improve this") | Defines done condition | |||
| Fixed structure | "Exactly 4 sections, each with 3 numbered items" | Prevents structural drift | |||
| Fixed change items | "Address exactly these 3 reviewer concerns" | Bounds modification scope | |||
| Constraint | Why It Fails | What Happens | |||
| ----------- | ------------- | ------------- | |||
| Word count alone | Not a scope constraint | False convergence — rejected for length, not quality | |||
| "Be concise" | Too vague | Ignored after 2-3 passes | |||
| "Be comprehensive" | Anti-constraint | Invites scope creep | |||
| No constraints at all | Unbounded improvement space | Synthesis dominates, no convergence | |||
| Task Type | Autoreason Works? | Why | |||
| ----------- | ------------------- | ----- | |||
| Tasks with genuine tradeoffs (strategy, policy) | Yes | Multiple valid approaches for tournament to select between | |||
| Constrained writing (pitch, memo, postmortem) | Mostly (2/3) | Bounded scope, clear evaluation criteria | |||
| Template-filling (incident postmortem) | No | One correct structure, minimal decision space | |||
| Competitive programming | Yes | Naturally scoped, test suite provides external verification | |||
| Open-ended unconstrained + frontier model | No | Synthesis drift, no convergence | |||
| Failure Mode | Condition | Detection | Evidence | ||
| ------------- | ----------- | ----------- | ---------- | ||
| Self-correction unreliable | No external evaluation signal | Baselines degrade below single pass | Haiku baselines: 16.3 avg vs 33.7 single pass | ||
| Drift / synthesis dominance | Unconstrained scope | A wins <15%, AB dominates | Sonnet 4.6 unconstrained: A wins 12%, AB wins 60%+ | ||
| Overfitting to visible feedback | Shallow revision loop (C&R) | High public/private divergence | C&R overfits 32% on hard code problems | ||
| No convergence | Broken judge pipeline | Parsing failures, <3 valid judges | Mixed panel parser failure: 11+ passes | ||
| Model too weak | Insufficient generation diversity | All candidates look similar | Llama 8B wins only 1/3 tasks | ||
| Failure | Recovery | ||||
| --------- | ---------- | ||||
| No convergence (drift) | Add scope constraints to the task | ||||
| No convergence (broken judges) | Fix parser, ensure 3 valid judges before continuing | ||||
| Quality degrades with iteration | Switch to single pass or add constraints | ||||
| Model too weak | Use a stronger model for generation, keep weak model for cheap roles | ||||
| Overfitting (code) | Use structured analysis step, not just test feedback | ||||
| Method | Calls per Pass | Typical Passes | Total Calls | Relative Cost | |
| -------- | --------------- | ---------------- | ------------- | --------------- | |
| Single pass | 1 | 1 | 1 | 1x | |
| Best-of-N | N | 1 | N | Nx | |
| Critique & revise | 2 | 15 | 30 | 30x | |
| Autoreason (in-loop) | ~6 | 10-15 | 60-90 | 60-90x | |
| Autoreason (with final eval) | ~6 + 7 | 10-15 + 1 | 67-97 | ~80x |
Cost-quality tradeoff: Autoreason uses ~6x more compute per pass and typically runs more passes. This is a real tradeoff. The method trades compute for evaluation quality. On constrained tasks with mid-tier models, this tradeoff is strongly positive. On unconstrained tasks with frontier models, it's negative.
CoT judges reduce cost: 1 CoT judge provides evaluation quality comparable to 3 standard judges, at ~40% cost savings. Always use CoT judges.