Experiment Design Patterns

Patterns and best practices distilled from running research experiments at scale with the Hermes agent. These cover experiment infrastructure, evaluation protocols, monitoring, and failure recovery.

Experiment Infrastructure

Directory Structure

Organize experiments with a consistent structure:

`

workspace/

experiments/

run_main.py # Core experiment runner

run_baselines.py # Baseline comparison

run_ablation.py # Ablation studies

strategies.py # Method implementations

config.yaml # Shared configuration

results/

/

/

/

result.json # Final metrics

final_output.md # Final output artifact

history.json # Full trajectory/log

pass_01/ # Per-iteration artifacts (if iterative)

intermediate.md

analysis/

analyze_results.py # Statistical analysis

compute_stats.py # Significance tests

make_charts.py # Visualization

paper/

paper.tex # LaTeX source

fig_*.pdf # Generated figures

`

Script Design Principles

1. Incremental Saving (Crash Recovery)

Every experiment script should save results after each unit of work, and skip already-completed work on restart:

`python

import json, os

from pathlib import Path

def run_experiment(problems, strategies, output_dir):

for problem in problems:

for strategy in strategies:

result_path = Path(output_dir) / problem["id"] / strategy / "result.json"

if result_path.exists():

print(f"Skipping {problem['id']}/{strategy} (already done)")

continue

# Run the experiment

result = execute_strategy(problem, strategy)

# Save immediately

result_path.parent.mkdir(parents=True, exist_ok=True)

with open(result_path, 'w') as f:

json.dump(result, f, indent=2)

`

This pattern makes re-runs safe and efficient. If a process crashes at problem 47/150, restarting skips the first 46.

2. Artifact Preservation

Save all intermediate outputs, not just final results. This enables post-hoc analysis without re-running:

`python

def save_pass_artifacts(output_dir, pass_num, artifacts):

"""Save all artifacts from a single pass of an iterative method."""

pass_dir = Path(output_dir) / f"pass_{pass_num:02d}"

pass_dir.mkdir(parents=True, exist_ok=True)

for name, content in artifacts.items():

with open(pass_dir / f"{name}.md", 'w') as f:

f.write(content)

`

3. Configuration Management

Use YAML configs for reproducibility:

`yaml

config.yaml

model: anthropic/claude-sonnet-4-20250514

author_temperature: 0.8

judge_temperature: 0.3

max_tokens: 4096

num_judges: 3

max_passes: 15

convergence_k: 2

`

`python

import yaml

with open("config.yaml") as f:

config = yaml.safe_load(f)

`

4. Separation of Concerns

Keep generation, evaluation, and visualization in separate scripts:

This lets you re-run evaluation without re-running expensive generation, and regenerate figures without re-running analysis.

Evaluation Protocols

Blind Judge Panels (for Subjective Tasks)

When evaluating subjective outputs (writing, analysis, recommendations), use a blind judge panel:

`python

import random

def run_blind_evaluation(outputs: dict, task_prompt: str, num_judges: int = 7):

"""

Run blind evaluation of multiple method outputs.

Args:

outputs: {"method_name": "output_text", ...}

task_prompt: The original task description

num_judges: Number of independent judge evaluations

"""

rankings = []

for judge_i in range(num_judges):

# Randomize labels and presentation order per judge

methods = list(outputs.keys())

random.shuffle(methods)

labels = {m: chr(65 + i) for i, m in enumerate(methods)} # A, B, C...

# Present to judge with randomized labels

prompt = f"Task: {task_prompt}\n\n"

for method in methods:

prompt += f"--- Proposal {labels[method]} ---\n{outputs[method]}\n\n"

prompt += "Rank all proposals from best to worst. Format: RANKING: [best], [second], [worst]"

ranking = call_judge(prompt)

rankings.append({"labels": labels, "ranking": ranking})

# Aggregate via Borda count

return compute_borda(rankings)

def compute_borda(rankings, n_methods=3):

"""Borda count: 3/2/1 points for 1st/2nd/3rd."""

scores = {}

points = {0: n_methods, 1: n_methods - 1, 2: n_methods - 2} # Adjust for n_methods

for r in rankings:

for position, method in enumerate(r["ranking"]):

scores[method] = scores.get(method, 0) + points.get(position, 0)

return scores

`

Key design decisions:

• - Randomize both labels AND order per judge to prevent position bias
• - Use odd number of judges (3, 5, 7) to break ties
• - Conservative tiebreak: Incumbent/baseline wins ties (prevents false positives)
• - CoT judges match non-CoT quality at ~40% cost (1 CoT judge ≈ 3 standard judges)

Code/Objective Evaluation

For tasks with ground-truth evaluation (code, math, factual):

`python

import subprocess

def evaluate_code(solution: str, test_cases: list, timeout: int = 30):

"""Run code solution against test cases with sandboxed execution."""

results = {"public": [], "private": []}

for test in test_cases:

try:

proc = subprocess.run(

["python3", "-c", solution],

input=test["input"],

capture_output=True,

timeout=timeout,

text=True

)

actual = proc.stdout.strip()

expected = test["expected"].strip()

passed = actual == expected

except subprocess.TimeoutExpired:

passed = False

category = "public" if test.get("public") else "private"

results[category].append(passed)

return {

"public_pass_rate": sum(results["public"]) / max(len(results["public"]), 1),

"private_pass_rate": sum(results["private"]) / max(len(results["private"]), 1),

}

`

Compute-Matched Comparison

Always compare methods at equal compute budget. If your method uses N API calls, baselines get N calls too:

Human Evaluation Design

Many ML/NLP papers require human evaluation, especially for subjective tasks (text generation, summarization, dialogue, creative writing). Poorly designed human evals are a common rejection reason.

#### When Human Evaluation Is Required

#### Annotation Protocol Design

`

Human Evaluation Protocol:

1. Define the evaluation dimensions (fluency, relevance, factual accuracy, etc.)

2. Create annotation guidelines with examples of each score level

3. Run a pilot with 2-3 annotators on 20-30 examples

4. Compute pilot inter-annotator agreement — if low, revise guidelines

5. Run full evaluation

6. Report: annotator count, agreement metrics, compensation, time per item

`

Evaluation dimensions (pick relevant subset):

Pairwise comparison (preferred over absolute scoring — more reliable):

#### Inter-Annotator Agreement

Always report agreement metrics. Without them, reviewers assume your annotations are unreliable.

`python

Krippendorff's alpha (preferred — handles missing data, any scale)

pip install krippendorffs-alpha

import krippendorff

Ratings: rows = annotators, columns = items, values = scores

ratings = [

[3, 4, 1, 2, 5, None, 3], # Annotator 1

[3, 5, 1, 3, 5, 2, 3], # Annotator 2

[4, 4, 2, 2, 4, 2, None], # Annotator 3

]

alpha = krippendorff.alpha(reliability_data=ratings, level_of_measurement="ordinal")

print(f"Krippendorff's alpha: {alpha:.3f}")

Interpretation: >0.80 good, 0.67-0.80 acceptable, <0.67 questionable

`

`python

Cohen's kappa (for exactly 2 annotators, categorical data)

from sklearn.metrics import cohen_kappa_score

annotator_1 = [1, 2, 3, 1, 2, 3, 2]

annotator_2 = [1, 2, 2, 1, 3, 3, 2]

kappa = cohen_kappa_score(annotator_1, annotator_2)

print(f"Cohen's kappa: {kappa:.3f}")

Interpretation: >0.80 excellent, 0.60-0.80 substantial, 0.40-0.60 moderate

`

#### Crowdsourcing Platforms

Ethics requirements:

#### What to Report in the Paper

`

Human Evaluation Section Checklist:

`

Statistical Analysis

Required Tests

Standard Analysis Script

`python

import numpy as np

from scipy import stats

from pathlib import Path

import json

def load_all_results(results_dir):

"""Load all results into a structured format."""

results = {}

for result_file in Path(results_dir).rglob("result.json"):

parts = result_file.relative_to(results_dir).parts

if len(parts) >= 3:

experiment, task, strategy = parts[0], parts[1], parts[2]

data = json.loads(result_file.read_text())

results.setdefault(experiment, {}).setdefault(strategy, {})[task] = data

return results

def pairwise_mcnemar(method_a_results, method_b_results):

"""McNemar's test for paired binary outcomes."""

a_win_b_lose = sum(1 for a, b in zip(method_a_results, method_b_results) if a and not b)

b_win_a_lose = sum(1 for a, b in zip(method_a_results, method_b_results) if b and not a)

n = a_win_b_lose + b_win_a_lose

if n < 25:

# Use exact binomial for small samples

result = stats.binomtest(a_win_b_lose, n, 0.5)

p_value = result.pvalue

else:

# Chi-squared approximation

chi2 = (abs(a_win_b_lose - b_win_a_lose) - 1)**2 / (a_win_b_lose + b_win_a_lose)

p_value = 1 - stats.chi2.cdf(chi2, df=1)

return {

"a_wins": a_win_b_lose,

"b_wins": b_win_a_lose,

"n_discordant": n,

"p_value": p_value,

"significant": p_value < 0.05

}

def bootstrap_ci(data, n_bootstrap=10000, ci=0.95):

"""Bootstrap confidence interval for mean."""

means = []

for _ in range(n_bootstrap):

sample = np.random.choice(data, size=len(data), replace=True)

means.append(np.mean(sample))

lower = np.percentile(means, (1 - ci) / 2 * 100)

upper = np.percentile(means, (1 + ci) / 2 * 100)

return {"mean": np.mean(data), "ci_lower": lower, "ci_upper": upper}

def cohens_h(p1, p2):

"""Cohen's h effect size for two proportions."""

return 2 np.arcsin(np.sqrt(p1)) - 2 np.arcsin(np.sqrt(p2))

`

Reporting Standards

Always include in the paper:

Monitoring (Cron Pattern)

Cron Prompt Template

For each experiment batch, create a monitoring prompt:

`

Check the status of the [EXPERIMENT_NAME] experiment:

1. Process check: ps aux | grep [PROCESS_PATTERN]

2. Log check: tail -30 [LOG_FILE]

3. Results check: ls [RESULT_DIR]/eval/ (or appropriate result location)

4. If results are available:

- Read the result JSON files

- Report metrics in a table (Borda scores, accuracy, etc.)

- Compute key comparisons between methods

5. If all experiments in this batch are complete:

- git add -A && git commit -m "[COMMIT_MESSAGE]" && git push

- Report final summary

6. Key question: [SPECIFIC ANALYTICAL QUESTION]

If nothing has changed since the last check, respond with [SILENT].

`

Monitoring Best Practices

1. Check processes first — don't read results if the experiment is still running and results are incomplete

2. Read the log tail — look for errors, progress indicators, completion messages

3. Count completed vs expected — "45/150 problems done" is more useful than "some results exist"

4. Report in structured tables — always include key metrics in a table

5. Answer the key question — each experiment should have a specific analytical question to answer when done

6. [SILENT] for no-news — suppress notifications when nothing has changed

7. Commit on completion — every completed batch gets committed with a descriptive message

Example Monitoring Report

`

Code Experiments (Haiku 3.5) - COMPLETE

Key finding: Autoreason shows +2pp improvement over single pass, while

best-of-6 collapses due to single-public-test selection issue.

Committed: git commit -m "Add Haiku code results (150 problems, 4 strategies)"

Next: Run significance tests on these results.

`

Failure Recovery

Common Failures and Recovery

Retry Naming Convention

When re-running failed experiments, use a suffix to track rounds:

`

logs/experiment_haiku_0_50.log # Round 1

logs/experiment_haiku_0_50_r2.log # Round 2 (after credit exhaustion)

logs/experiment_haiku_0_50_r3.log # Round 3 (after bug fix)

`

Pre-Flight Checklist

Before launching any experiment batch:

`

Pre-Flight:

`

Task/Benchmark Design

Open-Ended Tasks (Subjective Evaluation)

Design tasks that have clear objectives but subjective quality:

`markdown

Task: [Title]

Context

[Specific scenario with concrete details: company size, constraints, timeline]

Deliverable

[Exact format and structure required]

Requirements

`

Constrained Tasks (for Testing Scope Effects)

Constrained tasks test whether methods respect scope boundaries. Design with:

Do NOT use word count as a scope constraint. Word limits cause false convergence — outputs get rejected for length, not quality. Constrain scope (what to include) not length.

Example: Good vs Bad Constraints

Visualization Best Practices

Setup: SciencePlots + matplotlib

Install SciencePlots for publication-ready defaults:

`bash

pip install SciencePlots matplotlib numpy

`

Option A: SciencePlots styles (recommended — handles most defaults automatically):

`python

import matplotlib.pyplot as plt

import scienceplots # registers the styles

Pick a style:

'science' — clean, serif fonts, suitable for most venues

'science+ieee' — IEEE-style (good for two-column papers)

'science+nature' — Nature-style

Add 'no-latex' if LaTeX is not installed on the machine generating plots

with plt.style.context(['science', 'no-latex']):

fig, ax = plt.subplots(figsize=(3.5, 2.5)) # single-column width

# ... plot ...

fig.savefig('paper/fig_results.pdf', bbox_inches='tight')

`

Option B: Manual rcParams (when you need full control):

`python

import matplotlib.pyplot as plt

plt.rcParams.update({

'font.size': 10,

'font.family': 'serif',

'axes.labelsize': 11,

'axes.titlesize': 11,

'xtick.labelsize': 9,

'ytick.labelsize': 9,

'legend.fontsize': 9,

'figure.figsize': (3.5, 2.5), # single-column default

'figure.dpi': 300,

'savefig.dpi': 300,

'savefig.bbox': 'tight',

'savefig.pad_inches': 0.05,

'axes.linewidth': 0.8,

'lines.linewidth': 1.5,

'lines.markersize': 5,

'axes.grid': True,

'grid.alpha': 0.3,

'grid.linewidth': 0.5,

})

`

Standard Figure Sizes (Two-Column Format)

Colorblind-Safe Palette (Okabe-Ito)

Use this palette for all paper figures. It is distinguishable by people with all common forms of color vision deficiency:

`python

COLORS = {

'blue': '#0072B2',

'orange': '#E69F00',

'green': '#009E73',

'red': '#D55E00',

'purple': '#CC79A7',

'cyan': '#56B4E9',

'yellow': '#F0E442',

'black': '#000000',

}

As a list for cycling:

COLOR_CYCLE = ['#0072B2', '#D55E00', '#009E73', '#E69F00', '#CC79A7', '#56B4E9']

`

Also differentiate lines by marker and linestyle, not just color:

`python

STYLES = [

{'color': '#0072B2', 'marker': 'o', 'linestyle': '-'},

{'color': '#D55E00', 'marker': 's', 'linestyle': '--'},

{'color': '#009E73', 'marker': '^', 'linestyle': '-.'},

{'color': '#E69F00', 'marker': 'D', 'linestyle': ':'},

]

`

Complete Example: Method Comparison Bar Chart

`python

import matplotlib.pyplot as plt

import numpy as np

try:

import scienceplots

style = ['science', 'no-latex']

except ImportError:

style = 'default'

with plt.style.context(style):

methods = ['Single Pass', 'Critique+Revise', 'Best-of-N', 'Ours']

scores = [73.2, 74.1, 68.5, 77.0]

errors = [2.1, 1.8, 3.2, 1.5]

colors = ['#56B4E9', '#E69F00', '#CC79A7', '#0072B2']

fig, ax = plt.subplots(figsize=(3.5, 2.5))

bars = ax.bar(methods, scores, yerr=errors, capsize=3,

color=colors, edgecolor='black', linewidth=0.5)

# Highlight "Ours"

bars[-1].set_edgecolor('#0072B2')

bars[-1].set_linewidth(1.5)

ax.set_ylabel('Pass Rate (%)')

ax.set_ylim(60, 85)

ax.spines['top'].set_visible(False)

ax.spines['right'].set_visible(False)

fig.savefig('paper/fig_comparison.pdf', bbox_inches='tight')

`

Complete Example: Convergence/Trajectory Line Chart

`python

with plt.style.context(style):

fig, ax = plt.subplots(figsize=(3.5, 2.5))

passes = np.arange(1, 16)

ours = [65, 72, 78, 82, 85, 87, 88, 89, 89.5, 90, 90, 90, 90, 90, 90]

baseline = [65, 68, 70, 71, 69, 67, 66, 65, 64, 63, 62, 61, 60, 59, 58]

ax.plot(passes, ours, **STYLES[0], label='Ours', markersize=4)

ax.plot(passes, baseline, **STYLES[1], label='Critique+Revise', markersize=4)

# Mark convergence point

ax.axvline(x=10, color='gray', linestyle=':', alpha=0.5, linewidth=0.8)

ax.annotate('Converged', xy=(10, 90), fontsize=8, ha='center',

xytext=(10, 93), arrowprops=dict(arrowstyle='->', color='gray'))

ax.set_xlabel('Iteration')

ax.set_ylabel('Quality Score')

ax.legend(loc='lower right')

ax.spines['top'].set_visible(False)

ax.spines['right'].set_visible(False)

fig.savefig('paper/fig_trajectory.pdf', bbox_inches='tight')

`

Output Rules

Chart Types for Common Comparisons