[experiments] Daily Experiment Report — 2026-07-29

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-07-29

Analysed 47 experiments across 44 workflows in github/gh-aw. 24 are ready for outcome-metric review (balanced, min_samples reached), 23 still need more data, and 6 show a statistically significant assignment imbalance (p < 0.05) worth investigating.

⚡ Quick Stats

Metric	Value
Active experiments	47
Ready for analysis	24
Extend (need more samples)	23
Assignment imbalance flagged (p < 0.05)	6
Recommendations	🟡 EXTEND: majority · ✅ READY: ~half, pending outcome-metric review

Scope note: The CLI's chi-square test checks assignment balance (are variants getting roughly equal traffic?), not outcome-metric significance (success rate / duration by variant). Correlating per-run outcomes for all 44 workflows was out of scope for this pass — see Limitations at the end.

---

⚠️ Flagged: Significant Assignment Imbalance (p < 0.05)

Experiment	Workflow	Variants (n)	p-value	Issue
sub_agent_strategy	agentpersonaexplorer	batch(45), per_scenario(24)	1.11e-02	—
model_size	dailycachestrategyanalyzer	agent(1), gpt-5-codex(8), gpt-5-mini(2), gpt-5.4(25), gpt-5.4-mini(17), small-agent(1)	7.80e-10	—
model_size	dailycavemanoptimizer	agent(1), claude-haiku-4.5(30), claude-sonnet-4.6(22), small-agent(2)	5.10e-09	—
model_size	dailydochealer	agent(2), claude-haiku-4.5(28), claude-sonnet-4.6(24), small-agent(1)	1.01e-08	—
model_size	dailydocupdater	agent(1), claude-haiku-4.5(22), claude-sonnet-4.6(30), small-agent(2)	5.10e-09	—
model_size	dailyfunctionnamer	single variant only	0.00e+00	—

These reflect uneven traffic split (e.g. multi-arm model_size experiments with a dominant variant), not outcome significance — no PROMOTE/ABANDON action is implied.

📊 Full Experiments Table

View all 47 experiments

Experiment	Workflow	Variants (n)	min_samples	Balance p	Recommendation	Issue
prompt_compression	agentperformanceanalyzer	caveman(38), verbose(31)	20	0.4041	✅ READY	#33280
sub_agent_strategy	agentpersonaexplorer	batch(45), per_scenario(24)	20	0.0111	✅ READY	—
sub_agent_strategy	architectureguardian	single_agent(14), sub_agents(16)	20	0.7144	🟡 EXTEND	#39062
audit_decomposition	auditworkflows	phased_sub_agents(13), single_agent(11)	264	0.6853	🟡 EXTEND	#43177
tone_variant	awfailureinvestigator	assertive(75), clinical(86), narrative(70)	20	0.4218	✅ READY	#36105
prompt_style	blogauditor	concise(1), detailed(1)	20	1.0000	🟡 EXTEND	#32603
tone_variant	breakingchangechecker	neutral(10), urgent(6)	20	0.3187	🟡 EXTEND	#42467
prompt_style	cicoach	concise(23), detailed(30)	20	0.3385	✅ READY	#32335
output_format	copilotagentanalysis	prose(22), structured(29)	30	0.3288	🟡 EXTEND	—
sub_agent_strategy	dailyagentrxtraceoptimizer	single_agent(20), sub_agents(22)	20	0.7521	✅ READY	—
detail_level	dailyarchitecturediagram	brief(6), comprehensive(3)	20	0.3187	🟡 EXTEND	#31926
prompt_style	dailyastrostylelitemarkdownspellcheck	concise(47), detailed(40)	20	0.4594	✅ READY	—
model_size	dailycachestrategyanalyzer	agent(1), gpt-5-codex(8), gpt-5-mini(2), gpt-5.4(25), gpt-5.4-mini(17), small-agent(1)	20	0.0000	🟡 EXTEND	—
model_size	dailycavemanoptimizer	agent(1), claude-haiku-4.5(30), claude-sonnet-4.6(22), small-agent(2)	20	0.0000	🟡 EXTEND	—
output_format	dailycodemetrics	executive_summary(34), full_detail(39)	20	0.5658	✅ READY	#1
prompt_style	dailycommunityattribution	concise(42), verbose(43)	20	0.8785	✅ READY	—
output_format	dailycompilerquality	concise(36), detailed(38)	20	0.8017	✅ READY	#32390
model_size	dailydochealer	agent(2), claude-haiku-4.5(28), claude-sonnet-4.6(24), small-agent(1)	20	0.0000	🟡 EXTEND	—
model_size	dailydocupdater	agent(1), claude-haiku-4.5(22), claude-sonnet-4.6(30), small-agent(2)	20	0.0000	🟡 EXTEND	—
reasoning_depth	dailyfact	multi_candidate(22), single_pass(32)	30	0.1698	🟡 EXTEND	#31324
model_size	dailyfunctionnamer	n/a (single variant)	20	0.0000	🟡 EXTEND	—
output_format	dailyissuesreport	collapsible(40), inline(44)	20	0.6662	✅ READY	#30573
prompt_style	dailynews	concise(39), detailed(29)	20	0.2229	✅ READY	#31190
log_fetch_strategy	dailysafeoutputoptimizer	eager(16), lazy(17)	20	0.8386	🟡 EXTEND	—
reasoning_depth	dailysecurityredteam	iterative(47), single_pass(31)	30	0.0665	✅ READY	#31673
semgrep_output_format	dailysemgrepscan	bullet_list(27), prose(26), structured_sections(20)	30	0.6011	🟡 EXTEND	#32795
timeout_setting	dailysubagentoptimizer	default(2), relaxed(1), tight(1)	20	0.7814	🟡 EXTEND	—
caveman_mode	dataflowprdiscussiondataset	no(2), yes(5)	10	0.2557	🟡 EXTEND	#37102
output_format	deepreport	annotated_brief(15), executive_brief(26), full_briefing(20)	20	0.2233	🟡 EXTEND	—
summary_detail	dependabotcampaign	brief(6), detailed(6)	20	1.0000	🟡 EXTEND	—
prompt_style	dependabotgochecker	concise(9), detailed(7), step_by_step(10)	20	0.7672	🟡 EXTEND	—
tool_verbosity	gpclean	full_bash(30), minimal_toolset(36)	20	0.4668	✅ READY	—
prompt_style	issuearborist	concise(13), detailed(7)	20	0.1760	🟡 EXTEND	#30015
reasoning_depth	plan	baseline(3), deep(2), shallow(1)	137	0.5956	🟡 EXTEND	#42941
sub_agent_strategy	smokeantigravity	single_agent(96), sub_agents(107)	20	0.4462	✅ READY	—
caveman	smokecopilot	no(152), yes(151)	20	0.9089	✅ READY	—
subagent_model	smokecopilot	large(141), small(142)	20	0.9077	✅ READY	—
caveman	smokecopilotaoaiapikey	no(75), yes(76)	20	0.8946	✅ READY	—
subagent_model	smokecopilotaoaiapikey	large(76), small(75)	20	0.8946	✅ READY	—
caveman	smokecopilotaoaientra	no(62), yes(62)	20	1.0000	✅ READY	—
subagent_model	smokecopilotaoaientra	large(62), small(62)	20	1.0000	✅ READY	—
sub_agent_strategy	smokecopilotsubagents	delegated_sequential(1), inline_strict(1), single_agent_control(4)	30	0.2129	🟡 EXTEND	#47551
sub_agent_strategy	smokegemini	single_agent(107), sub_agents(127)	20	0.1877	✅ READY	—
sub_agent_decomposition	smokepi	parallel_sub_agents(42), single_agent(34)	20	0.3619	✅ READY	—
model_size	testqualitysentinel	claude-haiku-4.5(211), claude-sonnet-4.6(208)	20	0.8555	✅ READY	—
tone_style	typist	conversational(23), formal(23)	20	1.0000	✅ READY	#34032
prefetch_strategy	weeklyblogpostwriter	eager(2), lazy(5)	20	0.2557	🟡 EXTEND	#38590

✅ Ready for Outcome-Metric Analysis (min_samples reached, balanced)

prompt_compression/agentperformanceanalyzer, sub_agent_strategy/agentpersonaexplorer, tone_variant/awfailureinvestigator, prompt_style/cicoach, sub_agent_strategy/dailyagentrxtraceoptimizer, prompt_style/dailyastrostylelitemarkdownspellcheck, output_format/dailycodemetrics, prompt_style/dailycommunityattribution, output_format/dailycompilerquality, output_format/dailyissuesreport, prompt_style/dailynews, reasoning_depth/dailysecurityredteam, tool_verbosity/gpclean, sub_agent_strategy/smokeantigravity, caveman/smokecopilot, subagent_model/smokecopilot, caveman/smokecopilotaoaiapikey, subagent_model/smokecopilotaoaiapikey, caveman/smokecopilotaoaientra, subagent_model/smokecopilotaoaientra, sub_agent_strategy/smokegemini, sub_agent_decomposition/smokepi, model_size/testqualitysentinel, tone_style/typist

🟡 Needs More Data (below min_samples on ≥1 variant)

sub_agent_strategy/architectureguardian, audit_decomposition/auditworkflows, prompt_style/blogauditor, tone_variant/breakingchangechecker, output_format/copilotagentanalysis, detail_level/dailyarchitecturediagram, model_size/dailycachestrategyanalyzer, model_size/dailycavemanoptimizer, model_size/dailydochealer, model_size/dailydocupdater, reasoning_depth/dailyfact, model_size/dailyfunctionnamer, log_fetch_strategy/dailysafeoutputoptimizer, semgrep_output_format/dailysemgrepscan, timeout_setting/dailysubagentoptimizer, caveman_mode/dataflowprdiscussiondataset, output_format/deepreport, summary_detail/dependabotcampaign, prompt_style/dependabotgochecker, prompt_style/issuearborist, reasoning_depth/plan, sub_agent_strategy/smokecopilotsubagents, prefetch_strategy/weeklyblogpostwriter

⚠️ Limitations of This Pass

• The gh aw CLI extension was unavailable via its normal install path (release-asset download returned 403 Forbidden). It was rebuilt from source for this run only; no repository files were modified as part of this report.
• Outcome-metric statistics (success rate, mean duration, 95% CI, Welch t-test / two-proportion z-test vs. control) were not computed. Correlating the last 30 runs per experiment with variant assignments and downloading run_duration_ms via the GitHub MCP tools for all 44 workflows was out of scope for this run. Only the CLI's chi-square assignment-balance test is reported above.
• No guardrail_metrics: outcome data was evaluated; guardrail pass/fail could not be determined from available inputs.
• Charting libraries (matplotlib/scipy) could not be installed in this sandbox (no network access to PyPI binary wheels), so no bar charts were generated for this report.

Run: 30436674032

Warning

Firewall blocked 12 domains

The following domains were blocked by the firewall during workflow execution:

• charm.land
• cloud.google.com
• go.opentelemetry.io
• go.uber.org
• go.yaml.in
• golang.org
• google.golang.org
• gopkg.in
• goproxy.cn
• goproxy.io
• proxy.golang.org
• releaseassets.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "charm.land"
    - "cloud.google.com"
    - "go.opentelemetry.io"
    - "go.uber.org"
    - "go.yaml.in"
    - "golang.org"
    - "google.golang.org"
    - "gopkg.in"
    - "goproxy.cn"
    - "goproxy.io"
    - "proxy.golang.org"
    - "releaseassets.githubusercontent.com"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · sonnet50 · 187.5 AIC · ⌖ 12.3 AIC · ⊞ 15.8K · ◷

• expires on Aug 1, 2026, 1:01 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-01T09:01:49.116Z.

Closed by Workflow