[experiments] Daily Experiment Report — 2026-10-17

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-17

53 experiments across 50 workflows analysed. No PROMOTE or REJECT decisions; 43 are EXTEND and 10 are INCONCLUSIVE. 24 are 🟢 READY on sample size, but most are blocked by missing guardrail or outcome observations.

⚡ Quick Stats

Metric	Value
Active experiments	53
Ready (sample gate)	24
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 43 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 10
Interaction safety holds	0 (only smokecopilot* run two experiments; each pair is balanced, but cell-level diagnostics were not computed)

Reason codes: insufficient_observations 16 · insufficient_samples 14 · guardrail_unsupported 13 · unsupported_multi_variant 10.

📊 All experiments (core decisions, verbatim from experiments analyze)

Experiment	Workflow	Readiness	Usable samples	Core decision	Reason code
prompt_compression	agentperformanceanalyzer	🟢 READY	caveman:80, verbose:68	EXTEND	guardrail_unsupported
sub_agent_strategy	agentpersonaexplorer	🟡 COLLECTING	–	EXTEND	insufficient_observations
sub_agent_strategy	architectureguardian	🟡 COLLECTING	single_agent:17, sub_agents:21	EXTEND	insufficient_samples
audit_decomposition	auditworkflows	🟡 COLLECTING	phased_sub_agents:58, single_agent:44	EXTEND	insufficient_samples
tone_variant	awfailureinvestigator	🟡 COLLECTING	–	EXTEND	insufficient_observations
prompt_style	blogauditor	🟡 COLLECTING	concise:5, detailed:9	EXTEND	insufficient_samples
tone_variant	breakingchangechecker	🟡 COLLECTING	neutral:34, urgent:31	EXTEND	insufficient_samples
prompt_style	cicoach	🟢 READY	concise:47, detailed:64	EXTEND	guardrail_unsupported
output_format	copilotagentanalysis	🟡 COLLECTING	prose:39, ste:4, structured:48	INCONCLUSIVE	unsupported_multi_variant
nlp_prompt_style	copilotprnlpanalysis	🟡 COLLECTING	concise:0, structured:0	EXTEND	insufficient_samples
sub_agent_strategy	dailyagentrxtraceoptimizer	🟢 READY	single_agent:50, sub_agents:49	EXTEND	guardrail_unsupported
detail_level	dailyarchitecturediagram	🟡 COLLECTING	brief:13, comprehensive:8	EXTEND	insufficient_samples
prompt_style	dailyastrostylelitemarkdownspellcheck	🟢 READY	concise:85, detailed:80	EXTEND	guardrail_unsupported
model_size	dailycachestrategyanalyzer	🟡 COLLECTING	gpt-5.3-codex:23, gpt-5.3-codex-spark:18	EXTEND	insufficient_samples
model_size	dailycavemanoptimizer	🟡 COLLECTING	claude-haiku-4.5:79, claude-sonnet-5:17	EXTEND	insufficient_samples
output_format	dailycodemetrics	🟡 COLLECTING	executive_summary:54, full_detail:58, ste:16	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailycommunityattribution	🟢 READY	concise:76, verbose:76	EXTEND	insufficient_observations
output_format	dailycompilerquality	🟢 READY	concise:56, detailed:66, ste:22	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydochealer	🟢 READY	claude-haiku-4.5:68, claude-sonnet-5:21	EXTEND	guardrail_unsupported
model_size	dailydocupdater	🟡 COLLECTING	claude-haiku-4.5:55, claude-sonnet-5:19	EXTEND	insufficient_samples
reasoning_depth	dailyfact	🟢 READY	multi_candidate:47, single_pass:64	EXTEND	guardrail_unsupported
model_size	dailyfunctionnamer	🟡 COLLECTING	–	EXTEND	insufficient_observations
output_format	dailyissuesreport	🟡 COLLECTING	collapsible:67, inline:64, ste:23	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailynews	🟢 READY	concise:67, detailed:60	EXTEND	guardrail_unsupported
remove_redundant_context_v1	dailyrenderingscriptsverifier	🟡 COLLECTING	candidate:0, control:0	EXTEND	insufficient_samples
log_fetch_strategy	dailysafeoutputoptimizer	🟡 COLLECTING	–	EXTEND	insufficient_observations
reasoning_depth	dailysecurityredteam	🟢 READY	iterative:89, single_pass:68	EXTEND	guardrail_unsupported
semgrep_output_format	dailysemgrepscan	🟡 COLLECTING	bullet_list:28, prose:27, structured_sections:22	INCONCLUSIVE	unsupported_multi_variant
timeout_setting	dailysubagentoptimizer	🟡 COLLECTING	default:2, relaxed:1, tight:1	INCONCLUSIVE	unsupported_multi_variant
caveman_mode	dataflowprdiscussiondataset	🟡 COLLECTING	no:8, yes:11	EXTEND	insufficient_samples
output_format	deepreport	🟢 READY	annotated_brief:71, executive_brief:91, full_briefing:71, ste:53	INCONCLUSIVE	unsupported_multi_variant
summary_detail	dependabotcampaign	🟡 COLLECTING	brief:6, detailed:6	EXTEND	insufficient_samples
prompt_style	dependabotgochecker	🟡 COLLECTING	concise:22, detailed:19, step_by_step:20	INCONCLUSIVE	unsupported_multi_variant
prompt_tightening_v1	gofan	🟡 COLLECTING	candidate:0, control:0	EXTEND	insufficient_samples
tool_verbosity	gpclean	🟢 READY	full_bash:68, minimal_toolset:77	EXTEND	guardrail_unsupported
prompt_style	issuearborist	🟢 READY	concise:56, detailed:48	EXTEND	guardrail_unsupported
reasoning_depth	plan	🟡 COLLECTING	baseline:4, deep:2, shallow:1	INCONCLUSIVE	unsupported_multi_variant
remove_redundant_context_v1	prsouschef	🟡 COLLECTING	–	EXTEND	insufficient_observations
sub_agent_strategy	smokeantigravity	🟢 READY	single_agent:99, sub_agents:109	EXTEND	insufficient_observations
caveman	smokecopilot	🟢 READY	no:197, yes:197	EXTEND	insufficient_observations
subagent_model	smokecopilot	🟢 READY	large:187, small:187	EXTEND	insufficient_observations
caveman	smokecopilotaoaiapikey	🟢 READY	no:103, yes:102	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaiapikey	🟢 READY	large:102, small:103	EXTEND	insufficient_observations
caveman	smokecopilotaoaientra	🟢 READY	no:89, yes:89	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaientra	🟢 READY	large:89, small:89	EXTEND	insufficient_observations
sub_agent_strategy	smokecopilotsubagents	🟡 COLLECTING	delegated_sequential:19, inline_strict:18, single_agent_control:22	INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smokegemini	🟢 READY	single_agent:137, sub_agents:152	EXTEND	guardrail_unsupported
sub_agent_decomposition	smokepi	🟢 READY	parallel_sub_agents:42, single_agent:34	EXTEND	insufficient_observations
prompt_style_test	smokeproject	🟡 COLLECTING	–	EXTEND	insufficient_observations
sub_agent_strategy	smoketemporaryid	🟡 COLLECTING	–	EXTEND	insufficient_observations
model_size	testqualitysentinel	🟢 READY	claude-haiku-4.5:84, claude-sonnet-5:96	EXTEND	guardrail_unsupported
tone_style	typist	🟢 READY	conversational:58, formal:46	EXTEND	guardrail_unsupported
prefetch_strategy	weeklyblogpostwriter	🟡 COLLECTING	eager:7, lazy:11	EXTEND	insufficient_samples

⚠️ Imbalanced assignment (chi-square p < 0.05)

copilotagentanalysis/output_format (ste: 4 vs 39/48), dailycodemetrics, dailycompilerquality, dailyissuesreport, deepreport (all ste arms), dailydocupdater/model_size, dailysemgrepscan. Several output_format experiments with an ste arm show skewed allocation, which is worth checking in the assignment logic.

Per-variant outcome charts and duration CIs were not produced this run: they need per-run outcome data from the Actions API, and the decision layer does not use them. No tracking-issue comments or labels were posted (no experiment declares an issue: field).
Run: 39015155327

🔁 Self-Tuning Continuation Plan

Top experiment actions

1. 13 READY experiments are blocked on guardrail_unsupported (e.g. cicoach, dailynews, issuearborist, gpclean, typist). Wire native guardrail metrics into observations.
2. 16 experiments, including all smoke* caveman/subagent_model pairs, have insufficient_observations. Emit primary-metric observations from evals or graders.
3. 10 multi-variant experiments are INCONCLUSIVE (unsupported_multi_variant). Add K≥3 support in core analysis or reduce to two arms. Fix the skewed ste allocation in output_format experiments.

Eval actions

1. Add a grader that emits output_length_chars per run for the output_format experiments (deepreport, dailycompilerquality, dailyissuesreport).
2. Add success/empty-output graders for prompt_style workflows (cicoach, dailynews, issuearborist).
3. Add an eval for sub_agent_strategy workflows (smokegemini, dailyagentrxtraceoptimizer, smokeantigravity).

Decision-pipeline gaps

• Resolve guardrail_unsupported and insufficient_observations metrics (above).
• Promote the factorial interaction chi-square (smokecopilot* caveman × subagent_model) into a normalized core signal.

Generated by 🧪 daily-experiment-report · copilot · auto · 23.7 AIC · ⌖ 1.06 AIC · ⊞ 17K · ◷

• expires on Oct 19, 2026, 11:57 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #69896.