Software Engineering Empirical Research Radar

Repair 怎么开跑

从 Defects4J checkout 到 LLM feedback loop 的第一套可执行骨架。

第一目标不是一口气修很多 bug，而是让每个步骤都有 receipt：输入是什么、模型看到了什么、补丁是什么、跑了哪些测试、为什么失败、下一轮怎么继续。

目录骨架

workdirs/10-ai-guided-program-repair/
  experiments/
    d4j-smoke-001/
      manifest.csv
      prompts/
      patches/
      receipts/
      logs/
      results.csv
  scripts/
    d4j_checkout.sh
    run_repair_loop.py
    apply_patch_and_test.py
    summarize_failure.py

Defects4J 最小命令

# 1. checkout 一个 buggy version
defects4j checkout -p Lang -v 1b -w /tmp/d4j/Lang-1b

# 2. 编译
cd /tmp/d4j/Lang-1b
defects4j compile

# 3. 导出 triggering tests
defects4j export -p tests.trigger

# 4. 跑全部开发者测试
defects4j test

# 5. 跑单个测试，适合 feedback loop
defects4j test -t org.apache.commons.lang3.math.NumberUtilsTest::testCreateNumber

Repair Loop 伪代码

for bug in manifest:
  checkout_bug(bug)
  collect_context(bug)
  for arm in [vanilla, ochiai, diagnostic_advice]:
    state = build_initial_prompt(bug, arm)
    for round_id in range(1, max_rounds + 1):
      patch = call_llm(state)
      apply_result = apply_patch(patch)
      test_result = run_validation_layers()
      save_receipt(bug, arm, round_id, patch, test_result)
      if test_result.full_tests_pass:
        send_to_correctness_review()
        break
      state = append_failure_summary(state, test_result)

Prompt 模板

Task: repair the buggy Java project.
Do not change tests. Do not remove assertions.

Bug:
- project: {project}
- bug id: {bug_id}
- failing tests: {triggering_tests}
- stack/assert summary: {failure_summary}

Allowed context:
{code_context}

Guidance:
{arm_specific_guidance}

Return:
- unified diff only
- short rationale after diff
- no test edits

Receipt Schema

字段	用途
bug_id / arm / round	配对比较的主键。
model / temperature / seed	复现实验预算。
prompt_hash / input_tokens / output_tokens	成本与公平性。
patch_hash / changed_files / changed_lines	补丁大小与重复检测。
compile_status / triggering_status / relevant_status / full_status	validation 分层结果。
failure_summary	下一轮反馈给 LLM 的唯一失败信息。
plausible / correct_review_status	最终论文表格不能混淆。

第一天只做什么

Day 1

只让一个 bug 走完整闭环

选择一个编译快、triggering test 明确的 Defects4J bug。先手写一个空补丁和一个明显错误补丁，确认 apply/test/receipt 都工作。然后再接 LLM。这样可以避免把工程 bug 误认为模型能力问题。