Ask whether patch-free diagnostic evidence helps the same LLM repair bugs more accurately, faster, or with less overfitting than vanilla and Ochiai-guided baselines.
Software Engineering Empirical Research Radar
AI-Native SE Catch-up Tutorial
A practical route back into first-tier software engineering research from testing, diagnosis, and industry practice.
Purpose. This tutorial is for researchers who already understand software testing, fault localization, CI, or software quality, and want to catch up with the current AI-for-SE wave without losing their original strengths.
Main thesis. The most promising restart path is not generic code generation. It is AI-native testing, diagnosis, and repair for continuously evolving software systems.
Best Starting Directions
Turn test-case prioritization into an agentic-repair problem: which tests should an AI repair loop run after each candidate patch?
Compress traces, value changes, failing/passing behavior, and statement neighborhoods into diagnostic cards that humans and models can both use.
Connect logs, test history, change diffs, traces, and developer actions into reliable triage workflows for continuous software evolution.
Four-Week Catch-up Path
Rebuild the Field Map
First learn the shape of the current software-engineering field rather than memorizing isolated papers.
- Read the 2021-2025 ICSE/FSE/ASE/ISSTA topic map.
- Track Testing/QA, AI/LLM for SE, Debugging/FL/Diagnosis/Repair, and DevOps/CI as one connected cluster.
- Use scholar signals as a map of productive research styles, not as a leaderboard to imitate blindly.
Recover Testing and Diagnosis Intuition
Translate existing FL and TCP habits into modern AI-for-SE questions.
- FL asks where a fault is; repair asks what evidence helps generate and validate a patch.
- TCP asks which tests to run first; agentic repair asks which tests to run after every proposed patch.
- The bridge concept is budget-aware evidence: what should a human or model inspect next?
Learn Modern LLM Repair
Understand the current repair pipeline before proposing a new one.
- Direct prompting gives failing tests and code context to a model.
- FL-guided prompting adds suspicious statements or methods.
- Feedback-loop repair compiles, tests, and feeds failures back to the model.
- Agentic repair lets the model inspect files, run tools, and iterate under a fixed budget.
Study Benchmarks and Evaluation Traps
A repair or testing paper is only as credible as its evaluation protocol.
- Defects4J numbers are not comparable unless bug universe, version, oracle, and budget match.
- Plausible patches pass tests; correct patches require semantic judgment.
- Common traps include weak baselines, oracle leakage, unequal feedback budgets, and prompt-length confounds.
Design a First Paper Sprint
Convert the catch-up into a publishable, scoped experiment.
- Run paired comparisons: vanilla LLM, Ochiai-guided LLM, and diagnostic-advice-guided LLM.
- Measure fixed bugs, attempts, test cost, compile failures, patch size, and overfitting risk.
- If the signal is weak, reframe toward negative results, evaluation protocol, or agentic test-budget scheduling.
Scholar Signals to Watch
| Scholar / Line | Signal | What to Learn |
|---|---|---|
| Yang Liu | security, testing, analysis, AI and systems-facing SE work across many venues | High productivity often comes from building reusable benchmark/tool pipelines across adjacent problems. |
| Hongyu Zhang | empirical SE, reliability, AI/software analytics, performance and industry-connected problems | A durable research program can move across topics while keeping a stable empirical/software-quality lens. |
| Xin Xia and David Lo | fault localization, mining software repositories, software analytics, developer support | Frequent, careful empirical framing can turn tool ideas into broader SE claims. |
| Lingming Zhang | testing, patch validation, fault localization, program analysis | Strong technical infrastructure can anchor multiple testing and repair papers. |
Reading Queue
- Defects4J — Know the benchmark before trusting any repair number.
- SRepair — A modern function-level repair line; useful for understanding practical LLM repair settings.
- RepairAgent — A representative autonomous LLM repair agent; useful as an agentic baseline mindset.
- DebugRepair — A runtime-evidence repair direction; useful for comparing against diagnostic-evidence claims.
- On-the-fly patch validation — A bridge from testing/TCP to repair: validation cost matters as much as generation.
- Practitioners' expectations on fault localization — A reminder that diagnosis must be actionable, not merely well-ranked.
How to Read
The pipeline-first reading habit
For each paper, write down its input, oracle assumptions, model/tool actions, validation budget, benchmark universe, and failure cases before judging whether the result is impressive.
- Ask what information the method is allowed to see.
- Ask what the strongest fair baseline would be.
- Ask whether the result would survive a paired comparison under equal budgets.
- Ask which part of the pipeline is reusable for your next experiment.
Publication Compass
| Venue Type | What Usually Fits | Risk |
|---|---|---|
| ICSE / FSE / ASE / ISSTA | Strong method, strong benchmark, careful baselines, clear threat model. | Weak or unfair baselines are quickly exposed. |
| SANER / ICSME | Analysis, evolution, reengineering, repair, diagnosis, empirical maintenance workflows. | The paper still needs a crisp technical or empirical contribution. |
| ICST / ISSRE | Testing, validation, reliability, flakiness, cost-effective test execution. | AI framing alone is not enough; testing insight must be central. |
| Workshops / ERA / Short | Fast signals, new protocols, negative results, emerging agentic workflows. | Do not oversell pilot-scale evidence. |