AI-Assisted Development > AI-assisted coding > Prompt engineering for code generation
What This Covers
This scenario focuses on prompt engineering for AI-assisted code generation. The design challenge is evaluating whether a prompt strategy that looks strong in internal testing is robust enough for broader real-world developer use.
The System
Current State
- A code assistant is used by 420 internal engineers and handles about 18,000 code-generation requests per weekday.
- The current prompt template is a compact, general-purpose system prompt plus lightweight repository context, file excerpts, and the user’s task.
- The team tracks task completion rate, syntax-valid output rate, average first-response latency, and developer acceptance rate after manual review.
- On the current internal benchmark set of 600 tasks, the assistant achieves 78% acceptance rate with median latency of 4.1 seconds.
Proposed Change
- The team wants to replace the current prompt with a more structured prompt generated through iterative prompt tuning and benchmark optimization.
- The new prompt adds explicit planning steps, style constraints, repo-analysis instructions, and a required self-check section before final code output.
- The expected benefit is higher acceptance on benchmark tasks without changing the underlying model.
Worked Example: Design Tradeoff
In an earlier rollout, the team introduced a heavily optimized prompt for bug-fix generation. It performed very well on the validation tasks used during development, especially for short Python and TypeScript issues drawn from the team’s main services.
During review, a design concern came up: the prompt had accumulated many narrow instructions that matched the development set unusually well, but there was little evidence it would behave consistently on requests that looked different from those examples. The initial benchmark still looked excellent, so the concern was easy to dismiss.
What changed the discussion was a broader sample review. Quality remained high on familiar tasks, but dropped on requests with longer pasted stack traces, mixed natural language and code, and repositories using less common frameworks. Small wording changes in user requests also caused larger-than-expected differences in output structure. Interestingly, a slightly simplified version of the prompt gave up some peak benchmark performance but behaved more consistently across a wider task sample.
The team kept some of the structure but removed several tightly prescriptive instructions, added evaluation slices by language, prompt length, and repo type, and required pre-launch testing on a traffic-shaped sample rather than only the curated benchmark. The lesson was not “avoid optimization,” but to ask what exactly had been optimized and for which requests.
The Design Question
The team now wants to ship the new structured prompt as the default for all code-generation requests, including code completion, refactoring suggestions, unit test generation, and API migration tasks. Internal benchmark results are strong, but the proposed design increases prompt complexity and was tuned mostly on task-based generation requests rather than the full spread of production usage.
Should the team adopt one highly structured default prompt, or preserve a simpler default and use specialized prompting only for some request classes?
Anchor Data
| Prompt variant | Benchmark task set size | Acceptance rate | Syntax-valid rate | Median latency | Avg prompt tokens |
|---|---|---|---|---|---|
| Current general prompt | 600 | 78.0% | 96.1% | 4.1s | 890 |
| Structured prompt v2 | 600 | 84.7% | 97.4% | 5.3s | 1,640 |
| Structured prompt v2 + self-check | 600 | 86.2% | 97.8% | 6.1s | 2,080 |
Benchmark composition:
- 58% function implementation
- 21% unit test generation
- 14% bug fixes
- 7% refactoring
Languages in benchmark:
- 61% Python
- 24% TypeScript
- 10% Java
- 5% Go
Current Observations
- Reviewers say the structured prompt produces noticeably better outputs on benchmark-style implementation tasks.
- Two pilot users reported that it was less reliable on large pasted files and on requests phrased as terse IDE comments rather than full instructions.
- One engineer suggested the recent model-version update may explain the variation seen in a small production shadow test.
- In a limited shadow run of 1,200 live requests, acceptance improved overall, but gains were concentrated in Python repositories.
Constraints
- The team needs a launch recommendation within 10 business days to align with a quarterly developer productivity review.
- No model change is available this quarter; only prompt, routing, and evaluation changes are feasible.
- The developer tools group wants one default experience unless there is clear evidence that routing complexity is justified.
What You’ll Be Evaluated On
- Tradeoffs: How well you identify competing concerns and justify a design choice.
- Gaps: Whether you spot what evidence is missing from the current proposal and data.
- Prevention: Whether you anticipate risks in rollout and propose practical safeguards.
- Clarity: How clearly and directly you communicate your design reasoning.
- Prioritization: Whether you focus on the highest-impact questions first.
- Reasoning quality: Whether your conclusions follow from the evidence and acknowledge uncertainty.