The Productivity Question Was Always Hard to Answer
When AI coding assistants became mainstream in 2023, measuring their impact was harder than it looked. Developer productivity is not a single number - it involves writing code, reading code, debugging, testing, reviewing, documenting, and coordinating with teammates. A tool that speeds up code writing might slow down code review if the generated code is low quality. Aggregate measures like lines of code written are notoriously misleading: the goal is not more code but more working software.
After two years of data from organizations running formal studies, the picture is more nuanced than either the enthusiastic early claims or the skeptical pushback suggested.
Where the Gains Are Real
Boilerplate code generation is a clear win. Writing database schema migrations, API client code, test scaffolding, configuration files, and standard library usage is faster with AI assistance, and the quality is at least as good as manual writing. This is especially true for less experienced developers who might write boilerplate incorrectly on first attempt. Reducing the friction for this kind of mechanical work meaningfully changes how developers allocate their attention.
Code exploration and explanation is another genuine gain. Asking an AI to explain a function you are reading, trace a dependency chain, or suggest where to look for a bug is faster than reading through code manually. For onboarding onto a new codebase, AI assistants meaningfully accelerate the process. These are not dramatic productivity gains but consistent small wins that compound.
Test generation has improved substantially. Generating unit tests from functions, creating test cases that cover edge cases, and writing regression tests for bug fixes all benefit from AI assistance. The generated tests are not always production-quality but provide a useful starting point that developers refine rather than write from scratch.
Where the Tools Still Struggle
Complex architectural decisions remain beyond reliable AI assistance. A coding assistant can suggest an implementation approach for a well-defined function; it cannot architect a system, evaluate tradeoff decisions, or understand the implications of a design choice on other parts of a complex codebase. Teams that expected AI to handle significant portions of system design have been disappointed.
Debugging assistance is uneven. AI can suggest fixes for common error patterns and help trace through logic errors in contained functions. It struggles with bugs that emerge from system-level interactions, environment-specific issues, or incorrect assumptions embedded deep in a codebase. The same model that writes confident wrong code also suggests confident wrong fixes.
Context window limitations mean AI assistants work best on well-scoped tasks. Understanding the full implications of a code change across a large codebase requires context the models cannot hold. This is a fundamental limitation of the approach rather than a bug that will be patched away.
What This Means for Teams
The most effective teams treat AI coding tools as a skilled junior developer who works very fast, sometimes confidently produces wrong answers, and needs supervision. They automate repetitive tasks, accelerate exploration and learning, and generate good starting points for human refinement. They do not offload architectural decisions or trust the tools on complex tasks without review. The developers who get the most out of these tools are the ones who understand both their strengths and their failure modes.