MSFT Accenture and F100 SWE walk into a bar

Though each experiment is noisy, when data is combined across three experiments and 4,867 developers, our analysis reveals a 26.08% increase (SE: 10.3%) in completed tasks among developers using the AI tool. Notably, less experienced developers had higher adoption rates and greater productivity gains. … We find that Copilot significantly raises task completion for more recent hires and those in more junior positions but not for developers with longer tenure and in more senior positions.

Focused research on the productivity improvements that using AI made on software engineers working on real products. Even covers the observation that more builds/prs/commits could be a move to trial-and-error coding style driven by code generation. But authors indicate there is only weak support for this hypothesis.

The productivity gains seem material and significant.

Side note: There is a quote in here that the experiment was abandoned at Accenture due to a large layoff affecting 42% of participants. I guess proving even more effective engineers aren’t immune from layoffs.

I also found a great medium article summarizing the original pdf

_{Quote Citation: Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz, “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers”, 5 Sep 2024, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566}