GitHub Copilot — First Impressions from Real-World Use

In June 2021, GitHub launched a technical preview of Copilot — an AI assistant for writing code built on OpenAI Codex. We gained access among the first users and deployed it in real-world development. After two months, we have enough data for an objective assessment.

What Is Copilot and How Does It Work¶

GitHub Copilot is a VS Code extension (and other editors) that suggests code in real time based on context — comments, function names, surrounding code. Under the hood runs OpenAI Codex, a model trained on public GitHub repositories. It’s not autocomplete on steroids — Copilot can suggest entire functions, tests, and SQL queries.

Where Copilot Truly Helps¶

After two months of use across a team of five developers, we’re clear on where the value is greatest:

• Boilerplate code: CRUD operations, DTO mapping, REST controllers. Copilot generates 80% of the code correctly. Saves minutes per class.
• Unit tests: Write a test name and Copilot suggests the implementation. For simple methods, it nails assertions and edge cases. For complex ones, it’s a good starting point to refine.
• Regex and SQL: Write a comment “// find all emails in string” and Copilot generates the regex. Works surprisingly well even for complex SQL queries.
• Unfamiliar APIs: When working with a library you don’t know, Copilot often suggests the correct call. Saves time spent in documentation.

Where Copilot Falls Short¶

Copilot isn’t a silver bullet. We hit limits in several areas:

• Business logic: Copilot can’t handle complex domain logic. It lacks your business context — it suggests generic code that looks correct but solves the wrong problem.
• Security: Copilot occasionally suggests code with security flaws — SQL injection, hardcoded credentials, missing validation. A junior developer might not catch it.
• Architecture: Copilot doesn’t think in architectures. It suggests code function by function but doesn’t address separation of concerns, dependency injection, or design patterns in the context of the whole application.
• Proprietary code: If you use internal frameworks or custom conventions, Copilot doesn’t know them. It suggests standard solutions, not yours.

Measuring Productivity¶

We tracked metrics over two sprints — one without Copilot, one with it. Results:

• Commit count: +12% with Copilot (more smaller commits, faster iteration)
• Code review time: +8% (reviewers must also check AI-generated code)
• Story points completed: +15% (primarily due to faster boilerplate)
• QA bugs: no change (Copilot neither adds nor removes bugs if review works)

We estimate the net benefit at 10–15% speedup for experienced developers. For juniors, it’s more nuanced — they write code faster but understand less of what Copilot suggested.

Legal and Ethical Questions¶

Copilot is trained on public GitHub code — including code under copyleft licenses (GPL). It occasionally suggests code nearly identical to existing open-source projects. For enterprise use, this poses a potential legal risk. GitHub claims the suggested code is original, but the debate continues. In regulated industries (finance, healthcare), we recommend caution.

How to Use Copilot Properly¶

Based on our experience, we recommend:

1. Write quality comments — Copilot draws context from them. The better the description, the better the suggestion.
2. Always review — never accept a suggestion without reading it. Copilot is not a senior developer.
3. Use for boilerplate — that’s where ROI is highest. Write critical logic yourself.
4. Set coding standards — Copilot adapts to the style in a file but doesn’t enforce project-wide conventions.
5. Train your team — especially juniors. They must understand that Copilot is a tool, not a teacher.

AI Pair Programming Is Here — But Don’t Overestimate It¶

GitHub Copilot is the most useful developer tool of 2021. But it’s not a revolution — it’s an evolution. It saves time on routine tasks but doesn’t free you from the need to think. The biggest benefit? Less context switching between editor and documentation. The biggest risk? A false sense of confidence in less experienced developers.