ReportsAI Coding ToolsRun historyNov 15, 2026
AI Coding Tools · published run
Run · Nov 15, 2026
Download data
Full dataset (CSV)
Every product, model, topic and run, with confidence intervals.
Or get the summary CSV (ranked products, all models) without an email.
One snapshot. Every recommendation is traceable to an answer.
Prompt set v110 models2,500 / 2,500 · 100.0% validPublished
Permanent URL. This published snapshot will not change. A correction is a new revision with a note, not an edit of this page.
Rankings describe what AI models answered to a fixed set of prompts. They are not endorsements or quality ratings. Product names belong to their owners.
5 samples per prompt
Topic × product
Recommendation rate by topic. All models, equal weight.
| Topic | GitHub Copilot | Cursor | Claude Code | Windsurf | Amazon Q Developer | Aider | OpenAI Codex | Continue |
|---|---|---|---|---|---|---|---|---|
| Autonomous agents | 76% | 74% | 71% | 27% | 25% | 53% | 73% | 8% |
| Debugging & testing | 84% | 68% | 45% | 18% | 24% | 15% | 11% | 3% |
| Enterprise | 72% | 40% | 22% | 31% | 60% | 13% | 7% | 36% |
| IDE assistance | 98% | 84% | 44% | 55% | 23% | 14% | 4% | 45% |
| Developer workflow | 64% | 65% | 51% | 37% | 19% | 40% | 26% | 21% |
Each cell is the recommendation rate. Models are weighted equally inside a topic. A dash means the product was not in this slice of the fixture.
Model × product
Recommendation rate by model, across every topic.
| Model | GitHub Copilot | Cursor | Claude Code | Windsurf | Amazon Q Developer | Aider | OpenAI Codex | Continue |
|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 76% | 71% | 73% | 35% | 32% | 31% | 30% | 28% |
| DeepSeek V4.1 Flash | 84% | 60% | 39% | 38% | 33% | 28% | 26% | 38% |
| Gemini 4 Hydrogen | 86% | 77% | 56% | 65% | 28% | 40% | 26% | 27% |
| Muse Spark 1.3 | 84% | 61% | 20% | 49% | 53% | 26% | 17% | 28% |
| Mistral Large 4 | 75% | 61% | 30% | 39% | 42% | 34% | 8% | 30% |
| Kimi K3.1 Pro | 76% | 56% | 34% | 24% | 20% | 22% | 19% | 28% |
| GPT-6.1 Sol | 80% | 64% | 42% | 11% | 18% | 18% | 31% | 20% |
| Perplexity Sonar | 65% | 50% | 54% | 12% | 16% | 21% | 22% | 7% |
| Grok 4.7 | 83% | 74% | 36% | 31% | 34% | 26% | 19% | 28% |
| GLM-5.3 | 72% | 61% | 42% | 38% | 25% | 33% | 10% | 24% |
Each cell is the recommendation rate. Models are weighted equally inside a topic. A dash means the product was not in this slice of the fixture.
Prompt-level evidence
Short excerpts from the fixed prompt set. Product names in the answers are not endorsements.
Autonomous agents
Which AI coding agents can take a GitHub issue and open a working pull request on their own?
GPT-6.1 Sol
Yes. Several coding agents can take a GitHub issue, modify the repository, run tests, and open a pull request **without you driving each step**. The key distinction is whether you can assign the issue directly or need to hand the agent its URL. | Agent | Issue → PR workflow | Best fit | |---|---|---| | **[GitHub…
Claude Sonnet 5.5
Several tools can do this today. The space changes quickly, so check current docs for details. **Hosted or commercial agents** - **GitHub Copilot coding agent**: Assign an issue to Copilot. It works in a cloud environment powered by GitHub Actions, then opens a draft PR and responds to review comments. - **OpenAI…
Gemini 4 Hydrogen
Several AI coding agents can take a GitHub issue, explore the repository, write and test code in a sandboxed environment, and open a pull request autonomously. They generally fall into three categories: **major platform/lab agents**, **dedicated autonomous agent startups**, and **open-source frameworks**. --- ### 1.…
Debugging & testing
What AI tools are best for debugging failing tests and fixing bugs in a Python project?
GPT-6.1 Sol
For a Python project, the most useful AI tools are those that can **inspect the repository, run `pytest`, edit code, and rerun the failing tests**—not just explain a pasted traceback. Here’s a practical shortlist. Product features and pricing change quickly, so treat this as a workflow comparison rather than a fixed…
Claude Sonnet 5.5
# AI tools for debugging Python tests and bugs The landscape changes fast, so treat this as a snapshot and try a couple yourself. ## Agentic tools (best for failing tests) These can run your test suite, read the traceback, edit code, and re-run until tests pass. - **Claude Code**: terminal-based, strong at multi-file…
Gemini 4 Hydrogen
For debugging failing tests and fixing Python bugs, the single most important feature to look for is **agentic terminal execution**—meaning the AI can run `pytest` (or `unittest`), read the traceback, inspect the relevant files, edit the code, and re-run the test automatically until it passes. Copy-pasting tracebacks…
Enterprise
Which AI coding assistants are best for an enterprise with strict security and compliance requirements?
GPT-6.1 Sol
For a security-sensitive enterprise, the best choice depends first on **where code may be processed**—not on which assistant wins a coding benchmark. A regulated company that permits approved SaaS has different options from an organization that prohibits source code from leaving its network. **My shortlist:** GitHub…
Claude Sonnet 5.5
# AI Coding Assistants for Security-Conscious Enterprises Offerings change quickly, so verify certifications and features with each vendor before deciding. ## Leading options by deployment need **Mainstream SaaS with strong enterprise controls** - **GitHub Copilot Business/Enterprise**: No training on your data, IP…
Gemini 4 Hydrogen
For an enterprise with strict security and compliance requirements, the "best" AI coding assistant depends primarily on **where your code is allowed to travel**—specifically whether your security posture permits **zero-data-retention SaaS**, requires **private cloud (VPC) / Bring Your Own Model (BYOM)**, or mandates…
IDE assistance
What's the best AI coding assistant to use inside my code editor for everyday programming?
GPT-6.1 Sol
**For most people who want an AI assistant in their existing editor, GitHub Copilot is the safest default.** It’s a good fit for everyday autocomplete, explaining unfamiliar code, writing tests, and making routine changes—without requiring you to change your workflow. My shortlist: | Your priority | My pick | Why |…
Claude Sonnet 5.5
There's no single best one, since it depends on your editor, budget, and workflow. Here's how I'd break it down: **Best all-around picks** - **GitHub Copilot**: Works in VS Code, JetBrains, Neovim, and Visual Studio. It has solid autocomplete, chat, and agent features, and it's the lowest-friction option if you just…
Gemini 4 Hydrogen
There isn’t one single "best" assistant for everyone—it mostly comes down to **which editor you use**, **whether you're willing to switch editors**, and **how you prefer to work** (fast inline autocomplete vs. multi-file AI chat/agents). Here are the strongest options for everyday programming right now: --- ### 1.…
Developer workflow
What are the best terminal-based or command-line AI coding agents?
GPT-6.1 Sol
For **coding agents that work directly in your terminal**—reading a repository, editing files, and running commands—my shortlist is **Claude Code, Codex CLI, OpenCode, Gemini CLI, and Aider**. **Goose** and **GitHub Copilot CLI** are also worth considering. There’s no universal winner: model quality, permissions,…
Claude Sonnet 5.5
# Terminal-based AI coding agents This space moves fast, so check each project's current state. ## Top tier (commercial or vendor-backed) - **Claude Code** (Anthropic): Widely considered the leader. It has strong agentic loops, subagents, hooks, MCP support, plan mode, and a `CLAUDE.md` memory file. It requires a…
Gemini 4 Hydrogen
The best terminal-based AI coding agents balance codebase awareness, safe execution (running tests/commands, reviewing diffs), and model quality. Here are the top command-line coding agents, depending on your workflow and budget: --- ### 1. **Claude Code** (by Anthropic) * **Best for:** Deep codebase reasoning,…
Model memory
Model memory — what each model generation learned in training. Not dated observations; no web search. It never shares an axis with this run.
Open model memory