# GPT-6.1 Sol leaderboards: which developer tools GPT-6.1 Sol chooses in Codex

Source: https://armature.tech/leaderboards/sol-6-1 (also https://leaderboards.armature.tech/sol-6-1), Armature agent leaderboards. 1014 runs, 34 sectors, 97 apps, one model (GPT-6.1 Sol) in one agent (Codex), updated 2026-09-29. Interactive board with every run: https://armature.tech/leaderboards/sol-6-1#app

We tested GPT-6.1 Sol in Codex across 34 sectors. These boards contain 1014 runs.

- Open a sector to see its product rankings, key learnings, and evidence. Open a product to see where it wins and loses.
- The test set includes the GPT-6.1 Sol campaign and its sandbox rerun. Each recorded choice counts once. The agent frameworks sector is left out.
- Product choices count even when the implementation was incomplete. Runs without captured evidence are not counted.

## Sectors at a glance

| Sector | Runs | First choice | Share |
|---|---:|---|---:|
| [Agent runtime](#agent-runtime) | 60 | LangGraph | 15% |
| [AI gateway](#ai-gateway) | 24 | Portkey | 33% |
| [AI search](#ai-search) | 48 | Exa | 35% |
| [AI SRE](#ai-sre) | 18 | Resolve AI | 39% |
| [Auth](#auth) | 68 | Auth0 | 34% |
| [Background jobs](#background-jobs) | 16 | pg-boss | 19% |
| [Bot protection](#bot-protection) | 15 | Cloudflare Turnstile | 80% |
| [Cloud](#cloud) | 18 | AWS | 61% |
| [Code review](#code-review) | 30 | CodeRabbit | 63% |
| [Databases](#databases) | 18 | Neon | 44% |
| [Deploy](#deploy) | 30 | Render | 50% |
| [Document processing & OCR](#document-processing) | 60 | Azure AI Document Intelligence | 45% |
| [E-signature](#e-signature) | 18 | DocuSign | 56% |
| [LLM evals & observability](#evals) | 24 | Langfuse | 33% |
| [In-app chat & calls](#in-app-communication) | 18 | Stream Chat and Video | 39% |
| [Internationalization](#internationalization) | 36 | next-intl | 17% |
| [Email providers](#mail) | 18 | Resend | 39% |
| [Maps](#maps) | 58 | Google Maps | 24% |
| [MCP gateway](#mcp-gateway) | 15 | Amazon Bedrock AgentCore Gateway | 33% |
| [Message queues](#message-queues) | 48 | Amazon SQS | 17% |
| [Model choice](#model-choice) | 9 | OpenAI (audio) | 100% |
| [Observability](#observability) | 30 | Sentry | 30% |
| [OCR](#ocr) | 15 | Azure AI Document Intelligence | 53% |
| [Payments](#payments) | 33 | Stripe | 88% |
| [Performance in CI](#perf-ci) | 18 | Grafana k6 | 44% |
| [Product analytics](#product-analytics) | 30 | PostHog | 60% |
| [Agent sandboxes](#sandboxes) | 34 | Modal | 53% |
| [Search](#search) | 53 | Postgres Full-Text Search | 17% |
| [Serverless functions](#serverless) | 24 | AWS Lambda | 38% |
| [File storage](#storage) | 24 | Amazon S3 | 46% |
| [Usage-based billing](#usage-based-billing) | 33 | Metronome | 42% |
| [Vector search](#vector-search) | 18 | Typesense | 28% |
| [Voice](#voice) | 9 | Deepgram | 22% |
| [Voice Agents](#voice-agents) | 44 | Retell AI | 57% |

## Agent runtime

<a id="agent-runtime"></a>LangGraph leads the product choices. 60 runs on 12 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/agent-runtime

We asked GPT-6.1 Sol to add agent runtime to 12 apps. This board contains 60 runs.

- **The leading choice (15%).** LangGraph was chosen in 9 runs. Temporal followed with 7 runs.
- **Being named does not ensure adoption (60 mentions).** LangChain appeared in 60 runs. It was chosen in 0 runs.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | LangGraph (langchain.com) | 9 | 15% |
| 2 | Temporal (temporal.io) | 7 | 12% |
| 3 | E2B + Pydantic AI + Temporal | 5 | 8% |
| 4 | E2B + LangChain4j + Temporal | 3 | 5% |
| 5 | E2B + RubyLLM + Temporal | 3 | 5% |
| 6 | E2B + Temporal + Vercel AI SDK | 3 | 5% |
| 7 | E2B + FastMCP + Pydantic AI + Temporal | 3 | 5% |
| 8 | E2B + Temporal | 3 | 5% |
| 9 | Deep Agents + E2B + LangGraph | 3 | 5% |
| 10 | Pydantic AI (pydantic.dev) | 2 | 3% |
| | 18 more on the board | | |

## AI gateway

<a id="ai-gateway"></a>Portkey leads the product choices. 24 runs on 8 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/ai-gateway

We asked GPT-6.1 Sol to add an AI gateway to 8 apps. This board contains 24 runs.

- **The leading choice (33%).** Portkey was chosen in 8 runs. Cloudflare AI Gateway followed with 6 runs.
- **The project affects the choice (8 apps).** We tested 8 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Portkey (portkey.ai) | 8 | 33% |
| 2 | Cloudflare AI Gateway (cloudflare.com) | 6 | 25% |
| 3 | Vercel AI Gateway (vercel.com) | 4 | 17% |
| 4 | OpenRouter (openrouter.ai) | 4 | 17% |
| 5 | Amazon Bedrock (aws.amazon.com) | 2 | 8% |

## AI search

<a id="ai-search"></a>Exa leads the product choices. 48 runs on 12 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/ai-search

We asked GPT-6.1 Sol to get live web information into 12 apps. This board contains 48 runs.

- **The leading choice (35%).** Exa was chosen in 17 runs. Parallel followed with 8 runs.
- **The project affects the choice (12 apps).** We tested 12 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Exa (exa.ai) | 17 | 35% |
| 2 | Parallel (parallel.ai) | 8 | 17% |
| 3 | Firecrawl (firecrawl.dev) | 6 | 13% |
| 4 | Brave Search API (brave.com) | 6 | 13% |
| 5 | Tavily (tavily.com) | 6 | 13% |
| 6 | OpenAI web search (openai.com) | 3 | 6% |
| 7 | Perplexity Sonar (perplexity.ai) | 1 | 2% |
| 8 | NewsAPI.ai (newsapi.ai) | 1 | 2% |

## AI SRE

<a id="ai-sre"></a>Resolve AI leads the product choices. 18 runs on 6 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/ai-sre

We asked GPT-6.1 Sol to add automatic incident investigation and fixes to 6 apps. This board contains 18 runs.

- **The leading choice (39%).** Resolve AI was chosen in 7 runs. Sentry Seer followed with 6 runs.
- **The project affects the choice (6 apps).** We tested 6 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Resolve AI (resolve.ai) | 7 | 39% |
| 2 | Sentry Seer (sentry.io) | 6 | 33% |
| 3 | Datadog Bits AI Dev Agent + Datadog Bits Investigation | 3 | 17% |
| 4 | StackPilot (stack-pilot.com) | 1 | 6% |
| 5 | OneUptime AI (oneuptime.com) | 1 | 6% |

## Auth

<a id="auth"></a>Auth0 leads the product choices. 68 runs on 17 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/auth

We asked GPT-6.1 Sol to add auth to 17 apps. This board contains 68 runs.

- **The leading choice (34%).** Auth0 was chosen in 23 runs. Clerk followed with 8 runs.
- **The project affects the choice (17 apps).** We tested 17 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Auth0 (auth0.com) | 23 | 34% |
| 2 | Clerk (clerk.com) | 8 | 12% |
| 3 | Microsoft Entra ID (entra.microsoft.com) | 7 | 10% |
| 4 | WorkOS AuthKit (workos.com) | 6 | 9% |
| 5 | Built in-house (outcome) | 5 | 7% |
| 6 | Google Identity (developers.google.com) | 4 | 6% |
| 7 | Better Auth (better-auth.com) | 3 | 4% |
| 8 | Amazon Cognito (aws.amazon.com) | 3 | 4% |
| 9 | Keycloak (keycloak.org) | 3 | 4% |
| 10 | Laravel Fortify + Laravel Socialite | 2 | 3% |
| | 2 more on the board | | |

## Background jobs

<a id="background-jobs"></a>GPT-6.1 Sol most often built it in the app. 16 runs on 11 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/background-jobs

We asked GPT-6.1 Sol to add background jobs to 11 apps. This board contains 16 runs.

- **Building in the app comes first (25%).** The agents built the capability in the app in 4 runs. This was more common than any single product choice.
- **The project affects the choice (11 apps).** We tested 11 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Built in-house (outcome) | 4 | 25% |
| 2 | pg-boss (github.com) | 3 | 19% |
| 3 | Celery (celeryq.dev) | 2 | 13% |
| 4 | Supabase Queues (supabase.com) | 2 | 13% |
| 5 | BullMQ (bullmq.io) | 1 | 6% |
| 6 | Agenda (github.com) | 1 | 6% |
| 7 | GoodJob (github.com) | 1 | 6% |
| 8 | Inngest (inngest.com) | 1 | 6% |
| 9 | Procrastinate (procrastinate.readthedocs.io) | 1 | 6% |

## Bot protection

<a id="bot-protection"></a>Cloudflare Turnstile leads the product choices. 15 runs on 5 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/bot-protection

We asked GPT-6.1 Sol to add bot protection to 5 apps. This board contains 15 runs.

- **The leading choice (80%).** Cloudflare Turnstile was chosen in 12 runs. Vercel BotID followed with 1 run.
- **The project affects the choice (5 apps).** We tested 5 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Cloudflare Turnstile (cloudflare.com) | 12 | 80% |
| 2 | Vercel BotID (vercel.com) | 1 | 7% |

## Cloud

<a id="cloud"></a>AWS leads the product choices. 18 runs on 6 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/cloud

We asked GPT-6.1 Sol to pick a cloud for 6 apps. This board contains 18 runs.

- **The leading choice (61%).** AWS was chosen in 11 runs. Google Cloud followed with 3 runs.
- **The project affects the choice (6 apps).** We tested 6 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | AWS (aws.amazon.com) | 11 | 61% |
| 2 | Google Cloud (cloud.google.com) | 3 | 17% |
| 3 | Upstash (upstash.com) | 1 | 6% |
| 4 | E2B + Supabase + Trigger.dev | 1 | 6% |
| 5 | Inngest (inngest.com) | 1 | 6% |
| 6 | Cloudflare + Upstash | 1 | 6% |

## Code review

<a id="code-review"></a>CodeRabbit leads the product choices. 30 runs on 10 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/code-review

We asked GPT-6.1 Sol to add automated code review to 10 apps. This board contains 30 runs.

- **The leading choice (63%).** CodeRabbit was chosen in 19 runs. Qodo Merge followed with 9 runs.
- **The project affects the choice (10 apps).** We tested 10 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | CodeRabbit (coderabbit.ai) | 19 | 63% |
| 2 | Qodo Merge (qodo.ai) | 9 | 30% |
| 3 | Greptile (greptile.com) | 1 | 3% |
| 4 | Claude Code review (anthropic.com) | 1 | 3% |

## Databases

<a id="databases"></a>Neon leads the product choices. 18 runs on 6 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/databases

We asked GPT-6.1 Sol to pick a database for 6 apps. This board contains 18 runs.

- **The leading choice (44%).** Neon was chosen in 8 runs. Amazon RDS for PostgreSQL followed with 3 runs.
- **The project affects the choice (6 apps).** We tested 6 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Neon (neon.tech) | 8 | 44% |
| 2 | Amazon RDS for PostgreSQL (aws.amazon.com) | 3 | 17% |
| 3 | Supabase (supabase.com) | 2 | 11% |
| 4 | Azure Database for PostgreSQL Flexible Server (azure.microsoft.com) | 2 | 11% |
| 5 | DigitalOcean Managed PostgreSQL (digitalocean.com) | 1 | 6% |
| 6 | Azure SQL Database (azure.microsoft.com) | 1 | 6% |
| 7 | Scaleway Managed Database for PostgreSQL (scaleway.com) | 1 | 6% |

## Deploy

<a id="deploy"></a>Render leads the product choices. 30 runs on 6 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/deploy

We asked GPT-6.1 Sol to deploy 6 apps. This board contains 30 runs.

- **The leading choice (50%).** Render was chosen in 15 runs. Vercel followed with 10 runs.
- **The project affects the choice (6 apps).** We tested 6 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Render (render.com) | 15 | 50% |
| 2 | Vercel (vercel.com) | 10 | 33% |
| 3 | Cloudflare (cloudflare.com) | 4 | 13% |
| 4 | Netlify (netlify.com) | 1 | 3% |

## Document processing & OCR

<a id="document-processing"></a>Azure AI Document Intelligence leads the product choices. 60 runs on 12 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/document-processing

We asked GPT-6.1 Sol to read the documents that arrive at 12 apps. This board contains 60 runs.

- **The leading choice (45%).** Azure AI Document Intelligence was chosen in 27 runs. Amazon Textract followed with 7 runs.
- **The project affects the choice (12 apps).** We tested 12 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Azure AI Document Intelligence (azure.microsoft.com) | 27 | 45% |
| 2 | Amazon Textract (aws.amazon.com) | 7 | 12% |
| 3 | Mistral OCR (mistral.ai) | 5 | 8% |
| 4 | Docling (docling-project.github.io) | 5 | 8% |
| 5 | Reducto (reducto.ai) | 4 | 7% |
| 6 | Google Cloud Document AI (cloud.google.com) | 4 | 7% |
| 7 | Veryfi (veryfi.com) | 2 | 3% |
| 8 | Azure AI Content Understanding (azure.microsoft.com) | 2 | 3% |
| 9 | Google Gemini (ai.google.dev) | 1 | 2% |
| 10 | OpenAI models (openai.com) | 1 | 2% |
| | 2 more on the board | | |

## E-signature

<a id="e-signature"></a>DocuSign leads the product choices. 18 runs on 9 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/e-signature

We asked GPT-6.1 Sol to add e-signature to 9 apps. This board contains 18 runs.

- **The leading choice (56%).** DocuSign was chosen in 10 runs. Dropbox Sign followed with 3 runs.
- **The project affects the choice (9 apps).** We tested 9 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | DocuSign (docusign.com) | 10 | 56% |
| 2 | Dropbox Sign (sign.dropbox.com) | 3 | 17% |
| 3 | Documenso (documenso.com) | 2 | 11% |
| 4 | Skribble (skribble.com) | 1 | 6% |
| 5 | DocuSeal (docuseal.com) | 1 | 6% |
| 6 | YouTrust (youtrust.com) | 1 | 6% |

## LLM evals & observability

<a id="evals"></a>Langfuse leads the product choices. 24 runs on 2 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/evals

We asked GPT-6.1 Sol to add LLM evals to 2 apps. This board contains 24 runs.

- **The leading choice (33%).** Langfuse was chosen in 8 runs. Promptfoo followed with 7 runs.
- **The project affects the choice (2 apps).** We tested 2 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Langfuse (langfuse.com) | 8 | 33% |
| 2 | Promptfoo (promptfoo.dev) | 7 | 29% |
| 3 | Braintrust (braintrust.dev) | 3 | 13% |
| 4 | Arize Phoenix (arize.com) | 2 | 8% |
| 5 | Helicone (helicone.ai) | 1 | 4% |
| 6 | pydantic-evals (pydantic.dev) | 1 | 4% |
| 7 | LangSmith (smith.langchain.com) | 1 | 4% |
| 8 | Built in-house (outcome) | 1 | 4% |

## In-app chat & calls

<a id="in-app-communication"></a>Stream Chat and Video leads the product choices. 18 runs on 6 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/in-app-communication

We asked GPT-6.1 Sol to add in-app chat and calls to 6 apps. This board contains 18 runs.

- **The leading choice (39%).** Stream Chat and Video was chosen in 7 runs. Daily followed with 3 runs.
- **The project affects the choice (6 apps).** We tested 6 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Stream Chat and Video (getstream.io) | 7 | 39% |
| 2 | Daily (daily.co) | 3 | 17% |
| 3 | Supabase (supabase.com) | 3 | 17% |
| 4 | Daily + Twilio | 2 | 11% |
| 5 | Ably (ably.com) | 1 | 6% |
| 6 | Stream Chat and Video + Twilio | 1 | 6% |
| 7 | Twilio (twilio.com) | 1 | 6% |

## Internationalization

<a id="internationalization"></a>next-intl leads the product choices. 36 runs on 12 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/internationalization

We asked GPT-6.1 Sol to add internationalization to 12 apps. This board contains 36 runs.

- **The leading choice (17%).** next-intl was chosen in 6 runs. Weblate followed with 5 runs.
- **The project affects the choice (12 apps).** We tested 12 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | next-intl (next-intl.dev) | 6 | 17% |
| 2 | Weblate (weblate.org) | 5 | 14% |
| 3 | Django translation framework (docs.djangoproject.com) | 4 | 11% |
| 4 | Paraglide JS (inlang.com) | 3 | 8% |
| 5 | i18next (i18next.com) | 3 | 8% |
| 6 | Rails I18n (guides.rubyonrails.org) | 3 | 8% |
| 7 | Symfony Translation (symfony.com) | 3 | 8% |
| 8 | Lingui (lingui.dev) | 2 | 6% |
| 9 | Vue I18n (vue-i18n.intlify.dev) | 2 | 6% |
| 10 | i18next + Weblate | 1 | 3% |
| | 4 more on the board | | |

## Email providers

<a id="mail"></a>Resend leads the product choices. 18 runs on 9 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/mail

We asked GPT-6.1 Sol to add email sending to 9 apps. This board contains 18 runs.

- **The leading choice (39%).** Resend was chosen in 7 runs. Postmark followed with 5 runs.
- **The project affects the choice (9 apps).** We tested 9 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Resend (resend.com) | 7 | 39% |
| 2 | Postmark (postmarkapp.com) | 5 | 28% |
| 3 | Amazon SES (aws.amazon.com) | 2 | 11% |
| 4 | SendGrid (sendgrid.com) | 2 | 11% |
| 5 | Paubox (paubox.com) | 1 | 6% |
| 6 | Generic SMTP (nodemailer.com) | 1 | 6% |

## Maps

<a id="maps"></a>Google Maps leads the product choices. 58 runs on 14 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/maps

We asked GPT-6.1 Sol to add a map to 14 apps. This board contains 58 runs.

- **The leading choice (24%).** Google Maps was chosen in 14 runs. Mapbox followed with 10 runs.
- **The project affects the choice (14 apps).** We tested 14 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Google Maps (mapsplatform.google.com) | 14 | 24% |
| 2 | Mapbox (mapbox.com) | 10 | 17% |
| 3 | Apple Maps (MapKit) (developer.apple.com) | 7 | 12% |
| 4 | Leaflet (leafletjs.com) | 6 | 10% |
| 5 | Leaflet + MapTiler | 3 | 5% |
| 6 | MapTiler (maptiler.com) | 3 | 5% |
| 7 | MapLibre (maplibre.org) | 3 | 5% |
| 8 | react-native-maps (github.com) | 2 | 3% |
| 9 | OpenStreetMap (openstreetmap.org) | 2 | 3% |
| 10 | Azure Maps (azure.microsoft.com) | 2 | 3% |
| | 5 more on the board | | |

## MCP gateway

<a id="mcp-gateway"></a>Amazon Bedrock AgentCore Gateway leads the product choices. 15 runs on 5 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/mcp-gateway

We asked GPT-6.1 Sol to add mcp gateway to 5 apps. This board contains 15 runs.

- **The leading choice (33%).** Amazon Bedrock AgentCore Gateway was chosen in 5 runs. agentgateway followed with 2 runs.
- **The project affects the choice (5 apps).** We tested 5 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Amazon Bedrock AgentCore Gateway (aws.amazon.com) | 5 | 33% |
| 2 | agentgateway (agentgateway.dev) | 2 | 13% |
| 3 | Kong MCP Gateway (konghq.com) | 2 | 13% |
| 4 | Composio Connect | 1 | 7% |
| 5 | Lunar MCPX (lunar.dev) | 1 | 7% |
| 6 | MintMCP Gateway (mintmcp.com) | 1 | 7% |
| 7 | Composio (composio.dev) | 1 | 7% |
| 8 | Cloudflare MCP Server Portals (cloudflare.com) | 1 | 7% |
| 9 | Digitalocean Action Gateway | 1 | 7% |

## Message queues

<a id="message-queues"></a>Amazon SQS leads the product choices. 48 runs on 12 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/message-queues

We asked GPT-6.1 Sol to add a message queue to 12 apps. This board contains 48 runs.

- **The leading choice (17%).** Amazon SQS was chosen in 8 runs. Apache Kafka followed with 5 runs.
- **The project affects the choice (12 apps).** We tested 12 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Amazon SQS (aws.amazon.com) | 8 | 17% |
| 2 | Apache Kafka (kafka.apache.org) | 5 | 10% |
| 3 | Service Bus (azure.microsoft.com) | 4 | 8% |
| 4 | Amazon Kinesis (aws.amazon.com) | 3 | 6% |
| 5 | Confluent (confluent.io) | 3 | 6% |
| 6 | Cloudflare Queues (cloudflare.com) | 3 | 6% |
| 7 | Inngest (inngest.com) | 3 | 6% |
| 8 | Laravel Queues (laravel.com) | 3 | 6% |
| 9 | Cloud Tasks (cloud.google.com) | 2 | 4% |
| 10 | Graphile Worker (worker.graphile.org) | 2 | 4% |
| | 11 more on the board | | |

## Model choice

<a id="model-choice"></a>OpenAI (audio) leads the product choices. 9 runs on 3 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/model-choice

We asked GPT-6.1 Sol to add model choice to 3 apps. This board contains 9 runs.

- **The leading choice (100%).** OpenAI (audio) was chosen in 9 runs. No other product won a run.
- **The project affects the choice (3 apps).** We tested 3 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | OpenAI (audio) (openai.com) | 9 | 100% |

## Observability

<a id="observability"></a>Sentry leads the product choices. 30 runs on 11 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/observability

We asked GPT-6.1 Sol to add observability to 11 apps. This board contains 30 runs.

- **The leading choice (30%).** Sentry was chosen in 9 runs. New Relic followed with 5 runs.
- **The project affects the choice (11 apps).** We tested 11 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Sentry (sentry.io) | 9 | 30% |
| 2 | New Relic (newrelic.com) | 5 | 17% |
| 3 | Grafana (grafana.com) | 4 | 13% |
| 4 | Checkly (checklyhq.com) | 3 | 10% |
| 5 | Amazon CloudWatch (aws.amazon.com) | 3 | 10% |
| 6 | Better Stack (betterstack.com) | 2 | 7% |
| 7 | GlitchTip (glitchtip.com) | 1 | 3% |
| 8 | Datadog (datadoghq.com) | 1 | 3% |
| 9 | Azure Monitor (azure.microsoft.com) | 1 | 3% |
| 10 | AWS X-Ray (aws.amazon.com) | 1 | 3% |

## OCR

<a id="ocr"></a>Azure AI Document Intelligence leads the product choices. 15 runs on 5 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/ocr

We asked GPT-6.1 Sol to add ocr to 5 apps. This board contains 15 runs.

- **The leading choice (53%).** Azure AI Document Intelligence was chosen in 8 runs. Amazon Textract followed with 3 runs.
- **The project affects the choice (5 apps).** We tested 5 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Azure AI Document Intelligence (azure.microsoft.com) | 8 | 53% |
| 2 | Amazon Textract (aws.amazon.com) | 3 | 20% |
| 3 | Tesseract OCR (github.com) | 2 | 13% |
| 4 | PaddleOCR (paddleocr.ai) | 1 | 7% |
| 5 | Google Cloud Document AI (cloud.google.com) | 1 | 7% |

## Payments

<a id="payments"></a>Stripe leads the product choices. 33 runs on 11 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/payments

We asked GPT-6.1 Sol to add payments to 11 apps. This board contains 33 runs.

- **The leading choice (88%).** Stripe was chosen in 29 runs. Paddle followed with 2 runs.
- **The project affects the choice (11 apps).** We tested 11 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Stripe (stripe.com) | 29 | 88% |
| 2 | Paddle (paddle.com) | 2 | 6% |
| 3 | GoCardless (gocardless.com) | 1 | 3% |
| 4 | Mollie (mollie.com) | 1 | 3% |

## Performance in CI

<a id="perf-ci"></a>Grafana k6 leads the product choices. 18 runs on 8 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/perf-ci

We asked GPT-6.1 Sol to add a performance gate to 8 apps. This board contains 18 runs.

- **The leading choice (44%).** Grafana k6 was chosen in 8 runs. hyperfine followed with 2 runs.
- **The project affects the choice (8 apps).** We tested 8 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Grafana k6 (k6.io) | 8 | 44% |
| 2 | Built in-house (outcome) | 5 | 28% |
| 3 | hyperfine (github.com) | 2 | 11% |
| 4 | Gatling (gatling.io) | 1 | 6% |
| 5 | PHPBench (phpbench.readthedocs.io) | 1 | 6% |
| 6 | JMH (openjdk.org) | 1 | 6% |

## Product analytics

<a id="product-analytics"></a>PostHog leads the product choices. 30 runs on 10 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/product-analytics

We asked GPT-6.1 Sol to add product analytics to 10 apps. This board contains 30 runs.

- **The leading choice (60%).** PostHog was chosen in 18 runs. Metabase followed with 3 runs.
- **The project affects the choice (10 apps).** We tested 10 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | PostHog (posthog.com) | 18 | 60% |
| 2 | Metabase (metabase.com) | 3 | 10% |
| 3 | Mitzu (mitzu.io) | 2 | 7% |
| 4 | Mixpanel (mixpanel.com) | 2 | 7% |
| 5 | Datadog (datadoghq.com) | 2 | 7% |
| 6 | Matomo (matomo.org) | 1 | 3% |
| 7 | Datadog Product Analytics (datadoghq.com) | 1 | 3% |
| 8 | Umami (umami.is) | 1 | 3% |

## Agent sandboxes

<a id="sandboxes"></a>Modal leads the product choices. 34 runs on 5 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/sandboxes

We asked GPT-6.1 Sol to add a code sandbox to 5 apps. This board contains 34 runs.

- **The leading choice (53%).** Modal was chosen in 18 runs. Vercel Sandbox followed with 6 runs.
- **The project affects the choice (5 apps).** We tested 5 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Modal (modal.com) | 18 | 53% |
| 2 | Vercel Sandbox (vercel.com) | 6 | 18% |
| 3 | E2B (e2b.dev) | 6 | 18% |
| 4 | Daytona (daytona.io) | 4 | 12% |

## Search

<a id="search"></a>GPT-6.1 Sol most often built it in the app. 53 runs on 11 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/search

We asked GPT-6.1 Sol to add search to 11 apps. This board contains 53 runs.

- **Building in the app comes first (21%).** The agents built the capability in the app in 11 runs. This was more common than any single product choice.
- **The project affects the choice (11 apps).** We tested 11 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Built in-house (outcome) | 11 | 21% |
| 2 | Postgres Full-Text Search (postgresql.org) | 9 | 17% |
| 3 | OpenSearch (opensearch.org) | 7 | 13% |
| 4 | Typesense (typesense.org) | 6 | 11% |
| 5 | Elasticsearch (elastic.co) | 4 | 8% |
| 6 | Meilisearch (meilisearch.com) | 3 | 6% |
| 7 | Algolia (algolia.com) | 3 | 6% |
| 8 | Atlas Search (mongodb.com) | 3 | 6% |
| 9 | Oracle Indexed Search (oracle.com) | 2 | 4% |
| 10 | Redis Query Engine (redis.io) | 1 | 2% |
| | 2 more on the board | | |

## Serverless functions

<a id="serverless"></a>AWS Lambda leads the product choices. 24 runs on 8 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/serverless

We asked GPT-6.1 Sol to add serverless functions to 8 apps. This board contains 24 runs.

- **The leading choice (38%).** AWS Lambda was chosen in 9 runs. Vercel Functions followed with 6 runs.
- **The project affects the choice (8 apps).** We tested 8 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | AWS Lambda (aws.amazon.com) | 9 | 38% |
| 2 | Vercel Functions (vercel.com) | 6 | 25% |
| 3 | Azure Functions (azure.microsoft.com) | 3 | 13% |
| 4 | Cloudflare Workers (workers.cloudflare.com) | 3 | 13% |
| 5 | Trigger.dev (trigger.dev) | 2 | 8% |
| 6 | Google Cloud Functions (cloud.google.com) | 1 | 4% |

## File storage

<a id="storage"></a>Amazon S3 leads the product choices. 24 runs on 8 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/storage

We asked GPT-6.1 Sol to add file storage to 8 apps. This board contains 24 runs.

- **The leading choice (46%).** Amazon S3 was chosen in 11 runs. Cloudflare R2 followed with 7 runs.
- **The project affects the choice (8 apps).** We tested 8 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Amazon S3 (aws.amazon.com) | 11 | 46% |
| 2 | Cloudflare R2 (cloudflare.com) | 7 | 29% |
| 3 | Google Cloud Storage (cloud.google.com) | 3 | 13% |
| 4 | Azure Blob Storage (azure.microsoft.com) | 3 | 13% |

## Usage-based billing

<a id="usage-based-billing"></a>Metronome leads the product choices. 33 runs on 11 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/usage-based-billing

We asked GPT-6.1 Sol to bill customers for what they use in 11 apps. This board contains 33 runs.

- **The leading choice (42%).** Metronome was chosen in 14 runs. Orb followed with 8 runs.
- **The project affects the choice (11 apps).** We tested 11 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Metronome (metronome.com) | 14 | 42% |
| 2 | Orb (withorb.com) | 8 | 24% |
| 3 | Lago (getlago.com) | 3 | 9% |
| 4 | Oracle BRM (oracle.com) | 3 | 9% |
| 5 | m3ter (m3ter.com) | 2 | 6% |
| 6 | Zuora (zuora.com) | 1 | 3% |
| 7 | Flexprice (flexprice.io) | 1 | 3% |
| 8 | PortaBilling | 1 | 3% |

## Vector search

<a id="vector-search"></a>Typesense leads the product choices. 18 runs on 6 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/vector-search

We asked GPT-6.1 Sol to add vector search to 6 apps. This board contains 18 runs.

- **The leading choice (28%).** Typesense was chosen in 5 runs. Neon followed with 3 runs.
- **The project affects the choice (6 apps).** We tested 6 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Typesense (typesense.org) | 5 | 28% |
| 2 | Neon (neon.tech) | 3 | 17% |
| 3 | Pinecone (pinecone.io) | 3 | 17% |
| 4 | OpenAI Vector Stores (openai.com) | 3 | 17% |
| 5 | Supabase Vector (supabase.com) | 2 | 11% |
| 6 | sqlite-vec (alexgarcia.xyz) | 1 | 6% |
| 7 | Qdrant (qdrant.tech) | 1 | 6% |

## Voice

<a id="voice"></a>Deepgram shares the lead. 9 runs on 6 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/voice

We asked GPT-6.1 Sol to add voice to 6 apps. This board contains 9 runs.

- **The leading products are tied (22%).** Deepgram was chosen in 2 runs. OpenAI (audio) also won 2 runs.
- **The project affects the choice (6 apps).** We tested 6 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Deepgram (deepgram.com) | 2 | 22% |
| 2 | OpenAI (audio) (openai.com) | 2 | 22% |
| 3 | ElevenLabs (elevenlabs.io) | 2 | 22% |
| 4 | Speechmatics (speechmatics.com) | 1 | 11% |
| 5 | Kokoro (github.com) | 1 | 11% |
| 6 | Web Speech API (developer.mozilla.org) | 1 | 11% |

## Voice Agents

<a id="voice-agents"></a>Retell AI leads the product choices. 44 runs on 10 apps, updated 2026-09-29. Board: https://armature.tech/leaderboards/sol-6-1#app/voice-agents

We asked GPT-6.1 Sol to add a voice agent to 10 apps. This board contains 44 runs.

- **The leading choice (57%).** Retell AI was chosen in 25 runs. LiveKit Agents followed with 5 runs.
- **The project affects the choice (10 apps).** We tested 10 apps. Each result records a choice made for an existing project.

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Retell AI (retellai.com) | 25 | 57% |
| 2 | LiveKit Agents (livekit.io) | 5 | 11% |
| 3 | ElevenLabs Agents (elevenlabs.io) | 4 | 9% |
| 4 | Vapi (vapi.ai) | 4 | 9% |
| 5 | OpenAI Realtime API (openai.com) | 2 | 5% |
| 6 | Picovoice (picovoice.ai) | 1 | 2% |
| 7 | Rasa Pro (rasa.com) | 1 | 2% |
| 8 | Cognigy (cognigy.com) | 1 | 2% |
| 9 | OpenAI GPT-Live (openai.com) | 1 | 2% |

## How this was measured

Every number comes from a controlled experiment. GPT-6.1 Sol ran in Codex on 97 existing codebases, asked in several wordings and as different kinds of user to add one capability (the sector) and to choose the product itself. Each run happened in a sandbox at a pinned agent version, and a judge read the session to record the choice. A choice counts when the session captured it and the judge confirmed it, even when the implementation was incomplete; runs without captured evidence are left out. Shares are wins over all runs in the sector. Several tools adopted together count as one combination. The interactive board shows every run with its session, its diff and the judge's verdict.

Methodology and publications: https://armature.tech/publications

## Other boards

- All agents, sector by sector: https://armature.tech/leaderboards/sectors (Markdown: https://armature.tech/leaderboards/sectors.md)
- GPT-6.2 Sol: https://armature.tech/leaderboards/sol-6-2 (Markdown: https://armature.tech/leaderboards/sol-6-2.md)
- GPT-6.2 Astra: https://armature.tech/leaderboards/astra-6-2 (Markdown: https://armature.tech/leaderboards/astra-6-2.md)
- Claude Fable 5.2: https://armature.tech/leaderboards/fable-5-2 (Markdown: https://armature.tech/leaderboards/fable-5-2.md)
- Muse Ember: https://armature.tech/leaderboards/muse-ember (Markdown: https://armature.tech/leaderboards/muse-ember.md)
- Muse Wonder: https://armature.tech/leaderboards/muse-wonder (Markdown: https://armature.tech/leaderboards/muse-wonder.md)
- Grok 4.7: https://armature.tech/leaderboards/grok-4-7 (Markdown: https://armature.tech/leaderboards/grok-4-7.md)
- Claude Opus 5.5: https://armature.tech/leaderboards/opus-5-5 (Markdown: https://armature.tech/leaderboards/opus-5-5.md)
- Muse Spark 1.3: https://armature.tech/leaderboards/muse-spark-1-3 (Markdown: https://armature.tech/leaderboards/muse-spark-1-3.md)
- GPT-6 Sol: https://armature.tech/leaderboards/sol-6 (Markdown: https://armature.tech/leaderboards/sol-6.md)
