# Grok 4.7 Is Trending. A Leaderboard Is Still Not a Model Procurement Strategy

> Conflicting Grok 4.7 evaluations show why serious product teams should test models on their own code, harnesses, costs and failure budgets before switching.

Source: https://mcdougall-digital.de/en/blog/grok-4-7-model-procurement/
Published: 2026-09-22
Author: Kyluke McDougall, McDougall Digital
Language: en

---

Grok 4.7 is having the kind of launch that makes model selection look easy.

It is trending on X. The official numbers show large improvements over Grok 4.6. Independent evaluators describe strong price-performance. Coding platforms are adding it immediately. The API starts at $2 per million input tokens and $6 per million output tokens.

Then the results begin to disagree.

SpaceXAI reports 46.3% on CursorBench 4.0 and 38.0% on Terminal-Bench 4.0. Devin reports strong performance on hard backend-engineering tasks. Artificial Analysis places the model near the front of the price-performance discussion. Vals, however, reports a lower aggregate score than Grok 4.6. Early users praise its speed and value in some workflows while reporting weak results in others.

This does not mean one benchmark is wrong.

It means the question “Which model is best?” is too vague to support a serious purchasing decision.

For founders, CTOs and product owners, the useful lesson from the Grok 4.7 launch is not whether to switch today. It is how to decide whether any new model belongs in your delivery system.

## One model, several credible answers

[SpaceXAI's release data](https://x.ai/news/grok-4-7) makes a strong case for improvement over the previous version.

The vendor reports:

- 46.3% on CursorBench 4.0, up from 40.4% for Grok 4.6;
- 38.0% on Terminal-Bench 4.0, up from 20.3%;
- 71.0% on DeepSWE v1.1 at high effort, up from 65.2%;
- the same standard API price as Grok 4.6;
- better self-verification and longer-context handling.

Those are meaningful gains. They are not a universal verdict.

The same vendor table shows Fable 5.1 Max ahead on CursorBench and Terminal-Bench, while GPT-5.6 Sol Max leads on DeepSWE. Independent evaluations apply different task sets and produce different rankings. Real users then add another variable: the repository, instructions, tools and review process around the model.

A model that is good at a long backend migration may be poor at a pixel-accurate frontend task. A model that solves more tickets may consume enough tokens or wall-clock time to lose on cost. A model that succeeds inside its native harness may perform differently through a generic API wrapper. A high aggregate score may hide the one failure mode your regulated or revenue-critical workflow cannot tolerate.

There is no contradiction in a model being:

- a good everyday coding agent;
- a poor choice for one specialised workload;
- cheaper per accepted task;
- more expensive per token-heavy failed attempt;
- strong in its native harness;
- weaker in your existing toolchain.

“Best” only becomes meaningful after the job and the operating conditions are defined.

## The harness is part of the product

One detail in the Grok 4.7 announcement deserves more attention than another leaderboard position.

SpaceXAI says it trained the model to understand the Grok Bot harness. Team members also advised users to run it through the company's Build harness for the best results.

That matters because a coding model does not work alone.

The harness decides how the model sees the repository, searches code, edits files, runs commands, interprets failures, compresses context and decides when to stop. It may add planning, memory, sub-agents, retries and verification. Permission boundaries determine which actions are possible. Repository instructions determine which actions are desirable.

Change the harness and you change the system being evaluated.

This is why a public result rarely transfers directly into a company environment. A benchmark may test one model with high reasoning effort, a tuned prompt, a particular tool protocol and a generous token budget. Your team may use medium effort, a different agent shell, stricter timeouts and a monorepo with years of implicit conventions.

Buying a model based on a leaderboard is like buying a database from an isolated query benchmark while ignoring schema, workload, network, operations and failure recovery.

The component matters. The surrounding system determines whether it works.

## Measure accepted work, not generated work

Token prices are easy to compare. Engineering outcomes are harder.

The cheapest model per million tokens is not necessarily the cheapest model per accepted change. The model that generates a patch in three minutes may still be expensive if an engineer spends an hour correcting it. A slower model can be cheaper when it produces smaller diffs, passes tests more often and requires less review.

For coding work, useful unit economics look more like this:

**Total task cost = model cost + compute and tool cost + review time + rework + expected failure cost.**

That last term matters. A plausible but incorrect database migration has a different cost from a flawed internal script. A security regression in an authentication flow has a different failure budget from a CSS defect.

Track outcomes that connect to delivery:

- percentage of tasks accepted without material rework;
- tests passed on the first complete attempt;
- engineer review minutes per accepted change;
- escaped defects and rollback frequency;
- median and tail completion time;
- total model and infrastructure cost per accepted task;
- size and clarity of the resulting diff;
- policy or security violations;
- ability to explain and reproduce the change.

These measures will not fit into one attractive public score. They will tell you whether the model helps your company.

## A six-step evaluation for serious product teams

You do not need a research laboratory to evaluate a new model. You need a controlled portfolio and explicit acceptance criteria.

### 1. Define the job before choosing the model

Do not start with “We should test Grok 4.7.” Start with the bottleneck: backend tickets, dependency upgrades, regression tests, incident investigation, frontend implementation or pull-request review.

Choose one or two jobs where better performance would produce measurable value. Define what the model may access, what it may change and where human approval remains mandatory.

### 2. Build a representative task portfolio

A useful internal evaluation is not ten toy prompts.

Select completed tasks from your own history and remove the solutions. Include routine work, ambiguous work, cross-cutting changes and tasks that previously failed. Preserve the repository state, requirements and tests available at the time.

Twenty to fifty representative tasks can reveal more about fit than a global benchmark with hundreds of unrelated problems. Keep a protected holdout set so prompt tuning does not become test-set optimisation.

### 3. Test the model and harness together

Run the candidate in the environment you would actually operate.

Keep model effort, tool access, timeouts, retry policy and context rules explicit. If you compare native and third-party harnesses, treat them as separate systems. Record versions because both models and orchestration layers change quickly.

Compare the current model in the current harness, the candidate in the current harness and the candidate in its recommended harness. Add a lower-cost routine option and a frontier escalation option where useful.

This often leads to routing, not replacement.

### 4. Score the whole outcome

Automated tests are necessary but insufficient.

Use blinded human review where practical. Score correctness, maintainability, security, scope discipline and explanation quality. Record whether the model changed unrelated files, weakened tests, introduced dependencies or solved the symptom instead of the cause.

Then calculate cost per accepted outcome, not cost per run.

### 5. Introduce it through shadow and canary phases

A benchmark win does not justify immediate repository-wide access.

Start with shadow runs on real tasks while the existing process remains authoritative. Then allow a canary group to use it on bounded repositories and low-risk task classes.

Log prompts, tool calls, commands, changed files, test results, approvals and final outcomes. Establish stop conditions before rollout: security violations, rising review time, excessive retries, unstable cost or a regression in accepted-task rate.

### 6. Keep switching and rollback cheap

The model market moves too quickly for deep, irreversible coupling.

Use stable internal interfaces for task submission, tool permissions, telemetry and result review. Keep repository guidance portable. Store evaluation fixtures outside a vendor's product. Separate model choice from business logic.

Native features can still be valuable. The objective is to know where coupling exists and preserve a tested fallback for critical workflows.

## Use a routing policy, not a company-wide winner

The likely end state for many teams is not one approved model.

It is a small routing policy:

- low-cost model for narrow, well-tested maintenance;
- strong coding model for multi-file implementation;
- long-context or research model for investigation;
- specialised tool for security or data work;
- human review proportional to the possible damage.

Routing can be manual at first. The important part is that the choice follows the task and risk profile.

This also improves negotiating power. When evaluation data and interfaces are portable, a provider's new release becomes an option to test instead of an urgent migration event.

## What should a CTO do this week?

Do not hold a meeting to debate screenshots of leaderboards.

Pick one valuable workflow. Recover 20 representative historical tasks. Define acceptance criteria and a failure budget. Run your current setup and one candidate setup under the same conditions. Measure accepted-task cost and review effort. Keep the candidate in shadow mode until the result survives contact with real engineers and real repositories.

Grok 4.7 may win that test. Its price, longer-task training and early coding results make it a credible candidate.

It may also lose for your workload.

Both outcomes are useful because the goal is not to predict the industry's favourite model. The goal is to build a delivery system that can adopt a better component without outsourcing technical judgement to a trending chart.

Models will keep changing.

Your evaluation boundary should be the durable part.

---

This article was created with the help of AI. McDougall Digital is responsible for its content.