One of the most useful things an LLM benchmark can reveal is that your benchmark is wrong.

We recently tested 27 chat-capable open-source models through DigitalOcean's OpenAI-compatible Serverless Inference API.

The initial goal was simple:

Find models that offer a good balance of quality, latency, and reliability for real-time AI workloads.

We expected to discover which models were fast.

Which ones produced better answers.

Which ones failed under load.

We found all of that.

But we also found something more useful.

Some of the results that initially looked like model or infrastructure failures were actually caused by our own benchmark configuration.

That changed the way we interpreted the entire test.

What We Tested

Every model received the same six Turkish customer-support-style prompts:

1. greeting / simple conversation
2. how-to request
3. short or ambiguous question
4. customer complaint
5. detailed comparison request
6. off-topic question

The models were called through:

https://inference.do-ai.run/v1/chat/completions

using the OpenAI SDK and streaming chat completions.

Our initial baseline configuration was approximately:

temperature: 0.2
max_tokens: 500
stream: true

We measured:

• time to first token
• total completion time
• output length
• request success/failure
• answer quality

For quality evaluation, we used GPT-4o separately through the OpenAI API.

The judge received only the original prompt and generated answer.

It did not know which model produced the response.

Answers were scored from 1–10 for:

• fluency
• helpfulness
• tone
• conciseness

This was not intended to create a universal LLM leaderboard.

Six prompts in one language are nowhere near enough for that.

The question was narrower:

What actually happens when these models are used behind an interactive AI product?

The Results Were Closer Than Expected

Most models that completed normally produced surprisingly good answers.

Here is a selected view of the benchmark.

Model	Quality	Avg. Total Time	Observation
deepseek-v3.2	9.0/10	16.9s	Highest quality, slower
deepseek-v4-pro	9.0/10	13.9s	High quality, slower
glm-5.2 — corrected run	9.0/10	37.7s	Good output after increasing token budget
deepseek-v4-flash-0731	8.8/10	3.7s	Excellent speed/quality balance
nemotron-3-ultra-550b	8.8/10	4.5s	Excellent speed/quality balance
qwen3-coder-flash	8.7/10	2.9s	Very fast despite being code-oriented
glm-5.1 — corrected run	8.7/10	72.5s	Good quality, but extremely slow
gpt-oss-20b	8.5/10	3.4s	Strong latency
nemotron-3-nano-omni	8.5/10	2.6s	One of the fastest
qwen3.8-max	8.3/10	3.4s	Fast
Kimi K3 — initial reasoning run	7.3/10*	10.7s	Token budget constrained

*The Kimi K3 result should not be treated as directly comparable with ordinary chat models because the initial max_tokens: 500 budget was also being consumed by reasoning.

The interesting part of the table is not who ranked first.

It is how compressed the quality scores became.

Once many models are producing answers between roughly 8 and 9 out of 10, the difference between them starts moving somewhere else.

Latency.

Reliability.

Serving capacity.

Configuration.

And workload fit.

8.8 in Four Seconds Can Be Better Than 9.0 in Forty

Two models stood out immediately for interactive workloads.

DeepSeek V4 Flash 0731

Quality:

8.8 / 10

Average total response time:

3.7 seconds

Nemotron 3 Ultra 550B

Quality:

8.8 / 10

Average total response time:

4.5 seconds

Compare that with the corrected GLM-5.2 run.

GLM-5.2 scored:

9.0 / 10

That is technically better.

But its average total response time was:

37.7 seconds

So which model wins?

There is no universal answer.

If you are running an offline research task, waiting longer might be completely reasonable.

If a customer is staring at a live chat interface, four seconds and forty seconds are completely different products.

This is why a production benchmark cannot optimize for quality alone.

We Initially Thought GLM-5.1 and GLM-5.2 Were Broken

This was probably the most useful mistake in the benchmark.

During our first run, glm-5.1 and glm-5.2 returned essentially no visible answer.

With:

max_tokens: 500

five of the six prompts were ending with:

finish_reason: "length"

while producing no useful visible content.

The shortest prompt was the exception and ended normally with:

finish_reason: "stop"

At first, the behavior looked suspicious.

The original glm-5 worked using the same API integration.

The newer models appeared to complete requests but returned almost nothing.

It was tempting to classify this as a model-serving or streaming problem.

That would have been wrong.

We Increased the Budget to 2,500 Tokens

We reran the GLM models with:

max_tokens: 2500

The result changed dramatically.

GLM-5.2

The corrected run produced usable responses across all six prompts.

Average quality:

9.0 / 10

Average total latency:

37.7 seconds

The longest prompt still exhausted the larger token budget, but this time only after producing roughly 2,796 visible characters.

That is a completely different result from:

“GLM-5.2 returns empty responses.”

The model was capable of producing very good answers.

Our original budget simply did not give its generation behavior enough room.

GLM-5.1

GLM-5.1 also produced good-quality answers after the correction.

Quality:

8.7 / 10

But the latency problem remained.

Average total latency:

72.5 seconds

And three of the six requests hit our 90-second client timeout.

So the corrected conclusion was not:

GLM-5.1 is broken.

It was:

GLM-5.1 can produce good answers in this workload, but it is far too slow for the real-time path we were evaluating.

That distinction matters.

A lot.

The Benchmark Had Produced a False Positive

This was a useful reminder of how easy it is to misdiagnose LLM behavior.

The initial observation was technically correct:

The API returned almost no visible content.

But our explanation was wrong.

We had interpreted an evaluation-configuration problem as a possible serving problem.

The difference only became visible because we inspected:

finish_reason

and reran the same models with a larger generation budget.

This suggests a useful rule for LLM testing:

Never classify an empty or truncated response as a model failure before inspecting token usage and finish reason.

HTTP status codes are not enough.

Visible output length is not enough.

You need to understand why generation stopped.

Kimi K3 Exposed the Same Problem in a Different Way

Reasoning models introduced another complication.

Kimi K3, Kimi K2.x, and DeepSeek R1-style models can expose reasoning separately from the final visible answer.

That means a configuration like:

max_tokens: 500

does not necessarily mean:

500 tokens for the answer.

Part of that budget can disappear into reasoning before the final answer is produced.

In our initial Kimi K3 run we measured:

Metric	Kimi K3
Quality	7.3/10
Avg. first token	7.4s
Avg. total time	10.7s
Avg. visible output	357 characters

Those numbers look mediocre compared with some of the standard chat models.

But the comparison is not entirely fair.

The model was operating under a token budget that constrained its reasoning and visible response simultaneously.

So instead of concluding:

Kimi K3 is worse.

the more accurate conclusion was:

Reasoning models require a different benchmark configuration.

That is an important distinction if you are benchmarking a heterogeneous model catalog.

Identical Parameters Do Not Always Create a Fair Test

This sounds counterintuitive.

Benchmarks usually try to keep everything identical.

Same prompts.

Same parameters.

Same evaluation.

Same environment.

That feels scientifically correct.

But an identical configuration can actually make a model comparison less fair when the models behave fundamentally differently.

A standard chat model may be perfectly comfortable with:

max_tokens: 500

A reasoning model may spend most of that budget before reaching its final answer.

Another model may generate much longer internal sequences.

A third may require a specific temperature setting.

So our next benchmark should not have one universal configuration.

It should have model-class-aware configurations.

For example:

Standard chat models
→ normal token budget

Reasoning models
→ larger reasoning/output budget

Code models
→ code-specific prompts

Tool models
→ tool-calling evaluation

Structured-output models
→ schema-compliance tests

Fairness does not always mean treating every model identically.

Sometimes fairness means evaluating each architecture under conditions appropriate for its intended use.

Some Failures Were Still Real

Correcting our own benchmark did not explain everything.

We also observed genuine operational differences.

Qwen 3.5 397B A17B

During one test run, all six requests returned:

429 Platform overloaded.
Please try again later.

Our account-level headers still showed significant request quota.

On another run, the overload errors disappeared, but requests took approximately 46–53 seconds and returned empty visible output.

Unlike the GLM case, our saved benchmark did not establish a corrected successful run, so it would be wrong to draw a quality conclusion from that test.

The useful result is simply:

we could not obtain a reliably usable completion under those test conditions.

Arcee Trinity Large Thinking

This model appeared in the catalog available to our account but every completion attempt returned:

403
this model is not available for your subscription tier

This is less a model-quality issue than a discovery/entitlement issue.

If applications dynamically consume /v1/models, catalog visibility and actual entitlement need to be treated separately.

Nemotron 3 Super 120B

Two of six requests returned:

429 Platform overloaded

The successful requests were also substantially slower than most of the catalog.

Again, this does not prove anything universal about the model.

Serving conditions are temporal.

It does show why a single quality score is insufficient for production model selection.

HTTP 200 Is Not a Production Success Metric

One of the broader lessons from the experiment is that inference monitoring needs to go beyond transport-level success.

A useful completion might require:

HTTP success
        +
valid stream
        +
non-empty output
        +
acceptable finish reason
        +
valid structure
        +
acceptable latency
        =
usable response

An API request can succeed while the product experience fails.

For an AI application, the more meaningful metric might be:

usable responses
----------------
 total requests

rather than raw API success rate.

What We Would Change in the Next Benchmark

The first test was deliberately small.

A stronger benchmark would add:

• more prompts
• multiple languages
• repeated runs
• different times of day
• p50 / p95 / p99 latency
• separate reasoning-model configurations
• structured-output reliability
• tool-calling success
• retry behavior
• cost per successful response
• successful-useful-response rate

I would also keep the failed first runs.

Not in the final ranking.

But in the dataset.

Because failed experiments contain useful information.

The GLM result is a good example.

If we had simply deleted the original run after fixing it, we would have lost one of the most useful lessons from the entire benchmark.

The Benchmark Became an Infrastructure Test

We started by comparing 27 models.

Eventually, we realized we were evaluating a much larger system.

The model matters.

But so do:

• token configuration
• serving capacity
• latency
• streaming behavior
• entitlement
• finish reasons
• retries
• fallbacks
• observability
• workload characteristics

And the “best model” becomes increasingly difficult to define.

For customer-facing chat, I might prefer an 8.8 model that consistently answers in four seconds.

For deep research, I might happily wait forty seconds for a better result.

For coding, another model may win.

For classification, cost may dominate everything else.

So instead of asking:

Which model is best?

a production team probably needs to ask:

Which model is best for this execution path, under these latency, reliability, cost, and quality constraints?

That is a less exciting leaderboard.

But it is a much better engineering question.

And perhaps the most valuable result from our 27-model benchmark was not finding the winner.

It was discovering that benchmark configuration itself is part of the system being tested.

Disclosure: I’m building Tinten.ai, and these tests were performed while evaluating inference infrastructure for our AI workloads.