TL;DR — Four different failures get reported as "the model returned invalid JSON", and they need different fixes: prose or markdown fences around the object (use a structured-output mode), a truncated object (your token budget, not the model), syntactically valid JSON with the wrong shape (schema enforcement, not JSON mode), and valid JSON containing a hallucinated value (no output mode fixes this — validate the content). Find out which one you have before reaching for a regex.
The four failures
1. Prose or fences around the object
Here's the JSON you requested:
```json
{"name": "Ada"}
Let me know if you need anything else!
`json.loads` fails on the first character. This is the one people meet first and the one that is genuinely solved by asking for a structured output mode, which constrains the model to emit only the object.
**2. Truncated object**
```json
{"items": [{"id": 1, "title": "Something long
Valid JSON up to where it stops. This is a token-budget failure, not a formatting failure — the generation hit the cap mid-object. The tell is a length finish reason. Raising the completion budget fixes it; retrying without raising it produces the same truncation at the same place.
3. Valid JSON, wrong shape
{"user_name": "Ada"} // you expected "name"
{"age": "42"} // you expected a number
Parses cleanly, then breaks your code downstream. A plain JSON mode does not help here — it guarantees syntax, not structure. What you want is schema-constrained output, where the provider enforces your schema during generation.
4. Valid, correctly-shaped, and wrong
{"founded": 1847, "source": "company website"}
Perfect JSON. The number is invented. No output mode can fix this — structured output constrains form, never truth. If the value matters, it has to be checked against something real, or carry a field saying where it came from.
What to do
| Failure | Fix |
|---|---|
| Prose / fences | Structured-output or JSON mode |
| Truncated | Raise the completion budget; log finish_reason |
| Wrong shape | Schema-constrained output, plus client-side validation |
| Wrong content | Validation against a source of truth — not an output mode |
And regardless of mode, always validate after parsing. Provider-side schema enforcement is strong but it is not a reason to skip a validator you control; version drift and fallback models will eventually hand you something unexpected.
Two implementation notes
- Do not repair JSON with regex. Stripping fences is fine. Balancing braces on a truncated object invents data — you are guessing at what got cut off, and the result parses, which means nobody notices.
- Retry on parse failure, but change something. An identical retry at the same temperature and budget is likely to fail identically; raise the budget or tighten the instruction.
Prevention
- Log the raw response on every parse failure. The four cases above are instantly distinguishable from the raw text and nearly impossible to tell apart from the exception alone.
- Keep the schema in one place and generate both the request constraint and the client-side validator from it, so they cannot drift.
- If you fall back across models, remember that structured-output support is not uniform — a fallback path can silently land on a model without it, which is exactly when your parser meets case 1 again. Related: AI API gateway vs LLM router.