Get reasoning key when "json_validate_failed"

Feature Requests
1

Hello,

This is the current shape of an error when json validation doesn’t work :

"type":"invalid_request_error",
"code":"json_validate_failed",
"failed_generation":"{}"}}

(I removed the keys in the object failed_generation but there are well there.)

  • Knowing the reasoning the model used, even after a failure, would be really helpful.

  • I think I could still parse this payload when validation fails. Usually it’s just one field, all the others are good.

  • I’m about to parse it and add a fallback when validation fails but I wish the reasoning of the model was added in the payload too. It would help with debugging and tracking the behaviour.

What do you think of it ? I think it would be really helpful, to be able to control the model behavior even when there is an error.

1 Like
2

Hmm interesting you’re getting a null output for failed generation — could you share your system and message prompts? I’d love to see what’s going on that you’re not getting any output.

For this example, I’m getting the generated response from the server that failed validation:

{
    "error": {
        "message": "Generated JSON does not match the expected schema. Please adjust your prompt. See 'failed_generation' for more details. Error: jsonschema: '/choice' does not validate with /properties/choice/enum: value must be \"angry\"",
        "type": "invalid_request_error",
        "code": "json_validate_failed",
        "failed_generation": "{\"choice\":\"happy\"}"
    }
}

{
    "model": "moonshotai/kimi-k2-instruct-0905",
    "messages": [
        {
            "role": "system",
            "content": "Classify the following with one of the following moods: angry, bored, confident, dazed, embarrassed, energized, fine, flirty, focused, happy, inspired, playful, sad, scared, tense, uncomfortable. Return only one word."
        },
        {
            "role": "user",
            "content": "Yay I'm so happy!!"
        }
    ],
    "max_tokens": 15,
    "temperature": 0.6,
    "top_p": 0.9,
    "response_format": {
        "type": "json_schema",
        "json_schema": {
            "name": "thechoice",
            "schema": {
                "type": "object",
                "properties": {
                    "choice": {
                        "type": "string",
                        "description": "The choice",
                        "enum": [
                            "angry"
                        ]
                    }
                },
                "required": [
                    "choice"
                ],
                "additionalProperties": false
            },
            "strict": true
        }
    }
}
3

Hey,

I am receiving it, it’s not null. I removed it because the JSON wasn’t relevant.

The question is about having the reasoning field even when the json generation fails.

Do you think it would be possible ?

4

Or maybe the good approach would be to remove strict: true from my request and just parse it to get the keys that did work

5

Oh yes, sorry I misunderstood. With reasoning, we actually don’t support json_schema however we do support json_object — and with json_object you can have it output its reasoning as well as its json output.

In this mode, it does NOT adhere as strictly as json_schema but it does include reasoning. (I actually prefer this mode myself)

More details here: Structured Outputs - GroqDocs

Request:

{
  "messages": [
    {
      "role": "user",
      "content": "How many r's are in the word strawberry? output json: {\"count\":3,\"reasoning\":\"I examined each letter in 'strawberry' in order: s-t-r-a-w-b-e-r-r-y. The letter 'r' appears at positions 3, 8, and 9, for a total of 3 occurrences.\"}"
    }
  ],
  "model": "openai/gpt-oss-20b",
  "temperature": 0.6,
  "max_completion_tokens": 4096,
  "top_p": 0.95,
  "stream": false,
  "stop": null,
  "response_format": { "type": "json_object" }
}

Response:

{
    "id": "chatcmpl-ef04d684-7b31-4ed6-9126-9977f2bc8c3a",
    "object": "chat.completion",
    "created": 1764118440,
    "model": "openai/gpt-oss-20b",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "{\"count\":3,\"reasoning\":\"I examined each letter in 'strawberry' in order: s-t-r-a-w-b-e-r-r-y. The letter 'r' appears at positions 3, 8, and 9, for a total of 3 occurrences.\"}",
                "reasoning": "User requests: \"How many r's are in the word strawberry? output json: ...\"\n\nThey provide a JSON string: {\"count\":3,\"reasoning\":\"I examined each letter in 'strawberry' in order: s-t-r-a-w-b-e-r-r-y. The letter 'r' appears at positions 3, 8, and 9, for a total of 3 occurrences.\"}\n\nWe need to verify JSON is valid. Let's check: The JSON string: {\"count\":3,\"reasoning\":\"I examined each letter in 'strawberry' in order: s-t-r-a-w-b-e-r-r-y. The letter 'r' appears at positions 3, 8, and 9, for a total of 3 occurrences.\"}\n\nCheck quotes: The outermost JSON uses double quotes. The value for \"reasoning\" is a string containing single quotes around strawberry, and also contains double quotes? It has no double quotes inside. It contains single quotes, that's fine. It contains colon after \"strawberry\" inside the string: \"I examined each letter in 'strawberry' in order: s-t-r-a-w-b-e-r-r-y. The letter 'r' appears at positions 3, 8, and 9, for a total of 3 occurrences.\"\n\nThat string includes commas, periods, etc. That's fine. It ends with a double quote. The JSON seems valid.\n\nBut we need to output JSON. The user already provided JSON. We should output that JSON. Ensure it's compact. The JSON provided is compact? It has no spaces except after colon? Actually the JSON: {\"count\":3,\"reasoning\":\"I examined each letter in 'strawberry' in order: s-t-r-a-w-b-e-r-r-y. The letter 'r' appears at positions 3, 8, and 9, for a total of 3 occurrences.\"}\n\nIt has no spaces except after colon? Actually there are no spaces after colon. It's compact. It is valid.\n\nThus we can output exactly that JSON.\n\nBut we should verify that the JSON is valid. Let's quickly parse mentally: object with two properties: count: 3 (number), reasoning: string. The string contains \"I examined each letter in 'strawberry' in order: s-t-r-a-w-b-e-r-r-y. The letter 'r' appears at positions 3, 8, and 9, for a total of 3 occurrences.\"\n\nThis is valid. So we can output that JSON."
            },
            "logprobs": null,
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "queue_time": 0.017297228,
        "prompt_tokens": 167,
        "prompt_time": 0.0110158,
        "completion_tokens": 590,
        "completion_time": 0.58079991,
        "total_tokens": 757,
        "total_time": 0.59181571,
        "completion_tokens_details": {
            "reasoning_tokens": 522
        }
    },
    "usage_breakdown": null,
    "service_tier": "on_demand"
}