Gpt-oss-120b ignoring tools

Forum
21

Thanks for your patience. We have fixed this issue - it should be working as expected now.

22

Hey ​@benank I tried several times but still getting the 400 issue with tool_choice = 'required’ or tool_choice = 'any’. Moreover I can see from the failed_generation response that the tools are not picked up at all. The only way groq is working is with tool_choice = 'auto’ but it is unstable in terms of tool calling. Did anyone had a succesfull attempt with tool_choice = 'required' today?

Error code: 400 - {'error': {'message': 'Tool choice is required, but model did not call a tool', 'type': 'invalid_request_error', 'code': 'tool_use_failed', 'failed_generation': 'Is there anything ...?'}}

23

Hi,

Is there any update on 400 issue with tool_choice=None

24

Hi everyone, I don’t have any updates on this. I would recommend reading this earlier post to learn more about how this works: Gpt-oss-120b ignoring tools - #11 by benank

Setting tool_choice does not guarantee that the model will call or not call tools. The best way to encourage a model to call tools is to update your prompt to explicitly tell the model “You MUST call a tool.” Or if you don’t want it to call tools, the safest way would be to not include tools in your request at all. Otherwise, in tool_choice=auto cases, you can let the model decide when to use tools (and this will be largely based on your prompt and the model itself). I’m working on updating our documentation to be more clear about this; I understand that this can be frustrating when you expect it to call a tool and it doesn’t, and I totally agree with this.

We are investigating methods on our side to make tool calling more reliable, but changes like these need thorough testing since they affect all calls made on our platform. I don’t have any updates or timelines here.

If you have specific cases you would like me to investigate, please post the requests with reproductions that are failing here and I can provide updates.

1 Like
25

Hello, so I’m currently getting the following issue using gpt-oss-20b:
”Tool choice is none, but model called a tool”
→ my current model settings:

        response_stream = await self.client.chat.completions.create(
            model=self.model_name,
            messages=groq_messages,
            temperature=0.2,
            max_completion_tokens=160,
            reasoning_effort="low",
            stream=True,
            tools=None,
            stop=None
        )

So why is it trying to make a call to tools?

part of the error trace

python3.11/site-packages/groq/_streaming.py", line 147, in __aiter__
    async for item in self._iterator:
python3.11/site-packages/groq/_streaming.py", line 193, in __stream__
    raise APIError(
groq.APIError: Tool choice is none, but model called a tool
1 Like
26

Hi, this sounds like it could be a bug! If you’re able to, please share the exact request params (prompt, messages, etc) with me and I can take a look for you. Feel free to DM me if you don’t want to share it publicly.

27

Running into the same issue @KarimAboshamia mentioned when trying to use gpt-oss!

28

Please send me the exact request you’re using so I can look into it! :slightly_smiling_face: It’s very difficult for me to investigate without being able to reproduce it.

29

This request triggers the error for me:

from groq import Groq

client = Groq()
completion = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[
      {
        "role": "system",
        "content": "You are a development assistant expert agent with access to workspace tools. You will help the user with their super project here in Pequeroku.\n\nOnly reveal these facts **if asked**:\n\n* Your name is `Pequenin`.\n* You are inside Pequeroku, a PaaS where users can create VMs and interact with them like an online IDE.\n* People can deploy services on Pequeroku.\n\nBehavior rules (be strict, concise, chill):\n\n* Be extremely concise. Hate wasting words and time.\n* No yapping, no fluff, no emojis unless the user asks for them.\n* Do not invent facts. If you don’t know something and the user asks, reply: “I don’t know the answer” and continue.\n* Do not assume. Ask clarifying questions when anything is unclear.\n* Do not jump to conclusions. Never start coding until asked *and* you’ve asked necessary clarification questions.\n* Offer the user whether they want you to use any specific tool.\n\nInteraction flow (must follow exactly):\n\n1. Always start by asking precise clarifying questions when the user’s request could be interpreted multiple ways (e.g., what type of game, target platform, language, required features, constraints). Keep questions minimal and direct.\n2. If you think workspace tools are needed, call them. (e.g., browse the workspace, read files, create files). Use tools only when they concretely help.\n3. After the user answers, state concisely what you understood (one or two short sentences).\n4. Then perform the requested task using tools or by writing code/text in chat. Summarize what you did and where you wrote files.\n5. Finish with a question, can be something like: what do you want to do next?, do you want me do it for you?, etc.\n\nWhen coding:\n\n* Prefer readability and maintainability (clean code).\n* Propose a sensible file/project structure and filenames before creating files.\n* Use available workspace tools when they help (and ask permission if needed).\n* After changes, summarize exactly what you changed/created and where.\n\nExtra constraints:\n\n* Do not use tables in responses.\n* Keep markdown to the bare minimum.\n* Be extremely chill but useful — short, direct, no paja, no yapping.\n* If you have to create code from scratch use \\\"create_full_project\\\" if have to debug, rewrite, etc, use: \\\"read_workspace\\\", \\\"create_file\\\" and \\\"read_file\\\"\n* Don't call the same tool with the same path or task more than once in the same shift. Group changes into a single call.\n\nRequests you must satisfy from the user:\n\n* Ask clarification questions (no nonsense).\n* Call tools if you think they’re needed.\n* Tell the user what you understood.\n* Do the work using tools or chat.\n\nIf the user asks, comply: translate answers, run commands, create files, or scaffold projects — but only after step 1 (clarify) and step 3 (confirm understanding"
      },
      {
        "role": "user",
        "content": "De que va este proyecto?"
      },
      {
        "role": "assistant",
        "content": "ÂżQuĂ© proyecto especĂ­fico quieres que describa? ÂżHay algĂșn directorio o archivo en el workspace que deba revisar?"
      },
      {
        "role": "user",
        "content": "Este proyecto, dime que puedes ver"
      },
      {
        "role": "assistant",
        "content": "ÂżQuieres que explore el workspace y te muestre la estructura de archivos y el contenido principal?"
      },
      {
        "role": "user",
        "content": "Si"
      },
      {
        "role": "assistant",
        "content": "ÂżPuedo leer el workspace para listar la estructura de archivos y mostrarte los contenidos relevantes?"
      },
      {
        "role": "user",
        "content": "Si por favor"
      },
      {
        "role": "user",
        "content": ""
      }
    ],
    temperature=1,
    max_completion_tokens=8192,
    top_p=1,
    reasoning_effort="medium",
    stream=True,
    stop=None
)

for chunk in completion:
    print(chunk.choices[0].delta.content or "", end="")

I know that is a lot of text, but I hope it helps.

1 Like
30

I’ve been investigating this over the last week and it looks like we have a bug in our handling of tool_choice=required for the GPT-OSS models. On all other models, we return a 400 error when the model doesn’t call a tool when it’s required. On GPT-OSS models, it doesn’t return an error, but it should.

@benank how come this is the expected/default behaviour? The behaviour for e.g. OpenAI is that if a tool is required, the model HAS to return a tool - even if it normally wouldn’t; I believe they use constrained decoding to only allow tokens that produce json corresponding to one of the available tools. I think you already support this somehow, since you do support structured outputs/tool calls? Why is supporting the required setting different?

31

This is the expected behavior because we don’t have constrained decoding yet. Any provider that always returns a tool, such as OpenAI, uses constrained decoding to force the model to do so.

Our support for structured outputs and tool calls currently uses various forms of serverside validation, but we don’t have constrained decoding support yet, so you may see errors when the model isn’t able to output correct information.

But stay tuned! Constrained decoding is a top priority for our team and it’s coming soon :slight_smile:

2 Likes
32

I also agree with you, I tried couple of times but still the same error

33

We’re pushing really hard to get constrained decoding out, which should hopefully make this process a bit easier!

34

I’ve been running into the exact same problem since yesterday morning. Everything had been working fine, but starting yesterday I’m getting this error : 400 {“error”:{“message”:“Tool choice is none, but model called a tool”,“type”:“invalid_request_error”,“code”:“tool_use_failed”,“failed_generation”}}, Is there any update on this issue?

const response = await groq_client.chat.completions.create({

    messages: [

      {

role: "system",

content: systemPrompt,

      },

      {

role: "user",

content: userPrompt,

      },

    ],

model: "openai/gpt-oss-120b",

temperature: 0.1,

tool_choice:"none"

  });
35

Urgent: Livekit Agents + Groq Issue

We are experiencing the same critical issue, and it requires immediate resolution!

Problem Description

The problem lies within our Livekit Agents setup integrated with Groq.

Temporary Workaround

I have discovered a temporary workaround:

  • When calling generate_reply from within a function_tool, pass the instructions directly in the user_input parameter.

Example:

session.generate_reply(user_input="[SAY GOODBYE TO THE USER]")
1 Like
37

This unfortunately happens to my app too, occasionally – I usually have to update my system prompts to be more robust to prevent this, plus I’m adding more retries and other guard rails.

(Unfortunately that also adds to the speed; I’m also doing agentic voice, so every ms counts)

We’re so close to releasing constrained dec, and it’ll hopefully make many of these problems go away

38

What are you adding to your system prompt to be more robust? Thanks.

39

A few things I do:

  • give a 1-3 examples of the JSON I expect from the output, including a short example (with less metadata, but still valid) and a full example (of all the possible metadata). I’ll sometimes give it the Zod or Pydantic representation
  • for function names, I’ll usually have very pedantic function names, like instead of “generateTag” I’ll have “generateTagsForBetterSearchResults” — these kinds of function names for tool calling helps add some semantic meaning to the LLM and keeps it focused on the task
  • usually I’ll feed it just a few tools, like 1-5 tools, to choose from; if there’s a lot more tools I think it gets really confused, so I take a lot of ideas from french cooking hierarchy ideas, like having an “executive” chef to make decisions, and let different “stations” (or sub-agents) have a small number of tasks that they focus on, they go do their thing, and the executive chef figures out whether it’s good enough.
40

Is “none” passed to tool_choice param?

Working example: groq_tool_call_example.py · GitHub

41

yep!

tool_choice
string / object or null
Optional
Controls which (if any) tool is called by the model. none means the model will not call any tool and instead generates a message. auto means the model can pick between generating a message or calling one or more tools. required means the model must call one or more tools. Specifying a particular tool via {“type”: “function”, “function”: {“name”: “my_function”}} forces the model to call that tool.

source: API Reference - GroqDocs