Which tokeniser is used by Groq?

Forum
1

Hello
What tokeniser is used by Groq ? Is there a python library that can be added in the code which tells us the token count BEFORE the request is sent to Groq servers ? Will be very useful for context engineering and optimising tokens will benifit the entire ecosystem.

2

Hi! We use whatever tokenizer was used to train the model.

You can do this:

from tokenizers import Tokenizer
t = Tokenizer.from_pretrained("meta-llama/Meta-Llama-3-70B-Instruct")
t.encode("hello").ids

Docs: tokenizers · PyPI

2 Likes
3

Thanks for sharing this!

4

with some hit and trial, i arrived at the following for getting accurate prompt tokens count that matches the value in the api response :

from tokenizers import Tokenizer

# Load the tokenizer
t = Tokenizer.from_pretrained("openai/gpt-oss-20b")

# If messages is a list of dictionaries (like in chat completion)
text_content = ""
for msg in messages:
    text_content += f"{msg['role']}: {msg['content']}\n"
    
encoding = t.encode(text_content)
token_count = len(encoding.ids)
accurate_token_count = (len(messages)*17+1)+token_count
1 Like
5

Wow I didn’t realize there was more work involved to do that; if you have more tips and tricks on how you reverse engineered and built that out, I’d love if you cross posted that on the forum as a technical post — that might help a lot of people!