Cloudflare Blocking Urllib.request without User-Agent

Forum
1

I have a simple sublime editor plugin using Python standard library function urllib.request.urlopen(...), and I was getting consistent HTTP Error 403: Forbidden error code: 1010 for any api request, indicating cloudflare was blocking my python client.

Could not stream from groq-oss-120:
  model: openai/gpt-oss-120b
  url: https://api.groq.com/openai/v1/chat/completions
  options: {'stream': True}
Error: HTTP Error 403: Forbidden
error code: 1010

I added a user-agent to the request, and cloudflare stopped blocking it:

"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

# --- System ---
You are an expert programming agent. Focus on correctness and simplicity.

# --- User ---
hi

# --- Agent ---

## --- Thinking ---
>The user says "hi". Just respond friendly.

## --- Response ---
Hello! How can I help you today?

This is not good for an API which should be accessible from any programmatic client.

This issue only started happening after Dec 18.

2

Does this plugin call our API from your localhost, and are you able to call the API from your localhost (eg via cURL) otherwise?

If you’re able to do it via cURL but not via the plugin it must be something wonky the plugin is doing — if you’re just doing an API call then to our servers it should look the same. Do you know if it’s changing the params / shape of the API call in any way?

3

I have created a test program example to demontrate the error. It is using the python URLLIB package.

import os, json, urllib.request
URL = “https://api.groq.com/openai/v1/chat/completions”
API_KEY = os.getenv(“GROQ_API_KEY”)
HEADERS = {“Authorization”: "Bearer " + API_KEY, “Content-Type”: “application/json”}
PAYLOAD = {“model”:“openai/gpt-oss-120b”,“messages”:\[{“role”:“user”,“content”:“Hello, world!”}\],“temperature”:0.7}
body = json.dumps(PAYLOAD).encode(“utf-8”)
req = urllib.request.Request(url=URL, data=body, headers=HEADERS, method=“POST”)
resp = urllib.request.urlopen(req)
print(resp.read().decode(“utf-8”))

error:

$ python3 test.py 
Traceback (most recent call last):
  File "/home/alec/.config/sublime-text/Packages/Agentic/test.py", line 8, in <module>
    resp = urllib.request.urlopen(req)
  File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.10/urllib/request.py", line 563, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

no error:

# Modify the test program to include User-Agent in HEADERS
HEADERS = {"Authorization": "Bearer " + API_KEY, "Content-Type": "application/json",
	"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"}

$ python3 test.py 
{"id":"chatcmpl-2fc09dff-2c91-4b9c-8fac-107d0d4b8e56","object":"chat.completion","created":1766449633,"model":"openai/gpt-oss-120b","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! How can I assist you today?","reasoning":"The user says \"Hello, world!\" Probably just a greeting. We respond politely."},"logprobs":null,"finish_reason":"stop"}],"usage":{"queue_time":0.004695947,"prompt_tokens":75,"prompt_time":0.002884385,"completion_tokens":36,"completion_time":0.07572427,"total_tokens":111,"total_time":0.078608655,"completion_tokens_details":{"reasoning_tokens":18}},"usage_breakdown":null,"system_fingerprint":"fp_c652c0ffaa","x_groq":{"id":"req_01kd49mpksf0prvs1wzjzyj59h","seed":268796957},"service_tier":"on_demand"}
4

curl works fine for some reason, so something about the python urllib package is being flagged and rejected by cloudflare.

Also, I have experienced this on multiple computers, multiple OS, and muliple networks across different states in the US.

7

The OpenAI API default User-Agent (openai/openai-python/src/openai/_base_client.py) is something like:

'AsyncAPIClient/Python 2.14.0' or 'SyncAPIClient/Python 2.14.0'

for OpenAI API python module version 2.14.0

8

Apparently curl injects a custom user-agent:

curl -vvv …

…
user-agent: curl/7.81.0
…

The python user-agent is being flagged by cloudflare.

urrlib.request source (python/cpython/blob/main/Lib/urllib/request.py):

# used in User-Agent header sent
__version__ = '%d.%d' % sys.version_info[:2]
# ...
client_version = "Python-urllib/%s" % __version__
self.addheaders = [('User-agent', client_version)]

so 'User-Agent': 'Python-urllib/3.10' for example on my system.
[/quote]

10

ohhhh I think I’ve seen this before, is there a way to fake another user agent in python?

If it’s between how Cloudflare handles calls from the standard python API, I’m not sure there’s much we can do.

11

Yeah, I can manually set to another User-Agent. It is annoying that Cloudflare blocks this, but I suppose it makes since that cloudflare blocks this traffic automatically due to the ease of making a web crawler with Python…

12

Yeah Cloudflare seems to build a bigger and bigger wall… but I guess if it’s as easy as setting a User-Agent then maybe it’s not that big after all