Zenodo call throws error due to request rate limit

[pabloitu]

Zenodo requests causes error in request json parsing

floatcsep/floatcsep/utils/accessors.py

Line 23 in 5b3e00f

r = requests.get(f"https://zenodo.org/api/records/{record_id}", timeout=30)

download_urls = [f["links"]["self"] for f in r.json()["files"]]
                                                 ^^^^^^^^

    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 2 column 11 (char 11)

This is caused by a HTML reply rather than a JSON, due to Zenodo limiting
rate requests (https://blog.zenodo.org/2025/11/25/2025-11-14-search-api-updates/). This causes tests and tutorials to usually fail. This might be due to multiple model flavours calls in the same zenodo repository in quick succession, which label the IP to be causing suspicious traffic.

[mherrmann3]

Not cool, but fair enough. The actual culprit should be this limit:

Request frequency: Maximum 30 requests per minute

Right? So could a simple time.sleep(2) [2 secs] before the request call solve this issue? To max out the rate limit, I could imagine a tracker of the current request rate, so that the sleep time could be adapted accordingly. Maybe not necessary.
How many single request do the test suite and tutorials perform?

[pabloitu]

Thank you Marcus for checking! This issue came from a long time, in which also I had to "skip" the test of this tutorial in the CI.

The main culprits were:

• If models came from the same repo, the first model downloaded everything, and then the other models skipped the download (doesnt make too much sense).
• Having too many simultaneous requests, such as getting all files from a repo at once and get the HEAD (i.e., content size) to provide status bars. This was acceptable for "local" runs in the past, but the CI was always doing multiple requests from all py versions/os, so I skipped CI tests. However, Zenodo reduced the request rate to 30/min, so now even local fails.

I implemented your suggestions, reduced the amount of tutorial d) files, skipped all requests that were meant to get the HEAD and handled some server errors. Those changes were included in #118, and works okay now :D! THank you

[mherrmann3]

Great solution! It looks very adaptive to me and generally optimized :-)

I'm not sure, but I think you don't have to drop progress bars, since additionally requesting the HEAD was only supposed to get the total file size, correct?

I wonder if you could simplify download_file() further by passing the file size, which is already returned with the initial request. You can get it with:

download_sizes = [f["size"] for f in files]

Just an idea.