Skip to content

Conversation

@aviau
Copy link
Member

@aviau aviau commented May 15, 2025

  • Add scroll_events which includes the logic to call an an events api with all uids returned from a feed api.
  • We will be able to use this exact method even when we add a new events api.
  • This would be our only method that includes ratelimiting, we have no choice but to implement it ourselves since we don't pass control back to the user (because of empty pages).
  • For now, I created a basic ratelimiter, but I kept it as a private argument. So this assumes that the users do not use this in two threads, and that they are happy with our default ratelimits.

I don't want to expose the limiter arg as public right now because I think in the future this can just be replaced by reading ratelimit headers from our API. So, for now, I focus on supporting this simple common use case. We can revisit when we get more feedback and after we have implemented ratelimit headers.

This allow us to replace this example by:

import datetime

from flareio import FlareApiClient


api_client = FlareApiClient.from_env()

from_timestamp: str = (
    datetime.datetime.now(tz=datetime.timezone.utc) - datetime.timedelta(hours=1)
).isoformat()

for event, next in api_client.scroll_events(
    method="POST",
    pages_url="/firework/v4/events/global/_search",
    events_url="/firework/v2/activities/",
    json={
        "size": 10,
        "order": "asc",
        "from": last_from,
        "filters": {
            "type": ["chat_message"],
            "estimated_created_at": {"gte": from_timestamp},
        },
        "query": {
            "type": "query_string",
            "query_string": "hello",
        },
    },
):
    print(f"Full event data: {event}")
    print(f"The next execution could resume using from={next}")

@aviau aviau force-pushed the aviau/scroll_events branch 12 times, most recently from c9162c5 to c4a6802 Compare May 15, 2025 12:49
@gpratte-flare
Copy link

gpratte-flare commented May 15, 2025

lgtm!

@aviau aviau force-pushed the aviau/scroll_events branch from c4a6802 to 64a94c5 Compare November 19, 2025 22:44
@aviau aviau merged commit f510fb2 into main Nov 19, 2025
8 checks passed
@aviau aviau deleted the aviau/scroll_events branch November 19, 2025 23:12
@aviau aviau restored the aviau/scroll_events branch November 19, 2025 23:15
@aviau aviau deleted the aviau/scroll_events branch November 19, 2025 23:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants