[DRAFT] feat: pipeline from_query supports cursors #1149
Conversation
product-auto-label
bot
added
the
api: firestore
Summary of ChangesHello @daniel-sanche, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a powerful new 'pipeline' feature to the Firestore client library, enabling complex data transformations and queries through a series of chained stages. It provides both synchronous and asynchronous APIs for building and executing these pipelines, integrating existing query and aggregation functionalities into this new framework. This significantly enhances the flexibility and expressiveness of data manipulation within Firestore. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
product-auto-label
bot
added
size: m
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for cursors and limit_to_last in pipeline from_query. The implementation looks promising, but I've identified a couple of critical correctness bugs in the logic. One is related to the incorrect ordering of pipeline stages, where a sort stage is added before a where stage for cursors. The other is in the cursor condition generation, which doesn't account for the sort direction. I've provided detailed comments and code suggestions to address these issues. Additionally, the unit tests for the new cursor functionality could be more specific to catch such bugs in the future. Overall, this is a great feature addition, and with these fixes, it should be solid.
| # If limit_to_last is set, we need to reverse the orderings to find the | ||
| # "last" N documents (which effectively become the "first" N in reverse order). | ||
| if self._limit_to_last: | ||
| actual_orderings = _reverse_orderings(orderings) | ||
| ppl = ppl.sort(*actual_orderings) | ||
| else: | ||
| ppl = ppl.sort(*orderings) | ||
|
|
||
| # Apply cursor conditions. | ||
| # Cursors are translated into filter conditions (e.g., field > value) | ||
| # based on the orderings. | ||
| if start_at_val: | ||
| ppl = ppl.where( | ||
| _where_conditions_from_cursor( | ||
| start_at_val, orderings, is_start_cursor=True | ||
| ) | ||
| ) | ||
|
|
||
| # Cursors, Limit and Offset | ||
| if self._start_at or self._end_at or self._limit_to_last: | ||
| raise NotImplementedError( | ||
| "Query to Pipeline conversion: cursors and limit_to_last is not supported yet." | ||
| ) | ||
| else: # Limit & Offset without cursors | ||
| if self._offset: | ||
| ppl = ppl.offset(self._offset) | ||
| if self._limit: | ||
| if end_at_val: | ||
| ppl = ppl.where( | ||
| _where_conditions_from_cursor( | ||
| end_at_val, orderings, is_start_cursor=False | ||
| ) | ||
| ) |
There was a problem hiding this comment.
The order of pipeline stages is incorrect. The sort stage is being added before the where stage for cursor conditions. To ensure correctness and for performance reasons, all filtering (where stages) should be applied before sorting. The block for applying cursor conditions should be moved before the block for sorting.
# Apply cursor conditions.
# Cursors are translated into filter conditions (e.g., field > value)
# based on the orderings.
if start_at_val:
ppl = ppl.where(
_where_conditions_from_cursor(
start_at_val, orderings, is_start_cursor=True
)
)
if end_at_val:
ppl = ppl.where(
_where_conditions_from_cursor(
end_at_val, orderings, is_start_cursor=False
)
)
# If limit_to_last is set, we need to reverse the orderings to find the
# "last" N documents (which effectively become the "first" N in reverse order).
if self._limit_to_last:
actual_orderings = _reverse_orderings(orderings)
ppl = ppl.sort(*actual_orderings)
else:
ppl = ppl.sort(*orderings)There was a problem hiding this comment.
I think this is partially true. When limit_to_last is set, it needs to sort both before and after though. I made that change
product-auto-label
bot
added
size: l
1 participant
WIP
TODO:
Existsstatements. Blocked on revised sample code