Added collectNumberOrderedElements #45
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
benraha
changed the title
Contributor
|
Did you specify that DataFu build with Spark 3.3 or 3.4? I think your PR assumes a newer interface of DeclarativeAggregate than what we have currently, and that's why the build is failing in our CI. I'm planning on pushing code that will upgrade us to these versions, so that will probably make your PR pass tests. |
Contributor
|
Looks like after the last commit our CI passes, so now this works for Spark 3.0.x - 3.4.x. I'll merge it in. Great job! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In a lot of our solutions, we select only a fixed number of rows, based on ordering by a column, usually a small amount. Datafu has dedupTopN, which uses a window function, and dedupWithCombiner, which is limited to only taking one record per grouping. dedupTopN is using a window function, which is inefficient because it orders all of the rows per group, and is very susceptible to skew. DedupWithCombiner won't let us take more than one row.
This PR introduces a solution - a class that implements DeclarativeAggregate, to avoid declaring the schemas explicitly and using the combiner to avoid skew and Codegen.