-
Notifications
You must be signed in to change notification settings - Fork 501
fix(cost-model): add true above-diagonal samples to valueContains benchmark #7527
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
…chmark Two different "diagonal" concepts were confused: 1. Value structure diagonal (numPolicies vs tokensPerPolicy) 2. Costing diagonal (container totalSize vs contained totalSize) The previous `aboveDiagonalCombos` tested #1 but the cost model needs #2. Changes: - Remove misleading `aboveDiagonalCombos` variable - Add true above-diagonal samples where contained size > container size - Update comments to clarify both diagonal concepts - Generate 3 independent (container, contained) pairs for the constant-cost region Closes #7525
cc3fe66 to
4c8c0c3
Compare
kwxm
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's have a few more points above the diagonal!
| Grid: 10×10 power-of-2 below-diagonal (55 combos) | ||
| Samples per container: 10 evenly distributed contained sizes | ||
| Total points: ~570 | ||
| Above-diagonal: 3 independent (container, contained) pairs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe 10 or 20 with some of them containing larger values would be better, just to make sure that we get a decent average: 3 points seems a bit small.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
Kenneth requested more data points above the diagonal for a better average of the constant-cost region. Changes: - Increased from 3 to 20 above-diagonal samples - Added variety in container/contained size ratios: * Small containers (4-16 entries) with larger contained * Medium containers (25-100 entries) with larger contained * Large containers (150-1024 entries) with much larger contained - Total benchmark points: ~570 (550 below + 20 above diagonal)
Updated valueContains cost model parameters based on benchmark run with expanded above-diagonal coverage (3→20 samples). Cost parameter changes: - Above-diagonal constant: 233391 → 246085 (+5.4%) - Below-diagonal intercept: 658617 → 686640 (+4.3%) - Slope parameters adjusted for better fit
Summary
Fixes #7525
The
valueContainscost model usesconst_above_diagonalwhich requires benchmark data in two regions:However, the benchmark function only generated samples in the below-diagonal region (contained was always a subset of container).
Changes
Benchmark Generation
aboveDiagonalCombosvariable (referred to Value structure diagonal, not costing diagonal)Cost Model Updates
builtinCostModelA/B/C.jsonwith refined parameters:benching-conway.csvwith new benchmark data (~570 total points)