Fix NaN propagation in rate averages and cost totals (v2.0.1) #59
+50
−10
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
When simulating periods with sparse missing data (e.g., 30 half-hour slots out of 17,520 in a full year), the entire summary would show NaN values for rates and costs. This made annual simulations unusable when even 0.17% of imbalance pricing data was missing.
Root cause
Both aggregation paths used
skipna=False, causing any NaN to propagate:output.py:safe_average()usednp.average()which returns NaN if any input value is NaNbreakdown.py: cost totals used.sum(skipna=False).sum(skipna=False)which returns NaN if any cell is NaNSolution
Add a 5% NaN threshold to both aggregation functions:
safe_average(): Filter out NaN values and their weights before calculating weighted average. Only return NaN if >5% of data is missing.safe_sum(): New helper function that usesnp.nansum()to sum valid values. Only return NaN if >5% of data is missing.This allows simulations to complete with valid results when small amounts of data are missing, while still flagging unreliable results when too much data (>5%) is absent.
Files changed
output.py: Enhancedsafe_average()with NaN thresholdbreakdown.py: Addedsafe_sum()helper, replaced double sum callspyproject.toml: Bump version to 2.0.1Test plan