Monte-carlo algorithm

[mw2b]

Overview

The imbalance price is inherently hard to forecast directly because it changes significantly if the system moves between long and short states. I.e. the imbalance price semi-stochastically moves between the 'long price' and the 'short price'.

However, the 'long price' and the 'short price' themselves should be quite forecastable - based on standard price forecasting techniques based on wind data, gas prices, load forecasts, generation forecasts etc.

A monte-carlo algorithm would use the existing linear programming optimisation to execute many optimisation runs, each with different assumptions about when the system will be long and short. This could be as simple as assuming the system switches between long and short states every [30mins, 1hr, 2hr, 4hr, etc].

The monte-carlo algorithm would then look at the result from each of these optimisations runs to determine the best action in the current settlement period (statistically speaking).

There are many benefits to an approach like this:

• The system will know how to hedge its bets - i.e. is it better to charge overnight at 6p, or is it better to wait in case we get negative pricing due to solar in the day?
• DUoS payments are also 'automatically understood' by the LP optimisation (no need to dedicated 'peak approach logic').
• P415 payments can be understood by the LP optimisation by adding additional rates at the wholesale price. The LP optimisation could also quite easily be extended to understand that it has the option to not follow the P415 schedules - in some circumstances it might be better to go harder than the P415 schedule, or even back out of the P415 schedules, in order to maximise imbalance payments.
• Solar self-use and OSAM is understood by the LP optimiser
• The algorithm could easily integrate real-time imbalance price calculations from Modo for the current settlement period, whilst using forecasts for future settlement periods

Design

As illustrated below, a set of forecasters are required for:

• The 'short' imbalance price
• The 'long' imbalance price
• The expected solar generation (this could be added later)

These forecasts would then be fed into the monte-carlo algorithm, which would use the LP optimiser to make predictions

The monte-carlo algorithm would run scenarios assuming different combinations of long and short. In the illustration below:

• the red "S" means it's assumed the system will be short
• the green "L" means it's assumed the system will be long
• The red dotted line is the forecast short price
• The green dotted line is the forecast long price
• The black line is the price that is assumed - it flips between the forecast long and short prices

The black line is the price which is assumed for the optimisation run - many optimisations would then be run with different price assumptions.

Outturn short + long prices

The two imbalance price forecasters discussed above would need to be developed. To build a forecaster for these prices you'd first need the calculate the actual (outturn) prices and then you can try different techniques/models to forecast the outturn.

The outturns we need to calculate is not just the imbalance price: we need two prices (long and short) for every settlement period. The outturns we calculate could be validated by plotting alongside the imbalance price - the imbalance price should jump between the long and short prices that we calculate.

Modo's website has code snipits for how they calculate the 'reference price' from the datasets they host. By removing "SO flagged actions" from the reference price you can get pretty close to the short + long prices.
I tried this on the monte-carlo branch of skypro here: https://github.com/cepro/skypro/blob/monte-carlo/src/skypro/commands/simulator/algorithms/monte/reference_prices.py

There may be a better way to calculate these outturn prices - Elexon's guide to calculating the imbalance price is here, but it's a heavy read: https://bscdocs.elexon.co.uk/guidance-notes/imbalance-pricing-guidance

Once the outturns are known, then a forecaster could be build - perhaps using Sam's final year project code as inspiration - using EPEX intra-day prices, wind forecasts, gas prices, etc. There is quite a lot of data on the Elexon Insights portal.

However, before spending a lot of time developing accurate forecasts, it may be worth testing the principles of the monte-carlo algorithm by first feeding in the outturn short/long prices. This would be be a 'perfect hindsight' scenario, where we have perfect long and short price forecasts, but would allow us to validate the monte-carlo logic. If the monte-carlo logic gives good results with perfect forecasts, then we can circle back and develop the forecasters. This also allows us to see how accurate the forecasts need to be for good results - it might be that a 2p/kWh average error on the price forecasts get us 90% of the way there.

Existing work in progress

A scrappy work-in-progress for a monte-carlo algorithm is on a branch here: https://github.com/cepro/skypro/tree/monte-carlo/src/skypro/commands/simulator/algorithms/monte

It quickly became apparent the the number of optimisations required is large, and therefore slow to run. In the above I used a ProcessPoolExecutor to run optimisations in parallel, but more work is likely required to either simplify the LP optimisation problem (i.e. less variables, more constraints, less granular time intervals, smaller time frame, etc) or run less optimisations (i.e. do we need to run 10 sim scenarios, or 100, etc).

We can probbly tolerate a fairly slow system - it would not be a problem in real-time control as it doesn't really matter if a monte-carlo run takes 1 minute to run. But for running simulations over a year it could become a pain. Another option for larger simulation runs is to offload the job to a cloud resource with lots of cores. You can get one optimisation scenario per core - my laptop has 8 so can run 8 at a time, but you can get an AWS machine with 192 cores for 10$/hr.