zarr in forcingprocessor #17
Description
Forcingprocessor was originally designed to serve as the forcings generation for the NRDS system. This motivated the design choice to optimize for large domain (>10,000 catchments), short time (<1000 time steps) processing of NWM forcing files. Forcingprocessor's internal algorithm performs a multi processed read and processing of each netcdf required. This is performant at scale, but would be considered a network heavy solution in the context of small domain (<100 catchments) simulations. Also, forcingprocessor is not currently capable of processing anything other than the NWM forcing files provided by nwmurl.
From here, we can implement algorithms (that are already in related tooling) that will generalize forcingprocessor to use the most efficient internal algorithm based on the domain size and time step count. For example, if forcingprocessor starts up with a large geopackage and a few number of nwm files listed in filenamelist.txt, then the original algorithm is used. However if a small geopackage and a large number of nwm files, forcingprocessor could then switch to a zarr based algorithm that is more suitable in that case.
Action items:
- Implement zarr algorithm in forcingprocessor with switch based on geopackage and filenamelist size.
- Implement additional forcing sources like AORC, this may motivate creating another tool similar to nwmurl, but perhaps aorc_url or something more generally named so it could be used for any forcing source (i.e. forcing_url).
If these two are implemented, forcingprocessor could serve as a general purpose forcings tool that can be integrated in any workflow requiring forcings generation.