How to name online framefiles when there is non-integer aggregation latency?

[saleem-muhammed]

I faced this issue with the prototype model I have been using that if we are going with non-integer aggregation latency, the frame file will have to be named decimal points in the GPS times.

For example, the frame that comes from a kernel ending at 1230000000, the 1s frame that we write after allowing an aggregation latency of 0.25, will be named (under the standard naming strategy followed in LIGO) as something like H-H1_HOFT-1229999998.75-1.gwf

which follows H-H1_HOFT-{start}-{duration}.gwf

Earlier it appeared ugly to me and I kept the timestamps to integers as if we did not apply an aggregation latency while it was actually applied (and terribly forgot about it :( ). This is what eventually led to the inconsistencies in the PE tests as the injected signals were not found at the times they were expected to be.

What is the solution in the long term?

[alecgunny]

I don't think this is the case: each frame should correspond to the same GPS start times as the frame it came from. You just won't be able to write that frame until you have an extra .25 seconds worth of data from the future, right?

[saleem-muhammed]

If frames come in 1s durations, you will need to wait for 1s (not 0.25s) to get the next 0.25 seconds worth of data. That is why I thought of the other way: ss soon as a frame is available, use that to clean all but the last 0.25s of that frame and write it to disc which will have a different start time

[alecgunny]

Hmmm ok that's true, but you need to include extra latency to do filtering anyway, and so at that point you're introducing a source of potential complication and confusion over 0.5s worth of latency (since the last 0.5s of each frame won't be able to produce predictions until the next frame comes around anyway). Given the other sources of latency in the pipeline, it may be worth just eating that until we can get closer to the source anyway.

[saleem-muhammed]

That means the latency would be 1s + inference time
(can never have < 1s latency).
If the pipelines like GstLAL can take frames with different naming conventions, then it shouldn't be an issue, I suppose.

[alecgunny]

Yes that's true, but I guess what I'm saying is that once you factor in the filtering, around half the frame has to wait for that 1s anyway, and then has to wait for additional inference latency incurred by the first half of the next frame, and at that point the latency you're saving becomes pretty marginal wrt the other sources of latency in the pipeline.

The downside to moving the frames in production imo has less to do with the naming conventions and more to do with the overhead of coordinating the time offsets between raw strain (or other timeseries for things like data quality flags, injection times, etc.) and the cleaned strain, especially as we try to experiment with various levels of allowed latency and how see how they affect the quality of the clean. I can just imagine ways in which things get complicated pretty quickly, but I could be overthinking things.

[saleem-muhammed]

Maybe we are not on the same page: to have a better sense, would you call 0.5s or 0.25s gain in latency as marginal?

Also, I am not sure I followed what you meant by the time offsets.
I guess, regardless of the inference latency or the other things, the timestamps that go into the frame will always be the GPS times of the raw strain and hence just straightforwardly simple, no?

[alecgunny]

In the scale of the 7-12 seconds those frames are already behind, I would say so. But again you're not getting a free lunch: only some of that data gets a latency reduction, and some of it actually gets worse because now it has to wait for inference on data it otherwise wouldn't have to wait for.

In fact, if you've chosen your inference sampling rate such that you're saturating the throughput capacity (which I think we want to do, or at least shoot to do in the future), so that you're processing close to 1s of data per second, you don't end up gaining anything, because while the first 0.5s of data of a frame gets written 0.5s after that frame becomes available, the second 0.5s gets written 1.5s after that frame became available, so it ends up washing out.

But I think I'm agreed on your second point, I guess I'm just worried about errors or hacky code to handle realignment (e.g. something like what happened with the PE issue), but I suppose it should be pretty straightforward. It does make crawling through frames for monitoring more difficult though: if I have a raw strain file with some associated timestamped name, I either need to know the aggregation latency to find its cleaned analogue, or search through all the filenames in a folder until I find the two frames containing it, neither of which seems particularly robust.

In any case, the async file writer I've built and am testing uses the original timestamps and doesn't introduce and offset, so at least for the short term I'll probably keep things that way. However, this is worth keeping as an open discussion for a future PR if we end up not being happy with the associated latencies.

[saleem-muhammed]

Need to think more on your first two paragraphs. I agree with the later part and we should keep this issue open and revisit in future once we have everything working with 1s (+inference) latency.

Just to be clear, the way you write frames now take care of the times by themselves (in other words, you don't have to remember that you have applied an aggregation latency on them), right?

[alecgunny]

Ok cool sounds good.

More or less yeah. What happens in practice is that the postprocessor/writer throws away the first aggregation_latency // kernel_stride responses from the server, since with the delay introduced by aggregation those technically correspond to predictions from the past. It can then proceed as normal from there.

The aggregation_latency used to parameterize the number of thrown away responses is fed as a command line parameter at the moment, but I'm thinking of reformulating things to have that saved as metadata on the inference server that can be requested by client pipelines at inference time.