Online inference: noise added by DeepClean

[saleem-muhammed]

There are a few initial steps for systematically debugging this

• Make sure the data is properly read. Enforce the code to read the original data as well that is in the InMemoryDataSet and make sure it is consistent with the original data
• Write to the raw prediction of the noise on a small segment and a large segment and see what is the difference.
• Do the same on the offline testing step as well. This will show if the prediction before any postprocessing agrees with test results.
• If they agree, check for what sizes of the edges are they not in agreement
• If enough edge is discarded, then check the data after postprocessing and do this in both online and offline testing

[saleem-muhammed]

Make sure the data is properly read. Enforce the code to read the original data as well that is in the InMemoryDataSet and make sure it is consistent with the original data

The original data is the one that is not passed through the pipeline at all.
Unclean is the one that was sent to the pipeline for estimating the noise (specifically for doing the operation: cleaned = unclean - prediction). Though this is not cleaned, this has gone thorough the preprocessing and postprocessing steps and need to check if they have caused some issues.

Ideally, both unclean and original are supposed to be the same but we see a difference.

• Is it due to a bandpass?
• do we need to preprocess the original data at all except resampling?
• Is there a difference if we do not send the strain to the inmemorydataset?

[saleem-muhammed]

Removed the preprocessing and post-processing steps for the strain not needed. This means that the strain data is not scaled or de-scaled. The outcome does not change!

[saleem-muhammed]

Set up a simplistic model without the buffer and edge issues:

• Read a fixed 128s data from the low latency frames without using any crawler and buffer
• Preprocess and clean it
• discard the first and last one seconds (or more if needed)
• If this works, try it for any 3 seconds from the 128 s and compare the outcome. Get a sense of the size of the edge to be discarded
• if not, the problem must be with the postprocessing as that is where we differ with the offline code as we do not use overlapping kernels and Hann windows.
• If needed, try using them

[saleem-muhammed]

the strange behaviour in the asd of the unclean data above was caused by the precision issue - the same issue we has also encountered in the training problem. The updated version of the plot is here now

[saleem-muhammed]

The problem has now been resolved. It was due to sending the witness channel data in the wrong order to the model during inference. In the frame iterator, the channels are taken in the order it is given to (no sorting happens inside) whereas the channels being sent to the training is sorted. This has now been fixed.