Score calculation relative to what?

[krejciadam]

Hi!
I've noticed some rather odd results returned by cell-eval. So I experimented on a tiny examples and on those, it seems that MAE is always calculated relative to non-targeting cells in the reference cells object. At the same time, "overlap_at_N" and "discrimination_score_l1" are calculated relative to reference cells in the predicted cells object.

I'm using cell-eval run 0.6.0

cell-eval run -ap predicted.h5ad -ar true.h5ad --num-threads 64 --profile vcc

non-targeting cells in "predicted.h5ad" are completely disregarded for MAE calculation, and MAE is always relative to non-targeting cells in "true.h5ad". At the same time, non-targeting cells in "predicted.h5ad" are used to calculate "overlap_at_N" and "discrimination_score_l1".

I can provide a toy example demonstrating this.

Do I understand this correctly and is it the intended behavior? It just feels wrong to have different scores calculated on different baselines. Frankly, it fells wrong to have any score calculated based on predicted wild-type cells and not real ones. Thank you!

[KatarinaYuan]

Would jump in this issue and call for attention from the organizers.

Because the repo does not guarantee that the control cells in predicted adata is the same as the control cells in true data, this leads to a severe flaw where people can twist the control cells in the predicted adata to gain better "performance".

[noamteyssier]

Yes this is intended behavior. See my comment in #183 (comment)

[KatarinaYuan]

Hi @noamteyssier , thanks for the quick response and the comment. I still have some questions on this. Please see #183 (comment)