what is the reason for "fix precision"?

[XingyuLu206]

In the newest code, why the dtype for loading models is changed to float32 rather than bfloat16?

[cgq15]

Based on our experiments, loading PRM in bf16 may cause convergence issue, so we set it to fp32 by default.

[XingyuLu206]

Should we also use float32 for the ActorRolloutRefWorker?

[cgq15]

I think it's not necessary, we are still validating the effect of bf16 on PRM. On larger models (32B) it also converges well