Write the model in a parallel framework which we do not need to maintain

[BarKetPlace]

I found this library:
https://github.com/williamFalcon/pytorch-lightning

As I understand it, they make it easy to parallelize torch models.

[FirstHandScientist]

the library is really a nice tool. Saving our hand-crafted parallel interfaces.