Seed analysis #127
Seed analysis #127
Conversation
wbenoit26
left a comment
wbenoit26
left a comment
There was a problem hiding this comment.
Not sure if you were planning on adding more to this, but I think it looks good, just had a couple questions
| # which has is enough memory to write large temp | ||
| # files with luigi/law | ||
| env["TMPDIR"] = f"/local/{os.getenv('USER')}" | ||
| env["TMPDIR"] = os.getenv("AFRAME_TMPDIR") |
There was a problem hiding this comment.
Should mention this variable in the README along with the others that you suggest putting into a .env
|
|
||
| def requires(self): | ||
| seeds = np.random.randint(0, 1e5, size=self.num_seeds) | ||
| for seed in seeds: |
There was a problem hiding this comment.
Does this mean that the different seeds are done sequentially? Or does that scheduling happen sequentially and all the training/inference happens in parallel?
There was a problem hiding this comment.
So they'll still get scheduled in "parallel" , which is actually sort of problematic at inference time when they'll be fighting for gpus.
This is mostly why i'm holding off on merging this. Should find a cleaner solution.
| environment += f'PATH={os.getenv("PATH")} ' | ||
| environment += f"LAW_CONFIG_FILE={self.law_config} " | ||
| environment += f"USER={os.getenv('USER')} " | ||
| environment += f"TMPDIR={os.getenv('TMPDIR')} " |
There was a problem hiding this comment.
Should this be AFRAME_TMPDIR?
2 participants
Adds a
pipelinefor running multiple training jobs using different seeds.