PBS with headnode

[matthias-k]

Hi,

I would like to use ipython-cluster-helper on a cluster running PBS. The problem is that I have to ssh to the head node to be able to use qsub and qdel. The file system is the same as on my machine (nfs). Is there some easy way to accomplish this? I know that ipython.parallel is able to use ssh somehow, but I am not sure how to combine this with PBS. As ipython-cluster-helper improves ipython.parallel so much, is there some way of achieving this that I missed so far? If not, how could I build it into ipython-cluster-helper?

Best,
Matthias

[roryk]

Hi Matthias,

I don't have a great answer, I'd need to investigate it some more. I think something would have to sit on top of ipython-cluster-helper to do this; you'd have something that would ssh in and launch a cluster with ipython-cluster-helper and return the ip address and port that the controller is listening on that you can use to connect a client to from the outside. It is likely though that your machine won't be able to talk to the IPython cluster sitting behind the head node, so you'd have to set up a ssh tunnel through the head node to be able to connect to it. It might be possible to do something like this automatically with a custom profile, I haven't poked around at IPython 3 enough. starcluster has an implementation of doing what you were describing as well: http://star.mit.edu/cluster/docs/latest/plugins/ipython.html

[basnijholt]

@matthias-k did you get any further with this? We need the exact same thing and it will be really helpful to use this :)

[roryk]

Once an ipcluster is up, you can connect to it via opening up an outside client if you know what ipaddress it is running on. It might take a bit of mucking about to do but you could set up a profile to do it, some instructions are here: https://ipython.org/ipython-doc/dev/parallel/parallel_process.html. We don't have support currently for using a previously existing profile, but it would be easy to add. Then you could set it up so it is working with the profile and then just tell ipython-cluster-helper to use that profile for the details, I think that would get you there.

[tkoomar]

I apologize for what is essentially a bump to an old issue, but I want to assure you, @roryk that this feature would be greatly appreciated by anyone working on a cluster where qsub is only allowed on login or head nodes. My SGE sysadmins assure me the solution is as simple as " just use ssh $SGE_O_HOST 'cd $PWD; qsub next_job' ". If only . . .