[Iplant-api-dev] Running foundation api jobs from condor

[Darren Boss]

My Python code is coming along nicely now and I had a thought about
how to run Bonnie's pipeline from the DE in an interesting way that
should get around many of the road blocks we've been encountering. If
I was able to run a Python script via condor I would be able to launch
Foundation API jobs from within that Python script. I have the code to
monitor running foundation jobs and soon should be able to handle job
dependencies. Bonnie had brought up the issue of where to run very
short utility type portions of the pipeline and if our Python code was
running on condor I would just call out to the executable in the
Python script. In order for this to work the following would have to
be in place
- python on condor with the simplejson and reqests modules installed
- network access to the foundation api server from the condor node
- any utility binaries we call in the python code installed on condor

Does this seem like a reasonable method of execution?

Not required but could possibly speed up execution is the ability to
run additional condor jobs from within the python script. It sounds
like v2 of the api would make this possible but is there a command
line way to make it work right now? Would that work from a process
that was already running in condor? I can do it in sge if I make the
execution hosts submit hosts but I don't have any condor experience.