[Iplant-api-dev] Running foundation api jobs from condor

[Rion Dooley]

V2 will let you do this out of the box on Condor, atmosphere, a random cm, whatever. The quick and dirty solution you described is how api jobs are run in the DE now.

-
Rion

----- Reply message -----
From: "Darren Boss" <dboss at email.arizona.edu>
To: "Rion Dooley" <dooley at tacc.utexas.edu>
Cc: "iplant-api-dev at iplantcollaborative.org" <iplant-api-dev at iplantcollaborative.org>
Subject: [Iplant-api-dev] Running foundation api jobs from condor
Date: Tue, Jul 16, 2013 1:41 PM

My Python code is coming along nicely now and I had a thought about
how to run Bonnie's pipeline from the DE in an interesting way that
should get around many of the road blocks we've been encountering. If
I was able to run a Python script via condor I would be able to launch
Foundation API jobs from within that Python script. I have the code to
monitor running foundation jobs and soon should be able to handle job
dependencies. Bonnie had brought up the issue of where to run very
short utility type portions of the pipeline and if our Python code was
running on condor I would just call out to the executable in the
Python script. In order for this to work the following would have to
be in place
- python on condor with the simplejson and reqests modules installed
- network access to the foundation api server from the condor node
- any utility binaries we call in the python code installed on condor

Does this seem like a reasonable method of execution?

Not required but could possibly speed up execution is the ability to
run additional condor jobs from within the python script. It sounds
like v2 of the api would make this possible but is there a command
line way to make it work right now? Would that work from a process
that was already running in condor? I can do it in sge if I make the
execution hosts submit hosts but I don't have any condor experience.
_______________________________________________
Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
List Info and Archives: http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
One-click Unsubscribe: http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/dooley%40tacc.utexas.edu?unsub=1&unsubconfirm=1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.iplantcollaborative.org/pipermail/iplant-api-dev/attachments/20130716/0a1dff51/attachment.html