[Iplant-api-dev] File Inputs in Agave

[Rion Dooley]

It will stage whatever URI you give it.

You can pass in a url such as those below. In both cases, whatever is returned from that url will be staged as a file to the job work directory. Any redirects will be followed. non 2xx responses will result in a failed transfer, thus killing the job and sending an appropriate notification. Be careful, if the server does something tricky, a server response may be the file contents.

* http://bit.ly/V5UwoD
* https://bit.ly/V5UwoD

You can pass in an agave URI for authenticated systems.

* agave://data.iplantcollaborative.org/dennis/simple.fasta
* agave://qairods.iplantcollaborative.org/dennis/simple.fasta

Both storage and execution systems are valid. Many people will simply put their inputs and/or application data on the execution system to get better throughput since Agave will do server side copies whenever possible. Files as well as folders will be copied, so pass whatever works for you.

* agave://desertrat.example.com//mnt/fuse/dennis/simple.fasta
* agave://desertrat.example.com//mnt/fuse/dennis/jobs/daily/2014-08-07

You can use absolute or relative paths in leu of full URIs. They will resolve to your default storage system (data.iplantcollaborative.org<http://data.iplantcollaborative.org> in most cases) and the paths will be staged. The following are equivalent:

* dennis/simple.fasta => agave://data.iplantcollaborative.org/dennis/simple.fasta
* /dennis => agave://data.iplantcollaborative.org/dennis

If you need to pass in multiple inputs for a single input attribute, then you can semi-colon delimit them. Mixing and matching of URI and paths is fine. The following examples below should all be valid.

* dennis/fasta/foo1.fasta;dennis/fasta/foo2.fasta;shared/fasta/bar.fasta
* dennis/fasta/foo1.fasta;agave://data.iplantcollaborative.org/dennis/fasta/foo2.fasta;shared/fasta/bar.fasta
* agave://data.iplantcollaborative.org/dennis/simple.fasta;;;;;;;;;;;;;;;;;;;;agave://desertrat.example.com//mnt/fuse/dennis/jobs/daily/2014-08-07



FWIW, It’s on my short list to do this properly by supporting arrays, but I just haven’t got it tested thoroughly yet. Probably a week or so and it will be in production.


--
Rion




On Aug 7, 2014, at 5:14 PM, Dennis Roberts <dennis at iplantcollaborative.org<mailto:dennis at iplantcollaborative.org>> wrote:

This may be stupid questions. I apologize if I’m missing something obvious.

I haven’t been able to find this in the documentation yet, but does Agave support passing the path to a directory to an app in order to process all of the files or some subset of the files in that directory?

Also, I’ve noticed that Agave apps seem to support input parameters with cardinalities greater than 1, but I didn’t see an example of passing multiple file paths in. If the cardinality of an input can be greater than one should I pass an array in the job submission JSON? Perhaps something like this:

https://gist.github.com/dennisroberts71/44db8ebcb5b5a2680b64

Thanks,
Dennis

_______________________________________________
Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org<mailto:Iplant-api-dev at iplantcollaborative.org>
List Info and Archives: http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
One-click Unsubscribe: http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/dooley%40tacc.utexas.edu?unsub=1&unsubconfirm=1

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.iplantcollaborative.org/pipermail/iplant-api-dev/attachments/20140807/26e2d214/attachment.html