[Iplant-api-dev] /apps-v1/job/

[Joe Stubbs]

Hi Darren,

I was actually going to suggest the requests library; it is definitely a much friendlier API than urllib2.  Let me know if you have any trouble, and I will do my best to help.

Joe


________________________________________
From: Darren Boss
Sent: Thursday, July 11, 2013 12:10 PM
To: Rion Dooley
Cc: iplant-api-dev at iplantcollaborative.org; Joe Stubbs
Subject: Re: [Iplant-api-dev] /apps-v1/job/

I've installed the request module
(http://docs.python-requests.org/en/latest/index.html) and finding it
mush easier to use. With the job submission I was having trouble
accessing the json returned when I received an HTTP error using the
urllib2 library because an exception was thrown and I couldn't figure
out how to access the response. I think this will simplify my code and
speed development tremendously.

I'll post again if I get stuck with the reuqest module or if I have
something worth sharing with other developers.

On Thu, Jul 11, 2013 at 12:42 PM, Rion Dooley <dooley at tacc.utexas.edu> wrote:
> You can go by my email outlining the job service or try clearing your cache and refreshing the documentation page. In response to your earlier email, I updated the documentation with the most recent information. v1 is frozen now, so this won't change. I'm cc'ing Joe Stubbs on this email as he's also writing up a job submission routine in Python, though probably with a smaller scope than you taking on. Perhaps you two can be of help to each other.
>
> Rion
>
>
> ________________________________________
> From: Darren Boss [dboss at email.arizona.edu]
> Sent: Thursday, July 11, 2013 11:12 AM
> To: Rion Dooley
> Cc: iplant-api-dev at iplantcollaborative.org
> Subject: Re: [Iplant-api-dev] /apps-v1/job/
>
> I'm still confused about what attributes should be set in the job
> request. In the api docs it says:
>
> Parameter Role Required Notes
> application $appname-$version Yes Lexical name of an application in
> the app registry
> name Job name No String. Human-readable name for this job. Defaults to
> value in "application"
> processors Integer > 0 No How many processors to request
> memory String No Minimum memory required to complete the job. Uses
> same semantics as SGE l_vmem reservations. Example: 8G
> jobPostbackURL String, URL No Post status of job to this URL
> archiveOutput Boolean No If 'true' then copy job/output to $IPLANTHOME/x-name/
> archivePath String No Copy output files here instead of $IPLANTHOME/x-name/
> parameters     These parameters are devised by inpsecting the inputs,
> parameters, and outputs sections of the APPS description for
> $appname-$version
>
> In the php wireframe I see the code:
> //TODO CHANGE APPLICATION NAME TO MAKE SURE PROPER SOFTWARE IS CALLED
> $postField ="softwareName=APPNAMEHERE&archive=true";
>
> So is it application or softwareName?
>
> On Thu, Jul 11, 2013 at 11:04 AM, Rion Dooley <dooley at tacc.utexas.edu> wrote:
>> Thanks for letting us know about the errors on the documentation page. I have updated the page with appropriate changes.
>>
>> The v1 /job endpoint takes a standard form submission. File uploading is not supported on that resource, so multipart is not needed.
>>
>> Thanks for all your hard work putting together the library. Hopefully it will be help to you and the community.
>>
>> cheers,
>> Rion
>>
>> ________________________________________
>> From: Darren Boss [dboss at email.arizona.edu]
>> Sent: Thursday, July 11, 2013 9:29 AM
>> To: Rion Dooley
>> Cc: Discussion of iPlant API development
>> Subject: Re: [Iplant-api-dev] /apps-v1/job/
>>
>> There is an error in the docs regarding the url for returning
>> historical jobs. It indicates that the endpoint is /apps-v1/job/ with
>> a GET request but it should be /apps-v1/jobs/list.
>>
>> On that same page there is a section that states:
>>
>> You can also create a request to the Job manager via HTTP POST of
>> Content-type application/json to the /job-v1 endpoint. This supports
>> programmatic access to the API from applications that might write JSON
>> natively. Here's the format for that method, with specifics for the
>> "head" application.
>>
>> What endpoint is this referring to? If I call up
>> https://foundation.iplantcollaborative.org/jobs-v1/usage I get a
>> resource not available message so that's not it and I don't see
>> /apps-v1/jobs-v1 in the apps-v1 usage list. Is this not implemented? I
>> can very easily construct the json from Python dictionaries  in my
>> code so this would be very nice to use.
>>
>> The reason why I'm trying to use this endpoint is I'm having trouble
>> constructing the "Content-type: form/multipart" POST and sending an
>> authenticated request for the /apps-v1/job endpoint. If you think that
>> x-www-form-urlencoded (the default) should work as well I'll
>> reevaluate my code to see if I goofed up somewhere else. I think once
>> I get job submission working the remaining method implementations
>> should be trivial.
>>
>> On Wed, Jul 10, 2013 at 11:27 PM, Rion Dooley <dooley at tacc.utexas.edu> wrote:
>>> Hi Darren,
>>>
>>> Thanks for following up on the list. I believe it's just a matter of finding
>>> the right v1 urls. The apps, jobs, and systems service urls are listed in
>>> the v1 docs website at:
>>>
>>> https://foundation.iplantcollaborative.org/docs
>>>
>>> The service also provides its own (very basic) usage page with a list of
>>> valid urls:
>>>
>>> https://foundation.iplantcollaborative.org/auth-v1/usage
>>> https://foundation.iplantcollaborative.org/apps-v1/usage
>>> https://foundation.iplantcollaborative.org/io-v1/usage
>>> https://foundation.iplantcollaborative.org/postit-v1/usage/
>>> https://foundation.iplantcollaborative.org/profile-v1/usage
>>>
>>> In your case, the relevant operations for the job service are listed below.
>>> All of these URLs require authentication.
>>>
>>> POST /apps-v1/job
>>> Submit a new job
>>> PARAMETERS
>>> jobName: name of job
>>> softwareName: name of app you want to run
>>> processorCount: number of processors desired to run the app
>>> requestedTime: estimated wall time to run the job in HH:MM:SS format
>>> maxMemory: max memory desired to run the job in GB
>>> callbackUrl: URL (GET only) or email address that notification of job
>>> success or failure will be sent.
>>> archive: 1 to archive output, 0 not to. default: 1
>>> archivePath: path within the iPlant Data Store to archive the data. defaults
>>> to /{username}/archive/jobs/job-{job_id}-{jobName}
>>> [inputs]: enter each of the app input keys with appropriate values as
>>> independent parameters in the form
>>> [parameters]: enter each of the app parameters keys with appropriate values
>>> as independent parameters in the form
>>>
>>>
>>> GET /apps-v1/jobs/list
>>> List your job history. This will include shared jobs
>>>
>>>
>>> GET /apps-v1/jobs/{job_id}
>>> List detailed information about a specific job.
>>> URL STRUCTURE
>>> job_id: id of the job
>>>
>>>
>>> PUT /apps-v1/jobs/{job_id}
>>> Perform some type of action on a job.
>>> URL STRUCTURE
>>> job_id: id of the job
>>> PARAMETERS
>>> action: resubmit or stop
>>>
>>>
>>> DELETE /apps-v1/jobs/{job_id}
>>> Delete a job from a history. This will kill the job if it is currently
>>> running.
>>> URL STRUCTURE
>>> job_id: id of the job
>>>
>>>
>>> GET /apps-v1/jobs/{job_id}/share
>>> List all share permissions on the job
>>> URL STRUCTURE
>>> job_id: id of the job
>>>
>>>
>>> GET /apps-v1/jobs/{job_id}/share/{username}
>>> List the share permissions for a specific user on the job
>>> URL STRUCTURE
>>> job_id: id of the job
>>> username: username of the user whose permissions you want to retrieve
>>>
>>>
>>> POST /apps-v1/jobs/{job_id}/share
>>> Set and update share permissions on the job
>>> URL STRUCTURE
>>> job_id: id of the job
>>> PARAMETERS
>>> username: name of user to whom the permissions will be given
>>> permission: one of READ, WRITE, EXECUTE, READ_WRITE, READ_EXECUTE,
>>> WRITE_EXECUTE, ALL, READ_PERMISSION, WRITE_PERMISSION,
>>> READ_WRITE_PERMISSION, NONE
>>>
>>>
>>> POST /apps-v1/jobs/{job_id}/share/{username}
>>> Set and update share permissions for a specific user on the job
>>> URL STRUCTURE
>>> job_id: id of the job
>>> username: username of the user whose permissions you want to retrieve
>>> PARAMETERS
>>> permission: one of READ, WRITE, EXECUTE, READ_WRITE, READ_EXECUTE,
>>> WRITE_EXECUTE, ALL, READ_PERMISSION, WRITE_PERMISSION,
>>> READ_WRITE_PERMISSION, NONE
>>>
>>>
>>> DELETE /apps-v1/jobs/{job_id}/share
>>> Remove all share permissions on a job.
>>> URL STRUCTURE
>>> job_id: id of the job
>>>
>>>
>>> DELETE /apps-v1/jobs/{job_id}/share/{username}
>>> Remote share permissions for a user. This is the same as posting a
>>> permission of NONE to this url.
>>> URL STRUCTURE
>>> job_id: id of the job
>>> username: username of the user whose permissions you want to retrieve
>>>
>>> ________________________________________
>>> From: iplant-api-dev-bounces at iplantcollaborative.org
>>> [iplant-api-dev-bounces at iplantcollaborative.org] on behalf of Darren Boss
>>> [dboss at email.arizona.edu]
>>> Sent: Wednesday, July 10, 2013 7:54 PM
>>> To: Rion Dooley
>>> Cc: Discussion of iPlant API development
>>> Subject: Re: [Iplant-api-dev] /apps-v1/job/
>>>
>>>
>>> As a follow up to my emails to Matthew Vaughn which he just forwarded
>>> to this dev list, I'm doing this in Python and am running into some
>>> issues with the standard Python libraries. The first problem I
>>> encountered is the standard behavior with urllib2 is to always send an
>>> unauthenticated request first which won't work with the API because
>>> for some urls an unauthenticated request means something different
>>> than an authenticated request. I got around that by manually setting
>>> http headers. Now I'm trying to submit a job but I kept getting HTTP
>>> 400: Bad request errors when trying to set the content type to
>>> multipart. I saw a note on the wiki about sending a request to /job-v1
>>> with content-type application/json but got a 404. I also tried
>>> /apps-v1/job-v1 but I get a 401: Unauthorized error and I'm not sure
>>> if it a problem with my code, the url or if my userid isn't allowed to
>>> use this endpoint.
>>>
>>> On Wed, Jul 10, 2013 at 8:05 PM, Matthew Vaughn <vaughn at tacc.utexas.edu>
>>> wrote:
>>>> I just tried /apps-v1/job-v1 but I get a 401 Unauthorized. This is the
>>>> first time I'm adding more than just the one http header since I have
>>>> to both add the authorization and and content type so it could be my
>>>> code that's causing this issue.
>>>>
>>>> On Wed, Jul 10, 2013 at 7:35 PM, Darren Boss <darren.boss at arizona.edu>
>>>> wrote:
>>>>> In the doc at
>>>>> https://pods.iplantcollaborative.org/wiki/display/docs/FoundationAPI.JOB.Basic.1.0
>>>>> there is mention of using the /job-v1 endpoint together with
>>>>> Content-type application/json. When I post to /job-v1 I get a 404. Is
>>>>> this implemented? I'm not have any luck using the Python urllib2
>>>>> library together with multipart forms.
>>>>>
>>>>> On Wed, Jul 10, 2013 at 12:10 PM, Darren Boss <darren.boss at arizona.edu>
>>>>> wrote:
>>>>>> Is the get not implemented in v1 for returning a list of historical
>>>>>> jobs for a user as indicated in the docs?
>>> _______________________________________________
>>> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
>>> List Info and Archives:
>>> http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
>>> One-click Unsubscribe:
>>> http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/dooley%40tacc.utexas.edu?unsub=1&unsubconfirm=1