[Iplant-api-dev] /apps-v1/job/

[Darren Boss]

I just succeeded at submitting a job. I'm definitely ditching urllib2.

Thanks.

On Thu, Jul 11, 2013 at 1:15 PM, Joe Stubbs <jstubbs at tacc.utexas.edu> wrote:
> Hi Darren,
>
> I was actually going to suggest the requests library; it is definitely a much friendlier API than urllib2.  Let me know if you have any trouble, and I will do my best to help.
>
> Joe
>
>
> ________________________________________
> From: Darren Boss
> Sent: Thursday, July 11, 2013 12:10 PM
> To: Rion Dooley
> Cc: iplant-api-dev at iplantcollaborative.org; Joe Stubbs
> Subject: Re: [Iplant-api-dev] /apps-v1/job/
>
> I've installed the request module
> (http://docs.python-requests.org/en/latest/index.html) and finding it
> mush easier to use. With the job submission I was having trouble
> accessing the json returned when I received an HTTP error using the
> urllib2 library because an exception was thrown and I couldn't figure
> out how to access the response. I think this will simplify my code and
> speed development tremendously.
>
> I'll post again if I get stuck with the reuqest module or if I have
> something worth sharing with other developers.
>
> On Thu, Jul 11, 2013 at 12:42 PM, Rion Dooley <dooley at tacc.utexas.edu> wrote:
>> You can go by my email outlining the job service or try clearing your cache and refreshing the documentation page. In response to your earlier email, I updated the documentation with the most recent information. v1 is frozen now, so this won't change. I'm cc'ing Joe Stubbs on this email as he's also writing up a job submission routine in Python, though probably with a smaller scope than you taking on. Perhaps you two can be of help to each other.
>>
>> Rion
>>
>>
>> ________________________________________
>> From: Darren Boss [dboss at email.arizona.edu]
>> Sent: Thursday, July 11, 2013 11:12 AM
>> To: Rion Dooley
>> Cc: iplant-api-dev at iplantcollaborative.org
>> Subject: Re: [Iplant-api-dev] /apps-v1/job/
>>
>> I'm still confused about what attributes should be set in the job
>> request. In the api docs it says:
>>
>> Parameter Role Required Notes
>> application $appname-$version Yes Lexical name of an application in
>> the app registry
>> name Job name No String. Human-readable name for this job. Defaults to
>> value in "application"
>> processors Integer > 0 No How many processors to request
>> memory String No Minimum memory required to complete the job. Uses
>> same semantics as SGE l_vmem reservations. Example: 8G
>> jobPostbackURL String, URL No Post status of job to this URL
>> archiveOutput Boolean No If 'true' then copy job/output to $IPLANTHOME/x-name/
>> archivePath String No Copy output files here instead of $IPLANTHOME/x-name/
>> parameters     These parameters are devised by inpsecting the inputs,
>> parameters, and outputs sections of the APPS description for
>> $appname-$version
>>
>> In the php wireframe I see the code:
>> //TODO CHANGE APPLICATION NAME TO MAKE SURE PROPER SOFTWARE IS CALLED
>> $postField ="softwareName=APPNAMEHERE&archive=true";
>>
>> So is it application or softwareName?
>>
>> On Thu, Jul 11, 2013 at 11:04 AM, Rion Dooley <dooley at tacc.utexas.edu> wrote:
>>> Thanks for letting us know about the errors on the documentation page. I have updated the page with appropriate changes.
>>>
>>> The v1 /job endpoint takes a standard form submission. File uploading is not supported on that resource, so multipart is not needed.
>>>
>>> Thanks for all your hard work putting together the library. Hopefully it will be help to you and the community.
>>>
>>> cheers,
>>> Rion
>>>
>>> ________________________________________
>>> From: Darren Boss [dboss at email.arizona.edu]
>>> Sent: Thursday, July 11, 2013 9:29 AM
>>> To: Rion Dooley
>>> Cc: Discussion of iPlant API development
>>> Subject: Re: [Iplant-api-dev] /apps-v1/job/
>>>
>>> There is an error in the docs regarding the url for returning
>>> historical jobs. It indicates that the endpoint is /apps-v1/job/ with
>>> a GET request but it should be /apps-v1/jobs/list.
>>>
>>> On that same page there is a section that states:
>>>
>>> You can also create a request to the Job manager via HTTP POST of
>>> Content-type application/json to the /job-v1 endpoint. This supports
>>> programmatic access to the API from applications that might write JSON
>>> natively. Here's the format for that method, with specifics for the
>>> "head" application.
>>>
>>> What endpoint is this referring to? If I call up
>>> https://foundation.iplantcollaborative.org/jobs-v1/usage I get a
>>> resource not available message so that's not it and I don't see
>>> /apps-v1/jobs-v1 in the apps-v1 usage list. Is this not implemented? I
>>> can very easily construct the json from Python dictionaries  in my
>>> code so this would be very nice to use.
>>>
>>> The reason why I'm trying to use this endpoint is I'm having trouble
>>> constructing the "Content-type: form/multipart" POST and sending an
>>> authenticated request for the /apps-v1/job endpoint. If you think that
>>> x-www-form-urlencoded (the default) should work as well I'll
>>> reevaluate my code to see if I goofed up somewhere else. I think once
>>> I get job submission working the remaining method implementations
>>> should be trivial.
>>>
>>> On Wed, Jul 10, 2013 at 11:27 PM, Rion Dooley <dooley at tacc.utexas.edu> wrote:
>>>> Hi Darren,
>>>>
>>>> Thanks for following up on the list. I believe it's just a matter of finding
>>>> the right v1 urls. The apps, jobs, and systems service urls are listed in
>>>> the v1 docs website at:
>>>>
>>>> https://foundation.iplantcollaborative.org/docs
>>>>
>>>> The service also provides its own (very basic) usage page with a list of
>>>> valid urls:
>>>>
>>>> https://foundation.iplantcollaborative.org/auth-v1/usage
>>>> https://foundation.iplantcollaborative.org/apps-v1/usage
>>>> https://foundation.iplantcollaborative.org/io-v1/usage
>>>> https://foundation.iplantcollaborative.org/postit-v1/usage/
>>>> https://foundation.iplantcollaborative.org/profile-v1/usage
>>>>
>>>> In your case, the relevant operations for the job service are listed below.
>>>> All of these URLs require authentication.
>>>>
>>>> POST /apps-v1/job
>>>> Submit a new job
>>>> PARAMETERS
>>>> jobName: name of job
>>>> softwareName: name of app you want to run
>>>> processorCount: number of processors desired to run the app
>>>> requestedTime: estimated wall time to run the job in HH:MM:SS format
>>>> maxMemory: max memory desired to run the job in GB
>>>> callbackUrl: URL (GET only) or email address that notification of job
>>>> success or failure will be sent.
>>>> archive: 1 to archive output, 0 not to. default: 1
>>>> archivePath: path within the iPlant Data Store to archive the data. defaults
>>>> to /{username}/archive/jobs/job-{job_id}-{jobName}
>>>> [inputs]: enter each of the app input keys with appropriate values as
>>>> independent parameters in the form
>>>> [parameters]: enter each of the app parameters keys with appropriate values
>>>> as independent parameters in the form
>>>>
>>>>
>>>> GET /apps-v1/jobs/list
>>>> List your job history. This will include shared jobs
>>>>
>>>>
>>>> GET /apps-v1/jobs/{job_id}
>>>> List detailed information about a specific job.
>>>> URL STRUCTURE
>>>> job_id: id of the job
>>>>
>>>>
>>>> PUT /apps-v1/jobs/{job_id}
>>>> Perform some type of action on a job.
>>>> URL STRUCTURE
>>>> job_id: id of the job
>>>> PARAMETERS
>>>> action: resubmit or stop
>>>>
>>>>
>>>> DELETE /apps-v1/jobs/{job_id}
>>>> Delete a job from a history. This will kill the job if it is currently
>>>> running.
>>>> URL STRUCTURE
>>>> job_id: id of the job
>>>>
>>>>
>>>> GET /apps-v1/jobs/{job_id}/share
>>>> List all share permissions on the job
>>>> URL STRUCTURE
>>>> job_id: id of the job
>>>>
>>>>
>>>> GET /apps-v1/jobs/{job_id}/share/{username}
>>>> List the share permissions for a specific user on the job
>>>> URL STRUCTURE
>>>> job_id: id of the job
>>>> username: username of the user whose permissions you want to retrieve
>>>>
>>>>
>>>> POST /apps-v1/jobs/{job_id}/share
>>>> Set and update share permissions on the job
>>>> URL STRUCTURE
>>>> job_id: id of the job
>>>> PARAMETERS
>>>> username: name of user to whom the permissions will be given
>>>> permission: one of READ, WRITE, EXECUTE, READ_WRITE, READ_EXECUTE,
>>>> WRITE_EXECUTE, ALL, READ_PERMISSION, WRITE_PERMISSION,
>>>> READ_WRITE_PERMISSION, NONE
>>>>
>>>>
>>>> POST /apps-v1/jobs/{job_id}/share/{username}
>>>> Set and update share permissions for a specific user on the job
>>>> URL STRUCTURE
>>>> job_id: id of the job
>>>> username: username of the user whose permissions you want to retrieve
>>>> PARAMETERS
>>>> permission: one of READ, WRITE, EXECUTE, READ_WRITE, READ_EXECUTE,
>>>> WRITE_EXECUTE, ALL, READ_PERMISSION, WRITE_PERMISSION,
>>>> READ_WRITE_PERMISSION, NONE
>>>>
>>>>
>>>> DELETE /apps-v1/jobs/{job_id}/share
>>>> Remove all share permissions on a job.
>>>> URL STRUCTURE
>>>> job_id: id of the job
>>>>
>>>>
>>>> DELETE /apps-v1/jobs/{job_id}/share/{username}
>>>> Remote share permissions for a user. This is the same as posting a
>>>> permission of NONE to this url.
>>>> URL STRUCTURE
>>>> job_id: id of the job
>>>> username: username of the user whose permissions you want to retrieve
>>>>
>>>> ________________________________________
>>>> From: iplant-api-dev-bounces at iplantcollaborative.org
>>>> [iplant-api-dev-bounces at iplantcollaborative.org] on behalf of Darren Boss
>>>> [dboss at email.arizona.edu]
>>>> Sent: Wednesday, July 10, 2013 7:54 PM
>>>> To: Rion Dooley
>>>> Cc: Discussion of iPlant API development
>>>> Subject: Re: [Iplant-api-dev] /apps-v1/job/
>>>>
>>>>
>>>> As a follow up to my emails to Matthew Vaughn which he just forwarded
>>>> to this dev list, I'm doing this in Python and am running into some
>>>> issues with the standard Python libraries. The first problem I
>>>> encountered is the standard behavior with urllib2 is to always send an
>>>> unauthenticated request first which won't work with the API because
>>>> for some urls an unauthenticated request means something different
>>>> than an authenticated request. I got around that by manually setting
>>>> http headers. Now I'm trying to submit a job but I kept getting HTTP
>>>> 400: Bad request errors when trying to set the content type to
>>>> multipart. I saw a note on the wiki about sending a request to /job-v1
>>>> with content-type application/json but got a 404. I also tried
>>>> /apps-v1/job-v1 but I get a 401: Unauthorized error and I'm not sure
>>>> if it a problem with my code, the url or if my userid isn't allowed to
>>>> use this endpoint.
>>>>
>>>> On Wed, Jul 10, 2013 at 8:05 PM, Matthew Vaughn <vaughn at tacc.utexas.edu>
>>>> wrote:
>>>>> I just tried /apps-v1/job-v1 but I get a 401 Unauthorized. This is the
>>>>> first time I'm adding more than just the one http header since I have
>>>>> to both add the authorization and and content type so it could be my
>>>>> code that's causing this issue.
>>>>>
>>>>> On Wed, Jul 10, 2013 at 7:35 PM, Darren Boss <darren.boss at arizona.edu>
>>>>> wrote:
>>>>>> In the doc at
>>>>>> https://pods.iplantcollaborative.org/wiki/display/docs/FoundationAPI.JOB.Basic.1.0
>>>>>> there is mention of using the /job-v1 endpoint together with
>>>>>> Content-type application/json. When I post to /job-v1 I get a 404. Is
>>>>>> this implemented? I'm not have any luck using the Python urllib2
>>>>>> library together with multipart forms.
>>>>>>
>>>>>> On Wed, Jul 10, 2013 at 12:10 PM, Darren Boss <darren.boss at arizona.edu>
>>>>>> wrote:
>>>>>>> Is the get not implemented in v1 for returning a list of historical
>>>>>>> jobs for a user as indicated in the docs?
>>>> _______________________________________________
>>>> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
>>>> List Info and Archives:
>>>> http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
>>>> One-click Unsubscribe:
>>>> http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/dooley%40tacc.utexas.edu?unsub=1&unsubconfirm=1