[Iplant-api-dev] question about running api jobs
Rion Dooley
dooley at tacc.utexas.edu
Fri Jun 13 11:01:45 MST 2014
Well, you submitted the job, so you’d just use the job’s output listing endpoint to browse the files. If you’re trying to access a job for another user, then they would need to share the job with you.
They can share the job like this:
$ curl -sku "jseto" -XPOST -d "permission=read" "https://foundation.iplantcollaborative.org/apps-v1/job/52676/share/mkhalfan" | python -mjson.tool
{
"message": "",
"result": {},
"status": "success"
}
$ curl -sku "jseto" "https://foundation.iplantcollaborative.org/apps-v1/job/52676/share/mkhalfan" | python -mjson.tool
{
"message": "",
"result": {
"name": "fx2753",
"owner": "jseto",
"permissions": [
{
"permission": {
"read": true,
"write": true
},
"username": "jseto"
},
{
"permission": {
"read": true,
"write": false
},
"username": "mkhalfan"
}
]
},
"status": "success"
}
Provided you have permission to view their job, you can view the output with the jobs output service.
$ curl -sku "jseto" https://foundation.iplantcollaborative.org/apps-v1/job/52676/output/list | python -mjson.tool
{
"message": "",
"result": [
{
"format": "folder",
"lastModified": 1402552851000,
"length": 4096,
"mimeType": "application/binary",
"name": "jseto-52676-1402533661000",
"owner": "jseto",
"parent": "/",
"path": "/jseto-52676-1402533661000",
"permission": "READ",
"type": "dir",
"url": "https://foundation.iplantcollaborative.org/apps-v1/job/52676/output/list/jseto-52676-1402533661000"
}
],
"status": "success"
}
This response says the output folder only contains a single folder, "jseto-52676-1402533661000". Let’s see the contents of that folder. I’m going to just list the name fields for brevity in this email.
$ curl -sku "jseto" https://foundation.iplantcollaborative.org/apps-v1/job/52676/output/list/jseto-52676-1402533661000 | python -mjson.tool | grep "name"
"name": ".iplant.archive",
"name": "FastQC",
"name": "FastQC.tgz",
"name": "RNA17_S9_R1.fastq",
"name": "bin",
"name": "bin.tgz",
"name": "fastx.sh",
"name": "fastx_out",
"name": "fastx_stampede.json",
"name": "fx2753-52676.err",
"name": "fx2753-52676.out",
"name": "fx2753.ipcexe",
"name": "library”,
Now we see the "fx2753-52676.err” and "fx2753-52676.out” log files from the job. Now we can download them using the job output download service. Notice it’s identical to the listing service, but there is no “listing” token in the url.
$ curl -sku "jseto” -O https://foundation.iplantcollaborative.org/apps-v1/job/52676/output/jseto-52676-1402533661000/fx2753-52676.err
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
108 758 108 758 0 0 1730 0 --:--:-- --:--:-- --:--:— 58307
The file downloaded to our local system. We can tail the output to see what happened.
$ tail -n 10 fx2753-52676.err
Approx 80% complete for RNA17_S9_R1-fx2753.fastq
Approx 85% complete for RNA17_S9_R1-fx2753.fastq
Approx 90% complete for RNA17_S9_R1-fx2753.fastq
Approx 95% complete for RNA17_S9_R1-fx2753.fastq
Approx 100% complete for RNA17_S9_R1-fx2753.fastq
compressing output file with gzip
slurmd[c556-703]: *** JOB 3516934 CANCELLED AT 2014-06-11T22:41:47 DUE TO TIME LIMIT ***
Hope that helps! FWIW, the following are the corresponding commands in agave using curl and the cli. I’ve included the first few commands in case you haven’t used Agave or the cli before. The cli is available from: https://bitbucket.org/taccaci/foundation-cli
Initialize your cli to the agave tenant
$ tenants-init -b
Please select a tenant from the following list:
[0] araport.org<http://araport.org>
[1] iplantc.org<http://iplantc.org>
[2] vdjserver.org<http://vdjserver.org>
[3] wso2.agave.staging
[4] xsede.org<http://xsede.org>
[5] xsede.org.staging
Your choice [2]: 1
You are now configured to interact with the APIs at https://agave.iplantc.org/
Get your API keys
$ clients-create -V -u mkhalfan -N "mkhalfan-cli” -S
API password:
Calling curl -sku “mkhalfan" -X POST -d "clientName= mkhalfan-cli&tier=Unlimited&description=&callbackUrl=" https://agave.iplantc.org/clients/v2/?pretty=true
{
"message": "Client created successfully.",
"result": {
"_links": {
"self": {
"href": "https://agave.iplantc.org/clients/v2/mkhalfan-cli"
},
"subscriber": {
"href": "https://agave.iplantc.org/profiles/v2/mkhalfan"
},
"subscriptions": {
"href": "https://agave.iplantc.org/clients/v2/mkhalfan-cli/subscriptions/"
}
},
"callbackUrl": "",
"consumerKey": “your_api_key",
"consumerSecret": “your_api_secret",
"description": "",
"name": "mkhalfan-cli",
"tier": "Unlimited"
},
"status": "success",
"version": "2.0.0-SNAPSHOT-rc3fad"
}
Check to see if you’re authenticated yet.
$ auth-check
Current token is expired
tenant: iplantc.org<http://iplantc.org>
username: mkhalfan
time left: 0 seconds
expired at:
You are not authenticated, so go ahead and pull a token
$ auth-tokens-create -S -V
API secret [moOX8xR9ySh_mkTaOy02MYiq7bka]:
API key [3ekNmfvne9yN_wnfedCRMeA8XSUa]:
API username [mkhalfan]:
API password:
Calling curl -sku “your_api_key:XXXXXX" -X POST -d "grant_type=password&username=mkhalfan&password=XXXXXX&scope=PRODUCTION" -H "Content-Type:application/x-www-form-urlencoded" https://agave.iplantc.org/token
Token successfully refreshed and cached for 12514 seconds
{
"YOUR_ACCESS_TOKEN": "ACCESS_TOKEN",YOUR_ACCESS_TOKEN
"expires_in": 12514,
"refresh_token": “YOUR_REFRESH_TOKEN",
"token_type": "bearer"
}
You don’t have a job yet, so let’s submit a simple python job.
$ echo '{"name":"Rodeo Python job","appId":"docker-python-demo-0.1.0u2","notifications":[{"url":"mkhalfan at cshl.edu<mailto:mkhalfan at cshl.edu>","event":"FINISHED","persistent":false},{"url":"mkhalfan at cshl.edu<mailto:mkhalfan at cshl.edu>","event":"FAILED","persistent":false}]}' > mkhalfan.job
We submit jobs the same as before.
$ jobs-submit -V -F mkhalfan.job
Calling curl -sk -H "Authorization: Bearer YOUR_ACCESS_TOKEN -X POST -F "fileToUpload=@mkhalfan.job<mailto:fileToUpload=@mkhalfan.job>" https://agave.iplantc.org/jobs/v2/?pretty=true
{
"status" : "success",
"message" : null,
"version" : "2.0.0-SNAPSHOT-r838d8",
"result" : {
"id" : "0001402680372520-5056a550b8-0001-007",
"name" : "Rodeo Python job",
"owner" : "mkhalfan",
"appId" : "docker-python-demo-0.1.0u2",
"executionSystem" : "docker.iplantcollaborative.org<http://docker.iplantcollaborative.org>",
"batchQueue" : "debug",
"nodeCount" : 1,
"processorsPerNode" : 1,
"memoryPerNode" : 1.0,
"maxRunTime" : "24:00:00",
"archive" : true,
"retries" : 0,
"localId" : null,
"archivePath" : "mkhalfan/archive/jobs/job-0001402680372520-5056a550b8-0001-007",
"archiveSystem" : "data.iplantcollaborative.org<http://data.iplantcollaborative.org>",
"outputPath" : null,
"status" : "PENDING",
"submitTime" : "2014-06-13T12:26:12.525-05:00",
"startTime" : null,
"endTime" : null,
"inputs" : { },
"parameters" : {
"dockerImage" : "agaveapi/iwsg-python-demo"
},
"_links" : {
"self" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007"
},
"app" : {
"href" : "https://agave.iplantc.org/apps/v2/docker-python-demo-0.1.0u2"
},
"executionSystem" : {
"href" : "https://agave.iplantc.org/systems/v2/docker.iplantcollaborative.org"
},
"archiveSystem" : {
"href" : "https://agave.iplantc.org/systems/v2/data.iplantcollaborative.org"
},
"archiveData" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/outputs/listings"
},
"owner" : {
"href" : "https://agave.iplantc.org/profiles/v2/mkhalfan"
},
"permissions" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/pems"
},
"history" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/history"
},
"metadata" : {
"href" : "https://agave.iplantc.org/meta/v2/data/?q={\"associationIds\":\"0001402680372520-5056a550b8-0001-007\"}"
},
"notifications" : {
"href" : "https://agave.iplantc.org/notifications/v2/?associatedUuid=0001402680372520-5056a550b8-0001-007"
}
}
}
}
Is it done yet?
$ jobs-status -V 0001402680372520-5056a550b8-0001-007
Calling curl -sk -H "Authorization: Bearer YOUR_ACCESS_TOKEN" https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/status?pretty=true
{
"status" : "success",
"message" : null,
"version" : "2.0.0-SNAPSHOT-r838d8",
"result" : {
"id" : "0001402680372520-5056a550b8-0001-007",
"status" : "FINISHED",
"_links" : {
"self" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007"
}
}
}
}
Sweet! What happened? Let me check the job’s history
$ jobs-history -V 0001402680372520-5056a550b8-0001-007
Calling curl -sk -H "Authorization: Bearer YOUR_ACCESS_TOKEN" https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/history?pretty=true
{
"status" : "success",
"message" : null,
"version" : "2.0.0-SNAPSHOT-r838d8",
"result" : [ {
"created" : "2014-06-13T12:26:12.000-05:00",
"status" : "PENDING",
"description" : "Job accepted and queued for submission."
}, {
"created" : "2014-06-13T12:26:16.000-05:00",
"status" : "STAGED",
"description" : "Skipping staging. No input data associated with this job."
}, {
"created" : "2014-06-13T12:26:26.000-05:00",
"status" : "SUBMITTING",
"description" : "Preparing job for submission."
}, {
"created" : "2014-06-13T12:26:26.000-05:00",
"status" : "SUBMITTING",
"description" : "Attempt 1 to submit job"
}, {
"created" : "2014-06-13T12:26:35.000-05:00",
"status" : "RUNNING",
"description" : "Job started running"
}, {
"created" : "2014-06-13T12:26:35.000-05:00",
"status" : "CLEANING_UP"
}, {
"created" : "2014-06-13T12:26:36.000-05:00",
"status" : "ARCHIVING",
"description" : "Beginning to archive output."
}, {
"created" : "2014-06-13T12:26:36.000-05:00",
"status" : "ARCHIVING",
"description" : "Attempt 1 to archive job output"
}, {
"created" : "2014-06-13T12:26:36.000-05:00",
"status" : "ARCHIVING",
"description" : "Transferring job output to archive system"
}, {
"progress" : {
"averageRate" : 0,
"totalFiles" : 6,
"source" : "agave://docker.iplantcollaborative.org/mkhalfan/job-0001402680372520-5056a550b8-0001-007-rodeo-python-job",
"totalActiveTransfers" : 1,
"totalBytes" : 15644,
"totalBytesTransferred" : 15644
},
"created" : "2014-06-13T12:26:38.000-05:00",
"status" : "ARCHIVING",
"description" : "Archiving agave://docker.iplantcollaborative.org/mkhalfan/job-0001402680372520-5056a550b8-0001-007-rodeo-python-job to agave://data.iplantcollaborative.org/mkhalfan/archive/jobs/job-0001402680372520-5056a550b8-0001-007"
}, {
"created" : "2014-06-13T12:26:40.000-05:00",
"status" : "QUEUED",
"description" : "HPC job successfully placed into queue"
}, {
"created" : "2014-06-13T12:26:54.000-05:00",
"status" : "ARCHIVING_FINISHED",
"description" : "Job archiving completed successfully."
}, {
"created" : "2014-06-13T12:26:55.000-05:00",
"status" : "FINISHED",
"description" : "Job complete"
} ]
}
Ok, now for the output. What’s there?
$ jobs-output -V 0001402680372520-5056a550b8-0001-007
Calling curl -sk -H "Authorization: Bearer YOUR_ACCESS_TOKEN" https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/outputs/listings/?pretty=true
{
"status" : "success",
"message" : null,
"version" : "2.0.0-SNAPSHOT-r838d8",
"result" : [ {
"name" : "results",
"path" : "/results",
"lastModified" : "2014-06-13T12:26:38.000-05:00",
"length" : 0,
"permission" : "NONE",
"mimeType" : "text/directory",
"format" : "folder",
"type" : "dir",
"_links" : {
"self" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/outputs/media/results"
},
"system" : {
"href" : "https://agave.iplantc.org/systems/v2/data.iplantcollaborative.org"
},
"parent" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007"
}
}
}, {
"name" : "rodeo-python-job.err",
"path" : "/rodeo-python-job.err",
"lastModified" : "2014-06-13T12:26:47.000-05:00",
"length" : 785,
"permission" : "NONE",
"mimeType" : "application/octet-stream",
"format" : "unknown",
"type" : "file",
"_links" : {
"self" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/outputs/media/rodeo-python-job.err"
},
"system" : {
"href" : "https://agave.iplantc.org/systems/v2/data.iplantcollaborative.org"
},
"parent" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007"
}
}
}, {
"name" : "rodeo-python-job.out",
"path" : "/rodeo-python-job.out",
"lastModified" : "2014-06-13T12:26:49.000-05:00",
"length" : 162,
"permission" : "NONE",
"mimeType" : "application/octet-stream",
"format" : "unknown",
"type" : "file",
"_links" : {
"self" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/outputs/media/rodeo-python-job.out"
},
"system" : {
"href" : "https://agave.iplantc.org/systems/v2/data.iplantcollaborative.org"
},
"parent" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007"
}
}
}, {
"name" : "rodeo-python-job.pid",
"path" : "/rodeo-python-job.pid",
"lastModified" : "2014-06-13T12:26:52.000-05:00",
"length" : 6,
"permission" : "NONE",
"mimeType" : "application/octet-stream",
"format" : "unknown",
"type" : "file",
"_links" : {
"self" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/outputs/media/rodeo-python-job.pid"
},
"system" : {
"href" : "https://agave.iplantc.org/systems/v2/data.iplantcollaborative.org"
},
"parent" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007"
}
}
} ]
}
I see the output file. Let me grab that
$ jobs-output -V -D -P rodeo-python-job.err 0001402680372520-5056a550b8-0001-007
Calling curl -k -H "Authorization: Bearer YOUR_ACCESS_TOKEN -o "rodeo-python-job.err" https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/outputs/media/rodeo-python-job.err
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 785 0 785 0 0 946 0 --:--:-- --:--:-- --:--:-- 945
Successfully downloaded rodeo-python-job.err from job 0001402680372520-5056a550b8-0001-007 to rodeo-python-job.err
Let’s look at the file.
$ tail -n 10 rodeo-python-job.err
Dload Upload Total Spent Left Speed
100 81 100 81 0 0 407 0 --:--:-- --:--:-- --:--:-- 417
+ DOCKER_IMAGE=agaveapi/iwsg-python-demo
+ '[' -n '' ']'
+ '[' -z agaveapi/iwsg-python-demo ']'
+ mkdir results
++ pwd
+ docker run -i -t -v /home/mkhalfan/vhome/mkhalfan/job-0001402680372520-5056a550b8-0001-007-rodeo-python-job:/home/ubuntu/scratch -w /home/ubuntu/scratch agaveapi/iwsg-python-demo
+ '[' '!' 0 ']'
+ curl -sk https://agave.iplantc.org/jobs/v2/trigger/job/0001402680372520-5056a550b8-0001-007/token/28c76405-f1ae-490f-ab39-3cf3c68aba90/status/CLEANING_UP
I gotta tell my buddy about this. Let me share the job with him.
$ jobs-pems-update -V -u dooley -p READ 0001402680372520-5056a550b8-0001-007
Calling curl -sk -H "Authorization: Bearer YOUR_ACCESS_TOKEN -X POST -d "permission=READ" https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/pems/dooley?pretty=true
{
"status" : "success",
"message" : null,
"version" : "2.0.0-SNAPSHOT-r838d8",
"result" : {
"username" : "dooley",
"internalUsername" : null,
"permission" : {
"read" : false,
"write" : false
},
"_links" : {
"self" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/pems/dooley"
},
"parent" : {
"href" : "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007"
},
"profile" : {
"href" : "https://agave.iplantc.org/profiles/v2/dooley"
}
}
}
}
Better yet, let me just send him a link to the output graph. I remember it was given in the hypermedia response from the output listing as: https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/outputs/media/results/myHistogram.png. I’ll just create a postit and email it to him so he can download it directly.
$ postits-create -m 10 -V https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/outputs/media/results/myHistogram.png
Calling curl -sk -H "Authorization: Bearer YOUR_ACCESS_TOKEN -X POST -d "maxUses=10&method=GET" -d "url=https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/outputs/media/results/myHistogram.png" https://agave.iplantc.org/postits/v2/?pretty=true
{
"status": "success",
"message": "",
"result": {
"creator": “mkhalfan",
"internalUsername": null,
"authenticated": true,
"created": "2014-06-13T12:48:56-05:00",
"expires": "2014-07-13T12:48:56-05:00",
"remainingUses": 10,
"postit": "036c9f3bf2739ac4d7a60539a6bc49e2",
"noauth": false,
"url": "https://agave.iplantc.org/jobs/v2/0001402680372520-5056a550b8-0001-007/outputs/media/results/myHistogram.png",
"method": "GET",
"_links": {
"self": {
"href": "https://agave.iplantc.org/postits/v2/036c9f3bf2739ac4d7a60539a6bc49e2"
},
"profile": {
"href": "https://agave.iplantc.org/profiles/v2/iplantc/mkhalfan"
}
}
},
"version": "2.0.0-SNAPSHOT-ra1036"
}
Before I send it, let’s make sure it works.
$ curl -o myHistogram.png https://agave.iplantc.org/postits/v2/036c9f3bf2739ac4d7a60539a6bc49e2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 14691 0 14691 0 0 8988 0 --:--:-- 0:00:01 --:--:— 8985
$ open myHistogram.png
[cid:BEFAFE81-FCF7-47BE-A206-48FFF7CFC5CE at tacc.utexas.edu]
--
Rion
On Jun 13, 2014, at 11:07 AM, Khalfan, Mohammed <mkhalfan at cshl.edu<mailto:mkhalfan at cshl.edu>> wrote:
Hey Rion,
How would I view the error log for a job for another user? Is this possible? In the case below for example...
Thank you,
Mohammed
-----Original Message-----
From: Rion Dooley [mailto:dooley at tacc.utexas.edu]
Sent: Thursday, June 12, 2014 1:54 PM
To: Khalfan, Mohammed
Cc: iPlant API Developers Mailing List
Subject: Re: [Iplant-api-dev] question about running api jobs
Looks like the jobs were cancelled due to exceeding the time limit. This is in the error logs.
...
Approx 95% complete for RNA17_S9_R1-fx2753.fastq Approx 100% complete for RNA17_S9_R1-fx2753.fastq compressing output file with gzip
slurmd[c556-703]: *** JOB 3516934 CANCELLED AT 2014-06-11T22:41:47 DUE TO TIME LIMIT ***
--
Rion
On Jun 12, 2014, at 12:32 PM, Khalfan, Mohammed <mkhalfan at cshl.edu<mailto:mkhalfan at cshl.edu>> wrote:
Hi,
I have a user with 2 jobs (52676, 52679) which were launched last night along with a few others jobs. The other jobs completed, but these two are still RUNNING, well after the requested time (3hrs) has expired. Are you able to look into this to see if it's normal? Or what's happening..?
Thank you,
Mohammed
_______________________________________________
Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org<mailto:Iplant-api-dev at iplantcollaborative.org>
List Info and Archives:
http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
One-click Unsubscribe:
http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/doo
ley%40tacc.utexas.edu?unsub=1&unsubconfirm=1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.iplantcollaborative.org/pipermail/iplant-api-dev/attachments/20140613/7a4a352b/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 036c9f3bf2739ac4d7a60539a6bc49e2.png
Type: image/png
Size: 14691 bytes
Desc: 036c9f3bf2739ac4d7a60539a6bc49e2.png
Url : http://mail.iplantcollaborative.org/pipermail/iplant-api-dev/attachments/20140613/7a4a352b/attachment-0001.png
More information about the Iplant-api-dev
mailing list