[Iplant-api-dev] status FINISHED vs. ARCHIVING_FINISHED, no data
Rion Dooley
dooley at tacc.utexas.edu
Wed Dec 11 12:04:44 MST 2013
You can pull the entire output of the job even if it's not archived by using the job/<jobid>/output/list endpoint I gave earlier. Here is the error log from your other job. Same issue. Seems to be symptomatic of the script:
$> curl -sku "mkhalfan:1908d2f671fb6df4b25c4d04245bbe2e" https://foundation.iplantc.org/apps-v1/job/36121/output/mkhalfan-36121-1386783031000/cd547-36121.err
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
103 2680 103 2680 0 0 7320 0 --:--:-- --:--:-- --:--:-- 174k
Inspecting list of GTF files to merge...
Executing cuffmerge -p 16 -o cuffmerge_out -g annotation.gtf -s genome.fas gtf_to_merge.txt...
[Wed Dec 11 11:34:02 2013] Beginning transcriptome assembly merge
-------------------------------------------
[Wed Dec 11 11:34:02 2013] Preparing output location cuffmerge_out/
[Wed Dec 11 11:34:09 2013] Converting GTF files to SAM
[11:34:09] Loading reference annotation.
[11:34:10] Loading reference annotation.
[11:34:12] Loading reference annotation.
[11:34:14] Loading reference annotation.
[11:34:15] Loading reference annotation.
[11:34:16] Loading reference annotation.
[11:34:18] Loading reference annotation.
[11:34:20] Loading reference annotation.
[11:34:21] Loading reference annotation.
[Wed Dec 11 11:34:24 2013] Quantitating transcripts
You are using Cufflinks v2.1.1, which is the most recent release.
Command line:
cufflinks -o cuffmerge_out/ -F 0.05 -g annotation.gtf -q --overhang-tolerance 200 --library-type=transfrags -A 0.0 --min-frags-per-transfrag 0 --no-5-extend -p 16 cuffmerge_out/tmp/mergeSam_fileqbngnV
[bam_header_read] EOF marker is absent.
[bam_header_read] invalid BAM binary header (this is not a BAM file).
File cuffmerge_out/tmp/mergeSam_fileqbngnV doesn't appear to be a valid BAM file, trying SAM...
[11:34:24] Loading reference annotation.
[11:34:27] Inspecting reads and determining fragment length distribution.
Processed 98978 loci.
> Map Properties:
> Normalized Map Mass: 358518.00
> Raw Map Mass: 358518.00
> Fragment Length Distribution: Truncated Gaussian (default)
> Default Mean: 200
> Default Std Dev: 80
[11:34:29] Assembling transcripts and estimating abundances.
Processed 98978 loci.
[Wed Dec 11 11:35:10 2013] Comparing against reference file annotation.gtf
You are using Cufflinks v2.1.1, which is the most recent release.
No fasta index found for genome.fas. Rebuilding, please wait..
Fasta index rebuilt.
[Wed Dec 11 11:36:16 2013] Comparing against reference file annotation.gtf
You are using Cufflinks v2.1.1, which is the most recent release.
DONE!
total 295M
drwx------ 3 iplant G-802086 4.0K Dec 11 11:36 .
drwxrwxr-x 7 iplant G-802086 4.0K Dec 11 11:36 ..
drwx------ 2 iplant G-802086 4.0K Dec 11 11:34 logs
-rw------- 1 iplant G-802086 148M Dec 11 11:36 merged.gtf
-rw------- 1 iplant G-802086 150M Dec 11 11:37 merged_with_ref_ids.gtf
/tmp/slurmd/job2214920/slurm_script: line 276: syntax error near unexpected token `;'
/tmp/slurmd/job2214920/slurm_script: line 276: `if ! [[ -s $SAM_1F ]]; then;'
The concept of FAILED that you're referring to is only applicable in your application logic and the logic of your wrapper. Foundation has no idea whether your computation was successful. The status you see is purely a reflection of whether Foundation was able to submit, monitor, and take appropriate cleanup action. You can, to a certain extent, think of the Jobs service like a scheduler. The scheduler tells you when your job is valid to add to the queue, what the status is, and when it's done. It can't really tell you whether it was a success or not.
One thing we could improve on is checking status codes on exit, but that would require a different monitoring mechanism than Foundation uses. Broken record, I know, but this is available in Agave.
Rion
________________________________
From: iplant-api-dev-bounces at iplantcollaborative.org [iplant-api-dev-bounces at iplantcollaborative.org] on behalf of Khalfan, Mohammed [mkhalfan at cshl.edu]
Sent: Wednesday, December 11, 2013 12:46 PM
To: Rion Dooley
Cc: iPlant API Developers Mailing List
Subject: Re: [Iplant-api-dev] status FINISHED vs. ARCHIVING_FINISHED, no data
If it was a failure, shouldn’t it return FAILED with an error? It’s returning FINISHED, which makes me think the job finished ok, and the problem was at the archiving of the data stage.
This app has worked previously, I’m not sure why it would stop working all of a sudden, unless there was a change in something on the API side and the app needs to be updated.
Can you please help me to resolve this.
Thank you.
From: Matthew Vaughn [mailto:vaughn at tacc.utexas.edu]
Sent: Wednesday, December 11, 2013 1:42 PM
To: Khalfan, Mohammed
Cc: iPlant API Developers Mailing List
Subject: Re: [Iplant-api-dev] status FINISHED vs. ARCHIVING_FINISHED, no data
It very well may be. If $SAM1_F has not been defined by the line line 276 is executed then bash will see the following
if [[ -s ]]; then;
which is invalid bash syntax.
On Dec 11, 2013, at 12:35 PM, Khalfan, Mohammed <mkhalfan at cshl.edu<mailto:mkhalfan at cshl.edu>> wrote:
Is this an issue with our wrapper script?
I just tried another job (id: 36121) and I got the same result (FINISHED, no data in output folder).
From: Rion Dooley [mailto:dooley at tacc.utexas.edu]
Sent: Wednesday, December 11, 2013 1:00 PM
To: Khalfan, Mohammed
Cc: iPlant API Developers Mailing List
Subject: Re: [Iplant-api-dev] status FINISHED vs. ARCHIVING_FINISHED, no data
Looking at the error log it seems like the wrapper script bombed out somewhere around line 276 during execution.
curl -sku “mkhalfan:******" https://foundation.iplantc.org/apps-v1/job/36113/output/mkhalfan-36113-1386776891000/cd389-36113.err
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
102 2669 102 2669 0 0 5368 0 --:--:-- --:--:-- --:--:-- 45237
Inspecting list of GTF files to merge...
Executing cuffmerge -p 16 -o cuffmerge_out -g annotation.gtf -s genome.fas gtf_to_merge.txt...
[Wed Dec 11 10:09:08 2013] Beginning transcriptome assembly merge
-------------------------------------------
[Wed Dec 11 10:09:08 2013] Preparing output location cuffmerge_out/
[Wed Dec 11 10:09:15 2013] Converting GTF files to SAM
[10:09:15] Loading reference annotation.
[10:09:16] Loading reference annotation.
[10:09:17] Loading reference annotation.
[10:09:19] Loading reference annotation.
[10:09:21] Loading reference annotation.
[10:09:22] Loading reference annotation.
[10:09:23] Loading reference annotation.
[10:09:25] Loading reference annotation.
[10:09:27] Loading reference annotation.
[Wed Dec 11 10:09:29 2013] Quantitating transcripts
You are using Cufflinks v2.1.1, which is the most recent release.
Command line:
cufflinks -o cuffmerge_out/ -F 0.05 -g annotation.gtf -q --overhang-tolerance 200 --library-type=transfrags -A 0.0 --min-frags-per-transfrag 0 --no-5-extend -p 16 cuffmerge_out/tmp/mergeSam_filemtqH90
[bam_header_read] EOF marker is absent.
[bam_header_read] invalid BAM binary header (this is not a BAM file).
File cuffmerge_out/tmp/mergeSam_filemtqH90 doesn't appear to be a valid BAM file, trying SAM...
[10:09:29] Loading reference annotation.
[10:09:33] Inspecting reads and determining fragment length distribution.
Processed 98978 loci.
> Map Properties:
> Normalized Map Mass: 358518.00
> Raw Map Mass: 358518.00
> Fragment Length Distribution: Truncated Gaussian (default)
> Default Mean: 200
> Default Std Dev: 80
[10:09:35] Assembling transcripts and estimating abundances.
Processed 98978 loci.
[Wed Dec 11 10:10:16 2013] Comparing against reference file annotation.gtf
You are using Cufflinks v2.1.1, which is the most recent release.
No fasta index found for genome.fas. Rebuilding, please wait..
Fasta index rebuilt.
[Wed Dec 11 10:11:22 2013] Comparing against reference file annotation.gtf
You are using Cufflinks v2.1.1, which is the most recent release.
DONE!
total 295M
drwx------ 3 iplant G-802086 4.0K Dec 11 10:12 .
drwxrwxr-x 7 iplant G-802086 4.0K Dec 11 10:12 ..
drwx------ 2 iplant G-802086 4.0K Dec 11 10:09 logs
-rw------- 1 iplant G-802086 148M Dec 11 10:12 merged.gtf
-rw------- 1 iplant G-802086 150M Dec 11 10:12 merged_with_ref_ids.gtf
/tmp/slurmd/job2214500/slurm_script: line 276: syntax error near unexpected token `;'
/tmp/slurmd/job2214500/slurm_script: line 276: `if ! [[ -s $SAM_1F ]]; then;'
--
Rion
On Dec 11, 2013, at 10:33 AM, Khalfan, Mohammed <mkhalfan at cshl.edu<mailto:mkhalfan at cshl.edu>> wrote:
Hi,
One of my jobs, 36113, seems to have completed with status “FINISHED”. There is no data in the specified archive path. I have seen this before. Can you tell me why this is happening/help me get this working properly?
Thank you,
Mohammed
Mohammed Khalfan
Bioinformatics Developer
DNA Learning Center
Cold Spring Harbor Laboratory
1 Bungtown Road
Cold Spring Harbor NY 11724
(516)367-5162
www.dnalc.org<http://www.dnalc.org/>
_______________________________________________
Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org<mailto:Iplant-api-dev at iplantcollaborative.org>
List Info and Archives: http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
One-click Unsubscribe: http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/dooley%40tacc.utexas.edu?unsub=1&unsubconfirm=1
_______________________________________________
Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org<mailto:Iplant-api-dev at iplantcollaborative.org>
List Info and Archives: http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
One-click Unsubscribe: http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/vaughn%40tacc.utexas.edu?unsub=1&unsubconfirm=1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.iplantcollaborative.org/pipermail/iplant-api-dev/attachments/20131211/277f0bee/attachment-0001.html
More information about the Iplant-api-dev
mailing list