[api-dev] [Fwd: jobs stuck in PROCESSING_INPUTS]

[Ghiban, Cornel]

-------- Forwarded Message --------
From: Cornel Ghiban <ghiban at cshl.edu<mailto:Cornel%20Ghiban%20%3cghiban at cshl.edu%3e>>
To: Discussion of iPlant API development <iplant-api-dev at cyverse.org<mailto:Discussion%20of%20iPlant%20API%20development%20%3ciplant-api-dev at cyverse.org%3e>>
Subject: jobs stuck in PROCESSING_INPUTS
Date: Tue, 2 Feb 2016 14:44:56 -0500

Hi,

Im' seeing quite a few jobs stuck in PROCESSING_INPUTS and at the same time I see that there is some data transferred that is not reflected in the job's history.



$ jobs-history 8188858798255968741-ee4acae9fffff7a7-0001-007
Job accepted and queued for submission.
Attempt 1 to stage job inputs
Identifying input files for staging
READ permission granted to abc123

$ jobs-output-list -L 8188858798255968741-ee4acae9fffff7a7-0001-007
-                ghiban         2100068352                   hy5_rep1.fastq 2016-02-02T13:22:19.000-06:00
$ jobs-output-list -L 8188858798255968741-ee4acae9fffff7a7-0001-007
-                ghiban         2355822592                   hy5_rep1.fastq 2016-02-02T13:23:12.000-06:00

$ jobs-history 8188858798255968741-ee4acae9fffff7a7-0001-007
Job accepted and queued for submission.
Attempt 1 to stage job inputs
Identifying input files for staging
READ permission granted to abc123

$ jobs-output-list -L 8188858798255968741-ee4acae9fffff7a7-0001-007
-                ghiban         2404188160                   hy5_rep1.fastq 2016-02-02T13:23:26.000-06:00
$ jobs-output-list -L 8188858798255968741-ee4acae9fffff7a7-0001-007
-                ghiban         2734475054                   hy5_rep1.fastq 2016-02-02T13:24:24.000-06:00

$ jobs-history 8188858798255968741-ee4acae9fffff7a7-0001-007
Job accepted and queued for submission.
Attempt 1 to stage job inputs
Identifying input files for staging
READ permission granted to abc123


After a while I have to manually stop these jobs..

I could stop them and re-submit them programatically  but I'm not sure how much to wait.

Thanks,
Cornel