[Iplant-api-dev] Jobs Stuck At Pending

[Duvick, Jonathan P [GDCBS]]

Thanks for your work; my test jobs are completing but output is empty, and I'm seeing a lot of the 'Text file busy' messages like before...

Example:

job_id 5306259608984949221-e0bd34dffff8de6-0001-007

Also, 500 errors are pretty common when sending API requests to jobs and output/listings.

(Separate issue: the Arizona ntp server appears to be running 7 hours fast.)

Thanks,

Jon Duvick
PlantGDB Manager
http://www.plantgdb.org/
Department of Genetics, Development and Cell Biology
2258 Molecular Biology Building
Iowa State University
Ames IA 50011

(515) 294-2360
(515) 294-6755 FAX
________________________________________
From: iplant-api-dev-bounces at iplantcollaborative.org <iplant-api-dev-bounces at iplantcollaborative.org> on behalf of John Fonner <jfonner at tacc.utexas.edu>
Sent: Friday, July 10, 2015 3:13 AM
To: Duvick, Jonathan P [GDCBS]
Cc: Discussion of iPlant API development
Subject: Re: [Iplant-api-dev] Jobs Stuck At Pending

Hi everyone,

Just wanted to send out a quick status update.  Agave is back online and
everything appears to be healthy.  Jobs are flowing, and hopefully Rion
can end his vigil and catch some sleep.  More info on bug fixes and
features will come next week.  For now, though, please test things out and
let us know if the service is working for you.

Thanks to the Agave team for all the hard work!

Thanks,
Fonner

On 7/9/15, 12:21 PM, "iplant-api-dev-bounces at iplantcollaborative.org on
behalf of Brian Corrie" <iplant-api-dev-bounces at iplantcollaborative.org on
behalf of bcorrie at sfu.ca> wrote:

>Thanks for the update Rion... We will leave you alone, good luck... 8-)
>
>Brian
>
>On 09/07/2015 10:18 AM, Rion Dooley wrote:
>> Submission is paused due to a critical issue in the api stemming from
>> several factors. We have been working around the clock to address the
>> problem and restore full service to iPlant as well as several other
>> tenants. We have identified the problem, written a patch, and are
>> currently load testing it on our staging servers. We hope to have
>> service back up after lunch.
>>
>> ‹
>> Rion
>>
>>> On Jul 9, 2015, at 12:13 PM, Barthelson, Roger A - (rogerab)
>>> <rogerab at email.arizona.edu <mailto:rogerab at email.arizona.edu>> wrote:
>>>
>>> All jobs I have run from the DE in the past 4 days have stalled at
>>> submitted. Most of the time the data gets staged, but then nothing
>>> happens. No outputs, logs, etc returned. Jobs are still listed in the
>>> DE as submitted, but I think they have essentially failed because of
>>> system errors.
>>>
>>> Roger
>>> --
>>> Roger Barthelson Ph.D.
>>> Scientific Analyst
>>> iPlant Collaborative
>>> BIO5 Institute, University of Arizona
>>> Phone: 520-977-5249
>>> Email: rogerab at email.arizona.edu <mailto:rogerab at email.arizona.edu>
>>> Web: www.iplantcollaborative.org/ <http://www.iplantcollaborative.org/>
>>>
>>> On July 9, 2015 at 8:24:55 AM, Fritz-Waters, Eric R [AN S]
>>> (ercfrtz at iastate.edu <mailto:ercfrtz at iastate.edu>) wrote:
>>>
>>>> I submitted some jobs via the api on Tuesday. They are still stuck at
>>>> the Pending status. I also have some previous jobs that are stuck at
>>>> the states they are in, both Staged and Pending.
>>>>
>>>> -Eric Fritz-Waters
>>>> _______________________________________________
>>>> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
>>>> <mailto:Iplant-api-dev at iplantcollaborative.org>
>>>> List Info and Archives:
>>>> http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
>>>> One-click Unsubscribe:
>>>>
>>>>http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/roge
>>>>rab%40email.arizona.edu?unsub=1&unsubconfirm=1
>>>>
>>> _______________________________________________
>>> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
>>> <mailto:Iplant-api-dev at iplantcollaborative.org>
>>> List Info and Archives:
>>> http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
>>> One-click Unsubscribe:
>>>
>>>http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/doole
>>>y%40tacc.utexas.edu?unsub=1&unsubconfirm=1
>>>
>>
>>
>>
>> _______________________________________________
>> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
>> List Info and Archives:
>>http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
>> One-click Unsubscribe:
>>http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/bcorri
>>e%40sfu.ca?unsub=1&unsubconfirm=1
>>
>_______________________________________________
>Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
>List Info and Archives:
>http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
>One-click Unsubscribe:
>http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/jfonner
>%40tacc.utexas.edu?unsub=1&unsubconfirm=1


_______________________________________________
Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
List Info and Archives: http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
One-click Unsubscribe: http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/jduvick%40iastate.edu?unsub=1&unsubconfirm=1