[Iplant-api-dev] Jobs Stuck At Pending

[Rion Dooley]

<expelative>
I’m aware and working on it. Your jobs are getting submitted twice and the latter is overwriting the former. 
</expelative>

—
Rion 

> On Jul 11, 2015, at 10:47 AM, Duvick, Jonathan P [GDCBS] <jduvick at iastate.edu> wrote:
> 
> Thanks for your work; my test jobs are completing but output is empty, and I'm seeing a lot of the 'Text file busy' messages like before...
> 
> Example:
> 
> job_id 5306259608984949221-e0bd34dffff8de6-0001-007
> 
> Also, 500 errors are pretty common when sending API requests to jobs and output/listings.
> 
> (Separate issue: the Arizona ntp server appears to be running 7 hours fast.)
> 
> Thanks,
> 
> Jon Duvick
> PlantGDB Manager
> http://www.plantgdb.org/
> Department of Genetics, Development and Cell Biology
> 2258 Molecular Biology Building
> Iowa State University
> Ames IA 50011
> 
> (515) 294-2360
> (515) 294-6755 FAX
> ________________________________________
> From: iplant-api-dev-bounces at iplantcollaborative.org <iplant-api-dev-bounces at iplantcollaborative.org> on behalf of John Fonner <jfonner at tacc.utexas.edu>
> Sent: Friday, July 10, 2015 3:13 AM
> To: Duvick, Jonathan P [GDCBS]
> Cc: Discussion of iPlant API development
> Subject: Re: [Iplant-api-dev] Jobs Stuck At Pending
> 
> Hi everyone,
> 
> Just wanted to send out a quick status update.  Agave is back online and
> everything appears to be healthy.  Jobs are flowing, and hopefully Rion
> can end his vigil and catch some sleep.  More info on bug fixes and
> features will come next week.  For now, though, please test things out and
> let us know if the service is working for you.
> 
> Thanks to the Agave team for all the hard work!
> 
> Thanks,
> Fonner
> 
> On 7/9/15, 12:21 PM, "iplant-api-dev-bounces at iplantcollaborative.org on
> behalf of Brian Corrie" <iplant-api-dev-bounces at iplantcollaborative.org on
> behalf of bcorrie at sfu.ca> wrote:
> 
>> Thanks for the update Rion... We will leave you alone, good luck... 8-)
>> 
>> Brian
>> 
>> On 09/07/2015 10:18 AM, Rion Dooley wrote:
>>> Submission is paused due to a critical issue in the api stemming from
>>> several factors. We have been working around the clock to address the
>>> problem and restore full service to iPlant as well as several other
>>> tenants. We have identified the problem, written a patch, and are
>>> currently load testing it on our staging servers. We hope to have
>>> service back up after lunch.
>>> 
>>> ‹
>>> Rion
>>> 
>>>> On Jul 9, 2015, at 12:13 PM, Barthelson, Roger A - (rogerab)
>>>> <rogerab at email.arizona.edu <mailto:rogerab at email.arizona.edu>> wrote:
>>>> 
>>>> All jobs I have run from the DE in the past 4 days have stalled at
>>>> submitted. Most of the time the data gets staged, but then nothing
>>>> happens. No outputs, logs, etc returned. Jobs are still listed in the
>>>> DE as submitted, but I think they have essentially failed because of
>>>> system errors.
>>>> 
>>>> Roger
>>>> --
>>>> Roger Barthelson Ph.D.
>>>> Scientific Analyst
>>>> iPlant Collaborative
>>>> BIO5 Institute, University of Arizona
>>>> Phone: 520-977-5249
>>>> Email: rogerab at email.arizona.edu <mailto:rogerab at email.arizona.edu>
>>>> Web: www.iplantcollaborative.org/ <http://www.iplantcollaborative.org/>
>>>> 
>>>> On July 9, 2015 at 8:24:55 AM, Fritz-Waters, Eric R [AN S]
>>>> (ercfrtz at iastate.edu <mailto:ercfrtz at iastate.edu>) wrote:
>>>> 
>>>>> I submitted some jobs via the api on Tuesday. They are still stuck at
>>>>> the Pending status. I also have some previous jobs that are stuck at
>>>>> the states they are in, both Staged and Pending.
>>>>> 
>>>>> -Eric Fritz-Waters
>>>>> _______________________________________________
>>>>> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
>>>>> <mailto:Iplant-api-dev at iplantcollaborative.org>
>>>>> List Info and Archives:
>>>>> http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
>>>>> One-click Unsubscribe:
>>>>> 
>>>>> http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/roge
>>>>> rab%40email.arizona.edu?unsub=1&unsubconfirm=1
>>>>> 
>>>> _______________________________________________
>>>> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
>>>> <mailto:Iplant-api-dev at iplantcollaborative.org>
>>>> List Info and Archives:
>>>> http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
>>>> One-click Unsubscribe:
>>>> 
>>>> http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/doole
>>>> y%40tacc.utexas.edu?unsub=1&unsubconfirm=1
>>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
>>> List Info and Archives:
>>> http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
>>> One-click Unsubscribe:
>>> http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/bcorri
>>> e%40sfu.ca?unsub=1&unsubconfirm=1
>>> 
>> _______________________________________________
>> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
>> List Info and Archives:
>> http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
>> One-click Unsubscribe:
>> http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/jfonner
>> %40tacc.utexas.edu?unsub=1&unsubconfirm=1
> 
> 
> _______________________________________________
> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
> List Info and Archives: http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
> One-click Unsubscribe: http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/jduvick%40iastate.edu?unsub=1&unsubconfirm=1
> 
> _______________________________________________
> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
> List Info and Archives: http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev  
> One-click Unsubscribe: http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/dooley%40tacc.utexas.edu?unsub=1&unsubconfirm=1