[Iplant-api-dev] Error submitting jobs to stampede

[Darren Boss]

This is happening in the DE as well. This output is from the
condor-stderr-0 file in my job log. Scroll right down to the bottom to
see the Unknown CA error.


LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: POST
https://foundation.iplantcollaborative.org/auth-v1/
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 222 bytes
LWP::UserAgent::request: Simple response: OK
Application_Id: cd-hit-4.6.1
LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET
https://foundation.iplantcollaborative.org/apps-v1/apps/cd-hit-4.6.1
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 3614 bytes
LWP::UserAgent::request: Simple response: OK
https://foundation.iplantcollaborative.org/io-v1/io/list/dboss/analyses
LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET
https://foundation.iplantcollaborative.org/io-v1/io/list/dboss/analyses
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 3500 bytes
LWP::UserAgent::request: Simple response: OK
LWP::UserAgent::new: ()
Checking that stampede.tacc.xsede.org is accepting job submissions
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET
https://foundation.iplantcollaborative.org/apps-v1/systems/list/stampede.tacc.xsede.org
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 454 bytes
LWP::UserAgent::request: Simple response: OK
LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: POST
https://foundation.iplantcollaborative.org/apps-v1/job
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 835 bytes
LWP::UserAgent::request: Simple response: OK
LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET
https://foundation.iplantcollaborative.org/apps-v1/job/26240
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 835 bytes
LWP::UserAgent::request: Simple response: OK
apps-v1/jobs/26240    PENDING    re-poll: 5s
LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET
https://foundation.iplantcollaborative.org/apps-v1/job/26240
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 845 bytes
LWP::UserAgent::request: Simple response: OK
apps-v1/jobs/26240    PROCESSING_INPUTS    re-poll: 5s
LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET
https://foundation.iplantcollaborative.org/apps-v1/job/26240
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 836 bytes
LWP::UserAgent::request: Simple response: OK
apps-v1/jobs/26240    STAGING_JOB    re-poll: 5s
LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET
https://foundation.iplantcollaborative.org/apps-v1/job/26240
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 1058 bytes
LWP::UserAgent::request: Simple response: OK
apps-v1/jobs/26240    FAILED    re-poll: 5s
    Failed to submit job 26240: org.globus.common.ChainedIOException:
Authentication failed [Caused by: Failure unspecified at GSS-API level
[Caused by: Unknown CA]]s

On Fri, Aug 30, 2013 at 7:39 AM, Darren Boss <dboss at email.arizona.edu> wrote:
> Any information on this issue? I'm still unable to submit jobs this morning.
>
> On Thu, Aug 29, 2013 at 2:43 PM, Jennewein, Douglas M
> <Doug.Jennewein at usd.edu> wrote:
>> I'm seeing these as well today for lonestar, job 26176.  Jobs fail immediately with " Failed to retrieve credential from MyProxy"
>>
>> Doug
>>
>> -----Original Message-----
>> From: iplant-api-dev-bounces at iplantcollaborative.org [mailto:iplant-api-dev-bounces at iplantcollaborative.org] On Behalf Of Darren Boss
>> Sent: Thursday, August 29, 2013 11:50 AM
>> To: Jennewein, Douglas M
>> Cc: Discussion of iPlant API development
>> Subject: Re: [Iplant-api-dev] Error submitting jobs to stampede
>>
>> I'm getting a different error now:
>>
>> {u'message': u'',
>>  u'result': {u'archive': True,
>>   u'archivePath': u'/dboss/archive/jobs/job-26148-head-test',
>>   u'callbackUrl': None,
>>   u'endTime': 1377794821000,
>>   u'id': 26148,
>>   u'inputs': [],
>>   u'memory': 1,
>>   u'message': u'Failed to submit job 26148: Failed to retrieve credential from MyProxy',
>>   u'name': u'Head test',
>>   u'outputPath': None,
>>   u'outputUrl':
>> u'https://foundation.iplantcollaborative.org/io-v1/io/list/dboss/archive/jobs/job-26148-head-test',
>>   u'owner': u'dboss',
>>   u'parameters': [{u'numberofbytes': u''}, {u'numberoflines': u'10'}],
>>   u'permissions': [],
>>   u'processors': 1,
>>   u'requestedTime': u'1:00:00',
>>   u'retries': 2,
>>   u'software': u'head-stampede-5.97u2',
>>   u'startTime': None,
>>   u'status': u'FAILED',
>>   u'submitTime': 1377794806000,
>>   u'system': u'stampede.tacc.xsede.org'},
>>  u'status': u'success'}
>>
>> On Thu, Aug 29, 2013 at 11:56 AM, Darren Boss <dboss at email.arizona.edu> wrote:
>>> I'm seeing this error message in the json returned by a job submission request.
>>>
>>> org.globus.common.ChainedIOException: Authentication failed [Caused
>>> by: Failure unspecified at GSS-API level [Caused by: Unknown CA]
>>>
>>> Here is the full message. Sorry about the formatting:
>>> {u'message': u'',
>>>  u'result': {u'archive': True,
>>>   u'archivePath': u'/dboss/archive/jobs/job-26132-head-test',
>>>   u'callbackUrl': None,
>>>   u'endTime': 1377791001000,
>>>   u'id': 26132,
>>>   u'inputs': [],
>>>   u'memory': 1,
>>>   u'message': u'Failed to submit job 26132:
>>> org.globus.common.ChainedIOException: Authentication failed [Caused
>>> by: Failure unspecified at GSS-API level [Caused by: Unknown CA]]',
>>>   u'name': u'Head test',
>>>   u'outputPath': None,
>>>   u'outputUrl':
>>> u'https://foundation.iplantcollaborative.org/io-v1/io/list/dboss/archive/jobs/job-26132-head-test',
>>>   u'owner': u'dboss',
>>>   u'parameters': [{u'numberofbytes': u''}, {u'numberoflines': u'10'}],
>>>   u'permissions': [],
>>>   u'processors': 1,
>>>   u'requestedTime': u'1:00:00',
>>>   u'retries': 2,
>>>   u'software': u'head-stampede-5.97u2',
>>>   u'startTime': None,
>>>   u'status': u'FAILED',
>>>   u'submitTime': 1377790990000,
>>>   u'system': u'stampede.tacc.xsede.org'},
>>>  u'status': u'success'}
>> _______________________________________________
>> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
>> List Info and Archives: http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
>> One-click Unsubscribe: http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/doug.jennewein%40usd.edu?unsub=1&unsubconfirm=1
>>
>> _______________________________________________
>> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
>> List Info and Archives: http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
>> One-click Unsubscribe: http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/dboss%40email.arizona.edu?unsub=1&unsubconfirm=1