[Iplant-api-dev] fAPI Jobs fail top stage intermittently

[Rion Dooley]

Both failures are due to stampede's gridftp server being unavailable after several attempts. Globus online was probably ddos'ing that node or someone else was probably pushing a lot of data through it. This happens from time to time. Foundation automatically retries the submission multiple times, but during times of spiked utilization, it can still fail, as you see. It's best to implement a retry mechanism on your side when you see this happen. There is generally no harm in doing so, and it's a best practice when interacting with distributed systems.

-
Rion

On Jan 27, 2014, at 1:48 PM, "Jennewein, Douglas M" <Doug.Jennewein at usd.edu<mailto:Doug.Jennewein at usd.edu>> wrote:

Job 41118 failed to stage, but an identically launched job (41127) staged and ran normally 5 minutes later.

Similarly, job 41120 failed to stage, but later job 41129 ran successfully with the same parameters.

Both failed jobs give errors like the following:
Failed to submit job 41129: org.globus.ftp.exception.ServerException: Server refused performing the request. Custom message: (error code 1) [Nested exception message: Custom message: Unexpected reply: 451 active connection to server failed java.net<http://java.net>.SocketException: Connection reset java.net<http://java.net>.SocketException: Connection reset at java.net<http://java.net>.SocketInputStream.read(SocketInputStream.java:168) at org.globus.gsi.gssapi.SSLUtil.read(SSLUtil.java:42) at org.globus.gsi.gssapi.net.impl.GSIGssInputStream.readToken(GSIGssInputStream.java:69) at org.globus.gsi.gssapi.net.impl.GSIGssInputStream.readHandshakeToken(GSIGssInputStream.java:59) at org.globus.gsi.gssapi.net.impl.GSIGssSocket.readToken(GSIGssSocket.java:65) at org.globus.gsi.gssapi.net.GssSocket.authenticateClient(GssSocket.java:115) at org.globus.gsi.gssapi.net.GssSocket.startHandshake(GssSocket.java:145) at org.globus.gsi.gssapi.net.GssSocket.getOutputStream(GssSocket.java:166) at org.globus.ftp.extended.GridFTPServerFacade.authenticate(GridFTPServerFacade.java:550) at org.globus.ftp.dc.GridFTPActiveConnectTask.execute(GridFTPActiveConnectTask.java:86) at org.globus.ftp.dc.TaskThread.run(TaskThread.java:71) at java.lang.Thread.run(Thread.java:662) ] [Nested exception is org.globus.ftp.exception.UnexpectedReplyCodeException: Custom message: Unexpected reply: 451 active connection to server failed java.net<http://java.net>.SocketException: Connection reset java.net<http://java.net>.SocketException: Connection reset at java.net<http://java.net>.SocketInputStream.read(SocketInputStream.java:168) at org.globus.gsi.gssapi.SSLUtil.read(SSLUtil.java:42) at org.globus.gsi.gssapi.net.impl.GSIGssInputStream.readToken(GSIGssInputStream.java:69) at org.globus.gsi.gssapi.net.impl.GSIGssInputStream.readHandshakeToken(GSIGssInputStream.java:59) at org.globus.gsi.gssapi.net.impl.GSIGssSocket.readToken(GSIGssSocket.java:65) at org.globus.gsi.gssapi.net.GssSocket.authenticateClient(GssSocket.java:115) at org.globus.gsi.gssapi.net.GssSocket.startHandshake(GssSocket.java:145) at org.globus.gsi.gssapi.net.GssSocket.getOutputStream(GssSocket.java:166) at org.globus.ftp.extended.GridFTPServerFacade.authenticate(GridFTPServerFacade.java:550) at org.globus.ftp.dc.GridFTPActiveConnectTask.execute(GridFTPActiveConnectTask.java:86) at org.globus.ftp.dc.TaskThread.run(TaskThread.java:71) at java.lang.Thread.run(Thread.java:662) ]

_______________________________________________
Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org<mailto:Iplant-api-dev at iplantcollaborative.org>
List Info and Archives: http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
One-click Unsubscribe: http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/dooley%40tacc.utexas.edu?unsub=1&unsubconfirm=1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.iplantcollaborative.org/pipermail/iplant-api-dev/attachments/20140127/12a480d4/attachment-0001.html