[api-dev] Update: Agave API and general Cyverse Data Store issues

[Matthew Vaughn]

I wanted to bring to your collective attention a couple of issues that’ve emerged with API access to and management of data stored in the Cyverse Data Store. 

Users are reporting sporadic access errors to Data Store files when submitting jobs (occurs during the staging of assets or inputs portions of the job lifecycle) as well as ongoing issues with failed data transfers from jobs after they have finished. This is due to two interacting problems. 

First, somehow, and we don’t quite understand why, when Agave interacts with the Cyverse iRODS server, one of the Java libraries its based on is interpreting the access controls differently than it was a few days ago. This is mysterious, frankly, as we don’t believe anyone changed configurations or code. The important thing is that the fix has been identified and tested, released to Agave’s 2.1.6 master branch, and will be deployed during maintenance next Tuesday. 

https://bitbucket.org/taccaci/agave/commits/34db6850d99bdc32799208fdb7cd8a664746f1ec#chg-agave-postits

Second, there are network configuration issues between UA and TACC that are causing degradation or interruption of iRODS transfers. This has been a long-standing minor issue, but has gotten worse due to a new configuration somewhere along the route between the two institutions. We believe the solution is to move the Cyverse iRODS catalog server to a different section of the UA network, which should solve everyone’s issues (not just the APIs) on this front. We’re trying to schedule that move as soon as possible.

If you are currently using Cyverse’s Agave API in a way that interacts with the Cyverse Data Store, these issues will impact you and you may wish to consider pausing your activities until we announce the fixes are in place. If you are not interacting with Cyverse Data Store in your work (for instance, you are using your own storageSystems), you should be able to continue work without disruption.