[Iplant-api-dev] Problems adding filesystems / v2
Rion Dooley
dooley at tacc.utexas.edu
Wed Oct 9 19:13:04 MST 2013
I bet it's because we don't have your myproxy server's CA cert. There is a chicken and egg problem with myproxy. Do you have a tarball you support there? Perhaps you could send over the tarball and I can install it on our servers.
Rion
________________________________________
From: Markus Binsteiner [m.binsteiner at auckland.ac.nz]
Sent: Wednesday, October 09, 2013 8:50 PM
To: Rion Dooley
Cc: iPlant API Developers Mailing List
Subject: Re: [Iplant-api-dev] Problems adding filesystems / v2
Hi Rion,
thanks a lot for the info, the notification service sounds very useful.
I managed to add our irods instance, and I could browse my files. I also
added our gridftp server, using MyProxy authentication:
{
"id": "gram.test",
"name": "Pan cluster storage",
"status": "UP",
"type": "STORAGE",
"description": "Auckland Pan cluster storage",
"site": "auckland.ac.nz",
"storage": {
"auth": {
"type": "X509",
"password": "myproxy_password",
"username": "markus",
"server": {
"name": "NeSI MyProxy server",
"endpoint": "myproxy.nesi.org.nz",
"port": 7512,
"protocol": "MYPROXY"
}
},
"homeDir": "/home/mbin029",
"rootDir": "/",
"protocol": "GRIDFTP",
"port": 22,
"host": "gram.uoa.nesi.org.nz"
},
"default": false
}
But when I get try to list files I get:
Unable to establish a connection to the remote server.Failed to retrieve
a credential from myproxy.nesi.org.nz. GSI data will be unavailable.
{
"status" : "error",
"message" : "Unable to establish a connection to the remote
server.Failed to retrieve a credential from myproxy.nesi.org.nz. GSI
data will be unavailable.",
"version" : "2.1.8-SNAPSHOT-r8536"
}
Next I'll try to embedd the proxy directly in the system-creation json.
Do I need to replace all carriage returns with '\n' ( like in the
documentation on http://agaveapi.co/system-management/ ) or can I just
use the proxy as is with the newlines still there?
Best,
Markus
On Tue, 2013-10-08 at 09:48 +0000, Rion Dooley wrote:
> Hey Markus,
>
> I have a few updates going out today to Agave. One will address the bug you saw. It came from a new parameter we added, (system.storage.mirror) to deal with different system sharing scenarios. If set to true, permissions set in the api will be mirrored to the system. Currently this only makes sense for IRODS systems. It will also be supported in S3 instances in the future.
>
> Another feature rolling out is file histories. Prior to this there was no way for users to get provenance information about changes to files and folders made through Agave. With the new /files/history/system/<system_id>/<path> service you can get the complete history of a file's movement, permission grants and revocations, change operations, etc. This should be helpful when tracking down breaks in reproducability when the provenance change broke down due to user error.
>
> The last update to the /files service is the addition of proper metadata urls in the response objects. When querying a specific file or folder, a reference to the metadata listing url for that object is given in the _links.metadata.href value. We may introduce convenience urls into the files service in the future, but the plan right now is to push everyone to the metadata service as the sole place to handle those kinds of operations.
>
> Speaking of metadata, the metadata service got some TLC. Improved error handling, searching, and import mechanisms were added. I also updated the CLI to reflect the improved behavior.
>
> Lastly, we have a new notification service rolling out in the next day or two. This service is a beefed up pubsub service that allows you to subscribe to one or more events associated with the lifecycle of a resource. Notifications can be set at any time and accept both emails and webhooks. If you are using a webhook, the template variables described in the job and file tutorials are available along with a ton of other template variables specific to each resource. This will allow you to do things like set up a callback when a files starts and ends transferring with the same URL, for example, http://example.com/callback/?uuid=${UUID}&status=${STATUS}, http://example.com/callback/${JOB_ID}/${STATUS}, etc. In the near term, most people will benefit from being able to subscribe to multiple notification for a specific job. For other users, they will benefit from being able to use the notifications service to notify their users of status changes without having to run their own mail servers.
>
> I'll be pushing these changes in the morning after I get some sleep. I'll update the developer's site to reflect these changes during the day as well. If you run into any issues or have questions, please don't hesitate to let me know.
>
> cheers,
> Rion
>
>
> ________________________________________
> From: Markus Binsteiner [m.binsteiner at auckland.ac.nz]
> Sent: Monday, October 07, 2013 3:37 PM
> To: Rion Dooley
> Cc: iPlant API Developers Mailing List
> Subject: Re: [Iplant-api-dev] Problems adding filesystems / v2
>
> Thanks for this. So, now, my new system with id datafabric.auckland4
> shows in when I call systems-list:
>
> systems-list
> ✔
> blacklight.psc.teragrid.org
> datafabric.auckland4
> lonestar4.tacc.teragrid.org
> stampede.tacc.utexas.edu
> data.iplantcollaborative.org
> trestles.sdsc.teragrid.org
> stampede
> data
> condor.opensciencegrid.org
>
>
> But when I try to list the root directory of this system I get:
>
> files-list -vS datafabric.auckland4
> Calling curl -sku "markus:XXXXXX"
> https://iplant-dev.tacc.utexas.edu/v2/files/listings/system/datafabric.auckland4/?pretty=true
> ✖ {
> "status" : "error",
> "message" : "No resource found for user with system id
> 'datafabric.auckland4'",
> "version" : "2.1.8-SNAPSHOT-r8484",
> "result" : null
> }
>
> Did I miss something?
>
> Cheers,
> Markus
>
> On Mon, 2013-10-07 at 01:03 +0000, Rion Dooley wrote:
> > No worries. The Agave CLI is also helpful for this kind of thing. Just run any command with the -V option and it will enable verbose output including the corresponding curl command to perform the action.
> >
> > http://agaveapi.co/agave-cli/
> >
> > Rion
> >
> > ________________________________________
> > From: Markus Binsteiner [m.binsteiner at auckland.ac.nz]
> > Sent: Sunday, October 06, 2013 8:01 PM
> > To: Rion Dooley
> > Cc: iPlant API Developers Mailing List
> > Subject: Re: [Iplant-api-dev] Problems adding filesystems / v2
> >
> > Yup, that works. Thanks!
> >
> > On Mon, 2013-10-07 at 00:53 +0000, Rion Dooley wrote:
> > > The curl command is doing a get. Try adding -XPOST -F before the
> > > quoted file parameter.
> > >
> > >
> > > -
> > > Rion
> > >
> > > ----- Reply message -----
> > > From: "Markus Binsteiner" <m.binsteiner at auckland.ac.nz>
> > > To: "Rion Dooley" <dooley at tacc.utexas.edu>
> > > Cc: "Discussion of iPlant API development"
> > > <iplant-api-dev at iplantcollaborative.org>
> > > Subject: [Iplant-api-dev] Problems adding filesystems / v2
> > > Date: Sun, Oct 6, 2013 6:39 PM
> > >
> > >
> > > Hi,
> > >
> > > I'm trying to add an irods as well as a gridftp filesystem to v2 of
> > > the
> > > Agave API. When issuing:
> > >
> > > curl -sku $CLIENT_SECRET:$CLIENT_KEY
> > > "fileToUpload=@/home/markus/Documents/work/agave/datafabric.json"
> > > https://iplant-dev.tacc.utexas.edu/v2/systems?pretty=true
> > >
> > > On this json file:
> > >
> > > {
> > > "id": "datafabric.auckland2",
> > > "name": "DataFabric",
> > > "status": "UP",
> > > "type": "STORAGE",
> > > "description": "Auckland node of NeSI DataFabric",
> > > "site": "auckland.ac.nz",
> > > "storage": {
> > > "host": "df.auckland.ac.nz",
> > > "port": 1247,
> > > "protocol": "IRODS",
> > > "rootDir": "/BeSTGRID/home/markus.binsteiner2",
> > > "auth": {
> > > "type": "PASSWORD",
> > > "password": "<myPassword>",
> > > "username": "markus.binsteiner2"
> > > },
> > > "resource": "irods.ceres.auckland.ac.nz",
> > > "zone": "BeSTGRID"
> > > }
> > > }
> > >
> > > I'm getting a list of filesystems back, but the one I wanted to add is
> > > not included. Same goes for any subsequent systems-list, which only
> > > shows:
> > >
> > > blacklight.psc.teragrid.org
> > > lonestar4.tacc.teragrid.org
> > > stampede.tacc.utexas.edu
> > > data.iplantcollaborative.org
> > > trestles.sdsc.teragrid.org
> > > stampede
> > > data
> > > condor.opensciencegrid.org
> > >
> > >
> > > There is no error message either. Anything I'm missing?
> > >
> > > Cheers,
> > > Markus
> > >
> > >
> > > _______________________________________________
> > > Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
> > > List Info and Archives:
> > > http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
> > > One-click Unsubscribe:
> > > http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/dooley%40tacc.utexas.edu?unsub=1&unsubconfirm=1
> > >
> >
> >
>
>
More information about the Iplant-api-dev
mailing list