[Iplant-api-dev] Problems adding filesystems / v2

Tue Oct 8 02:48:03 MST 2013

Hey Markus,

I have a few updates going out today to Agave. One will address the bug you saw. It came from a new parameter we added, (system.storage.mirror) to deal with different system sharing scenarios. If set to true, permissions set in the api will be mirrored to the system. Currently this only makes sense for IRODS systems. It will also be supported in S3 instances in the future.

Another feature rolling out is file histories. Prior to this there was no way for users to get provenance information about changes to files and folders made through Agave. With the new /files/history/system/<system_id>/<path> service you can get the complete history of a file's movement, permission grants and revocations, change operations, etc. This should be helpful when tracking down breaks in reproducability when the provenance change broke down due to user error.

The last update to the /files service is the addition of proper metadata urls in the response objects. When querying a specific file or folder, a reference to the metadata listing url for that object is given in the _links.metadata.href value. We may introduce convenience urls into the files service in the future, but the plan right now is to push everyone to the metadata service as the sole place to handle those kinds of operations.

Speaking of metadata, the metadata service got some TLC. Improved error handling, searching, and import mechanisms were added. I also updated the CLI to reflect the improved behavior. 

Lastly, we have a new notification service rolling out in the next day or two. This service is a beefed up pubsub service that allows you to subscribe to one or more events associated with the lifecycle of a resource. Notifications can be set at any time and accept both emails and webhooks. If you are using a webhook, the template variables described in the job and file tutorials are available along with a ton of other template variables specific to each resource. This will allow you to do things like set up a callback when a files starts and ends transferring with the same URL, for example, http://example.com/callback/?uuid=${UUID}&status=${STATUS}, http://example.com/callback/${JOB_ID}/${STATUS}, etc. In the near term, most people will benefit from being able to subscribe to multiple notification for a specific job. For other users, they will benefit from being able to use the notifications service to notify their users of status changes without having to run their own mail servers. 

I'll be pushing these changes in the morning after I get some sleep. I'll update the developer's site to reflect these changes during the day as well. If you run into any issues or have questions, please don't hesitate to let me know.

cheers,
Rion

________________________________________
From: Markus Binsteiner [m.binsteiner at auckland.ac.nz]
Sent: Monday, October 07, 2013 3:37 PM
To: Rion Dooley
Cc: iPlant API Developers Mailing List
Subject: Re: [Iplant-api-dev] Problems adding filesystems / v2

Thanks for this. So, now, my new system with id datafabric.auckland4
shows in when I call systems-list:

systems-list
 ✔
blacklight.psc.teragrid.org
datafabric.auckland4
lonestar4.tacc.teragrid.org
stampede.tacc.utexas.edu
data.iplantcollaborative.org
trestles.sdsc.teragrid.org
stampede
data
condor.opensciencegrid.org

But when I try to list the root directory of this system I get:

files-list -vS datafabric.auckland4
Calling curl -sku "markus:XXXXXX"
https://iplant-dev.tacc.utexas.edu/v2/files/listings/system/datafabric.auckland4/?pretty=true
 ✖  {
  "status" : "error",
  "message" : "No resource found for user with system id
'datafabric.auckland4'",
  "version" : "2.1.8-SNAPSHOT-r8484",
  "result" : null
}

Did I miss something?

Cheers,
Markus

On Mon, 2013-10-07 at 01:03 +0000, Rion Dooley wrote:
> No worries. The Agave CLI is also helpful for this kind of thing. Just run any command with the -V option and it will enable verbose output including the corresponding curl command to perform the action.
>
> http://agaveapi.co/agave-cli/
>
> Rion
>
> ________________________________________
> From: Markus Binsteiner [m.binsteiner at auckland.ac.nz]
> Sent: Sunday, October 06, 2013 8:01 PM
> To: Rion Dooley
> Cc: iPlant API Developers Mailing List
> Subject: Re: [Iplant-api-dev] Problems adding filesystems / v2
>
> Yup, that works. Thanks!
>
> On Mon, 2013-10-07 at 00:53 +0000, Rion Dooley wrote:
> > The curl command is doing a get. Try adding -XPOST -F before the
> > quoted file parameter.
> >
> >
> > -
> > Rion
> >
> > ----- Reply message -----
> > From: "Markus Binsteiner" <m.binsteiner at auckland.ac.nz>
> > To: "Rion Dooley" <dooley at tacc.utexas.edu>
> > Cc: "Discussion of iPlant API development"
> > <iplant-api-dev at iplantcollaborative.org>
> > Subject: [Iplant-api-dev] Problems adding filesystems / v2
> > Date: Sun, Oct 6, 2013 6:39 PM
> >
> >
> > Hi,
> >
> > I'm trying to add an irods as well as a gridftp filesystem to v2 of
> > the
> > Agave API. When issuing:
> >
> > curl -sku $CLIENT_SECRET:$CLIENT_KEY
> > "fileToUpload=@/home/markus/Documents/work/agave/datafabric.json"
> > https://iplant-dev.tacc.utexas.edu/v2/systems?pretty=true
> >
> > On this json file:
> >
> > {
> >     "id": "datafabric.auckland2",
> >     "name": "DataFabric",
> >     "status": "UP",
> >     "type": "STORAGE",
> >     "description": "Auckland node of NeSI DataFabric",
> >     "site": "auckland.ac.nz",
> >     "storage": {
> >         "host": "df.auckland.ac.nz",
> >         "port": 1247,
> >         "protocol": "IRODS",
> >         "rootDir": "/BeSTGRID/home/markus.binsteiner2",
> >         "auth": {
> >             "type": "PASSWORD",
> >             "password": "<myPassword>",
> >             "username": "markus.binsteiner2"
> >         },
> >         "resource": "irods.ceres.auckland.ac.nz",
> >         "zone": "BeSTGRID"
> >     }
> > }
> >
> > I'm getting a list of filesystems back, but the one I wanted to add is
> > not included. Same goes for any subsequent systems-list, which only
> > shows:
> >
> > blacklight.psc.teragrid.org
> > lonestar4.tacc.teragrid.org
> > stampede.tacc.utexas.edu
> > data.iplantcollaborative.org
> > trestles.sdsc.teragrid.org
> > stampede
> > data
> > condor.opensciencegrid.org
> >
> >
> > There is no error message either. Anything I'm missing?
> >
> > Cheers,
> > Markus
> >
> >
> > _______________________________________________
> > Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org
> > List Info and Archives:
> > http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
> > One-click Unsubscribe:
> > http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/dooley%40tacc.utexas.edu?unsub=1&unsubconfirm=1
> >
>
>