[Iplant-api-dev] many jobs in PENDING/SUBMITTING

[Rion Dooley]

Hi Cornel,

Sure. For reference, here is the documentation on queues and quotas in the Agave Developer’s Site:

http://preview.agaveapi.co/documentation/tutorials/system-management-tutorial/#defining-batch-queues

The system in question is dnalc.lonestar.tacc.utexas.edu<http://dnalc.lonestar.tacc.utexas.edu>. This is a system you’ve published that essentially clones lonestar.tacc.utexas.edu<http://lonestar.tacc.utexas.edu>, but with different queue and quota policies. Explanation is inline below:

{
    ...
    "maxSystemJobs": 50, // The total number of jobs submitted through Agave
//  which can concurrently be active on the system across all queues
    "maxSystemJobsPerUser": 5, // The total number of jobs submitted through Agave which can
//  concurrently be active for a single user across all queues
    "queues": [
        {
            "customDirectives": "-A iPlant-Master",
            "default": true, // Tells Agave to use this queue if none is specified in a job request and
//  the job fits the specified bounds of memory, nodes, etc.
            "maxJobs": 50, // The total number of jobs submitted through Agave which can
//  concurrently be running in this queue. Note that this is equal to the
            "maxMemoryPerNode": 24, // The maximum number of nodes a job can request when using this queue
            "maxProcessorsPerNode": 12, // The maximum processors per node this job can specify in the request.
            "maxNodes": 2, // The maximum number of nodes a job can request when using this queue
            "maxProcessorsPerNode": 12, // The maximum processors per node this job can specify in the request.
//  When more advanced schedulers are used and jobs do not specify a node
//  request value, this can be used to calculate the node request
            "maxUserJobs": -1, // Maximum number of jobs a user may concurrently have active in this queue.
//  -1 means no limit, however the maxSystemJobsPerUser value of 5 will prevent
//  a user from having more than 5 jobs active.
            "name": "normal"
        },
        {
            "customDirectives": "-A iPlant-Master",
            "default": false,
            "maxJobs": 5, // The total number of jobs submitted through Agave which can
//  concurrently be running in this queue. Note that value is well below the
//  system limit and a user can fill this queue with their quota alone.
            "maxMemoryPerNode": 24, // The maximum number of nodes a job can request when using this queue
            "maxProcessorsPerNode": 12, // The maximum processors per node this job can specify in the request.
            "maxNodes": 1, // The maximum number of nodes a job can request when using this queue
            "maxProcessorsPerNode": 12, // The maximum processors per node this job can specify in the request.
            "maxProcessorsPerNode": 12, // The maximum processors per node this job can specify in the request.
//  When more advanced schedulers are used and jobs do not specify a node
//  request value, this can be used to calculate the node request
            "maxUserJobs": -1, // Maximum number of jobs a user may concurrently have active in this queue.
//  -1 means no limit, however the maxSystemJobsPerUser value of 5 will prevent
//  a user from having more than 5 jobs active.
            "name": "development"
        }
    ],
    …
}


—
Rion

On Jun 4, 2015, at 4:33 PM, Ghiban, Cornel <ghiban at cshl.edu<mailto:ghiban at cshl.edu>> wrote:

Hi Rion,

Could you, please, explain how the user quota works? Some of the users
have indeed submitted a lot of jobs.

Should I have a buffer queue that "releases" their jobs, lets say max 10
jobs/hr, so they don't clog Agave?

Thanks,
Cornel

On Thu, 2015-06-04 at 14:10 +0000, Rion Dooley wrote:
I imagine so. i only see 5 people with pending jobs and they are both
being throttled due to user-level quotas on the systems and queues
they are requesting.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.iplantcollaborative.org/pipermail/iplant-api-dev/attachments/20150604/6c8b606e/attachment-0001.html