[Iplant-api-dev] many jobs in PENDING/SUBMITTING

[Ghiban, Cornel]

Hi Rion,

This is very interesting.

So a user submitting 20 jobs to the "normal" queue, will have 5 active and 15 PENDING/SUBMITTING? Once the active jobs FINISH or FAIL, PENDING jobs will become active? Am I right? I just want to understand so I can explain it further.

Thanks,
Cornel

On Thu, 2015-06-04 at 14:58 +0000, Rion Dooley wrote:
Hi Cornel,


Sure. For reference, here is the documentation on queues and quotas in the Agave Developer’s Site:


http://preview.agaveapi.co/documentation/tutorials/system-management-tutorial/#defining-batch-queues


The system in question is dnalc.lonestar.tacc.utexas.edu<http://dnalc.lonestar.tacc.utexas.edu>. This is a system you’ve published that essentially clones lonestar.tacc.utexas.edu<http://lonestar.tacc.utexas.edu>, but with different queue and quota policies. Explanation is inline below:


{
    ...
    "maxSystemJobs": 50, // The total number of jobs submitted through Agave
//  which can concurrently be active on the system across all queues
    "maxSystemJobsPerUser": 5,// The total number of jobs submitted through Agave which can
//  concurrently be active for a single user across all queues
    "queues": [
        {
            "customDirectives": "-A iPlant-Master",
            "default": true, // Tells Agave to use this queue if none is specified in a job request and
//  the job fits the specified bounds of memory, nodes, etc.
            "maxJobs": 50, // The total number of jobs submitted through Agave which can
//  concurrently be running in this queue. Note that this is equal to the
            "maxMemoryPerNode": 24,// The maximum number of nodes a job can request when using this queue
            "maxProcessorsPerNode": 12,// The maximum processors per node this job can specify in the request.
            "maxNodes": 2,// The maximum number of nodes a job can request when using this queue
            "maxProcessorsPerNode": 12,// The maximum processors per node this job can specify in the request.
//  When more advanced schedulers are used and jobs do not specify a node
//  request value, this can be used to calculate the node request
            "maxUserJobs": -1,// Maximum number of jobs a user may concurrently have active in this queue.
//  -1 means no limit, however the maxSystemJobsPerUser value of 5 will prevent
//  a user from having more than 5 jobs active.
            "name": "normal"
        },
        {
            "customDirectives": "-A iPlant-Master",
            "default": false,
            "maxJobs": 5,// The total number of jobs submitted through Agave which can
//  concurrently be running in this queue. Note that value is well below the
//  system limit and a user can fill this queue with their quota alone.
            "maxMemoryPerNode": 24,// The maximum number of nodes a job can request when using this queue
            "maxProcessorsPerNode": 12,// The maximum processors per node this job can specify in the request.
            "maxNodes": 1,// The maximum number of nodes a job can request when using this queue
            "maxProcessorsPerNode": 12,// The maximum processors per node this job can specify in the request.
            "maxProcessorsPerNode": 12,// The maximum processors per node this job can specify in the request.
//  When more advanced schedulers are used and jobs do not specify a node
//  request value, this can be used to calculate the node request
            "maxUserJobs": -1,// Maximum number of jobs a user may concurrently have active in this queue.
//  -1 means no limit, however the maxSystemJobsPerUser value of 5 will prevent
//  a user from having more than 5 jobs active.
            "name": "development"
        }
    ],
    …
}




—
Rion

On Jun 4, 2015, at 4:33 PM, Ghiban, Cornel <ghiban at cshl.edu<mailto:ghiban at cshl.edu>> wrote:

Hi Rion,

Could you, please, explain how the user quota works? Some of the users
have indeed submitted a lot of jobs.

Should I have a buffer queue that "releases" their jobs, lets say max 10
jobs/hr, so they don't clog Agave?

Thanks,
Cornel

On Thu, 2015-06-04 at 14:10 +0000, Rion Dooley wrote:
I imagine so. i only see 5 people with pending jobs and they are both
being throttled due to user-level quotas on the systems and queues
they are requesting.