[Iplant-api-dev] many jobs in PENDING/SUBMITTING

[Rion Dooley]

Exactly. This allows you to submit everything you want to whenever you want to. Agave will stage data just in time for your job to run and keep a lookout for the next available opening. That’s why you don’t get rejected when you submit 20 jobs despite only actually being allowed to have 5 in queue on the HPC system at any given moment.

—
Rion 

> On Jun 4, 2015, at 5:10 PM, Ghiban, Cornel <ghiban at cshl.edu> wrote:
> 
> Hi Rion,
> 
> This is very interesting.
> 
> So a user submitting 20 jobs to the "normal" queue, will have 5 active and 15 PENDING/SUBMITTING? Once the active jobs FINISH or FAIL, PENDING jobs will become active? Am I right? I just want to understand so I can explain it further.
> 
> Thanks,
> Cornel
> 
> On Thu, 2015-06-04 at 14:58 +0000, Rion Dooley wrote:
> Hi Cornel,
> 
> 
> Sure. For reference, here is the documentation on queues and quotas in the Agave Developer’s Site:
> 
> 
> http://preview.agaveapi.co/documentation/tutorials/system-management-tutorial/#defining-batch-queues
> 
> 
> The system in question is dnalc.lonestar.tacc.utexas.edu<http://dnalc.lonestar.tacc.utexas.edu>. This is a system you’ve published that essentially clones lonestar.tacc.utexas.edu<http://lonestar.tacc.utexas.edu>, but with different queue and quota policies. Explanation is inline below:
> 
> 
> {
>    ...
>    "maxSystemJobs": 50, // The total number of jobs submitted through Agave
> //  which can concurrently be active on the system across all queues
>    "maxSystemJobsPerUser": 5,// The total number of jobs submitted through Agave which can
> //  concurrently be active for a single user across all queues
>    "queues": [
>        {
>            "customDirectives": "-A iPlant-Master",
>            "default": true, // Tells Agave to use this queue if none is specified in a job request and
> //  the job fits the specified bounds of memory, nodes, etc.
>            "maxJobs": 50, // The total number of jobs submitted through Agave which can
> //  concurrently be running in this queue. Note that this is equal to the
>            "maxMemoryPerNode": 24,// The maximum number of nodes a job can request when using this queue
>            "maxProcessorsPerNode": 12,// The maximum processors per node this job can specify in the request.
>            "maxNodes": 2,// The maximum number of nodes a job can request when using this queue
>            "maxProcessorsPerNode": 12,// The maximum processors per node this job can specify in the request.
> //  When more advanced schedulers are used and jobs do not specify a node
> //  request value, this can be used to calculate the node request
>            "maxUserJobs": -1,// Maximum number of jobs a user may concurrently have active in this queue.
> //  -1 means no limit, however the maxSystemJobsPerUser value of 5 will prevent
> //  a user from having more than 5 jobs active.
>            "name": "normal"
>        },
>        {
>            "customDirectives": "-A iPlant-Master",
>            "default": false,
>            "maxJobs": 5,// The total number of jobs submitted through Agave which can
> //  concurrently be running in this queue. Note that value is well below the
> //  system limit and a user can fill this queue with their quota alone.
>            "maxMemoryPerNode": 24,// The maximum number of nodes a job can request when using this queue
>            "maxProcessorsPerNode": 12,// The maximum processors per node this job can specify in the request.
>            "maxNodes": 1,// The maximum number of nodes a job can request when using this queue
>            "maxProcessorsPerNode": 12,// The maximum processors per node this job can specify in the request.
>            "maxProcessorsPerNode": 12,// The maximum processors per node this job can specify in the request.
> //  When more advanced schedulers are used and jobs do not specify a node
> //  request value, this can be used to calculate the node request
>            "maxUserJobs": -1,// Maximum number of jobs a user may concurrently have active in this queue.
> //  -1 means no limit, however the maxSystemJobsPerUser value of 5 will prevent
> //  a user from having more than 5 jobs active.
>            "name": "development"
>        }
>    ],
>    …
> }
> 
> 
> 
> 
> —
> Rion
> 
> On Jun 4, 2015, at 4:33 PM, Ghiban, Cornel <ghiban at cshl.edu<mailto:ghiban at cshl.edu>> wrote:
> 
> Hi Rion,
> 
> Could you, please, explain how the user quota works? Some of the users
> have indeed submitted a lot of jobs.
> 
> Should I have a buffer queue that "releases" their jobs, lets say max 10
> jobs/hr, so they don't clog Agave?
> 
> Thanks,
> Cornel
> 
> On Thu, 2015-06-04 at 14:10 +0000, Rion Dooley wrote:
> I imagine so. i only see 5 people with pending jobs and they are both
> being throttled due to user-level quotas on the systems and queues
> they are requesting.
> 
> 
> 
> 
>