[Iplant-api-dev] multiple jobs failing

[Ghiban, Cornel]

Not sure if it matters, but the input file for this job is shared with a bunch of other people and it's not public.

$ jobs-list -v 0001433293342818-5056a550b8-0001-007


{
    "_links": {
        "app": {
            "href": "https://agave.iplantc.org/apps/v2/dnasubway-fastx-lonestar-0.0.13.2.0u1"
        },
        "archiveData": {
            "href": "https://agave.iplantc.org/files/v2/listings//smukhtar/DNASubway/project-785/fx6282"
        },
        "archiveSystem": {
            "href": "https://agave.iplantc.org/systems/v2/data.iplantcollaborative.org"
        },
        "executionSystem": {
            "href": "https://agave.iplantc.org/systems/v2/dnalc.lonestar.tacc.utexas.edu"
        },
        "history": {
            "href": "https://agave.iplantc.org/jobs/v2/0001433293342818-5056a550b8-0001-007/history"
        },
        "metadata": {
            "href": "https://agave.iplantc.org/meta/v2/data/?q={\"associationIds\":\"0001433293342818-5056a550b8-0001-007\"}"
        },
        "notifications": {
            "href": "https://agave.iplantc.org/notifications/v2/?associatedUuid=0001433293342818-5056a550b8-0001-007"
        },
        "owner": {
            "href": "https://agave.iplantc.org/profiles/v2/smukhtar"
        },
        "permissions": {
            "href": "https://agave.iplantc.org/jobs/v2/0001433293342818-5056a550b8-0001-007/pems"
        },
        "self": {
            "href": "https://agave.iplantc.org/jobs/v2/0001433293342818-5056a550b8-0001-007"
        }
    },
    "appId": "dnasubway-fastx-lonestar-0.0.13.2.0u1",
    "archive": true,
    "archivePath": "/smukhtar/DNASubway/project-785/fx6282",
    "archiveSystem": "data.iplantcollaborative.org",
    "batchQueue": "normal",
    "endTime": "2015-06-02T21:45:44.000-05:00",
    "executionSystem": "dnalc.lonestar.tacc.utexas.edu",
    "id": "0001433293342818-5056a550b8-0001-007",
    "inputs": {
        "seq1": "agave://data.iplantcollaborative.org/shared/cshl_ngs_edu/fwg_2015/smukhtar/295236_merged_R1.fastq.gz"
    },
    "localId": null,
    "maxRunTime": "03:00:00",
    "memoryPerNode": 1.0,
    "message": "Unable to stage inputs for job after 3 attempts. Job cancelled.",
    "name": "fx6282",
    "nodeCount": 1,
    "outputPath": null,
    "owner": "smukhtar",
    "parameters": {
        "jobName": "fx6282",
        "min_length": 20,
        "min_quality": 20,
        "percent_bases": 50,
        "perform_indexing": 1,
        "quality_threshold": 20
    },
    "processorsPerNode": 12,
    "retries": 2,
    "startTime": null,
    "status": "FAILED",
    "submitTime": "2015-06-02T15:02:22.000-05:00"
}


Thanks,
Cornel

On Wed, 2015-06-03 at 14:27 +0000, Matthew Vaughn wrote:


Eric and Cornel,

We’re coming back from a center wide maintenance which also included a version update to Agave. Many things could be amiss. Can you each post the output of jobs-list -v JOBID for one representative failed job to the list?

Matt

> On Jun 3, 2015, at 9:14 AM, Fritz-Waters, Eric R [AN S] <ercfrtz at iastate.edu<mailto:ercfrtz at iastate.edu>> wrote:
>
> I am having multiple jobs fail right now too. So far I have 14 out of 27
> jobs submitted that have failed with the same error that Cornel is getting
> below and the rest are in the staging inputs step and I¹m expecting them
> to fail out soon as well.
>
>
> -Eric
>
>
>
> On 6/3/15, 9:10 AM, "Ghiban, Cornel" <ghiban at cshl.edu<mailto:ghiban at cshl.edu>> wrote:
>
>> Hi,
>>
>> I have multiple jobs failing with this message:
>>
>> 'message' => 'Unable to stage inputs for job after 3 attempts. Job
>> cancelled.',
>>
>> 0001433328461090-5056a550b8-0001-007
>> 0001433293342818-5056a550b8-0001-007
>> 0001433328461755-5056a550b8-0001-007
>> etc
>>
>> The files are there, but it's not clear to me what the problem is.
>>
>> Why is this happening and how do you recommend I should handle cases
>> like these?
>>
>> Thanks,
>> Cornel
>>
>> _______________________________________________
>> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org<mailto:Iplant-api-dev at iplantcollaborative.org>
>> List Info and Archives:
>> http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
>> One-click Unsubscribe:
>> http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/ercfrtz
>> %40iastate.edu?unsub=1&unsubconfirm=1
>
>
> _______________________________________________
> Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org<mailto:Iplant-api-dev at iplantcollaborative.org>
> List Info and Archives: http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
> One-click Unsubscribe: http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/vaughn%40tacc.utexas.edu?unsub=1&unsubconfirm=1


_______________________________________________
Iplant-api-dev Mailing List: Iplant-api-dev at iplantcollaborative.org<mailto:Iplant-api-dev at iplantcollaborative.org>
List Info and Archives: http://mail.iplantcollaborative.org/mailman/listinfo/iplant-api-dev
One-click Unsubscribe: http://mail.iplantcollaborative.org/mailman/options/iplant-api-dev/ghiban%40cshl.edu?unsub=1&unsubconfirm=1