Apps endpoints

Apps endpoints provide detailed information about your applications, similar to the Applications page in Unravel UI. You can collect data based on the application name, type, user, queues, and tags.

/apps/{app_id}/resource_usage

Gets an array of resource usage metrics for a given application.

Request

GET http://unravel-host:3000/api/v1/apps/{app_id}/resource_usage?metric_id={metric_id}

Path parameters

Name	Type	Description
`app_id`	string	App ID

Query parameters

Required parameters are highlighted.

Name

Type

Description

metric_id

string

Metric ID

Valid values are:

115: gcLoad

125: maxHeap

126: usedHeap

134: processCpuLoad

135: systemCpuLoad

137: availableMemory

138: vmRss

Response body

Name	Type	Description
`appId`	string	App ID
`unravel_metric_id`	integer	Metric ID
`system`	string
`entity_id`	string
`entity_name`	string
`host_name`	string
`sampleCount`	integer
`sum`	integer
`min`	integer
`max`	integer
`avg`	integer
`ts`	integer
`endTs`	integer
`unit`	string

Examples

Request:

curl -X GET "http://myserver.com:3000/api/v1/apps/20190626T032742Z--6535643260096439568/resource_usage?metric_id=138" -H "accept: application/json" -H  "Authorization: JWT token"

Response body:

[
   {
     "appId":"20190626T032742Z--6535643260096439568",
     "unravel_metric_id":138,
     "system":"mr",
     "entity_id":"a_1",
     "entity_name":"a_1",
     "host_name":"myserver.com",
     "sampleCount":2,
     "sum":772329472,
     "min":384274432,
     "max":388055040,
     "avg":386164736,
     "ts":1553500850000,
     "endTs":1553500860000,
     "unit":"BYTES"
   }
]

/apps/{app_id}/tags

Gets tags for a given app.

Request

GET http://unravel-host:3000/api/v1/apps/{app_id}/tags

Path parameters

Name	Type	Description
`app_id`	string	App ID

Query parameters

None.

Response body

Name	Type	Description
`realUser`	string	Username
`project`	string	Project name
`dept`	string	Department name

Examples

Request:

curl -X GET "http://myserver.com:3000/api/v1/apps/hive_20190531170145_12ccd59c-cecf-457b-b440-86b5f187097c/tags" -H "accept: application/json" -H  "Authorization: JWT token"

/apps/events/inefficient_apps

Gets total job count per status in a given time window (at one hour intervals).

Request

GET http://unravel-host:3000/api/v1/apps/events/inefficient_apps?start_time={timestamp}&end_time={timestamp}&entity_type={application type}

Path parameters

None.

Query parameters

Required parameters are highlighted.

Name

Type

Description

start_time

string

Start date. Format YYYY-MM-DD

end_time

string

End date. Format YYYY-MM-DD

entity_type

integer

Application type.

Valid values are:

0: MR

1: Hive

2 : Spark

16 : Impala

Response body

Name	Type	Description
`entity_type`	integer	Entity type
`event_name`	string	Event name
`count`	integer

Examples

Request:

curl -X GET "http://http://myserver.com:3000/api/v1/apps/events/inefficient_apps/?start_time=2019-03-01&end_time=2019-03-06&amp;entity_type=2" -H "accept: application/json"-H  "accept: application/json" -H  "Authorization: JWT token"

Response body:

[
  {
    "entity_type": 0,
    "event_name": "MRTooLargeMapEvent",
    "count": 10559
  },
  {
    "entity_type": 0,
    "event_name": "MRTimeBreakdownEvent",
    "count": 7868
  },
  {
    "entity_type": 0,
    "event_name": "MRTooLargeReduceEvent",
    "count": 4782
  },
  {
    "entity_type": 0,
    "event_name": "MRTooManyMapEvent2",
    "count": 2097
  },
  {
   "entity_type": 0,
    "event_name": "MRTooManyReduceEvent2",
    "count": 1066
  },
  {
    "entity_type": 0,
    "event_name": "LongReduceTasksStartBeforeMapFinishEvent",
    "count": 21
  },
  {
    "entity_type": 0,
    "event_name": "MRTooFewMapEvent2",
    "count": 3
  },
  {
    "entity_type": 0,
    "event_name": "MRReduceTimeSkewEvent",
    "count": 2
  },
  {
    "entity_type": 0,
    "event_name": "MRMapTimeSkewEvent",
    "count": 1
  }
]

/apps/resources/cpu/allocated

Gets a time series for CPU allocations by app type.

Request

GET http://unravel-host:3000/api/v1/apps/resources/cpu/allocated

Path parameters

None.

Query parameters

Required parameters are highlighted.

Name	Type	Description
`from`	string	Start date. Format `YYYY-MM-DD`
`to`	string	End date. Format `YYYY-MM-DD`

Response body

Name	Type	Description
`date`	integer	Timestamp in Unix epoch format
`SUCCEEDED`	integer

Examples

Request:

curl -X GET "http://playground3.unraveldata.com/api/v1/apps/events/inefficient_apps/?start_time=2019-03-01&end_time=2019-03-06&entity_type=2" -H  "accept: application/json" -H  "Authorization: JWT token"

Response body:

[
  {
    "date": 1551938400000,
    "SUCCEEDED": 86
  },
  {
    "date": 1551942000000,
    "SUCCEEDED": 20
  },
  {
    "date": 1551945600000,
    "SUCCEEDED": 34
  }
]

/apps/resources/memory/allocated

Gets a time series for allocated memory by app type.

Request

GET http://unravel-host:3000/api/v1/apps/resources/memory/allocated

Path parameters

None.

Query parameters

Required parameters are highlighted.

Name	Type	Description
`from`	string	Start date. Format `YYYY-MM-DD`
`to`	string	End date. Format `YYYY-MM-DD`

Response body

Name	Type	Description
`timestamp`	string

Examples

Request:

curl -X GET "http://myserver.com:3000/api/v1/apps/resources/memory/allocated?from=2019-03-01&to=2019-03-06" -H  "accept: application/json" -H  "Authorization: JWT token"

Response body:

{
   "1551787200000":"2"
}

/apps/search

Gets all apps filtered by app type, status, username, queue, and tags.

Request

POST http://unravel-host:3000/api/v1/apps/search -d '{"from":0,"appTypes":[{app_types}],"appStatus":[{app_status},"end_time":"{timestamp}","start_time":"{timestamp}","users":["{username}"],"queues":["{queuename} "],"appTags":{"taglist"}}'

Path parameters

None.

Query parameters

Required parameters are highlighted.

Name	Type	Description
`from`	integer	Set this to `0`.
`appTypes`	string	List of app types. Format: `["type1","type2"]` Valid values are `mr` \| `hive` \| `spark` \| `pig` \| `cascading` \| `impala` \| `tez`.
`appStatus`	string	App status. Format: `["status1","status2"]` Valid values are `S`uccess \| `F`ailed \| `K`illed \| `R`unning \| `W` \| `P`ending \| `U`nkown. Note For Hive and MR you must specify at least one status type.
`start_time`	string	Start time. Format: `YYYY-MM-DDTHH:MM:SS.NNNZ`
`end_time`	string	End time. Format: `YYYY-MM-DDTHH:MM:SS.NNNZ`
`users`	string	Usernames(s). Format: `["user1","user2"]`
`queues`	string	Cluster's queue names(s). Format: `["queue1","queue2"]`
`taglist`	string	Tag(s). Format: `"key":["value,value"]`. For example, `"dept":[finance, mktg]`

Response body

The JSON response body contains a metadata section and a results section that vary according to the request.

Examples

List all apps:

curl -X POST "http://myserver.com:3000/api/v1/apps/search" -H  "Content-Type: application/json" -H  "Authorization: JWT token"

List all failed, killed, or unknown Hive and Tez apps within a specific timeframe:

curl -X POST "http://myserver.com:3000/api/v1/apps/search" -H  "Content-Type: application/json" -H  "Authorization: JWT token" -d '{"from":0,"appTypes":["hive","tez"],"appStatus":["F","K","U"],"end_time":"2018-10-09T05:18:42.000Z","start_time":"2018-10-09T02:18:42.000Z"}'

List all apps owned by a user regardless of status, queue, or tags.

curl -X POST "http://myserver.com:3000/api/v1/apps/search" -H  "Content-Type: application/json" -H  "Authorization: JWT token" -d '{"from":0,"appTypes":["mr","hive","spark","pig","cascading","impala","tez"],"appStatus":["S","F","K","R","W","P","U"],"end_time":"20190626T042742.000Z","start_time":"20190626T032742.000Z","users":["root"]}'

Response body:

{
  "metadata": {
    "duration": {
      "max": 163367,
      "min": 4135
    },
    "resource": {
      "max": null,
      "min": null
    },
    "events": {
      "max": 3,
      "min": 0
    },
    "appTypes": {
      "spark": 28
    },
    "appStatus": {
      "S": 23,
      "F": 5
    },
    "users": {
      "unravel": 321
    },
    "queues": {
      "root.users.unravel": 113,
      "default": 94
    },
    "clusters": {
      "QACDH601A": 29
    },
    "totalRecords": 28
  },
  "results": [
    {
      "appId": "20190626T032742Z--6535643260096439568",
      "appType": "wfi",
      "appt": "wfi",
      "gotoId": "20190626T032742Z--6535643260096439568",
      "gotoLevel": "WFI",
      "id": "application_1558643494852_0916",
      "nick": "spark",
      "name": "multiple_jobs_stages.py",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 10:28:18",
      "start_time_long": "2019-06-26T10:28:18.403Z",
      "duration_long": 21658,
      "predDuration_long": 0,
      "io_long": 0,
      "read_long": 0,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 3,
      "numApps": 0,
      "numSparkApps": 25,
      "numMRJobs": 25,
      "numEvents": 3,
      "mrJobIds": [
        "stage-0",
        "stage-1",
        "stage-2",
        "stage-3",
        "stage-4",
        "stage-9",
        "stage-10",
        "stage-11",
        "stage-12",
        "stage-13",
        "stage-22",
        "stage-23",
        "stage-24",
        "stage-25",
        "stage-26",
        "stage-39",
        "stage-40",
        "stage-41",
        "stage-42",
        "stage-43",
        "stage-60",
        "stage-61",
        "stage-62",
        "stage-63",
        "stage-64"
      ],
      "appIds": [],
      "sm": 50,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": 50,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": 50,
      "totalReduceTasks": null,
      "totalSparkTasks": 50,
      "totalMapSlotDuration": 7650,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": 7650,
      "inputTables": null,
      "outputTables": [],
      "wi": "20190626T032742Z--6535643260096439568",
      "wn": "workflow_spark_test_20190626T032742Z",
      "wt": "1561519662000",
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 65779,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "multiple_jobs_stages.py",
      "goToApp": {
        "appId": "20190626T032742Z--6535643260096439568",
        "appType": "WFI"
      },
      "kind_url": "spark",
      "kind_parent_url": "workflow"
    },
    {
      "appId": null,
      "appType": null,
      "gotoId": null,
      "gotoLevel": null,
      "id": "application_1558643494852_0915",
      "nick": "spark",
      "name": "basic.py",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 10:25:43",
      "start_time_long": "2019-06-26T10:25:43.626Z",
      "duration_long": 31493,
      "predDuration_long": 0,
      "io_long": 785,
      "read_long": 785,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 3,
      "numApps": 0,
      "numSparkApps": 18,
      "numMRJobs": 18,
      "numEvents": 3,
      "mrJobIds": [
        "stage-0",
        "stage-1",
        "stage-2",
        "stage-3",
        "stage-4",
        "stage-5",
        "stage-6",
        "stage-8",
        "stage-10",
        "stage-12",
        "stage-14",
        "stage-15",
        "stage-16",
        "stage-17",
        "stage-18",
        "stage-19",
        "stage-20",
        "stage-21"
      ],
      "appIds": [],
      "sm": 214,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": 214,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": 214,
      "totalReduceTasks": null,
      "totalSparkTasks": 214,
      "totalMapSlotDuration": 10030,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": 10030,
      "inputTables": [
        "default.people"
      ],
      "outputTables": [],
      "wi": null,
      "wn": null,
      "wt": null,
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 86976,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "basic.py",
      "kind_url": "spark",
      "kind_parent_url": "app"
    },
    {
      "appId": null,
      "appType": null,
      "gotoId": null,
      "gotoLevel": null,
      "id": "application_1558643494852_0914",
      "nick": "spark",
      "name": "Spark Test App",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 10:23:45",
      "start_time_long": "2019-06-26T10:23:45.120Z",
      "duration_long": 18955,
      "predDuration_long": 0,
      "io_long": 0,
      "read_long": 0,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 2,
      "numApps": 0,
      "numSparkApps": 25,
      "numMRJobs": 25,
      "numEvents": 2,
      "mrJobIds": [
        "stage-0",
        "stage-1",
        "stage-2",
        "stage-3",
        "stage-4",
        "stage-9",
        "stage-10",
        "stage-11",
        "stage-12",
        "stage-13",
        "stage-22",
        "stage-23",
        "stage-24",
        "stage-25",
        "stage-26",
        "stage-39",
        "stage-40",
        "stage-41",
        "stage-42",
        "stage-43",
        "stage-60",
        "stage-61",
        "stage-62",
        "stage-63",
        "stage-64"
      ],
      "appIds": [],
      "sm": 50,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": 50,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": 50,
      "totalReduceTasks": null,
      "totalSparkTasks": 50,
      "totalMapSlotDuration": 7651,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": 7651,
      "inputTables": null,
      "outputTables": [],
      "wi": null,
      "wn": null,
      "wt": null,
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 49704,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "Spark Test App",
      "kind_url": "spark",
      "kind_parent_url": "app"
    },
    {
      "appId": null,
      "appType": null,
      "gotoId": null,
      "gotoLevel": null,
      "id": "application_1558643494852_0913",
      "nick": "spark",
      "name": "multiple_jobs_stages.py",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 10:21:53",
      "start_time_long": "2019-06-26T10:21:53.088Z",
      "duration_long": 21233,
      "predDuration_long": 0,
      "io_long": 0,
      "read_long": 0,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 2,
      "numApps": 0,
      "numSparkApps": 0,
      "numMRJobs": 0,
      "numEvents": 2,
      "mrJobIds": [],
      "appIds": [],
      "sm": null,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": null,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": null,
      "totalReduceTasks": null,
      "totalSparkTasks": null,
      "totalMapSlotDuration": null,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": null,
      "inputTables": null,
      "outputTables": [],
      "wi": null,
      "wn": null,
      "wt": null,
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 63554,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "multiple_jobs_stages.py",
      "kind_url": "spark",
      "kind_parent_url": "app"
    },
    {
      "appId": null,
      "appType": null,
      "gotoId": null,
      "gotoLevel": null,
      "id": "application_1558643494852_0912",
      "nick": "spark",
      "name": "multiple_jobs_stages.py",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 10:19:22",
      "start_time_long": "2019-06-26T10:19:22.725Z",
      "duration_long": 21803,
      "predDuration_long": 0,
      "io_long": 0,
      "read_long": 0,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 2,
      "numApps": 0,
      "numSparkApps": 0,
      "numMRJobs": 0,
      "numEvents": 2,
      "mrJobIds": [],
      "appIds": [],
      "sm": null,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": null,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": null,
      "totalReduceTasks": null,
      "totalSparkTasks": null,
      "totalMapSlotDuration": null,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": null,
      "inputTables": null,
      "outputTables": [],
      "wi": null,
      "wn": null,
      "wt": null,
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 63723,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "multiple_jobs_stages.py",
      "kind_url": "spark",
      "kind_parent_url": "app"
    },
    {
      "appId": null,
      "appType": null,
      "gotoId": null,
      "gotoLevel": null,
      "id": "application_1558643494852_0911",
      "nick": "spark",
      "name": "multiple_jobs_stages.py",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 10:16:03",
      "start_time_long": "2019-06-26T10:16:03.395Z",
      "duration_long": 16942,
      "predDuration_long": 0,
      "io_long": 0,
      "read_long": 0,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 1,
      "numApps": 0,
      "numSparkApps": 10,
      "numMRJobs": 10,
      "numEvents": 1,
      "mrJobIds": [
        "stage-0",
        "stage-1",
        "stage-3",
        "stage-4",
        "stage-7",
        "stage-8",
        "stage-12",
        "stage-13",
        "stage-18",
        "stage-19"
      ],
      "appIds": [],
      "sm": 20,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": 20,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": 20,
      "totalReduceTasks": null,
      "totalSparkTasks": 20,
      "totalMapSlotDuration": 4628,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": 4628,
      "inputTables": null,
      "outputTables": [],
      "wi": null,
      "wn": null,
      "wt": null,
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 44537,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "multiple_jobs_stages.py",
      "kind_url": "spark",
      "kind_parent_url": "app"
    },
    {
      "appId": null,
      "appType": null,
      "gotoId": null,
      "gotoLevel": null,
      "id": "application_1558643494852_0910",
      "nick": "spark",
      "name": "multiple_jobs_stages.py",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 10:12:43",
      "start_time_long": "2019-06-26T10:12:43.909Z",
      "duration_long": 17429,
      "predDuration_long": 0,
      "io_long": 0,
      "read_long": 0,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 1,
      "numApps": 0,
      "numSparkApps": 10,
      "numMRJobs": 10,
      "numEvents": 1,
      "mrJobIds": [
        "stage-0",
        "stage-1",
        "stage-3",
        "stage-4",
        "stage-7",
        "stage-8",
        "stage-12",
        "stage-13",
        "stage-18",
        "stage-19"
      ],
      "appIds": [],
      "sm": 20,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": 20,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": 20,
      "totalReduceTasks": null,
      "totalSparkTasks": 20,
      "totalMapSlotDuration": 4770,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": 4770,
      "inputTables": null,
      "outputTables": [],
      "wi": null,
      "wn": null,
      "wt": null,
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 45714,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "multiple_jobs_stages.py",
      "kind_url": "spark",
      "kind_parent_url": "app"
    },
    {
      "appId": null,
      "appType": null,
      "gotoId": null,
      "gotoLevel": null,
      "id": "application_1558643494852_0909",
      "nick": "spark",
      "name": "multiple_jobs_stages.py",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 10:10:24",
      "start_time_long": "2019-06-26T10:10:24.015Z",
      "duration_long": 17154,
      "predDuration_long": 0,
      "io_long": 0,
      "read_long": 0,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 1,
      "numApps": 0,
      "numSparkApps": 10,
      "numMRJobs": 10,
      "numEvents": 1,
      "mrJobIds": [
        "stage-0",
        "stage-1",
        "stage-3",
        "stage-4",
        "stage-7",
        "stage-8",
        "stage-12",
        "stage-13",
        "stage-18",
        "stage-19"
      ],
      "appIds": [],
      "sm": 20,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": 20,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": 20,
      "totalReduceTasks": null,
      "totalSparkTasks": 20,
      "totalMapSlotDuration": 4631,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": 4631,
      "inputTables": null,
      "outputTables": [],
      "wi": null,
      "wn": null,
      "wt": null,
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 44999,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "multiple_jobs_stages.py",
      "kind_url": "spark",
      "kind_parent_url": "app"
    },
    {
      "appId": null,
      "appType": null,
      "gotoId": null,
      "gotoLevel": null,
      "id": "application_1558643494852_0908",
      "nick": "spark",
      "name": "multiple_jobs_stages.py",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 10:08:03",
      "start_time_long": "2019-06-26T10:08:03.004Z",
      "duration_long": 19386,
      "predDuration_long": 0,
      "io_long": 0,
      "read_long": 0,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 3,
      "numApps": 0,
      "numSparkApps": 10,
      "numMRJobs": 10,
      "numEvents": 3,
      "mrJobIds": [
        "stage-0",
        "stage-1",
        "stage-3",
        "stage-4",
        "stage-7",
        "stage-8",
        "stage-12",
        "stage-13",
        "stage-18",
        "stage-19"
      ],
      "appIds": [],
      "sm": 20,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": 20,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": 20,
      "totalReduceTasks": null,
      "totalSparkTasks": 20,
      "totalMapSlotDuration": 5345,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": 5345,
      "inputTables": null,
      "outputTables": [],
      "wi": null,
      "wn": null,
      "wt": null,
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 55511,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "multiple_jobs_stages.py",
      "kind_url": "spark",
      "kind_parent_url": "app"
    },
    {
      "appId": null,
      "appType": null,
      "gotoId": null,
      "gotoLevel": null,
      "id": "application_1558643494852_0907",
      "nick": "spark",
      "name": "multiple_jobs_stages.py",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 10:05:41",
      "start_time_long": "2019-06-26T10:05:41.564Z",
      "duration_long": 19017,
      "predDuration_long": 0,
      "io_long": 0,
      "read_long": 0,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 3,
      "numApps": 0,
      "numSparkApps": 10,
      "numMRJobs": 10,
      "numEvents": 3,
      "mrJobIds": [
        "stage-0",
        "stage-1",
        "stage-3",
        "stage-4",
        "stage-7",
        "stage-8",
        "stage-12",
        "stage-13",
        "stage-18",
        "stage-19"
      ],
      "appIds": [],
      "sm": 20,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": 20,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": 20,
      "totalReduceTasks": null,
      "totalSparkTasks": 20,
      "totalMapSlotDuration": 5137,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": 5137,
      "inputTables": null,
      "outputTables": [],
      "wi": null,
      "wn": null,
      "wt": null,
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 53504,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "multiple_jobs_stages.py",
      "kind_url": "spark",
      "kind_parent_url": "app"
    },
    {
      "appId": null,
      "appType": null,
      "gotoId": null,
      "gotoLevel": null,
      "id": "application_1558643494852_0906",
      "nick": "spark",
      "name": "multiple_jobs_stages.py",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 10:03:20",
      "start_time_long": "2019-06-26T10:03:20.433Z",
      "duration_long": 20112,
      "predDuration_long": 0,
      "io_long": 0,
      "read_long": 0,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 1,
      "numApps": 0,
      "numSparkApps": 25,
      "numMRJobs": 25,
      "numEvents": 1,
      "mrJobIds": [
        "stage-0",
        "stage-1",
        "stage-2",
        "stage-3",
        "stage-4",
        "stage-9",
        "stage-10",
        "stage-11",
        "stage-12",
        "stage-13",
        "stage-22",
        "stage-23",
        "stage-24",
        "stage-25",
        "stage-26",
        "stage-39",
        "stage-40",
        "stage-41",
        "stage-42",
        "stage-43",
        "stage-60",
        "stage-61",
        "stage-62",
        "stage-63",
        "stage-64"
      ],
      "appIds": [],
      "sm": 50,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": 50,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": 50,
      "totalReduceTasks": null,
      "totalSparkTasks": 50,
      "totalMapSlotDuration": 6988,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": 6988,
      "inputTables": null,
      "outputTables": [],
      "wi": null,
      "wn": null,
      "wt": null,
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 58648,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "multiple_jobs_stages.py",
      "kind_url": "spark",
      "kind_parent_url": "app"
    },
    {
      "appId": null,
      "appType": null,
      "gotoId": null,
      "gotoLevel": null,
      "id": "application_1558643494852_0905",
      "nick": "spark",
      "name": "multiple_jobs_stages.py",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 10:00:55",
      "start_time_long": "2019-06-26T10:00:55.666Z",
      "duration_long": 20682,
      "predDuration_long": 0,
      "io_long": 0,
      "read_long": 0,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 3,
      "numApps": 0,
      "numSparkApps": 25,
      "numMRJobs": 25,
      "numEvents": 3,
      "mrJobIds": [
        "stage-0",
        "stage-1",
        "stage-2",
        "stage-3",
        "stage-4",
        "stage-9",
        "stage-10",
        "stage-11",
        "stage-12",
        "stage-13",
        "stage-22",
        "stage-23",
        "stage-24",
        "stage-25",
        "stage-26",
        "stage-39",
        "stage-40",
        "stage-41",
        "stage-42",
        "stage-43",
        "stage-60",
        "stage-61",
        "stage-62",
        "stage-63",
        "stage-64"
      ],
      "appIds": [],
      "sm": 50,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": 50,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": 50,
      "totalReduceTasks": null,
      "totalSparkTasks": 50,
      "totalMapSlotDuration": 7419,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": 7419,
      "inputTables": null,
      "outputTables": [],
      "wi": null,
      "wn": null,
      "wt": null,
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 62845,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "multiple_jobs_stages.py",
      "kind_url": "spark",
      "kind_parent_url": "app"
    },
    {
      "appId": null,
      "appType": null,
      "gotoId": null,
      "gotoLevel": null,
      "id": "application_1558643494852_0904",
      "nick": "spark",
      "name": "multiple_jobs_stages.py",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 09:58:33",
      "start_time_long": "2019-06-26T09:58:33.028Z",
      "duration_long": 20967,
      "predDuration_long": 0,
      "io_long": 0,
      "read_long": 0,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 3,
      "numApps": 0,
      "numSparkApps": 25,
      "numMRJobs": 25,
      "numEvents": 3,
      "mrJobIds": [
        "stage-0",
        "stage-1",
        "stage-2",
        "stage-3",
        "stage-4",
        "stage-9",
        "stage-10",
        "stage-11",
        "stage-12",
        "stage-13",
        "stage-22",
        "stage-23",
        "stage-24",
        "stage-25",
        "stage-26",
        "stage-39",
        "stage-40",
        "stage-41",
        "stage-42",
        "stage-43",
        "stage-60",
        "stage-61",
        "stage-62",
        "stage-63",
        "stage-64"
      ],
      "appIds": [],
      "sm": 50,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": 50,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": 50,
      "totalReduceTasks": null,
      "totalSparkTasks": 50,
      "totalMapSlotDuration": 7688,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": 7688,
      "inputTables": null,
      "outputTables": [],
      "wi": null,
      "wn": null,
      "wt": null,
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 63076,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "multiple_jobs_stages.py",
      "kind_url": "spark",
      "kind_parent_url": "app"
    },
    {
      "appId": null,
      "appType": null,
      "gotoId": null,
      "gotoLevel": null,
      "id": "application_1558643494852_0903",
      "nick": "spark",
      "name": "multiple_jobs_stages.py",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 09:56:10",
      "start_time_long": "2019-06-26T09:56:10.706Z",
      "duration_long": 20906,
      "predDuration_long": 0,
      "io_long": 0,
      "read_long": 0,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 2,
      "numApps": 0,
      "numSparkApps": 0,
      "numMRJobs": 0,
      "numEvents": 2,
      "mrJobIds": [],
      "appIds": [],
      "sm": null,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": null,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": null,
      "totalReduceTasks": null,
      "totalSparkTasks": null,
      "totalMapSlotDuration": null,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": null,
      "inputTables": null,
      "outputTables": [],
      "wi": null,
      "wn": null,
      "wt": null,
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 63967,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "multiple_jobs_stages.py",
      "kind_url": "spark",
      "kind_parent_url": "app"
    },
    {
      "appId": null,
      "appType": null,
      "gotoId": null,
      "gotoLevel": null,
      "id": "application_1558643494852_0902",
      "nick": "spark",
      "name": "multiple_jobs_stages.py",
      "queue": "root.users.unravel",
      "status": "S",
      "status_long": "Success",
      "user": "unravel",
      "raw_user": "unravel",
      "submitHost": "-",
      "remarks": [
        "-"
      ],
      "aa2Badge": false,
      "inBadge": false,
      "clusterId": "QACDH601A",
      "clusterTag": "1558643494852",
      "start_time": "06/26/19 09:53:46",
      "start_time_long": "2019-06-26T09:53:46.591Z",
      "duration_long": 19830,
      "predDuration_long": 0,
      "io_long": 0,
      "read_long": 0,
      "write_long": 0,
      "resource": 0,
      "service": 0,
      "events": 0,
      "numApps": 0,
      "numSparkApps": 0,
      "numMRJobs": 0,
      "numEvents": 0,
      "mrJobIds": [],
      "appIds": [],
      "sm": null,
      "sr": 100,
      "fm": null,
      "fr": null,
      "km": null,
      "kr": null,
      "ss": null,
      "fs": null,
      "ks": null,
      "fsu": null,
      "ksu": null,
      "totalMapTasks": null,
      "totalReduceTasks": null,
      "totalSparkTasks": null,
      "totalMapSlotDuration": null,
      "totalReduceSlotDuration": null,
      "totalSparkSlotDuration": null,
      "inputTables": null,
      "outputTables": [],
      "wi": null,
      "wn": null,
      "wt": null,
      "type": "REGULAR",
      "type_regular": true,
      "jobId": "-",
      "runName": "-",
      "runId": "-",
      "db": "-",
      "output": "-",
      "aid": "-",
      "userType": "-",
      "cents": 0,
      "metrics": null,
      "totalProcessingTime": 0,
      "memorySeconds": 56824,
      "cpuTime": 0,
      "storageWaitTime": 0,
      "networkSendWaitTime": 0,
      "networkReceiveWaitTime": 0,
      "elastic": true,
      "kind": "spark",
      "kindLong": "Spark",
      "name_long": "multiple_jobs_stages.py",
      "kind_url": "spark",
      "kind_parent_url": "app"
    }
  ]
}

/apps/status/finished

Gets a time series of finished apps.

Request

GET http://unravel-host:3000/api/v1/apps/status/finished

Path parameters

None.

Query parameters

None.

Response body

Name	Type	Description
`date`	string	Time stamp.
`SUCCEEDED`	integer	Number of apps with sucessful run.
`FAILED`	integer	Number of failed apps.
`KILLED`	integer	Number of killed apps.

Examples

Request:

curl -X GET "http://http://myserver.com:3000/api/v1/apps/status/finished" -H  "accept: application/json" -H  "Authorization: JWT token"

Response:

[
  {
    "date": 1561482000000,
    "SUCCEEDED": 2
  },
  {
    "date": 1561485600000,
    "SUCCEEDED": 2
  },
  {
    "date": 1561489200000
  },
  {
    "date": 1561492800000
  },
  {
    "date": 1561496400000
  },
  {
    "date": 1561500000000
  },
  {
    "date": 1561503600000
  },
  {
    "date": 1561507200000
  },
  {
    "date": 1561510800000
  },
  {
    "date": 1561514400000
  },
  {
    "date": 1561518000000
  },
  {
    "date": 1561521600000
  },
  {
    "date": 1561525200000
  },
  {
    "date": 1561528800000
  },
  {
    "date": 1561532400000
  },
  {
    "date": 1561536000000
  },
  {
    "date": 1561539600000,
    "SUCCEEDED": 11,
    "FAILED": 5
  },
  {
    "date": 1561543200000,
    "SUCCEEDED": 12
  }
]

/apps/status/running

Gets a list of running apps.

Request

GET http://unravel-host:3000/api/v1/apps/status/running

Path parameters

None.

Query parameters

None.

Response body

Name	Type	Description
`date`	string	Timestamp in Unix epoch format
`appsRunning`	integer	Number of apps running
`appsPending`	integer	Number of apps pending

Examples

Request:

curl -X GET "http://http://myserver.com:3000/api/v1/apps/status/running" -H  "accept: application/json" -H  "Authorization: JWT token"

Response body:

{
  "date": 1561619001184,
  "appsRunning": 0,
  "appsPending": 0
}

In this section:

Home

Apps endpoints

/apps/{app_id}/resource_usage

/apps/{app_id}/tags

/apps/events/inefficient_apps

/apps/resources/cpu/allocated

/apps/resources/memory/allocated

/apps/search

Note

/apps/status/finished

/apps/status/running

Search results