Reports
These endpoints return key performance indicators (KPIs) for clusters, the small files report, chargeback reports (by user, queue, app), and cluster workload reports.
Cluster Reports
Gets a summary of the cluster's users, apps, or queues.
GET http://unravel-host:3000/api/v1/reports/operational/clusterstats?st={timestamp}&et={timestamp}&mode={summary_type}None.
Name | Type | Description |
|---|---|---|
| string | Start time, in Unix epoch format. Required. |
| string | End time, in Unix epoch format. Required. |
| string | Summary type. Required. Valid values are |
JSON schema when mode is user or queue:
{
"userStats": [
// array of users | queues currently on cluster
{
"root": {
"running": {
"min": minimum applications running,,
"max": minimum applications running,
"mean": average appication number running,
"stddev": standard deviation
},
"memory": { // see running above },
"pending": { // see running above },
"vcores": { // see running above }
}
}
]
}Request with mode=user:
curl -X GET "http://myserver.com:3000/api/v1/reports/operational/clusterstats?st=1536658189000&et=1536744589000&mode=user" -H "accept: application/json" -H "Authorization: JWT token"Response for mode=user:
{
"userStats": [
{
"root": {
"running": {
"min": 1,
"max": 1,
"mean": 1,
"stddev": 0
},
"memory": {
"min": 1024,
"max": 16384,
"mean": 7040,
"stddev": 7099.0974074174
},
"pending": {
"min": 0,
"max": 0,
"mean": 0,
"stddev": 0
},
"vcores": {
"min": 1,
"max": 3,
"mean": 2,
"stddev": 0.81649658092773
}
}
}
]
}Request with mode=queue:
curl -X GET "http://myserver.com:3000/api/v1/reports/operational/clusterstats?st=1536658189000&et=1536744589000&mode=queue" -H "accept: application/json" -H "Authorization: JWT token"Response for mode=queue:
{
"queueStats": [
{
"root.users.root": {
"running": {
"min": 1,
"max": 1,
"mean": 1,
"stddev": 0
},
"memory": {
"min": 1024,
"max": 16384,
"mean": 7040,
"stddev": 7099.0974074174
},
"pending": {
"min": 0,
"max": 0,
"mean": 0,
"stddev": 0
},
"vcores": {
"min": 1,
"max": 3,
"mean": 2,
"stddev": 0.81649658092773
}
}
}
]
}Response for mode=app:
{
[
appID : application id,
type : memorySeconds/vcoreSeconds
vcore : vcore value
memory : memory value
]
}Gets a summary of the cluster's workloads.
GET http://unravel-host:3000/api/v1/reports/operational/clusterworkload?gte={timestamp}<e={timestamp}&reportBy={interval}None.
Name | Type | Description |
|---|---|---|
| string | Start time, in Unix epoch format. Required. |
| string | End time, in Unix epoch format. Required. |
| string | Report interval. Required. Valid values are |
// work load by month one of more months with the application count for month
// minimum of one month
{ timestamp: appcount[,timestamp: appcount] }
// work load by hour array of 25 hours
[
{ timestamp: appcount }, ... { timestamp: appcount }
]
// work load by day - array for each days contained with the time period (Mon - Sun)
// minimum of one day
[
{ timestamp: appcount }
]
// work load by hour/day:array for each day (Mon - Sun) by hour
// minimum of 24 hours for one day
[
{ timestamp: appcount }
]
Request:
curl -X GET "http://myserver.com:3000/api/v1/reports/operational/clusterworkload?gte=1536777000Z<e=1536863400Z&reportBy=month" -H "accept: application/json" -H "Authorization: JWT token"Response:
{"1536777000":96,"1536863400":4}
[
{"1536813000000":0},{"1536816600000":0},{"1536820200000":0},{"1536823800000":10},{"1536827400000":7},{"1536831000000":13},{"1536834600000":10},{"1536838200000":31},{"1536841800000":0},{"1536845400000":0},{"1536849000000":4},{"1536852600000":2},{"1536856200000":0},{"1536859800000":0},{"1536863400000":0},{"1536867000000":0},{"1536870600000":0},{"1536874200000":0},{"1536877800000":0},{"1536881400000":0},{"1536885000000":0},{"1536888600000":0},{"1536892200000":0},{"1536895800000":6},{"1536899400000":2}]
[
{"1536777000000":96},{"1536863400000":7}
]
[{"1536813000000":0},{"1536816600000":0},{"1536820200000":0},{"1536823800000":10},{"1536827400000":7},{"1536831000000":13},{"1536834600000":10},{"1536838200000":31},{"1536841800000":0},{"1536845400000":0},{"1536849000000":4},{"1536852600000":2},{"1536856200000":0},{"1536859800000":0},{"1536863400000":0},{"1536867000000":0},{"1536870600000":0},{"1536874200000":0},{"1536877800000":0},{"1536881400000":0},{"1536885000000":0},{"1536888600000":0},{"1536892200000":0},{"1536895800000":6},{"1536899400000":2}]
Chargeback Reports
Gets chargeback reports by application type. Gives the count of all applications in all queues for all users across all clusters.
GET http://unravel-host:3000/api/v1/search/cb/appt?from={date}&to={date}None.
Name | Type | Description |
|---|---|---|
| string | Start date. Format: |
| string | End date. Format: |
{
"cb": [
// array organizing by application type
{
"ms": memory usage seconds,
"count": application count,
"v1": "application type (mr | spark)",
"vs": vcore usage in second
}
]
}Request:
curl -X GET "http://myserver.com:3000/api/v1/search/cb/appt?from=1536670860&to=1536757260" -H "accept: application/json" -H "Authorization: JWT token"Response:
{
"cb": [
{
"ms": 2324825154,
"count": 4332,
"v1": "mr",
"vs": 1512734
},
{
"ms": 298989641,
"count": 75,
"v1": "spark",
"vs": 120404
}
]
}Gets chargeback reports by user.
GET http://unravel-host:3000/api/v1/search/cb/appt/user?from={date}&to={date}None.
Name | Type | Description |
|---|---|---|
| string | Start date. Format: |
| string | End date. Format: |
{
"cb": [
// array by application type
{
"count": application count,
"v1": application type (mr, | spark), ,
"cb": [
//array of users or queues (depending on command)
{
"ms": memory usage in milliseconds,,
"count": application count,,
"v2": userName | queueName,,
"vs": vcore usage in seconds
}
]
}
]
Request:
curl -X GET "http://myserver.com:3000/api/v1/search/cb/appt/user?from=1536676860&to=1536763260" -H "accept: application/json" -H "Authorization: JWT token"Response:
{
"cb": [
{
"count": 27,
"v1": "mr",
"cb": [
{
"ms": 1974235,
"count": 27,
"v2": "root.users.root",
"vs": 798
}
]
}
]
}Gets chargeback reports by queue.
GET http://unravel-host:3000/api/v1/search/cb/appt/queue?from={date}&to={date}None.
Name | Type | Description |
|---|---|---|
| string | Start date. Format: |
| string | End date. Format: |
{
"cb": [
// array by application type
{
"count": application count,
"v1": application type (mr, | spark), ,
"cb": [
//array of users or queues (depending on command)
{
"ms": memory usage in milliseconds,,
"count": application count,,
"v2": userName | queueName,,
"vs": vcore usage in seconds
}
]
}
]
}Request:
curl -X GET "http://myserver.com:3000/api/v1/search/cb/appt/queue?from=1536676860&to=1536763260" -H "accept: application/json" -H "Authorization: JWT token"Response:
{
"cb": [
{
"count": 27,
"v1": "mr",
"cb": [
{
"ms": 1974235,
"count": 27,
"v2": "root.users.root",
"vs": 798
}
]
}
]
}Gets chargeback reports by application type for a specific queue.
GET http://unravel-host:3000/api/v1/search/cb/user?appt={app_type}>e={date}<e={date}&queue={queue_name}None.
Name | Type | Description |
|---|---|---|
| string | Application type. Required. Valid values are |
| string | Start date. Format: |
| string | End date. Format: |
| string | Queue name. Required. |
{
"cb": [
// array by application type
{
"ms": memory usage in milliseconds,
"count": application count,
"v2": userName | queueName,
"vs": vcore usage in seconds
}
]
}Request:
curl -X GET "http://myserver.com:3000/api/v1/search/cb/user?appt=mr>e=1536677580<e=1536763980&queue=root.users.root" -H "accept: application/json" -H "Authorization: JWT token"Response:
{
"cb": [
{
"ms": 2324825154,
"count": 4332,
"v1": "mr",
"vs": 1512734
},
{
"ms": 298989641,
"count": 75,
"v1": "spark",
"vs": 120404
}
]
}Gets chargeback reports by application type. Gives the count of all applications in all queues for all users across all clusters.
Compatibility
This endpoint is available on Unravel 4.5.0.9+. If you're using an older version of Unravel, see /search/cb/appt.
GET http://unravel-host:3000/api/v1/chargeback/cb/app?gte={date}<e={date}None.
Name | Type | Description |
|---|---|---|
| string | Start date. Format: |
| string | End date. Format: |
{
"cb": [
// array organizing by application type
{
"ms": memory usage seconds,
"count": application count,
"v1": "application type (mr | spark)",
"vs": vcore usage in second
}
]
}Request:
curl -X GET "http://myserver.com:3000/api/v1/chargeback/cb/appt?gte=2019-01-02<e=2019-04-02" -H "accept: application/json" -H "Authorization: JWT token"Response:
[
{
"type": "mr",
"appCount": 39,
"cpuHours": "4.14",
"memoryHours": "2Y 11M 3D 14h 27m 33s"
},
{
"type": "tez",
"appCount": 39,
"cpuHours": "87.16",
"memoryHours": "82Y 8M 26D 9h 14m 12s"
},
{
"type": "spark",
"appCount": 29,
"cpuHours": "2.14",
"memoryHours": "5M 13D 5h 9m 26s"
}
]Gets chargeback reports by user.
Compatibility
This endpoint is available on Unravel 4.5.0.9+. If you're using an older version of Unravel, see /search/cb/appt/user.
GET http://unravel-host:3000/api/v1/chargeback/cb/appt/user?gte={date}<e={date}None.
Name | Type | Description |
|---|---|---|
| string | Start date. Format: |
| string | End date. Format: |
{
"cb": [
// array by application type
{
"count": application count,
"v1": application type (mr, | spark), ,
"cb": [
//array of users or queues (depending on command)
{
"ms": memory usage in milliseconds,,
"count": application count,,
"v2": userName | queueName,,
"vs": vcore usage in seconds
}
]
}
]
Request:
curl -X GET "http://myserver.com:3000/api/v1/chargeback/cb/appt/user?gte=2019-01-02<e=2019-04-02" -H "accept: application/json" -H "Authorization: JWT token"Response:
[
{
"type": "mr",
"appCount": 37,
"cpuHours": "3.66",
"user": "hdfs",
"memoryHours": "2Y 8M 8D 16h 44m 45s"
},
{
"type": "mr",
"appCount": 2,
"cpuHours": "0.48",
"user": "hbase",
"memoryHours": "2M 24D 21h 42m 48s"
},
{
"type": "tez",
"appCount": 38,
"cpuHours": "87.14",
"user": "hdfs",
"memoryHours": "82Y 8M 18D 13h 30m 38s"
},
{
"type": "tez",
"appCount": 1,
"cpuHours": "0.02",
"user": "unravel",
"memoryHours": "7D 19h 43m 34s"
},
{
"type": "spark",
"appCount": 24,
"cpuHours": "1.60",
"user": "hdfs",
"memoryHours": "3M 27D 22h 4m 31s"
},
{
"type": "spark",
"appCount": 3,
"cpuHours": "0.52",
"user": "user1",
"memoryHours": "1M 14D 10h 47m 48s"
},
{
"type": "spark",
"appCount": 2,
"cpuHours": "0.01",
"user": "unravel",
"memoryHours": "20h 17m 7s"
}
]Gets chargeback reports by queue.
Compatibility
This endpoint is available on Unravel 4.5.0.9+. If you're using an older version of Unravel, see /search/cb/appt/queue.
GET http://unravel-host:3000/api/v1/chargeback/cb/appt/queue?gte={date}<e={date}None.
Name | Type | Description |
|---|---|---|
| string | Start date. Format: |
| string | End date. Format: |
{
"cb": [
// array by application type
{
"count": application count,
"v1": application type (mr, | spark), ,
"cb": [
//array of users or queues (depending on command)
{
"ms": memory usage in milliseconds,,
"count": application count,,
"v2": userName | queueName,,
"vs": vcore usage in seconds
}
]
}
]
}Request:
curl -X GET "http://myserver.com:3000/api/v1/chargeback/cb/appt/queue?gte=2019-01-02<e=2019-04-02" -H "accept: application/json" -H "Authorization: JWT token"Response:
[
{
"type": "mr",
"appCount": 37,
"cpuHours": "3.66",
"queue": "root.hdfs",
"memoryHours": "2Y 8M 8D 16h 44m 45s"
},
{
"type": "mr",
"appCount": 2,
"cpuHours": "0.48",
"queue": "root.hbase",
"memoryHours": "2M 24D 21h 42m 48s"
},
{
"type": "tez",
"appCount": 38,
"cpuHours": "87.14",
"queue": "root.hdfs",
"memoryHours": "82Y 8M 18D 13h 30m 38s"
},
{
"type": "tez",
"appCount": 1,
"cpuHours": "0.02",
"queue": "root.unravel",
"memoryHours": "7D 19h 43m 34s"
},
{
"type": "spark",
"appCount": 24,
"cpuHours": "1.60",
"queue": "root.hdfs",
"memoryHours": "3M 27D 22h 4m 31s"
},
{
"type": "spark",
"appCount": 3,
"cpuHours": "0.52",
"queue": "root.user1",
"memoryHours": "1M 14D 10h 47m 48s"
},
{
"type": "spark",
"appCount": 2,
"cpuHours": "0.01",
"queue": "root.unravel",
"memoryHours": "20h 17m 7s"
}
]