Home

Report endpoints

These endpoints return key performance indicators (KPIs) for clusters, the small files report, chargeback reports (by user, queue, app), and cluster workload reports.

Cluster reports
Chargeback reports
/reports/archives/get_report_archives

Under construction.

/reports/celeryreports

Under construction.

/reports/celeryreports/metadata

Under construction.

/reports/clusteroptimization/metadata

Under construction.

/reports/data/diskusage/capacity_task_details

Under construction.

/reports/data/diskusage/get_latest_reports
/reports/data/diskusage/get_latest_reports?task_name=small_files_report

Gets the latest small file report.

Request
GET http://unravel-host:3000/api/v1/reports/data/diskusage/get_latest_reports?task_name=small_files_report
Path Parameters

None.

Query Parameters

Name

Type

Description

task_name

string

Task name. Set this to small_files_report.

Response Fields
{
  "date": date created EPOCH_timestamp,
  "isSuccess": report generation sucess/failure,

// report parameters
  "avg_size_threshold": small file size in bytes,
  "num_files_threshold": minimum number of small files,
  "top_n_small_files": # of directories to show,

  "report_id": report name",
  "root": [
    // array[top_n_small files] directory
    {
      "MaxFilesize": maximum size file in directory,
      "MinFilesize": minimum size file in directory,
      "NumFiles": # of small files in the directory,
      "DirPath": directory path,
      "TotalFilesize": sum of file size in directory,
      "AvgFilesize": average file size
    }
  ]
}
Examples

Request:

curl -X GET "http://myserver.com:3000/api/v1/reports/data/diskusage/get_latest_reports?task_name=small_files_report" -H  "accept: application/json" -H  "Authorization: JWT token"

Response:

{
  "date": 1536899695,
  "isSuccess": true,
  "avg_size_threshold": 100000,
  "num_files_threshold": 100,
  "top_n_small_files": 5,
  "report_id": "small_files_1536927781_5674",
  "root": [
    {
      "MaxFilesize": "0",
      "MinFilesize": "0",
      "NumFiles": "21523359",
      "DirPath": "\/97ovg",
      "TotalFilesize": "0",
      "AvgFilesize": "0"
    },
    {
      "MaxFilesize": "0",
      "MinFilesize": "0",
      "NumFiles": "11302185",
      "DirPath": "\/98esm",
      "TotalFilesize": "0",
      "AvgFilesize": "0"
    },
    {
      "MaxFilesize": "0",
      "MinFilesize": "0",
      "NumFiles": "7174452",
      "DirPath": "\/99acx",
      "TotalFilesize": "0",
      "AvgFilesize": "0"
    },
    {
      "MaxFilesize": "0",
      "MinFilesize": "0",
      "NumFiles": "7174452",
      "DirPath": "\/97ovg\/97zqu",
      "TotalFilesize": "0",
      "AvgFilesize": "0"
    },
    {
      "MaxFilesize": "0",
      "MinFilesize": "0",
      "NumFiles": "7174452",
      "DirPath": "\/97ovg\/99kgg",
      "TotalFilesize": "0",
      "AvgFilesize": "0"
    }
  ]
}
/reports/data/diskusage/get_latest_success_reports

Under construction.

/reports/data/kpis

Generates a Key Performance Indicator (KPI) report.

Request
GET http://unravel-host:3000/api/v1/reports/data/kpis?numDays=num-days
Path parameters

None.

Query parameters

Required parameters are highlighted.

Name

Type

Description

numDays

integer

Number of days to include.

Response body

Name

Type

Description

st

integer

Start time in Unix epoch format.

et

integer

End time in Unix epoch format.

nlaTb

integer

Number of tables accessed.

nlaPr

integer

Number of partitions accessed.

nlaQr

integer

Number of queries accessing the table.

nlaRi

integer

Total Read I/O due to accessing the tables.

nlcTb

integer

Number of tables created.

nlcPr

integer

Number of partitions created.

nlcTz

integer

Size of tables created.

nlcPz

integer

Size of partitions created.

ntoTb

integer

Total number of tables in the system.

ntoPr

integer

Total number of partitions in the system.

nhtTb

integer

nwaTb

integer

ncoTb

integer

nhtPr

integer

nwaPr

integer

ncoPr

integer

rp

integer

rs

integer

fs

integer

users

array

Comma-separated list of users.

Examples

Request:

curl -X GET "http://myserver.com:3000/api/v1/reports/data/kpis?numDays=1" -H  "accept: application/json" -H  "Authorization: JWT token"

Response:

[
  {
    "st": 1536795705,
    "et": 1536882105,
    "nlaTb": 15,
    "nlaPr": 0,
    "nlaQr": 57,
    "nlaRi": 101489411821,
    "nlcTb": 7,
    "nlcPr": 0,
    "nlcTz": 304768890,
    "nlcPz": 0,
    "ntoTb": 378,
    "ntoPr": 30061,
    "nhtTb": 19,
    "nwaTb": 0,
    "ncoTb": 359,
    "nhtPr": 1823,
    "nwaPr": 0,
    "ncoPr": 28238,
    "rp": 30061,
    "rs": 92544293666,
    "fs": 92544293666,
    "users": [
      "root",
      "hdfs"
    ]
  }
]
/reports/data/small_file_report_details

Gets details about the small file report.

Request
GET http://unravel-host:3000/api/v1/reports/data/small_file_report_details
Path parameters

None.

Query parameters

None.

Response body

Name

Type

Description

[no name]

array

JSON structure containing details about the small file report.

Examples

Request:

curl -X GET "http://myserver.com:3000/api/v1/reports/data/small_file_report_details" -H  "accept: application/json" -H  "Authorization: JWT token"
/reports/files/reports/latest

Under construction.

/reports/operational/clusterstats?st={timestamp}&et={timestamp}&mode={summary_type}

Gets a summary of the cluster's users, apps, or queues.

Request
GET http://unravel-host:3000/api/v1/reports/operational/clusterstats?st={timestamp}&et={timestamp}&mode={summary_type}
Path parameters

None.

Query parameters

Required parameters are highlighted.

Name

Type

Description

st

string

Start time, in Unix epoch format.

et

string

End time, in Unix epoch format.

mode

string

Summary type.

Valid values are user, app, queue.

Response body

JSON schema when mode is user or queue:

{
  "userStats": [
// array of users | queues currently on cluster
    {
      "root": {
        "running": {
          "min":  minimum applications running,,
          "max": minimum applications running,
          "mean": average appication number running,
          "stddev": standard deviation
        },
        "memory":  { // see running above },
        "pending": {  // see running above },
        "vcores":  {  // see running above }
      }
    }
  ]
}
Examples

Request with mode=user:

curl -X GET "http://myserver.com:3000/api/v1/reports/operational/clusterstats?st=1536658189000&et=1536744589000&mode=user" -H  "accept: application/json" -H  "Authorization: JWT token"

Response for mode=user:

{
  "userStats": [
    {
      "root": {
        "running": {
          "min": 1,
          "max": 1,
          "mean": 1,
          "stddev": 0
        },
        "memory": {
          "min": 1024,
          "max": 16384,
          "mean": 7040,
          "stddev": 7099.0974074174
        },
        "pending": {
          "min": 0,
          "max": 0,
          "mean": 0,
          "stddev": 0
        },
        "vcores": {
          "min": 1,
          "max": 3,
          "mean": 2,
          "stddev": 0.81649658092773
        }
      }
    }
  ]
}

Request with mode=queue:

curl -X GET "http://myserver.com:3000/api/v1/reports/operational/clusterstats?st=1536658189000&et=1536744589000&mode=queue" -H  "accept: application/json" -H  "Authorization: JWT token"

Response for mode=queue:

{
  "queueStats": [
    {
      "root.users.root": {
        "running": {
          "min": 1,
          "max": 1,
          "mean": 1,
          "stddev": 0
        },
        "memory": {
          "min": 1024,
          "max": 16384,
          "mean": 7040,
          "stddev": 7099.0974074174
        },
        "pending": {
          "min": 0,
          "max": 0,
          "mean": 0,
          "stddev": 0
        },
        "vcores": {
          "min": 1,
          "max": 3,
          "mean": 2,
          "stddev": 0.81649658092773
        }
      }
    }
  ]
}

Response for mode=app:

{
   [
     appID : application id,
     type : memorySeconds/vcoreSeconds
     vcore : vcore value
     memory : memory value
   ]
}
/reports/operational/clusterworkload?gte={timestamp}&lte={timestamp}&reportBy={interval}

Gets a summary of the cluster's workloads.

Request
GET http://unravel-host:3000/api/v1/reports/operational/clusterworkload?gte={timestamp}&lte={timestamp}&reportBy={interval}
Path parameters

None.

Query parameters

Required parameters are highlighted.

Name

Type

Description

gte

string

Start time, in Unix epoch format.

lte

string

End time, in Unix epoch format.

reportBy

string

Report interval.

Valid values are month, hour, hourday.

Response body
// work load by month one of more months with the application count for month
// minimum of one month
  { timestamp: appcount[,timestamp: appcount] }

// work load by hour array of 25 hours
   [
      { timestamp: appcount }, ...  { timestamp: appcount }
   ]

// work load by day - array for each days contained with the time period (Mon - Sun)
// minimum of one day
   [
      { timestamp: appcount }
   ]

// work load by hour/day:array for each day (Mon - Sun) by hour
// minimum of 24 hours for one day
  [
    { timestamp: appcount }
  ]

Examples

Request:

curl -X GET "http://myserver.com:3000/api/v1/reports/operational/clusterworkload?gte=1536777000Z&lte=1536863400Z&reportBy=month" -H  "accept: application/json" -H  "Authorization: JWT token"

Response:

{"1536777000":96,"1536863400":4}  
[
{"1536813000000":0},{"1536816600000":0},{"1536820200000":0},{"1536823800000":10},{"1536827400000":7},{"1536831000000":13},{"1536834600000":10},{"1536838200000":31},{"1536841800000":0},{"1536845400000":0},{"1536849000000":4},{"1536852600000":2},{"1536856200000":0},{"1536859800000":0},{"1536863400000":0},{"1536867000000":0},{"1536870600000":0},{"1536874200000":0},{"1536877800000":0},{"1536881400000":0},{"1536885000000":0},{"1536888600000":0},{"1536892200000":0},{"1536895800000":6},{"1536899400000":2}]
[
{"1536777000000":96},{"1536863400000":7}
]
[{"1536813000000":0},{"1536816600000":0},{"1536820200000":0},{"1536823800000":10},{"1536827400000":7},{"1536831000000":13},{"1536834600000":10},{"1536838200000":31},{"1536841800000":0},{"1536845400000":0},{"1536849000000":4},{"1536852600000":2},{"1536856200000":0},{"1536859800000":0},{"1536863400000":0},{"1536867000000":0},{"1536870600000":0},{"1536874200000":0},{"1536877800000":0},{"1536881400000":0},{"1536885000000":0},{"1536888600000":0},{"1536892200000":0},{"1536895800000":6},{"1536899400000":2}]
/reports/queueanalysis/get_latest_report_by_queue_name

Under construction.

/reports/schedules

Under construction.