Errors: Lists all errors associated with the app. The errors are color coded (fatal , errors , warnings ) and the number for each type noted. The total number of errors is listed at the top. They are grouped by executors with the number and types of errors noted. Errors are categorized by severity, and include keywords and details associated with each. Keywords help to extract important details from the error messages/log data to help quickly root out the cause of a problem. If there are no errors, then "No errors found" is listed.

Logs: The critical logs that were collected for this Spark app. “No Logs Found” is displayed when Unravel is unable to load any log information.

Click the log name to see it, this example is an excerpt of the executor-20 log.

Conf: The configuration parameters and their values. The tab opens listing all the properties, according to app/task/job. The number of properties displayed is noted above the list.

You can narrow the list by choosing the configuration type to display; to see the Spark version select metadata. Metadata and driver are selected in this example and the list narrowed from 1042 to two properties. Some properties appear in multiple categories, for example, spark.executor.extraJavaOptions is listed under Memory, Driver and Executor. You can search by name; searching on YARN displays every property containing the word yarn. Click Reset to display all properties.

Program: Displays the program's source code if available. Both the Spark Stage and Execution graph link to this tab and display code related to the tab. See Uploading Spark Programs.

Note: This tab only loads for spark-submit job types, see

Graphs: This tab is only displayed when there is available data. If a graph has less than four data points, it is converted to a bar graph. The graphs use wall clock time as opposed to computer usage time.

Unravel gather this data from the Resource Manager which it polls every 90 seconds while the app is running. If the app has a runtime less than 90 seconds there is likely no data.

It has three subtabs:

Resource: Graphs the resources the app or stage consumed. Unravel gathers this data from the Resource Manager which is polled every 90 seconds while the app is running. If the app's runtime is less than 90 seconds there is likely to be no data collected.

When no resource data is available the tab displays “No metrics found. Metrics are displayed after Unravel has processed the app completely and Unravel's application sensors are enabled.”

By default, the graph displays all executors using the metric availableMemory. Beneath the list notes the page, if more than one, that is being displayed. Click the arrow to page down or up to display a page. Choose the metric for the graph using the pull-down menu. The metrics fall into one of the following categories: OS Memory, CPU, and Java Memory. Click Get Data to see the resource information displayed in JSON. You can toggle whether a resource is displayed by clicking on it; it is greyed out when it's not displayed.

You choose one or more series to display using the Select box. Clicking within the box brings up a list of all resources available. Click the resource you want to display. You can choose to display from one to all resources. Only the chosen resources are listed at the bottom of the box. Click the X next to the resource to remove it from the list.

Example JSON file obtained by clicking Get Data.

Task Attempts: Graphically and textually notes the number of tasks and their status. The donut graphs show the percentage of successful (green), failed (orange), and killed (red) tasks. A legend on the right lists the number of each. The following graph on the left shows a job in which all tasks succeeded, while the on the right has all failed tasks. Typically, the result is a combination. Hovering over the chart shows you the percentage of each.

Timings: Graphically presents the time spent by an application for various tasks; the application’s wall clock time as well as the time spent by different tasks on each executor. The tab has two views:

Events: is the number, if any, of Unravel insights for the query. See the here for more information.

Duration: The "wall clock" time it took for the app to complete.

Data I/O: The memory used.

Number of Stages: The number of stages in the query.

Navigation: The app's jobs with their relevant KPIs: Status, Start Time, Duration, Partitions/Tasks (), DFS Read, DFS Write, # Stages. See above for more details.

Execution: The execution graph shows a RDD DAG of the app. The app's RDDs are represented by the vertices while the edges are the Operations to be applied. Any particular RDD can involve multiple stages so there isn't a one-to-one mapping between a RDD and stages. The graph is "linked" to the Program tab. If the source code is available, clicking on a vertex brings up the source line associated with the RDD.

If the program tab displays the code it is linked with the DAG. Display the execution and program tab simultaneously, click the vertex to highlight the relevant code. Here we see the corresponding code for vertex 18.

We expanded the above area, and vertices 15-19 are shown (the vertex number is noted in the circle). The vertex lists the type of RDD, partitions used, Spark call, and the number of stages which were involved. The RDD represented by vertices 17-16 involved two stages, while 15-16 had five. Hover over the vertex to bring up an information box, containing the RDD description and CallSite (source line) which called the RDD transformation.

Gantt Chart:

Displays the stages using a Gantt Chart. The table is sorted on Start Time in ascending order.

Errors, Log and Conf Tabs: For an explanation of these tabs see Errors, Logs, and Conf.

Program: When available it displays the program associated with the app. See above for an example of this tab and its relation to the Execution graph.

Task Attempts, Graphs, and Resources: For an explanation of these tabs see Task Attempts, Graphs, and Resource.

Events: is the number, if any, of Unravel insights for the query. See here for more information.

Duration: The "wall clock" time it took for the app to complete.

Data I/O: The memory used.

# of Stages: The number of stages in the query.

Navigation: The app's jobs with their relevant KPIs: Status, Start Time, Duration, Partitions/Tasks (), DFS Read, DFS Write, # Stages. See above for more details.

Execution: A execution graph of the query. There are times when the DAG is too large to display and it will be noted. See above for more details.

Gantt Chart: Displays the stages using a Gantt Chart. For more details see above.

Errors, Log, and Conf Tabs: For an explanation of these tabs see Errors, Logs, and Conf.

Program: This tab connects all the pieces of the SQL query. The table lists all queries with significant KPIs and the top five stages, i.e., the stages with the longest duration. Beneath the list are two tabs, SQL and Program The SQL tab displays the SQL query in either query or plan mode. The Program tab lists the query's corresponding Program. For a particular query, the SQL query text, Program, and stages are linked.

By default:

Click the Query ID to display its SQL and program. Click the stage to display its Spark Stage Detail. See Spark Stage Details for further information on the stage view. To switch between SQL Query and Plan view, click the plan/query button in the upper right-hand corner of the SQL window. To copy the SQL query to your clipboard click the Copy button. You don't have to have the query view showing in order to copy it.

This screenshot shows the default window, the SQL query for Query ID 4.

Scroll down to see the entire SQL Plan. Click Query to return to the program.

Task Attempts, Graphs, and Resources: For an explanation of these tabs see Task Attempts, Graphs, and Resource.

Events: Is the number, if any, of Unravel insights for the query. See the here for more information.

Duration: The total time to process all stages. (The wall clock time can be calculated from the start and end times are on the left-hand side of the job bar.)

Data I/O: Total data read and written by the query.

Number of Jobs: The number of jobs that make up the streaming.

Number of Stages: The number of stages that make up the streaming.

Stream: Displays the core of a Streaming Application. From here you drill down into the batches, the main processing unit for Spark streaming. The graph displays the number of events/second (the bars) with a superimposed line graph of the chosen metric. By default, Scheduling Delay is used. You can change the date range to display or the metric used via the Metric pull-down menu (1). You can choose Scheduling Delay, Processing Time or Total Delay. In the following example, the graph is displaying Scheduling Delay Time.

It has two sections; by default, they display the entire run over the last 7 days. You can zoom in on a section of graph by pulling the tabs left or right (2). The table lists the Completed Batches relevant to the time period selected. Each batch has its KPI's listed. In the view above, the entire stream time is displayed, therefore all Completed Batches are displayed and in this case there are seven pages. In the example, we have zoomed in on the last two minutes, the table now lists the batches completed in that time period. The tables now contains only one page, versus the seven above.

The table lists only the first three batches, but you can page through the table (3). By default, the streams are sorted on start time in ascending order. When you sort the batches, they are sorted across all tables, i.e., if Start Time is switched to descending list, nth batch becomes the first, the n-1 the second, etc.

Click a batch to bring up the Spark Stream Batch tile. You can only open one batch job at a time. The batch window lists all the jobs associated with the batch and the batch's metadata. Its title bar notes it's a Spark Stream Batch view and that it's part of a Spark Streaming app. The KPI's, Duration, Processing Delay, Scheduling Delay, and Total Delay refer to this Batch job. It has two tabs, Output Operation and Input.

This example is of the batch in the first line of the table above. The Stream Batch has two calls and the first call has two jobs. Since these jobs are run in parallel, the job with the longest time determines the duration of the batch. The description notes the RDD and the call line; clicking on the description displays the associated code in the program window. Click the Job ID to see the Spark Job information.

The Input Tab lists all the Topics consumed.

Errors, Log, and Conf: For an explanation of these tabs see Errors, Logs, and Conf.

Program: The program (if uploaded by the user) is shown in this tab.

Task Attempts, Graphs, and Resources: For an explanation of these tabs see Task Attempts, Graphs, and Resource.