Pipelines
This tab lists all the running workflows in your cluster which are known as pipelines. For implementing pipelines In a multi-cluster setup, you must configure the com.unravel.workflows.compareupdate.disable to true on the Core node.
Note
A Multi-cluster monitoring feature is implemented for on-prem clusters in Unravel. However, in Pipelines, a pipeline can only contain apps that belong to the same cluster. Currently, a scenario wherein the apps belonging to the same pipeline but are sourced from different clusters is not supported.
To filter and view the pipelines, specify any of the following, and the matching pipelines are displayed.
Click in the Search by pipeline name or user text box and specify a pipeline name or user name.
Click in the Cluster text box for available clusters and select the required cluster.
From the date picker, select a time range.
The status of the pipeline is noted by a colored bar on the left-hand side of the row.
Color code | Description |
|---|---|
| Running applications. |
| Successful applications. |
| Failed applications. |
| Unknown applications. NoteTagged pipelines are always displayed with Unknown status. |
The table columns in the pipeline list are:
Columns | Description |
|---|---|
Pipeline Name | Name of the pipeline |
User | Pipeline owner |
Start Time | The start time of the nth instance. The time displayed is rounded to the nearest second. |
Instances | The number of times the pipeline has run. |
Duration | Length of time for the pipeline to complete execution. This shows a trend line mapping from the first to the last instance. You can hover over the trend line to view the date, time, duration of the instance. Adjacent to the trendline is the percent of duration change between the last two runs (if there was any delta), and the absolute value of the last run. |
Write | Total amount of memory written by the pipeline. This shows a trend line mapping from the first to the last instance. You can hover over the trend line to view the date, time, and write bytes of the instance. Adjacent to the trendline is the percent of write changes between the last two runs (if there was any delta), and the absolute value of the last run. |
Read | Total amount of memory read by the pipeline. This shows a trend line mapping from the first to the last instance. You can hover over the trend line to view the date, time, duration, and read bytes of the instance. Adjacent to the trendline is the percent of read change between the last two runs (if there was any delta), and the absolute value of the last run. |
Cluster Id | The cluster where the pipeline is running. |
In the following example, there are two pipelines. The first job shows a difference of +99.22% increase in time with the absolute duration of 43s.
Hover over the trendline to view more details.
Click a pipeline row to bring up the corresponding pipeline details page.
Pipeline details page
The Pipeline details page provides a summary of the instances that are run for a specific pipeline.
The following details are noted on the upper right corner of the page:
Pipeline owner
Cluster ID
Queue
The following details are provided in the upper left side of the pipeline details page:
Item | Description |
|---|---|
1 | Shows the name of the pipeline. |
2 | Shows the status of the application in the pipeline. |
3 | Start time of the application run. End time of the application run. Duration of the application run. |
4 | A drop-down to select any of the following metrics to update the graph
|
5 | Graph that plot the metrics that you have selected. Click a point in the graph to view more details. |
6 | Click to toggle between bar graph and line graph. |
7 | Click to expand the Instance Summary view. |
The details page initially opens displaying the insights for the last instance within the Analysis tab. In case there are no insights, the Resources page is displayed.
Navigation
The Navigation tab lists the pipelines in a cluster with the following details;
Type: Type of the Pipeline
ID: Name of the pipeline.
Start Time: Time when the pipeline started.
Duration: Time when the pipeline ended.
I/O: Input/Output data processed.
: The number of insights/recommendations for a specific pipeline.
Click a row within the pipeline list. The corresponding applications within a pipeline are displayed. When you click a row within the applications list the corresponding app details page is shown.
Gantt Chart
The Gantt tab visually displays the progress of all the pipelines in the cluster along with their corresponding status in the color-coded bar against a timescale.
You can expand each pipeline to view applications in a pipeline and the progress of their corresponding jobs Further, if you click an application, the details of the app can viewed in the app details page.
Instance Summary
The Instance Summary page provides the details of a pipeline instance from the following tabs:
The Unravel intelligence engine helps you manage your jobs and pipelines by analyzing its run and providing insights based upon the runs. Unravel's analysis results show where the efficiencies of your app can be improved at cluster level as well as at the application level.
The number of efficiencies found is noted in the tab. When an Efficiency leads to a suggested parameter/property change, those suggestions are listed as a Recommendation.
Efficiency (
): Suggest areas to examine to improve efficiency. For example, inefficient usage of resources. Expand an Efficiency to read the in-depth analysis. While some are only configuration property suggestions, others call your attention to areas you should examine. For example, inefficient storage, problems with a query etc.Recommendations (
): Suggestions to update a property or configuration parameter. Click to view the current and recommended value.
Resources
The resources contain the following charts:
Task Attempt (Map): Graphs the successful and failed Map attempts.
Task Attempt (Reduce): Graphs the successful and failed Reduce attempts.
Attempt: Graphs Slot over Time for Map and Reduce tasks.
Pipeline Dag Graph
This tab presents the Dag graph of the pipeline. You can click on the stages to view the details and status of each stage.
Errors
This tab lists the errors that were encountered during the instance run.
Tags
This tab lists the tags used in the pipeline instances. Refer to Tagging.
