Skip to main content

Home

Pipelines

This tab lists all the tagged workflows, also called pipelines, in the workspace.

To filter and view the pipelines, specify any of the following, and the matching pipelines are displayed.

  • Click in the Search by job name or user text box and specify a pipeline name or user name.

  • From the date picker, select a time range.

    Pipeline-screen1.png

The pipeline status is noted by a colored bar on the left-hand side of the row.

Color code

Description

blue.png

Running applications.

green.png

Successful applications.

orange.png

Failed applications.

grey.png

Unknown applications.

Note

Tagged pipelines are always displayed with Unknown status.

The table columns in the pipeline list are:

Columns

Description

Pipeline Name

Name of the pipeline

User

Pipeline owner

Start Time

The start time of the nth instance. The time displayed is rounded to the nearest second.

Instances

The number of times the pipeline has run.

Duration

Length of time for the pipeline to complete execution. This shows a trend line mapping from the first to the last instance. You can hover over the trend line to view the instance's date, time, and duration.

Adjacent to the trendline is the percent of duration change between the last two runs (if there was any delta), and the absolute value of the last run.

Write trend

Total amount of memory written by the pipeline. This shows a trend line mapping from the first to the last instance. You can hover over the trend line to view the instance's date, time, and write bytes.

Adjacent to the trendline is the percent of write changes between the last two runs (if there was any delta), and the absolute value of the last run.

Read trend

Total amount of memory read by the pipeline. This shows a trend line mapping from the first to the last instance. You can hover over the trend line to view the date, time, duration, and read bytes of the instance.

Adjacent to the trendline is the percent of read change between the last two runs (if there was any delta), and the absolute value of the last run.

Example of Duration

In the following example, there are two pipelines. The first job shows a difference of +99.22% increase in time with the absolute duration of 43s.

Jobs-Pipelines.png

Example of Read and Write trend

Hover over the trendline to view more details.

jobs-pipeline-hoverover-trendline.png
Pipeline details page

The Pipeline details page provides a summary of the instances that are run for a specific pipeline.

Click a pipeline row to bring up the corresponding pipeline details page.

jobs-pipeline-example1.png

The following details are noted on the upper right corner of the page:

  • Pipeline owner

  • Cluster ID

  • Queue

The following details are provided in the upper left side of the pipeline details page:

jobs-pipeline-apms-left.png

Item

Description

1

Shows the name of the pipeline.

2

Shows the status of the application in the pipeline.

3

Start time of the application run.

End time of the application run.

Duration of the application run.

4

A drop-down to select any of the following metrics to update the graph:

  • Duration

  • IO

  • Number of Apps

  • Number of Events

5

Graph that plot the metrics that you have selected. Click a point in the graph to view more details.

6

Click to toggle between bar graph and line graph.

7

Click to expand the Instance Summary view.

The details page initially opens displaying the insights for the last instance within the Analysis tab. In case there are no insights, the Resources page is not displayed.

Navigation

The Navigation tab lists the pipelines in a cluster with the following details;

jobs-pipeline-navigation.png
  • Type: Type of the pipeline.

  • ID: Name of the pipeline.

  • Start Time: Time when the pipeline started.

  • Duration: Time when the pipeline ended.

  • I/O: Input/Output data processed.

  • insights-icon.png: The number of insights/recommendations for a specific pipeline.

Click a row within the pipeline list. The corresponding applications within a pipeline are displayed. When you click a row within the applications list the corresponding app details page is shown.

Gantt Chart

The Gantt tab visually displays the progress of all the pipelines in the cluster along with their corresponding status in the color-coded bar against a timescale.

jobs-pipeline-gantt.png

You can expand each pipeline to view applications in a pipeline and the progress of their corresponding jobs. Further, if you click an application, you can view the details of the app in the app details page.

Instance Summary

The Instance Summary page provides the details of a pipeline instance from the following tabs:

Resources

The resources contain the following charts:

jobs-pipeline-resources-tab.png
  • Task Attempt (Map): Graphs the successful and failed Map attempts.

  • Task Attempt (Reduce): Graphs the successful and failed Reduce attempts.

  • Attempt: Graphs Slot over Time for Map and Reduce tasks.

Pipeline Dag Graph
pipeline-dag-chart.png

This tab presents the Dag graph of the pipeline. You can click on the stages to view the details and status of each stage.

Errors

This tab lists the errors that were encountered during the instance run.

jobs-pipeline-errors.png
Tags

This tab lists the tags used in the pipeline instances. Refer to Tagging.Tagging

pipeline-tags.png