Skip to main content

Home

Dataflow Migration report

A report to analyze the catalog of apps and tables and derive a sprint plan to migrate them to another environment. You can create this report either from the UI or the command line.

You can use this report to analyze Spark, Impala, Hive, and MR data.

  1. Click Generate Reports > New button.

  2. In the New Report dialog box, enter the following details.

    Items

    Description

    General

    Name

    Name of the report.

    Environment

    Select your platform. The reports corresponding to the selected platform are only listed in the Report type. If you select the All option, all the reports are listed.

    Report type

    Type of report. Select Dataflow Migration Report.

    Schedule

    Select the checkbox to schedule the report to run daily, hourly, weekly, or monthly. You can also set the schedule can using a cron expression. You can expand the Example drop-down and select the corresponding options. The next four sample run times are displayed for reference.

    Retention

    The number of days to retain the report files. All the reports are stored in the unity-one/src/assets/reports/jobs directory. After completion of the retention period, the report files are automatically purged.

    Parameters

    Application Kind

    Select the type of application. Options are Impala, Hive, Spark, and MR.

    Look Back

    The period range in days when applications can be selected for report generation. A notification is displayed above this option that informs about the duration when data is available for the report.

    Use Exact Date-Time

    Check this option and select the from and to dates to schedule the report for the exact time range specified. The only anomaly with this option is that the same report will be scheduled to be generated repeatedly.

    Users

    Select the users who must be included in the report. You can select multiple users. If none are selected, all the users will be included in the report.

    Queues

    Select the queues that you want to be included in the report. You can select multiple queues. If none are selected, all the queues will be included in the report.

    Clusters

    Select the clusters that you want to be included in the report. You can select multiple clusters. If none are selected, all the clusters will be included in the report.

    Tag Names

    Select the tag names that you want to be included in the report. You can select multiple tag names. If none are selected, all the tag names will be included in the report.

    Tag Values

    Select the tag values corresponding to the tag names you have selected that you want to include in the report. Options for tag values will only show when you select a tag. You can choose multiple tag values. If none are selected, all tag values detected for a tag will be included in the report.

    Number of Waves

    Set of applications that can be migrated together. Several turns are required for migration.

    Number of hours per engineer per sprint

    The number of hours that the engineers worked in a sprint.

    Features filter

    Specify the Key-Value pair. You can add or remove a pair.

    Notifications

    Email to

    Email ID to send the notification when the report is generated. You can specify multiple email IDs with comma separation. You can also select the Attach Files to Email checkbox to receive the reports as an attachment.

    Advance Options

    Profile Memory

    Select this option if you want to generate logs that help to troubleshoot scenarios where the report takes excessive time to generate or fails to generate.

    Caution

    The option will increase the report run time excessively.

  3. Click OK. The generated reports will be listed under Reports on the App UI.

  1. From the terminal, change directories to unity-one/src directory.

    cd /opt/unravel/data/apps/unity-one/src

  2. Use a screen editor, such as vi editor, to edit the job.json file and copy-paste the following text into the file. 

    {
    "enabled": false,
    "retention_days": 10,
    "report_type": "dataflow_migration",
    "params": {
        "kind": [
            "spark",
            "impala",
            "hive",
            "mr"
        ],
        "days": 300,
        "num_waves": 10,
        "num_hours_engg": 75
    },
    "notifications": {}
}

    Parameters

    Description

    start_date

    Can be null or a valid start date.

    end_date

    Can be null or a valid start date.

    days

    The number of days to look back while downloading data.

    topk

    The number of applications to be shown in the report.

    all_filters

    Filter options.

  3. Save and exit the screen editor.

  4. Run the following command to generate the report:

    ./unity-cli.sh report generate job.json dataflowMigReport true

The Dataflow Migration report generates one file for each application type selected and one file to summarize all the data.

  • Dataflow Migration Report

    It contains the number of apps, total hours, and number of engineers for each wave.

    unity-dataflow-migration.png

  • Dataflow Migration Report for hive Data

    It contains an overall summary and rank list, and estimated effort with links to sample apps and signature.

    unity-dataflow-migration1.png
Viewing metrics and insights

  1. Open the report and click any app link from the Signature column.

    The Metrics tab displays the trend charts for the Duration, IO, MemorySeconds, and CPU metrics. The dots plotted on the trend charts are the app runs. The app info is provided in the right panel.

  2. You can select any of the following features to view the metric trends:

    • status

    • cluster id

    • cluster UID

  3. Click any dot on any of the Metric trend charts. This becomes the app target run. The details of the app target run appear in the App Run info panel on the right.

  4. In the App Run info panel, click the ID link. This is the ID link of the app target run. The Metric tab is shown with the app target run you have chosen in a different color. The details of the app target run are displayed in the App Target Run panel on the right. Click the View in Unravel link to view the app's details in Unravel.

    targetapprun-highlighted.png

  5. Select any dot (other than the dot for the app target run) on any of the Metric trend charts. This becomes the app baseline run. This is highlighted in a different color. The details of the app target run are displayed in the App Baseline Run panel on the right. Click the View in Unravel link to view the app's details in Unravel.

    baseapprun-highlighted.png

  6. In the App Baseline Run panel, click Compare.

    1. Select an app from the App Type options:

      • Hive on Tez

      • Spark

      • Impala

    2. From the Cluster, select a cluster.

    3. Specify the app runs that you want to compare.

      • Baseline App Run: Specify the ID of the baseline app.

      • Target App Run: Specify the ID of the target app with which you want to compare the baseline app.

    4. Click Compare. The following app details are given for the baseline and target app run:

      • App details

      • Cluster resource usage when these apps were running

      • App statistics comparison details

      • App config comparison details

      • Query string comparison details

      • Query plan comparison details

      unity-app-app-comparator-results.png

  7. In the App Baseline Run panel, click Refresh Insights. The insights are provided in the following sections.

    • Baseline comparison

      The results of the comparison are displayed.

    • Concurrent heavy hitter apps

      This section will show up only when the information is available.

      Select this option to view the top five applications in a specific category.

      1. Click the Concurrent heavy hitter apps option.

      2. Select any option from the Top-5 Applications option.

        troubleshooting-tuning-concurrent-heavy-hitter.png

        The corresponding details are displayed.

        troubleshooting-tuning-concurrent-heavy-hitter1.png