Unravel's CI/CD platforms integration overview
Overview
Unravel offers Continuous Integration and Continuous Deployment (CI/CD) integration for Azure DevOps and GitHub platforms. This new capability enhances data application development by integrating Unravel's intelligence and monitoring tools into the CI/CD pipeline. This integration allows you to catch performance problems early on, helps you to take actions to mitigate the issues proactively, and promotes cost and performance awareness.
Unravel gathers essential details about applications like workspace, run ID, and job ID from the cluster and renders data about the app ID within Databricks, cluster ID, estimated cost, executor node type, driver node, applied tags, auto-scale settings, and Unravel's insights associated with the cluster.
What to expect
Unravel CI/CD integration allows for early detection and prevention of performance issues.
Developers can proactively address bottlenecks, anomalies, and failures before they impact users.
Teams can meet financial and performance goals by analyzing critical metrics during development.
Bottlenecks are identified and resolved promptly, reducing cycle times.
Real-time visibility helps in making data-driven decisions.
Refer to the following video to understand the functioning of Unravel's CI/CD integration for your organization.
CI/CD integration use cases
Scenario
The developer makes code changes and commits them.
A pull request (PR) is raised with links to Databricks job runs.
Unravel CI/CD retrieves relevant data.
Unravel-Bot (Unravel account) renders crucial performance insights within the PR page.
Impact
By leveraging the links developers provide to test Databricks job runs within PRs, Unravel CI/CD enhances operational excellence by facilitating early issue detection. This ensures potential problems are identified and addressed before impacting users, improving performance and actionability.
Early detection of issues within PRs ensures that code is thoroughly reviewed and tested before merging, reducing the chances of post-deployment disruptions and user-experience problems.
Improved issue detection enhances application reliability, providing a consistent and dependable user experience.
Scenario
The developer makes code changes and commits them.
A pull request (PR) is raised with links to Databricks job runs.
Unravel CI/CD retrieves relevant data and renders actionable real-time insight for cost optimization. This scenario calculates the potential cost savings based on resource usage patterns from past job runs.
Unravel-Bot (Unravel account) suggests specific instances that can be downsized to save costs. For example, you are prompted to decrease certain worker instances from "standard_xyz" to "standard_abc" to achieve a cost saving of *.** dollars.
By comparing the current instance (e.g., standard_xyz) with the recommended instance (e.g., standard_abc). Unravel-Bot (Unravel account) highlights differences in memory, CPU cores, hourly costs, and purpose (e.g., general purpose vs. compute optimized).
The developer reviewing the PR gains real-time insights into the financial impact of the code changes. This helps the developer to make informed decisions regarding resource allocation and utilization.
The developer opts to implement the recommended changes and make adjustments to optimize resource usage and saves costs.
Impact
Unravel CI/CD empowers teams to make cost-conscious decisions, potentially saving significant resources. Teams can optimize resource allocation, ensuring that resources are used effectively and minimize wastage.
Integrating cost considerations into the development process leads to cost-efficient application development and enhances financial management within the development team. This improves financial efficiency and ensures that projects stay on budget, and financial goals are met, contributing to better financial management practices.
Scenario
The developer makes code changes and commits them.
A pull request (PR) is raised with links to Databricks job runs.
Unravel CI/CD retrieves relevant data and renders the following actionable real-time insight for performance accuracy. In this scenario, it highlights the bottlenecks in the application.
Unravel-Bot (Unravel account) points out that a significant portion of the application's execution time (63%) is spent on the driver component, potentially affecting overall efficiency.
The developer reviewing the PR gains real-time actionable recommendations to address the identified issue. Unravel-Bot (Unravel account) suggests revisiting the application code to investigate why the driver component consumes much processing time.
Impact
Unravel CI/CD's pipeline accuracy insight identifies performance bottlenecks, such as excessive time spent on the driver component, contributing to enhanced code quality and adherence to pipeline standards.
By identifying bottlenecks and providing actionable recommendations, Unravel CI/CD helps developers optimize code, which results in more efficient application delivery. This translates to performance improvement and an improved user experience.
Improved pipeline accuracy ensures that code changes align with standards, which reduces the risk of post-deployment issues and conflicts. This promotes a culture of quality within the development team and enhances code reliability.