Microsoft Azure Data Factory is a service that allows to automate and orchestrate data retrieval and publish the results.
This step-by-step guide explains how to setup and monitor Azure Data Factory using CloudMonix.
In this article
1. Monitoring setup
2. Collect, understand and use your data
2.1 Metrics
2.2 Alerts
2.3 Actions
3. Setup verification and troubleshooting
Did you know?
CloudMonix extends native Azure Data Factory monitoring with advanced metrics and features. Default metrics and features may vary depending on the Data Factory version. Metrics and features available for the Data Factory V2 template are provided below. Noteworthy:
- CloudMonix receives data from the Azure Management API and Azure Monitor API
- pre-configured metrics: all activities, activity count, failed activity count, integration runtimes, pipelines count, pipelines, recommended actions, trigger failed runs count, triggers
- alerts on failed activities detected and failed trigger runs detected
- ability to start / stop an integration runtime and to start / stop a trigger in Azure Data Factory
Setup
a. Run the Setup Wizard in the portal (preferred way):
This article explains how to add resources to CloudMonix via the Setup Wizard.
b. Tweak settings in the Definition tab (optional):
Definition tab for an existing resource can be accessed by clicking the resource's monitoring settings in the performance dashboard:
Definition tab provides optional settings for the resource name, Azure resource management token, Azure resource group, Azure resource name, Data Factory version, configuration template and categories:
Best Practices
Configuration Template setting provides pre-defined configuration templates available in CloudMonix by default as well as previously stored custom templates. See predefined templates for Azure Data Factory for reference.
c. Manual setup (optional instead of the Setup Wizard route):
Click the Add New button in the top right corner of your dashboard:
Fill in required information in the Definition tab as described in the previous step.
d. Advanced configuration:
Advanced configuration tab provides additional monitoring settings, which are already set as default for most use-cases.
Collect, understand and use your data
Specific Metrics, Templates, Alerts and Automation Actions for Azure Data Factory:
a. Metrics:
Diagnostic data points retrieved from the monitored resource are referred to as metrics. CloudMonix provides default templates for the metrics recommended for common configurations. Metrics can be further added, removed or customized in the Metrics tab of the Azure Data Factory resource configuration dialog:
b. Alerts:
CloudMonix features a sophisticated alert engine that allows alerts to be published for very particular conditions pre-defined by a template configuration or custom based on any of the available metrics. Alerts can be further added, removed or customized in the Alerts tab of the Azure Data Factory resource configuration dialog:
c. Actions:
Actions are automation features that can be configured to fire based on specific conditions or schedule. Actions can be added and configured in the Actions tab of the Azure Data Factory resource configuration dialog:
Available actions include the ability start / stop an integration runtime and to start / stop a trigger in Azure Data Factory resource based on conditions or schedule and execute custom WebRequest to a specified URL.
Setup verification and troubleshooting
a. Setup verification:
Successful resource setup can be verified by clicking Test button in the resource configuration dialog and visiting the Test Results tab:
b. Troubleshooting monitoring issues:
CloudMonix provides deep insights into resource monitoring issues via the Status Dashboard screen. The screen allows to overview resources that have raised alerts and troubleshoot them by diving into the monitoring logs.
Read the full article on how to use Status Dashboard to diagnose resource monitoring issues.