Overview
Information in this article is related to using CloudMonix for monitoring Azure Data Factories. The article covers the following topics:
common use cases where CloudMonix can help with monitoring
what is needed to connect to and monitor Azure Data Factory
-
what metrics can be tracked, visualized and monitored
Why use CloudMonix for Azure Data Factory?
Popular usages of CloudMonix include the following examples:
Tracking Azure Data Factory activities, pipelines, gateways and datasets
Alerting on failed activities
Configuration
CloudMonix will use Azure Monitoring API for monitoring Azure Data Factories.
Azure Data Factory monitoring can be configured either via Setup Wizard or by using the “Add New” button in the dashboard. It’s highly recommended to use Setup Wizard when configuring permissions for the first time, as that will simplify authorization. Learn more about authorizing with Setup Wizard here.
Metrics
CloudMonix provides a default template for monitoring Azure Data Factories that contains most useful Metrics and Alerts.
The metrics can be added, removed and customized in the Metrics tab in the resource configuration dialog.
Built-in Metrics
DataFactoryAllActivities
Tracks data from all actions in all pipelines.
-
Data Type: array of objects with the following properties:
ActivityName (string): Pipeline activity name.
PipelineName (string): Pipeline name.
Status (string): Window state.
LastRunTime (DateTime): Start time of the last run, or 01/01/0001.
LastEndTime (DateTime): End time of the last run, or 01/01/0001.
Duration (double): Duration of the last run in minutes, or 0.
PercentComplete (int): Percent completion of activity window execution.
RunAttempts (int): Number of activity run attempts.
WindowStart (DateTime): Start time of the activity window.
WindowEnd (DateTime): End time of the activity window.
Included in sample profile: yes, tracked as a metric called Activities
Included in default alerts: no
DataFactoryTotalActivityCount
Tracks count of activities in Data Factory.
Data Type: int
Included in sample profile: yes, tracked as a metric called ActivityCount
Included in default alerts: no
DataFactoryFailedActivityCount
Tracks count of failed activities in Data Factory.
Data Type: int
Included in sample profile: yes, tracked as a metric called FailedActivityCount
Included in default alerts: yes, in an alert Failed Activities Detected (Warning): Triggered whenever failed activities count is larger than zero.
DataFactoryDatasets
Tracks data from all datasets in data factory.
-
Data Type: array of objects with the following properties:
Name (string)
LatestError (string)
Included in sample profile: yes, tracked as a metric called Datasets
Included in default alerts: no
DataFactoryLinkedServices
Tracks data from all Linked Services in data factory.
-
Data Type: array of objects with the following properties:
Name (string)
LatestError (string)
Included in sample profile: yes, tracked as a metric called LinkedServices
Included in default alerts: no
DataFactoryGateways
Tracks data from all gateways in data factory.
-
Data Type: array of objects with the following properties:
Name (string): Gateway name.
LastConnectTime (DateTime): Last connect time of the data factory gateway.
RegisterTime (DateTime): Register time of the data factory gateway.
Status (string): Gateway status, possible values: NeedRegistration, Online, Offline.
Included in sample profile: yes, tracked as a metric called Gateways
Included in default alerts: no
DataFactoryPipelines
Tracks data from all pipelines in data factory.
-
Data Type: array of objects with the following properties:
Name (string): Pipeline name.
LastError (string): Error in processing pipeline request.
StartTime (DateTime): The start time of the pipeline, or 01/01/0001.
EndTime (DateTime): The end time of the pipeline, or 01/01/0001.
IsPaused (bool): Indicates if the pipeline is paused.
PipelineState (string): State of the pipeline, possible values: Completed, Deleting, Inactive, Initializing, NotSpecified, Paused, Running, Updating.
DeploymentTime (DateTime): The deployment time of the pipeline.
Included in sample profile: yes, tracked as a metric called Pipelines
Included in default alerts: no
DataFactoryPipelinesCount
Tracks total pipelines count.
Data Type: int
Included in sample profile: yes, tracked as a metric called PipelineCount
-
Included in default alerts: no
DataFactoryPipelineActivities
Tracks state of activities in one pipeline.
-
Data Type: array of objects with the following properties:
ActivityName (string): Pipeline activity name.
PipelineName (string): Pipeline name.
Status (string): Window state.
LastRunTime (DateTime): Start time of the last run, or 01/01/0001.
LastEndTime (DateTime): End time of the last run, or 01/01/0001.
Duration (double): Duration of the last run in minutes, or 0.
PercentComplete (int): Percent completion of activity window execution.
RunAttempts (int): Number of activity run attempts.
WindowStart (DateTime): Start time of the activity window.
WindowEnd (DateTime): End time of the activity window.
Requires specifying a pipeline.
Included in sample profile: no
Included in default alerts: no
DataFactoryLastActivityDurationInMin
Tracks work's duration of one activity.
Data Type: double
Requires specifying a pipeline and activity
Included in sample profile: no
Included in default alerts: no
DataFactoryLastActivityStartDate
Tracks start date of one activity.
Data Type: DateTime
Requires specifying a pipeline and activity
Included in sample profile: no
Included in default alerts: no
DataFactoryLastActivityEndDate
Tracks end date of one activity.
Data Type: DateTime
Requires specifying a pipeline and activity
Included in sample profile: no
Included in default alerts: no
DataFactoryLastActivityStatus
Tracks status of one activity.
Data Type: string
Requires specifying a pipeline and activity
Included in sample profile: no
Included in default alerts: no
Alerts
Users can create alerts based on changes in any value tracked by CloudMonix (including custom metrics). Each resource template includes alerts which are suitable for a given resource. The predefined alerts for Azure Data Factories are listed in the Metrics section. Refer to the Alerts article to learn more about alerts in general.
Alerts are available during the Trial period or in Professional and Ultimate plans only.