Overview

Information in this article is related to using CloudMonix for monitoring Azure Data Factories. The article covers the following topics:

  • common use cases where CloudMonix can help with monitoring

  • what is needed to connect to and monitor Azure Data Factory

  • what metrics can be tracked, visualized and monitored


Why use CloudMonix for Azure Data Factory?

Popular usages of CloudMonix include the following examples:

  • Tracking Azure Data Factory activities, pipelines, gateways and datasets

  • Alerting on failed activities


Configuration 

CloudMonix will use Azure Monitoring API for monitoring Azure Data Factories.


Azure Data Factory monitoring can be configured either via Setup Wizard or by using the “Add New” button in the dashboard. It’s highly recommended to use Setup Wizard when configuring permissions for the first time, as that will simplify authorization. Learn more about authorizing with Setup Wizard here.




Metrics


CloudMonix provides a default template for monitoring Azure Data Factories that contains most useful Metrics and Alerts.



The metrics can be added, removed and customized in the Metrics tab in the resource configuration dialog.


Built-in Metrics 

DataFactoryAllActivities

Tracks data from all actions in all pipelines.

  • Data Type: array of objects with the following properties:

    • ActivityName (string): Pipeline activity name.

    • PipelineName (string): Pipeline name.

    • Status (string): Window state.

    • LastRunTime (DateTime): Start time of the last run, or 01/01/0001.

    • LastEndTime (DateTime): End time of the last run, or 01/01/0001.

    • Duration (double): Duration of the last run in minutes, or 0.

    • PercentComplete (int): Percent completion of activity window execution.

    • RunAttempts (int): Number of activity run attempts.

    • WindowStart (DateTime): Start time of the activity window.

    • WindowEnd (DateTime): End time of the activity window.

  • Included in sample profile: yes, tracked as a metric called Activities

  • Included in default alerts: no


DataFactoryTotalActivityCount

Tracks count of activities in Data Factory.

  • Data Type: int

  • Included in sample profile: yes, tracked as a metric called ActivityCount

  • Included in default alerts: no


DataFactoryFailedActivityCount

Tracks count of failed activities in Data Factory.

  • Data Type: int

  • Included in sample profile: yes, tracked as a metric called FailedActivityCount

  • Included in default alerts: yes, in an alert Failed Activities Detected (Warning): Triggered whenever failed activities count is larger than zero.


DataFactoryDatasets

Tracks data from all datasets in data factory.

  • Data Type: array of objects with the following properties:

    • Name (string)

    • LatestError (string)

  • Included in sample profile: yes, tracked as a metric called Datasets

  • Included in default alerts: no


DataFactoryLinkedServices

Tracks data from all Linked Services in data factory.

  • Data Type: array of objects with the following properties:

    • Name (string)

    • LatestError (string)

  • Included in sample profile: yes, tracked as a metric called LinkedServices

  • Included in default alerts: no


DataFactoryGateways

Tracks data from all gateways in data factory.

  • Data Type: array of objects with the following properties:

    • Name (string): Gateway name.

    • LastConnectTime (DateTime): Last connect time of the data factory gateway.

    • RegisterTime (DateTime): Register time of the data factory gateway.

    • Status (string): Gateway status, possible values: NeedRegistration, Online, Offline.

  • Included in sample profile: yes, tracked as a metric called Gateways

  • Included in default alerts: no


DataFactoryPipelines

Tracks data from all pipelines in data factory.

  • Data Type: array of objects with the following properties:

    • Name (string): Pipeline name.

    • LastError (string): Error in processing pipeline request.

    • StartTime (DateTime): The start time of the pipeline, or 01/01/0001.

    • EndTime (DateTime): The end time of the pipeline, or 01/01/0001.

    • IsPaused (bool): Indicates if the pipeline is paused.

    • PipelineState (string): State of the pipeline, possible values: Completed, Deleting, Inactive, Initializing, NotSpecified, Paused, Running, Updating.

    • DeploymentTime (DateTime): The deployment time of the pipeline.

  • Included in sample profile: yes, tracked as a metric called Pipelines

  • Included in default alerts: no


DataFactoryPipelinesCount

Tracks total pipelines count.

  • Data Type: int

  • Included in sample profile: yes, tracked as a metric called PipelineCount

  • Included in default alerts: no


DataFactoryPipelineActivities

Tracks state of activities in one pipeline.

  • Data Type: array of objects with the following properties:

    • ActivityName (string): Pipeline activity name.

    • PipelineName (string): Pipeline name.

    • Status (string): Window state.

    • LastRunTime (DateTime): Start time of the last run, or 01/01/0001.

    • LastEndTime (DateTime): End time of the last run, or 01/01/0001.

    • Duration (double): Duration of the last run in minutes, or 0.

    • PercentComplete (int): Percent completion of activity window execution.

    • RunAttempts (int): Number of activity run attempts.

    • WindowStart (DateTime): Start time of the activity window.

    • WindowEnd (DateTime): End time of the activity window.

  • Requires specifying a pipeline.

  • Included in sample profile: no

  • Included in default alerts: no


DataFactoryLastActivityDurationInMin

Tracks work's duration of one activity.

  • Data Type: double

  • Requires specifying a pipeline and activity

  • Included in sample profile: no

  • Included in default alerts: no


DataFactoryLastActivityStartDate

Tracks start date of one activity.

  • Data Type: DateTime

  • Requires specifying a pipeline and activity

  • Included in sample profile: no

  • Included in default alerts: no


DataFactoryLastActivityEndDate

Tracks end date of one activity.

  • Data Type: DateTime

  • Requires specifying a pipeline and activity

  • Included in sample profile: no

  • Included in default alerts: no


DataFactoryLastActivityStatus

Tracks status of one activity.

  • Data Type: string

  • Requires specifying a pipeline and activity

  • Included in sample profile: no

  • Included in default alerts: no


Alerts

Users can create alerts based on changes in any value tracked by CloudMonix (including custom metrics). Each resource template includes alerts which are suitable for a given resource. The predefined alerts for Azure Data Factories are listed in the Metrics section. Refer to the Alerts article to learn more about alerts in general.


Alerts are available during the Trial period or in Professional and Ultimate plans only.