Others

How to Trigger an Azure Data Factory Pipeline from Azure ML SDK

Kushani Nimanthika Oct 16, 2026 869 views

Question details

The user needs to initiate an Azure Data Factory (ADF) pipeline programmatically using the Azure Machine Learning Python SDK to handle ETL processes.

How to Trigger an Azure Data Factory Pipeline from the Azure Machine Learning SDK

Product: Azure Machine Learning SDK / Azure Data Factory
Device & OS: not provided
Scenario: Running an ETL process within an automated machine learning workflow.
Observed behavior: Needs to start an ADF pipeline run from a Python script or notebook within an Azure Machine Learning workspace.

Before you start

Ensure you have the azure-mgmt-datafactory and azure-identity Python packages installed in your Azure ML environment, and that your managed identity has Data Factory Contributor permissions.

Solution 1Recommended

Use DataFactoryManagementClient in Python SDK

The standard way to trigger an ADF pipeline from any Python environment, including Azure ML, is by utilizing the Azure Data Factory Management SDK.

By authenticating through Azure Identity and using the DataFactoryManagementClient, you can programmatically trigger, monitor, and pass parameters to your ETL pipelines directly from an Azure Machine Learning notebook or script.

Install required Azure SDK packages

Run 'pip install azure-mgmt-datafactory azure-identity' in your Azure ML notebook or pipeline script to ensure the required management libraries are available.

Import libraries and authenticate

Import 'DataFactoryManagementClient' and 'DefaultAzureCredential'. Initialize the credential object to automatically authenticate using your Azure ML workspace's managed identity.

Initialize the Data Factory Client

Create an instance of 'DataFactoryManagementClient' using your Azure subscription ID and the credential object you initialized in the previous step.

Trigger the pipeline run

Call the 'pipelines.create_run' method on your client object, passing the 'resource_group_name', 'factory_name', and 'pipeline_name'. You can also pass a dictionary of pipeline parameters if your ETL process requires dynamic inputs.

Use DataFactoryManagementClient in Python SDK

Microsoft Q&A Resource: For highly customized or complex Azure integration workflows, consulting the official Microsoft Q&A forums or Azure ML SDK documentation is highly recommended.

Free Microsoft Office alternative

Document Your Azure Workflows and Analyze Data with WPS Office

While Azure handles your heavy ETL and machine learning pipelines, you need a reliable, lightweight suite to document your architecture and analyze exported datasets. WPS Office provides a free, highly compatible alternative to Microsoft Office for all your reporting needs.

1. Download and install WPS Office: Visit the official WPS Office website to download the free suite for your operating system.
2. Open your exported Azure data: Launch WPS Spreadsheet and open the .csv or .xlsx files generated by your ETL pipelines.
3. Analyze and visualize your results: Use built-in pivot tables and charts to summarize your machine learning datasets effortlessly.

Fully compatible with Microsoft Excel (.xlsx), Word (.docx), and PowerPoint (.pptx) formats.Easily open and analyze large CSV data exports generated by Azure Data Factory.Lightweight installation with a familiar, tabbed user interface.Free built-in PDF tools for sharing machine learning architecture reports.

Frequently Asked Questions

Can I trigger an Azure Data Factory pipeline via REST API instead of the Python SDK?

Yes, you can use the Azure REST API to trigger a pipeline run by sending a POST request to the pipeline's 'createRun' endpoint, authenticated with an Azure AD bearer token.

How do I pass parameters from Azure ML to the Data Factory pipeline?

When calling the pipelines.create_run method in the Python SDK, you can pass a dictionary of parameters to the parameters argument. These will map directly to the parameters defined in your ADF pipeline.

What permissions are required to run an ADF pipeline from Azure ML?

The managed identity of the Azure ML workspace (or the service principal used for authentication) must be assigned the 'Data Factory Contributor' role, or a custom role with the Microsoft.DataFactory/factories/pipelines/createRun/action permission.

Office Quick Solutions