Home› Platform Support› Azure Data Factory

Azure Data Factory

AnalyticsCreator generates Azure Data Factory pipelines and supporting orchestration artefacts from the metadata-driven data warehouse design. The generated implementation is deployed to the customer's Azure environment and executed by Azure Data Factory without requiring an AnalyticsCreator production runtime.

Overview

AnalyticsCreator supports Azure Data Factory as an orchestration technology for metadata-driven data warehouse implementations.

Source mappings, transformations, historisation rules, processing steps and dependencies are defined in the AnalyticsCreator project. AnalyticsCreator uses this metadata to generate the corresponding Azure Data Factory orchestration artefacts.

AnalyticsCreator is a design-time application. Azure Data Factory executes the deployed pipelines, while database processing is performed by the relevant source or target technology.

What AnalyticsCreator Generates

Depending on the project design and deployment configuration, AnalyticsCreator can generate Azure Data Factory artefacts including:

  • pipelines
  • datasets
  • linked service definitions
  • source ingestion workflows
  • copy operations
  • execution of generated database processing logic
  • historisation and persistence workflows
  • dependency-based orchestration
  • coordination of multiple processing stages

The generated orchestration reflects the processing dependencies maintained in the AnalyticsCreator project. Teams therefore do not need to recreate the complete workflow manually in Azure Data Factory.

ARM Template Generation

AnalyticsCreator can package the generated Azure Data Factory implementation as an Azure Resource Manager (ARM) template as part of the deployment output.

The generated ARM template can be deployed to the target Data Factory to create the configured Azure Data Factory resources. Depending on the project, these resources can include pipelines, datasets and linked service definitions required by the generated workflow.

Generation of the ARM template and configuration of the target Azure environment are separate activities. Environment-specific connections, credentials, Integration Runtimes and permissions must still be configured or validated for the target environment.

Pipeline Generation

AnalyticsCreator generates pipelines that coordinate the processing steps required by the data warehouse design.

Generated workflows can use supported Azure Data Factory activities to perform tasks such as:

  • copying data from source systems
  • executing generated stored procedures
  • performing lookups required by the workflow
  • executing dependent pipelines
  • coordinating downstream processing stages

The exact pipeline structure depends on the source systems, mappings, transformations, historisation rules and processing dependencies defined in the project.

Datasets and Linked Services

AnalyticsCreator can generate the Azure Data Factory definitions required to connect generated pipelines with the datasets and services used by the solution.

Linked service definitions describe the connection required by the generated workflow, but environment-specific connection values and security configuration remain part of the Azure environment.

Depending on the implementation, this can include configuration for:

  • source and target server names
  • database names
  • authentication and credentials
  • Integration Runtime assignment
  • network connectivity
  • environment-specific parameters

AnalyticsCreator does not replace the identity, security or network configuration managed by Azure.

Integration Runtimes

Azure Data Factory uses Integration Runtimes to provide the compute and connectivity required for data movement and activity execution.

The appropriate Integration Runtime depends on the systems involved. For example, a self-hosted Integration Runtime can be required to access an on-premises source, while Azure-hosted resources can use an Azure Integration Runtime where appropriate.

After deployment, generated linked services must reference the correct Integration Runtime for the target environment. This binding is part of the Azure environment configuration rather than the AnalyticsCreator data warehouse design.

Orchestration Model

Azure Data Factory provides orchestration for the generated solution. The underlying processing logic is executed by the appropriate source or target technology.

A generated workflow can coordinate activities such as:

  • source extraction
  • data import
  • warehouse loading
  • transformation processing
  • historisation where configured
  • persistence where configured
  • downstream processing

Dependencies defined in the AnalyticsCreator project determine the required processing sequence.

These stages describe possible generated workflow components. They do not require every AnalyticsCreator project to use the same warehouse-layer topology.

Generation, Deployment and Execution

AnalyticsCreator separates design and generation from deployment and production execution.

  1. Design. Source mappings, transformations, processing rules and dependencies are maintained in AnalyticsCreator.
  2. Generate. AnalyticsCreator generates the configured Azure Data Factory artefacts and deployment output.
  3. Deploy. The generated ARM template or other configured deployment artefacts are applied to the target Azure environment.
  4. Configure. Environment-specific linked-service settings, Integration Runtimes, credentials and permissions are applied or validated.
  5. Execute. Azure Data Factory executes the deployed pipelines and coordinates the required source and target processing.

Generating or deploying a pipeline does not itself process business data. Data movement and processing occur when the deployed workflow is executed.

Typical Generated Workflow

A warehouse implementation can use a generated Data Factory workflow to coordinate several processing stages.

For example, a workflow can:

  1. start the required source import pipelines
  2. copy source data to the configured warehouse structures
  3. execute historisation where it is enabled
  4. execute generated transformation or persistence logic
  5. start dependent downstream processes in the required sequence

The actual workflow is determined by the AnalyticsCreator project. The example above is a common generated pattern, not a mandatory architecture.

Changes and Regeneration

When AnalyticsCreator-owned definitions change, the corresponding Azure Data Factory orchestration can be regenerated from the updated metadata.

Teams should review the resulting changes before deployment, particularly where the target Data Factory also contains manually maintained resources or environment-specific configuration.

Generated orchestration should normally be changed through the AnalyticsCreator design where AnalyticsCreator owns that definition. This helps keep subsequent generation consistent with the maintained project model.

CI/CD and Version Control

AnalyticsCreator project definitions and generated Azure Data Factory deployment artefacts can be included in controlled source-control and CI/CD processes.

Teams can use this approach to:

  • version AnalyticsCreator project definitions
  • review changes before deployment
  • include generated Data Factory artefacts in deployment workflows
  • manage development, test and production environments separately
  • regenerate orchestration when the underlying model changes

The exact CI/CD implementation depends on the customer's source-control system, Azure environment and deployment tooling.

Source control, ARM template generation, Azure deployment and pipeline execution remain separate steps.

Connectors and Source Systems

Azure Data Factory orchestration can be generated for workflows involving source systems supported by the AnalyticsCreator project and the selected deployment architecture.

The source connection itself is defined through the relevant AnalyticsCreator connector and Azure Data Factory configuration. Refer to the AnalyticsCreator connector documentation for supported source types and their individual requirements.

Prerequisites and Considerations

  • An Azure Data Factory environment and the required Azure permissions must be available.
  • The required source and target systems must be reachable from the selected Integration Runtime.
  • Linked services must be configured with the appropriate environment-specific connection information.
  • Authentication, credentials and network access remain part of the Azure environment configuration.
  • Generated pipelines depend on the capabilities of Azure Data Factory and the connected source and target technologies.
  • Generated deployment artefacts should be reviewed before promotion between environments.
  • Pipeline execution and resulting data should be validated after deployment.

Example Use Cases

  • Generate Azure Data Factory pipelines for source ingestion.
  • Orchestrate generated data warehouse loading and processing.
  • Coordinate historisation and persistence workflows.
  • Generate dependency-aware processing sequences from AnalyticsCreator metadata.
  • Reduce repetitive manual construction of Data Factory pipelines.
  • Include generated ARM templates and Data Factory artefacts in controlled deployment processes.

Related Content

  • Microsoft Azure
  • Connectors
  • Deployment
  • SQL Server
  • Microsoft Fabric
  • Power BI

Key Takeaway

AnalyticsCreator generates Azure Data Factory orchestration from the maintained data warehouse design and can package the generated resources as an ARM template for deployment. Azure Data Factory then executes the deployed workflows in the customer's Azure environment, while environment-specific connectivity, security and Integration Runtime configuration remain under Azure control.