Terragrunt is an orchestrator for Terraform/OpenTofu and is designed for scalable deployments across multiple environments (e.g. DEV, TEST, UAT, PROD)

The three patterns we’ll explore:

Pattern 1: Simple Input Mapping

Example scenario: You’re managing platform resources across multiple business units, and each team wants to maintain their own configuration without stepping on each other’s toes.

This is the pattern I typically recommend when you’re starting your Terragrunt journey or when you have simple data that needs to be aggregated from multiple sources.

The traditional approach would be one massive configuration file that becomes a bottleneck for changes. Not ideal!

Here’s how the simple input mapping pattern might solve this problem. This approach allows different teams to maintain their own configuration files independently. Each business unit can modify their settings without creating merge conflicts or requiring coordination with other teams.

For context I used this pattern to manage GitHub repositories and teams across 9 business units with approximately 150+ repositories and 50+ teams.

First we define each source environment file separately to pull in all HCL data from them into our Terragrunt deployment.

locals {
  # Read each file separately
  business_unit1_env = read_terragrunt_config("business-unit1-env.hcl")
  business_unit2_env = read_terragrunt_config("business-unit2-env.hcl")
  business_unit3_env = read_terragrunt_config("business-unit3-env.hcl")
  business_unit4_env = read_terragrunt_config("business-unit4-env.hcl")
  business_unit5_env = read_terragrunt_config("business-unit5-env.hcl")
  business_unit6_env = read_terragrunt_config("business-unit6-env.hcl")
  business_unit7_env = read_terragrunt_config("business-unit7-env.hcl")
  business_unit8_env = read_terragrunt_config("business-unit8-env.hcl")
  business_unit9_env = read_terragrunt_config("business-unit9-env.hcl")
}

read_terragrunt_config() as shown above allows each team to manage their own config file. The Business Unit 2 team might manage business-unit2-env.hcl without affecting other team configurations.

Here’s where we combine the data from the multiple sources above while handling missing or malformed files gracefully. The concat() function safely combines lists from multiple sources. The try() function here can be essential for handling missing config files that might cause deployment failures.

# Concatenate the lists from all files into the single variable
repositories = concat( #concat used for combining lists/arrays. Use merge for combining maps/objects.
  try(local.business_unit1_env.locals.repositories, []),
  try(local.business_unit2_env.locals.repositories, []),
  try(local.business_unit3_env.locals.repositories, []),
  try(local.business_unit4_env.locals.repositories, []),
  try(local.business_unit5_env.locals.repositories, []),
  try(local.business_unit6_env.locals.repositories, []),
  try(local.business_unit7_env.locals.repositories, []),
  try(local.business_unit8_env.locals.repositories, []),
  try(local.business_unit9_env.locals.repositories, [])
)

# Concatenate the lists from all files into the single variable
teams = concat( #concat used for combining lists/arrays. Use merge for combining maps/objects.
  try(local.business_unit1_env.locals.teams, []),
  try(local.business_unit2_env.locals.teams, []),
  try(local.business_unit3_env.locals.teams, []),
  try(local.business_unit4_env.locals.teams, []),
  try(local.business_unit5_env.locals.teams, []),
  try(local.business_unit6_env.locals.teams, []),
  try(local.business_unit7_env.locals.teams, []),
  try(local.business_unit8_env.locals.teams, []),
  try(local.business_unit9_env.locals.teams, [])
)

The final step is straightforward - we pass our aggregated and processed data as inputs to the Terraform module. This keeps the interface clean and predictable.

inputs = {
  # Map module variables to local variables
  repositories = local.repositories
  teams        = local.teams
}

Pattern 1: Complete Example

# terragrunt.hcl - GitHub Platform Onboarding
# Pattern 1: Simple Input Mapping

# Include the root configuration
include {
  path = find_in_parent_folders("root-terragrunt.hcl")
}

# Set Terraform stack source location
terraform {
  source = "${get_repo_root()}/stacks/onboarding"
}

locals {
  # Read each file separately
  business_unit1_env = read_terragrunt_config("business-unit1-env.hcl")
  business_unit2_env = read_terragrunt_config("business-unit2-env.hcl")
  business_unit3_env = read_terragrunt_config("business-unit3-env.hcl")
  business_unit4_env = read_terragrunt_config("business-unit4-env.hcl")
  business_unit5_env = read_terragrunt_config("business-unit5-env.hcl")
  business_unit6_env = read_terragrunt_config("business-unit6-env.hcl")
  business_unit7_env = read_terragrunt_config("business-unit7-env.hcl")
  business_unit8_env = read_terragrunt_config("business-unit8-env.hcl")
  business_unit9_env = read_terragrunt_config("business-unit9-env.hcl")

  # Concatenate the lists from all files into the single variable
  repositories = concat( #concat used for combining lists/arrays. Use merge for combining maps/objects.
    try(local.business_unit1_env.locals.repositories, []),
    try(local.business_unit2_env.locals.repositories, []),
    try(local.business_unit3_env.locals.repositories, []),
    try(local.business_unit4_env.locals.repositories, []),
    try(local.business_unit5_env.locals.repositories, []),
    try(local.business_unit6_env.locals.repositories, []),
    try(local.business_unit7_env.locals.repositories, []),
    try(local.business_unit8_env.locals.repositories, []),
    try(local.business_unit9_env.locals.repositories, [])
  )

  # Concatenate the lists from all files into the single variable
  teams = concat( #concat used for combining lists/arrays. Use merge for combining maps/objects.
    try(local.business_unit1_env.locals.teams, []),
    try(local.business_unit2_env.locals.teams, []),
    try(local.business_unit3_env.locals.teams, []),
    try(local.business_unit4_env.locals.teams, []),
    try(local.business_unit5_env.locals.teams, []),
    try(local.business_unit6_env.locals.teams, []),
    try(local.business_unit7_env.locals.teams, []),
    try(local.business_unit8_env.locals.teams, []),
    try(local.business_unit9_env.locals.teams, [])
  )
}

inputs = {
  # Map module variables to local variables
  repositories = local.repositories
  teams        = local.teams
}

remote_state {
  backend = "azurerm"
  generate = {
    path      = "_backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    subscription_id      = "__TFSTATE_SUBSCRIPTION_ID__"
    tenant_id            = "__AZURE_TENANT_ID__"
    client_id            = "__AZURE_CLIENT_ID__"
    use_azuread_auth     = true
    use_oidc             = true
    resource_group_name  = "__TFSTATE_RESOURCE_GROUP_NAME__"
    storage_account_name = "__TFSTATE_STORAGE_ACCOUNT_NAME__"
    container_name       = "__TFSTATE_AZURE_PLATFORM_CONTAINER__"
    key                  = "github/terraform.tfstate"
  }
}

Pattern 2: TFVars Generation for Large Datasets

Example scenario: You’re managing enterprise infrastructure with thousands of resources and assignments, and your CI/CD workflows keep failing with “argument list too long” errors.

This pattern addresses enterprise scale implementations where CLI argument length limits become a constraint. It provides a practical approach for managing large datasets effectively. When you pass large datasets through Terragrunt inputs, they get converted to command-line arguments for the underlying Terraform execution. And there are limits!

I used this pattern to manage Azure RBAC across multiple Azure subscriptions with 2000+ role assignments, 500+ service principals, and 200+ Azure AD groups. But only after I started hitting the limits described above, and in hindsight we should have split up and managed the non-dependent resources separately instead of combining them.

First, we load configuration from multiple HCL environment inputs sources in the directory hierarchy. Using find_in_parent_folders() here is effective for certain folder hierarchies as it automatically discovers configuration files up the directory tree.

locals {
  # Automatically load environment-level variables
  rbac_groups_azure_env_vars  = read_terragrunt_config(find_in_parent_folders("rbac-groups-azure-env.hcl"))
  rbac_groups_github_env_vars = read_terragrunt_config(find_in_parent_folders("rbac-groups-github-env.hcl"))
  rbac_spns_env_vars          = read_terragrunt_config(find_in_parent_folders("rbac-spns-env.hcl"))
  platform_kv_appreg_env_vars = read_terragrunt_config(find_in_parent_folders("platform-kv-appreg-env.hcl"))

  # Map local variables to environment variables
  subscription_id                             = local.rbac_groups_azure_env_vars.locals.subscription_id
  groups                                      = local.rbac_groups_azure_env_vars.locals.groups
  groups_github                               = local.rbac_groups_github_env_vars.locals.groups_github
  spn_app_registrations                       = local.rbac_spns_env_vars.locals.spn_app_registrations
  spn_role_assignments                        = local.rbac_spns_env_vars.locals.spn_role_assignments
  managed_identity_directory_role_assignments = local.rbac_spns_env_vars.locals.managed_identity_directory_role_assignments
}

This is the key insight - we handle small and large datasets differently. Small variables use normal Terragrunt inputs, while large datasets get written to auto-loaded files to avoid CLI limits. This demonstrates the core functionality of Terragrunt’s generate blocks. Small, simple variables might use the normal inputs mechanism, while large, complex datasets could be written to .auto.tfvars.json files that Terraform automatically loads.

# Generate a tfvars file with the large variable inputs to avoid command-line length limits 
# e.g. 'argument list too long' errors in GitHub workflow runs
# Terraform will automatically load *.auto.tfvars.json files
generate "large_vars" {
  path              = "large_vars.auto.tfvars.json"
  if_exists         = "overwrite"
  disable_signature = true
  contents  = jsonencode({
    spn_app_registrations                       = local.spn_app_registrations
    spn_role_assignments                        = local.spn_role_assignments
    managed_identity_directory_role_assignments = local.managed_identity_directory_role_assignments
    # add more large variables here as needed if you are seeing 'argument list too long' errors in the GitHub workflow runs.
  })
}

inputs = {
  # Map smaller variables via inputs (these are safe for command-line)
  subscription_id      = local.subscription_id
  groups               = local.groups
  groups_github        = local.groups_github
  app_reg_key_vault_id = local.platform_kv_appreg_env_vars.locals.app_reg_key_vault_id

  # Large variables (spn_app_registrations, spn_role_assignments, managed_identity_directory_role_assignments)
  # are written to .terraform/large_vars.auto.tfvars.json by the generate 'large_vars' block above.
  # Terraform will automatically load them from the file
}

Pattern 2: Complete Example

# terragrunt.hcl - Enterprise RBAC Management
# Pattern 2: TFVars Generation for Large Datasets

# Include the root configuration
include {
  path = find_in_parent_folders("root-terragrunt.hcl")
}

# Set Terraform stack source location
terraform {
  source = "${get_repo_root()}/stacks/rbac"
}

locals {
  # Automatically load environment-level variables
  rbac_groups_azure_env_vars  = read_terragrunt_config(find_in_parent_folders("rbac-groups-azure-env.hcl"))
  rbac_groups_github_env_vars = read_terragrunt_config(find_in_parent_folders("rbac-groups-github-env.hcl"))
  rbac_spns_env_vars          = read_terragrunt_config(find_in_parent_folders("rbac-spns-env.hcl"))
  platform_kv_appreg_env_vars = read_terragrunt_config(find_in_parent_folders("platform-kv-appreg-env.hcl"))

  # Map local variables to environment variables
  subscription_id                             = local.rbac_groups_azure_env_vars.locals.subscription_id
  groups                                      = local.rbac_groups_azure_env_vars.locals.groups
  groups_github                               = local.rbac_groups_github_env_vars.locals.groups_github
  spn_app_registrations                       = local.rbac_spns_env_vars.locals.spn_app_registrations
  spn_role_assignments                        = local.rbac_spns_env_vars.locals.spn_role_assignments
  managed_identity_directory_role_assignments = local.rbac_spns_env_vars.locals.managed_identity_directory_role_assignments
}

# Generate a tfvars file with the large variable inputs to avoid command-line length limits
# e.g. 'argument list too long' errors in GitHub workflow runs
# Terraform will automatically load *.auto.tfvars.json files
generate "large_vars" {
  path              = "large_vars.auto.tfvars.json"
  if_exists         = "overwrite"
  disable_signature = true
  contents  = jsonencode({
    spn_app_registrations                       = local.spn_app_registrations
    spn_role_assignments                        = local.spn_role_assignments
    managed_identity_directory_role_assignments = local.managed_identity_directory_role_assignments
    # add more large variables here as needed if you are seeing 'argument list too long' errors in the GitHub workflow runs.
  })
}

inputs = {
  # Map smaller variables via inputs (these are safe for command-line)
  subscription_id      = local.subscription_id
  groups               = local.groups
  groups_github        = local.groups_github
  app_reg_key_vault_id = local.platform_kv_appreg_env_vars.locals.app_reg_key_vault_id

  # Large variables (spn_app_registrations, spn_role_assignments, managed_identity_directory_role_assignments)
  # are written to .terraform/large_vars.auto.tfvars.json by the generate 'large_vars' block above.
  # Terraform will automatically load them from the file
}

# Generate an Azure provider block
generate "provider" {
  path      = "_provider.tf"
  if_exists = "overwrite" #overwriting root terragrunt file
  contents  = <<EOF
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">=4.41.0"
    }
    azuread = {
      source  = "hashicorp/azuread"
      version = "3.5.0"
    }
  }
}

provider "azurerm" {
  use_oidc                        = true
  resource_provider_registrations = "extended"
  storage_use_azuread             = true
  subscription_id                 = var.subscription_id
  features {}
}

provider "azuread" {
  tenant_id = "__AZURE_TENANT_ID__"
}

EOF
}

Pattern 3: Advanced Input Aggregation

Example scenario: You’re managing complex infrastructure configurations across multiple applications and services, each with different requirements and schemas.

This represents the most advanced pattern for managing complex, multi-service architectures. While this pattern requires careful consideration, it becomes essential for maintaining organised configurations at scale.

I used this pattern to manage an Azure Application Gateway with 50+ backend pools, 30+ SSL certificates, 100+ routing rules across multiple applications, plus Azure Front Door with 20+ custom domains and 40+ origins

First we explicitly define which configuration files to process.

locals {
  app_gateway_base = read_terragrunt_config("app-gateway-env.hcl") # Contains all the base level configuration for App Gateway
  frontdoor_base   = read_terragrunt_config("frontdoor-env.hcl")   # Contains all the base level configuration for Front Door

  # List of config files that have various app specific configs, such as Backend pools, HTTP listeners, SSL certs, etc. that need to be merged
  app_gateway_config_files = [
    "app-gateway-env.hcl",           // Base App Gateway Config
    "app-gateway-application1.hcl",  // Application1 specific config
    "app-gateway-application2.hcl",  // Application2 specific config
    "app-gateway-application3.hcl",  // Application3 specific config
    // Add other config files here as they are added
  ]

  # List of config files that have various frontdoor specific configs, such as Origins, Routes, Custom Domains, etc. that need to be merged
  frontdoor_config_files = [
    "frontdoor-env.hcl",           // Base Front Door Config
    "frontdoor-application1.hcl",  // Application1 specific config
    "frontdoor-application3.hcl",  // Application3 specific config
    // Add other config files here as they are added
  ]

  # Read in all the app gateway and frontdoor config files
  app_gateway_configs = [for file in local.app_gateway_config_files : read_terragrunt_config(file)] # Read in all the app gateway config files
  frontdoor_configs   = [for file in local.frontdoor_config_files : read_terragrunt_config(file)]   # Read in all the frontdoor config files
}

Rather than reading each file individually, we use for expressions to process all configuration files in a single operation. This reduces repetitive code and makes the pattern more maintainable. Using for expressions allows you to process all configuration files efficiently rather than manually reading each one individually.

# Read in all the app gateway and frontdoor config files
app_gateway_configs = [for file in local.app_gateway_config_files : read_terragrunt_config(file)]
frontdoor_configs   = [for file in local.frontdoor_config_files : read_terragrunt_config(file)]

This is where the real complexity lies - combining data from multiple sources while respecting data types and handling missing values. Each data type requires a specific approach.

Different data types require specific merging approaches:

• Lists get flattened with flatten()
• Maps get merged with merge() and the spread operator
• Everything is wrapped with try() for safe access

inputs = {
  # Map application gateway module variables to local variables
  subscription_id                       = local.app_gateway_base.locals.subscription_id
  name                                  = local.app_gateway_base.locals.name
  location                              = local.app_gateway_base.locals.location
  resource_group_name                   = local.app_gateway_base.locals.resource_group_name
  tags                                  = local.app_gateway_base.locals.tags
  
  # Complex merging strategies for different data types
  per_app_waf_policies                  = merge([for config in local.app_gateway_configs : try(config.locals.waf_policies, {})]...)
  backend_address_pools                 = flatten([for config in local.app_gateway_configs : try(config.locals.backend_address_pools, [])])
  backend_http_settings                 = flatten([for config in local.app_gateway_configs : try(config.locals.backend_http_settings, [])])
  ssl_certificates                      = flatten([for config in local.app_gateway_configs : try(config.locals.ssl_certificates, [])])
  http_listeners                        = flatten([for config in local.app_gateway_configs : try(config.locals.http_listeners, [])])
  request_routing_rules                 = flatten([for config in local.app_gateway_configs : try(config.locals.request_routing_rules, [])])
  
  # Map frontdoor module variables to local variables  
  frontdoor_custom_domains    = merge([for config in local.frontdoor_configs : try(config.locals.frontdoor_custom_domains, {})]...)
  origins                     = merge([for config in local.frontdoor_configs : try(config.locals.origins, {})]...)
  origin_group_configuration  = merge([for config in local.frontdoor_configs : try(config.locals.origin_group_configuration, {})]...)
  routes                      = merge([for config in local.frontdoor_configs : try(config.locals.routes, {})]...)
}

Some key callouts:

• Data type compatibility: Using concat() on maps or merge() on lists will cause errors
• Spread operator usage: merge([map1, map2]...) flattens the list of maps before merging

This pattern is appropriate when:

• Multiple services have different configuration schemas
• Adding new services requires touching multiple configuration areas
• Team members frequently need to locate specific configuration settings
• Configuration management overhead impacts development productivity

Pattern 3: Complete Example

# terragrunt.hcl - Application Connectivity (App Gateway + Front Door)
# Pattern 3: Advanced Input Aggregation

# Include the root configuration
include {
  path = find_in_parent_folders("root-terragrunt.hcl")
}

# Set Terraform stack source location
terraform {
  source = "${get_repo_root()}/stacks/app-connectivity"
}

locals {
  app_gateway_base = read_terragrunt_config("app-gateway-env.hcl") # Contains all the base level configuration for App Gateway
  frontdoor_base   = read_terragrunt_config("frontdoor-env.hcl")   # Contains all the base level configuration for Front Door

  # List of config files that have various app specific configs, such as Backend pools, HTTP listeners, SSL certs, etc. that need to be merged
  app_gateway_config_files = [
    "app-gateway-env.hcl",           // Base App Gateway Config
    "app-gateway-application1.hcl",  // Application1 specific config
    "app-gateway-application2.hcl",  // Application2 specific config
    "app-gateway-application3.hcl",  // Application3 specific config
    // Add other config files here as they are added
  ]

  # List of config files that have various frontdoor specific configs, such as Origins, Routes, Custom Domains, etc. that need to be merged
  frontdoor_config_files = [
    "frontdoor-env.hcl",           // Base Front Door Config
    "frontdoor-application1.hcl",  // Application1 specific config
    "frontdoor-application3.hcl",  // Application3 specific config
    // Add other config files here as they are added
  ]

  # Read in all the app gateway and frontdoor config files
  app_gateway_configs = [for file in local.app_gateway_config_files : read_terragrunt_config(file)] # Read in all the app gateway config files
  frontdoor_configs   = [for file in local.frontdoor_config_files : read_terragrunt_config(file)]   # Read in all the frontdoor config files
}

inputs = {
  # Map application gateway module variables to local variables
  subscription_id                       = local.app_gateway_base.locals.subscription_id
  name                                  = local.app_gateway_base.locals.name
  location                              = local.app_gateway_base.locals.location
  resource_group_name                   = local.app_gateway_base.locals.resource_group_name
  tags                                  = local.app_gateway_base.locals.tags
  sku                                   = local.app_gateway_base.locals.sku
  gateway_ip_configuration_name         = local.app_gateway_base.locals.gateway_ip_configuration_name
  gateway_ip_configuration_subnet_id    = local.app_gateway_base.locals.gateway_ip_configuration_subnet_id
  frontend_port_name                    = local.app_gateway_base.locals.frontend_port_name
  frontend_port_number                  = local.app_gateway_base.locals.frontend_port_number
  frontend_ip_configuration             = local.app_gateway_base.locals.frontend_ip_configuration
  private_link_configurations           = local.app_gateway_base.locals.private_link_configurations
  enable_diagnostics                    = local.app_gateway_base.locals.enable_diagnostics
  diagnostic_log_category_groups        = local.app_gateway_base.locals.diagnostic_log_category_groups
  diagnostic_metrics                    = local.app_gateway_base.locals.diagnostic_metrics
  diagnostic_log_analytics_workspace_id = local.app_gateway_base.locals.diagnostic_log_analytics_workspace_id
  alerts_resource_group_name            = local.app_gateway_base.locals.alerts_resource_group_name
  action_groups                         = local.app_gateway_base.locals.action_groups
  application_gateway_base_alert_config = local.app_gateway_base.locals.base_alert_config
  trusted_root_certificates             = local.app_gateway_base.locals.trusted_root_certificates
  zones                                 = local.app_gateway_base.locals.zones
  waf_policy                            = local.app_gateway_base.locals.waf_policy
  
  # Complex merging strategies for different data types
  per_app_waf_policies                  = merge([for config in local.app_gateway_configs : try(config.locals.waf_policies, {})]...)
  backend_address_pools                 = flatten([for config in local.app_gateway_configs : try(config.locals.backend_address_pools, [])])
  backend_http_settings                 = flatten([for config in local.app_gateway_configs : try(config.locals.backend_http_settings, [])])
  ssl_certificates                      = flatten([for config in local.app_gateway_configs : try(config.locals.ssl_certificates, [])])
  http_listeners                        = flatten([for config in local.app_gateway_configs : try(config.locals.http_listeners, [])])
  request_routing_rules                 = flatten([for config in local.app_gateway_configs : try(config.locals.request_routing_rules, [])])
  rewrite_rule_sets                     = flatten([for config in local.app_gateway_configs : try(config.locals.rewrite_rule_sets, [])])
  probes                                = flatten([for config in local.app_gateway_configs : try(config.locals.probes, [])])

  # Map frontdoor module variables to local variables
  deploy_frontdoor            = local.frontdoor_base.locals.deploy_frontdoor
  frontdoor_profile_name      = local.frontdoor_base.locals.frontdoor_profile_name
  frontdoor_sku               = local.frontdoor_base.locals.frontdoor_sku
  frontdoor_endpoints         = local.frontdoor_base.locals.frontdoor_endpoints
  enable_frontdoor_waf_policy = local.frontdoor_base.locals.enable_frontdoor_waf_policy
  frontdoor_base_alert_config = local.frontdoor_base.locals.base_alert_config
  frontdoor_custom_domains    = merge([for config in local.frontdoor_configs : try(config.locals.frontdoor_custom_domains, {})]...)
  origins                     = merge([for config in local.frontdoor_configs : try(config.locals.origins, {})]...)
  origin_group_configuration  = merge([for config in local.frontdoor_configs : try(config.locals.origin_group_configuration, {})]...)
  routes                      = merge([for config in local.frontdoor_configs : try(config.locals.routes, {})]...)
}

# Generate an Azure provider block
generate "provider" {
  path      = "_provider.tf"
  if_exists = "overwrite" #overwriting root terragrunt file
  contents  = <<EOF
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.35"
    }
    azuread = {
      source  = "hashicorp/azuread"
      version = "~> 3.4"
    }
  }
}

provider "azurerm" {
  subscription_id                 = var.subscription_id
  use_oidc                        = true
  resource_provider_registrations = "extended" #A larger set of resource providers that provides coverage for the most common supported resources. See this doco https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#resource-provider-registrations
  resource_providers_to_register  = [] #Pass a list of strings here to register specific resource providers for example "Microsoft.AlertsManagement". See this list https://github.com/hashicorp/terraform-provider-azurerm/blob/main/internal/resourceproviders/required.go
  storage_use_azuread             = true
  features {}
}

provider "azuread" {
  tenant_id = "__AZURE_TENANT_ID__"
}

EOF
}

Choosing the Right Pattern for Your Team

After implementing these patterns across different organisations and use cases, here’s my decision framework:

Start with Simple Input Mapping if you have

• Configuration that’s relatively small (under 50 resources per environment)
• Data structure that’s uniform across sources

Move to TFVars Generation for Large Datasets when you encounter

• Large datasets that are hitting CLI limits (you’ll know when it happens!)

Use Advanced Input Aggregation when you’re managing

• Multi-service architectures with varying configuration schemas
• Services that need flexible addition/removal capabilities
• Configuration complexity that varies significantly between components

Conclusion

These three Terragrunt patterns shown above address common infrastructure management challenges at different scales. Each pattern emerged from practical requirements for managing Terraform configurations effectively.

Terragrunt can provide flexibility to adapt configuration patterns as your Terraform/OpenTofu requirements grow.

Thanks for reading, looking forward to your thoughts below,

Jesse

Before we get cracking my assumption is that you have a basic to intermediate understanding of how Azure networking works so I can make this writeup as streamlined as possible!

Firstly let’s look at a potential high-level migration architecture diagram as shown in the below. This helps set the scene for what’s to come.

• A brownfields hub and spoke architecture exists with IaaS-based shared services deployed into the hub VNET alongside an existing ExpressRoute Circuit providing connectivity to on-premises.
• A greenfields virtual WAN hub will be deployed with PaaS-based shared services and connected to the brownfields ExpressRoute Circuit to achieve hub to hub connectivity and hybrid connectivity to on-premises.

Based on Microsoft’s published guidance on migrating to Azure Virtual WAN the ideal sequence is:

1. Deploy Virtual WAN hub(s)
2. Connect remote sites (ExpressRoute/VPN) to Virtual WAN
3. Test hybrid connectivity via Virtual WAN
4. Transition connectivity to Virtual WAN hub
5. Old hub becomes shared services spoke
6. Optimise on-premises connectivity to fully utilise Virtual WAN

Whilst I won’t go into specific instructions/detail on each of the above steps (because Microsoft’s doco is solid with diagrams to help understand the changes) I will touch on a few related points that caught me out during a recent project.

Multi Region Architecture

If you’re implementing a primary region and secondary region deployment of vWAN to cover multi-region requirements the thing which might not be obvious is that you will need to have vWAN Hub in both regions tied to the same vWAN.

So for example:

• Australia East Region - AE vWAN + AE vWAN Hub deployed
• Australia SouthEast Region - ASE vWAN Hub deployed (and associated to AE vWAN)

Having your vWAN Hubs associated to the same vWAN ensures that hub to hub connectivity and routing is automatically established and there’s no need to create VPN/ExpressRoute connections directly between your vWAN Hubs in either region.

Allowing Traffic from Non-Virtual WAN Networks

If you’re about to deploy Virtual WAN you should know that the default behaviour for enabling traffic from Non-Virtual WAN networks has changed (since 2024 I think) and takes a few more steps to achieve.

I was caught out by this after creating an ExpressRoute connection to link my new vWAN hub to the old hub’s ExpressRoute Circuit then noticing that none of the old hub’s routes were being received on vWAN side :)

To resolve this you need to firstly ensure that your vWAN hub has this option “Allow traffic from non Virtual WAN networks” turned on.

Additionally your vWAN ExpressRoute Gateway needs to have the “allowNonVirtualWanTraffic” property set to true.

And finally on your old hub side - ensure the virtual network gateway for ExpressRoute has this option for “Allow traffic from remote Virtual WAN networks” turned on.

So there’s a couple non-default hoops to jump through there but once you do enable the above you’ll get the delightful result of seeing your old hub’s routes coming through to your vWAN side.

Micro Segmentation Traffic Inspection

If you’re using Routing Intent on your vWAN hub to centrally manage routing for your connected spokes the vWAN’s default route table will automatically include all RFC1918 super-nets (10.0.0.0/8, 192.168.0.0/16 and 172.16.0.0/12) but did you know that traffic between subnets in the same VNET will not go through these RFC1918 routes and hence not be inspected by your hub firewall? The same applies for traffic between hosts in the same subnet.

This is because each VNET has a system route which, because it’s often a smaller prefix than the RFC1918 super-nets in the vWAN default route table, will remain active and in use.

To demonstrate this scenario in my lab I created a secured vWAN hub with routing intent policies enabled and a connected spoke VNET (10.1.0.0/16) with 2 subnets 10.1.1.0/24 and 10.1.2.0/24.

Below you can see my next hop test result between subnets showing traffic sending to the VirtualNetwork via that system route I mentioned earlier. This is the default behaviour for subnet to subnet traffic even within a vWAN architecture with routing intent policies enabled, and it means traffic between these subnets and hosts in the same VNET is not being inspected by the hub firewall.

Now to achieve micro segmentation I’ve created a route table associated to my subnets and a UDR for my VNET 10.1.0.0/16 to send to the firewall IP address 10.0.0.132. You can see based on the test result below that traffic between subnets is now directed to the firewall in the hub.

This final test result also shows that with the route table and UDR addition, traffic between hosts in the same subnet is also now directed to the firewall in the hub.

As demonstrated above, implementing micro segmentation so that host to host and subnet to subnet traffic within the same VNET is inspected at your hub firewall can be achieved within a vWAN architecture however I don’t recommend doing this at scale for every workload in Azure as it may not be supported for certain PaaS architectures. Consult the Microsoft documentation on this before jumping in!

Transitioning Azure DNS from Old Hub to New Hub

When adopting Azure Virtual WAN in a greenfields deployment coming from a traditional hub and spoke architecture a common scenario may resemble the below diagram.

• A brownfields hub and spoke with existing Private DNS Zones to service the brownfields landing zones and on-premises AD DC conditional forwarders setup to the brownfields hub.
• A greenfields Virtual WAN with Private DNS Resolver and new Private DNS Zones to service the greenfields landing zones.

Because it’s a high management overhead to run two Azure DNS hubs side by side for a long period of time I recommend planning and executing a transition to your greenfields deployment. Some key activities to consider and sequence are:

• Scripted syncing of Private DNS records from existing brownfields hub Private DNS zones to greenfields hub Private DNS zones
• Updating any brownfields Private Endpoints to point to greenfields Private DNS zones.
• Enabling Route Table network policies for brownfields private endpoints on the subnets where Private Endpoints are deployed to prevent asymmetric routing from on-prem traffic to private endpoints
• Updating on-premises AD DNS conditional forwarders to point to the greenfields Private DNS Resolver service.
• Removing brownfields Private DNS Zone virtual network links to the brownfields hub VNET to avoid private DNS resolution conflicts.

Conclusion

I hope you enjoyed reading, looking forward to your thoughts below.

Cheers, Jesse

This blog will cover the IaC side of things and to keep this short and sweet I’m assuming you already have working knowledge of how to deploy your IaC via CICD workflows/pipelines.

Be sure to also checkout the Palo Alto documentation on Cloud NGFW to fill in any gaps for the overall deployment prerequisities and configuration. One thing that caught me out early on was onboarding the Azure Tenant to the Strata Cloud Manager account and ensuring the required Azure resource providers were registered on the target subscription.

Provider Config

These are the providers I pinned to for my Palo Alto deployment. Note that I am using AzAPI provider because at the time of writing AzureRM did not have a resource covering my usecase to deploy a Palo Alto NGFW with Strata Cloud Manager (SCM) integration.

terraform {
  required_version = ">= 1.3"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 4.39.0"
    }
    azapi = {
      source  = "Azure/azapi"
      version = ">= 2.4"
    }
  }
}

AzureRM Resources

Here I’m creating the Palo Alto virtual network appliance which I’m associating to my virtual WAN hub ID.

There’s also 2 separate baseline public IPs resources to cover the Egress NAT traffic and standard traffic. Additional public IPs can be easily created by incrementing the related count variable value.

Finally there’s a user assigned managed identity which is required by the Palo Alto NGFW as it didn’t support a system managed identity.

data "azurerm_client_config" "current" {}

resource "azurerm_palo_alto_virtual_network_appliance" "this" {
  name           = var.appliance_name
  virtual_hub_id = var.virtual_hub_id
}

resource "azurerm_public_ip" "egress_nat" {
  for_each            = { for i in range(var.egress_nat_ip_address_count) : i => i }
  name                = "${var.firewall_name}-egress-nat-pip-${each.key}"
  location            = var.location
  resource_group_name = var.resource_group_name
  sku                 = "Standard"
  allocation_method   = "Static"
  tags                = var.tags
}

resource "azurerm_public_ip" "this" {
  for_each            = { for i in range(var.public_ip_address_count) : i => i }
  name                = "${var.firewall_name}-pip-${each.key}"
  location            = var.location
  resource_group_name = var.resource_group_name
  sku                 = "Standard"
  allocation_method   = "Static"
  tags                = var.tags
}

resource "azurerm_user_assigned_identity" "this" {
  name                = "${var.firewall_name}-uami"
  location            = var.location
  resource_group_name = var.resource_group_name
  tags                = var.tags
}

AzAPI Resources

The AzAPI resource here is fairly simple:

• set the parent_id to the RG
• pass through the user assiged identity ID
• configure the FW to use Strata Cloud Manager or Panorama based on a variable value (var.cloud_managed_type)
• pass through the Strata Cloud Manager config or Panorama config based on a variable value (var.cloud_managed_type)
• for each on the public IPs based on the count
• link the FW to the network virtual appliance and VWAN Hub

resource "azapi_resource" "this" {
  type      = "PaloAltoNetworks.Cloudngfw/firewalls@2025-05-23"
  name      = var.firewall_name
  parent_id = "/subscriptions/${data.azurerm_client_config.current.subscription_id}/resourceGroups/${var.resource_group_name}"
  location  = var.location
  tags      = var.tags
  identity {
    type         = "UserAssigned" # Only user assigned identity is supported at this time
    identity_ids = [azurerm_user_assigned_identity.this.id]
  }
  body = {
    properties = {
      dnsSettings = {
        dnsServers = [
          for dns_server in var.dns_settings.dns_servers : {
            address    = dns_server.address
            resourceId = dns_server.resourceId
          }
        ]
        enabledDnsType = var.dns_settings.enabled_dns_type
        enableDnsProxy = var.dns_settings.enable_dns_proxy
      }
      isStrataCloudManaged = var.cloud_managed_type == "Strata" ? true : false
      isPanoramaManaged    = var.cloud_managed_type == "Panorama" ? true : false
      networkProfile = {
        egressNatIp = [
          for i in range(var.egress_nat_ip_address_count) : {
            address    = azurerm_public_ip.egress_nat[i].ip_address
            resourceId = azurerm_public_ip.egress_nat[i].id
          }
        ]
        enableEgressNat = var.enable_egress_nat
        networkType     = var.network_type
        publicIps = [
          for i in range(var.public_ip_address_count) : {
            address    = azurerm_public_ip.this[i].ip_address
            resourceId = azurerm_public_ip.this[i].id
          }
        ]
        trustedRanges = var.trusted_ranges
        vwanConfiguration = {
          networkVirtualApplianceId = azurerm_palo_alto_virtual_network_appliance.this.id
          vHub = {
            resourceId = var.virtual_hub_id
          }
        }
      }
      panoramaConfig = {
        configString = var.cloud_managed_type == "Panorama" ? var.panorama_config_string : ""
      }
      strataCloudManagerConfig = {
        cloudManagerName = var.cloud_managed_type == "Strata" ? var.strata_cloud_manager_name : ""
      }
      marketplaceDetails = {
        offerId     = var.marketplace_details.offer_id
        publisherId = var.marketplace_details.publisher_id
      }
      planData = {
        billingCycle = var.plan_data.billing_cycle
        planId       = var.plan_data.plan_id
      }
    }
  }
  depends_on = [azurerm_user_assigned_identity.this, azurerm_palo_alto_virtual_network_appliance.this, azurerm_public_ip.egress_nat, azurerm_public_ip.this]
}

Variables

These variables below complete the full IaC codebase for the Palo Alto Cloud NGFW deployment and I used them to ensure the module for the deployment was repeatable and as generic as possible.

variable "resource_group_name" {
  type        = string
  description = "The name of the Resource Group where the Palo Alto Cloud NGFW Firewall will be deployed."
}

variable "appliance_name" {
  type        = string
  description = "The name which should be used for this Palo Alto Local Network Virtual Appliance. Changing this forces a new Palo Alto Local Network Virtual Appliance to be created."
}

variable "virtual_hub_id" {
  type        = string
  description = "The ID of the Virtual Hub to deploy this appliance onto. Changing this forces a new Palo Alto Local Network Virtual Appliance to be created."
}

variable "firewall_name" {
  type        = string
  description = "The name of the Palo Alto Cloud NGFW Firewall."
}

variable "location" {
  type        = string
  description = "The Azure region where the Palo Alto Cloud NGFW Firewall will be deployed."
}

variable "tags" {
  type        = map(string)
  description = "A map of tags to assign to the Palo Alto Cloud NGFW Firewall."
  default     = {}
}

variable "network_type" {
  type = string
  description = "The type of network to deploy the Palo Alto Cloud NGFW Firewall onto. Allowed values: 'VWAN' or 'VNET'. Defaults to VWAN."
  default     = "VWAN"
  validation {
    condition     = contains(["VWAN", "VNET"], var.network_type)
    error_message = "network_type must be either 'VWAN' or 'VNET'."
  }
}

variable "enable_egress_nat" {
  type = string
  description = "Enable Egress NAT for the Palo Alto Cloud NGFW Firewall. Allowed values: 'ENABLED' or 'DISABLED'. Defaults to 'ENABLED'."
  default     = "ENABLED"
  validation {
    condition     = contains(["ENABLED", "DISABLED"], var.enable_egress_nat)
    error_message = "enable_egress_nat must be either 'ENABLED' or 'DISABLED'."
  }
}

variable "egress_nat_ip_address_count" {
  type        = number
  description = "The number of Egress NAT IP addresses to allocate for the firewall."
  default     = 1
}

variable "public_ip_address_count" {
  type        = number
  description = "The number of Public IP addresses to allocate for the firewall."
  default     = 1
}

variable "dns_settings" {
  type = object({
    dns_servers = list(object({
      address    = string
      resourceId = string
    }))
    enabled_dns_type = string
    enable_dns_proxy = string
  })
  description = "DNS settings for the Palo Alto Cloud NGFW Firewall."
}

variable "trusted_ranges" {
  type        = list(string)
  description = "List of NON-RFC 1918 trusted ranges for the Palo Alto Cloud NGFW Firewall."
}

variable "cloud_managed_type" {
  type = string
  description = "Is this appliance managed by Panorama or Strata Cloud Manager? Allowed values: 'Panorama' or 'Strata'."
  validation {
    condition     = contains(["Panorama", "Strata"], var.cloud_managed_type)
    error_message = "cloud_managed_type must be either 'Panorama' or 'Strata'."
  }
}

variable "panorama_config_string" {
  type        = string
  description = "Base64 encoded string representing Panorama parameters to be used by Firewall to connect to Panorama. This string is generated via azure plugin in Panorama"
  default     = ""
}

variable "strata_cloud_manager_name" {
  type        = string
  description = "The name of the Strata Cloud Manager which is intended to manage the policy for this firewall."
  default     = ""
}

variable "marketplace_details" {
  type = object({
    offer_id     = string # e.g. "pan_swfw_cloud_ngfw"
    publisher_id = string # e.g. "paloaltonetworks"
  })
  description = "Marketplace details for the Palo Alto Cloud NGFW Firewall."
}

variable "plan_data" {
  type = object({
    billing_cycle = string # e.g. "MONTHLY"
    plan_id       = string # e.g. "panw-cloud-ngfw-payg"
  })
  description = "Plan data for the Palo Alto Cloud NGFW Firewall."
}

I hope you enjoyed reading, looking forward to your thoughts below.

Cheers, Jesse

Intro To ADB IP Access Lists

Using ADB IP access lists allows us to control which networks can connect to our ADB account and workspaces - by default all connections from any IP address are allowed so most enterprises will want to further secure network access by configuring this feature either via CICD and DevOps processes or manually as a portal-driven change.

Currently ADB has two IP access list features:

• IP access lists for the account console (currently Public Preview) - IP access lists for the account console to allow users to connect to the account console UI and account-level REST APIs only through a set of approved IP addresses.
• IP access lists for workspaces - IP access lists for Azure Databricks workspaces to allow users to connect to the workspace or workspace-level APIs only through a set of approved IP addresses.

Access is checked according to this flow below. Image Source: Microsoft Docs

Problem Statement and Rabbit Holes

When working on a project to deploy and manage multiple ADB workspaces via code, I had already deployed the workspaces using Terraform AzureRM and I briefly explored the option of managing the workspace IP access configuration using the available Terraform Databricks IP Access List resource. Ultimately I didn’t go down the path of using the Terraform Databricks provider for various reasons specific to that project but I am interested in getting hands-on with that provider in future if there was the right opportunity.

So I still needed to manage the ADB workspace IP access lists via CICD and the next best option was to leverage the latest Databricks CLI (currently Public Preview). This first led me down a bit of a rabbit hole of various Databricks CLI authentication options until I finally figured out that ‘magic’ set of environment variables to use for OAuth machine-to-machine (M2M) authetication to successfully work to the workspaces. If you’re interested in knowing more about Databricks CLI authentication, I have some high level details in my repo’s README.

The second rabbit hole was related to Databricks CLI itself. During this project I often found myself confused about what the actual supported cmdlets and inputs were for the CLI. I’ll try not to bore you too much with the details but in general I found the Databricks CLI commands documentation to be a bit incomplete in terms of available/supported commands that map to the REST APIs. This was further validated when I discovered various //TODO comments in the CLI’s .go files for the workspace/ip-access-lists command as shown here. With Databricks saying the CLI is still in public preview I’m hoping these doco gaps are fleshed out and completed prior to it going GA.

Managing Azure Databricks Workspace IP Access Lists via CICD

So now let’s dig into how exactly I managed ADB IP access lists via CICD.

As part of my Azure DevOps build and release pipeline templates I needed to install the Databricks CLI to my Ubuntu agent pool. This was fairly easy following the available doco.

curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

I then added a single step in my build and release pipeline template to:

1. echo the existing IP access lists to the logs (this is also how we can see the ip_access_list_id which is needed for update and delete operations)
2. enable or disable the IP access lists based on a parameter input from the calling pipeline
3. echo the IP access list enablement status to the logs

The below cmdlets gave me an easy way to toggle enablement or disablement of the workspace IP access lists via the Azure DevOps release pipeline.

Now onto the fun bit – standing up the capability to trigger create, update, and delete operations within the IP access list.

Noticing that the Databricks CLI currently only supports JSON inputs for IP access list changes I chose to store the workspace IP ACLs in dedicated .JSON files per environment within a single folder in my repo e.g. ./workspace-ip-access-lists with each JSON object aligning to the below inputs as shown here.

Now that I had the IP ACLs defined in .JSON files within the repo I then added a single step in my release pipeline template which:

1. iterates over JSON objects: The jq -c '.[]' "$json_file" command extracts each object from the JSON file ($json_file) as a compact, single-line JSON string. The while loop reads each of these JSON objects one by one into the variable json_object_creation
2. extracts the operation field: For each JSON object, the script uses jq -r '.operation' to extract the value of the operation field. This value is stored in the operation variable. The -r flag ensures that the extracted value is output as a raw string, without quotes
3. checks for “Create” operations: The script converts the operation value to lowercase using ${operation,,} and checks if it starts with the word "create". This is done using the if [[ ${operation,,} == "create"* ]]; then condition. If the condition is true, the script proceeds to execute the block of code inside the if statement
4. creates Databricks IP Access Lists: If the operation is a "create" operation, the script logs a message to the console indicating that it is creating a new Databricks IP Access List. It then invokes the databricks ip-access-lists create command, passing the JSON object ($json_object_creation) as input using the –json flag. The command also
5. uses two parameters, BUNDLE_TARGET and DATABRICKS_LOG_LEVEL, which are dynamically substituted from the pipeline’s parameters ($ and $)
6. uses || true at the end of the databricks command ensuring that the script does not fail if the command encounters an error

This is the full pipeline step to create a new ADB IP access list:

An example of a valid JSON object block which would be read and used by the above pipeline step is below.

[
  {
    "label": "ALLOW_EXAMPLE_CORP_NETWORK1",
    "list_type": "ALLOW",
    "ip_addresses": ["192.168.0.0/23"],
    "operation": "CREATE"
  }
]

To support the 'update' operation I repeated the above pipeline step logic as above, but added additional input and logic to support extracting and passing in the ip_access_list_id which is required.

An example of a valid JSON object block which would be read and used by the above pipeline step is below. Note the ip_access_list_id value and operation value.

[
  {
    "label": "ALLOW_EXAMPLE_CORP_NETWORK1",
    "list_type": "ALLOW",
    "ip_addresses": ["192.168.0.0/23", "192.168.100.0/23"],
    "ip_access_list_id": "a559572d-1730-4ce4-203z-75506242f04h",
    "operation": "UPDATE"
  }
]

The 'delete' operation shown below follows a similar implementation logic as the ‘update’ operation to handle ip_access_list_id which is required.

An example of a valid JSON object block which would be read and used by the above pipeline step is below. Note the ip_access_list_id value and operation value.

[
  {
    "label": "ALLOW_EXAMPLE_CORP_NETWORK1",
    "list_type": "ALLOW",
    "ip_addresses": ["192.168.0.0/23", "192.168.100.0/23"],
    "ip_access_list_id": "a559572d-1730-4ce4-203z-75506242f04h",
    "operation": "DELETE"
  }
]

Now that I had the basic foundations to manage this all from a pipeline and repo, a typical DevOps flow to operationally consume the above capability would be to:

1. create a new branch of the github.com/globalbao/azure-databricks-cicd repo
2. update an existing .JSON file within ./workspace-ip-access-lists with the desired ADB IP access list operation (create, update, delete)
3. optionally, update the 'ADB_ENABLE_IP_ACCESS_LISTS' parameter value to true or false from the calling pipeline
4. pull request and merge the new branch to main
5. approve the latest pipeline run from main to deploy the IP access list changes

Final Thoughts

Some improvements I can think of if I were to get a chance at this again:

1. add JSON input validation as part of the build pipeline step to catch typos and errors early on before a PR/merge to main
2. add capability to manage ADB Account IP Access Lists as it appears to be supported through Databricks CLI
3. add logic for finding and mapping the ip_access_list_id from created IP access lists to any ‘update’ and ‘delete’ operations so you don’t need to manually input that value into the .JSON files.
4. experiment using the Terraform Databricks provider as an alternative to Databricks CLI (assuming both Workspace and Account IP Access Lists are supported)

The past project, and this related blog post, were both fun and engaging pieces for me personally. I felt that I’ve been able to leverage past skills and experience in the DevOps space to prototype something fairly quickly and navigate through several rabbit holes that appeared.

I hope you enjoyed reading, looking forward to your thoughts below.

Cheers, Jesse

G’day folks. I recently had the pleasure of presenting a livestream session via Microsoft Reactor Sydney on a subject close to my heart.

It was a great opportunity to share my experience in the subject domain of Security Governance and Azure Policy as Code workflows using Bicep language. And also a good opportunity to receive some questions and feedback in real time from an audience.

I’ve included some links below if you’re interested to dive into this subject a bit more.

Watch

Watch the recorded session from Microsoft Reactor’s livestream event.

• YouTube: Flexing Your Security Governance with Azure Policy As Code

Catchup on the latest (Dec 2021) Azure Governance & Deployments news from Microsoft.

• YouTube: Azure Governance & Deployments Quarterly Customer Panel December 2021

Watch a session on managing the logging and security of Azure Key Vaults at-scale with Azure Policy and Microsoft Sentinel.

• YouTube: Azure Sentinel and Policy as Code from Jesse Loudon and Casey Mullineaux

Watch a session on deploying Azure Policy as Code with Bicep walking through levels 1-2-3 of configuration.

• YouTube: Policy as Code with Bicep for Enterprise Scale

Learn & Discover

Learn about what exactly is Azure Policy from Microsoft Docs.

• Microsoft Docs: What is Azure Policy?

Learn about how Azure Policy effects work from Microsoft Docs.

• Understand Azure Policy effects

Learn about scoping your Azure Policy assignments from Microsoft Docs.

• Understand scope in Azure Policy

Learn about the Azure Security Benchmark from Microsoft Docs.

• Microsoft Docs: Azure Security Benchmark introduction

Learn about Microsoft Security Best Practices from Microsoft Docs.

• Microsoft Docs: What’s inside Microsoft Security Best Practices?

Learn about what is Microsoft Defender for Cloud from Microsoft Docs..

• Microsoft Docs: What is Microsoft Defender for Cloud?

Discover how to apply and enforce Zero Trust concepts with Azure Policy from Microsoft Blogs.

• Microsoft Blogs: Enforcing Policy for Zero Trust with Azure Policy

Discover how to enable Defender for Cloud plans to protect workflows in Azure from the community.

• Community Blog: Automatically Enable and Audit Microsoft Defender for Cloud Enhanced Security Features

Discover how to create custom Microsoft Defender for Cloud recommendations with Azure Policy from the community.

• Community Blog: Create a custom Azure Security Center recommendation with Azure Policy

Discover how to create custom security recommendations within Microsoft Defender for Cloud from the community.

• Community Blog: Custom security recommendation within Microsoft Defender for Cloud

Discover Security Posture Management with Azure Policy and Microsoft Defender for Cloud from the community.

• Community Blog: Security Posture Management with Azure Policy and Microsoft Defender for Cloud

Find

Find samples of built-in definitions for Azure Policy from Microsoft GitHub.

• GitHub: Repository for Azure Policy built-in definitions and samples

Find all the best Azure Policy official and community content available across the internet.

• GitHub: A curated list of AWESOME Azure Policy learning resources and links

Find and adopt working modules and CI/CD pipelines for Azure Policy as Code workflows.

• GitHub: Bicep and Terraform code examples for policy-as-code workflows

Find the related (this blog post) code used for the Bicep demo on Security Governance with Azure Policy as Code.

• GitHub: Bicep Demo Security Governance

Find a list of past security mistakes by CSPs.

• GitHub: List of security mistakes by cloud service providers (AWS, GCP, and Azure)

Keep Informed

Keep up to date with the latest changes and releases for Azure Governance capabilties (Policies, Aliases, RBAC roles, etc).

• Site: AzAdvertizer - Release and change tracking on Azure Governance capabilities

Cheers,

Jesse

My first experience with Azure Policy was mid-2020 engaged as a consultant for an enterprise client in the retail industry. They had been steadily growing their Microsoft Azure footprint since before the start of the COVID pandemic. Then there appeared to be somewhat of an explosion in projects to onboard and migrate applications and infrastructure into their Azure tenancies. Due to this explosion of growth there also happened to be what I would call ‘governance drift’. Examples of which included:

• Tags - Missing completely, incorrect or outdated keys/values, no inheritence of tags from resource group to resources.
• Data Protection - Gaps in compliance for virtual machine backups and disk encryption.
• Monitoring - No baseline logging and alerting for infrastructure.
• Security - RBAC drift on resource groups, resource locks missing from some critical infrastructure such as ExpressRoute.

At the same I was learning HashiCorp Terraform for the first time in my life and being a complete novice (I still am!) I’m grateful for the help I received from Michael O’Leary (my colleague on the same project) and the Terraform community through their blogs and tutorials.

In the first week of January 2022 I was bouncing about the #TechTwitter space and was well down the rabbit hole with Open Policy Agent (OPA). My interest in OPA is largely thanks to having reviewed a couple chapters covering possible use cases and solutions with OPA. Courtesy of my role as a technical reviewer for PackT publishing which began in August 2021.

Long story short, if you’re also interested in OPA you may have come across github.com/anderseknert/awesome-opa by Anders Eknert. It’s a curated list of OPA related tools, frameworks, and articles. So as a newcomer to OPA I found this public list of resources helpful and having followed Anders on Twitter I have come to trust that he is a good source of information on the subject matter.

There was inspiration from that list and it was a great starting point for what I had in mind for an Azure Policy list.

You may be wondering how to go about creating a list of something where no list exists? For me, because I’ve been actively engaged in the Microsoft Azure community for a few years, through various social media channels and I’ve come to know of people who are leaders in their field and publicly sharing their knowledge with the community.

Regardless of whether you are searching for docs, code repos, tools, blogs, or videos, I have a feeling that 80% of content out there has been created with the blood sweat and tears from real people. Yes there are exceptions, but for the majority of stuff out there people are the ones with the ideas and experiences and they are what drive community engagement and growth. Find your people and you’ll find the community and the content you need to build your own list.

So here’s where you should start:

1. People - identify leaders in the community, stalk them (in a nice way), follow their announcements and content.
2. Blogs - Search articles of content and look for links/references to other articles, tools, blogs, authors, etc.
3. Twitter - use keyword/hashtag searching for public content.
4. GitHub - use keyword searching for public repositories.
5. YouTube - use keyword searching for published videos.
6. Events - use keyword searching of annual events run by the community for public content.
7. Browser - maybe you have bookmarked some gems? :).

Guiding principles for organsing the README:

• List items sorted in alphabetical order
• Links cannot be behind a paywall / must be freely accessible to everyone
• Sentence case for Blogs, Videos, Docs, etc
• Lower case for GitHub repositories
• No duplicates of items in lists
• Official links placed at the top

That’s all I have for now. I hope you found this article interesting and I hope you find the Awesome Azure Policy list helpful. If you have any feedback or suggestions please feel free to leave comments below. Thanks for reading!

Cheers Jesse

The Ctrl+Alt+Azure podcast was launched back in October 2019 by Microsoft Azure MVPs Tobias Zimmergren and Jussi Roine.

Tobias had reached out to me over twitter to check my availability and interest in contributing to an upcoming episode dedicated to Azure Policy as Code.

Admittedly I was surprised by Tobias’s message, and other community events were already on my calendar, but the fact of the matter is that having actively advocated on this exact subject matter over the past 12+ months I would not be passing up such a wonderful opportunity!

:studio_microphone: You can tune into the episode at https://ctrlaltazure.com/episodes/109-azure-policy-as-code. There you’ll also find helpful links to various tools and resources mentioned during our talk.

Starting from ground zero with Azure Policy? Check out episode 25 of the Ctrl+Alt+Azure podcast Azure Policies - the how, the what, and the why?

Azure Policy as Code - Episode 109 - Session Abstract

:bulb: Don’t have 40 mins to listen to the entire episode but want to know what’s discussed? I’ve got you covered!

Here’s 13 points of Azure Policy as Code goodness covered during episode 109! :muscle:

1. Brief introduction to Azure Policy and Policy as Code using Bicep language - Microsoft’s Domain Specific Language for transpiling into ARM Templates.
2. At what point might you decide to move from managing your Azure Policies via the Azure Portal (click-ops) to a Policy as Code workflow based on Infrastructure as Code and DevOps methodologies?
3. Using Bicep’s loadtextcontext function to quickly onboard new policies to your Bicep modules.
4. Is there a difference in capability for developers when managing your Azure Policies from the Azure Portal versus managing them as code via say Bicep/ARM templates?
5. What are some key concepts to know about when working with Azure Policy as Code workflows?
6. Designing your Service Principal RBAC roles adhering to the principle of least privilege.
7. How to approach testing methdologies for new custom Azure Policies before deploying them to prod.
8. Programmatically triggering an Azure Policy on-demand compliance scan at your desired scope.
9. Recommended tools for getting started with Azure Policy as Code workflows using Bicep language.
10. Available options for migrating your Azure Policies from the Azure Portal to a code-driven workflow.
11. Free community tools which can assist with your Azure Policy as Code journey.
12. Infrastructure as Code tooling comparison.
13. A surprise question to Jesse.

:studio_microphone: Now that I’ve piqued your curiousity :smile: tune into the episode at https://ctrlaltazure.com/episodes/109-azure-policy-as-code.

The Ctrl+Alt+Azure podcast journey

My own research of the Ctrl+Alt+Azure podcast’s early beginnings and it’s journey from episode 1 to episode 52 (late last year) has yielded many insightful blog posts from both Tobias and Jussi.

I highly recommend checking them out! For example I found a couple beautiful gems in their documented journey some of which include:

• Choosing a podcast name. (one of my favourites!)
• Finding time to record an episode.
• Staying flexible with scheduling.
• Leveraging peer pressure to collaborate.
• Finding their audience via distribution.
• Technology and tooling choices.

Launch annoucements

• Announcing a new podcast Ctrl Alt Azure - Tobias Zimmergren
• Announcing our new podcast Ctrl Alt Azure the first episode is out now - Jussi Roine

Getting started blog posts

• How we planned and launched a podcast the technical story - Tobias Zimmergren
• Getting started with a podcast the equipment setup and logistics - Jussi Roine

Podcast anniversaries

• The first anniversary of the Ctrl Alt Azure podcast - Jussi Roine

Conclusion

It was an amazing opportunity to talk Azure Policy as Code with Tobias Zimmergren and Jussi Roine. I personally learned much from the discussions. Also very grateful for the wide scope of questions from Tobias and Jussi related to the topic and hope we were inclusive towards various perspectives, scenarios, and skillsets!

As to my own experience with contributing to a podcast episode let’s just say I now have a great appreciation for the amount of effort required to plan, setup, and record just one of these episodes! :smile:

Thanks for tuning in folks. I hope you found the episode interesting, informative, and helpful; as always I welcome your feedback and comments.

Cheers,

Jesse

I had a scope requirement for a recent customer engagement to implement diagnostic settings for several resource types - one of which was Azure Kubernetes Service (AKS) clusters.

These diagnostic settings needed to be customised per the design document and ultimately logs were to be forwarded to a log analytics workspace – perfect fit for leveraging policy-as-code and deployIfNotExists policies!

The following builtin policy seemed to fit my requirements above perfectly so I set about testing it in my development subscription.

  "properties": {
    "displayName": "Deploy - Configure diagnostic settings for Azure Kubernetes Service to Log Analytics workspace",
    "policyType": "BuiltIn",
    "mode": "Indexed",
    "description": "Deploys the diagnostic settings for Azure Kubernetes Service to stream resource logs to a Log Analytics workspace.",
    "metadata": {
      "version": "1.0.0",
      "category": "Kubernetes"
    },

When testing deployIfNotExists policies you should verify (1) the Azure Resource Manager (ARM) template deployment for your non-compliant resources was successful, and (2) the resource is marked as compliant post-remediation task.

In case you’re not familiar with deployIfNotExists policies this snippet gives a high-level overview of the JSON:

So here’s a simplistic image illustrating my flow when troubleshooting the root cause of this non-compliant policy. I’ll cover each highlighted point in more detail soon.

Fighting The ‘Non-Compliance’ Enemy

Screenshot #1 (or SS #1) shows a non-compliant AKS cluster. This evaluation result is AFTER a remediation task had successfully configured the diagnostic settings per my requirements on the resource. Initially I was puzzled to see this non-compliant result but it became clear why this was was happening as I investigated the policy’s existenceCondition.

SS #2 shows the reason for non-compliance is because target value and current value for the evaluated field is not matching. The path for the evaluated field is also an array “properties.logs[*].enabled” which basically means there’s more than one element to evaluate.

I was able to view the reason for non-compliance by clicking into the Details link under the Compliance reason column – a crucial piece of evidence for troubleshooting – in the future I hope we’ll be able to query this exact data programmatically.

SS #3 shows my verification of the successful remediation task on the resource. This confirms the policy’s nested Azure Resource Manager (ARM) template deployment was a SUCCESS.

SS #4 shows our root cause issue with one of the existenceCondition blocks where the alias “Microsoft.Insights/diagnosticSettings/logs.enabled” needs to equal to “True” for the resource to be marked as compliant.

Again, based on the fact that this alias maps to an array (as seen in SS #2) this condition is basically saying that every element in the array needs equal to “True” for the resource to be marked as compliant after an evaluation scan. A bit of an ‘opps’ moment here :smile:

Winning The Battle

SS #5 and SS #6 show how I resolved this issue by:

1. changing the condition from “equals” to “in”
2. referencing each AKS log parameter name in the array aka “[parameters(‘kube-apiserver’)]” etc

These parameter names also need to be in the right order. And as the original policy was a builtin type, I duplicated the JSON into a custom policy and modified the existenceCondition as shown in SS #5.

After checking numerous builtin policies for configuring diagnostic settings I can confirm Microsoft have paramaterised the individual logs/metrics so you can specify during your policy assignment which logs/metrics you want to configure (by default they are all set to “True”).

This is great, as it allows developers/admins to be flexible with the policy’s settings without having to change/duplicate the policy definition JSON to get a desired result. However I believe most of these builtin policies have the same design flaw with the existenceCondition as outlined in this blog post.

Battle Report

I found that the builtin policy’s existenceCondition shown below only 100% works if the logs/metric parameter default values do not change e.g. from “True” to “False”.

"existenceCondition": {
    "allOf": [
    {
        "field": "Microsoft.Insights/diagnosticSettings/logs.enabled",
        "equals": "True"
    },
    {
        "field": "Microsoft.Insights/diagnosticSettings/metrics.enabled",
        "equals": "True"
    },
    {
        "field": "Microsoft.Insights/diagnosticSettings/workspaceId",
        "equals": "[parameters('logAnalytics')]"
    }
  ]
},

My definition of an 100% working deployIfNotExists policy is one which:

• successfully deploys the policy’s nested ARM template to your non-compliant resource
• post-remediation marks the resource as compliant after an evaluation scan

Now for my use-case I needed to set a few of these parameters to “False” per below example.

parameter_values = {
    "AllMetrics" : { value = "False" }
    "kube-apiserver" : { value = "False" }
    "kube-controller-manager" : { value = "False" }
    "kube-scheduler" : { value = "False" }
    "cluster-autoscaler" : { value = "False" }
    }

Because the builtin policy’s existenceCondition shown previously expected all values in the array alias for “Microsoft.Insights/diagnosticSettings/logs.enabled” to equal to “True” I was never going to get a compliant resource. The remediation tasks would be successful but the policy was only really 50% working ‘out-of-the-box’. Not cool!

This is good example of incomplete policy authoring and why testing new policies against a proven methodology and framework can surface the issues as described in this blog post, particularly when the policy’s parameter values are changing.

The new existenceCondition shown below 100% works even if the logs/metric parameter default values have changed e.g. from “True” to “False”. This is what we need to aim for across all diagnostic settings policies using the deployIfNotExists effect.

"existenceCondition" : {
    "allOf" : [
    {
        "field" : "Microsoft.Insights/diagnosticSettings/logs.enabled",
        "in" : [
            "[parameters('kube-apiserver')]", 
            "[parameters('kube-audit')]", 
            "[parameters('kube-controller-manager')]", 
            "[parameters('kube-scheduler')]", 
            "[parameters('cluster-autoscaler')]", 
            "[parameters('kube-audit-admin')]", 
            "[parameters('guard')]"
            ]
    },
    {
        "field" : "Microsoft.Insights/diagnosticSettings/metrics.enabled",
        "equals" : "[parameters('AllMetrics')]"
    },
    {
        "field" : "Microsoft.Insights/diagnosticSettings/workspaceId",
        "equals" : "[parameters('logAnalytics')]"
    }
  ]
},

Thanks for joining me today, I look forward to your feedback and questions.

Read more about Azure Kubernetes Service (AKS) logs here

Interested in authoring policies which evaluate array aliases? I recommend reading this

For a breakdown of available policy evaluation conditions check out this link

Keep fighting the good fight!

Jesse

Firstly, here’s what you can expect from this article:

• further context not captured by the recorded session
• insights into pros and cons (very important to know these before diving in)
• another usage pattern for your Terraform infrastructure-as-code journey

You can find my HashiTalk video link/slides at the end of this post

Further Context + Pros/Cons

CSVs and ForEach are not a ‘silver bullet’ for all your IaC scaling problems. I’m absolutely not saying go out and use CSVs and ForEach for ALL your Terraform IaC initiatives. If you’re reading this you’re hopefully taking the time to do your research and testing to evaluate the pattern for yourself!

There’ll be situations where you should be using native HCL for your data/inputs to avoid complexity whilst adhering to the KISS principle. What I’ve learnt from deploying large datasets are that these situations are not as common as I would like. Particularly if you need to manage an application that is NOT cloud-native and requires a vast range of IaaS resources to operate in your cloud of choice e.g. Microsoft Azure.

An elephant in the room is the question of why use CSV when you have YAML? The short answer is that, having tried both data formats, I’ve found having my inputs/variables stored within CSVs makes management of large datasets easier at scale and helps achieve a DRY coding methodology.

What I’ve also learnt in the CSV vs YAML debate:

• When using YAML for IaC, the keys need to be duplicated for each new resource. Not very DRY. CSVs don’t have this issue because each header column represents the unique key in the resulting map.
• YAML info is presented vertically to you, so if you have 100s to 1000s of resources to manage from a single YAML file this makes management at scale difficult and tiresome! I prefer having the info horizontally presented to me using a CSV/Excel file so I can scroll across rather than scroll down. I’ve leveraged my IDE’s ctrl+find and fold/unfold options heavily with YAML datasets, but these shortcuts don’t change the overall experience much when it comes to large datasets.
• With CSV datasets I can open the files using Excel to use all the wonderful native Excel functions and features to manage/manipulate my dataset e.g. concat strings, freeze panes, column filters, pivot tables, etc! To my knowledge this exact functionality isn’t possible today with YAML datasets!
• With YAML it seems easier to spot extra spaces/rows in your datasets that shouldn’t be there. When using CSVs I’ve leveraged the ‘Edit CSV’ vscode extension to assist me in this area with great success.

Take a look at these 2x NSG rules represented in two data formats. Then imagine managing 100s to 1000s of these NSG rules across multiple NSGs and multiple environments.

YAML example

nsg1:
  - action: Allow
    description: Allow RDP from Corp
    destination_details: VirtualNetwork
    destination_port_ranges: "3389"
    destination_type: Tag
    direction: Inbound
    priority: 100
    protocol: TCP
    rule_name: AllowRDPfromCorp
    source_details: "10.10.0.0/24"
    source_port_ranges: "*"
    source_type: IP Address
  - action: Deny
    description: Deny RDP from Internet
    destination_details: VirtualNetwork
    destination_port_ranges: "3389"
    destination_type: Tag
    direction: Inbound
    priority: 200
    protocol: TCP
    rule_name: DenyRDPfromInternet
    source_details: Internet
    source_port_ranges: "*"
    source_type: Tag

CSV example

action,description,destination_details,destination_port_ranges,destination_type,direction,priority,protocol,rule_name,source_details,source_port_ranges,source_type
Allow,Allow RDP from Corp,VirtualNetwork,3389,Tag,Inbound,100,TCP,AllowRDPfromCorp,10.10.0.0/24,*,IP Address
Deny,Deny RDP from Internet,VirtualNetwork,3389,Tag,Inbound,200,TCP,DenyRDPfromInternet,Internet,*,Tag

Regardless of if you choose YAML or CSV watch out for these common IaC issues relating to:

• Bad data - this is where, due to the human-factor, you’ll have incorrect/errorneous values to investigate and ultimately fix-up before your IaC workflow is healthy.
• Multiple sources of truth - working with customers to build environments as IaC often means working across teams of people who have their own process and habits for maintaining their IaC source of truth. This often differs from your own which leads to confusion/frustration.
• Exceptions to your usage patterns - there’ll be times when the dataset and Terraform consumption pattern isn’t fit-for-purpose to meet every use-case and you’ll need to create an exception/ad-hoc implementation pattern.
• Inconsistent naming - this often happens organically for your Terraform modules as they grow in scale and has been the cause of many ‘terraform state mv’ cmdlets in my experience.
• API timeouts and resource dependencies - the larger your IaC implementation gets (I’m talking hundreds of resources as code in a single TFstate file) you’ll need to manage your changes in a staggered fashion to avoid seeing API timeouts and errors relating to dependencies not being ready.
• TF provider parity with ARM - I’ve had a few cases where the AzureRM provider didn’t support something I needed as IaC and needed to use ARM templates or AzureCLI or even click-ops to acheive the desired outcome!

I’ve enjoyed diving into this usage pattern and exploring the pros and cons with you today. I hope you’ve gained some value from my ranting and can relate to my IaC struggles of today :smile:

Cheers,

Jesse

Session Slides:

Recorded Session:

HashiTalks: Australia / New Zealand - YouTube

VSCode extensions:

Edit csv

Rainbox CSV

Additional Reading:

Terraform + CSVs = Netflix

CSVDecode

ForEach

Global Azure is a community event about the Microsoft Azure platform. On April 15-17 the Global Azure community goes online to share, learn, and have community Azure fun together.

This year I was fortunate to have a session accepted for Global Azure 2021 titled: Policy as Code with Bicep for Enterprise Scale

Recently I’ve been diving into Microsoft’s new DSL Bicep and as I’m passionate about Azure Policy I thought why not combine both aspects into a session?

Admittedly, I left my sessionize submission very late and probably just scraped in. Even so it was a wonderful surprise to see my session accepted followed by a frantic scramble to prepare my presentation (YT links below) and code which can be found here https://github.com/globalbao/azure-policy-as-code.

Presentation Structure

I structured my presentation into 3x levels so that beginners could start with a small proof of concept deployment and then scale up in complexity and advanced logic as their comfort levels increased with Bicep and Azure Policy.

More experienced users can jump straight to Level 3 and learn/adopt a Policy as Code workflow with Bicep.

Links to skip to specific content levels in the recorded session are included below!

Level 1

• Uses built-in policies
• Uses an initiative and assignment
• 1x main.bicep
• Manual CLI deployment

YouTube Video Timestamp 16m 10s

Level 2

• Uses built-in policies and custom policies
• Uses multiple initiatives and assignments
• 1x main.bicep
• Manual CLI deployment
• Targeting multiple Azure environments
• Uses parameter files for environment-specfic values passed during deployment

YouTube Video Timestamp 50m 3s

Level 3

• Uses built-in policies and custom policies
• Uses multiple initiatives and assignments
• Custom policyDefinitionReferenceId for initiatives
• Custom non-compliance msgs for assignments targeted to the policyDefinitionReferenceId
• Advanced modules organised per resource type
• CI/CD workflow automation with GitHub Actions YAML
• Targeting multiple Azure environments with authentication via GitHub secrets

YouTube Video Timestamp 1h 11m 45s

Kudos

Finally, I’d like to thank Rahul Nath for reaching out to me prior to the session and helping me out with YouTube streaming/gearing tips. Much appreciated Rahul!

This year’s Global Azure event was the biggest yet with over 560 speakers and 530 sessions from across the world! So a big kudos to the organisers and session reviewers who contributed their time to make this happen!

Looking forward to next year’s Global Azure!

Jesse