Recently, I embarked on a journey to build a complete observability pipeline from scratch to learn the nuances of the Grafana LGTM stack. In this post, I’ll break down how I configured Prometheus for metrics, Loki for logs, and the new Grafana Alloy as the ultimate telemetry collector.This was Proof of concept project, so i tested in locally, using Rancher Desktop.

You can find the full source code and configuration files for this project in my GitHub repository.

The Architecture

The goal was to move away from legacy configurations and toward a modern, “flow-based” pipeline. Here is the setup:

• Metrics: Using the kube-prometheus-stack to collect node and container metrics.

• Logs: Using Grafana Alloy (the successor to the Grafana Agent) to scrape logs directly from the Kubernetes API.

• Alerting: A sophisticated Alertmanager tree that routes notifications to Slack based on specific team labels.

Metrics and Smart Alerting

One of the biggest hurdles in monitoring is “Alert Fatigue.” To solve this, I implemented a label-based routing strategy.

By adding a team label to my PrometheusRule manifests, I can ensure that a database memory spike goes to the DBA Slack channel, while a CoreDNS failure goes to the Infrastructure channel, and Fronted app related alerts goes to alerts fronted slack channel.

# Example of a team-specific alert
- alert: SQLPodMemoryUsage
  expr: (container_memory_working_set_bytes{container="mssql"} / kube_pod_container_resource_limits{resource="memory"}) * 100 > 80
  labels:
    severity: critical
    team: database  # This label determines the Slack destination

Grafana Alloy

As of March 2, 2026, Promtail has officially reached its End-of-Life (EOL).While the grafana/promtail chart might still exist in the repository, it is no longer receiving security patches or updates. Grafana has replaced it with a new, unified agent called Grafana Alloy.Alloy is the successor to Promtail. It does everything Promtail did but is faster, uses less memory, and can handle metrics and traces in the same pipeline.Unlike Promtail,Alloy can automatically discover pod log paths. Alloy uses a “Flow” configuration. I configured it to:

1. Discover pods in specific namespaces.

2. Filter for specific application containers (mvc and mssql).

3. Process logs to drop noisy SQL error paths before they even hit storage.

This keeps the storage costs in Loki low and the signal-to-noise ratio high.

Alertmanager

The final piece of the puzzle was the AlertmanagerConfig. I set up a hierarchy where every alert is caught by a “Global SRE” receiver, but specific sub-routes handle the specialized teams.

Using continue: true in the routing logic allows an alert to be sent to both a general “All Systems” channel and a specific team channel simultaneously—ensuring no critical issue is ever missed.

The prometheus-overrides.yaml file is the “instruction manual” for Helm deployment. When we install a complex stack like kube-prometheus-stack, it comes with hundreds of default settings. The override file allows us to change only the specific parts we need without modifying the original chart.

Connecting the “Operator” to Rules

The Prometheus Operator acts like a watchman. It doesn’t just load every file it finds; it looks for specific labels. In prometheus-overrides.yaml, we define a Rule Selector.

• I set it to look for prometheus: prometheus.

• This ensures that when we apply cluster-alerts.yaml or microservices-alerts.yaml (which both have that label), the Operator “sees” them and injects them into the Prometheus engine.

Configuring Alertmanager Routing

By default, Alertmanager might not know how to handle specific Kubernetes Custom Resources (CRDs).

• I configured the alertmanagerConfigMatcherStrategy to type: None.

• This allows Alertmanager to treat labels from AlertmanagerConfig (like the team labels for Slack) as global labels. Without this override, Alertmanager would ignore specific routing instructions for different Slack channels.

Enabling/Disabling Default Metrics

The standard Prometheus stack tries to monitor everything (CoreDNS, etcd, KubeProxy, etc.). However, in environments like Rancher Desktop, some of these components aren’t accessible.

• Under defaultRules, i set etcd: false and kubeScheduler: false.

• This prevents dashboard from being filled with “Red” failing alerts for services that don’t actually exist in local setup.

Infrastructure & Persistence

Prometheus is a database. If the Pod restarts and we haven’t configured storage, we lose all historical graphs.

• A volumeClaimTemplate is defined using the local-path storage class with a 2Gi limit.

• This ensures that metrics survive a computer reboot or a pod crash by saving the data to the disk.

SQL deployment monitoring

The metrics for MS SQL database didn’t come directly from the SQL engine; instead, they were collected through a Sidecar or Exporter pattern.

• The Exporter is A specialized tool (like the sql_exporter or mssql-exporter) runs alongside the database. It connects to SQL Server, runs system queries (like SELECT * FROM sys.dm_os_process_memory), and converts the results into a format Prometheus can read (HTTP/text).

• Prometheus is configured to “scrape” (visit) the exporter’s endpoint every few seconds.

• In microservices-alerts.yaml, i defined a PrometheusRule. Prometheus takes the raw bytes collected from the exporter and runs the math written in the expr (expression) field:

Fronted deployment monitoring

Monitoring the “Health” of a URL (like checking if a website is up) is called Probing or Synthetic Monitoring. This was handled by the Prometheus Blackbox Exporter.

• Unlike standard metrics where a pod tells you its health, the Blackbox Exporter acts like an outside user. It tries to “hit” the URL you specify.

• It sends an HTTP GET request to the frontend service. If the service responds with a 200 OK status code, the probe returns a 1 (Success). If the site is down or returns a 500 Error, it returns a 0 (Failure).

• In your microservices-alerts.yaml, we have an alert named MvcFrontendDown. It looks specifically at the result of that probe:

- alert: MvcFrontendDown
  expr: probe_success{job="mvc-frontend-check"} == 0
  for: 2m

To find out what labels prometheus/alertmanager expects, run this command to find out:

kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.ruleSelector}'                                                       
# output
{"matchLabels":{"prometheus":"prometheus"}}    

It is telling the Operator: “I will only load rules that have the label prometheus: prometheus

For blackbox exporter (used to monitor frontend app URL) and services

kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorSelector}'
{"matchLabels":{"release":"prometheus"}}

Check if prometheus/alert manager rules are loaded

# restart prometheus/alert manager deployments after config changes
kubectl rollout restart statefulset alertmanager-prometheus-kube-prometheus-alertmanager -n monitoring
kubectl rollout restart statefulset prometheus-prometheus-kube-prometheus-prometheus -n monitoring
kubectl logs -n monitoring -l app.kubernetes.io/name=prometheus --tail=50 | grep -i "rule"
kubectl logs -n monitoring -l app.kubernetes.io/name=alertmanager --tail=50

Check if Aloy rules are loaded:

kubectl logs -n monitoring -l app.kubernetes.io/name=alloy --tail=100
# restart alloy after config changes
kubectl rollout restart ds alloy -n monitoring 

Subscribe now

Share

Leave a comment

I built this project as a comprehensive learning journey to master the intersection of Node.js development and modern Cloud Native operations. My goal was to move beyond basic tutorials and implement a real-world, production-ready environment using the 'Everything-as-Code' philosophy."

In this post, I’ll walk through a comprehensive project that bridges the gap between development and operations. We’ll look at a Node.js microservice backed by MongoDB, deployed via Helm, and automated through a custom Jenkins Shared Library that integrates with AWS Parameter Store.Check out the full source code on GitHub.

The project is built on the philosophy of Everything-as-Code.

1. Application Layer: A Node.js API.

2. Orchestration: Kubernetes managed via Helm charts.

3. Secrets Management: External Secrets Operator (ESO) syncing from AWS Parameter Store.

4. CI/CD Pipeline: A modular Jenkins Shared Library to avoid code duplication across multiple repositories.

Pipeline breakdown

When code is pushed to a branch that is not master and has no open Pull Request, only the baseline quality checks run. This is designed to give developers fast feedback without wasting registry space or deployment time.

• Stages that run:

  • Tests & Coverage: Spins up the temporary MongoDB, runs yarn coverage, and publishes the HTML report.

  • Analysis & Lint: Runs yarn lint and the SonarCloud scan. It will wait for the Quality Gate to pass.

  Everything else is skipped (Nexus, Docker, ECR, EKS).

Once we open a Pull Request (PR), Jenkins identifies this as a changeRequest(). The pipeline now adds the “Publishing” and “Validation” steps.

• Stages that run:

  • Tests & Coverage & Analysis & Lint (Always run).

  • Publish to Nexus: NPM package is published to private Nexus repository.

  • Build Docker Image: A Docker image is built using the Jenkins Build Number as a tag.

  • Image Analysis: The image is scanned for vulnerabilities (Trivy/Clair/etc.) with a focus on CRITICAL issues.

• Post-Action:

  • Because env.CHANGE_ID is now active, the post block executes the GitHub PR Comment logic. It collects the test results and coverage % and posts them directly onto your PR conversation so reviewers can see the stats without leaving GitHub.

After the PR is approved and merged, the pipeline runs on master. This is the only time the actual deployment takes place.

• Stages that run:

  • All CI stages (Tests, Lint, Sonar).

  • All PR stages (Nexus, Docker Build, Image Scan).

  • Approval Gate: The pipeline pauses. It waits up to 1 hour for a human to click “Deploy” in the Jenkins UI.

  • Push to ECR: Authenticates with AWS and pushes the latest and BUILD_NUMBER tagged images to Amazon ECR.

  • Deploy to EKS: Runs helm upgrade --install to update the Kubernetes cluster with the new image.

• Post-Action: * Since it’s no longer a PR (CHANGE_ID is null), it skips the GitHub comment and simply prints “Pipeline completed successfully!”

Instead of static YAML files, this project uses Helm to parameterize the environment. This allows us to use the same chart for Dev, Staging, and Production by simply swapping values.yaml.

We’ve implemented an HPA (app-hpa.yaml) that monitors CPU utilization. If the load spikes, Kubernetes automatically scales our Node.js replicas up to 10.

The deployment uses a RollingUpdate strategy, ensuring that at least two pods are always healthy while a new version is being rolled out.

A Headless Service handles internal communication for MongoDB, while a LoadBalancer exposes the Node.js API to the world.

Hardcoding secrets is a not a good practice in DevOps. This project utilizes the External Secrets Operator.

YAML

# From secrets.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
spec:
  secretStoreRef:
    name: aws-parameter-store
  data:
  - secretKey: password
    remoteRef:
      key: /my-aks-cluster/mongodb/root-password

By using this pattern, our Kubernetes secrets are automatically synced from AWS SSM, keeping sensitive data out of our Git history and ensuring that rotation is handled at the source.

Jenkins Shared Libraries

The standout feature of this setup is the Jenkins Shared Library. Instead of writing 100-line Jenkinsfiles for every service, we define the logic once in a centralized repository.

Why Shared Libraries?

• Standardization: Every microservice follows the same build, test, and deploy logic.

• Easier Maintenance: Updating a deployment step in the library instantly updates all connected pipelines.

• Readability: The application-level Jenkinsfile becomes a simple configuration call.

Shared libraries code is also on GitHub

To ensure our cluster doesn’t run out of memory, we define strict resource limits:

YAML

nodesApp:
  requests:
    cpu: "50m"
    memory: "64Mi"
  limits:
    cpu: "200m"
    memory: "128Mi"

We dynamically construct the MongoDB URL to support different namespaces and service names:

data:
  MONGODB_URL: {{ printf "mongodb://%s-mongodb-0.%s-mongodb:27017/%s" ... }}

By implementing this stack, we achieve:

• Scalability: The HPA ensures we only pay for the compute we use.

• Reliability: Readiness and Liveness probes ensure traffic only hits healthy pods.

• Portability: The Helm chart can be deployed to EKS, GKE, or AKS with minimal changes.

Subscribe now

Share

Leave a comment

I created a small project as part of my journey into learning DevOps,and the code is available on GitHub.

Here is a deep dive into the architecture and the automation logic behind it.

1. The Root Orchestrator (main.tf)

The root main.tf is the “brain” of the project. Instead of just creating resources, it manages the lifecycle and dependencies between the different layers of the stack.the root main.tf doesn't just deploy a cluster; it bootstraps an entire environment. By using the Helm provider directly in the root, we have created a tight coupling between the EKS cluster's health and the application's availability.

1. Bridge Between Infrastructure and App

The root file calls module.eks first. Once the cluster is ready, it passes the resulting cluster endpoints and security credentials into the module.addons. This ensures that we never try to install software before the “hardware” (the VPC and EKS) exists.

2. Automated Secret Management

This is where the MongoDB security is defined. Inside the root main.tf, we use the random_password resource to generate a 16-character complex password.

• The password is never hardcoded.

• Terraform immediately saves this password into the AWS SSM Parameter Store as a SecureString. This allows your developers or other applications to retrieve the password securely via the AWS Console or CLI without ever seeing it in the git logs.

3. The MongoDB Deployment (Helm)

The actual creation of the MongoDB database happens here using the Helm Provider.

• It configures MongoDB to use the gp3 StorageClass (created in the EKS module), ensuring that if a database pod restarts, your data is safe on an AWS EBS volume.

• It maps the generated random password directly into the Helm chart values.

• It ensures MongoDB is deployed into a clean, dedicated namespace to keep the cluster organized.

2. The EKS Module (module_eks)

This is the foundation of the environment. It handles the “Heavy Lifting” of AWS networking and compute.

• Provisions a dedicated VPC with public subnets, an Internet Gateway, and optimized route tables.

• Deploys the EKS cluster using API and ConfigMap authentication mode for modern access management.

• Uses t3.medium instances with automated scaling to balance cost and performance.

• Configures the OIDC provider and IAM roles for service accounts, allowing Kubernetes pods to interact with AWS services securely.

• Automatically installs the EBS CSI Driver and sets up gp3 as the default StorageClass.

3. The Add-ons Module (module_addons)

Once the cluster is up, the Add-ons module “decorates” it with necessary services.

• This module is the “Security Vault” for Kubernetes cluster. Instead managing infrastructure, it focuses on how applications safely consume sensitive data from AWS.

  We are using the new EKS Pod Identity feature rather than the older IRSA (OIDC) method.

  • We create a specialized IAM role (eso-role) that trusts the pods.eks.amazonaws.com service.

  • The aws_eks_pod_identity_association creates a direct link between the Kubernetes Service Account in the external-secrets namespace and our AWS IAM role.

  The module defines a strict IAM policy (eso_secrets_policy) that grants the cluster permission to reach out and “read” from two specific AWS services:

  • AWS Secrets Manager: For high-rotation application secrets.

  • AWS SSM Parameter Store: For configuration and passwords (like MongoDB root password).

  We deploy the External Secrets Operator via a Helm chart.

  • ESO acts as a controller that synchronizes secrets from AWS into native Kubernetes Secrets.

  • By setting installCRDs = true, Terraform automatically prepares the cluster to understand custom secret resources.

• The module concludes with a null_resource that executes a local-exec provisioner.

  • It applies a ClusterSecretStore manifest using kubectl.

  • This tells the entire cluster: “If you need a secret from AWS Parameter Store in this region, use this specific provider.” This allows MongoDB deployment to automatically fetch its password from AWS SSM without you ever having to manually create a secret in Kubernetes.

4. The CI/CD Pipeline (Jenkinsfile)

The pipeline is what turns “code” into “infrastructure.”

• Runs fmt and validate to catch syntax errors before they ever reach AWS.

• Uses AquaSec Trivy to scan the Terraform code for misconfigurations (like open S3 buckets or overly permissive IAM roles).

• A custom Groovy script extracts the terraform plan and posts it as a comment on the GitHub Pull Request. This allows reviewers to see exactly what will change before they hit “Merge.”

• For the production (master) branch, the pipeline pauses. A human must manually click “Deploy” to trigger the final apply.

• Uses tfplan binary files to ensure that the exact plan approved in the PR is the one applied to AWS.

Subscribe now

Share

Leave a comment

In modern cloud-native environments, managing sensitive data like database passwords or API keys can be a logistical nightmare. While Kubernetes has a native Secret object, it lacks a secure, automated way to pull those secrets from external vaults like AWS Secrets Manager or SSM Parameter Store.

This is where the External Secrets Operator (ESO) comes in.

What is External Secrets Operator (ESO)?

The External Secrets Operator (ESO) is a Kubernetes operator that integrates external secret management systems into Kubernetes. Instead of manually creating Kubernetes Secrets, you define where the secret lives in cloud provider (like AWS), and ESO automatically fetches it, synchronizes it, and injects it into cluster as a standard Kubernetes Secret.

How it Works:

1. Poll & Fetch: ESO monitors cloud provider (AWS, GCP, Azure, etc.) for changes to specific secrets.

2. Mapping: It uses a custom resource called a ClusterSecretStore to define how to connect to the cloud provider (e.g., using specific AWS regions and IAM roles).

3. Synchronization: It creates a native Kubernetes Secret. If you update the password in the AWS Console, ESO detects the change and updates the Kubernetes Secret automatically.

This example deployment uses a modular Terraform approach to set up ESO with EKS Pod Identity, which is the most secure way to grant permissions to pods without managing long-lived IAM keys.

Security First: IAM Roles and Policies

To implement the External Secrets Operator (ESO) with EKS Pod Identity, the configuration is spread across IAM roles, policies, and the Helm deployment.To allow ESO to “read” from AWS, we first create an IAM role with a trust policy specifically for EKS Pod Identity.

# IAM policy allowing ESO to read from Secrets Manager and SSM
resource "aws_iam_policy" "eso_secrets_policy" {
  name = "${var.cluster_name}-eso-secrets-policy"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["ssm:GetParameter", "ssm:GetParameters", "ssm:GetParametersByPath"]
        Resource = "arn:aws:ssm:${var.region}:${data.aws_caller_identity.current.account_id}:parameter/*"
      }
    ]
  })
}

# ESO IAM Role using Pod Identity trust policy
resource "aws_iam_role" "eso_role" {
  name = "${var.cluster_name}-eso-role"
  
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "pods.eks.amazonaws.com" }
      Action    = ["sts:AssumeRole", "sts:TagSession"]
    }]
  })
}

# Attach the policy to the role
resource "aws_iam_role_policy_attachment" "eso_attach" {
  role       = aws_iam_role.eso_role.name
  policy_arn = aws_iam_policy.eso_secrets_policy.arn
}

EKS Pod Identity Association

This section links the IAM role to the specific Kubernetes ServiceAccount used by the operator. Note that the Pod Identity Agent must be installed as an EKS add-on for this to function.

# The Pod Identity Agent add-on must be active on the cluster
resource "aws_eks_addon" "pod_identity_agent" {
  cluster_name = var.cluster_name
  addon_name   = "eks-pod-identity-agent"
}

# Link the IAM role to the 'external-secrets' ServiceAccount in the 'external-secrets' namespace
resource "aws_eks_pod_identity_association" "eso_association" {
  cluster_name    = var.cluster_name
  namespace       = "external-secrets"
  service_account = "external-secrets"
  role_arn        = aws_iam_role.eso_role.arn
}

We use the aws_eks_pod_identity_association to link the IAM role we created to the specific external-secrets ServiceAccount in the Kubernetes cluster. This ensures that only the ESO pods have the permission to fetch secrets.The operator itself is deployed using the official Helm chart. We ensure that Custom Resource Definitions (CRDs) are installed so Kubernetes understands the new "ExternalSecret" objects.

Helm Deployment and Cluster Configuration

This code deploys the operator and configures the ClusterSecretStore to tell ESO to look in the AWS Parameter Store for secrets.

# Deploy ESO via Helm
resource "helm_release" "external_secrets" {
  name             = "external-secrets"
  repository       = "https://charts.external-secrets.io"
  chart            = "external-secrets"
  namespace        = "external-secrets"
  create_namespace = true
  version          = "0.9.11"

  values = [
    yamlencode({
      installCRDs = true
      serviceAccount = {
        create = true
        name   = "external-secrets"
      }
    })
  ]
  
  depends_on = [aws_eks_pod_identity_association.eso_association]
}

# Apply the ClusterSecretStore manifest using local-exec (PowerShell example)
resource "null_resource" "apply_manifest" {
  depends_on = [helm_release.external_secrets]

  provisioner "local-exec" {
    interpreter = ["PowerShell", "-Command"]
    command = <<EOT
      aws eks update-kubeconfig --region ${var.region} --name ${var.cluster_name}
      
      $manifest = @"
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-parameter-store
spec:
  provider:
    aws:
      service: ParameterStore
      region: ${var.region}
"@
      $manifest | Out-File -FilePath manifest.yaml -Encoding ascii
      kubectl apply -f manifest.yaml
      Remove-Item manifest.yaml
EOT
  }
}

Finally, we apply a ClusterSecretStore manifest. This is the configuration that tells ESO: “Whenever you need to find a secret, look in the AWS Parameter Store in this specific region”.

While the External Secrets Operator (ESO) has a role to fetch secrets, your application needs its own role to authorize the retrieval of that specific secret data through the ESO mechanism.

# IAM Policy allowing the app to read specific parameters
resource "aws_iam_policy" "node_app_secrets_policy" {
  name        = "${var.cluster_name}-node-app-secrets-policy"
  description = "Allows Node app to read its specific MongoDB password from SSM"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ssm:GetParameter"
        ]
        # Restrict specifically to the path used by the app
        Resource = "arn:aws:ssm:${var.region}:${data.aws_caller_identity.current.account_id}:parameter/${var.cluster_name}/mongodb/*"
      }
    ]
  })
}

# IAM Role for the Application
resource "aws_iam_role" "node_app_role" {
  name = "${var.cluster_name}-node-app-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "pods.eks.amazonaws.com" }
      Action    = ["sts:AssumeRole", "sts:TagSession"]
    }]
  })
}

# Attach the policy to the application role
resource "aws_iam_role_policy_attachment" "node_app_attach" {
  role       = aws_iam_role.node_app_role.name
  policy_arn = aws_iam_policy.node_app_secrets_policy.arn
}

# Pod Identity Association for the Node App
resource "aws_eks_pod_identity_association" "node_app_association" {
  cluster_name    = var.cluster_name
  namespace       = var.namespace
  service_account = "node-app-sa" # Matches the SA in your deployment
  role_arn        = aws_iam_role.node_app_role.arn
}

How to “Get” the Secret: The ExternalSecret Manifest

Once the IAM roles are ready, we define an ExternalSecret custom resource. This is the bridge that tells ESO: “Go to this AWS path, find the value, and put it in a Kubernetes Secret named mongodb-credentials.”

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: mongodb-pass-sync
  namespace: default
spec:
  refreshInterval: 1h             # How often to sync from AWS
  secretStoreRef:
    name: aws-parameter-store     # Defined in your ClusterSecretStore
    kind: ClusterSecretStore
  target:
    name: mongodb-credentials     # The name of the K8s Secret to create
    creationPolicy: Owner
  data:
  - secretKey: password           # The key inside the K8s Secret
    remoteRef:
      key: /my-aks-cluster/mongodb/root-password  # The path in AWS SSM

Finally, we map that synchronized Kubernetes Secret your application's environment variables within deployment manifest (password for Mongo DB).

# Deployment example
spec:
  template:
    spec:
      containers:
      - name: node-app
        env:
        - name: MONGODB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mongodb-credentials
              key: password
        # This password is then used to build your MONGODB_URL
        - name: MONGODB_URL
          value: "mongodb://root:$(MONGODB_PASSWORD)@mongodb.default.svc.cluster.local:27017/node-app-db?authSource=admin&ssl=false&directConnection=true"

Summary:

• Terraform generates a random password and stores it in AWS SSM Parameter Store.

• IAM Roles & Pod Identity give permission to ESO to read from SSM.

• ExternalSecret tells ESO which AWS path to watch.

• ESO creates a local Kubernetes Secret with the fetched value.

• The Pod injects that secret as an environment variable (MONGODB_PASSWORD).

Why to use this ?

• Security: No secrets are stored in your Git repository or Terraform state in plain text.

• Automation: Change a password in AWS, and your app receives the update without a manual kubectl apply.

• Compliance: Centralize all your audit logs in AWS CloudTrail to see exactly who accessed which secret and when.

Synchronizing secret updates from AWS parameter store to Kubernetes

Updating a password in the AWS Parameter Store doesn’t automatically reach your application pods because of how Kubernetes and the External Secrets Operator (ESO) work. The sync process happens in two distinct stages.

Stage 1: AWS to Kubernetes (ESO Sync)

The External Secrets Operator periodically “polls” AWS to check for updates. By default, this happens every 1 hour, unless specified a different refreshInterval in ExternalSecret manifest.

To Sync Immediately (Force Refresh)

If you don’t want to wait for the next polling cycle, you can force ESO to reconcile immediately by “touching” the ExternalSecret resource. The most common way is to add or update an annotation:

kubectl annotate externalsecret <your-es-name> force-sync=$(date +%s) --overwrite
Check the status of the sync to ensure it pulled the new value:

kubectl describe externalsecret <your-es-name>

Look for: Status: Ready, Message: Secret was synced

Stage 2: Kubernetes Secret to Application Pods

Even after the Kubernetes Secret is updated, your application pods likely still have the old password in memory.

• If using Environment Variables: Kubernetes does not update environment variables in running containers. You must restart the pods to pick up the new value.

  kubectl rollout restart deployment <your-app-deployment>

• If using Mounted Volumes: Kubernetes will eventually update the file in the pod (usually within 60 seconds), but your application code must be designed to “hot-reload” that file from disk. Most applications are not, so a restart is usually still required.

To make this truly hands-off, many DevOps teams use a tool called Reloader. It watches Secrets and automatically triggers a rollout restart of Deployment whenever the Secret changes.

If you have Reloader installed, you just add this annotation to your Deployment:

metadata:
  annotations:
    reloader.stakater.com/auto: "true"

Share

Leave a comment

For years, the standard way to connect a private server (like a Jenkins node) to AWS was to create an IAM User, generate an access_key_id and secret_access_key, and hide them in a config file.Those keys are permanent. If your server is compromised, those keys are stolen. If you forget to rotate them, you’re out of compliance. If you’re using a server with no public IP, you can’t even use OIDC easily because AWS can’t “call home” to verify your identity.

AWS IAM Roles Anywhere treats your on-premises server like an EC2 instance. Instead of static keys, it uses Public Key Infrastructure (PKI). Your server proves its identity using a digital certificate, and AWS hands back temporary, short-lived credentials.

To set this up, you just need:

1. A Trust Anchor: A Root Certificate Authority (CA) that you tell AWS to trust.

2. An IAM Role: A role with a Trust Policy allowing rolesanywhere.amazonaws.com to assume it.

3. A Profile: A configuration in AWS that links your Trust Anchor to your IAM Role.

4. Client Certificates: A private key and certificate on your Jenkins server, signed by your Root CA.

5. AWS Signing Helper: A small open-source tool from AWS that handles the “handshake.”

The Workflow

1. Jenkins wants to run a Terraform plan.

2. The AWS CLI calls the Signing Helper.

3. The Helper signs a request using the local Private Key.

4. AWS verifies the signature against the Trust Anchor and sends back temporary keys (valid for 1 hour).

5. Jenkins performs the task and the keys expire automatically.

Benefits vs. Disadvantages

The Benefits (Why you should do this)

• Zero Static Secrets: No AWS_ACCESS_KEY_ID is ever stored on your disk.

• No Inbound Access Required: Perfect for servers behind strict firewalls or NAT.

• Auditability: Every login is logged in AWS CloudTrail, showing exactly which certificate was used.

• Automated Lifecycle: No more "Rotating Keys".When a certificate expires, the access simply stops.

The Disadvantages

• Initial Complexity: Setting up a Private CA (OpenSSL or AWS Private CA) takes more effort than clicking “Create Access Key.”

• Certificate Management: You are now responsible for renewing certificates before they expire. If your Root CA expires, all your servers lose access.

• Regionality: Roles Anywhere is a regional service. If you operate in multiple regions, you may need to set up Trust Anchors in each.

Creating Private CA and certificates

# Switch to the jenkins user
sudo su -s /bin/bash jenkins

# Create a secure directory
mkdir -p ~/.aws_certs
cat <<EOF > ca.conf
[ v3_ca ]
authorityKeyIdentifier=keyid,issuer
basicConstraints=critical,CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
subjectKeyIdentifier = hash
EOF
 
chmod 700 ~/.aws_certs
cd ~/.aws_certs

# Generate a root CA
openssl genrsa -out rootCA.key 2048
# Generate the Root Certificate with the v3_ca extensions
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 3650 \
  -out rootCA.pem \
  -config <(cat /etc/ssl/openssl.cnf ca.conf) \
  -extensions v3_ca \
  -subj "/C=US/ST=State/L=City/O=MyOrg/CN=MyPrivateRootCA"

cat <<EOF > jenkins_ext.conf
authorityKeyIdentifier=keyid,issuer
basicConstraints=critical,CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
subjectKeyIdentifier = hash
EOF

# Create the Jenkins Private Key and CSR
openssl genrsa -out jenkins.key 2048
openssl req -new -key jenkins.key -out jenkins.csr -subj "/CN=jenkins-prod-01"

 
# Sign the Jenkins cert with Root CA
openssl x509 -req -in jenkins.csr -CA rootCA.pem -CAkey rootCA.key \
  -CAcreateserial -out jenkins.crt -days 365 -sha256 \
  -extfile jenkins_ext.conf


# Creating Jenkins server certificates
 
openssl x509 -req -in jenkins.csr -CA rootCA.pem -CAkey rootCA.key \
  -CAcreateserial -out jenkins.crt -days 365 -sha256 \
  -extfile jenkins_ext.conf

Above commands set up a Private Mini-Certificate Authority (CA) on your server and use it to issue a unique “Security ID” (Certificate) to Jenkins so it can talk to AWS.

Here is the breakdown by section:

1. Environment Setup

• sudo su -s /bin/bash jenkins: Switches your current session to the jenkins user so all files are owned by the right account.

• mkdir & chmod 700: Creates a “vault” folder and locks it down so only the jenkins user can read the sensitive keys inside.

2. The Root CA (The "Stamp")

• cat <<EOF > ca.conf: Writes the rules for your "Master Authority" (defining that it’s a CA).

• openssl genrsa (rootCA.key): Creates the Master Private Key (the “seal” used to sign everything).

• openssl req -x509 (rootCA.pem): Creates the actual Root Certificate—this is what you upload to AWS to tell them, “I trust any certificate signed by this file.”

3. Jenkins Identity Request (The "Application")

• cat <<EOF > jenkins_ext.conf: Defines the specific “badges” (Digital Signature, Client Auth) AWS requires for a server to log in.

• openssl genrsa (jenkins.key): Creates a unique private key just for the Jenkins server.

• openssl req -new (jenkins.csr): Creates a Certificate Signing Request—essentially an "application form" that includes Jenkins’ identity.

4. Issuing the ID (The "Signing”)

• openssl x509 -req: This is the final step where the Root CA "stamps" the Jenkins Application. It produces jenkins.crt, which is the actual ID card Jenkins will show to AWS to get temporary permissions.

Creating Trusted anchor

In AWS console, in Search bar,type Roles anywhere-Create Trust anchor.

Select "External certificate bundler" and copy content of root CA file (rootCA.pem)

Now click "Create a new role"-Custom trust policy.We need a role (Identity) that AWS "anywhere" is allowed to give to Jenkins server.

Paste this code:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "rolesanywhere.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession",
                "sts:SetSourceIdentity"
            ]
        }
    ]
}

attach desired policy and give role a name (Jenkins-RolesAnywhere-Role). Save Role ARN.

Create a Profile (The Connection)

The Profile bridges Trust Anchor and Role.Click create a profile.

Select the role we just created (Jenkins-RolesAnywhere-Role),save Profile and Trusted anchor ARNs

To obtain temporary security credentials from AWS Identity and Access Management Roles Anywhere, we need credential helper tool that IAM Roles Anywhere provides.

wget https://rolesanywhere.amazonaws.com/releases/1.7.3/X86_64/Linux/Amzn2023/aws_signing_helper
chmod +x aws_signing_helper && mv aws_signing_helper /usr/local/bin/

# testing access

/usr/local/bin/aws_signing_helper credential-process \
  --certificate ~/.aws_certs/jenkins.crt \
  --private-key ~/.aws_certs/jenkins.key \
  --trust-anchor-arn Trusted-Anchor-ARN \
  --profile-arn profile-ARN \
  --role-arn Role-ARN

If all is good,output should be long JSON output.

Configuring the AWS CLI to use the Helper

We need to tell the AWS CLI to "call" the signing helper whenever it needs to do something.

sudo su -s /bin/bash jenkins
mkdir -p ~/.aws

cat <<EOF > ~/.aws/config
[profile roles-anywhere]
region = eu-central-1
credential_process = /usr/local/bin/aws_signing_helper credential-process --certificate /var/lib/jenkins/.aws_certs/jenkins.crt --private-key /var/lib/jenkins/.aws_certs/jenkins.key --trust-anchor-arn arn:aws:rolesanywhere:eu-central-1:477568783935:trust-anchor/017451d4-1bdf-4ccb-88dd-86a07cdab61b --profile-arn arn:aws:rolesanywhere:eu-central-1:477568783935:profile/6f2e004f-cc1b-4d60-8047-0f301c06c3d5 --role-arn arn:aws:iam::477568783935:role/Jenkins-RolesAnywhere-Role
EOF

Jenkins pipeline example how to use Role anywhere

 pipeline {
    agent any
    environment {
        // Force the AWS CLI to use the profile we created
        AWS_PROFILE = 'roles-anywhere'
    }
    stages {
        stage('AWS Task') {
            steps {
                // No 'withCredentials' block needed! 
                // The AWS CLI will trigger the signing helper automatically.
                sh 'aws s3 cp my-artifact.zip s3://my-jenkins-backups/'
            }
        }
    }
}

Subscribe now

Share

Leave a comment

Connecting Jenkins to Azure Kubernetes Service (AKS) often leads developers toward using Client Secrets (passwords). However, for a “Least Privilege” security model, X.509 Certificates are the gold standard. In this post, I’ll show you how to bypass common plugin issues and build a robust, secure CI/CD pipeline.

Most tutorials suggest the Azure Credentials plugin, but it can often fail with MissingPropertyException or null path errors when handling certificates. Here i used more reliable method: the Secret File binding.

Jenkins server is located outside Azure (in this case, on-prem Hyper-V server as VM)

Create service principal with right to push images to Azure container registry

az ad sp create-for-rbac --name jenkins-runner \
                         --role AcrPush \
                         --scopes $(az acr show --name myregistry205 --query id -o tsv) \
                         --create-cert

Output should be like this

“appId”: “33sasssS”,

  “displayName”: “jenkins-runner”,

  “fileWithCertAndPrivateKey”: “/var/lib/jenkins/tmpfzcl619j.pem”,

  “password”: null,

  “tenant”: “123-4445-tttttt”

On Jenkins,store content of PEM file as secret file

Credentials-Kind: Secret file

Store TenantID and Azure Service principal AppID as Jenkins secrets (Credential type-Secret text)

pipeline {
    agent any
   
    environment {
        REGISTRY_NAME = "myregistry205"
        IMAGE_NAME = "myapp"
        
    }
   stage('Build and push docker image') {
            steps {
                  withCredentials([file(credentialsId: 'azure-cert-pem', variable: 'CERT_FILE'),
                                  string(credentialsId: 'tenant_id', variable: 'TENANT_ID'),
                                  string(credentialsId: 'app_id', variable: 'CLIENT_ID')])
                {
                    script {
                        def fullImage = "${env.REGISTRY_NAME}.azurecr.io/${env.IMAGE_NAME}:${env.BUILD_NUMBER}"
                       
                        // 1. Build your image
                        sh "docker build -t ${fullImage} ./docker"
                        
                        sh '''
                            az login --service-principal \
                                -u ${env.CLIENT_ID} \
                                -t ${env.TENANT_ID} \
                                --certificate "${CERT_FILE}"
                        '''
                
                        sh "az acr login --name ${env.REGISTRY_NAME}"
                        sh "docker push ${fullImage}"
                      }
                }
            }
        }
     }
   
 post {
        success {
            echo "Image pushing to ACR successful!"
            
        }
        
        failure {
            echo "Pushing failed"
        }

        always {
            script {
                // 1. Always logout to clear the local Azure CLI token
                sh "az logout || true"
            }
            
            // 3. Clean the workspace files (keep this as the last step)
            cleanWs()
        }
    }
}

The az login allows the Azure CLI to prove its identity so it can gain permission to push images to ACR or modify the AKS cluster.

withCredentials([
    file(credentialsId: 'azure-cert-pem', variable: 'CERT_FILE'),
    string(credentialsId: 'tenant_id', variable: 'TENANT_ID'),
    string(credentialsId: 'app_id', variable: 'CLIENT_ID')
])

Jenkins fetches secrets from its encrypted database.

• For the certificate, Jenkins takes the binary uploaded data (PEM file content), writes it to a temporary file on the disk (e.g., /tmp/secret123.pem), and stores that path in the CERT_FILE variable.

• As soon as the code inside the { } block finishes, Jenkins deletes that temporary file

az login --service-principal \
    -u "$CLIENT_ID" \
    -t "$TENANT_ID" \
    --certificate "$CERT_FILE"

• --service-principal: Tells Azure we're an application rather than a human user with an email/password.

• -u “$CLIENT_ID”: Application (Client) ID. This is like username.

• -t “$TENANT_ID”: Azure Directory ID. This tells the CLI which specific Azure organization to log into.

• --certificate “$CERT_FILE”: Instead of a password, the CLI uses the private key inside pem file to sign a request. Azure validates this signature against the public key you uploaded to the App Registration.

Once this command succeeds:

1. An Access Token is stored in the jenkins user’s home directory (~/.azure/msal_token_cache.bin).

2. Subsequent commands like az acr login or az aks get-credentials look at that token to see if they are allowed to run.

3. Because we are logged in as a Service Principal, these commands will run non-interactively (no “Please open a browser” messages).

   Subscribe now

   Share

   Leave a comment

Unlike traditional Helm installs, using the ECK Operator allows Kubernetes to manage Elasticsearch as a native resource. We’ve also swapped Fluentd for Fluent Bit to keep our resource footprint tiny, ensuring that logging doesn’t eat into our application’s CPU and RAM.

We use volumeBindingMode: Immediate to ensure storage is ready the moment Elasticsearch requests it.

Using the ECK Operator, our elasticsearch.yaml is surprisingly simple. The operator automatically handles the generation of internal TLS certificates and the default elastic user password.

• Highlights: We’ve limited the JVM heap to 512MB (ES_JAVA_OPTS) to stay cost-effective while keeping the deployment stable with 10Gi of EBS storage.

Step 3: Visualizing with Kibana

Kibana is our window into the logs. In kibanaDeployment.yaml, we don’t need to manually provide Elasticsearch URLs; the elasticsearchRef field tells the Operator to link them automatically.

• Access: To make it reachable, we use kibanaService.yaml which creates an AWS LoadBalancer, giving us a public URL to access our dashboards.

Step 4: The Log Shipper (Fluent Bit)

This is where the heavy lifting happens. We deploy Fluent Bit as a DaemonSet, meaning one pod runs on every single worker node in the EKS cluster.

• fluent-bit-DaemonSet.yaml includes an initContainer. It “pings” Elasticsearch and waits for it to be fully healthy before Fluent Bit starts shipping logs.

• Security: We don’t hardcode passwords. Fluent Bit pulls the ELASTIC_PASSWORD directly from the Secret that the ECK Operator created.

Key Configuration: Fluent Bit Logic

The magic is in the fluent-bit-config.yaml. We’ve broken it down into four stages:

Input Section (input-kubernetes.conf)

This section defines how Fluent Bit reads the raw log files from the Kubernetes nodes.

• Name tail: Tells Fluent Bit to use the “tail” input plugin, which reads files like the Linux tail command.

• Path /var/log/containers/*.log: Specifies the exact location on the node where Kubernetes stores container logs.

• Tag kube.*: Assigns a tag starting with kube. to all logs collected from this path, allowing them to be targeted by specific filters later.

• Parser docker: Uses a predefined JSON parser to structure raw logs coming from Docker/CRI-O.

• Mem_Buf_Limit 5MB: Limits memory usage for this input to 5MB to prevent the Fluent Bit agent from consuming too much RAM.

• Skip_Long_Lines On: Skips lines that exceed the buffer size instead of crashing the plugin.

• DB /var/log/flb_kube.db: Maintains an SQLite database to track which parts of the log files have already been read, ensuring no logs are lost if the pod restarts.

• Read_from_Head False: Starts reading from the end of the file (new logs) rather than the beginning of old files.

2. Filter Section (filter-kubernetes.conf)

This section enriches raw logs with Kubernetes-specific information like pod names, namespaces, and labels.

• Name kubernetes: Uses the Kubernetes filter plugin to communicate with the API server.

• Match kube.*: Applies this filter only to logs that were tagged with kube. in the input step.

• Kube_URL / Kube_CA_File / Kube_Token_File: Standard credentials used by the filter to securely authenticate with the Kubernetes API.

• Kube_Tag_Prefix kube.var.log.containers.: Helps the filter extract the pod name and namespace directly from the file path.

• Merge_Log On / Merge_Log_Key log_processed: If the log is a JSON string, it merges those fields into the main log record under the key log_processed.

• Keep_Log Off: Discards the original unparsed “log” field after successful merging to save space.

3. Output Section (output-elasticsearch.conf)

This section defines where the processed logs are sent for storage and visualization.

• Name es: Uses the Elasticsearch output plugin.

• Host / Port: Points to the Elasticsearch service address and port (9200) within the cluster.

• Index / Logstash_Prefix k8s-logs: Sets the naming convention for the indices created in Elasticsearch.

• tls On / tls.verify On / tls.ca_file: Enables encrypted communication and specifies the certificate to verify the Elasticsearch server.

• HTTP_User / HTTP_Passwd: Authenticates with Elasticsearch using the default elastic user and a password pulled from a secure environment variable.

• Logstash_Format On: Formats the index names as prefix-YYYY.MM.DD, which is the standard format used by Kibana.

• Retry_Limit False: Tells Fluent Bit to keep trying to send logs if the connection to Elasticsearch is temporarily lost.

Manifest files are available here

Prerequisites

Install the ECK Operator

The Operator itself is installed cluster-wide. We’ll use the official manifest files from Elastic.These teach your Kubernetes cluster what an Elasticsearch and Kibana resource looks like.

kubectl create -f https://download.elastic.co/downloads/eck/3.2.0/crds.yaml

This deploys the actual Operator logic (the controller) into the elastic-system namespace.

kubectl apply -f https://download.elastic.co/downloads/eck/3.2.0/operator.yaml

Verify Operator Status:

kubectl get -n elastic-system pods
NAME                 READY   STATUS    RESTARTS   AGE
elastic-operator-0   1/1     Running   0          135m

Deploy EFK stack:

kubectl apply -f namespace.yaml
kubectl apply -f storageClass.yaml
kubectl apply -f elasticsearch.yaml
kubectl apply -f kibanaService.yaml
kubectl apply -f kibanaDeployment.yaml
kubectl apply -f fluent-bit-RBAC.yaml
kubectl apply -f fluent-bit-config.yaml
kubectl apply -f fluent-bit-DaemonSet

Check deployment:

kubectl get all -n efklog
NAME                                     READY   STATUS    RESTARTS   AGE
pod/elasticsearch-logging-es-default-0   1/1     Running   0          108m
pod/fluent-bit-9qhsl                     1/1     Running   0          68m
pod/fluent-bit-cpvfb                     1/1     Running   0          68m
pod/kibana-logging-kb-658c9bb7bb-fcxjt   1/1     Running   0          79m

NAME                                             TYPE           CLUSTER-IP      EXTERNAL-IP                                                                PORT(S)          AGE
service/elasticsearch-logging-es-default         ClusterIP      None            <none>                                                                     9200/TCP         108m
service/elasticsearch-logging-es-http            ClusterIP      10.100.34.105   <none>                                                                     9200/TCP         108m
service/elasticsearch-logging-es-internal-http   ClusterIP      10.100.187.1    <none>                                                                     9200/TCP         108m
service/elasticsearch-logging-es-transport       ClusterIP      None            <none>                                                                     9300/TCP         108m
service/kibana-logging-external                  LoadBalancer   10.100.17.186   a37a47b7e1cd2441fa8d78dfb0e9b041-64884497.eu-central-1.elb.amazonaws.com   5601:31515/TCP   105m
service/kibana-logging-kb-http                   ClusterIP      10.100.38.21    <none>                                                                     5601/TCP         79m

NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/fluent-bit   2         2         2       2            2           <none>          76m

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kibana-logging-kb   1/1     1            1           79m

NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/kibana-logging-kb-658c9bb7bb   1         1         1       79m

NAME                                                READY   AGE
statefulset.apps/elasticsearch-logging-es-default   1/1     108m

Get Kibana password:

kubectl get secret elasticsearch-logging-es-elastic-user -n efklog -o jsonpath=’{.data.elastic}’ | base64 --decode

Login to Kibana Web Gui using Load Balancer service

https://a37a47b7e1cd2441fa8d78dfb0e9b041-64884497.eu-central-1.elb.amazonaws.com:5601 in this case, username elastic

password from above step

kubectl get svc -n efklog
NAME                                     TYPE           CLUSTER-IP      EXTERNAL-IP                                                                PORT(S)          AGE
elasticsearch-logging-es-default         ClusterIP      None            <none>                                                                     9200/TCP         110m
elasticsearch-logging-es-http            ClusterIP      10.100.34.105   <none>                                                                     9200/TCP         110m
elasticsearch-logging-es-internal-http   ClusterIP      10.100.187.1    <none>                                                                     9200/TCP         110m
elasticsearch-logging-es-transport       ClusterIP      None            <none>                                                                     9300/TCP         110m
kibana-logging-external                  LoadBalancer   10.100.17.186   a37a47b7e1cd2441fa8d78dfb0e9b041-64884497.eu-central-1.elb.amazonaws.com   5601:31515/TCP   107m
kibana-logging-kb-http                   ClusterIP      10.100.38.21    <none>                                                                     5601/TCP         81m

When you access Kibana, the certificate warning occurs because the ECK (Elastic Cloud on Kubernetes) Operator automatically generates a self-signed CA to secure communication. Since your browser does not recognize or trust this internal Elastic CA, it flags the connection as “Not Secure.”

To resolve this and use your own trusted certificate (e.g., from Let’s Encrypt or your organization’s CA), you need to provide your certificate as a Kubernetes Secret and update the Kibana resource.First, you must package your certificate (tls.crt) and private key (tls.key) into a Secret within the efklog namespace.

kubectl create secret tls kibana-custom-cert \
  --cert=path/to/your/cert.crt \
  --key=path/to/your/key.key \
  -n efklog

We need to modify Kibana manifest (similar to elasticsearch.yaml structure) to tell the Operator to use your secret instead of generating its own.

Add the http section under spec:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-logging
  namespace: efklog
spec:
  version: 8.12.2
  count: 1
  elasticsearchRef:
    name: elasticsearch-logging
  http:
    tls:
      selfSignedCertificate:
        disabled: true  # Disable the auto-generated cert
      certificate:
        secretName: kibana-custom-cert # Use your created secret

In Kibana Dashboard-Kibana-Data views-Create Data view

• Name: Enter a name (e.g., k8s-logs-new).

• Index pattern: Enter k8s-logs-*. This must match the Logstash_Prefix defined in your output-elasticsearch.conf.

• Timestamp field: Select @timestamp from the dropdown menu.

• Click Save data view to Kibana.

• Click the Hamburger Menu (☰) and go to Analytics > Discover.

• Ensure your new data view is selected in the dropdown on the left.

• You should now see a clean timeline of logs coming from your EKS cluster.

Subscribe now

Share

Leave a comment

The code provided defines a comprehensive Azure DevOps Continuous Integration/Continuous Deployment (CI/CD) pipeline that automates the deployment of both infrastructure (Infrastructure as Code) and a .NET web application to Azure, following a secure Git flow branching strategy (develop and main).

The overall process is to:

1. Build and Test the application.

2. Scan the infrastructure code for security issues.

3. Deploy the infrastructure (Azure SQL, Web App networking, etc.) using Terraform.

4. Deploy the web application to the Azure Web App, using a Blue/Green (slot swap) strategy for zero-downtime deployment.

5. Secure the application’s database connection using Managed Identity (MSI) to eliminate the need for storing secrets.

1. CI/CD Pipeline Flow

The pipeline is structured into multiple stages that run conditionally based on the branch:

A. CI Stages (Runs on both develop and main)

• CI_Build_and_Tests:

  • Installs the .NET 8.0 SDK.

  • Restores NuGet packages and runs unit tests.

  • Calculates and publishes code coverage results and generates an HTML report.

  • Builds and publishes the final web application as a deployable artifact named drop.

• Static_Analysis_Checks:

  • Installs TFLint and Checkov tools.

  • Runs TFLint to lint the Terraform files for syntax and style compliance.

  • Runs Checkov to scan the infrastructure code for cloud security and compliance errors.

  • Publishes the security reports (JSON/HTML) as artifacts for review.

B. Development Environment Flow (Runs on develop branch)

• Deploy_Development_Infra:

  • Installs Terraform and the sqlcmd utility (required for database setup).

  • Runs terraform init, plan, and apply to provision the Development infrastructure using Azure Service Principal credentials (ARM_CLIENT_ID, etc.).

• Deploy_Development_Config:

  • Retrieves the provisioned infrastructure outputs (SQL FQDN, DB Name, Web App Name) from Terraform state.

  • Assembles the SQL Connection String and sets it as an output variable (CONN_STRING).

    • It accepts User ID=$SQL_USER;Password=$SQL_PASSWORD; but can use Managed Identity format (Authentication=Active Directory Managed Identity;).

• Deploy_Web_App:

  • Downloads the application artifact (drop).

  • Deploys the application code to the staging slot of the Web App, injecting the connection string as an App Setting (-ConnectionStrings__DefaultConnection).

  • Performs a slot swap (AzureAppServiceManage@0) to move the newly deployed code from staging to production, completing the blue/green deployment for the Development environment.

C. Production Environment Flow (Runs on main branch)

This flow utilizes a gated deployment pattern, separating plan, approval, and application.

• Plan_Prod_Infra: Runs terraform init and terraform plan for the Production environment and publishes the plan.out artifact for mandatory review (e.g., via a manual approval gate in a subsequent stage).

• Deploy_Prod_Infra: Downloads the previously approved plan.out artifact and executes terraform apply to provision the Production infrastructure.

• Deploy_Staging_Config: Identical to the Development config stage, retrieving Production outputs and assembling the connection string.

• Deploy_Web_App_Staging: Deploys the application code to the staging slot of the Production Web App.

• Deploy_Production_Swap: This stage is designed to be the final gate. It performs the slot swap from staging to production in the production environment. This stage is typically linked to a manual approval in the Azure DevOps environment to ensure a human confirms the deployment before affecting live traffic.

2. Infrastructure as Code (Terraform) Functionality

The fundamental reason for needing the separate Azure DevOps Variable Group (containing client_id, client_secret, and tenant_id) is that the standard Azure DevOps Service Connection mechanism does not expose the Service Principal’s secrets to the pipeline tasks that need them.

The Terraform files define the Azure resources and their secure configuration:

• Azure SQL Server and Database: Creates an Azure SQL Server with an Azure AD Administrator configured. The server is configured with a System-Assigned Managed Identity (MSI).

• Database User Mapping (MSI): Uses the mssql_user resource to create a user inside the SQL Database that is explicitly mapped to the Azure Web App’s Managed Identity (var.webapp_principal_id), granting it db_datareader and db_datawriter roles. This is the key step for secrets-less application authentication.

• Database Schema Initialization: The null_resource executes the setup.sql script via the sqlcmd utility, using Active Directory Service Principal authentication, to create the initial Artifacts table and insert sample data.

• Networking: Enforces security by creating an Azure Private Endpoint for the SQL Server, linking it to a Virtual Network via a Private DNS Zone (privatelink.database.windows.net), ensuring that database traffic stays within the private Azure network.

• Role Assignment: Assigns the SQL Server’s own System-Assigned Managed Identity to the Directory Readers Azure AD role, which is a common requirement for Azure SQL to resolve external Azure AD users (like the Web App’s MSI).

• A main Virtual Network is created and segmented into two purpose-built subnets:

  • Web App Subnet: Specifically delegated (Microsoft.Web/serverFarms) for App Service VNet Integration, which allows the Web App to join the private network.

  • Private Endpoint Subnet: A separate, dedicated subnet is reserved for hosting the Private Endpoint connection to the Azure SQL Database.

• Network Security Group (NSG) Enforcement: A stringent Network Security Group (azurerm_network_security_group.webapp_sql_nsg) is created and associated with the Web App Subnet. This acts as a private firewall, controlling all incoming and outgoing traffic:

  • Inbound Denial: Explicitly denies all inbound traffic from the public Internet (Internet Service Tag), ensuring the Web App is only accessible via Azure’s internal services or an authorized front door.

  • Outbound SQL Access: Limits outbound traffic to port 1433 (SQL) only when the destination is within the Virtual Network (VirtualNetwork Service Tag). This means the application can only connect to the Azure SQL Private Endpoint, blocking connections to any public SQL server.

  • Outbound Security Access: Explicitly permits outbound HTTPS (port 443) traffic to the AzureActiveDirectory Service Tag (required for the Web App to acquire Managed Identity tokens) and the AzureKeyVault Service Tag (if the application needs to fetch other configuration secrets).

While the network configuration locks down where the Web App can connect, the infrastructure code also prepares for who can connect to the database.

• SQL Server & Web App Identity (Setup): This infrastructure is designed to be paired with a Web App that uses a System-Assigned Managed Identity.

• Automated User Mapping: The process includes using the Web App’s Managed Identity Principal ID (retrieved from an output variable) to automatically create a corresponding user within the SQL Database itself. This step grants the Web App’s identity specific roles (like db_datareader and db_datawriter), establishing a principle of least privilege.

• The Result: The application’s final connection string will not contain any username or password, relying entirely on the Azure platform to exchange the Managed Identity for a temporary, secure access token.

Code link: here

Subscribe now

Share

Leave a comment

This repository provides a modular, reusable Terraform module for provisioning a standard, highly available AWS Virtual Private Cloud (VPC) with public and private subnets across multiple Availability Zones (AZs).

The Terraform module creates a best-practice VPC topology designed for high availability and security:

• VPC: The main network container.

• Public Subnets: Provisioned for public-facing resources (e.g., Load Balancers) and spread across multiple AZs.

• Private Subnets: Provisioned for application servers and databases, providing isolated, secure network space.

• Internet Gateway (IGW): Provides internet access for the VPC.

• NAT Gateways (NAT-GWs): One NAT-GW is deployed in each public subnet, allowing resources in the private subnets to initiate outbound internet traffic (e.g., for updates) without being publicly accessible.

• Routing: Public subnets route directly to the IGW, while each Private Subnet routes to its corresponding NAT-GW within the same AZ index.

It includes a robust two-stage GitHub Actions CI/CD pipeline for validating infrastructure changes on Pull Requests and safely deploying them on merge.

Two-stage GitHub Actions workflows exist:

1. Validation Pipeline (terraform-plan.yml): Runs on Pull Requests (PRs) to validate code and preview changes.

2. Deployment Pipeline (terraform-apply.yaml): Runs on merges (pushes) to safely deploy validated infrastructure.

Stage 1: The PR Validation Pipeline (terraform-plan.yml)

This workflow focuses on pre-deployment validation and transparency. It runs whenever a Pull Request is opened, synced, or reopened against the dev, staging, or main branches.

Key Features and Steps:

1. Environment Determination

The pipeline determines the target environment based on the PR’s destination branch (github.base_ref): dev/prod/staging are folder names as well as branch names as wella,each calls terraform module.

• If the base branch is main, the environment is set to prod.

• Otherwise, it uses the branch name (e.g., dev or staging). This is defined by the TF_ENV_FOLDER environment variable.

2. Secure Authentication with OIDC

Instead of long-lived keys, the pipeline uses OpenID Connect (OIDC) to assume a secure AWS IAM Role (arn:aws:iam::...:role/GitHubActions-AWS-Admin). This is a security best practice.

Here is guide how to set up OIDC, but i’ll describe it here step by step:

In AWS console, in IAM click Identity providers-add provider

Select OpenID connect

Provider URL: https://token.actions.githubusercontent.com

Audience: sts.amazonaws.com

Click next and click Create role

On Trusted entity type page click Web Identity and chose Identity provider, audience, specify Your GitHub organization and GitHub repository

Add administrator role click next, copy Identity ARN, you’ll need it later on.

In Trusted relationship property of just created role, modify trusted policy to allow this Identity provider to asume this admin role

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::12344565:oidc-provider/token.actions.githubusercontent.com"
},
"Action”: "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringLike": {
"token.actions.githubusercontent.com:sub": [
repo:dragan-actions-course/actions:*",
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
     ]
    }
   }
  }
 ]
}

S3 bucket permissions

I use S3 bucket for terraform TF state, so i created IAM policy, attached to this User entity Role, which allows access to this S3 bucket

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Sid": "VisualEditor0",

            "Effect": "Allow",

            "Action": [

                "s3:PutObject",

                "s3:GetObject",

                "s3:ListBucket"

            ],

            "Resource": [

                "arn:aws:s3:::terraform-tfstate-bucket-tatooine",

                "arn:aws:s3:::terraform-tfstate-bucket-tatooine/*"

            ]

        }

    ]

}

Now Managed Identity,when called by GitHub actions workflow, assumes Admin role and have access to S3 bucket to store Terraform state.

3. The Core Validation: Terraform Plan

The job executes a Python wrapper (tf_wrapper.py) to run terraform plan for the calculated environment folder (e.g., environments/staging).

• It captures the full output and the exit code.

• The job fails only if the exit code is 1 (indicating a hard configuration error), ensuring bad code doesn’t proceed.

4. Pull Request Visibility

The captured plan output is automatically posted back as a comment on the Pull Request. This gives reviewers an immediate, transparent view of exactly what infrastructure changes will occur if the code is merged.

5. Static Code Analysis

To catch issues before deployment, the workflow runs static analysis tools:

• TFLint: Ensures adherence to style and module structure.

• Checkov: Performs security scanning on the Terraform code, flagging potential misconfigurations.

• The raw JSON and a converted HTML report from Checkov are saved as artifacts for review.

Stage 2: The Deployment Pipeline (terraform-apply.yaml)

This workflow is the gatekeeper for deployment. It only runs on a push event (i.e., when code is merged) to the dev, staging, or main branches.

Key Features and Steps:

1. Environment and Concurrency Control

Similar to the plan, the environment is determined using the branch name (github.ref_name), mapping main to prod. Crucially, the concurrency group is set using the branch name, ensuring that only one deployment to a specific environment (e.g., staging) can run at any given time. This prevents race conditions and state file corruption.

2. Deployment Notification

Before execution, a commit comment is posted with the deployment start time and status. This provides immediate feedback to the team on the merge commit itself.

3. The Core Deployment: Terraform Apply

The workflow uses the Python wrapper to execute the full Terraform workflow (init, plan, and apply) for the specified environment.

• The TF_VAR_environment variable is explicitly passed to allow modules to tag resources with the correct environment name (e.g., dev, staging, or prod). So it pickups names of folders in GitHub repository

• Exit codes 0 (no changes) and 2 (changes applied) are considered successes.

• Only exit code 1 triggers a critical failure and stops the deployment.

Transparency on Pull Requests (The Validation Stage)

The primary goal of the Validation Pipeline (terraform-plan.yml) is to ensure every reviewer understands the precise impact of the code changes before they are merged.

• Automatic Plan Output: When a Pull Request is opened or updated, the workflow automatically executes the terraform plan. It captures the full output, which details exactly what AWS resources will be created, updated, or destroyed.

• PR Comment: The pipeline then posts this captured output as a detailed comment directly on the Pull Request. This means reviewers don’t have to leave the PR page or dig through CI logs; the critical decision-making information is right there.

This transparency helps answer the crucial question: “What is this change actually doing?” with an unalterable, machine-generated report.

Notifications on Merge (The Deployment Stage)

Once the code is approved and merged, the Deployment Pipeline (terraform-apply.yaml) takes over, focusing on keeping the team updated on the deployment status.

• Deployment Start Notification: As soon as the merge triggers the deployment workflow, the pipeline posts a commit comment on the merge commit. This comment provides immediate notification that the infrastructure update has begun, including the start time and the target environment (e.g., DEV, STAGING, or PROD).

• Status Tracking: This initial notification acts as an easy-to-find marker, allowing team members to monitor the job’s progress directly via a link to the workflow run.

The Terraform Python Wrapper (tf_wrapper.py)

While GitHub Actions provides the environment, the Python wrapper script (tf_wrapper.py) is the true orchestrator that manages the multi-step execution of Terraform commands within that environment.

This wrapper is primarily designed to enforce a consistent workflow and handle the tricky logic of capturing output and managing exit codes.

How the Wrapper Executes Terraform

The Python script does not run the Terraform logic itself; instead, it uses Python’s subprocess module to execute the native Terraform CLI binary on the runner machine.

The execution flow within tf_wrapper.py for a deployment or plan is a simple but powerful sequence:

Argument Parsing and Environment Setup

The wrapper first accepts the target environment path (e.g., environments/staging) as an argument from the GitHub Action:

• It changes the current working directory (cd) to this environment folder. All subsequent Terraform commands run relative to this path, ensuring the correct configuration files (.tf) and state are used.

The Python Utility Script (convert_json_to_html.py)

This Python script is a utility run during the terraform-plan.yml CI workflow to improve the usability of the security audit results.

• The script’s sole function is to read the raw JSON output generated by the Checkov security scanner and convert it into a styled, self-contained HTML report. This makes the security findings easy to review directly in a browser without needing to parse complex JSON data.

• It leverages the Jinja2 templating library to process the JSON data into a defined HTML_TEMPLATE.

• The script extracts and formats the following key information from the Checkov JSON:

  • Summary: Total number of failed, passed, and skipped checks.

  • Failed Checks Details: For every failed security rule, it lists the Check ID, Check Name, the offending File Path, the specific Resource (e.g., aws_vpc.main), and the line number of the violation.

Complete code is available here

Subscribe now

Share

Leave a comment

This script is structured into several steps and uses a robust function set for logging and state management.

Resumable Execution via State Management

This script ensures that you can run it, reboot when necessary, and the script will pick up exactly where it left off, making DC provisioning repeatable and reliable.

The core of this script’s reliability is the Get-ScriptState and Set-ScriptState functions.

State File (C:\Logs\state.json): Before a mandatory reboot (like after server rename or AD promotion), the script saves its current step number and all essential parameters (like $DomainName, $DSRMPassword, etc.) to a JSON file.

The Set-ScheduledTask function creates a task (DCSetupTask) that executes the script at startup. This is critical for catching the script execution immediately after the server reboots following the rename or DC promotion. Once the script is complete, the Remove-ScheduledTask function cleans up the environment.

The script uses a sequential flow controlled by the $State.CurrentStep variable. Crucially, each step includes checks to ensure it only performs an action if necessary (idempotency):

Rename Server

Checks $env:COMPUTERNAME -ne $ServerName. If successful, it sets the scheduled task and reboots.

Install AD DS & Promote

Checks if the AD-Domain-Services feature is installed and if the server is already a DC for the target domain.

Create AD Objects

Polls until the Active Directory is fully ready. Creates required OUs (TestUsers, Groups, Servers) only if they don’t exist. Creates standard users and a dedicated administrator (adm-setup) and adds the admin to Domain Admins.

Install & Configure IIS

Checks if IIS features are installed and only creates the default index.html if it doesn’t exist. It also ensures the “Default Web Site” is started.

param(
    [Parameter(Mandatory=$true)]
    [string]$DomainName,
    [Parameter(Mandatory=$true)]
    [string]$DSRMPassword,
    [Parameter(Mandatory=$true)]
    [string]$DefaultUserPassword,
    [string]$ServerName,
    [string]$AdminUsername = "adm-setup" 
)


# 1. General Variables
$ScriptPath = $MyInvocation.MyCommand.Path
$LogFilePath = "C:\Logs\build.log"
$StateFilePath = "C:\Logs\state.json"
$TaskName = "DCSetupTask" # Used by Set/Remove-ScheduledTask

# Ensure Logs directory exists
if (-not (Test-Path "C:\Logs")) {
    New-Item -Path "C:\Logs" -ItemType Directory | Out-Null
}

# Functions for Logging and State Management
function Write-Log {
    param(
        [Parameter(Mandatory=$true)]
        [string]$Message,
        [Parameter(Mandatory=$false)]
        [string]$Level = "INFO" # INFO, WARN, ERROR
    )
    $Timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
    $LogEntry = "[$Timestamp] [$Level] $Message"
    Add-Content -Path $LogFilePath -Value $LogEntry
    Write-Host “$LogEntry”
}

function Get-ScriptState {
       
    if (Test-Path $StateFilePath) {
       
        try {
            Write-Host "Found state file. Resuming setup."
            $State = Get-Content $StateFilePath | ConvertFrom-Json
            # Add default values if missing
            if (-not $State.CurrentStep) { $State.CurrentStep = 1 }
            if (-not $State.RebootPending) { $State.RebootPending = $false }
            return $State
        } catch {
            Write-Host "WARNING: State file was found but appears corrupt. Starting fresh." -ForegroundColor Yellow
        }
    }
    
    # If Test-Path was false, OR if the try block failed, return the Initial state
    Write-Host "No valid state file found. Starting fresh setup."
    return [PSCustomObject]@{
        CurrentStep = 1
        RebootPending = $false
        DomainName = $DomainName
        DSRMPassword = $DSRMPassword
        DefaultUserPassword = $DefaultUserPassword
        ServerName = $ServerName
    }
}

function Set-ScriptState {
    param(
        [Parameter(Mandatory=$true)]
        [PSCustomObject]$State
    )
    $State | ConvertTo-Json -Depth 5 | Out-File $StateFilePath -Force
}

function Set-ScheduledTask {
    param(
        [Parameter(Mandatory=$true)]
        [string]$PathToScript,
        [Parameter(Mandatory=$true)]
        [string]$DomainName,
        [Parameter(Mandatory=$true)]
        [string]$DSRMPassword,
        [Parameter(Mandatory=$true)]
        [string]$DefaultUserPassword,
        [Parameter(Mandatory=$true)]
        [string]$ServerName
    )
    Write-Log "Creating scheduled task '$TaskName' for self-resumption after reboot."
    
    # Construct arguments string for the scheduled task
    $Arguments = "-ExecutionPolicy Bypass -File `"$PathToScript`" -DomainName `"$DomainName`" -DSRMPassword `"$DSRMPassword`" -DefaultUserPassword `"$DefaultUserPassword`" -ServerName `"$ServerName`""

    $Action = New-ScheduledTaskAction -Execute "PowerShell.exe" -Argument $Arguments
    $Trigger = New-ScheduledTaskTrigger -AtStartup
    $Settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -StartWhenAvailable
    
    Register-ScheduledTask -TaskName $TaskName -Action $Action -Trigger $Trigger -Settings $Settings -User “System” -Force | Out-Null
    Write-Log "Scheduled task created. The script will resume automatically."
}

function Remove-ScheduledTask {
    if (Get-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue) {
        Write-Log "Removing scheduled task '$TaskName'."
        Unregister-ScheduledTask -TaskName $TaskName -Confirm:$false
    }
}


Write-Log "--- Starting DC Setup Script ---"

$State = Get-ScriptState

# Override initial parameters if we are resuming
if ($State.CurrentStep -gt 1) {
    Write-Log "Resuming from state: Step $($State.CurrentStep)" -Level "WARN"
    $DomainName = $State.DomainName
    $DSRMPassword = $State.DSRMPassword
    $DefaultUserPassword = $State.DefaultUserPassword
    $ServerName = $State.ServerName
    # If the state indicates reboot is pending, and we’ve just booted, we’ve successfully rebooted.
    if ($State.RebootPending -eq $true) {
        $State.RebootPending = $false
        $State.CurrentStep = 3 # Skip steps 1 (Rename) and 2 (DC Promo/Reboot)
        Set-ScriptState -State $State
    }
}

# Rename Server to DC01 (If necessary)
if ($State.CurrentStep -eq 1) {
    Write-Log "Step 1: Renaming server to $ServerName."

    if ($env:COMPUTERNAME -ne $ServerName) {
        Write-Log “Current name is '$($env:COMPUTERNAME)'. Renaming to '$ServerName'."

        try {
            # Attempt the rename operation
            Rename-Computer -NewName $ServerName -Force -ErrorAction Stop

            # If rename succeeds, prepare for reboot
            $State.RebootPending = $true
            $State.CurrentStep = 1
            Set-ScriptState -State $State
            Set-ScheduledTask -PathToScript $ScriptPath -DomainName $DomainName -DSRMPassword $DSRMPassword -DefaultUserPassword $DefaultUserPassword -ServerName $ServerName
            Write-Log "Server renamed successfully. Initiating reboot now..." -Level "WARN"
            Restart-Computer -Force
            exit # Wait for reboot

        } catch {
            $ErrorMessage = "Failed to rename the server: $($_.Exception.Message)"
            Write-Log “FATAL ERROR: $ErrorMessage. Setting state to 99 and exiting.” -Level "ERROR"

            # Set state to 99 (Error State)
            $State.CurrentStep = 99
            Set-ScriptState -State $State

            # Write a standard PowerShell error and exit the script
            Write-Error -Message $ErrorMessage -ErrorAction Stop
            exit 1 
        }
    } else {
        Write-Log "Server already named ‘$ServerName’. Skipping rename."
        $State.CurrentStep = 2
        Set-ScriptState -State $State
    }
}

# Install AD DS and Promote to DC
if ($State.CurrentStep -eq 2) {
    Write-Log "Step 2: Installing AD DS Role and promoting to DC."
    
    # Check if AD DS is already installed
    if ((Get-WindowsFeature AD-Domain-Services).Installed -eq $false) {
        Write-Log "Installing AD-Domain-Services feature."
        Install-WindowsFeature -Name AD-Domain-Services -IncludeManagementTools -Confirm:$false -Restart:$false | Out-Null
    } else {
        Write-Log "AD-Domain-Services feature already installed."
    }

    # Check if we are already a DC by testing for AD domain existence
    $IsDC = $false
    try {
        $Domain = Get-ADDomain -ErrorAction Stop
        if ($Domain.DNSRoot -eq $DomainName) {
            $IsDC = $true
        }
    } catch {
        # AD Domain cmdlets not available or domain doesn’t exist yet
        Write-Log "AD Domain not yet configured. Proceeding with promotion."
    }

    if (-not $IsDC) {
        $SafeModePassword = ConvertTo-SecureString $DSRMPassword -AsPlainText -Force
        
        Write-Log “Promoting server to a Domain Controller for new forest '$DomainName'."
        
        try {
            Install-ADDSForest `
                -CreateDnsDelegation:$false `
                -DatabasePath "C:\Windows\NTDS" `
                -DomainName $DomainName `
                -DomainMode "WinThreshold" `
                -DomainNetbiosName $DomainName.Split('.')[0].ToUpper() `
                -ForestMode "WinThreshold" `
                -LogPath "C:\Windows\NTDS" `
                -NoRebootOnCompletion:$false `
                -SYSVOLPath "C:\Windows\SYSVOL" `
                -Force:$true `
                -InstallDns:$true `
                -SafeModeAdministratorPassword $SafeModePassword `
                -ErrorAction Stop
            
            # If we reach here, promotion initiated successfully
            $State.RebootPending = $true
            $State.CurrentStep = 2
            Set-ScriptState -State $State
            Write-Log "AD DS promotion initiated. Waiting for automatic reboot..." -Level "WARN"
            exit # Wait for automatic reboot by Install-ADDSForest
            
        } catch {
            $ErrorMessage = "Failed to promote server to DC: $($_.Exception.Message)"
            Write-Log “FATAL ERROR: $ErrorMessage” -Level "ERROR"
            $State.CurrentStep = 99
            Set-ScriptState -State $State
            Write-Error -Message $ErrorMessage -ErrorAction Stop
            exit 1
        }
    } else {
        Write-Log "Server is already a DC for '$DomainName'. Skipping promotion."
        $State.CurrentStep = 3
        Set-ScriptState -State $State
    }
}

# Create OUs, Users, and Admin User (Requires AD DS to be fully functional)
if ($State.CurrentStep -eq 3) {
    Write-Log "Step 3: Creating AD OUs and Users."

    try {
        Import-Module ActiveDirectory -ErrorAction Stop
    } catch {
        Write-Log "FATAL ERROR: Could not import ActiveDirectory module. Is the feature installed?" -Level “ERROR”
        $State.CurrentStep = 99
        Set-ScriptState -State $State
        exit 1
    }
    
    $MaxAttempts = 700 # Check for 700 seconds
    $Attempt = 0
    $ADReady = $false
    $BaseDN = $null
    
    Write-Log "Polling for Active Directory Domain Services readiness (Max 120 seconds)..."
    
    while (-not $ADReady -and $Attempt -lt $MaxAttempts) {
        try {
            $BaseDN = (Get-ADDomain -ErrorAction Stop).DistinguishedName
            $ADReady = $true
        } catch {
            $Attempt++
            Write-Log "Attempt $Attempt/$MaxAttempts AD DS not ready. Waiting 2 seconds..." -Level "WARN"
            Start-Sleep -Seconds 2
        }
    }
    
    if (-not $ADReady) {
        $ErrorMessage = "AD DS did not become ready after $($MaxAttempts*2) seconds. Cannot proceed to create OUs/Users."
        Write-Log "FATAL ERROR: $ErrorMessage. Setting state to 99 and exiting." -Level "ERROR"
        $State.CurrentStep = 99
        Set-ScriptState -State $State
        Write-Error -Message $ErrorMessage -ErrorAction Stop
        exit 1
    }
    Write-Log "Active Directory is ready. Proceeding with OU and User creation."
    
    # Create OUs
    Write-Log "Creating required Organizational Units."
    $OUs = @("TestUsers", "Groups", "Servers")
    
    foreach ($OUName in $OUs) {
        $OUPath = "OU=$OUName,$BaseDN"
        
        
        $OUExists = $false
        try {
            $null = Get-ADOrganizationalUnit -Identity $OUPath -ErrorAction Stop
            $OUExists = $true
        } catch {
            # OU doesn’t exist, we’ll create it
        }
        
        if (-not $OUExists) {
            Write-Log "Creating OU: $OUName"
            New-ADOrganizationalUnit -Name $OUName -Path $BaseDN -ProtectedFromAccidentalDeletion $true
        } else {
            Write-Log "OU '$OUName' already exists."
        }
    }
    
    # Create Standard Users
    Write-Log "Creating standard users."
    $PasswordSecure = ConvertTo-SecureString $DefaultUserPassword -AsPlainText -Force
    $UserCounter = 1
    
    foreach ($OUName in $OUs) {
        $OUPath = "OU=$OUName,$BaseDN"
        for ($i = 1; $i -le 3; $i++) {
            $UserName = "stduser$UserCounter"
            
            $UserExists = $false
            try {
                $null = Get-ADUser -Identity $UserName -ErrorAction Stop
                $UserExists = $true
            } catch {
                # User doesn’t exist, we’ll create it
            }
            
            if (-not $UserExists) {
                Write-Log "Creating standard user: $UserName in OU '$OUName'."
                New-ADUser -Name $UserName -GivenName "Standard" -Surname "User$UserCounter" -SamAccountName $UserName `
                    -UserPrincipalName "$UserName@$DomainName" -Path $OUPath `
                    -AccountPassword $PasswordSecure -Enabled $true `
                    -ChangePasswordAtLogon $false
                # Set password to never expire
                Set-ADAccountControl -Identity $UserName -PasswordNeverExpires $true
            } else {
                Write-Log "Standard user '$UserName' already exists."
            }
            $UserCounter++
        }
    }
    
    # 3c: Create Domain Admin User
    Write-Log "Creating dedicated Domain Administrator user '$AdminUsername'."
    
    # Use try-catch to completely suppress error output
    $AdminExists = $false
    try {
        $null = Get-ADUser -Identity $AdminUsername -ErrorAction Stop
        $AdminExists = $true
    } catch {
        # Admin doesn’t exist, we’ll create it
    }
    
    if (-not $AdminExists) {
        New-ADUser -Name $AdminUsername -GivenName "Setup" -Surname "Admin" -SamAccountName $AdminUsername `
            -UserPrincipalName "$AdminUsername@$DomainName" -Path "OU=TestUsers,$BaseDN" `
            -AccountPassword $PasswordSecure -Enabled $true `
            -ChangePasswordAtLogon $false
            
        # Add to Domain Admins group
        Add-ADGroupMember -Identity "Domain Admins" -Members $AdminUsername
        Set-ADAccountControl -Identity $AdminUsername -PasswordNeverExpires $true
        Write-Log "Domain Admin user '$AdminUsername' created and added to 'Domain Admins'."
    } else {
        Write-Log "Domain Admin user '$AdminUsername' already exists.”
    }

    $State.CurrentStep = 4
    Set-ScriptState -State $State
}

# STEP 4: Install IIS and Configure Landing Page
if ($State.CurrentStep -eq 4) {
    Write-Log "Step 4: Installing IIS Web-Server role and configuring landing page."

    try {
        # Install IIS (Web-Server and Management Tools)
        $IISFeatures = @("Web-Server", "Web-Mgmt-Tools")
        foreach ($Feature in $IISFeatures) {
            if ((Get-WindowsFeature $Feature).Installed -eq $false) {
                Write-Log “Installing Windows Feature: $Feature”
               Install-WindowsFeature -Name $Feature -Confirm:$false -Restart:$false -ErrorAction Stop | Out-Null
            } else {
                Write-Log "Windows Feature '$Feature' already installed."
            }
        }
        
        # Check if the site is already running 
        $SitePath = "C:\inetpub\wwwroot\index.html"
        $SiteContent = "<html><body><h1>Hello World from Server!</h1></body></html>"
        
        # Configure Landing Page
     
        if (-not (Test-Path $SitePath)) {
            Write-Log "Creating default $SitePath with a simple landing page."
            $SiteContent | Out-File $SitePath -Encoding UTF8 -Force -ErrorAction Stop
        } else {
            Write-Log "Landing page $SitePath already exists. Skipping replacement."
        }
        
        # Ensure Default Site is Started and Listening on Port 80
        Write-Log "Ensuring Default Web Site is started."
        $DefaultSite = Get-Website -Name "Default Web Site" -ErrorAction SilentlyContinue
        if (-not $DefaultSite) {
            throw "Default Web Site does not exist after IIS installation."
        }

        if ($DefaultSite.State -ne "Started") {
            Start-Website -Name "Default Web Site" -ErrorAction Stop
        } else {
            Write-Log "Default Web Site is already started."
        }

        $State.CurrentStep = 5
        Set-ScriptState -State $State
        Write-Log "IIS installation and configuration completed successfully."

    } catch {
        $ErrorMessage = "Failed during IIS installation or configuration: $($_.Exception.Message)"
        Write-Log "FATAL ERROR: $ErrorMessage. Setting state to 99 and exiting." -Level "ERROR"

        # Set state to 99 (Error State)
        $State.CurrentStep = 99
        Set-ScriptState -State $State

        # Write a standard PowerShell error and exit the script
        Write-Error -Message $ErrorMessage -ErrorAction Stop
        exit 1
    }
}

# Finalization and Cleanup
if ($State.CurrentStep -eq 5) {
    Write-Log "Step 5: Finalization and Cleanup."

    # Remove the state file
    Remove-Item -Path $StateFilePath -Force -ErrorAction SilentlyContinue
    # Ensure the scheduled task is removed
    Remove-ScheduledTask

    Write-Log "--- Script Complete! DC01 setup is finished. ---"
}

Subscribe now

Share

Leave a comment

While Microsoft handles the underlying infrastructure and control plane (like the etcd database for the cluster state), Azure customares are responsible for protecting:

• Application Data: The persistent volumes (PVs) and persistent volume claims (PVCs) that store the vital data for your stateful applications (databases, message queues, file storage).

• Cluster Resources: The key configuration elements, like Deployments, Services, ConfigMaps, and Secrets, that define how your application runs.

Without a solid backup, a simple misconfiguration, an accidental deletion, or a broader regional disaster could lead to significant data loss and extended downtime, impacting business continuity and hitting your bottom line.

This post will explore how to implement a cloud-native, application-aware backup and restore solution for your AKS clusters, focusing on solutions like Azure Backup for AKS.

Azure Backup for AKS DOES NOT backup:

• The Kubernetes Control Plane Software: The version of the core Kubernetes components (like the API server, scheduler, and controller manager) running your cluster. This is entirely managed by the AKS service.

• The Operating System (OS) Image: The underlying OS image (e.g., Ubuntu or Windows Server) used by your node pool VMs.

• Networking Configuration: Virtual Networks, Subnets, Load Balancer rules external to the cluster, etc.

Azure Backup captures the state and data defined by the running version:

• Cluster State (etcd data equivalent): All the Kubernetes API resources (the YAML manifests) like Deployments, Services, ConfigMaps, Secrets, RBAC roles, and PVCs.

• Persistent Data: Snapshots of the underlying Azure Disks (Persistent Volumes) used by your stateful applications.

Required roles:

Below is code for creating Storage acocunt,Blob container,Key vault,permissions/roles needed for Azure backup and Resource group for storing backup snapshot.

$RESOURCE_GROUP="aks"
$CLUSTER_NAME="myaks"
$Location="westeurope"


# Create snapshot resource group
$SNAPSHOT_RG_NAME="snapshot-rg"
az group create `
  --name $SNAPSHOT_RG_NAME `
  --location $Location

# Check the status of your CSI driver. The required add-on is azurekeyvaultkms
az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query "storageProfile.diskCsiDriver.enabled"
# If not enabled, enable it using the following command:
az aks update --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --enable-disk-csi-driver

# Register the necessary resource provider for the Backup Extension.
az provider register --namespace Microsoft.KubernetesConfiguration
az provider register --namespace Microsoft.ContainerService 

$backupVaultRG="backup-vault"
$vaultName="MyAKSBackupVault"
$blobContainerName="aks-backup-container"

az group create `
  --name $backupVaultRG  `
  --location $Location

# Configure the CLI to automatically install extensions without asking
az config set extension.use_dynamic_install=yes_without_prompt

az dataprotection backup-vault create `
  --resource-group $backupVaultRG `
  --vault-name $vaultName `
  --location $Location `
  --type SystemAssigned `
  --storage-settings type="LocallyRedundant" datastore-type="VaultStore"
  # type="GeoRedundant" 2x more expensive than LocallyRedundant
  # ZoneRedundant best for high availability but not supported in all regions


#  az dataprotection backup-vault update `
#    --resource-group $backupVaultRG `
#    --vault-name $vaultName `
#    --set identity.type=SystemAssigned

# Create the Storage Account (Standard_LRS is typically sufficient and cost-effective)
$storageAccountName="mystorageaccount20251104"

az storage account create `
  --name $storageAccountName `
  --resource-group $backupVaultRG `
  --location $Location `
  --sku Standard_LRS `
  --kind StorageV2

# Create the Blob Container
az storage container create `
  --name $blobContainerName `
  --account-name $storageAccountName `
  --auth-mode login


# Get the Backup Vault ID
$vaultId = az dataprotection backup-vault show `
  --resource-group $backupVaultRG `
  --vault-name $vaultName `
  --query id -o tsv

# Get the AKS Cluster Resource ID
$aksId = az aks show `
  --resource-group $RESOURCE_GROUP `
  --name $CLUSTER_NAME `
  --query id -o tsv

# Get the Principal ID of the Backup Vault’s Managed Identity

$vaultPrincipalId = az dataprotection backup-vault show `
--resource-group $backupVaultRG `
--vault-name $vaultName `
--query identity.principalId -o tsv

# Get the Snapshot Resource Group ID
$snapshotRgId = az group show `
  --name $SNAPSHOT_RG_NAME `
  --query id -o tsv

# Install the Backup Extension on AKS

az k8s-extension create `
    --name azure-aks-backup `
    --extension-type microsoft.dataprotection.kubernetes `
    --scope cluster `
    --cluster-type managedClusters `
    --cluster-name $CLUSTER_NAME `
    --resource-group $RESOURCE_GROUP `
    --configuration-settings blobContainer=$blobContainerName storageAccount=$storageAccountName storageAccountResourceGroup=$backupVaultRG storageAccountSubscriptionId=$(az account show --query id -o tsv)



# Get the Principal ID of the Backup Extension’s identity
$EXTENSION_IDENTITY_ID=$(az k8s-extension show `
    --name azure-aks-backup `
    --cluster-name $CLUSTER_NAME `
    --resource-group $RESOURCE_GROUP `
    --cluster-type managedClusters `
    --query aksAssignedIdentity.principalId `
    --output tsv)



# Get the Storage Account Resource ID
$STORAGE_ID=$(az storage account show `
    --name $storageAccountName `
    --resource-group $backupVaultRG `
    --query id `
    --output tsv)

# Get the Principal ID (Object ID) of your Backup Vault’s Managed Identity
#    (You should have this from previous steps: $vaultPrincipalId)

$VaultPrincipalId = (az backup vault show --name “MyAKSBackupVault” --resource-group “backup-vault” --query identity.principalId --output tsv)

# Assign the ‘Storage Blob Data Contributor’ role
az role assignment create `
    --assignee-object-id $EXTENSION_IDENTITY_ID `
    --role 'Storage Blob Data Contributor' `
    --scope $STORAGE_ID

# The Backup Vault needs Trusted Access to the AKS cluster to interact with the extension.

az aks trustedaccess rolebinding create `
    --resource-group $RESOURCE_GROUP `
    --cluster-name $CLUSTER_NAME `
    --name "backup-vault-binding" `
    --source-resource-id $vaultId `
    --roles "Microsoft.DataProtection/backupVaults/backup-operator"


# Grant the Azure Backup Vault’s Managed Identity the Reader role permissions on the AKS cluster resource.

az role assignment create `
    --assignee-object-id $vaultPrincipalId `
    --role "Reader" `
    --scope $aksId `
    --assignee-principal-type "ServicePrincipal" `
    --name "18f7d05d-daf0-44f3-8aef-7df0b1b4b98c"



# Assign the ‘Data Operator for Managed Disks’ role to the Vault’s MSI on the Snapshot Resource Group
az role assignment create `
    --assignee-object-id $VaultPrincipalId `
    --role "Data Operator for Managed Disks" `
    --scope $snapshotRgId `
    --assignee-principal-type "ServicePrincipal"

Hooks in AKS backup

In the context of Azure Backup for Azure Kubernetes Service (AKS), hooks are a mechanism to ensure application-consistent backups of data stored in Persistent Volumes (PVs), particularly for stateful applications like databases.

Hooks are commands executed within a container before and after the Persistent Volume snapshots are taken. They are deployed as a Kubernetes Custom Resource (CR) called BackupHook.

The primary goal is to momentarily quiesce (stop writing to) the database, ensuring the snapshot captures a clean, consistent state that can be successfully restored.

How It Works (Example: MySQL/PostgreSQL)

1. Deployment: You define the hook logic in a BackupHook Custom Resource (CRD) and apply it to the AKS cluster using kubectl apply -f <hook-file>.yaml.

2. Backup Job Starts: Azure Backup initiates a job.

3. PreHook Execution: The Backup Extension executes the preHooks command inside the designated application container. The goal of application-consistent backup is to ensure the database’s data files are static while the volume snapshot is being taken. In MySQL, the command to achieve this is FLUSH TABLES WITH READ LOCK;

4. Snapshot: The CSI driver takes an incremental snapshot of the Azure Disk Persistent Volume.

5. PostHook Execution: The Backup Extension executes the postHooks command inside the container (e.g., executing UNLOCK TABLES;).

6. Backup Completes: The cluster state is copied to the blob container, and the backup job finishes.

Apply below yaml before starting backup, modify namespaces which need to be backed up and container name (to match ones in your deployments/statefull sets).

apiVersion: clusterbackup.dataprotection.microsoft.com/v1alpha1
kind: BackupHook
metadata:
  name: bkphook-mysql
  namespace: default
spec:
  backupHook:
    - name: mysql-quiesce-hook
      includedNamespaces: 
      - default # The namespace where your MySQL StatefulSet is running
      
      preHooks:
        - exec:
            container: mysql
            command:
            - /usr/bin/mysql
            - -u
            - root
            - -pmypassword # Replace ‘mypassword’ with your actual root password
            - -e
            - FLUSH TABLES WITH READ LOCK;
          
      
      postHooks:
        - exec:
            container: mysql
            command:
            - /usr/bin/mysql
            - -u
            - root
            - -pmypassword # Replace ‘mypassword’ with your actual root password
            - -e
            - UNLOCK TABLES;
            

Check if BackupHooks are running:

kubectl get pods -n dataprotection-microsoft

NAME                                                         READY   STATUS    RESTARTS   AGE

dataprotection-microsoft-controller-5bc5d797f6-g4nr9         2/2     Running   0          3h30m

dataprotection-microsoft-geneva-service-5cccf95bc9-jxftn     2/2     Running   0          3h30m

dataprotection-microsoft-kubernetes-agent-8549cf9bf4-qh4qr   2/2     Running   0          3h30m

# Check BackupHooks logs
kubectl logs dataprotection-microsoft-controller-5bc5d797f6-g4nr9 | Select-String -Pattern "fsfreeze|hook"

In AKS resource,click on Backup blade

Select Azure backup vault and AKS cluster and click Next

Configure backup frequency and click Next, in Datasources click add

Select what namespaces to include in backup

In Backup Hooks, specify namespace and Hook CR

Namespace: In hook YAML, namespace: default under metadata

apiVersion: clusterbackup.dataprotection.microsoft.com/v1alpha1

kind: BackupHook

metadata:

name: bkphook-mysql

namespace: default

Hook CR Deployed: This is the name of the specific BackupHook Custom Resource

• Value to enter: bkphook-mysql

• Reason: In hook YAML, set name: bkphook-mysql under metadata.

Select resource group for snapshot and click validate, if some permission are missing,just click on Assign missing roles and click configure backup

Click on three dots in Protection status and then click Backup Now

Check backup status

Restoring from Backup

On backup blade of AKS resource,click again on 3 dots on protection status-Restore

Select restore point

Subscribe now

Share

Leave a comment

Azure file shares offer persistent data that can be accessed by multiple pods simultaneously.Unlike Azure Disks, which are typically restricted to a single pod, Azure File Shares provide ReadWriteMany access, making them perfect for scenarios like shared configuration files, content management systems, or multi-instance applications needing a common data store.

Storage account mysa202510 in aks resource group with share name myshare was used.azure-secret is necessary for Persistent Volume (PV):

The Persistent Volume (PV) definition is merely metadata in Kubernetes. To actually connect to and mount the remote Azure File Share, the underlying mechanism—the Azure File CSI Driver running on the node—must present valid credentials to the Azure Storage service.

1. Azure File Uses SMB/CIFS: Azure File Shares are accessed using the SMB (Server Message Block) protocol (also known as CIFS). This protocol requires a username and a password to establish a secure connection, even if the storage account is publicly accessible.

2. Required Credentials: The credentials used are the Azure Storage Account Name (which acts as the username) and the Storage Account Key (which acts as the password).

3. Kubernetes Secret’s Role: The Kubernetes Secret resource (azure-secret in this case) is the secure, native way to store these sensitive credentials. It holds the Storage Account Name under the key azurestorageaccountname and the Storage Account Key under azurestorageaccountkey.

The nodeStageSecretRef section in PV acts as a pointer to these credentials:

• PV Definition: The nodeStageSecretRef points to the azure-secret in the default namespace.

• Kubelet Action: When a Pod needs to mount the PV, the Kubelet on the worker node reads the PV definition.

• CSI Driver Access: The Azure File CSI Driver sees the secret reference, retrieves the credentials from the secret, and constructs the necessary mount -t cifs command:

  mount -t cifs -o username=mysa202510,password=<STORAGE_KEY>,... //mysa202510.file.core.windows.net/myshare /mnt/azure

Without the Secret, the CSI driver has no way to authenticate the mount requests.

$AZURE_STORAGE_CONNECTION_STRING=$(az storage account show-connection-string --name mysa202510 --resource-group aks --query connectionString --output tsv)
# Create azure share
az storage share create --name myshare --connection-string $AZURE_STORAGE_CONNECTION_STRING
# Get storage key
$STORAGE_KEY=$(az storage account keys list --account-name mysa202510 --resource-group aks --query “[0].value” --output tsv)
# Create Kubernetes secret to tell AKS how to authenticate to storage account
kubectl create secret generic azure-secret `
  --from-literal=azurestorageaccountname=$AKS_PERS_STORAGE_ACCOUNT_NAME `
  --from-literal=azurestorageaccountkey=$STORAGE_KEY

The PersistentVolume (PV) object is Kubernetes’ way of knowing about existing Azure File Share. It’s a cluster-scoped resource that describes the actual cloud storage.

Create a file named azurefile-pv.yaml and apply it.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: azurefile-pv
spec:
  capacity:
    storage: 5Gi  # Size of your Azure File Share
  accessModes:
    - ReadWriteMany  # Crucial: Allows multiple pods to read/write
  persistentVolumeReclaimPolicy: Retain  # Don’t delete the Azure File Share if PVC is deleted
  storageClassName: azurefile-sc  # Logical name to match with PVC
  csi:
    driver: file.csi.azure.com  # The Azure File CSI driver
    volumeHandle: "mysa202510/myshare"  # Format: <StorageAccountName>/<FileShareName>
    readOnly: false
    volumeAttributes:
      shareName: "myshare"
    nodeStageSecretRef:
      name: azure-secret  # Reference to our secret
      namespace: default  # Namespace where the secret lives
  mountOptions:
    - dir_mode=0777 # Directory permissions
    - file_mode=0777 # File permissions
    - uid=1000 # Map files to UID 1000 inside container
    - gid=1000 # Map files to GID 1000 inside container

The PersistentVolumeClaim (PVC) is a request for storage from an application. In static provisioning, the PVC explicitly claims a specific PV.

Create a file named azurefile-pvc.yaml and apply it

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: azurefile
spec:
  accessModes:
    - ReadWriteMany  # Must match the PV’s access mode
  storageClassName: azurefile-sc  # Must match the PV’s StorageClass
  volumeName: azurefile-pv  # Explicitly binds to our 'azurefile-pv'
  resources:
    requests:
      storage: 5Gi  # Requested storage (must be <= PV capacity)

Finally, let’s deploy a simple Nginx pod that will mount Azure File Share.

Create a file named nginx-pod.yaml and apply it

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
    - name: mypod
      image: mcr.microsoft.com/oss/nginx/nginx:1.21.6
      resources:
        limits:
          memory: "128Mi"
          cpu: "500m"
        requests:
          memory: "64Mi"
          cpu: "250m"
      volumeMounts:
        - name: azure-storage
          mountPath: "/mnt/azure"  # The path inside the container
  volumes:
    - name: azure-storage
      persistentVolumeClaim:
        claimName: azurefile  # Reference to our PVC

Create example data inside pod

kubectl exec -it mypod-nginx -- touch touch /mnt/azure/test_file_$(date +%s).txt
# verify data are present
kubectl exec -it mypod -- ls -l /mnt/azure/      
total 0
-rwxrwxrwx 1 1000 1000 0 Nov  3 12:22 test_file_1762172563.txt

Data should be visible in Azure portal as well

Subscribe now

Share

Leave a comment

An Azure AKS snapshot is a point-in-time, read-only copy of an Azure Managed Disk that is used as a Persistent Volume (PV) by a stateful application running in Azure Kubernetes Service (AKS) cluster.

This feature is implemented using the Kubernetes Container Storage Interface (CSI) Volume Snapshot API.

Key Characteristics

• Data Protection: Its primary role is for backup, recovery, and data protection. It captures the data on the persistent disk at an exact moment, allowing you to restore the volume to that previous state.

• Kubernetes-Native: It’s managed through standard Kubernetes resources: the VolumeSnapshotClass (which defines parameters like incremental backup and the Azure Disk driver) and the VolumeSnapshot object (which requests the snapshot of a specific Persistent Volume Claim, or PVC).

• Storage Source: The snapshot lives as an object within Azure subscription’s resource group. It can be used as the data source to create a new Persistent Volume (Azure Managed Disk).

Use Cases

• Quickly roll back a volume to a clean state after accidental data corruption or deletion.

• Create a new, isolated environment (e.g., development or testing) with a copy of production data for debugging or validation.

• The underlying Azure Disk snapshot can be used to copy data to a different region or migrate workloads between AKS clusters.

• It is the technical foundation used by advanced backup solutions like Azure Backup for AKS and Velero to capture the data portion of an application backup.

First, let’s deploy our MySQL database using a Kubernetes StatefulSet. This ensures our database gets stable identities and persistent storage (an Azure Managed Disk).

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: default
spec:
  serviceName: "mysql-headless" # A dedicated Headless Service is required for StatefulSets
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        ports:
        - containerPort: 3306
          name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: my-secret-password # Should use a Kubernetes Secret in production!
        volumeMounts:
        - name: mysql-data # Links to the volumeClaimTemplate below
          mountPath: /var/lib/mysql
  # --- 1. THE VOLUME CLAIM TEMPLATE ---
  volumeClaimTemplates:
  - metadata:
      name: mysql-data # The name referenced in volumeMounts above
    spec:
      accessModes:
        - ReadWriteOnce # Correct for a database (Azure Disk)
      storageClassName: managed-premium # Your chosen StorageClass
      resources:
        requests:
          storage: 5Gi # Your chosen storage size
---
# Dedicated Headless Service (Required by StatefulSet)
apiVersion: v1
kind: Service
metadata:
  name: mysql-headless
  namespace: default
  labels:
    app: mysql
spec:
  ports:
  - port: 3306
    name: mysql
  clusterIP: None # Makes this a Headless Service
  selector:
    app: mysql

Test pods:

kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
mysql-0   1/1     Running   0          9m45s

Create and copy SQL script into running pod

kubectl exec -it mysql-0 -- bash -c "cat <<EOF > /tmp/test-data.sql
>> CREATE DATABASE IF NOT EXISTS snapshot_test_db;
>> USE snapshot_test_db;
>> CREATE TABLE IF NOT EXISTS inventory (
>>     id INT AUTO_INCREMENT PRIMARY KEY,
>>     item_name VARCHAR(100) NOT NULL,
>>     quantity INT NOT NULL,
>>     data_creation_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP
>> );
>> INSERT INTO inventory (item_name, quantity) VALUES ('Widget A', 100);
>> INSERT INTO inventory (item_name, quantity) VALUES ('Gadget B', 50);
>> EOF"

Execute the script to create test database and table

kubectl exec -it mysql-0 -- bash -c "mysql -u root -p'my-secret-password' < /tmp/test-data.sql"

Verify the data

kubectl exec -it mysql-0 -- bash -c "mysql -u root -p'my-secret-password' < /tmp/test-data.sql"
mysql: [Warning] Using a password on the command line interface can be insecure.
+----+-----------+----------+---------------------+
| id | item_name | quantity | data_creation_time  |
+----+-----------+----------+---------------------+
|  1 | Widget A  |      100 | 2025-11-03 09:21:34 |
|  2 | Gadget B  |       50 | 2025-11-03 09:21:34 |
+----+-----------+----------+---------------------+

Create the VolumeSnapshotClass

The VolumeSnapshotClass defines the driver and parameters for how the snapshot should be created on Azure. Since you are using Azure Disks, you’ll specify the disk.csi.azure.com provisioner.

Create a file named azuredisk-vsc.yaml with the following content:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-azuredisk-vsc
driver: disk.csi.azure.com # This is the Azure Disk CSI driver
deletionPolicy: Delete # Set to Delete to automatically remove the Azure snapshot when the K8s object is deleted, or Retain to keep it.
parameters:
  incremental: "true" # Creates incremental snapshots, which are faster and more storage-efficient.

Apply and verify the class

apply -f azuredisk-vsc.yaml
kubectl get volumesnapshotclass
NAME                DRIVER               DELETIONPOLICY   AGE
csi-azuredisk-vsc   disk.csi.azure.com   Delete           44m

Create the VolumeSnapshot

Now create the VolumeSnapshot resource, which references the VolumeSnapshotClass just created and the specific Persistent Volume Claim (PVC) associated with MySQL StatefulSet.

The name of PVC is mysql-data-mysql-0 (based on StatefulSet configuration: volumeClaimTemplates name mysql-data + StatefulSet name mysql + ordinal 0).

Create a file named mysql-snapshot.yaml with the following content:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: mysql-data-initial-snapshot # Give snapshot a descriptive name
  namespace: default
spec:
  # Reference the VolumeSnapshotClass created in previous step.
  volumeSnapshotClassName: csi-azuredisk-vsc
  source:
    # Reference the specific PVC of your StatefulSet Pod
    persistentVolumeClaimName: mysql-data-mysql-0

Apply and check snapshot

kubectl apply -f mysql-snapshot.yaml
kubectl get volumesnapshot mysql-data-initial-snapshot     
NAME                          READYTOUSE   SOURCEPVC            SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS       SNAPSHOTCONTENT                                    CREATIONTIME   AGE
mysql-data-initial-snapshot   true         mysql-data-mysql-0                           5Gi           csi-azuredisk-vsc   snapcontent-ff1edf44-3104-4ae6-a5f4-7eda16f86c8f   48m           
 48m

Simulate data loss

First, delete the test data to simulate the accident and verify data were lost

kubectl exec -it mysql-0 -- mysql -u root -p"my-secret-password" -e "DROP DATABASE snapshot_test_db;"
kubectl exec -it mysql-0 -- mysql -u root -p"my-secret-password" -e "SELECT * FROM snapshot_test_db.inventory;"
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1049 (42000) at line 1: Unknown database 'snapshot_test_db'
command terminated with exit code 1

Restore from Snapshot

Scale your MySQL StatefulSet to zero replicas. This cleanly shuts down the Pod and unmounts the corrupt Azure Disk from the node.

kubectl scale statefulset mysql --replicas=0

Delete original PVC

kubectl delete pvc mysql-data-mysql-0

Create a new Persistent Volume Claim (PVC) that uses the existing VolumeSnapshot as its data source and name matched original PVC we jusr deleted.Create a file named mysql-restore-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-data-mysql-0 # Match the original PVC name
  namespace: default
spec:
  # The new PVC MUST match the original StorageClass and size
  storageClassName: managed-premium 
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi # MUST be equal to or greater than the original PVC size
  dataSource: # <--- This block is the key to restoration!
    name: mysql-data-initial-snapshot # The name of your VolumeSnapshot object
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

kubectl apply -f mysql-restore-pvc.yaml
kubectl describe pvc mysql-data-mysql-0
Name:          mysql-data-mysql-0
Namespace:     default
StorageClass:  managed-premium
Status:        Pending
Volume:
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
DataSource:
  APIGroup:  snapshot.storage.k8s.io
  Kind:      VolumeSnapshot
  Name:      mysql-data-initial-snapshot
Used By:     <none>
Events:
  Type    Reason                Age               From                         Message
  ----    ------                ----              ----                         -------
  Normal  WaitForFirstConsumer  4s (x4 over 45s)  persistentvolume-controller  waiting for first consumer to be created before binding

volumeBindingMode: WaitForFirstConsumer.

What this means: Kubernetes will not provision the actual Azure Disk (or the PV derived from the snapshot) until a Pod (the “consumer”) is scheduled and requests to use this PVC. This helps the scheduler choose the best node first, preventing the node affinity conflicts we discussed earlier.

Current State: We have created the new PVC (mysql-data-mysql-0), but we have not yet scaled up the StatefulSet. Since no Pod is requesting this PVC yet, it remains in the Pending state.

Now that the PVC with the expected name (mysql-data-mysql-0) is present and sourced from the good snapshot data, we can scale the StatefulSet back up.

kubectl scale statefulset mysql --replicas=1

Testing that data were restored

kubectl exec -it mysql-0 -- mysql -u root -p"my-secret-password" -e "SELECT * FROM snapshot_test_db.inventory;"
mysql: [Warning] Using a password on the command line interface can be insecure.
+----+-----------+----------+---------------------+
| id | item_name | quantity | data_creation_time  |
+----+-----------+----------+---------------------+
|  1 | Widget A  |      100 | 2025-11-03 09:21:34 |
|  2 | Gadget B  |       50 | 2025-11-03 09:21:34 |
+----+-----------+----------+---------------------+

Leave a comment

Share

Simple Web frontend Go application and backed MySQL server were aimed for a production-ready setup. Application is deployed to Azure Kubernetes cluster and Azure Docker registry.Along the way, this pipeline includes:

• Setting up Go-specific CI tasks (testing, linting, code coverage).

• Building and scanning Docker images.

• Automating deployments to Dev and Production namespaces using Helm.

• Configuring Nginx Ingress for traffic routing.

• Automating SSL/TLS certificates with Cert-Manager.

Variables are stored in AzureDevOps portal

Code location: my github repository

The CI Stage: Ensuring Code Quality and Container Security

Our pipeline begins with a Build stage, focused on ensuring our Go application’s quality and container integrity.

1. Code Quality (SonarCloud):

   • SonarCloudPrepare@3: Configures SonarCloud for static code analysis, integrating test results and coverage reports.

   

   • Bash@3 (go fmt): Enforces consistent code formatting.

   • Bash@3 (go mod tidy): Manages Go modules, ensuring dependencies are clean and accurate.

1. Testing and Coverage:

   • Bash@3 (Go Tests): Runs unit tests, generates JUnit XML reports (for test results) and Cobertura XML reports (for code coverage) using go-junit-report and gocover-cobertura. It also generates an HTML coverage report.

   • PublishTestResults@2 & PublishCodeCoverageResults@2: Integrates these reports directly into Azure DevOps for easy visualization.

2. Application Build (Go@0): Compiles the Go application.

3. SonarCloud Analysis & Publish: Runs the actual analysis and publishes results to SonarCloud.

4. Docker Image Build & Push (DockerCompose@1):

   • Builds the Docker image for the Go application.

   • Pushes the image to Azure Container Registry (ACR) with a unique Build.BuildId tag, and importantly, also with latest for easy local testing.

5. Container Security Scan (Trivy):

   • Downloads the Trivy JUnit template.

   • Logs into ACR.

   • Bash@3 (Trivy Scan): Scans the newly built Docker image for vulnerabilities (High and Critical severity) and generates a JUnit XML report. We even included logic to create a dummy report if the scan fails or finds no issues, ensuring the pipeline doesn’t break.

   • PublishTestResults@2: Publishes Trivy results to Azure DevOps.

In this stage Build and publish artifacts tasks are actually not needed because building is done in next stage,but posted anyway for demonstrations

The CD Stages: Deploying to Dev and Production

After a successful build, the pipeline moves to the deployment stages. We maintain strict separation between Dev and Production environments.

Prerequisites (Manual Setup - Run Once Per Cluster):

Before these stages run, four critical components must be installed directly in Kubernetes cluster:

1. Nginx Ingress Controller: This acts as the external entry point (LoadBalancer) for all the traffic. It handles routing requests based on hostnames.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install nginx-ingress ingress-nginx/ingress-nginx `
--namespace ingress-nginx --create-namespace `
--set controller.replicaCount=1 `
--set controller.service.type=LoadBalancer `
--set controller.service.externalTrafficPolicy=Local `
--set controller.admissionWebhooks.enabled=false

Get Nginx controller public IP and set it in DNS registar

kubectl get services -n ingress-nginx
NAME                                     TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)                      AGE
nginx-ingress-ingress-nginx-controller   LoadBalancer   10.0.116.127   132.220.178.206   80:31906/TCP,443:31611/TCP   76s

2. Cert-Manager: This essential tool automates the provisioning, management, and renewal of TLS certificates from sources like Let’s Encrypt.

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install `
 cert-manager jetstack/cert-manager `
 --namespace cert-manager `
 --version v1.18.1 `
 --set installCRDs=true   

3. Kubrenetes cluster permissions to ACR:Make sure AKS cluster managed entity has ACRPull role assigned to Azure Container registry.

az aks update --resource-group <RESOURCE_GROUP_NAME> --name <AKS_CLUSTER_NAME> --attach-acr <ACR_NAME>

4. ClusterIssuer resource for cert-manager:Its purpose is to obtain SSL/TLS certificates from Let’s Encrypt, using the HTTP-01 challenge, which is essential for enabling HTTPS on dev and prod Kubernetes services

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: drvucanovic@gmail.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    # Enable the HTTP-01 challenge mechanism
    solvers:
    - http01:
        ingress:
          class: nginx

Deploy to Dev Stage (Deploy_to_dev)

This stage deploys the application to the dev namespace, making it accessible for testing and verification.

Key Steps in the Dev Deployment:

1. kubectl Installer: Ensures kubectl is available.

2. Namespace Creation: Creates the dev namespace if it doesn’t already exist.

3. Secrets Creation (app-config): Creates a Kubernetes Secret to hold sensitive application configurations (like database credentials).

4. Helm Deployment (HelmDeploy@1):

   • This is the core of the deployment. It uses our Helm chart (my-go-app-chart) and the values.dev.yaml file.

   • Crucially, this step triggers Cert-Manager. The ingress.yaml template, when rendered by Helm, contains annotations (cert-manager.io/cluster-issuer) and tls blocks that Cert-Manager watches. Upon detecting these, Cert-Manager automatically requests and stores the TLS certificate (bigfirm-online-tls) for dev.bigfirm.online in the dev namespace.

5. Deployment Restart: Performs a kubectl rollout restart to ensure the application deployment picks up any new image or secret changes.

Deploy to Production Stage (Deploy_to_production)

This stage mirrors the Dev deployment but targets the prod namespace with production-specific configurations.Key Steps in the Production Deployment:

1. kubectl Installer: Same as Dev.

2. Namespace Creation: Creates the prod namespace.

3. Secrets Creation (app-config): Creates production-specific secrets. Important: These should come from separate, more secure pipeline variables (PROD_MYSQL_PASSWORD, etc.) to maintain environment isolation.

4. Helm Deployment (HelmDeploy@1):

   • Deploys the Helm chart using values.prod.yaml.

   • This file specifies prod.bigfirm.online as the production hostname and a dedicated TLS Secret name (bigfirm-prod-tls).

   • Again, Cert-Manager automatically provisions the TLS certificate for prod.bigfirm.online and stores it in bigfirm-prod-tls within the prod namespace. No manual certificate steps are needed!

5. Deployment Restart: Restarts the production Go application.

Subscribe now

Share

Leave a comment

One of the biggest challenges in maintaining Infrastructure as Code (IaC) pipelines is avoiding repetition. If you have five environments (Dev, QA, Staging, Prod-A, Prod-B), copying the 50-line Terraform Plan logic five times is a recipe for error.

The solution would be Azure DevOps YAML Stage Templates.

This post breaks down a powerful, reusable, and secure pattern for running Terraform operations that only proceeds to the deployment stage if actual infrastructure changes are detected.

The main pipeline file azure-pipelines.yml serves as the “Table of Contents.” It sets global variables and, crucially, defines the stages by referencing templates. It passes dynamic information (like the working directory) to the reusable templates using parameters.

Key Points:

• DRY Principle: We call a template, we don’t duplicate the steps.

• Variable vs. Parameter: We use $(tfWorkingDir) (a variable) in the main file and pass its value to the template using the tfWorkingDir parameter.

• Stage Dependency: The Deploy stage explicitly sets dependsOnStage: Plan, which we use in the template to enforce order.

Variables ($() syntax)

• Variables are typically processed during the runtime of the pipeline (after the compilation/template expansion phase).

• They are used for values that might change during the job execution (like iteration counts, file paths, or values set by previous tasks).

• They are referenced using the $(variableName) syntax.

• $(tfWorkingDir) is defined in the variables: block of the main azure-pipelines.yml. It’s a pipeline-level setting that remains constant throughout the run, but YAML treats it as a runtime variable.

Parameters (${{ }} syntax)

• Parameters are evaluated and substituted before the pipeline starts running, during a process called template expansion (or compilation). They are compile-time concepts.

• They are essential for defining reusability and controlling the structure of the pipeline. They allow you to dynamically set job names, stage names, conditions, and other structural YAML elements.

We cannot directly use a runtime variable ($(tfWorkingDir)) to define a compile-time structural element (like the workingDirectory of a task input within the template) because the compiler needs to know the final structure of the pipeline before any variables are resolved at runtime.

The process works like this:

1. Main File (azure-pipelines.yml): Passes the value of the variable $(tfWorkingDir) (which is 'app-service-and-sql-database') as an argument to the template’s parameter block:

   YAML

# Main File
- template: templates/terraform-plan-stage.yml
  parameters:
    tfWorkingDir: '$(tfWorkingDir)' # Passes the VALUE 'app-service-and-sql-database'

1. Template Expansion: Azure DevOps takes the value 'app-service-and-sql-database' and substitutes it everywhere the parameter ${{ parameters.tfWorkingDir }} is used in the template.

2. Final Pipeline: The resulting, expanded YAML file (which is what actually runs) now has the hardcoded path where the parameter was used. This ensures the structural parts of the pipeline (like the artifact path or job inputs) are correctly defined before execution.

The Plan Template:templates/terraform-plan-stage.yml

This template defines the entire Plan Stage logic, including initialization, planning, and, most critically, detecting changes.

A. Parameters Block

The template declares the inputs it expects from the orchestrator:

parameters:
- name: stageName
  type: string
# ...
- name: tfWorkingDir
  type: string

Conditional Output Variable

The magic of conditional deployment happens here. We use a PowerShell task to execute terraform show -json tfplan, analyze the output for any resource changes (create, update, delete), and then set an output variable:

# Inside the PowerShell task:
script: |
  # ... logic to determine $anyTfChanges ...
  Write-Host “##vso[task.setvariable variable=anyTfChanges;isOutput=true]$anyTfChanges”
name: ‘SetChangesVariable’

This template runs the terraform apply command, but only if the previous Plan stage was successful and detected changes.

The Deploy Template: templates/terraform-deploy-stage.yml

This template runs the terraform apply command, but only if the previous Plan stage was successful and detected changes.

Code is published on my GitHub repo and code is already explained in my previous post,if you’re interested,check bellow text.

Subscribe now

Share

Leave a comment

Example terraform file which deploy SQL server and Azure Windows services:

main.tf

terraform {
  required_providers {
    # 1. This is the resource provider namespace and name.
    azurerm = {
      source  = "hashicorp/azurerm"
      # 2. Specify a version constraint (use a recent version)
      version = "~> 3.0”"
    }
    
  }
  }

provider "azurerm" {
  # This block is mandatory.
  # It tells Terraform how to configure the features of the AzureRM provider.
  features {}
}

resource "azurerm_resource_group" "RG-Terraform" {
  name     = “terraform-resource-group”
  location = “West Europe”
}

resource "azurerm_service_plan" "ASP-TerraForm" {
  name                = “terraform-appserviceplan”
  location            = azurerm_resource_group.RG-Terraform.location
  resource_group_name = azurerm_resource_group.RG-Terraform.name
 os_type             = “Windows”
  sku_name            = “P1v2”
}

resource "azurerm_windows_web_app" "AS-Terraform" {
  name                = "app-service-terraform-2027-10-08"
  location            = azurerm_resource_group.RG-Terraform.location
  resource_group_name = azurerm_resource_group.RG-Terraform.name
  service_plan_id = azurerm_service_plan.ASP-TerraForm.id
 
  
  site_config {}

  app_settings = {
    “SOME_KEY” = "some-value"
  }

  connection_string {
    name  = "Database"
    type  = "SQLServer"
    value = "Server=tcp:${azurerm_mssql_server.terraform-sqlserver.fully_qualified_domain_name} Database=${azurerm_mssql_database.terraform-sqldatabase.name};User ID=${azurerm_mssql_server.terraform-sqlserver.administrator_login};Password=${azurerm_mssql_server.terraform-sqlserver.administrator_login_password};Trusted_Connection=False;Encrypt=True;"
  }
}

resource “azurerm_mssql_server” "terraform-sqlserver" {
  name                         = "terraform-sqlserver"
  resource_group_name          = azurerm_resource_group.RG-Terraform.name
  location                     = azurerm_resource_group.RG-Terraform.location
  version                      = "12.0"
  administrator_login          = "houssem"
  administrator_login_password = "4-v3ry-53cr37-p455w0rd"
   minimum_tls_version          = "1.2"
}

resource "azurerm_mssql_database" "terraform-sqldatabase" {
  name                = "terraform-sqldatabase"
  server_id           = azurerm_mssql_server.terraform-sqlserver.id
  collation    = "SQL_Latin1_General_CP1_CI_AS"
  license_type = "LicenseIncluded"
  max_size_gb  = 2
  sku_name     = "S0"
  enclave_type = "VBS"
  tags = {
    environment = "production"
  }
}

Terraform state file is stored in Azure storage account,in container tfstate, Azure DevOps ServiceConnection PrincipalID has Storage Blob Data Contributor role on the container where terraform.tfstate file is hosted.

This YAML file defines an Azure DevOps multi-stage pipeline for deploying infrastructure to Azure using Terraform. It implements a robust, two-stage Plan and Apply workflow that includes conditional deployment.

YAML Pipeline Description

pool:
  vmImage: 'ubuntu-latest'

variables:
  
  azureServiceConnectionName: '(1234ereee-daf0-etetttetec)'
  storageAccountResourceGroup: resourcegroup-tfstate
  storageAccountName: terraformstate20251009
  containerName: tfstate
  tfstateFile: terraform.tfstate
  tfWorkingDir: 'app-service-and-sql-database'

stages:
- stage: Plan
  displayName: 'Terraform Plan'
  jobs:
  - job: TerraformPlan
    displayName: 'Run Plan'
    
    steps:
      # 1. INSTALL TERRAFORM
      - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
        displayName: 'Install Terraform'
        inputs:
          terraformVersion: '1.13.3'
          
      # INIT AND PLAN (Combined and simplified script)
      - task: TerraformTaskV4@4
        displayName: 'Terraform Init'
        inputs:
          provider: 'azurerm'
          command: 'init'
          workingDirectory: '$(tfWorkingDir)'
          backendServiceArm: '$(azureServiceConnectionName)'
          backendAzureRmResourceGroupName: '$(storageAccountResourceGroup)'
          backendAzureRmStorageAccountName: '$(storageAccountName)'
          backendAzureRmContainerName: '$(containerName)'
          backendAzureRmKey: '$(tfstateFile)'
     
     # PLAN: Use the 'TerraformCLI' task for the plan command
      - task: TerraformTaskV4@4
        displayName: 'Terraform Plan'
        inputs:
          provider: 'azurerm'
          command: 'plan'
          workingDirectory: '$(tfWorkingDir)'
          backendServiceArm: '$(azureServiceConnectionName)'         
          environmentServiceNameAzureRM: '$(azureServiceConnectionName)'
          commandOptions: '-out=tfplan'
        name: 'TerraformPlanOutput'

      # CHECK PLAN OUTPUT FOR CHANGES AND SET A PIPELINE VARIABLE
      - task: PowerShell@2
        displayName: 'Detect Terraform Changes via JSON and Set Variable'
        inputs:
          # Use the correct working directory where tfplan resides:
          workingDirectory: '$(tfWorkingDir)'
          targetType: 'inline'
          script: |
            # Set initial variable value
            $anyTfChanges = "false”"

            # 1. Show the plan file and convert the output from JSON to a PowerShell object
            Write-Host “--- Running 'terraform show -json tfplan' ---”
            # Use -ErrorAction Stop to fail the script if 'tfplan' is missing or unreadable
            $plan = terraform show -json tfplan | ConvertFrom-Json -ErrorAction Stop
            
            # 2. Extract the list of unique actions (including 'replace')
            # Using Select-Object -Unique ensures each action type is listed only once.
            # Add 'replace' as it signifies a destroy/create change.
            $actions = $plan.resource_changes.change.actions | Select-Object -Unique
            
            # 3. Check for any action that indicates a change
            $hasChanges = $false
            if ($actions -contains 'create' -or $actions -contains 'delete' -or $actions -contains 'update' -or $actions -contains 'replace')
            {
              $hasChanges = $true
              $anyTfChanges = "true"
              # Join the unique actions array into a comma-separated string for a cleaner message
              $actionsString = $actions -join ', '
              Write-Host "Terraform changes detected! Actions: $actionsString"
            }
            else
            {
              Write-Host "No change detected in Terraform tfplan file."
            }
            
            # 4. Set the output variable
            Write-Host "--- Setting Pipeline Variable ---"
            Write-Host "##vso[task.setvariable variable=anyTfChanges;isOutput=true]$anyTfChanges"
            
            # 5. Output the final value for immediate verification
            Write-Host "SetChangesVariable.anyTfChanges is set to: $anyTfChanges"

          pwsh: true # Ensure PowerShell Core is used
        name: 'SetChangesVariable'
       

      # PUBLISH THE PLAN ARTIFACT (Required for the Apply stage)
      - publish: '$(tfWorkingDir)/tfplan'
        artifact: drop-tfplan
        displayName: 'Publish Plan Artifact
        # Publish only if changes exist
        condition: eq(variables['SetChangesVariable.anyTfChanges'], 'true')
       
          

- stage: Deploy
  displayName: 'Terraform Apply'
  dependsOn: Plan
  condition: and(succeeded('Plan'), eq(dependencies.Plan.outputs['TerraformPlan.SetChangesVariable.anyTfChanges'], 'true'))
  jobs:
  
  - job: TerraformApply
    displayName: 'Run Apply'
    steps:
      # 1. DOWNLOAD ARTIFACT
      - task: DownloadPipelineArtifact@2
        displayName: 'Download tfplan Artifact'
        inputs:
          artifactName: 'drop-tfplan'
          targetPath: '$(tfWorkingDir)'

      # INSTALL TERRAFORM
      - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
        displayName: 'Install Terraform'
        inputs:
          terraformVersion: '1.13.3'

      # ESTABLISH AZURE CREDENTIALS (Optional but safe)
      - task: AzureCLI@2
        displayName: 'Establish Azure Credentials (ARM_* variables)'
        inputs:
          azureSubscription: '$(azureServiceConnectionName)'
          scriptType: 'bash'
          scriptLocation: 'inlineScript'
          inlineScript: 'echo "Azure credentials re-established."'

      # TERRAFORM INIT
      - task: TerraformTaskV4@4
        displayName: 'Terraform Init for Apply'
        inputs:
          provider: 'azurerm'
          command: 'init'
          workingDirectory: '$(tfWorkingDir)'
          backendServiceArm: '$(azureServiceConnectionName)'
          backendAzureRmResourceGroupName: '$(storageAccountResourceGroup)'
          backendAzureRmStorageAccountName: '$(storageAccountName)'
          backendAzureRmContainerName: '$(containerName)'
          backendAzureRmKey: '$(tfstateFile)'

      # TERRAFORM APPLY
      - task: TerraformTaskV4@4
        displayName: 'Terraform Apply'
        inputs:
          provider: 'azurerm'
          command: 'apply'
          workingDirectory: '$(tfWorkingDir)'
          environmentServiceNameAzureRM: '$(azureServiceConnectionName)'
          commandOptions: 'tfplan'


      

The pipeline is structured into two main stages: Plan and Deploy.

Global Configuration and Variables

• pool: vmImage: 'ubuntu-latest': Specifies that all jobs will run on a Microsoft-hosted agent using the latest Ubuntu image.

• variables:: Defines reusable parameters, primarily for Azure Service Connection details and Terraform backend configuration (storing the state file in an Azure Storage Account, which is best practice).

  • The tfWorkingDir variable points to the location of the Terraform configuration files ('app-service-and-sql-database').

Stage 1: Plan (Plan)

This stage’s purpose is to install Terraform, initialize the backend, generate an execution plan, check if the plan contains any changes, and save the plan for the next stage.

1. Install Terraform: Uses the TerraformInstaller@0 task to ensure the specific version (1.13.3) is available on the agent.

2. Terraform Init: Uses TerraformTaskV4@4 to run terraform init. This task downloads the Azure provider and configures the Azure Backend to manage the state file (terraform.tfstate) in the specified Azure Storage Account.

3. Terraform Plan: Uses TerraformTaskV4@4 to run terraform plan with the option -out=tfplan. This saves the proposed changes to a binary file named tfplan, ensuring the exact planned operations are used later for deployment.

4. Detect Changes (PowerShell): A PowerShell@2 task runs terraform show -json tfplan to inspect the saved plan. It checks the resources for actions (create, delete, update, replace) and sets an output variable (anyTfChanges) to true or false.

5. Publish Artifact: The publish task creates a pipeline artifact (drop-tfplan) containing the tfplan file. This step is conditional (condition: eq(variables['SetChangesVariable.anyTfChanges'], 'true')), meaning the plan artifact is only created if actual changes were detected.

Stage 2: Deploy (Deploy)

This stage’s purpose is to execute the deployment, but only if changes were detected in the Plan stage.

• Conditional Execution: The entire stage runs only if the Plan stage succeeded AND the output variable from the Plan stage indicates changes:

condition: and(succeeded('Plan'), eq(dependencies.Plan.outputs['TerraformPlan.SetChangesVariable.anyTfChanges'], 'true'))

• Download Artifact: Downloads the tfplan file created in the previous stage.

• Terraform Init (Again): Re-initializes Terraform to ensure the backend and providers are correctly set up on the new agent running the Deploy job.

• Terraform Apply: Uses TerraformTaskV4@4 to run terraform apply. Crucially, it uses commandOptions: tfplan; to apply the pre-approved and verified plan file, preventing any drift or unexpected changes since the plan was created.

Why We Need Two TerraformTaskV4@4 Tasks ?

The reason is that Terraform commands are executed independently based on the command: input.

The two tasks perform two distinct, required steps in the standard Terraform workflow:

1. TerraformTaskV4@4 (for init):

   • Command: init

   • Purpose: Initializes the Terraform working directory, downloads the necessary providers (for Azure in this case), and configures the remote backend (the Azure Storage Account). This must be done first.

2. TerraformTaskV4@4 (for plan):

   • Command: plan

   • Purpose: Compares the desired state (your HCL code) with the current remote state in Azure, and generates a detailed execution plan. This cannot run until the directory is initialized.

It’s necessary to have a separate task for each command (init and plan) because the task is designed to wrap a single terraform command at a time. This modular approach is standard practice in CI/CD pipelines.

This code is being re-factored into AzureDevOps template,see bellow post:

Subscribe now

Share

Leave a comment

Bicep is a domain-specific language (DSL) for deploying Azure resources declaratively. It's a more readable and concise alternative to JSON-based Azure Resource Manager (ARM) templates.

Why Use Bicep?

You should use Bicep because it simplifies the process of authoring Infrastructure as Code (IaC) for Azure. Its key advantages include:

• Simplified Syntax: Bicep's syntax is much cleaner and less verbose than ARM templates, which reduces boilerplate and makes your code easier to read and maintain. You can create resources without the repetitive type, apiVersion, and properties sections found in JSON.

• Modularity: Bicep makes it easy to break down large deployments into reusable modules. This promotes code reuse across projects, improves readability, and allows different team members to work on separate components without affecting the main template.

• Improved Safety: The Bicep VS Code extension provides a rich authoring experience with IntelliSense, syntax highlighting, and real-time validation. This allows you to catch errors and typos before you even run the deployment.

• Automatic Dependency Management: Bicep automatically detects and manages dependencies between resources. By referencing one resource in another (for example, a virtual machine using a virtual network's ID), Bicep understands the correct deployment order, eliminating the need for explicit dependsOn arrays.

• Ease of Adoption: You can decompile existing ARM templates into Bicep code using the Bicep CLI, making it easy to transition your current infrastructure and start using the new language.

Prerequisites

Make sure you have Powershell 5.6.0 or higher (Check it with $PSVersionTable.PSVersion command)

If needed,install PowerShell 7:winget install --id Microsoft.PowerShell --source winget

Install AZ module:

Install-module az

import-module az

Connect to Azure:

Connect-AzAccount

Install Bicep tool

Not needed but,highly recommended,install VS code and Bicep extension

Bicep code is available here.

What are Bicep Modules?

Think of a Bicep module as a self-contained unit of deployment. It's a Bicep file (.bicep) that can be called from another Bicep file, known as the parent or root template. This approach promotes several key benefits:

• Reusability: Write a module once and reuse it across multiple projects. For example, a module for a virtual network can be used in every deployment that requires one, ensuring consistency and saving time.

• Readability: Large, monolithic templates can be hard to read and understand. Modules help by logically organizing your code, making the main template easier to follow.

• Maintainability: Changes only need to be made in one place. If you need to update a resource configuration, you can modify the relevant module without touching the entire codebase.

• Reduced Complexity: By abstracting away the details of individual resource creation, the main template focuses on the overall architecture and how different components fit together.

Deconstructing a Bicep Deployment: A Practical Example

Let's examine the main.bicep file you provided to see how these principles work in practice. The main file acts as an orchestrator, defining the overall architecture and calling individual modules for each major component.

🛠️ The main.bicep Orchestrator

The main file starts by defining parameters and variables that are passed down to the modules. This centralizes configuration and makes the deployment flexible. For instance, the location and environment parameters are used throughout the deployment.

param location string = 'westeurope'
param environment string = 'dev'
// ... and many more

Next, it creates resource groups to logically organize the deployed resources.

resource vnetRg 'Microsoft.Resources/resourceGroups@2021-04-01' = {
  name: vnetRgName
  location: location
}
// ... and other resource groups

Calling the Modules

The main.bicep file calls modules for each part of the infrastructure: a virtual network, a bastion host, a virtual machine, a key vault, and a load balancer.

1. Virtual Network (VNet) Module 🌐

The vnetModule call deploys the network infrastructure. Notice how the parent template simply passes parameters to the module without needing to know the low-level details of how the VNet and its subnets are created.

Фрагмент кода

module vnetModule 'module/vnet.bicep' = {
  name: 'deployVNet'
  scope: resourceGroup(vnetRgName)
  params: {
    vnetName: vnetName
    // ... other parameters
    deployWANSubnet: deployWANSubnet
  }
  dependsOn: [
    vnetRg
  ]
}

The dependsOn: [ vnetRg ] line is crucial. It tells Bicep that the vnetModule deployment can't start until the vnetRg resource group has been successfully created. This establishes the correct deployment order.

2. Bastion Host Module 🔒

The bastionModule is called conditionally using an if statement. This demonstrates how modules can be deployed based on a parameter's value, making your templates even more dynamic.

module bastionModule 'module/bastion.bicep' = if (deployBastionSubnet) {
  name: 'deployBastionHost'
  scope: resourceGroup(bastionRgName)
  params: {
    bastionSubnetID: vnetModule.outputs.bastionSubnetId
    // ... other parameters
  }
}

This module requires the bastionSubnetId from the VNet module. Bicep automatically understands this dependency, so the VNet module will always deploy first. The vnetModule.outputs.bastionSubnetId syntax is how a parent template accesses values exported by a module.

The same pattern is repeated for the other components:

• vmModule deploys the virtual machines, also using a conditional if statement and a lookup map to get the correct subnet ID.

• vaultModule handles the Key Vault deployment, a standard practice for securely storing secrets.

• balancerModule deploys the load balancer, demonstrating how complex resources can be encapsulated in a single module.

Outputs

Finally, the main.bicep file defines outputs, which allow the results of the deployment (like resource IDs or public IP addresses) to be returned to the user. This is particularly useful for scripting and automation.

output vnetId string = vnetModule.outputs.vnetId
output bastionHostId string = deployBastionSubnet ? bastionModule.outputs.bastionHostId 

Module for deploying an Azure Load Balancer

This Bicep module provides a flexible and comprehensive solution for deploying an Azure Load Balancer. It is designed to be highly configurable, allowing you to define various aspects of your load balancer deployment, including the environment, project name, location, and key resource properties.

With this module, you can easily:

• Create a Public IP Address and associate it with the load balancer.

• Define Load Balancing Rules for different ports and protocols, ensuring traffic is distributed effectively to your backend resources.

• Configure Health Probes to monitor the health of your backend instances, ensuring traffic is only sent to healthy virtual machines.

• Set up Inbound NAT Rules to map specific ports on the load balancer's public IP to individual virtual machines in the backend pool.

Module for Deploying Azure Bastion

Azure Bastion provides a managed, secure, and seamless way to connect to your VMs directly from the Azure portal over a private network connection. The module creates two primary resources:

• Public IP Address (bastionPublicIp): Every Bastion Host requires a public IP address to allow secure communication from the Azure portal. This module automatically creates a Standard SKU Public IP with a static allocation method, which is a requirement for Bastion. It also uses the zoneCount parameter to make the IP address zone-redundant, matching the Bastion Host's availability settings.

• Bastion Host (bastionHost): This is the core resource. It's configured to use the public IP address and the dedicated Bastion subnet provided as parameters. The tags variable ensures that the resource is properly labeled for your organization. The ipConfigurations property links the Bastion Host to the public IP address and the AzureBastionSubnet, establishing the network connectivity it needs to function.

Module for Deploying Azure Vault

🔐 Secure by Default

The module configures the Key Vault with enableRbacAuthorization: true, which uses Azure Role-Based Access Control instead of the older access policies model. This provides more granular control and aligns with Azure's recommended security practices. It also sets defaultAction: 'Deny' for network access, meaning only traffic from explicitly whitelisted IPs or trusted Azure services can connect to the vault.

👥 Granular Access Control

Instead of granting a single broad "Key Vault Administrator" role, the module assigns two more specific built-in roles to the specified objectId:

• Key Vault Crypto Officer: Manages cryptographic keys.

• Key Vault Secrets Officer: Manages secrets.

This follows the principle of least privilege, ensuring the principal only has the permissions it needs. The module also automatically assigns the required roles for Azure Disk Encryption to a fixed service principal, enabling it to access secrets and keys for encrypting virtual machine disks.

🔑 Secret Provisioning

The module also provisions a new secret within the vault. The secretName and secretValue parameters allow you to create an initial secret as part of the deployment, which is a common practice for bootstrap configurations or storing initial credentials.

Module for deploying Azure VM

The module leverages Bicep's advanced features to create resources conditionally and dynamically:

• Multiple VMs: The numberOfVMs parameter allows you to deploy multiple VMs using a single for loop. The module will create a corresponding VM, network interface, data disk, and public IP (if enabled) for each instance.

• Public IP Addresses: The createPublicIp boolean parameter determines whether a public IP address is provisioned and attached to the VM's network interface, giving you control over public exposure.

• OS-Specific Configuration: Using Bicep's ternary operators (?:), the module dynamically selects the correct OS image and profile settings (e.g., SSH key for Linux vs. password for Windows), reducing the need for separate templates.

• Availability Zones: The zoneCount parameter ensures VMs are distributed across availability zones for high availability and resilience. The mod operator (%) is used to cycle through the zones, distributing VMs evenly.

Network Integration & Load Balancer Support 🌐

The module seamlessly integrates the VMs into your existing network infrastructure:

• Subnet Integration: It uses the subnetId parameter to place the VMs in a pre-existing subnet, ensuring they are part of your established virtual network.

• Load Balancer Backend Pool: The loadBalancerBackendPoolId parameter allows you to automatically add the VM's network interface to a load balancer backend pool. This is a crucial step for deploying web servers or other scalable applications behind a load balancer.

Module for deploying Virtual Network

Using a set of boolean parameters (e.g., deployWANSubnet, deployDMZSubnet), you can choose which subnets to deploy. This allows you to use the same Bicep file to provision a wide range of network topologies, from a simple development environment with just a couple of subnets to a complete production environment with multiple security zones.

🔐 Automated Security with NSGs

The module enforces a secure, multi-tier architecture by automatically creating and associating Network Security Groups (NSGs) with each deployed subnet. The NSGs come pre-configured with a set of security rules that define the flow of traffic between the different tiers. This ensures that:

• WAN Subnet can be accessed from the Management subnet for administrative purposes.

• DMZ Subnet is exposed to the internet for public-facing services (e.g., web servers).

• App Subnet can only receive traffic from the DMZ.

• Database Subnet can only be accessed from the App subnet.

This "deny-by-default" approach ensures that traffic is strictly controlled, significantly reducing your attack surface. It also includes specific rules to allow connections from a conditionally deployed AzureBastionSubnet for secure remote access.

Leave a comment

Share

In this WordPress deployment on Kubernetes, an Azure Private Endpoint is used to enhance the security and connectivity of the Azure Database for MySQL Flexible Server.Let’s encrypt SSL cert is assigned to wodpress instance.

Here's why Private Endpoint was used:

• Secure Connectivity: A Private Endpoint provides a private IP address for Azure MySQL Flexible Server within your Azure Virtual Network (VNet). This means that traffic between AKS cluster (which is integrated into the same VNet) and the MySQL database travels entirely within the Microsoft Azure backbone network, rather than over the public internet.

• Reduced Data Exfiltration Risk: By keeping database traffic private, it significantly reduces the risk of data exfiltration and provides a more isolated and secure environment for your sensitive database.

• Simplified Network Configuration: It simplifies your network architecture by removing the need for public IP addresses or complex firewall rules to access database from within VNet.

Private DNS zone:

Is used to enable secure, private connections to the backend Azure Database for MySQL.

• When we secure your Azure database instance using a Private Endpoint, that endpoint is only accessible via a private IP address within Azure Virtual Network (VNet).

• Name Resolution: The database service must resolve to that private IP address, not the public one. A Private DNS Zone is the authoritative source for this private name resolution. It contains the necessary “A” records to tell the Pods in AKS cluster (where WordPress runs) the correct private IP for the database.

Without the Private DNS Zone, WordPress Pods would try to resolve the database name publicly, fail to connect over the private network, or potentially connect to a public endpoint if one is available, which is highly insecure.

Using Azure Database for MySQL Flexible Server offers several significant advantages over deploying MySQL as a Kubernetes pod directly within cluster, particularly for production-grade WordPress deployments:

1. Reduced Operational Overhead:

   • Managed Service: Azure handles routine database administration tasks such as patching, backups, security updates, monitoring, and scaling. If we were running MySQL in a Kubernetes pod, we would be responsible for all these tasks, requiring significant expertise and effort.

   • No Kubernetes Complexity for Database: Managing stateful applications like databases in Kubernetes can be complex (e.g., persistent volumes, stateful sets, upgrades, replication). A managed service abstracts this complexity.

2. Built-in High Availability and Disaster Recovery:

   • Azure MySQL Flexible Server offers built-in high availability with automatic failover to a hot standby in a different availability zone (or within the same zone), ensuring database remains accessible even during infrastructure failures.

   • Automated backups with point-in-time restore capabilities provide robust disaster recovery. Replicating a production-ready HA/DR setup for MySQL within Kubernetes would be a substantial engineering effort.

3. Scalability and Performance:

   • Flexible Server is designed for scalable performance, allowing easily scale compute (vCores) and storage independently, often with minimal downtime.

   • It's optimized for database workloads, potentially offering better performance than a generic pod running on a shared Kubernetes node.

4. Enhanced Security and Isolation:

   • As highlighted before, using a private endpoint with Flexible Server ensures that traffic between AKS cluster and the database travels over Azure's private backbone network, significantly reducing exposure to the public internet and enhancing data security.

   • Azure applies security best practices and compliance certifications to its managed services.

5. Cost Efficiency (Total Cost of Ownership):

   • While there's a direct cost for the managed service, the total cost of ownership (TCO) can be lower due to reduced operational effort, less need for specialized database administrators, and optimized resource utilization compared to self-managing a highly available MySQL cluster on Kubernetes.

Code can be found here.

• cluster-issuer.yaml: This file defines a ClusterIssuer resource for cert-manager. Its purpose is to obtain SSL/TLS certificates from Let's Encrypt, using the HTTP-01 challenge, which is essential for enabling HTTPS on your WordPress site.

• mysql-azure-secret.yaml: This file contains two Kubernetes Secret resources. One secret (wordpress-azure-mysql-secret) stores sensitive connection details for your Azure MySQL Flexible Server, including the host, username, password, and database name. The other secret (wordpress-auth-secret) stores WordPress authentication unique keys and salts.

• wordpress-config-map.yml: This file defines a ConfigMap named php-config. It holds custom PHP settings such as upload_max_filesize, memory_limit, and max_execution_time, which are applied to the WordPress pods.

• wordpress-deployment.yaml: This file defines the Kubernetes Deployment for the WordPress application. It specifies that 3 replicas of the WordPress pod should be created, includes an initContainer to correct permissions on the wp-content directory, mounts persistent storage and the PHP configuration, and injects database credentials and WordPress authentication salts as environment variables.

• wordpress-files-pvc.yaml: This file defines both a StorageClass and a PersistentVolumeClaim (PVC) for WordPress. The StorageClass (azurefile-wordpress) configures Azure Files to allow multiple WordPress pods to share the same wp-content directory (ReadWriteMany access mode). The PersistentVolumeClaim (wordpress-content-pvc) requests 5Gi of storage for the wp-content directory.

• wordpress-ingress.yaml: This file defines the Kubernetes Ingress resource. It configures NGINX Ingress to direct external traffic to the wordpress-service, automatically provisions a TLS certificate for blog.bigfirm.online using cert-manager, enforces SSL redirection, and sets proxy timeouts and body size limits.

• wordpress-service.yaml: This file defines a Kubernetes Service of type ClusterIP for the WordPress deployment. This service exposes port 80 internally within the cluster, allowing the Ingress controller to forward traffic to the WordPress pods.

When trying to secure WordPress with Let's Encrypt, i encountered a common hurdle: certificate issuance failures. A key step in resolving this involved a seemingly small but impactful flag during NGINX Ingress Controller installation: --set controller.admissionWebhooks.enabled=false. This setting disables the Ingress controller's admission webhooks, which are typically responsible for validating and mutating Kubernetes Ingress resources. In this case, disabling them removed a potential conflict point, allowing Cert-Manager to successfully create its temporary Ingress resources for the HTTP-01 challenge and obtain our SSL certificates. While admission webhooks offer valuable validation, understanding when and why to temporarily disable them can be a crucial troubleshooting technique in complex Kubernetes deployments."

After provisioning Let’s encrypt SSL cert kubectl apply -f wordpress-ingress.yaml -n wordpress, monitor cert deloyment:

If cert was not issues, ran below commands:

# Check if the Cert-Manager pods are running,You should see cert-manager, cert-# manager-cainjector, and cert-manager-webhook pods.
kubectl get pods -n cert-manager
# Check cert status
kubectl get certificate -n wordpress
kubectl describe certificate blog-bigifirm-online-tls -n wordpress
# Check cert request status
kubectl get certificaterequest -n wordpress
# Check cert order status
kubectl get order -n wordpress
# Most important, check challenge status,it will tell why cert has not been issued
kubectl get challenges -n wordpress
kubectl describe challenge <challenge-name> -n wordpress
# If all is fine, you'll get as below
kubectl get certificate -n wordpress
NAME                      READY   SECRET                    AGE
blog-bigfirm-online-tls   True    blog-bigfirm-online-tls   3m2s

Share

Leave a comment

Following PowerShell script monitors various log files and sends output to JSON file, due to large number of logs, although script executes for about 16 seconds, was getting “Timed out” errors in Zabbix, that’s why output is saved in file.Each line in log file starts with datetime stamp, for example:

2025.06.27 12:35:12 Attempting to fetch Azure AD audit sign-in logs for

Script search log file section created last time log was modified: if last time script was running today, it will search today’s date, if yesterday, then yesterday’s date and so on.

# Script for monitoring varius log files and sending results
# to JSON file which is send to Zabbix



# Define the log file for THIS script's errors (for the WriteLog function)
$log = "D:\ScriptsLogs\CheckLogErrorsForZabbix.txt"

# Define the array of log file paths to scan.
$LogFiles = @(
    "D:\ScriptsLogs\1.txt",
    "D:\ScriptsLogs\2.txt",
    "D:\ScriptsLogs\3.txt",
    "D:\ScriptsLogs\4.txt"
)

$SearchStrings = @(
    "error",
    "exception",
    "[ERROR]",
    "Exception calling"
)

# Define a BASE set of ignore patterns (these apply to ALL files)
$BaseIgnorePatterns = @(
    "Could not get FGPP for"
    "Insufficient access a to perform the operation"
    "Insufficient access rights to perform the operation"
    "Cannot find an object with identity"
    # Add other truly global ignore patterns here
)

# Define a Hashtable for file-specific ignore patterns.
# Key: Full path to the log file.
# Value: Array of strings to ignore ONLY for that specific file.
$FileSpecificIgnorePatternsMap = @{
    "D:\ScriptsLogs\1.txt" = @(
        "More than one contract"
    );
    # Add more file paths and their specific literal ignore patterns here as needed
}

# Define the path where the JSON output file will be saved
$JsonOutputFilePath = "D:\ScriptsLogs\ZabbixLogErrors.json"

# Compile search patterns into a single regex object
$combinedSearchRegexPattern = ($SearchStrings | ForEach-Object { [regex]::Escape($_) }) -join '|'

# Regex to match the date part at the very beginning of a line (e.g., 2025.06.24)
# Also allows for timestamp and then a space or tab, and captures the date itself in group 1.
$dateLineRegex = '^(\d{4}\.\d{2}\.\d{2})(?:\s+\d{2}:\d{2}:\d{2})?[\s\t]*'

# Array to store results for Zabbix - Keep this OUTSIDE the loop
$zabbixResults = @()

# region --- Main Processing Loop ---

foreach ($logFilePath in $LogFiles) {
    # Initialize a temporary result object for current file's processing status
    # This object holds the actual analysis results (status, message, date)
    $currentFileAnalysisResult = [PSCustomObject]@{
        "status"           = 0 # Default: No error found
        "latest_error_message" = ""
        "latest_error_date"    = ""
    }

    if (-not (Test-Path -LiteralPath $logFilePath -PathType Leaf)) {
        $currentFileAnalysisResult.status = 0
        $currentFileAnalysisResult.latest_error_message = "Log file not found: $logFilePath"
        $currentFileAnalysisResult.latest_error_date = (Get-Date -Format 'yyyy.MM.dd')
        WriteLog "Warning: Log file not found - $logFilePath" $log # Log this specific warning
    } else {
        # --- Step 1: Initialize current file's ignore patterns with base patterns ---
        $currentFileIgnorePatterns = [System.Collections.ArrayList]::new($BaseIgnorePatterns)

        # --- Step 2: Add file-specific ignore patterns from the map ---
        if ($FileSpecificIgnorePatternsMap.ContainsKey($logFilePath)) {
            $patternsToAdd = $FileSpecificIgnorePatternsMap[$logFilePath]
            foreach ($pattern in $patternsToAdd) {
                $currentFileIgnorePatterns.Add($pattern) | Out-Null
            }
        }

        # --- Step 3: Compile the combined ignore regex pattern for the *current file* ---
        $combinedIgnoreRegexPattern = ($currentFileIgnorePatterns | ForEach-Object { [regex]::Escape($_) }) -join '|'

        $latestDateInFile = $null # Stores the most recent date found in the current log file

        # --- Pass 1: Determine the absolute latest date in the file ---
        try {
            $reader = [System.IO.File]::OpenText($logFilePath)
            while (($line = $reader.ReadLine()) -ne $null) {
                if ($line -match $dateLineRegex) {
                    $lineDate = $Matches[1]
                    if ($latestDateInFile -eq $null -or $lineDate -gt $latestDateInFile) {
                        $latestDateInFile = $lineDate
                    }
                }
            }
            $reader.Close()
        } catch {
            $currentFileAnalysisResult.status = 0
            $currentFileAnalysisResult.latest_error_message = "Script Error (Pass 1) for $logFilePath - $($_.Exception.Message)"
            $currentFileAnalysisResult.latest_error_date = (Get-Date -Format 'yyyy.MM.dd')
            WriteLog "Error (Pass 1) for $logFilePath $($_.Exception.Message)" $log # Corrected WriteLog call
        }

        # Only proceed to Pass 2 if no errors occurred in Pass 1 and dates were found
        if ($currentFileAnalysisResult.latest_error_message -eq "" -and $latestDateInFile -eq $null) {
            # No date-stamped entries found, so no errors to report for this file
            $currentFileAnalysisResult.status = 0
            $currentFileAnalysisResult.latest_error_message = "No date-stamped entries found."
            $currentFileAnalysisResult.latest_error_date = (Get-Date -Format 'yyyy.MM.dd')
        } elseif ($currentFileAnalysisResult.latest_error_message -eq "") { # If no errors in Pass 1 and date found
            # --- Pass 2: Collect all entries for the determined latest date and process ---
            try {
                $entriesForTargetDate = [System.Collections.ArrayList]::new()
                $currentEntryLines = [System.Collections.ArrayList]::new()
                $collectingForTargetDate = $false

                $reader = [System.IO.File]::OpenText($logFilePath)
                while (($line = $reader.ReadLine()) -ne $null) {
                    if ($line -match $dateLineRegex) {
                        $lineDate = $Matches[1]
                        if ($lineDate -eq $latestDateInFile) {
                            if ($currentEntryLines.Count -gt 0) {
                                $entriesForTargetDate.Add(($currentEntryLines -join "`n")) | Out-Null
                                $currentEntryLines.Clear()
                            }
                            $currentEntryLines.Add($line) | Out-Null
                            $collectingForTargetDate = $true
                        } elseif ($collectingForTargetDate -and $lineDate -lt $latestDateInFile) {
                            if ($currentEntryLines.Count -gt 0) {
                                $entriesForTargetDate.Add(($currentEntryLines -join "`n")) | Out-Null
                            }
                            $collectingForTargetDate = $false
                            break
                        } else {
                            $currentEntryLines.Clear()
                            $collectingForTargetDate = $false
                        }
                    } elseif ($collectingForTargetDate) {
                        $currentEntryLines.Add($line) | Out-Null
                    }
                }
                $reader.Close()

                if ($currentEntryLines.Count -gt 0) {
                    $entriesForTargetDate.Add(($currentEntryLines -join "`n")) | Out-Null
                }

                $latestError = $null

                # Process entries in reverse order to find the latest non-ignored error
                for ($i = $entriesForTargetDate.Count - 1; $i -ge 0; $i--) {
                    $entry = $entriesForTargetDate[$i]
                    if ($entry -match $combinedIgnoreRegexPattern) {
                        continue
                    }
                    # Check if this entry matches any search pattern
                    if ($entry -match $combinedSearchRegexPattern) {
                        $latestError = $entry
                        break
                    }
                }

                if ($latestError) {
                    $firstLineOfError = $latestError.Split("`n")[0]
                    $currentFileAnalysisResult.status = 1
                    $currentFileAnalysisResult.latest_error_message = $firstLineOfError
                    $currentFileAnalysisResult.latest_error_date = $latestDateInFile
                } else {
                    $currentFileAnalysisResult.status = 0
                    $currentFileAnalysisResult.latest_error_message = ""
                    $currentFileAnalysisResult.latest_error_date = $latestDateInFile
                }

            } catch {
                $errorMessage = $_.Exception.Message
                if ($errorMessage -isnot [string]) {
                    $errorMessage = $errorMessage.ToString()
                }
                $currentFileAnalysisResult.status = 0
                $currentFileAnalysisResult.latest_error_message = "Script Error (Pass 2) for $logFilePath - $errorMessage"
                $currentFileAnalysisResult.latest_error_date = (Get-Date -Format 'yyyy.MM.dd')
                WriteLog "Error (Pass 2) for $logFilePath $($_.Exception.Message)" $log # Corrected WriteLog call
            }
        }
    }

    # After all processing for the current logFilePath is done,
    # create the final result object with the correct Zabbix macro and collected data.
    $fileName = (Get-Item $logFilePath).Name # Extract filename here

    $finalZabbixEntry = [PSCustomObject]@{
        "{#LOGFILE_NAME}"       = $fileName
        "original_path"          = $logFilePath # Optional: useful for troubleshooting in Zabbix GUI
        "status"                 = $currentFileAnalysisResult.status
        "latest_error_message"   = $currentFileAnalysisResult.latest_error_message
        "latest_error_date"      = $currentFileAnalysisResult.latest_error_date
    }

    # Add the result for the current file to the overall results array
    $zabbixResults += $finalZabbixEntry
}

# endregion

# region --- Save Zabbix Output to File ---

# Format the results as JSON
$zabbixOutput = @{
    "data" = $zabbixResults
} | ConvertTo-Json -Depth 100

# Save the JSON to the specified file path
try {
    # Ensure the directory exists
    $OutputDirectory = Split-Path $JsonOutputFilePath
    if (-not (Test-Path -LiteralPath $OutputDirectory -PathType Container)) {
        New-Item -Path $OutputDirectory -ItemType Directory -Force | Out-Null
    }
   $zabbixOutput | Out-File $JsonOutputFilePath -Encoding UTF8 -Force
    # Optionally, you can add a log entry here if you have a WriteLog function
    WriteLog "JSON output successfully saved to $JsonOutputFilePath" $log # Corrected WriteLog call
} catch {
    # Write to standard error if saving fails (Zabbix won't see this, but it helps debugging scheduled task)
    Write-Error "Failed to save JSON output to $JsonOutputFilePath $($_.Exception.Message)"
    WriteLog "CRITICAL: Failed to save JSON output to $JsonOutputFilePath $($_.Exception.Message)" $log
}

The script uses a two-pass approach for processing each log file primarily to ensure that it only reports the latest relevant error from the latest date present in the log file.

1. Pass 1: Determine the Absolute Latest Date in the File

   • The first pass reads through the entire log file to identify the most recent date stamp present in any log entry.

   • It uses the $dateLineRegex to find lines starting with a date (e.g., "YYYY.MM.DD").

   • The purpose of this pass is to establish a temporal context. If a log file contains entries from multiple days, the script is only interested in potential errors that occurred on the very last day activity was recorded. This prevents it from reporting old, resolved errors.

2. Pass 2: Collect and Process Entries for the Determined Latest Date

   • Once the latestDateInFile is determined, the second pass reads the log file again.

   • This time, it collects all log entries (potentially multi-line entries) that correspond to the latestDateInFile.

   • It then processes these collected entries in reverse chronological order (from newest to oldest).

   • For each entry on the latest date, it first checks if the entry matches any of the combinedIgnoreRegexPattern (base or file-specific). If it does, the entry is skipped.

   • If the entry is not ignored, it then checks if it matches any of the combinedSearchRegexPattern (error/exception keywords).

   • The first error found in this reverse search (meaning the latest non-ignored error on the latest date) is then captured as the latest_error_message.

   • If no errors are found for the latest date after filtering, the status remains clear.

Script Output:

• For each log file, the script creates a PowerShell custom object containing:

  • {#LOGFILE_NAME}: The name of the log file (for Zabbix discovery).

  • original_path: The full path to the log file.

  • status: 1 if an error was found, 0 otherwise.

  • latest_error_message: The first line of the latest non-ignored error message, if found.

  • latest_error_date: The date of the latest_error_message.

• Finally, all these individual file results are compiled into a single JSON object under a "data" key and saved to the JsonOutputFilePath. This JSON file is what Zabbix would typically consume to update its monitoring dashboards and trigger alerts.

Zabbix configuration

Edit Zabbix agent configuration file:"C:\Program Files\Zabbix Agent\zabbix_agentd.conf" then restart Zabbix service.

# UserParameter for JSON file last modification time
UserParameter=file.mtime[*],powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "Get-Item '$1' | Select-Object -ExpandProperty LastWriteTimeUtc | ForEach-Object { [long][double]((Get-Date $_ -UFormat %s) + 0.5) }"


# UserParameter for Log Error Discovery (used by Zabbix LLD)
# This will now read the pre-generated JSON from the file
UserParameter=log.errors.discover,powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "Get-Content \"D:\ScriptsLogs\ZabbixLogErrors.json\""

# UserParameter for Log Error Status (per discovered log file)
# This reads from the file, converts JSON, filters, and gets the status.
UserParameter=log.errors.status[*],powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "((Get-Content 'D:\ScriptsLogs\ZabbixLogErrors.json' | ConvertFrom-Json).data | Where-Object {$_.'{#LOGFILE_NAME}' -eq \"$1\"}).status"

# UserParameter for Latest Log Error Message (per discovered log file)
# This reads from the file, converts JSON, filters, and gets the message.
UserParameter=log.errors.message[*],powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "((Get-Content 'D:\ScriptsLogs\ZabbixLogErrors.json' | ConvertFrom-Json).data | Where-Object {$_.'{#LOGFILE_NAME}' -eq \"$1\"}).latest_error_message"

# UserParameter for Latest Log Error Date (per discovered log file)
# This reads from the file, converts JSON, filters, and gets the date.
UserParameter=log.errors.date[*],powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "((Get-Content 'D:\ScriptsLogs\ZabbixLogErrors.json' | ConvertFrom-Json).data | Where-Object {$_.'{#LOGFILE_NAME}' -eq \"$1\"}).latest_error_date"

• UserParameter=file.mtime[*],powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "Get-Item '$1' | Select-Object -ExpandProperty LastWriteTimeUtc | ForEach-Object { [long][double]((Get-Date $_ -UFormat %s) + 0.5) }"

  • Purpose: This UserParameter is designed to monitor the last modification time (mtime) of any file specified as an argument ($1). It’s used to check how recently the ZabbixLogErrors.json file was updated by PowerShell script.

  • Why it's needed: Zabbix can use this item to ensure that the log error monitoring script is running regularly and successfully generating its output file. If the file's modification time is too old, it could indicate that the script has failed or stopped running.

• UserParameter=log.errors.discover,powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "Get-Content \"D:\ScriptsLogs\ZabbixLogErrors.json\""

  • Purpose: This UserParameter is specifically for Zabbix's Low-Level Discovery (LLD) feature. It reads the entire content of the ZabbixLogErrors.json file, which contains the analysis results for all monitored log files.

  • Why it's needed: Zabbix will consume this JSON output to automatically discover individual log files (using {#LOGFILE_NAME} as the discovery key) and then create specific monitoring items and triggers for each discovered log file (e.g., status, message, date). This avoids the need to manually configure each log file in Zabbix.

• UserParameter=log.errors.status[*],powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "((Get-Content 'D:\ScriptsLogs\ZabbixLogErrors.json' | ConvertFrom-Json).data | Where-Object {$_.'{#LOGFILE_NAME}' -eq \"$1\"}).status"

  • Purpose: This UserParameter retrieves the status (0 for no error, 1 for error) for a specific log file. The [*] indicates that it takes an argument, which will be the {#LOGFILE_NAME} discovered by the LLD rule.

  • Why it's needed: This is the core metric for monitoring. Zabbix will use this item to know if a particular log file has reported an error. A trigger can be set up to alert when this status becomes 1.

• UserParameter=log.errors.message[*],powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "((Get-Content 'D:\ScriptsLogs\ZabbixLogErrors.json' | ConvertFrom-Json).data | Where-Object {$_.'{#LOGFILE_NAME}' -eq \"$1\"}).latest_error_message"

  • Purpose: This UserParameter retrieves the latest_error_message for a specific log file.

  • Why it's needed: When an error is detected, this item provides the actual error message, which is crucial for troubleshooting and understanding the nature of the problem directly from the Zabbix interface or in alert notifications.

• UserParameter=log.errors.date[*],powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "((Get-Content 'D:\ScriptsLogs\ZabbixLogErrors.json' | ConvertFrom-Json).data | Where-Object {$_.'{#LOGFILE_NAME}' -eq \"$1\"}).latest_error_date"

  • Purpose: This UserParameter retrieves the latest_error_date for a specific log file.

  • Why it's needed: This provides the exact date when the latest error occurred in that specific log file, which is important for context and for quickly identifying if an error is recent or persistent.

Creating Zabbix template

We need to create Zabbix template,item and alert because it provides a structured and efficient way to apply a consistent set of monitoring configurations to one or more hosts. Instead of manually configuring each item, trigger, and discovery rule for every server that needs log error monitoring.The template would contain the log.errors.discover UserParameter as part of an LLD rule. This rule would automatically discover each log file reported in ZabbixLogErrors.json output and create individual monitoring items (like log.errors.status, log.errors.message, log.errors.date) for each discovered log file. Without a template using LLD, you'd have to manually define items for every single log file on every server.

Expand Data collection and click Templates-Create Template

After template is created,select it and click Discovery.

In a Zabbix template, discovery (specifically referred to as Low-Level Discovery or LLD) is a powerful feature that allows Zabbix to automatically detect components on a host and then create corresponding monitoring items, triggers, and graphs for them.

Populate is as below

In Discovery rules,select rule and click on Item prototypes

In Zabbix, item prototypes are blueprints or templates for creating actual monitoring items automatically through Low-Level Discovery (LLD). Instead of defining each monitoring item manually for every discovered component (like a log file, a file system, or a network interface), you define an item prototype within an LLD rule.

Create 3 prototypes as below, one for each Value in PowerShell script:error message,error date and error status, change Update interval if needed.

Now create a trigger prototype

# expression to detect error
last(/Template Log Error Monitoring/log.errors.status["{#LOGFILE_NAME}"])=1
# expression to fix the error
last(/Template Log Error Monitoring/log.errors.status["{#LOGFILE_NAME}"])=0

• last(...): This is a Zabbix trigger function. It returns the last (most recent) value collected for the specified item.

• /Template Log Error Monitoring/: This specifies the template that the item belongs to. In this case, it's a template likely named "Template Log Error Monitoring". This helps Zabbix locate the correct item even if multiple templates have similarly named items.

• log.errors.status["{#LOGFILE_NAME}"]: This refers to a specific item that Zabbix is monitoring.

  • log.errors.status is the item key, which corresponds to one of the UserParameter definitions you provided earlier: UserParameter=log.errors.status[*],powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "((Get-Content 'D:\ScriptsLogs\ZabbixLogErrors.json' | ConvertFrom-Json).data | Where-Object {$_.'{#LOGFILE_NAME}' -eq \"$1\"}).status". This item retrieves the error status (0 or 1) for a specific log file.

  • ["{#LOGFILE_NAME}"] is where the Low-Level Discovery (LLD) macro comes into play. When Zabbix's LLD rule discovers a particular log file,it automatically creates actual items using this prototype. For instance, {#LOGFILE_NAME} would be replaced with 1.txt, resulting in an actual item key like log.errors.status["1.txt"].

• =1: This is the condition for the trigger to fire. It means "if the last value collected by the log.errors.status item for this specific log file is equal to 1.",if 0,clear the error

In summary, the entire expression last(/Template Log Error Monitoring/log.errors.status["{#LOGFILE_NAME}"])=1 means:

"If the most recent status value reported for any discovered log file (as determined by the log.errors.status item in the 'Template Log Error Monitoring' template) is 1 (indicating an error), then activate this trigger.",otherwise,close it.

Now click on host where PowerShell script is running on, in templates section click Select and find the template we just created,now Template is linked to the host and Items and triggers will be created.

Creating Item for monitoring JSON file age

We need to monitor JSON file modification date so we can verify that script for monitoring script file is working and that Zabbix receives updated information.

In UserParameters section in Zabbix agent config file we already implemented JSON file monitoring

# UserParameter for JSON file last modification time
UserParameter=file.mtime[*],powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "Get-Item '$1' | Select-Object -ExpandProperty LastWriteTimeUtc | ForEach-Object { [long][double]((Get-Date $_ -UFormat %s) + 0.5) }"

On the same host we attached template to click in Items-Create New item

This item has path to JSON file and it tells powershell script which file to monitor $1 is parameter, it this case it’s Item key.

UserParameter=file.mtime[*],powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "Get-Item '$1

Creating trigger

Now click Triggers-Create new trigger

# problem expression
nodata(/server.example.com/file.mtime[D:/ScriptsLogs/ZabbixLogErrors.json],3h)=1
# recovery expression
nodata(/server.example.com/file.mtime[D:/ScriptsLogs/ZabbixLogErrors.json],3h)=1

If there are no new data in 3 hours,create alert.

Share

Leave a comment

Script published on MS Web site has some issues, so i modified it a bit.

This SQL script is designed to set up and configure a WifiMonitoring database in Microsoft SQL Server. It includes steps to:

1. Database Creation and Configuration

• Drop Existing Database: It first checks if a database named WifiMonitoring already exists. If it does, the script drops it to ensure a clean slate for the new database creation. This is useful for development or redeployment scenarios.

• Create Database: It then creates the WifiMonitoring database with specific file paths for its data (.MDF) and log (.LDF) files. It sets initial sizes and defines how the files should grow (10% increments). The database uses the SQL_Latin1_General_CP1_CI_AS collation, which defines rules for sorting and comparing character data.

• Alter Database Settings: A series of ALTER DATABASE commands follow to configure various settings for the WifiMonitoring database:

  • AUTO_CLOSE OFF: Prevents the database from automatically shutting down after the last user disconnects. This keeps it online for faster access.

  • TORN_PAGE_DETECTION ON: Helps detect incomplete I/O operations that can lead to data corruption.

  • READ_WRITE: Sets the database to be in read/write mode, allowing data modifications.

  • AUTO_SHRINK OFF: Prevents the database from automatically shrinking its files, which can cause performance issues due to fragmentation.

  • ANSI_NULL_DEFAULT OFF, ANSI_NULLS OFF, CONCAT_NULL_YIELDS_NULL OFF, QUOTED_IDENTIFIER OFF, ANSI_WARNINGS OFF: These settings control various ANSI standard behaviors related to null values, string concatenation, and identifier quoting. Setting them to OFF often indicates compatibility with older SQL Server versions or specific application requirements.

  • RECURSIVE_TRIGGERS OFF: Disables triggers from firing recursively.

  • CURSOR_CLOSE_ON_COMMIT OFF and CURSOR_DEFAULT LOCAL: Control the behavior of cursors.

  • AUTO_CREATE_STATISTICS ON and AUTO_UPDATE_STATISTICS ON: Enable automatic creation and updating of statistics, which the query optimizer uses to create efficient execution plans.

  • DB_CHAINING OFF: This conditional statement checks the SQL Server version and, if it matches certain older versions (SQL Server 2000 or 7.0), disables cross-database ownership chaining. This is a security-related setting.

2. Database Object Preparation

• Switch Context: use [WifiMonitoring] ensures that subsequent commands are executed within the newly created WifiMonitoring database.

• Drop Existing Objects: Similar to the database drop, the script checks for and drops existing stored procedures (report_event), tables (accounting_data), and user-defined data types (ipaddress) to ensure a clean deployment.

• Create Custom Data Type: EXEC sp_addtype N'ipaddress', N'nvarchar (15)', N'not null' creates a custom data type named ipaddress based on nvarchar(15). This is likely to enforce a consistent format for storing IP addresses.

3. Table Creation

• Create accounting_data Table: This is the core table for storing Wi-Fi monitoring data. It includes columns such as:

  • id (int, IDENTITY): A unique identifier for each record, auto-incrementing.

  • timestamp (datetime): The time of the event.

  • Computer_Name (nvarchar): The name of the computer involved.

  • Packet_Type (int): The type of network packet.

  • User_Name, F_Q_User_Name, SAM_Account_Name: Various user-related identifiers.

  • Called_Station_Id, Calling_Station_Id: Identifiers for the called and calling stations (e.g., MAC addresses).

  • NAS_Identifier, NAS_IP_Address: Information about the Network Access Server.

  • Client_IP_Address, Client_Friendly_Name: Information about the client.

  • Reason_Code, Connect_Info, Service_Type, Authentication_Type: Details about the connection and authentication.

  • NP_Policy_Name, Proxy_Policy_Name: Policy names.

  • Provider_Type column is most likely intended to store information about the source or type of the Wi-Fi.

• A PRIMARY KEY constraint named PK_accounting_data is defined on the id column, ensuring data integrity and fast lookups.

4. Stored Procedure Creation

• Create dbo.report_event Stored Procedure: This procedure is designed to insert data into the accounting_data table.

  • It takes an XML document (@doc NVARCHAR(MAX)) as input, which is expected to contain event details.

  • sp_xml_preparedocument: Parses the input XML into an in-memory representation, allowing it to be queried.

  • INSERT INTO ... SELECT ... FROM OPENXML: This is the core of the data insertion. It uses OPENXML to extract data from the prepared XML document and insert it into the corresponding columns of the accounting_data table. The WITH clause maps XML elements/attributes to table columns.

  • sp_xml_removedocument: Cleans up the in-memory XML document.

  • Error Handling and Auditing: The procedure includes a TRY...CATCH block for robust error handling:

    • If the data insertion is successful, it logs a success entry into a dbo.report_event_audit table (which is not defined in this script but is expected to exist). This audit includes information like host name, login name, and input length.

    • If an error occurs, it captures detailed error information (number, severity, state, line, procedure, message), logs it to the dbo.report_event_audit table (including the full XML input), and then re-raises the error to the caller, allowing the calling application to know about the failure.

IF EXISTS (SELECT name FROM master.dbo.sysdatabases WHERE name = N'WifiMonitoring')
    DROP DATABASE [WifiMonitoring]
GO

CREATE DATABASE [WifiMonitoring]  ON (NAME = N'WifiMonitoring_Data', 
                               FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL\data\WifiMonitoring_Data.MDF' , 
                               SIZE = 1, FILEGROWTH = 10%) 
                       LOG ON (NAME = N'WifiMonitoring_Log', 
                               FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL\data\WifiMonitoring_Log.LDF' , 
                               SIZE = 1, FILEGROWTH = 10%)
 COLLATE SQL_Latin1_General_CP1_CI_AS
GO

ALTER DATABASE [WifiMonitoring]
SET AUTO_CLOSE OFF
GO

ALTER DATABASE [WifiMonitoring]
SET TORN_PAGE_DETECTION ON
GO

ALTER DATABASE [WifiMonitoring]
SET READ_WRITE
GO

ALTER DATABASE [WifiMonitoring]
SET AUTO_SHRINK OFF
GO

ALTER DATABASE [WifiMonitoring]
SET ANSI_NULL_DEFAULT OFF
GO

ALTER DATABASE [WifiMonitoring]
SET RECURSIVE_TRIGGERS OFF
GO

ALTER DATABASE [WifiMonitoring]
SET ANSI_NULLS OFF
GO

ALTER DATABASE [WifiMonitoring]
SET CONCAT_NULL_YIELDS_NULL OFF
GO

ALTER DATABASE [WifiMonitoring]
SET CURSOR_CLOSE_ON_COMMIT OFF
GO

ALTER DATABASE [WifiMonitoring]
SET CURSOR_DEFAULT LOCAL
GO

ALTER DATABASE [WifiMonitoring]
SET QUOTED_IDENTIFIER OFF
GO

ALTER DATABASE [WifiMonitoring]
SET ANSI_WARNINGS OFF
GO

ALTER DATABASE [WifiMonitoring]
SET AUTO_CREATE_STATISTICS ON
GO

ALTER DATABASE [WifiMonitoring]
SET AUTO_UPDATE_STATISTICS ON
GO

if( ( (@@microsoftversion / power(2, 24) = 8) and (@@microsoftversion & 0xffff >= 724) ) or 
    ( (@@microsoftversion / power(2, 24) = 7) and (@@microsoftversion & 0xffff >= 1082) ) )
    ALTER DATABASE [WifiMonitoring] SET DB_CHAINING OFF
GO

use [WifiMonitoring]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[report_event]') and 
                                              OBJECTPROPERTY(id, N'IsProcedure') = 1)
drop procedure [dbo].[report_event]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[accounting_data]') and 
                                              OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[accounting_data]
GO

if exists (select * from dbo.systypes where name = N'ipaddress')
exec sp_droptype N'ipaddress'
GO

EXEC sp_addtype N'ipaddress', N'nvarchar (15)', N'not null'
GO

IF OBJECT_ID('dbo.accounting_data', 'U') IS NOT NULL
BEGIN
    DROP TABLE [dbo].[accounting_data];
    PRINT 'Dropped existing dbo.accounting_data table.';
END
GO

PRINT 'Creating new dbo.accounting_data table with selected columns...';
CREATE TABLE [dbo].[accounting_data](
    [id] [int] IDENTITY(1,1) NOT NULL,
    [Provider_Type] [NVARCHAR(50)] NULL,
    [timestamp] [datetime] NOT NULL,
    [Computer_Name] [nvarchar](255) NOT NULL,
    [Packet_Type] [int] NOT NULL,
    [User_Name] [nvarchar](255) NULL,
    [F_Q_User_Name] [nvarchar](255) NULL,
    [Called_Station_Id] [nvarchar](255) NULL,
    [Calling_Station_Id] [nvarchar](255) NULL,
    [NAS_Identifier] [nvarchar](255) NULL,
    [NAS_IP_Address] [ipaddress] NULL,
    [Client_IP_Address] [ipaddress] NULL,
    [Client_Friendly_Name] [nvarchar](255) NULL,
    [Reason_Code] [int] NULL, 
    [Connect_Info] [nvarchar](255) NULL,
    [Service_Type] [int] NULL,
    [Authentication_Type] [int] NULL,
    [NP_Policy_Name] [nvarchar](255) NULL,
    [Proxy_Policy_Name] [nvarchar](255) NULL,
    [SAM_Account_Name] [nvarchar](255) NULL,
CONSTRAINT [PK_accounting_data] PRIMARY KEY CLUSTERED
(
    [id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, OPTIMIZE_FOR_SEQUENTIAL_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY];
GO
PRINT 'New dbo.accounting_data table created successfully.';
GO


Create PROCEDURE dbo.report_event
    @doc NVARCHAR(MAX)
AS

SET NOCOUNT ON;

DECLARE @idoc INT;
DECLARE @record_timestamp DATETIME = GETUTCDATE();

BEGIN TRY
    -- Prepare XML document
    -- This creates an in-memory representation of the XML document.
    EXEC sp_xml_preparedocument @idoc OUTPUT, @doc;

    INSERT INTO dbo.accounting_data (
        [timestamp],
        Computer_Name,
        Provider_Type,
        Packet_Type,
        [User_Name],
        F_Q_User_Name,
        Called_Station_Id,
        Calling_Station_Id,
        NAS_Identifier,
        NAS_IP_Address,
        Client_IP_Address,
        Client_Friendly_Name,
        Reason_Code,
        Connect_Info,
        Service_Type,
        Authentication_Type,
        NP_Policy_Name,
        Proxy_Policy_Name,
        SAM_Account_Name
    )
    SELECT
        @record_timestamp,
        Computer_Name,
        Packet_Type,
        [User_Name],
        Fully_Qualifed_User_Name,
        Called_Station_Id,
        Calling_Station_Id,
        NAS_Identifier,
        NAS_IP_Address,
        Client_IP_Address,
        Client_Friendly_Name,
        Reason_Code,
        Connect_Info,
        Service_Type,
        Authentication_Type,
        NP_Policy_Name,
        Proxy_Policy_Name,
        SAM_Account_Name
    FROM OPENXML(@idoc, '/Event')
    WITH (
        Computer_Name NVARCHAR(255) './Computer-Name',
        Packet_Type INT './Packet-Type', 
        [User_Name] NVARCHAR(255) './User-Name',
        Fully_Qualifed_User_Name NVARCHAR(255) './Fully-Qualifed-User-Name',
        Called_Station_Id NVARCHAR(255) './Called-Station-Id',
        Calling_Station_Id NVARCHAR(255) './Calling-Station-Id',
        NAS_Identifier NVARCHAR(255) './NAS-Identifier',
        NAS_IP_Address dbo.ipaddress './NAS-IP-Address',
        Client_IP_Address dbo.ipaddress './Client-IP-Address',
        Client_Friendly_Name NVARCHAR(255) './Client-Friendly-Name',
        Reason_Code INT './Reason-Code', 
        Provider_Type NVARCHAR(50) './Provider_Type',
        Connect_Info NVARCHAR(255) './Connect-Info',
        Service_Type INT './Service-Type', 
        Authentication_Type INT './Authentication-Type', 
        NP_Policy_Name NVARCHAR(255) './NP-Policy-Name',
        Proxy_Policy_Name NVARCHAR(255) './Proxy-Policy-Name',
        SAM_Account_Name NVARCHAR(255) './SAM-Account-Name'
    );

    -- Clean up XML document from memory
    EXEC sp_xml_removedocument @idoc;

    -- Log successful execution
    INSERT INTO dbo.report_event_audit (success, host_name, login_name, input_length)
    VALUES (1, HOST_NAME(), SYSTEM_USER, DATALENGTH(@doc));

END TRY
BEGIN CATCH
    -- Clean up XML document in case of error
    IF @idoc IS NOT NULL
    BEGIN
        EXEC sp_xml_removedocument @idoc;
    END

    -- Capture richer error details
    DECLARE @error_number INT = ERROR_NUMBER();
    DECLARE @error_severity INT = ERROR_SEVERITY();
    DECLARE @error_state INT = ERROR_STATE();
    DECLARE @error_line INT = ERROR_LINE();
    DECLARE @error_procedure NVARCHAR(200) = ERROR_PROCEDURE();
    DECLARE @error_message NVARCHAR(MAX) = ERROR_MESSAGE();

    -- Create a more detailed error message for logging and raising
    DECLARE @detailed_error_msg NVARCHAR(MAX) =
        N'Error ' + CAST(@error_number AS NVARCHAR(20)) +
        N', Level ' + CAST(@error_severity AS NVARCHAR(20)) +
        N', State ' + CAST(@error_state AS NVARCHAR(20)) +
        N', Procedure ' + @error_procedure +
        N', Line ' + CAST(@error_line AS NVARCHAR(20)) +
        N': ' + @error_message;

    -- Log the new, detailed error information, including the full XML input
    INSERT INTO dbo.report_event_audit (
        success,
        error_message,
        error_line,
        host_name,
        login_name,
        input_length,
        xml_input
    )
    VALUES (
        0,
        @detailed_error_msg,
        @error_line,
        HOST_NAME(),
        SYSTEM_USER,
        DATALENGTH(@doc),
        @doc
    );

    -- Re-raise the detailed error to the caller, so the calling application knows the operation failed.
    RAISERROR(@detailed_error_msg, 16, 1);

END CATCH;
GO
PRINT 'dbo.report_event stored procedure altered successfully.';
GO

Subscribe now

Share

Leave a comment