Tag: AWS

Container Security: AI-Powered Golden Base Image Auto-Patching

In today’s fast-paced cloud-native world, containerization has become the backbone of modern applications. However, maintaining the security of container images, especially the underlying “golden base images” is a persistent challenge. Manually tracking and patching vulnerabilities is a time-consuming, error-prone process that leaves critical exposure windows open.

This post is about how this challenge was tackled head-on with an innovative, AI-powered solution that automates the detection and patching of critical and high vulnerabilities in our Amazon ECR container images. This not only drastically reduces Mean Time To Patch (MTTP) but also frees up time to focus on innovation rather than reactive security tasks.

The Challenge: Manual Vulnerability Management

Before this solution, the process for handling base image vulnerabilities involved:

  • Regular scans from tools like AWS Inspector.
  • Manual review of findings by security and operations teams.
  • Manually creating Dockerfile patches.
  • Triggering new image builds and testing cycles.

This sequential, human-dependent workflow meant that even with the best intentions, the time from vulnerability detection to deployment of a patched image could span days, sometimes even weeks, especially for non-critical but high-priority vulnerabilities. This was simply not sustainable for our rapidly growing infrastructure.

Solution: AI-Powered, Fully Automated Patching

The solution envisioned a system that could not only detect vulnerabilities but also intelligently propose and execute the patches autonomously. This solution, managed entirely through Infrastructure as Code (IaC) in a dedicated infra-terraform-image-builder repository, integrates several key AWS services to create a seamless, end-to-end automation pipeline.

Here’s how this works:

The Workflow at a Glance

Key Components of Auto-Patching Pipeline

AWS Inspector2: The Sentinel – The first line of defense. AWS Inspector2 continuously scans AWS ECR repositories, detecting critical and high vulnerabilities in our container images. When a new finding emerges or an existing one escalates, Inspector2 alerts the system.

Amazon DynamoDB: The Central Brain & Trigger – Inspector2 findings are streamed into a dedicated DynamoDB table. This acts as the centralized source of truth for all vulnerabilities. Crucially, DynamoDB’s Streams feature directly feeds into the AWS Lambda function, acting as the primary trigger for automation whenever a new or updated critical/high severity finding is recorded.

AWS Lambda: The Orchestrator – This is the heart of the automation. A Python-based AWS Lambda function is invoked by DynamoDB Streams.

  • It parses the finding, identifying the vulnerable package and image.
  • It determines the base image that needs patching.
  • It orchestrates the entire patching process, from AI command generation to signaling the image build.

Amazon ECR: The Image Repository – The central repository for all container images. Lambda interacts with ECR to fetch image metadata (tags, manifest) necessary for the patching process.

AWS Bedrock (Generative AI): The Intelligent Patch Creator – This is where the magic happens! The Lambda function sends the vulnerability details (CVE ID, package name, affected version, base OS) to an AWS Bedrock model. Bedrock, leveraging its generative AI capabilities, intelligently analyzes this information and generates the precise shell commands (e.g., apt-get update && apt-get install -y <package-name>=<fixed-version>) required to patch the vulnerability within the Dockerfile context. This eliminates manual script creation and dramatically speeds up the patching process.

Amazon S3: The Patch Script Store – The dynamically generated patch commands from Bedrock are stored as temporary patch scripts in an S3 bucket. This ensures an auditable trail and provides a robust, accessible location for the next step. The Lambda function updates these patch scripts in S3.

AWS Systems Manager (SSM) Parameter Store: The Signal Tower – To gracefully signal the image build process, SSM Parameter Store is used. The Lambda function updates a specific SSM parameter for the relevant base image. This parameter acts as a signal to the AWS Image Builder pipelines, indicating that a new patch script has been generated and a rebuild is required. The Lambda function updates this SSM parameter, which is then used by the Image Builder pipeline.

AWS Image Builder: The Automated Forge – AWS Image Builder pipelines are configured to monitor specific SSM parameters. Upon detecting an update to the relevant parameter, it springs into action. It retrieves the base image, injects the generated patch script from S3 into the Dockerfile/build process, and then builds a new, patched container image. This newly built image is then pushed back to ECR with updated tags.

FINAL THOUGHTS

This AI-powered golden base image auto-patching solution marks a significant leap forward in container security posture. Embracing generative AI with AWS Bedrock and integrating it with existing AWS ecosystem, not only drastically reduced the exposure window to critical and high vulnerabilities but also empowered teams by taking away a significant operational burden. This approach demonstrates the power of combining modern cloud services with cutting-edge AI to build resilient, secure, and future-proof infrastructure.

Manipulating JSON with jq

Recently I was working on a project using AWS CLI and happened to come across some cool jq techniques of manipulating JSON. I will describe what my use-case was, but similar techniques can be applied whenever JSON is involved.

jq is a lightweight JSON processor written in C. You can find more information about the tool and how to install it in their official documentation. It’s quite powerful and is capable of doing quite a lot – i.e parsing, manipulating and processing json files. Now let’s see how it was kinda easy to get my work done using jq.

The Use Case

We use AWS Image Builder to build EC2 AMI’s and this requirement was to explicitly update the ami_name field of an existing Image Builder Distribution Configuration whenever a pipeline is triggered. There are some options in the distribution configuration to specify what we want to name the output AMI, but we wanted it to be very custom (with the base ami version) and dynamic. I will not be talking about how Image Builder works as its a separate topic in itself, however you can read about it here.

The Problem

So to achieve this (using AWS CLI), we just needed to run 2 AWS CLI commands:

  1. Get the existing distribution configuration:
aws imagebuilder get-distribution-configuration --output json --distribution-configuration-arn $distribution_config_arn

The above will return a json response of the existing distribution configuration like below: 

{
    "requestId": "42b6bcf5-9505-4c42-ad38-7efd8177f2ac",
    "distributionConfiguration": {
        "arn": "arn:aws:imagebuilder:us-east-1:123456789012:distribution-configuration/amazon-eks-node-latest-pipeline-distribution-config",
        "name": "amazon-eks-node-latest-pipeline-distribution-config",
        "description": "amazon-eks-node-latest image builder pipeline",
        "distributions": [
            {
                "region": "us-east-1",
                "amiDistributionConfiguration": {
                    "name": "amazon-eks-node-latest-golden-ami-{{ imagebuilder:buildDate }}",
                    "description": "amazon-eks-node-latest image builder pipeline",
                    "amiTags": {
                        "family": "amazon-eks-node-latest",
                        "Name": "amazon-eks-node-latest-golden-ami-{{ imagebuilder:buildDate }}"
                    },
                    "launchPermission": {}
                }
            }
        ],
        "dateCreated": "2024-03-06T14:59:37.286Z",
        "tags": {
            "owner": "platform",
            "project": "image-builder",
            "env": "prod",
            "family": "amazon-eks-node-latest",
            "managedby": "terraform"
        }
    }
}

2. Update only the amiDistributionConfiguration.name field of this distribution configuration.

Now it is not possible to just update one field of the distribution configuration. If you see distributions is a list in the response and to update it we need to get the full response, change the fields we want and then pass this json as an input to the update-distribution-configuration CLI command.

But wait, if you see the input json syntax that update-distribution-configuration expects (shown below), it does not exactly match the response got from the get-distribution-configuration command.

{
    "distributionConfigurationArn": "arn:aws:imagebuilder:us-east-1:123456789012:distribution-configuration/amazon-eks-node-latest-pipeline-distribution-config",
    "description": "amazon-eks-node-latest image builder pipeline",
    "distributions": [
        {
            "region": "us-east-1",
            "amiDistributionConfiguration": {
                "name": "Name {{imagebuilder:buildDate}}",
                "description": "An example image name with parameter references"
            }
        },
        {
            "region": "eu-east-2",
            "amiDistributionConfiguration": {
                "name": "My {{imagebuilder:buildVersion}} image {{imagebuilder:buildDate}}"
            }
        }
    ]
}

This problem can be true for a variety of other scenarios (AWS or non AWS). Comes jq to the rescue! With jq, it’s just a one line command to process the json and transform it the way we want.

The Solution

The solution is, to transform the response from get-distribution-configuration to the required syntax using jq and then passing it to the update-distribution-configuration command, which can be effectively done with the below commands:

# 1. Get the existing distribution configuration
distribution_config=`aws imagebuilder get-distribution-configuration --output json --distribution-configuration-arn $arn`

# 2. Update the JSON syntax of the response to match the target commands requirements
updated_distribution_config=$(echo "$distribution_config" | jq '{ distributionConfigurationArn: .distributionConfiguration.arn, description: .distributionConfiguration.description, distributions: [.distributionConfiguration.distributions[] | .amiDistributionConfiguration.name = UPDATED_NAME]}')

aws imagebuilder update-distribution-configuration --cli-input-json $updated_distribution_config

The transformation happens in the 2nd command. I have split the command into new-lines to make it clearer. Let’s see how it works:

echo "$distribution_config" | 
jq '{ 
  distributionConfigurationArn: .distributionConfiguration.arn, 
  description: .distributionConfiguration.description, 
  distributions: [.distributionConfiguration.distributions[] |.amiDistributionConfiguration.name = UPDATED_NAME]
}'

The above filter tells jq to create a JSON object containing:

  • A distributionConfigurationArn attribute containing the value of .distributionConfiguration.arn
  • A description attribute containing the value of .distributionConfiguration.description
  • A distributions attribute containing a list of distributions from .distributionConfiguration.distributions[]
  • Also we update all .amiDistributionConfiguration.name fields from this list of distributions to our desired value (UPDATED_NAME here) for all occurrences.

As you see, the new json structure is created and existing json is parsed to get the required fields, manipulated to replace the fields items as per requirement in this new structure simultaneously. This ends up creating a json like below, which is what the update-distribution-configuration command expects:

{
  "distributionConfigurationArn": "arn:aws:imagebuilder:us-east-1:123456789012:distribution-configuration/amazon-eks-node-latest-pipeline-distribution-config",
  "description": "amazon-eks-node-latest image builder pipeline",
  "distributions": [
    {
      "region": "us-east-1",
      "amiDistributionConfiguration": {
        "name": "UPDATED_NAME" }}",
        "description": "amazon-eks-node-latest image builder pipeline",
        "amiTags": {
          "family": "amazon-eks-node-latest",
          "Name": "amazon-eks-node-latest-golden-ami-{{ imagebuilder:buildDate }}"
        },
        "launchPermission": {}
      }
    }
  ]
}

Though this is just a simple example, jq as you see is quite powerful. 🙂

Design a site like this with WordPress.com
Get started