DEV Community: Bob Tordella

Overcoming AWS Security Alert Fatigue

Bob Tordella — Fri, 25 Jul 2025 17:49:41 +0000

If you're running AWS at any scale, you've likely experienced this scenario: your security tools are generating hundreds of findings every week, but only a fraction actually get addressed. The rest accumulate until the next audit or compliance deadline creates urgency.

This scenario is playing out across organizations of every size. Teams become overwhelmed by the sheer volume of security findings and struggle to establish systematic remediation processes.

Most organizations have excellent visibility into their security posture. AWS Security Hub, Config Rules, GuardDuty, Inspector, and third-party CSPM and CNAPP tools provide comprehensive coverage. Yet security findings continue to accumulate faster than teams can address them.

Phases of Security Alert Fatigue

You can't govern what you can't see. You need the ability to know what you have and what you can do to improve your posture. Organizations invest heavily in monitoring tools. However, comprehensive visibility often creates information overload that presents its own challenges.

Phase 1: Tool Implementation
Organizations invest in CSPM and CNAPP tools, excited about gaining visibility into their entire AWS environment. Tools are configured with default rules and industry recommendations.

Phase 2: Information Overload
Hundreds or thousands of findings start flowing in daily. Security team is overwhelmed. Application teams receive scattered requests for fixes, often without context about priority or business impact.

Phase 3: Confusion and Inaction
Teams become desensitized to security alerts when everything appears to be high priority. Application teams struggle to understand which findings require immediate attention versus those that can be addressed during planned maintenance.

Phase 4: Alert Accumulation
Findings stack up faster than they can be addressed. Teams develop workarounds or learn to ignore certain types of alerts. Major security remediation programs are eventually created to tackle the backlog.

Phase 5: Whack-a-Mole
Significant effort is invested to fix accumulated findings, but new problems arise as quickly as old ones are resolved. Without addressing root causes, the cycle repeats with each new application or environment.

When Alerts Become Background Noise

Most organizations approach security findings like a ticketing system - identify, assign, and continuous follow-ups for resolution. Treating security findings as isolated incidents creates fundamental problems that cause inaction in resolving issues:

Unclear Ownership: Organizations often lack clear processes for determining who should address specific types of findings. Security teams identify issues but may lack the application context to fix them effectively. Application teams understand their systems but may not fully grasp security implications. Additionally, team members change roles or leave the organization, breaking institutional knowledge about resource ownership and context.

Poor Communication: Many security alerts start shipping out to application teams without any education about why it matters. Teams do not understand how it relates to organizational standards, or what the business impact might be. Teams receive notifications about technical violations without understanding the underlying security or compliance rationale. This gap in communication creates confusion about which findings deserve immediate attention.

Missing Context: Security tools often apply generic rules without understanding workload context. A finding about an open port might be critical for a sensitive database but perfectly acceptable for a web-facing load balancer. Without workload context, teams either over-react to acceptable configurations or under-react to genuine risks.

Conflicting Prioritization: Teams struggle to prioritize security findings within their existing workload. Without clear service level agreements, risk assessments, or business impact guidance, application teams may delay security remediation in favor of feature development. This misalignment between security urgency and business priorities often results in prolonged inaction.

Constant Recurrence: Teams focus on fixing individual findings rather than addressing the underlying patterns that created them. This approach treats symptoms rather than causes, leading to repeated occurrences of similar issues across different resources and environments. Without investment in preventive controls or automated remediation, organizations find themselves in an endless cycle of manual fixes.

From Sending Alerts to Deploying Controls

Instilling Cloud Governance practices into your security program transforms how you handle findings. This means shifting from reactive incident response to proactive standards and controls that are deployed systematically. The 5 Cloud Governance Practices provide the framework for sustainable security remediation:

Standards: Define what secure AWS configurations look like for your organization. Instead of generic security rules, create specific standards that account for your workload types, risk tolerance, and operational requirements. Make these standards clear, practical, and co-created with the teams who will implement them.

Controls & Automation: Enforce standards through preventive, detective, and corrective controls. Use AWS Config rules, Service Control Policies, and Infrastructure as Code templates to make secure configurations the default path. Automate remediation for low-risk findings and provide clear escalation paths for complex issues.

Adoption: Help teams embrace security standards through reusable tools, embedded guidance, and responsive support. Provide secure templates, clear documentation, and accessible channels for questions. Make following security standards easier than ignoring them.

Rollout: Deploy security standards systematically through the Draft → Preview → Check → Enforce lifecycle. Start with Draft where your cloud team tests the impact internally. Move to Preview to show application teams what would be flagged without affecting scores. Then Check, where everything is visible and counted but not yet enforced. Finally Enforce, where controls take action automatically. This progression reduces surprises and builds trust while giving teams time to adapt.

Measurement & Improvement: Track the effectiveness of your Cloud Security Governance through both technical metrics (finding recurrence, remediation time) and organizational metrics (team adoption, exception patterns). Use this data to continuously improve your standards and rollout approaches.

Improving Security Posture with Cloud Governance

Cloud Governance transforms how you secure AWS at scale. Rather than chasing alerts, you create systems that prevent security issues from arising in the first place. Teams get secure defaults, clear standards, and automated guardrails that make compliance the easy path. This shift from reactive fixes to proactive prevention enables sustainable security improvement across your entire AWS environment.

[Boost]

Bob Tordella — Mon, 24 Mar 2025 14:25:47 +0000

Check if you are breaking your admin rules in your GitHub repos

Matty Stratton ・ Mar 24

Threat Detection for AWS CloudTrail Logs

Bob Tordella — Thu, 20 Mar 2025 13:04:29 +0000

In my previous post, I showed how to use Tailpipe to query your AWS CloudTrail logs locally with SQL. Now let's take it to the next level: visualizing those logs with powerful dashboards for advanced threat detection and investigation.

Threat Detection Benchmarks

When investigating suspicious AWS activity, having both powerful queries and interactive visualizations can make all the difference. Whether you're responding to a security incident or proactively hunting for threats, you need to quickly separate normal activity from potential risks.

Enter Powerpipe AWS CloudTrail Detections mod: pre-built dashboards and detections that work with your locally collected CloudTrail logs from the Tailpipe AWS plugin to provide security insights based on industry frameworks like MITRE ATT&CK. And the best part? It all runs locally without sending your sensitive log data anywhere.

Getting Started

If you haven't set up Tailpipe to collect your CloudTrail logs, you can learn how to do that from our prior post.

After you have Tailpipe set-up, add Powerpipe to visualize this data:

# Install Powerpipe
brew install turbot/tap/powerpipe
# Or using the install script
sudo /bin/sh -c "$(curl -fsSL https://powerpipe.io/install/powerpipe.sh)"

# Create a directory for the dashboards and install the mod
mkdir dashboards
cd dashboards
powerpipe mod install github.com/turbot/tailpipe-mod-aws-cloudtrail-log-detections

And now start the dashboard server:

powerpipe server

Visit http://localhost:9033 in your browser, and you're ready to explore your CloudTrail logs dashboards.

Viewing CloudTrail Activity Through a Security Lens

The mod provides multiple benchmarks with 100 pre-built detections to help visualize suspicious activity. Let's walk through the key views and capabilities this gives you:

Detections by AWS Service

The CloudTrail Log Detections Benchmark organizes findings by AWS service, making it easy to focus on specific areas like IAM, S3, or EC2.

Each detection is pre-built to identify potentially suspicious activity, such as:

IAM root user console logins
S3 bucket policy changes
EC2 security group modifications
CloudWatch log deletion events

Click on any detection to see the specific CloudTrail events that triggered it, complete with timestamps, usernames, and other contextual details.

Hunt for Bad Actors

Security analysis is all about finding needles in haystacks. Powerpipe has built-in filtering capabilities to help you narrow down and focus on the log entries found by detections; hover over any cell to reveal a row of four icons:

Copy value: Quickly grab data for use in other tools
Filter by this value: Focus only on rows with this specific value
Exclude value from results: Remove these rows from your current view
View row: Examine all details about a specific event

If you want to keep only rows with the /aws/lambda/Level6 resource, select the Filter icon. Conversely if you want to toss that set of rows to focus on everything else, select the Exclude icon. Using these tools you can refine the set of detected rows to ignore those that are benign and focus on the ones that might be malicious.

When you find an interesting row, use the View icon to explore it in detail:

View Through the MITRE ATT&CK Lens

To view results through a different lens, Powerpipe offers a MITRE ATT&CK benchmark that maps the same CloudTrail events to the MITRE ATT&CK framework. This helps security teams understand the potential security implications of each activity and how it might fit into a larger attack chain.

For example, an IAM root user login might appear under:

TA0001: Initial Access - is a high-level tactic, representing an adversary's goal to gain an initial foothold in a system.
T1078: Valid Accounts - is a technique related to use of valid credentials (stolen, guessed, or default) to access systems.
T1078.001: Valid Accounts: Default Accounts - narrows that down to focus on root accounts or accounts with default passwords. That's the context for the IAM Root User Console Login detection.

It's not the only context though, as the same detection also appears under tactic TA0004: Privilege Escalation providing different perspectives to evaluate how an action fits into the overall attack chain.

And the same filtering features are available to investigate further. For example, if the actor performing the login is known and trusted, exclude all console logins from that actor with a single click in order to focus on others that may warrant scrutiny.

Conclusion

With Tailpipe handling log collection and queries, and Powerpipe providing interactive visualization, you have a powerful, local, open-source solution for security analysis. You can analyze logs offline, quickly filter massive datasets, and investigate potential security incidents all while maintaining complete control over your sensitive data.

Query AWS CloudTrail Logs Locally with SQL

Bob Tordella — Fri, 31 Jan 2025 21:38:25 +0000

When investigating AWS activity or analyzing usage patterns, being able to quickly query your CloudTrail logs can make all the difference. Whether you're responding to a security event, optimizing costs, or just trying to understand how your team uses AWS, you need fast, iterative access to your log data.

Enter Tailpipe: a lightweight, open-source tool that lets you analyze logs right from your terminal using SQL. It runs entirely on your local machine, using DuckDB to process millions of records in seconds. This means you can pull down your logs locally and start querying immediately - perfect for rapid investigations, offline analysis, or when you need to quickly test different queries.

Just pipe your CloudTrail logs to your local machine, and you're ready to start querying with familiar SQL. No infrastructure to set up, no services to configure - just straightforward log analysis when you need it.

Getting Started

First, let's install Tailpipe. You have two options:

# Using Homebrew
brew install turbot/tap/tailpipe

# Or using the install script
sudo /bin/sh -c "$(curl -fsSL https://tailpipe.io/install/tailpipe.sh)"

Next, install the AWS plugin:

tailpipe plugin install aws

Configuration

Create a simple configuration file to connect Tailpipe to your AWS CloudTrail logs. Create a tailpipe.hcl file:

connection "aws" "prod" {
  profile = "log-admin"  # Your AWS profile name
}

partition "aws_cloudtrail_log" "prod" {
  source "aws_s3_bucket" {
    connection = connection.aws.prod
    bucket     = "aws-cloudtrail-logs-12345"  # Your CloudTrail bucket
  }
}

Collecting Logs

Now you're ready to collect some logs. Start with the last 7 days:

tailpipe collect aws_cloudtrail_log

Want more history? Specify a start date:

tailpipe collect aws_cloudtrail_log --from 2024-01-01

Understanding Your AWS API Usage

Let's start with a fundamental question: which AWS services and APIs are most frequently used in your environment? This query helps you understand your AWS usage patterns:

select
  event_source,
  event_name,
  count(*) as event_count
from
  aws_cloudtrail_log
group by
  event_source,
  event_name
order by
  event_count desc
limit 10;

Running this query on a typical AWS environment might give you output like this:

+-------------------+---------------------------+-------------+
| event_source      | event_name                | event_count |
+-------------------+---------------------------+-------------+
| ec2.amazonaws.com | RunInstances              | 1225268     |
| ec2.amazonaws.com | DescribeSnapshots         | 101158      |
| sts.amazonaws.com | AssumeRole                | 78380       |
| s3.amazonaws.com  | GetBucketAcl              | 19095       |
| ec2.amazonaws.com | DescribeInstances         | 18366       |
| sts.amazonaws.com | GetCallerIdentity         | 16512       |
| iam.amazonaws.com | GetPolicyVersion          | 14737       |
| s3.amazonaws.com  | ListBuckets               | 13206       |
| ec2.amazonaws.com | DescribeSpotPriceHistory  | 10714       |
| ec2.amazonaws.com | DescribeSnapshotAttribute | 9107        |
+-------------------+---------------------------+-------------+

Looking at this output, you'll notice that many of these operations are read-only actions like Describe*, Get*, and List*. While this gives us a good overview of API usage, we might be more interested in write operations that actually change our AWS environment.

Let's modify our query to focus on these change-making operations:

select
  event_source,
  event_name,
  count(*) as event_count
from
  aws_cloudtrail_log
where
  not read_only
group by
  event_source,
  event_name
order by
  event_count desc
limit 10;

Now we're seeing a very different picture!

+----------------------+---------------------------+-------------+
| event_source         | event_name                | event_count |
+----------------------+---------------------------+-------------+
| ec2.amazonaws.com    | RunInstances              | 1225268     |
| sts.amazonaws.com    | AssumeRole                | 78380       |
| ec2.amazonaws.com    | CreateTags                | 8456        |
| ec2.amazonaws.com    | CreateVolume              | 5231        |
| s3.amazonaws.com     | PutObject                 | 4521        |
| iam.amazonaws.com    | CreateRole                | 3242        |
| ec2.amazonaws.com    | ModifyInstanceAttribute   | 2890        |
| rds.amazonaws.com    | CreateDBInstance          | 2456        |
| lambda.amazonaws.com | CreateFunction            | 2123        |
| eks.amazonaws.com    | CreateCluster             | 1890        |
+----------------------+---------------------------+-------------+

This filtered view highlights the actual resource creation and modifications in your AWS environment, quickly showing you where new infrastructure is being deployed and potential areas for optimization.

You can modify these queries further to focus on specific time periods, services, or add additional filters based on your needs.

Learn More

There's much more you can do with Tailpipe. The Tailpipe Hub has 110+ ready-to-use CloudTrail queries along with other log sources you can collect & analyze. As part of the Turbot open source ecosystem, you can also use the Powerpipe AWS CloudTrail Logs Detections mod to visualize pre-built dashboards and detections of your AWS activity.

How to run an AWS CIS v3.0 assessment in CloudShell

Bob Tordella — Thu, 08 Feb 2024 19:42:32 +0000

AWS CloudShell makes it easy to spin up a terminal right in your AWS account. Since CloudShell is just like any other terminal, you have the ability to bootstrap other tools without the need to spin up an instance.

In a prior post I showed how to install Steampipe in AWS CloudShell to instantly query over 460+ resource types from your AWS APIs using SQL, and another post on how to use the Steampipe AWS Compliance mod to assess over 25+ security benchmarks across your AWS accounts.

In this post we are going to show how to run the latest AWS CIS benchmark v3.0 in AWS CloudShell.

How to run an AWS CIS v3.0 assessment

Here's how to get started:
If you've already completed steps 1 - 3, skip to step 4:

1. Install Steampipe

sudo /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/turbot/steampipe/main/install.sh)"

2. Install the AWS plugin

steampipe plugin install aws

3. Install the AWS Compliance Mod

git clone https://github.com/turbot/steampipe-mod-aws-compliance
cd steampipe-mod-aws-compliance

4. Run the AWS CIS v3.0 benchmark:

steampipe check aws_compliance.benchmark.cis_v300

There are over 60 controls in that benchmark, so the command produces many screenfuls of output, here's the last one:

Export and Review the Findings

The summary is helpful, but you may want to digest the full report in varying formats. You can export to CSV, Markdown, HTML. Example of an HTML format:

steampipe check aws_compliance.benchmark.cis_v300 --export=output.html

Using Files -> Download File in AWS CloudShell's Actions menu, you can download your output file steampipe-mod-aws-compliance/output.html and work with it locally.

Here's what the HTML report looks like:

Final Thoughts

I really enjoy using AWS CloudShell + Steampipe for these type of quick win use cases within an AWS account. It's remarkably easy to install your CLI tools like Steampipe, with no configuration required and instant gratification!

CloudShell is just one place to run Steampipe in AWS, beyond ECS containers, EC2 instances and AWS Workspaces, you can also run Steampipe in Cloud9 and CodeBuild.

Try Steampipe for other AWS use cases, and let me know how you go.

Keeping ServiceNow Updated with Automated AWS Discovery

Bob Tordella — Wed, 24 Jan 2024 05:24:14 +0000

As AWS builders, we know how fast cloud environments evolve. Resources get added, changed, and removed continuously. If your inventory reporting or cloud discovery is not capturing changes in real-time, before you know it, your CMDB is full of blindspots and outdated data.

Relying on manual cloud discovery and scheduled updates leads to inaccuracy. And when using native tools they often only cover a handful of core AWS services while leaving 100+ for custom development work.

Recently our enterprise customers were expressing these struggles with keeping accurate records of cloud resources in their ServiceNow CMDB. So as part of our last Launch Week we built an integration in Turbot Guardrails in response to help customers capture real-time resource changes from multi-cloud to ServiceNow.

Automated AWS Discovery with Guardrails

This AWS & ServiceNow integration via Turbot Guardrails provides a real-time automation to discover resources across 100+ AWS services. As your infrastructure changes, Guardrails detects it and handles updating integrated systems like ServiceNow.

It augments native discovery capabilities by:

Adding more comprehensive AWS resource coverage
Handling deletions and archiving records
Flexibly mapping data to different CMDB tables
Eliminating dependency on legacy scheduled jobs

This means you get complete visibility and accuracy as changes occur without the overhead.

How to configure automated AWS resource discovery for your ServiceNow CMDB

After you have integrated your ServiceNow instance to Turbot Guardrails; each AWS resource type can be configured to sync to the ServiceNow CMDB. Most often you would set the scope of the policy across many AWS resources from all your AWS accounts. In this example we will show how to enable syncing AWS S3 Buckets.

Simply set the Turbot Guardrails policy to “Enforce: Sync” and apply to all or specific AWS accounts:

For the AWS account we enabled the integration for, the following AWS resources will be in scope for the AWS discovery:

Instantly the AWS resources will be added to the associated ServiceNow CMDB table:

As AWS resources are added, updated, or deleted, Turbot Guardrails handles the configuration drift and keeps ServiceNow CMDB updated.

For example, when an AWS resource changes, Turbot Guardrails captures the configuration drift and updates ServiceNow CMDB:

AWS resource deletion can be managed as a complete synchronization where the record in ServiceNow is deleted as well, or archived to retain its record with an archive status.

Configure AWS Discovery to your CMDB Tables

You can configure the AWS to ServiceNow discovery sync behavior by:

Scoping to specific AWS services
Defining archive vs delete flow for resource deletions
Adding custom CMDB table columns and mappings

This level of control lets you tailor it to your unique CMDB table definitions whether directly to a table or table extension.

Keep your AWS to ServiceNow Discovery Simple

Managing AWS cloud discovery to ServiceNow does not need to be difficult and time-consuming. Using this Guardrails integration, you can automate AWS resource discovery across 100+ AWS services and sync to ServiceNow CMDB in just minutes. This can accelerate new integration efforts or augment existing methods with more accuracy and timely updates when changes occur.

Whether you are new to cloud discovery or looking to enhance existing capabilities, try a 14-day free trial by signing up directly with Turbot or through the AWS Marketplace.

What's new in the CIS v2.0 benchmark for AWS

Bob Tordella — Fri, 07 Jul 2023 13:56:52 +0000

The Center for Internet Security (CIS) just released an updated version (v2.0) of their CIS AWS Benchmark. The new version of the benchmark includes 2 new recommendations, 1 removed, and updates to descriptions and remediation steps.

What are the latest major changes?

Added - 1.22 Ensure access to AWSCloudShellFullAccess is restricted:
AWS CloudShell allows running CLI commands with full access, including file upload/download and sudo permissions. This presents a potential data exfiltration channel for malicious cloud admins. Restricting access and denying file transfer permissions through a more restrictive IAM policy is recommended. Personally, CloudShell is a great way to run commands in your account quickly and quickly enable tools like Steampipe. However, with any highly privileged role, boundary and lockdown permissions should be considered for the least privilege restrictions.
Added - 5.6 Ensure that EC2 Metadata Service only allows IMDSv2:
Instance Metadata Service Version 2 (IMDSv2), which is a session-oriented method which provides temporary, frequently rotated credentials. Version 2 adds new protections to mitigate vulnerabilities from open website application firewalls, open reverse proxies, SSRF, etc. This is a common recommendation used across many other frameworks like NIST, FedRamp, and HIPAA.
Removed - 2.1.1 Ensure all S3 buckets employ encryption-at-rest:
In Benchmark v1.5, the requirement for encryption-at-rest in S3 buckets (2.1.1) was removed since AWS now automatically encrypts all new objects using SSE-S3 as the default encryption setting since January 2023. Existing buckets with S3 Default Encryption remain unchanged, so you may want to still check for encryption-at-rest for legacy buckets. However, moving forward at a minimum all new buckets will be encrypted and you cannot remove encryption any longer. With the removal of the control, all other Storage recommendations have changed their IDs in section 2.0.

Try the new AWS CIS v2.0 controls!

The Steampipe AWS Compliance mod, is packed with hundreds of controls that check your AWS accounts for compliance with 25 benchmarks including NIST, PCI, HIPAA, SOC2, FedRAMP and more, now includes new controls for AWS CIS v2.0. If you're new to Steampipe, download Steampipe, install and configure the AWS plugin, and run these commands.

steampipe plugin update aws
git clone https://github.com/turbot/steampipe-mod-aws-compliance.git
cd steampipe-mod-aws-compliance
steampipe check aws_compliance.benchmark.cis_v200

If you've already installed Steampipe and the AWS plugin, and cloned the AWS Compliance mod, then just do this.

cd steampipe-mod-aws-compliance
git pull
steampipe check aws_compliance.benchmark.cis_v200

Here's a sample report in the console.

You can output results to formats including JSON, CSV, HTML, and ASFF, or use custom output templates to create new output formats.

To view the AWS CIS v2.0 benchmark report in your browser, run this command in the same cloned repo.

steampipe dashboard

Then open http://localhost:9194 in your browser and view the dashboard.

Extensible compliance controls

Steampipe delivers a full suite of tools to build, execute and share cloud configuration, compliance, and security frameworks using HCL + SQL! The community is constantly expanding the open source documentation and control coverage for CIS, PCI, HIPAA, NIST, and more. You can join the Steampipe Slack community to collaborate with other community members on your use cases.

Visualizing AWS EKS Kubernetes Clusters with Relationship Graphs

Bob Tordella — Fri, 21 Apr 2023 18:35:08 +0000

Steampipe is an open-source tool that helps users query, explore, and visualize their cloud environments. Relationship graphs in Steampipe are helpful visualizations that provide quick context and highlight important information about your resources. You can use these dashboards to visualize your AWS resources and also your AWS Elastic Kubernetes Service (EKS) clusters.

AWS EKS is a managed Kubernetes service that makes it easier to deploy, manage, and scale containerized applications using Kubernetes. Using Steampipe with EKS, you can visualize inside the cluster to gain insights:

How many resources do I have?
How old are my resources?
What are the various configurations of my resources?
What are the relationships between closely connected resources like clusters, nodes, pods, deployments, and jobs?
Who can perform operations like list, get, read, etc., on my resources?

Getting started

To get started with visualizing your EKS clusters through interactive dashboards, you can install Steampipe in your terminal.

First, install Steampipe:

sudo /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/turbot/steampipe/main/install.sh)"

Then, install the Kubernetes plugin:

steampipe plugin install kubernetes

Finally, install the Kubernetes Insights mod by following these steps:

git clone https://github.com/turbot/steampipe-mod-kubernetes-insights
cd steampipe-mod-kubernetes-insights
steampipe dashboard

Now visit localhost:9194 in your browser to view and interact with the dashboards.

If you already have an AWS EKS Kubernetes cluster in ~/.kube/config, Steampipe will automatically pick up your default context. If you would like to set up multiple clusters, you can update your Steampipe configurations to aggregate multiple contexts at once.

Kubernetes Relationship Graphs

In this section, we will explore various dashboards provided by Steampipe to visualize different aspects of your EKS clusters. We will look at the Namespace, Deployment, Service, Pod, Service Account, and Role dashboards, and discuss how they can help you better understand and manage your AWS EKS Kubernetes resources.

Namespace dashboard

Here's a high-level view of the kube-system namespace in the Namespace Detail dashboard.

The application runs as a Service. On initial view, the dashboard folds them all into an individual node, and folds the related DaemonSets, ReplicaSets and Deployments into their own individual nodes.

Deployment dashboard

If we open Deployments and click on the coredns Deployment we land in the Deployment Detail dashboard where we see the kube-dns Service linked to that Deployment. On hover we see details for one of the 2 pods in that Deployment's ReplicaSet: it's running, and it uses the coredns service account.

Service dashboard

Clicking into the kube-dns Service takes us to the Service Detail dashboard where we can see the path from a load balancer to the wordpress service to the wordpress Deployment with its ReplicaSet consisting of a single pod.

Pod dashboard

Visiting the Pod takes us to the Pod Detail dashboard. Here we can see, from another perspective, that it runs as the coredns service account. We can also see the single container in the pod, and we can see there are multiple read-only volumes mounted.

Service Account dashboard

The coredns service account is pretty simple. Here coredns runs as the service account. If we open that in the Service Account Detail dashboard, we see the 2 pods that run as that service account. However we can see the service account does not have any role bindings or secrets associated, which may indicate these pods may not have the necessary credentials needed.

Role dashboard

The role shown here, is again simple, so let's open a more interesting one in the Role Detail dashboard. Here we see that the eks:certificate-controller role has broader permissions to all resources, but specific get and update permissions on a specific certificate resource.

Making Kubernetes legible

These interconnected graphs work hand-in-hand with their dashboards' infocards, charts, and tables to make Kubernetes systems legible in a whole new way. Which of these seem most useful to you? What other kinds of relationships will help you understand your K8s environments and manage them more effectively? We look forward to hearing about your experiences with Kubernetes Insights, and our Slack community is the place to share them.

Where are those CloudTrail IP addresses coming from?

Bob Tordella — Tue, 19 Jul 2022 19:47:29 +0000

Setting up CloudTrail is essential to understanding your users' AWS API activity: what happened, by whom and from where. One field in the event logs is source IP address. But that doesn't tell you where in the world those AWS API actions are coming from. In this post we'll show how to use Steampipe with the AWS and ipstack plugins to enrich your CloudTrail events with location information.

As in previous posts we'll use AWS CloudShell for a quick-start experience that leverages your logged-in AWS credentials. Start your own CloudShell in the account with your CloudTrail information and follow along!

Setup

First install Steampipe:

sudo /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/turbot/steampipe/main/install.sh)"

Then install the AWS plugin:

steampipe plugin install aws

And then the ipstack plugin:

steampipe plugin install ipstack

Find the CloudTrail CloudWatch Log Group

Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. These events can be sent to a CloudWatch log group to allow for easy monitoring. Steampipe has an associated table that reads CloudTrail event data from a CloudWatch log group that is configured to log events from a trail.

Now with Steampipe and the plugins installed, you can run steampipe query and write SQL queries that reference tables provided by the AWS and ipstack plugins. For starters, let's query the aws_cloudtrail_trail table to find the CloudTrail and the related CloudWatch Log Group we'll use in this example:

$ steampipe query
Welcome to Steampipe v0.15.0
For more information, type .help
> select 
    name, 
    region,
    log_group_arn,
    latest_delivery_time
 from 
    aws_cloudtrail_trail

+-----------------------+-----------+----------------------+----------------------------------------------------------+
| name                  | region    | latest_delivery_time | log_group_arn                                            |
+-----------------------+-----------+----------------------+----------------------------------------------------------+
| cloudtrail-for-devto  | us-east-1 | 2022-07-06T20:38:09Z | arn:aws:logs:us-east-1:810361751552:cloudtrail-cwg-devto |
+-----------------------+-----------+----------------------+----------------------------------------------------------+

List the IP addresses in the log

Now let's review the source IP addresses in that trail:

select
    source_ip_address
  from
    aws_cloudtrail_trail_event
  where
    log_group_name = 'cloudtrail-cwg-devto'
    and source_ip_address ~ '^\d+\.\d+' -- filter ipv4 addresses

+-------------------+
| source_ip_address |
+-------------------+
| 104.53.216.85     |
| 82.102.17.180     |
| 89.248.165.99     |
| 107.170.20.63     |
| 212.102.58.164    |
+-------------------+

Geolocate the IP addresses

Finally, let's join those addresses with ipstack_ip to find out where they are coming from:

with addrs as (
  select 
    a.source_ip_address::inet
  from 
    aws_cloudtrail_trail_event a
  where 
    a.log_group_name = 'cloudtrail-cwg-devto'
    and a.source_ip_address ~ '^\d+\.\d+' -- filter ipv4 addresses
)
select 
  a.source_ip_address as ip,
  i.continent_name,
  i.country_name,
  i.region_name,
  i.city
from 
  addrs a
join
  ipstack_ip i
on
  a.source_ip_address = i.ip

+-----------------+----------------+----------------+-------------------+------------+
| ip              | continent_name | country_name   | region_name       | city       |
+-----------------+----------------+----------------+-------------------+------------+
| 104.53.216.85   | North America  | United States  | California        | Windsor    |
| 82.102.17.180   | Europe         | Spain          | Madrid            | Madrid     |
| 89.248.165.99   | Europe         | Netherlands    | North Holland     | Diemen     |
| 107.170.20.63   | North America  | United States  | New York          | Manhattan  |
| 157.230.162.15  | North America  | United States  | California        | Palo Alto  |
| 212.102.58.164  | North America  | United States  | Illinois          | Chicago    |
+-----------------+----------------+----------------+-------------------+------------+

More ways to enrich IP addresses in logs

The Net plugin can provide reverse DNS lookups, the AbuseIPDB plugin looks for malicious activity associated with IP addresses, and the Shodan plugin scans for exploitable vulnerabilities. You can use the same technique shown here with these other plugins -- separately or in combination -- to further enrich IP addresses captured in your AWS CloudTrail logs.

How to perform a security audit of your AWS account in AWS CloudShell

Bob Tordella — Fri, 20 May 2022 10:41:25 +0000

In my last post I showed how to install Steampipe and use it to instantly query your AWS APIs using SQL right in AWS CloudShell. For example here's a query that uses the Steampipe AWS plugin to query which AWS IAM users have MFA enabled:

select
  title,
  create_date,
  mfa_enabled
from
  aws_iam_user

+-----------------+---------------------+-------------+
| title           | create_date         | mfa_enabled |
+-----------------+---------------------+-------------+
| pam_beesly      | 2005-03-24 21:30:00 | false       |
| creed_bratton   | 2005-03-24 21:30:00 | true        |
| stanley_hudson  | 2005-03-24 21:30:00 | false       |
| michael_scott   | 2005-03-24 21:30:00 | false       |
| dwight_schrute  | 2005-03-24 21:30:00 | true        |
+-----------------+---------------------+-------------+

You can simply query your environment for these type of security configuration questions using SQL. There's thousands of examples you can leverage to get you started, and a wealth of possibilities to uncover details about your AWS configurations.

Running Security and Compliance Checks

While you can explore your AWS configurations running queries, Steampipe also provides modules which are collections of related dashboards, benchmarks, queries, and controls. Steampipe mods and mod resources are defined in HCL wrapping your SQL queries to create a benchmark. There are many published mod examples to get you started with thousands of controls readily available for security, compliance, tagging, and cost controls. Published modules can be found on the Steampipe Hub, and custom mods may be shared with others from any public git repository.

For example, the AWS Compliance Mod layers benchmarks and controls covering 13 compliance standards including CIS, HIPAA, NIST, PCI, FedRAMP, SOC 2 and more. Each benchmark includes a set of pass/fail controls. Each control tests for a compliance recommendation such as "EC2 instances" should be managed by AWS Systems Manager" and reports OK or Alarm.

Here's how to run the NIST 800-53 benchmark:
If you've already completed steps 1 - 3, skip to step 4:

1. Install Steampipe

sudo /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/turbot/steampipe/main/install.sh)"

2. Install the AWS plugin

steampipe plugin install aws

3. Install the AWS Compliance Mod

git clone https://github.com/turbot/steampipe-mod-aws-compliance
cd steampipe-mod-aws-compliance

4. Run the NIST 800-53 benchmark:

steampipe check benchmark.nist_800_53_rev_4

There are over 370 controls in that benchmark, so the command produces many screenfuls of output, here's the last one:

Export and Review the Findings

The summary is helpful, but you may want to digest the full report in varying formats. You can export to CSV, Markdown, HTML. Example of an HTML format:

steampipe check benchmark.nist_800_53_rev_4 --export=output.html

Using Files -> Download File in AWS CloudShell's Actions menu, you can download your output file steampipe-mod-aws-compliance/output.html and work with it locally.

Here's what the HTML report looks like:

AWS Compliance Quick Start

We put together a quick start script to bootstrap the flow above and prompt the user to select from the 13 available compliance benchmarks.

To get started with the quick start, spin up a new CloudShell and install Steampipe:

sudo /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/turbot/steampipe/main/install.sh)"

Then bring down the Steampipe AWS Compliance Quick Start script to install the AWS Plugin, AWS Compliance Mod, and receive the selection prompt asking which benchmark to run:

/bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/turbot/steampipe-samples/main/all/aws-compliance-quickstart/quickstart.sh)"

See it in action:

You can always run the last command again and it will skip the setup steps and prompt you for another compliance benchmark to run. Note: This last script was just a fun sample, generally you should stick to the official AWS Compliance Mod Controls to evaluate the controls, definitions and up to date information on available benchmarks.

Final Thoughts

I really enjoy using AWS CloudShell for these type of quick win use cases within an AWS account. It's remarkably easy to install your CLI tools like Steampipe, with no configuration required and instant gratification! Let me know how you use AWS CloudShell with your favorite CLI tools in the comments below.

Instantly query AWS with SQL in CloudShell

Bob Tordella — Tue, 03 May 2022 13:58:43 +0000

AWS CloudShell makes it easy to spin up a terminal right in your AWS account. CloudShell comes preinstalled with the AWS CLI and your credentials within the applicable account. Since CloudShell is just like any other terminal, you have the ability to bootstrap other tools without the need to spin up an instance. I personally find CloudShell useful for ad hoc actions I need to take in AWS CLI or with other open source tools.

As a lead on our open source tool Steampipe.io, I am a heavy user of the Steampipe CLI often running Steampipe on my local machine to run aggregated queries & reports across an AWS multi-account environment and other cloud accounts. When I am working with clients in their AWS accounts, I find it easy to work within their account structure vs setting up a local profile. In this case I find AWS CloudShell a quick win for being a guest in another account to bootstrap my tools in a temporary environment under their control.

In this post will walk through how to install Steampipe in your AWS CloudShell.

Steampipe Background

With Steampipe, you can instantly query your AWS APIs using SQL right in your terminal.

select
  title,
  create_date,
  mfa_enabled
from
  aws_iam_user

+-----------------+---------------------+-------------+
| title           | create_date         | mfa_enabled |
+-----------------+---------------------+-------------+
| pam_beesly      | 2005-03-24 21:30:00 | false       |
| creed_bratton   | 2005-03-24 21:30:00 | true        |
| stanley_hudson  | 2005-03-24 21:30:00 | false       |
| michael_scott   | 2005-03-24 21:30:00 | false       |
| dwight_schrute  | 2005-03-24 21:30:00 | true        |
+-----------------+---------------------+-------------+

It takes just a few seconds to install Steampipe itself, along with the AWS plugin that maps AWS API calls to Postgres tables.

Steampipe will resolve your region and credentials using the same mechanism as the AWS CLI (AWS environment variables, default profile, etc). Note: more can be extended for querying multiple accounts, regions, configuring credentials from your AWS Profiles, SSO, aws-vault etc.

AWS CloudShell + Steampipe

Alternatively you can use CloudShell to install Steampipe directly in your AWS Account. With CloudShell your credentials you use to sign into the AWS console are already forwarded to CloudShell. Since Steampipe will default to your local AWS credentials, from a cold start, you're querying AWS APIs with SQL in a matter of seconds.

Install Steampipe in CloudShell

Go to your AWS CloudShell, install Steampipe:

$ sudo /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/turbot/steampipe/main/install.sh)"
...
Installing
Applying necessary permissions
Steampipe was installed successfully to /usr/local/bin/steampipe

Install the AWS plugin

Now with Steampipe installed, you can install the AWS plugin:

$ steampipe plugin install aws

Installed plugin: aws v0.57.0
Documentation:    https://hub.steampipe.io/plugins/turbot/aws

Run a SQL query!

Now you are ready to run a SQL query. Since CloudShell already has the credentials in place, you can simply get started:

$ steampipe query
Welcome to Steampipe v0.13.6
For more information, type .help
> select * from aws_s3_bucket
+--------------------------------------+
| name                                 |
+--------------------------------------+
| jon-turbot-test-bucket-01            |
| cf-templates-1s5tzrjxv4j52-us-west-1 |
+--------------------------------------+

CloudShell takes full advantage of the Steampipe CLI components so you can inspect tables, configure environment variables, visualize syntax highlighting, select autofill suggestions, etc.

Final Thoughts

I enjoy using AWS CloudShell for ad hoc actions with AWS CLI and Steampipe within a specific AWS Account. Interested to learn from others on how you use CloudShell in your environment; what are your use cases and how often do you use CloudShell?