Introduction: The Spark

I was at the AI connect meetup with the title Data Science for Social Good. Came to know how college students and professionals can contribute to the social good. When they shared about the hackathon. Like a kid in a candy store, I was excited to join the hackathon. I had been looking for a way to give back to the community and this was the perfect opportunity.

The Team: A Symphony of Skills

Before the hackathon

As soon as I signed up for the hackathon, I started looking for a team from the participant list.

Based on what I understood from the AI connect meetup, I was in the mindset of having a team with necessary skills. I have architectural, DevOps and design thinking skills that can break down the problem into actionable items for each team member. But I don’t want to sign up to be a developer, especially as a front-end developer. So I was looking for

• frontend developer (UI/UX is important for the demo)
• backend developer (with API design skills),
• data scientist (in case we get large dataset, we need to analyze the data and build a model to predict)
• product owner / scrum master (who can’t code but can focus on team building, passionate to solve problem and importantly take us towards deliverable required).
• a presenter (who has powerpoint skills and can present the solution to the judges).

Even though I connected with a few participants, I wasn’t able to form a team but made a few friends who were open to discuss and decide at the Mixer.

At the hackathon Mixer

After the welcome speech, we started the mixer to find a team. It was recommended to find a team with 4-6 members.

Initially I found a team with smart and young developers. In the middle of forming a team, I went to look for a product owner/scrum master who can identify smaller scope that is achivable and manage coordination when friction occurs. By the time I come around, this team had more members. Everyone loves doers.

I restarted looking for a team where I could be an architect, DevOps engineer, or at worst case, a backend developer. As I approached and introduced myself by telling them who I am and what I can do, they were more than happy to have me as part of the team.

Team formed and my teammates are

- Anthony Chapman: Frontend developer
- Lucas Wyman: CEO type with marketing skills, lots of energy, passion to solve problems, and strong presentation skills
- Matthew Lucas: Backend developer
- Don Huovinen: Product owner type; skilled in low-code/no-code development
- Danny Logsdon: CEO type with expertise in generating backend code using .NET
- and myself (Mahendran Mookkiah): Architect, AWS DevOps

As teams were seated, there were a few members looking for a team. At the stage, they were asked to share their skills and what they could do. As soon as Bill Davis shared that he is a PowerBI expert, Danny raised his hand to welcome Bill to the team. I sensed Danny’s people skills and his supportive, “not leaving anyone behind” mindset.

Excited to see the problem cases.

The Problem Cases

We had three problem cases to solve. This is not the full case details. The section I understand.

Case 1: Measuring Non-Electoral Civic Engagement

To fill the gap between civic engagement and electoral engagement. Research the given dataset and design a method, tool, or platform to measure non-electoral civic engagement in Duval County. Use creative data sourcing to answer: “In what ways and how many North Floridians are engaging in civic life outside of voting?”

Case 2: Estimating the Local Self-Sufficiency Standard

Develop a Self-Sufficiency standard framework that helps gather data that will allow the community to better estimate the true cost of self-sufficiency in Duval County. This helps incoming residents find the region within our county that suits their needs and budget. This also helps local leaders make funding decisions, policy development, and design regional development programs to improve the community.

Case 3: Community Connections

The goal is to foster collaboration between non-profits, local organizations and volunteers/donations to maximize resource allocation. We need a platform to bring the donating people and organization to list their donations. Reduce the friction in delivering the donations to the people who is looking for it. Increase the transparency of how a donation reaches who and when so that the donors trust the platform and give more.

This is the case we have chosen to implement.

The Obstacles: Pushing Boundaries

As we started digesting “Community Connections” problem, our team started seeing the complexity and large scope of this project. This includes…

• Onboarding non-profit and organizations
• Helping donor’s tax document needs
• How to categorize the donation types (good, service, volunteer hours, money etc.,)
• How to outreach/attract people who needs
• What do we collect from users and how to validate?
• How do we match donations and needs?
• Should we build a blockchain method to track donations and their delivery until they reach the needy person?

The Breakthrough: From Idea to Impact

In order to build a working demo in 48 hours, we had to find a smaller and impactful idea. We decided to focus on organizations as customers and bring them a screen to show opportunities to donate. We also wanted to give every one of our team members the opportunity to contribute with their existing skills or something quick to learn. We were able to take on these tasks:

• Build UI to onboard organizations
• Allow the organization to login
• Scrape instagram for needs based on hashtags
• Build a chatbot to talk in natural language with a no-code platform
• Design a schema that includes needs with description, organization, services, and donations with description
• Reach out to local non-profits to discover go-to-market strategy and develop an impactful presentation

By Saturday evening at 3 PM (~24 hours), we had a clear idea of everyone’s task and were able to dream about putting it all together and presenting.

Gratitude: The Village Behind the Vision

Thank everyone involved. This is your chance to show appreciation.

• The Team: The team’s support and ability to adapt to change helped a lot for our team’s success.
• The Organizers & Mentors: Thanks to Jen, Chris, Pete and the team behind them.
• The Sponsors & Supporters: Thanks to iR for giving us space to work, AnuVision for the space to present the demo, AWS for providing a platform, and every sponsor.

The Victory: 2nd Place and Giving Back

We were thrilled to win 2nd place in the hackathon! This recognition validated our team’s hard work and the impact potential of our Community Connections platform.

As a team, we decided to donate a portion of our winnings to Mission House, a local Jacksonville Beach non-profit that empowers and compassionately supports neighbors in need. Their mission of helping people achieve stable, healthy, and self-sufficient lives aligns perfectly with our hackathon’s goal of fostering community connections and resource allocation.

This donation represents our commitment to not just building solutions during the hackathon, but continuing to support the community we’re trying to help.

Conclusion: Beyond the Hackathon

Hackathons are for quickly prototyping a challenge with technical experts. We were proud to contribute. The success is in capturing the thoughts of every team which tackles this problem case and brainstorming to build a product. We’re happy to hear this project will continue beyond the hackathon to create impact in the community.

AWS provides easy to setup email platform that enables capability to send emails from application. Application developers can create template that managed outside of application code. DevOps can create configuration set to based on use cases that limits, adds secrity and improve reputation.

Purpose

In this page, we are sharing basics of SES setup and value of properly managing suppression lists and virtual delivery manager.

Identities

Identity is about expressing our ownership towards a domain or individual email addresses that we work with. AWS platform allows us to begin work with sandbox environment to begin the development without waiting. It guides us setup DKIM, SPF, DMARC and BINI to increase reputation and guarentee the deliverability at each identity level.

Creating identity (easy to do in console with workflow - preferred)

$ aws sesv2 create-email-identity --email-identity mm-notes.com
{
    "IdentityType": "DOMAIN",
    "VerifiedForSendingStatus": false,
    "DkimAttributes": {
        "SigningEnabled": true,
        "Status": "NOT_STARTED",
        "Tokens": [
            "token-1",
            "token-2",
            "token-3"
        ],
        "SigningAttributesOrigin": "AWS_SES",
        "NextSigningKeyLength": "RSA_2048_BIT",
        "CurrentSigningKeyLength": "RSA_2048_BIT",
        "LastKeyGenerationTimestamp": 1754653048.28
    }
}
$ aws sesv2 get-email-identity --email-identity mm-notes.com
{
    "IdentityType": "DOMAIN",
    "FeedbackForwardingStatus": true,
    "VerifiedForSendingStatus": false,
    "DkimAttributes": {
        "SigningEnabled": true,
        "Status": "PENDING",
        "Tokens": [
            "token-1",
            "token-2",
            "token-3"
        ],
        "SigningAttributesOrigin": "AWS_SES",
        "NextSigningKeyLength": "RSA_2048_BIT",
        "CurrentSigningKeyLength": "RSA_2048_BIT",
        "LastKeyGenerationTimestamp": 1754653048.28
    },
    "MailFromAttributes": {
        "BehaviorOnMxFailure": "USE_DEFAULT_VALUE"
    },
    "Policies": {},
    "Tags": [],
    "VerificationStatus": "PENDING",
    "VerificationInfo": {}
}

Once SES identity created, it will be in Verification pending status. To validate, we need to create TXT DNS records based on the Tokens under DkimAttributes. These record value ends with dkim.amazonses.com. If you are using AWS Route53 as your DNS, you may push a button Publish DNS records to Route53 on the console to publish. Alternatively you can download the record set and work with your DNS provider.

We have to wait for AWS to validate the DNS entries. Until then if we attempt sending email, we get error message like below.

dig CNAME token._domainkey.mm-notes.com
nslookup -type=CNAME token._domainkey.mm-notes.com

$ aws ses send-email \
     --from "[email protected]" \
     --destination "[email protected]" \
     --message "Subject={Data=Test Email},Body={Text={Data=Hello World}}"

An error occurred (MessageRejected) when calling the SendEmail operation: Email address is not verified. The following identities failed the check in region US-EASTT-1: [email protected], [email protected]

Once the setup successful,

$ aws ses send-email \
>     --from "[email protected]" \
>     --destination "[email protected]" \
>     --message "Subject={Data=Test Email},Body={Text={Data=Hello World}}"
{
    "MessageId": "random-uuid-000000"
}

Suppression list

Suppression list is being built by SES over a period of time based on its attempt to deliver mails to target list. We can also import suppression list which is helpful if we have multiple environment and when needed to follow the “Unsubscribe / Do Not Distrub” guidelines.

• Review periodically
• Share data with application/business team which sends email

While AWS console is not allowing to export all suppression list, we can export it with below python script. This is useful to share with business team to aware and act.

import boto3
import csv
from datetime import datetime

def get_ses_suppressions():
    # Initialize SESv2 client for SES service region
    ses_client = boto3.client('sesv2', region_name='us-east-1')
    
    # Lists to store different types of suppressions
    suppressions = []
    
    # Get current timestamp for filename
    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
    
    try:
        # Get all suppressed destinations with manual pagination using NextToken
        next_token = None
        
        while True:
            # Prepare parameters for API call
            params = {'Reasons': ['BOUNCE', 'COMPLAINT']}
            if next_token:
                params['NextToken'] = next_token
                
            # Make API call
            response = ses_client.list_suppressed_destinations(**params)
            
            # Process results
            for item in response.get('SuppressedDestinationSummaries', []):
                suppressions.append({
                    'email': item['EmailAddress'],
                    'reason': item['Reason'],
                    'last_update_time': item['LastUpdateTime'].strftime('%Y-%m-%d %H:%M:%S'),
                    'attributes': str(item.get('Attributes', {}))
                })
            
            # Check if there are more results
            next_token = response.get('NextToken')
            if not next_token:
                break
                
        # Write to CSV file
        output_file = f'ses_suppressions_{timestamp}.csv'
        
        with open(output_file, 'w', newline='') as csvfile:
            fieldnames = ['email', 'reason', 'last_update_time', 'attributes']
            writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
            
            writer.writeheader()
            for suppression in suppressions:
                writer.writerow(suppression)
                
        print(f"Successfully exported {len(suppressions)} suppressions to {output_file}")
        
    except Exception as e:
        print(f"Error occurred: {str(e)}")

if __name__ == "__main__":
    get_ses_suppressions()

We can also use AWS cli to periodically check new supressions

$ aws sesv2 list-suppressed-destinations --start-date $(date -d "14 days ago" +%s)

Virtual Delivery Manager

• Additional cost but valuable service.
• Help to find reasons for mail delivery failures
• provides dashboard with metrics about send, open and click rates.

$ aws sesv2 create-export-job \
    --export-data-source '{
        "MessageInsightsDataSource": {
            "StartDate": '$(date -d "1 days ago" +%s)',
            "EndDate": '$(date +%s)',
            "Include": {
                "LastDeliveryEvent": ["PERMANENT_BOUNCE","TRANSIENT_BOUNCE", "COMPLAINT", "UNDETERMINED_BOUNCE"]
            }
        }
    }' \
    --export-destination '{
        "DataFormat": "CSV"
    }'

The above command will return an job-id. We can check the job detais with following command. When it completes, it will have a pre-signed url for the export. (Not in our AWS account - Security and Privacy?)

$ aws sesv2 get-export-job --job-id <your-job-id>

Configuration Sets

• By using configuration sets, we can address individual application team needs in logging, suppression, archiving by overriding account level settings.
• Having separate configuration sets for application teams, environments helps to manage (pause/monitor) smaller subset.
• If you have a need of keeping mail audit logs (not the body of email) for long time, you can setup SNS event destinations.

Motivation

Single Sign On (SSO) is considered as a security best practice. Managing users and their access rights centrally help smooth and quick onboarding and offboarding employess in the organization. I wanted to share notes to enable this integration and strategy to implement few use cases.

Agile style requirements

1. As a organization employee, I should use my office 365 login credentials to login into AWS console.
2. As a infrastructure admin, I should be able to give birth rights access only for employees who needs AWS access.
3. As a AWS user, I should be able to login into AWS console with view only access when I dont intent to make changes.
4. As a infrastructure owner, I want to use SCIM to provision users to have connected user principles across systems.
5. As a infrastructrue owner, I want AWS access assigned to users based on security groups which are provisioned via SCIM.
6. As a infrastructure admin, I should be able to let the application/support team manage Security Group members.

SSO integration

As both Microsoft and AWS are favouring the need of SSO integration, the steps to integrate is getting easy as time passes. Here are steps and screenshots.

In technical terms, This integration uses SAML based SSO where Microsoft Entra ID act as IDP and AWS IAM Identity Center act as SP. AWS IAM Identity Center also accepting SCIM provisioing to keep User-Group setup in sync with Microsoft Entra ID.

Create Enterprise Application

1. Go to Azure Portal -> Entra ID -> Manage Enterprise applications – https://portal.azure.com/#view/Microsoft_AAD_IAM/StartboardApplicationsMenuBlade/~/AppAppsPreview/menuId~/null
2. Click New Application, Enter “aws” in search. Take time to see options and select the one which says `AWS IAM Identity Center’.

Now you have successfully created the application.

Download SP Metadata

IAM Identity Center act as SP which is allowed to configure only one IDP. By default it is using Identity Center Directory. We have to change it by going to Identity Ceter -> Settings –> Identity Source Actions -> Change identity source

Once you select External identity provider, you can download metadata file. If you are not managing IDP Entra ID, you should share this with respective team.

Upload SP Metadata and Download IDP Metadata

In Azure Portal, Entra ID –> Enterprise applications –> Go to the same application created in earlier setup –> Manage –> Single Sign-on – Upload (SP) Metadata file

Once you upload SP metadata, you can download Federation (IDP) Metadata XML. If you dont manage AWS Identity Center, share it with respective team.

Upload IDP Metadata

IDP metadata which is required to upload in the same change identiy source page in AWS console.

User Group planning

In Entra ID create security groups to represent the different teams which require AWS access.

Sample naming format:

• sg-infra-aws--- --> map with the specific account associated with the app and environmet.

where role can be developers, data scientists, architects, support etc.,

• sg-infra-aws-users –> map with basic/birth right access (make all the above groups added to this group as members). Test hierarchy works.

SCIM provisioning

SCIM - System for Cross-domain Identity Management

AWS Access Setup

Resources

• Micorsoft Tutorial
• SCIM Provisioning
• https://scim.cloud/
• https://docs.aws.amazon.com/singlesignon/latest/developerguide/what-is-scim.html

Notes

• Use namespaces (key name separated by colon :). Example: `company-name:department-name:standard-key’
• Cover RACI (Responsible, Accountable, Consulted and Informed) details with multiple tags.
• Define tags based on the need. What questions we are going to answer and what tags needed for integration.
• Implement process for Tag strategy iteration and improvement cycle
• Identify and promote a team member who is responsible for implementing and guiding tag strategy.
• Define mandatory tags and apply tagging policies to report or prevent missing tags.
• Plan review/measure the effectiveness of tags against target questions and goals. Downgrade mandatory tags to optional.
• Avoid analysis paralasis on keys as it can be altered with commands/scripting or tools like tag editor.

Quotes from resources

While we(AWS) strongly recommend that you adopt the practices outlined in Organizing Your AWS Environment Using Multiple Accounts whitepaper, realistically customers often find themselves with mixed and complex account structures that take time to migrate away from. Implementing a rigorous and effective tagging strategy is the key in this scenario, followed by activating relevant tags for cost allocation in the Billing and Cost Management console (in AWS Organizations, tags can be activated for cost allocation only from the Management Payer account).

Cost Allocation Tags

Once you defined the taging strategy, define cost allocation tags if the tag can answer any FINOPS questions.

Pricing Awareness

AWS pricing pages has very detaild documentation with samples. However it takes time to understand and figure out what is the cost of our resource based on the option we have choosen while creating.

We can tag the cost factor in the tag based on the provision we have made. This creates awareness to the engineers. For example cost-factor: $16/month

Use Tag Editor

Tag editor can be very useful to bulk edit tags to refactor existing tags or add tags by filtering resources.

Brownfield Tagging - Use programs

Tagging existing infrastructure with 1000s of resources will be a overwhelming task. Especially if the resources are not part of infrastructure as code. If you can derive the tag values based on existing attributes or names of the resource, then we could write program to select the resource with filtering and apply tags programatically. Here are few python example bulk tag examples.

VPC Endpoints

import boto3
import logging

# Configure logging
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s - %(levelname)s - %(message)s')

def list_vpc_endpoints(region_name):
    """
    Lists all VPC endpoints in a given AWS region.

    Args:
        region_name (str): The AWS region to list VPC endpoints in (e.g., 'us-east-1').

    Returns:
        list: A list of dictionaries, where each dictionary represents a VPC endpoint.
              Returns an empty list if no VPC endpoints are found or if an error occurs.
    """
    try:
        # Create an EC2 client for the specified region
        ec2_client = boto3.client('ec2', region_name=region_name)

        # Describe VPC endpoints
        response = ec2_client.describe_vpc_endpoints()

        # Extract VPC endpoint information
        vpc_endpoints = response.get('VpcEndpoints', [])  # Safe handling if 'VpcEndpoints' key is missing

        if vpc_endpoints:
            logging.info(f"Found {len(vpc_endpoints)} VPC endpoints in region {region_name}.")
            return vpc_endpoints
        else:
            logging.info(f"No VPC endpoints found in region {region_name}.")
            return []

    except Exception as e:
        logging.error(f"An error occurred: {e}")
        return []



def tag_vpc_endpoint(region_name, endpoint_id, endpoint_service_name):
    """
    Tags a VPC endpoint with the tag "app-name" and the endpoint name as the tag value.

    Args:
        region_name (str): The AWS region where the VPC endpoint exists.
        endpoint_id (str): The ID of the VPC endpoint to tag.
        endpoint_service_name (str): The desired name for the endpoint to be used as the tag value.
    """
    try:
        ec2_client = boto3.client('ec2', region_name=region_name)

        tags = [{
            'Key': 'service-name',
            'Value': endpoint_service_name
        },
        {
            'Key': 'resource-type',
            'Value': 'vpc-endpoint'
        },
        {
            'Key': 'approx-cost',
            'Value': '$16/month' # modify based on your observation/calculation
        }]

        ec2_client.create_tags(Resources=[endpoint_id], Tags=tags)
        logging.info(f"Successfully tagged VPC endpoint {endpoint_id} with app-name:{endpoint_service_name}")

    except Exception as e:
        logging.error(f"Error tagging VPC endpoint {endpoint_id}: {e}")


def main():
    """
    Main function to list VPC endpoints and tag them.
    """

    region = input("Enter the AWS region (e.g., us-east-1): ")
    if not region:
        print("Region cannot be empty. Please provide a valid AWS region.")
        return

    vpc_endpoints = list_vpc_endpoints(region)

    if vpc_endpoints:
        print("\nVPC Endpoints:")
        for endpoint in vpc_endpoints:
            endpoint_id = endpoint['VpcEndpointId']
            endpoint_service_name = endpoint.get('ServiceName', 'UnknownEndpoint') #Default value if ServiceName is empty
            print("------------------------------------")
            print(f"VPC Endpoint ID: {endpoint_id}")
            print(f"VPC ID: {endpoint['VpcId']}")
            print(f"Service Name: {endpoint['ServiceName']}")
            print(f"State: {endpoint['State']}")
            print(f"Creation Timestamp: {endpoint['CreationTimestamp']}")
            # Tag the VPC endpoint
            tag_vpc_endpoint(region, endpoint_id, endpoint_service_name)

    else:
        print(f"No VPC endpoints found or an error occurred in region {region}.")

if __name__ == "__main__":
    main()

DynamoDB

import boto3

sts_client = boto3.client('sts')  # Create STS client
account_id = sts_client.get_caller_identity()['Account']
print(account_id)

def tag_dynamodb_tables():
    """Tags all DynamoDB tables with the 'app-name' tag,
    using the table name as the tag value.
    """
    dynamodb = boto3.client('dynamodb')
    tables = getAllTableByPaginating(dynamodb)
    tagTables(dynamodb, tables)




def getAllTableByPaginating(dynamodb):
    table_names = []
    last_evaluated_table_name = ''

    try:
        # List all DynamoDB tables using pagination
        while True:
            if last_evaluated_table_name:
                response = dynamodb.list_tables(ExclusiveStartTableName=last_evaluated_table_name)
            else:
                response = dynamodb.list_tables()

            table_names.extend(response.get('TableNames', []))

            last_evaluated_table_name = response.get('LastEvaluatedTableName')

            if not last_evaluated_table_name:
                break
    except Exception as e:
        print(f"Error listing tables: {e}")
        return []

    return table_names

def tagTables(dynamodb, tables):
    for table_name in tables:
        try:
                # Check if the table already has the 'app-name' tag.  This avoids unnecessary API calls.
            existing_tags = dynamodb.list_tags_of_resource(
                    ResourceArn=f'arn:aws:dynamodb:{boto3.session.Session().region_name}:{account_id}:table/{table_name}'
                )

            app_name_tag_exists = False
            if 'Tags' in existing_tags:
                for tag in existing_tags['Tags']:
                    if tag['Key'] == 'app-name':
                        app_name_tag_exists = True
                        break

            if not app_name_tag_exists: # Only tag if it doesn't exist.
                    # Tag the table
                dynamodb.tag_resource(
                        ResourceArn=f'arn:aws:dynamodb:{boto3.session.Session().region_name}:{account_id}:table/{table_name}',
                        Tags=[
                            {
                                'Key': 'app-name',
                                'Value': table_name
                            },
                        ]
                    )
                print(f"Tagged table: {table_name}")
            else:
                print(f"Table {table_name} already has the app-name tag. Skipping.")

        except Exception as e:
            print(f"Error tagging table {table_name}: {e}")


if __name__ == "__main__":
    tag_dynamodb_tables()

Workspaces

import boto3

def tag_workspaces():
    """
    Lists all AWS WorkSpaces and adds a tag with Key='primary-owner' and Value equal to the Workspace's username.

    Prerequisites:
        - The IAM role/user running this script needs permission for:
            workspaces:DescribeWorkspaces
            workspaces:CreateTags
    """
    try:
        workspaces = boto3.client('workspaces')
        all_workspaces = []
        next_token = ''

        while True:
            if next_token:
                response = workspaces.describe_workspaces(NextToken=next_token)
            else:
                response = workspaces.describe_workspaces()

            workspaces_data = response.get('Workspaces', [])

            for workspace in workspaces_data:
                workspace_id = workspace.get('WorkspaceId')
                username = workspace.get('UserName')

                if workspace_id and username:
                    try:
                        # Tag the workspace
                        workspaces.create_tags(
                            ResourceId=workspace_id,
                            Tags=[
                                {
                                    'Key': 'primary-owner',
                                    'Value': username
                                }
                            ]
                        )
                        print(f"Successfully tagged Workspace '{workspace_id}' with primary-owner='{username}'")
                    except Exception as tag_err:
                        print(f"Error tagging Workspace '{workspace_id}': {tag_err}")

            next_token = response.get('NextToken')
            if not next_token:
                break


    except Exception as e:
        print(f"Error listing or tagging WorkSpaces: {e}")


if __name__ == "__main__":
    tag_workspaces()

Resources

• https://docs.aws.amazon.com/whitepapers/latest/tagging-best-practices/tagging-best-practices.html
• https://wellarchitectedlabs.com/cost-optimization/
• https://workshops.aws/categories/Cost%20Management

Google offers free services for nominal use for both personal and business purpose. For businesses who wants to have their own office workspace solution (like Microsoft O365), google offers GSuite. But it

Often businesses needs to share Google services (ex: GDrive) to outside organization. At the sametime not to allow personal accounts (with gmail domain) from accessing the business documents/resources.

Also it is good not to use personal accounts and avoid mixing personal and business documents, it is recommended getting separate google account.Even better, create a google account so your business partners can simply use your business mail address to share their google storage contents and collaborate.

Setup global access policy

Configure global setting for Google Shared Drive based on the need.

• Completely disable sharing outside domain
• Allow only whitelisted domains
• Allow only with Google accountss
• Allow public if they know the link

Protecting GSuite/GDrive by whitelisting domains

https://admin.google.com/ac/domains/allowlisted

Creating Google account with external domain

• Logout previously logged in google accounts or open browser in “private browsing / incognito” mode

• At the sign in screen, follow create account flow. Choose “For work or my business”

• Fill the required fields

• Google recommends few email address with gmail domain. Because we want to use our business email based google account, select User your existing email

• Provide work mail address

• Verify the ownership of the email you provided

• Create password

• Missing screenshot for setting up MFA.

• Confirm the ownership of MFA device

• Skip setting up business unless you want a GSuite account (This is only required if you are the org admin)

For admins global settings

• Individual shared drive members and access can be setup under managedrive settings.

  

Resources

• https://apps.google.com/supportwidget/articlehome?hl=en&article_url=https%3A%2F%2Fsupport.google.com%2Fa%2Fanswer%2F60781%3Fhl%3Den&assistant_event=welcome&assistant_id=mega-bot-shared-drive&product_context=60781&product_name=UnuFlow&trigger_context=a&fragment=restrict_sharing_to_domains

Useful AWS commands

To list all manual and automated Redshift snapshots created in the last 30 days:

aws redshift describe-cluster-snapshots \
  --snapshot-type manual \
  --start-time $(date -v-365d +"%Y-%m-%dT%H:%M:%S")

To limit few fiels with jq

aws redshift describe-cluster-snapshots | jq ".Snapshots[] | {SnapshotIdentifier, ClusterIdentifier, SnapshotCreateTime, SnapshotType, TotalBackupSizeInMegaBytes, ManualSnapshotRetentionPeriod}"

List snapshots without retention period

$ aws redshift describe-cluster-snapshots --snapshot-type manual,automated --no-retention-period

--no-retention-period is NOT working as Amazon Q explains or unofficial blog posts. It is not implemented

$ aws redshift describe-cluster-snapshots --snapshot-type manual  | jq  ".Snapshots |= map(select (.ManualSnapshotRetentionPeriod == -1))" | jq ".Snapshots[] | {SnapshotIdentifier, ClusterIdentifier, SnapshotCreateTime, SnapshotType, TotalBackupSizeInMegaBytes, ManualSnapshotRetentionPeriod}"

To view snapshot schedules

$ aws redshift describe-snapshot-schedules

Create cluster

Node Types

• economic - dc2.large
• Production - choose any ra3

aws redshift describe-clusters | jq ".Clusters[] |= {ClusterIdentifier,NodeType, ClusterStatus }"

Backup & Snapshots

Before making any infrastructure or DBA kind changes, it is recommended to take backup.

aws redshift create-cluster-snapshot --cluster-identifier <cluster-name> --snapshot-identifier <snapshot-name> [--snapshot-cluster-identifier <cluster-name>]
aws redshift create-cluster-snapshot --cluster-identifier myredshift-sandbox --snapshot-identifier myredshift-sandbox-001

Sample Queries

create table driver (name VARCHAR, age integer, candrivecar boolean);
insert into driver values ('Mahendran', 55, true);

Manipulate sample DB

select * from users limit 10
delete from users where username = 'AEB55QTM'
select * from users where username = 'AEB55QTM'

aws redshift create-cluster-snapshot --cluster-identifier myredshift-sandbox --snapshot-identifier myredshift-sandbox-after-change

Pause Cluster

$ aws redshift pause-cluster --cluster-identifier <cluster-name>
$ aws redshift pause-cluster --cluster-identifier myredshift-sandbox
$ aws redshift describe-clusters | jq ".Clusters[] |= {ClusterIdentifier,NodeType, ClusterStatus }"
{
  "Clusters": [
    {
      "ClusterIdentifier": "myredshift-sandbox",
      "NodeType": "dc2.large",
      "ClusterStatus": "pausing"
    }
  ]
}
$ aws redshift describe-clusters | jq ".Clusters[] |= {ClusterIdentifier,NodeType, ClusterStatus }"
{
  "Clusters": [
    {
      "ClusterIdentifier": "myredshift-sandbox",
      "NodeType": "dc2.large",
      "ClusterStatus": "paused"
    }
  ]
}

Describe Events for activity monitoring

$ aws redshift describe-events --source-identifier myredshift-sandbox --source-type cluster

Resume cluster

$ aws redshift resume-cluster --cluster-identifier <cluster-name>
$ aws redshift resume-cluster --cluster-identifier myredshift-sandbox
$ aws redshift describe-clusters | jq ".Clusters[] |= {ClusterIdentifier,NodeType, ClusterStatus }"
{
  "Clusters": [
    {
      "ClusterIdentifier": "myredshift-sandbox",
      "NodeType": "dc2.large",
      "ClusterStatus": "resuming"
    }
  ]
}
$ aws redshift describe-clusters | jq ".Clusters[] |= {ClusterIdentifier,NodeType, ClusterStatus }"
{
  "Clusters": [
    {
      "ClusterIdentifier": "myredshift-sandbox",
      "NodeType": "dc2.large",
      "ClusterStatus": "available"
    }
  ]
}

Delete cluster

Delete cluster using console to confirm after reviewing. Paused cluster can be deleted; Create final snapshot to avoid data loss.

$ aws redshift describe-cluster-snapshots --snapshot-type manual  | jq  ".Snapshots |= map(select (.ManualSnapshotRetentionPeriod == -1))" | jq ".Snapshots[] | {SnapshotIdentifier, ClusterIdentifier, SnapshotCreateTime, SnapshotType, TotalBackupSizeInMegaBytes, ManualSnapshotRetentionPeriod}"
{
  "SnapshotIdentifier": "myredshift-sandbox-final-snapshot",
  "ClusterIdentifier": "myredshift-sandbox",
  "SnapshotCreateTime": "2024-08-26T22:19:17.623000+00:00",
  "SnapshotType": "manual",
  "TotalBackupSizeInMegaBytes": 4206,
  "ManualSnapshotRetentionPeriod": -1
}
{
  "SnapshotIdentifier": "myredshift-sandbox-after-change",
  "ClusterIdentifier": "myredshift-sandbox",
  "SnapshotCreateTime": "2024-08-26T21:59:49.967000+00:00",
  "SnapshotType": "manual",
  "TotalBackupSizeInMegaBytes": 3931,
  "ManualSnapshotRetentionPeriod": -1
}
{
  "SnapshotIdentifier": "myredshift-sandbox-001",
  "ClusterIdentifier": "myredshift-sandbox",
  "SnapshotCreateTime": "2024-08-26T21:45:05.894000+00:00",
  "SnapshotType": "manual",
  "TotalBackupSizeInMegaBytes": 707,
  "ManualSnapshotRetentionPeriod": -1
}

Restore from snapshot

$ aws redshift restore-from-cluster-snapshot \
    --snapshot-identifier myredshift-sandbox-final-snapshot \
    --cluster-identifier myredshift-sandbox-new \
    --node-type dc2.large \
    --allow-version-upgrade
$ aws redshift describe-clusters | jq ".Clusters[] |= {ClusterIdentifier,NodeType, ClusterStatus }"
{
  "Clusters": [
    {
      "ClusterIdentifier": "myredshift-sandbox-new",
      "NodeType": "dc2.large",
      "ClusterStatus": "creating"
    }
  ]
}

While restoring from snapshot, we can change the cluster configuration. To find what options available…

aws redshift describe-node-configuration-options --snapshot-identifier myredshift-sandbox-final-snapshot --region eu-west-1 --action-type restore-cluster

$ aws redshift restore-from-cluster-snapshot --snapshot-identifier as-is-as-of-aug-23 --cluster-identifier mysandbox-3 --node-type dc2.large --allow-version-upgrade

An error occurred (InvalidSubnet) when calling the RestoreFromClusterSnapshot operation: No default subnet detected in VPC. Please contact AWS Support to recreate default Subnets.

Internet accessible

To make/block the cluster accessible from internet (outside VPC), change the cluster properties –> Network and security settings –> Publicly accessible settings

To make/block the serverless workgroup accessible from internet (outside VPC), go to your workgroup and edit Data access –> Publicly accessible settings

Create user and grant access

select * from pg_user;
CREATE USER awsdbuser WITH PASSWORD 'YourSecurePassword';
GRANT ROLE sys:dba TO awsdbuser;
GRANT ROLE sys:superuser TO awsdbuser;
select * from pg_user;

Resources

• https://docs.aws.amazon.com/cli/latest/reference/redshift/
• https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT-examples.html

General User

• You accidentally post something publicly which meant for a private message. What is it and how do you recover?
• You wake up to find your most popular social media post ever! What is it and why do you think it went viral?
• You’re on a deserted island and can only bring one social media app. Which one do you choose and why?
• How has social media impacted the way you connect with friends and family?
• How can social media be a force for positive change in the world?
• Imagine a future without social media. What are the pros and cons?
• You inherit a social media account with a million followers but no content. What kind of content do you create and why?
• Pick a social media influencer you admire. Explain what qualities make them successful and how you could apply those to your own social media presence

Expert: Public Relationship Officer, Communication Leader, Has more followers, Posts often

• Imagine we are having open house event? How do you plan to promote the event to get more guests and what are the hashtags for target audience?
• Share us some tips and tricks for social media posts to attract more followers
• How do you measure the success of a social media campaign? What metrics would you track and how would you use that data to inform future strategies?
• A negative comment about our brand goes viral on social media. How would you approach the situation? What steps would you take to address the issue and minimize reputational damage?
• Imagine we’re about to launch a new product. Walk me through the steps you would take to develop a social media campaign to generate pre-launch buzz and excitement.
• How do you stay up-to-date on the latest social media trends and platforms? How do you determine which platforms are most relevant for a specific PR campaign?
• What are the biggest differences between managing social media for a personal brand and for an organization?
• Describe your approach to creating engaging social media content that resonates with a target audience. How do you tailor content for different platforms?
• What social media management tools and platforms are you familiar with? How would you utilize them to streamline your workflow and track campaign performance?
• What are your thoughts on the future of social media? What emerging trends do you see impacting the way organizations use social media to communicate?
• Share few checks you would apply before posting a message and why is it more important?

If you dont have business problem, then you can’t possibiliy come up with machine learning algorithm and model to answer what you are trying figure out.

Terminology

What is a model?

Model is a trained algorithm which is

• Used to identify patterns in your data, and
• Does not require explicit, manually set rules.

Supervised

Learn by identifying data that is already been labeled. We have/had data to train

Classification:

• Binary (Yes or No)
• Multiclass Classfication (like tags. Ex: possible answers for call center - “IT Support”, “Returns”, “Accounting”)
• Regression problem (Range)

Unsupervised

There is no existing data points to train. We just have raw data. The machine has to uncover and create the labels itself.

• use data grouping.
• detects annomolies.

Reinforcement Learning

Learning through trial and error.

Resources

• https://training.resources.awscloud.com/on-demand-events/vod-ve-machine-learning-basics#main-content

Purpose

Why/When do we need VPC?

• Isolating networks

Characteristics

• Limitted to one AWS Region
• One default VPC comes with each AWS Region.
• Span accross all Avalability Zones within the Region.
• Default CIDR 172.31.0.0/16 (Range 172.31.0.0 - 172.31.255.255 - 65,536 IPs in this range)
• Availability zone name (ex: us-east-1) does not represent physical location of the servers. Its a logical name per aws account.
• AvailabilityZoneId (ex: use1-az1) associated with a physical location which is common for all AWS account. This is best to compare if we ever work with two different accounts/partners.

AWS VPC FAQ

• Is there a way to generate diagram from existing infrastructure?
  • No out of the box solution from AWS.
• How to look at the AS-IS network map?
  • Go to your VPC and explore the Resource map.
• What resources are available within a given VPC?
  • Drill through the subnet which is might span across availability zones.
• How do I debug challenging network issue?
  • Flow logs is available at VPC and Subnet level.

VPC Diagram

Useful AWS commands

aws ec2 describe-vpcs
aws ec2 describe-internet-gateways
aws ec2 describe-subnets

AWS VPC User Guide

VPC and Features

• Logically isolated virtual network.
• Limitted to one region
• Gateways connects your VPC to another network.
• VPC peering - routes traffic between two VPCs.
• Transit gateways - central hub.

Route tables

• One or more can be defined
• used to direct the network traffic from your VPC
• subnet implicitly associated with main route table

Designing VPC

• IAM permissions
• Define strategy to maintain IaC
• Define policies to get the team informed about change in network.
• Choose the size of your VPC - how many IP addresses you will need across your AWS accounts and VPCs.
• IP Address Manager (IPAM) makes it easier to plan, track and monitor IP addresses of your application.

Internet traffic

• To receive traffic from internet, VPC must have an internet gateway.
• Subnet route table must have an entry to route to internet gateway.
• AWs resournces (instances) have public IP address and associated security group that allow traffic from internet.

Aleternatively, use ELB with with public NAT gateway.

Monitoring

• VPC flow logs helps to monitor, debug
• VPC flow log can be stored/published into CloudWatch, S3 and Firehouse.
• The cost depends on where it is published and how long we keep the log.

Why and Who

Because enabling VPC logs will incur cost, it is recommended to identify the need and proper process in place to ensure the logs are monitored or audited. It is important to let the stakeholders involved. Depends on the stakeholder commitment we should enable right level of logging.

Cyber Security - SOC - Security Operation Center needs

• Security team often needs network logs to analyze the trafic, identify unusual activity, protect the infrastructure.
• Many SIEM tools available that are designed to work with VPC flow logs.
• Cyber Security team should request to enable the log based on the level of security needs.
• May ask to keep the logs for long time - prefer cold storage after live analysis.

Network Engineer

• Diagnose performance or network issues.
• May ask on the need basis.
• May not required to keep the log for long time.
• Limit the log collection to specific service / subnet to make the work productive, cost effective.

Complaince and Certification

• If your organization has compliance team, they may ask for recording the VPC logs based on the level.
• If your organization seeking for any certification to meet customer demands, it may need VPC flow logs monitored.
• Example: Payment Card Industry (PCI) Compliance woud require VPC flow logs to be enabled.

Auditor

• It is good to start with auditor to engage commitment to audit the logs.
• Both cyber and complaince team should have regular auditing process.

Common Understanding and Excuses for not enabling VPC flow logs

• We use GuardDuty.
• We use CloudTrail
• We use WAF, Load Balancer and application log.
• Log collections expensive.
• We dont have people and tools to consume the logs to get value.
• No auditors asked for details which is only found in VPC flow logs.
• No business decision taken by understanding the value or risk.

Questions

• How do we plan IP addressing when HA is a major concern?
• 750 hours of Public IPv4 usage is free. How do we monitor its usage?
• For production environment, are you deploying AWS resources evenly in more than one Avalability Zone?
• For a development or test environment, Are you saving by using one Availability Zone?
• How do you maintain network security settings in compliance with best practices and cyber sec team’s requirement all the time?
• What is the Amazon VPC quotas?

Resources

• https://docs.aws.amazon.com/vpc/latest/userguide/configure-your-vpc.html
• https://mxtoolbox.com/subnetcalculator.aspx
• https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpc.html
• https://docs.aws.amazon.com/vpc/latest/userguide/how-it-works.html
• https://docs.aws.amazon.com/vpc/latest/ipam/
• https://www.netskope.com/blog/a-real-world-look-at-aws-best-practices-logging
• https://www.devopsschool.com/blog/what-is-cloudtrail-and-use-cases-of-cloudtrail/

$ atlas auth login
$ atlas setup --clusterName myAtlasClusterEDU --provider AWS --currentIp --skipSampleData --username myAtlasDBUser --password myatlas-001 | tee atlas_cluster_details.txt

$ atlas clusters sampleData load myAtlasClusterEDU

Self-Managed Database Security

• https://learn.mongodb.com/learn/course/mongodb-self-managed-database-security/lesson-1-introduction-to-security/learn

Lesson 1: Introduction to Security:

• Authentication
  • Verify identity
  • Human
  • Software service
  • Methods
    • Prompting user for user and password
    • Integrated IDP
• Authorization
  • Limit what authenticated user can access
  • Roll-Based
• Auditing
  • Audit DB
  • Purpose
    • Regulatory requirement
    • Security incident

Lesson 2: Enabling Authentication for a Self-Managed MongoDB Deployment

• SCRAM - Salted Challenge Response Authentication Mechanism. Default authentication method.

vi /etc/mondod.conf

security:
  authorization: enabled

mongod --config /etc/mongod.conf

mongosh localhost:27017/admin
> use admin
> db.createUser(...)
> db.getUsers()

Lesson 3: Establishing Authorization for a Self-Managed MongoDB Deployment

https://learn.mongodb.com/learn/course/mongodb-self-managed-database-security/lesson-3-establishing-authorization-for-a-self-managed-mongodb-deployment/learn?client=customer&page=2

• builtin rolls

db.createUser(
    {
        user: "financeUser",
        pwd: passwordPrompt(),
        roles: [
            { role: "readWrite", db: "sample_analytics" },
            { role: "read", db: "sample_supplies" }
        ]
    }
)

mongosh "mongodb://analyst@localhost:27017/sample_analytics?authSource=admin"

db.revokeRolesFromUser(
    "financeUser",
    [
        {
            role: "read",
            db: "sample_supplies"
        }
    ]
)

mongosh localhost:27017/admin –username globalUserAdmin

Lesson 4: Security Auditing in MongoDB

https://learn.mongodb.com/learn/course/mongodb-self-managed-database-security/lesson-4-security-auditing-in-mongodb/learn?client=customer&page=2

• Location of the audit log file can be found in /etc/mondod.conf under auditLog section.
• tail the log. Use jq for pretty print. sudo tail /var/log/mongodb/auditLog.json | jq

Lesson 5

• Encryption at-rest (Encypted Storage Engine - native)
• Encryption during transfer (TLS)
• Encryption in-use and at-rest (CSFLE - Clinet-Side Field Level Encryptions)

Lesson 7: Enabling Network Encryption for a Self-Managed MongoDB Deployment

TLS certificate for each server in a deployment to enable TLS in a self-managed environment. You also learned that the net.tls.mode configuration file setting must be set to requireTLS to specify that a server uses and accepts only TLS-encrypted connections.

https://learn.mongodb.com/learn/course/mongodb-self-managed-database-security/lesson-7-enabling-network-encryption-for-a-self-managed-mongodb-deployment/learn?client=customer&page=2

• TLS in .pem file format
• In prod use certificate signed by CA.

Questions

• What authentication are you using?
• Why or why not to use integrated IDP?
• Is Auditing enabled by default?
• How to enable and how to monitor the auditing events?
• How often do you review the audit report?
• Who is incharge of reviewing the audit? There is no value if it is not planned to do.
• Did you enable encryption at-rest, in-transit and in-use?

MongoDB Atlas Security

Lesson 1: Intro

https://learn.mongodb.com/learn/course/mongodb-atlas-security/lesson-1-introduction-to-security/learn

• Authentication
• Authorization
• Auditing
  • to analyze security incidents
  • for complaince purpose

Lesson 2: The Atlas User Management Model

• Atlas User
  • Responsible for managing
    • Organization
    • Projects
    • Database Users
    • Billing
  • Built-In Roles
    • Project Owner
    • Project Cluster Manager
    • Project Data Access Admin
    • Project Data Access Read/Write
    • Project Data Access Read Only
• Database User
  • Authentication
    • SCRAM
    • X.509
    • AWS IAM
  • Built-In Roles
    • atlasAdmin
    • readWriteAnyDatabase
    • readAnyDatabase

Lesson 3: Atlas User Management

• Atlas provides hierarchy of access
• provides multi-factor authentication (MFA)

CLI Commands

atlas projects list --orgID 1234
atlas projects organizations invitations invite ...
atlas projects users list
atlas projects user delete ...

https://learn.mongodb.com/learn/course/mongodb-atlas-security/lesson-3-atlas-user-management/learn?client=customer&page=2

root@mongodb:~$ atlas projects invitations invite [email protected] --role GROUP_READ_ONLY
root@mongodb:~$ atlas projects invitations list
ID                         USERNAME                CREATED AT                      EXPIRES AT
65dc68f235aac70ddc602409   [email protected]   2024-02-26 10:33:22 +0000 UTC   2024-03-27 10:33:22 +0000 UTC
root@mongodb:~$ atlas organizations invitations list --orgId 1234567890
ID                         USERNAME                CREATED AT                      EXPIRES AT
65dc68f235aac70ddc602409   [email protected]   2024-02-26 10:33:22 +0000 UTC   2024-03-27 10:33:22 +0000 UTC
root@mongodb:~$ atlas project invitations delete 65dc68f235aac70ddc602409
? Are you sure you want to delete: 65dc68f235aac70ddc602409 Yes
Invitation '65dc68f235aac70ddc602409' deleted
root@mongodb:~$ atlas organizations invitations list --orgId 1234567890
ID    USERNAME   CREATED AT   EXPIRES AT

Questions

• How often do you review/audit the principle of least privilege
• Are you using the expiring access for your higher environments?