The goal of this project is to extract performance metrics (such as CPU utilisation etc.) from a Raspberry Pi and feed it into AWS IoT Core. IoT Core rules will publish events to AWS middleware which will in turn transform and store the data in DynamoDB. A simple FastAPI app will be hosted on ECS for user access to the data.

Below is the data flow description:

1. Raspberry Pi publishes CPU usage data as MQTT messages to AWS IoT Core.
2. IoT Core applies an IoT Rule to extract relevant fields and forward the message to SNS.
3. SNS (Simple Notification Service) pushes the message to an SQS Queue.
4. Lambda Function polls the SQS Queue, processes the message, and stores it in DynamoDB.
5. DynamoDB keeps a historical record of CPU metrics, allowing for future queries and analysis.
6. A FastAPI app hosted on ECS provides a REST API interface for the client to retrieve CPU usage data from DynamoDB.

1. Clone the repository onto your Local PC.

git clone https://github.com/TomAston1996/raspi-streamer.git
cd rasppi-streamer

2. Configure your AWS credentials in the aws CLI using aws configure

AWS Access Key ID [None]: YOUR_ACCESS_KEY_ID
AWS Secret Access Key [None]: YOUR_SECRET_ACCESS_KEY
Default region name [None]: YOUR_REGION (e.g., us-east-1)
Default output format [None]: json (or text, table)

3. Move into the sam directory with cd .\aws\sam
4. Compile and package code with sam build
5. Deploy your code to AWS with sam deploy
6. Navigate to AWS to set up IoT Core or this can also be done in the terminal. The following steps are required:
   • Create an IoT Thing
   • Generate certificates for AWS certificate, private key, public key and root CA and attach to the IoT Thing.
   • Create an IoT Policy and attach it to the certificate.
   • Save the IoT Core endpoint for Raspberry Pi configuration later.
   • You can test the set up later using MQTT Test Client once the Raspberry Pi has been set up.

1. Create a cognito user pool in AWS.
2. Enable the ALLOW_USER_PASSWORD_AUTH OAuth flow in your App Client.
3. Ensure the following parameters are updated in your .env file

COGNITO_USER_POOL_ID=<your_user_pool_id>
COGNITO_USER_POOL_CLIENT_ID=<your_app_client_id>
COGNITO_USER_POOL_REGION=<your_aws_region>
COGNITO_CLIENT_SECRET=<your_client_secret_for_secret_hash>

1. Build docker image locally with docker build -t my-app .
2. Tag the image docker tag my-app:latest <aws_account_id>.dkr.ecr.<your-region>.amazonaws.com/my-app:latest
3. Push the image to ECR docker push <aws_account_id>.dkr.ecr.<your-region>.amazonaws.com/my-app:latest
4. Create an ECS cluster.
5. Create a new task definition with the appropriate port mappings.
6. Create and attach an IAM role to allow your ECS task to perform CRUD operations on DynamoDB.
7. Ensure the following environment variables are set in your .env file for dev and prod environments.

DB_TABLE_NAME=<your_table_name>
DYNAMODB_ENDPOINT=<your_local_docker_endpoint>
DYNAMODB_REGION=<your_aws_region>

1. Clone the repository onto your Raspberry Pi

git clone https://github.com/TomAston1996/raspi-streamer.git
cd raspi-streamer

2. Install dependencies with UV

uv pip install -r pyproject.toml
#or
uv sync

3. Create a certificates directory in the app folder to hold IoT Core certificates and ensure to add the folder to your .gitignore
4. Place the following certificates and keys in the folder for use when authenticating with IoT Core:

7. Create a .env file in the root directory with the certificate file paths like below:

RPI_AWS_IOT_ENDPOINT=<string>-ats.iot.eu-east-1.amazonaws.com
RPI_AWS_IOT_CERTIFICATE=<path>-certificate.pem.crt
RPI_AWS_IOT_PRIVATE_KEY=<path>--private.pem.key
RPI_AWS_IOT_ROOT_CA=<path>AmazonRootCA1.pem

6. Run uv run .\raspberry_pi\src\main.py to start sending CPU metric data to the IoT Core topic from your Raspberry Pi
7. Now you can check you're data is being entered into DynamoDB.