DEV Community: Binat

Using Bash to upload files to AWS S3

Binat — Fri, 13 Dec 2024 01:01:30 +0000

This blog post describes how to create a Bash script which can be used from the command line to upload files on to AWS S3. In order to follow along, you will need:

An AWS Account
The AWS CLI Downloaded and installed on your machine
The AWS CLI configured with your credentials
Git installed on your machine

For guidance on how to download and configure AWS CLI for Windows, see my post here.

Use cases

We will satisfy the following use cases in this script:

Parse command line arguments $1, $2, to take in the file and bucket names.
Validate the input.
Provide error messages for incorrect or missing inputs.
Before uploading check if the file exists locally, if not provide feedback.
Before uploading, check if the bucket exists, if not create it, provide feedback.
Before uploading, check if the file exists in the bucket
If there is an error, capture it and display it to the user.

Create a bash file

Open a git bash prompt and navigate to the folder you will be working in.
Create a bash script using the touch command.

touch upload_to_s3.sh

Use the chmod command inside a git bash terminal to give the script executable permissions.

chmod+x upload_to_s3.sh

Open up the script in an editor. I am using Visual Studio Code.
All bash scripts start with a shebang. Add one to the top of the script.

#!/bin/bash

Parse command line arguments

In bash arguments are represented by $1 and $2. Let's set two variables, the name of the file we wish to upload, and the name of the S3 bucket we want to upload to, to $1 and $2 respectively.

!#/bin/bash

FILENAME=$1
S3BUcKETNAME=$2

echo $FILENAME
echo $S3BUCKETNAME

Run the script with the following command, remember to pass in arguments, ./upload_to_s3.sh arg1 arg2

Create a test file to upload. touch testfile.txt

Check if inputs are valid

Add an if statement to check if the two inputs are valid.

if [[ -z $FILENAME ]]
then
    echo Please provide a filename.
fi

if [[ -z $S3BUCKETNAME ]]
then
    echo Please provide a bucketname.
fi

Run the script without inputs to check the conditions work. You should get these responses:

It's a good idea to test the script with correct and incorrect inputs at each stage. This will ensure you catch any errors.

If a filename is provided, we want to check if the file exists. If it does not, return the error message.

if [[ ! -f $FILENAME ]]
then
    echo File not found!
fi

Check if the bucket exists, if not, create it.

Next we will check if the bucket already exists or not, and provide feedback depending on the result.

If the bucket does not exist and the name of the bucket is the appropriate length: create the bucket.
If the bucket exists but we do not own it: return an error message.
If the bucket does not exist but the proposed name is not within the required parameters: return an error message.
If the bucket already exists and we do own it: return a message.

# check if the bucket exists
bucketstatus=$(aws s3api head-bucket --bucket "${S3BUCKETNAME}" 2>&1 )
# search the bucketstatus result for the Not Found result, if the bucket is
# indeed not found, echo that as a response, then create the bucket

if echo $bucketstatus | grep -q "Not Found";
then
    echo Bucket not found.
    # this command creates the bucket
    aws s3 mb s3://$S3BUCKETNAME

elif echo $bucketstatus | grep -q "Forbidden";
then
    echo Bucket exists but not owned.
elif echo $bucketstatus | grep "Bad Request";
then
    echo Bucket name is less than 3 or greater than 63 characters.
else
    echo Bucket already exists.
fi

Create a function to upload the file

Making this a function will allow us to reuse this piece of code without repeating it. This will also check if the file upload was successful, and return and error message if it fails.

UPLOAD() {
    echo Uploading file.
    aws s3 cp $FILENAME s3://$S3BUCKETNAME
    # this checks if the previous command is equal to 0 or not, in bash a 0 return indicates a successful completion of command, while a non-zero return indicates an error occured.
    if [[ $? -eq 0 ]]
    then
        echo File uploaded successfully
        exit 0
    else
        echo An error occured. 
        exit 1
    fi
}

Check if the file already exists inside the bucket

Check if the file already exists inside the bucket. If it does, offer an option to replace the existing file, respond appropriately based on the user's input.

# this checks if the file exists inside the bucket
FILE_EXISTS=$(aws s3api head-object --bucket $S3BUCKETNAME --key $FILENAME)
# if the file does exist
if [[ !$FILE_EXISTS ]];
then
    echo File already exists
    echo Do you want replace the file in the bucket with this version? Y/N?
    # take in the user's answer and if it is yes, then upload the new version of the file.
    read ANSWER
    if [[ $ANSWER = Y ]]
    then
        UPLOAD $FILENAME $S3BUCKETNAME
    else
        exit
    fi
# if the file does not already exist in the bucket, then upload the file.
else
    echo Uploading
    UPLOAD $FILENAME $S3BUCKETNAME  
fi

Test the script

Run the script ./bash-uploader-cli.sh filename bucketname to test it. Use the following command to check if the file has been uploaded to the correct bucket.

aws s3 ls s3://bucketname

You can use this command to delete the bucket and files inside them after testing.

aws s3 rb --force s3://bucketname

Run the script again with the same file, or a different bucket name to ensure it responds appropriately.

You can find the script at this GitHub repository.

Thanks for reading this guide, I hope it was helpful.

Install and Configure AWS CLI on Windows

Binat — Tue, 22 Oct 2024 16:21:32 +0000

Welcome to this guide on how to install and configure the AWS command line interface. The command line interface is a powerful tool for programmatically managing and configuring AWS services and automating processes for those resources using scripts.

Download and run the installer

Download the installer here

Follow the Installation Wizard to complete the installation.

Verify Installation

Open up a command prompt and run aws --version

You should see the version number in response.

Configure Access

Firstly, we need to create a user with permissions for the AWS resources we want to access programmatically. In AWS permissions are explicitly set for users with the IAM (Identity Access Management) console.

-> Log into AWS console. Find the IAM dashboard by typing IAM into the search bar.

-> On the left hand panel, click users, and then click create user.

-> Give the user a name, and click next. Do not tick the console access button, we only want this user to have programmatic access.

-> For permissions, attach an existing policy, search for and add AmazonS3FullAccess.

-> Review and create user.

Create Access Key

-> From the users dashboard select the user we just created. Then click on create access key.

-> Select other and click next.

-> Tags are optional so leave as is and click create access key.

-> You should be on the retrieve access key page. Stay on this page for the next step. Please note the security warnings regarding access keys.

CLI configuration

Inside your command prompt run aws configure

As the command requests, copy and paste in your access key and secret access key. Type in your region, which is available in the top right corner of the AWS console.

It should look something like this

Click done on the retrieve access key page.

Test configuration

To ensure your permissions are working correctly, create an s3 bucket from the CLI with the following command. AWS S3 bucket names must be unique.

aws s3 mb s3://nat-blog-test-1

List the contents of the bucket, it is empty so nothing is listed.

aws s3 ls s3://nat-blog-test-1

Create a file to upload to the s3 bucket.

echo hello aws > testfile.txt

Use the cp (copy) command to upload it to the bucket.

aws s3 cp <path to file> s3://bucket-name

You can either list the contents of the bucket again or go to S3 on the AWS console and confirm the bucket has been created and the file has been uploaded. If the file is there, then the configuration has been successful.

Thanks for reading :)

Installing Docker On Windows

Binat — Wed, 12 Jun 2024 17:16:06 +0000

It was a little finicky for me to use Docker from the Windows Command Line.

This tutorial will ensure you have no such problems.

There are three main steps:

Install the Windows Subystem (WSL) 2 Linux distribution
Download and Install Docker Desktop for Windows
Configure Docker Desktop to access Docker from WSL

Installing WSL 2

Note: For a general tutorial see the WSL documentation here

Open the command prompt in administrator mode by right clicking and selecting run as administrator.

Run the following command wsl --install

Check which version is installed with wsl -l -v

If it is not already version 2, set WSL to the correct distribution with wsl --set-default-version 2

Download and Install Docker Desktop

Download and install Docker Desktop for Windows from here

Once installation is finished, you should be able to open the Docker Desktop application.

Configure Docker Desktop

In Docker Desktop, go to Settings, then select the resources tab. Set the terminal to be integrated with the WSL distribution.

Docker should now work from the command line.

Thank you for reading my first post!