React and Django Tutorial

Docker compose with Django 4, Celery, Redis and Postgres

2022-11-17T00:00:00+00:00

Deploying Django application that is using Celery and Redis might be challenging. Thanks to docker-compose tool we can prepare docker containers locally and make deployment much easier. I would like to show you my approach for constructing docker-compose configuration that can be reused in other web applications. I will be using Django version 4.1.3 and Celery 5.2.7. The nginx server will be used as reverse proxy for Django and to serve static files. Postgres database and Redis will be in the docker-compose.

The docker-compose architecture created in this article is presented in the diagram below:

All code from this article is available in the GitHub repository. In case of any problems or questions, please add a GitHub issue there. Good luck!

Example project

We will build a simple web application that will add two numbers in the background. We will use Django Rest Framework API viewer as our User Interface. There will be an Assignment model in the database. It will have following fields:

first_term,
second_term,
sum.

The first_term and second_term will be provided by REST API. The sum will be computed in the backgorund by Celery after object creation. Below is a screenshot from application view:

The whole project will be packed with docker-compose. It will contain following containers: ngnix, server, worker, redis, db.

Setup environment

Let’s start by configuring local environment. I’m using Python 3.8 and Ubuntu 20.04. Please create a new directory for the project or ideally a new git repository and start a new virtual environment there:

# create a new virtual environemnt
virtualenv venv 

# activate virtual environment
source venv/bin/activate

Please create a requirements.txt file with following packages:

django

djangorestframework
markdown
django-filter

celery[redis]

psycopg2-binary

We install Celery with all required packages to work with Redis. The psycopg2-binary is needed to connect Django with Postgres database. Please install required packages:

pip install -r requirements.txt

Please make sure that you have Redis available locally (link to official Redis documentation on how to install it). You can check if you have Redis installed correctly by typing:

redis-cli

You should see connection open to Redis server. We will not use Postgres locally, we will stick to SQLite. However, you can use Postgres locally (docker-compose configuration will be the same).

Please make sure that you have docker and docker-compose installed.

Start Django project

We need to bootstrap a Django project with django-admin tool (it is provided with Django package):

django-admin startproject backend

The above command will initialize an empty Django project. You should see directory structure like below:

backend/
├── backend
│   ├── asgi.py
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── manage.py

I call all my Django projects as backend. I usually use Django with React, so I have frontend and backend directories. You can name it as you want, but for similicity you can stick now with backend :)

The next step is to change working directory to the backend. We will add a new Django app:

python manage.py startapp assignments

The new app name is assignments. You should see directory structure like below:

backend/
├── assignments
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── backend
│   ├── asgi.py
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── manage.py

We can start a development server and see a launching rocket at 127.0.0.1:8000:

Assignment database model

Please update INSTALLED_APPS variable in the backend/backend/settings.py:

# rest of the code ...

INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    "rest_framework",             # DRF 
    "assignments",                # new app
]

# rest of the code ...

Let’s create a database model for Assignment objects. Please edit backend/assignments/models.py file:

from django.db import models


class Assignment(models.Model):

    first_term = models.DecimalField(
        max_digits=5, decimal_places=2, null=False, blank=False
    )

    second_term = models.DecimalField(
        max_digits=5, decimal_places=2, null=False, blank=False
    )

    # sum should be equal to first_term + second_term
    # its value will be computed in Celery
    sum = models.DecimalField(max_digits=5, decimal_places=2, null=True, blank=True)

We need to create and apply migrations:

# create migrations
python manage.py makemigrations

# apply migrations
python manage.py migrate

We will be using DRF ModelViewSet for creating a CRUD REST API for our models. We will need to add serializers.py file in the backend/assignments directory:

from rest_framework import serializers

from assignments.models import Assignment


class AssignmentSerializer(serializers.ModelSerializer):
    class Meta:
        model = Assignment
        read_only_fields = ("id", "sum")
        fields = ("id", "first_term", "second_term", "sum")

The id and sum fields are read-only. The id is automatically set in the database. The sum value will be computed in the Celery.

The backend/assignments/views.py file content:

from django.db import transaction

from rest_framework import viewsets
from rest_framework.exceptions import APIException

from assignments.models import Assignment
from assignments.serializers import AssignmentSerializer
from assignments.tasks import task_execute


class AssignmentViewSet(viewsets.ModelViewSet):

    serializer_class = AssignmentSerializer
    queryset = Assignment.objects.all()

    def perform_create(self, serializer):
        try:
            with transaction.atomic():
                # save instance
                instance = serializer.save()
                instance.save()

                # create task params
                job_params = {"db_id": instance.id}

                # submit task for background execution
                transaction.on_commit(lambda: task_execute.delay(job_params))

        except Exception as e:
            raise APIException(str(e))

The ModelViewSet is doing the job here with basic CRUD implementation - not much to code for us :) We overwrite the perform_create function to submit a background task after object creation. Please notice that object creation and submitting a background tasks are inside transaction.

We need to implement the task_execute function. Please add tasks.py file in the backend/assignments directory:

from celery import shared_task

from assignments.models import Assignment

@shared_task()
def task_execute(job_params):

    assignment = Assignment.objects.get(pk=job_params["db_id"])

    assignment.sum = assignment.first_term + assignment.second_term

    assignment.save()

The task_execute is a shared task, it will be discovered by Celery. It accepts one argument job_params that has db_id, which is id of Assignment object. We simply get the object by id, compute sum and save the object.

One more thing to do, we need to wire assignments URL endpoints. Please add a new file urls.py in the backend/assignments:

from django.urls import re_path
from rest_framework.routers import DefaultRouter

from assignments.views import AssignmentViewSet

router = DefaultRouter()
router.register(r"assignments", AssignmentViewSet, basename="assignments")
assignments_urlpatterns = router.urls

We are using DRF DefaultRouter to generate CRUD API endpoints. We need to add assignments_urlpatterns in the main urls.py. Please edit the file backend/backend/urls.py:

from django.contrib import admin
from django.urls import path
from assignments.urls import assignments_urlpatterns

urlpatterns = [
    path("admin/", admin.site.urls),
]

# add new urls
urlpatterns += assignments_urlpatterns

Configure Celery

We need to setup Celery to work with Django. Please add a new file celery.py in the backend/backend directory:

import os

from celery import Celery
from django.conf import settings

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "backend.settings")

app = Celery("backend")

app.config_from_object("django.conf:settings", namespace="CELERY")

app.autodiscover_tasks()

The next thing is to update backend/backend/__init__.py file:

from .celery import app as celery_app

__all__ = ("celery_app",)

We need to add configuration variables at the end of backend/backend/settings.py:

# rest of the code ...

# celery broker and result
CELERY_BROKER_URL = "redis://localhost:6379/0"
CELERY_RESULT_BACKEND = "redis://localhost:6379/0"

Running locally

We need two terminals open to run the project locally. Please run the Django development server in the first terminal:

python manage.py runserver

In the second terminal, please start a Celery worker:

# please run in the backend directory (the same dir as runserver command)

celery -A backend worker --loglevel=info --concurrency 1 -E

Please open A web browser at 127.0.0.1:8000/assignments and create a new Assignment object by clicking POST button. After object creation the sum should be null, like in the image below:

Please click GET in the right upper corner, to get list of all assignments. You should see the sum value computed:

It was computed in the background with Celery framework. We are using here super simple example, that is fast. In real world, the background task can take even hours to complete - depends on the project.

Create Dockerfile

It’s time to create a Dockerfile for server and worker. Server and worker will run in separate containers, but they will have the same Dockerfile. Please add new directories:

docker
docker/backend.

Please add a new file Dockerfile in docker/backend:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
FROM python:3.8.15-alpine

RUN apk update && apk add python3-dev gcc libc-dev

WORKDIR /app

RUN pip install --upgrade pip
RUN pip install gunicorn
ADD ./requirements.txt /app/
RUN pip install -r requirements.txt

ADD ./backend /app/backend
ADD ./docker /app/docker

RUN chmod +x /app/docker/backend/server-entrypoint.sh
RUN chmod +x /app/docker/backend/worker-entrypoint.sh

line 1: we are using python:3.8.15-alpine image as base.
line 3: we install gcc and python development libraries to install and build new packages.
line 5: we add app directory and set it as working directory.
line 6: copy requirements.txt file to image.
lines 8-10: update pip, install gunicorn and required packages.
lines 12-13: copy backend and docker directories to the docker image.
lines 15-16: add execution rights to entrypoint scripts.

Entrypoint scripts

I prefer to keep execution commands in separate scripts (they can be defined in docker-compose). Let’s add script that will start gunicorn server. Please add server-entrypoint.sh file in the docker/backend directory:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#!/bin/sh

until cd /app/backend
do
    echo "Waiting for server volume..."
done


until python manage.py migrate
do
    echo "Waiting for db to be ready..."
    sleep 2
done


python manage.py collectstatic --noinput

# python manage.py createsuperuser --noinput

gunicorn backend.wsgi --bind 0.0.0.0:8000 --workers 4 --threads 4

# for debug
#python manage.py runserver 0.0.0.0:8000

Explanations for server-entrypoint.sh script:

lines 3-6: we wait till /app/backend directory is ready.
lines 9-13: we wait till database is ready and run migration.
line 16: collect static files.
line 20: start gunicorn server.
line 23: I left the command to start development server. It is sometimes needed for debugging. When using development server please comment out line 20.

The worker will have separate script in worker-entrypoint.sh:

1
2
3
4
5
6
7
8
9
#!/bin/sh

until cd /app/backend
do
    echo "Waiting for server volume..."
done

# run a worker :)
celery -A backend worker --loglevel=info --concurrency 1 -E

The script just start a single worker. You can increase number of workers by changing --concurrency parameter.

Nginx configuration

We will use Nginx server to proxy requests to gunicron (Django) server and to serve static files. We will use default docker image (nging:1.23-alpine). We need to provide configuration file to customize server work. Please create a new directory docker/nginx and add there default.conf file there:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
server {
    listen 80;
    server_name _;
    server_tokens off;

    client_max_body_size 20M;

    location / {
        try_files $uri @proxy_api;
    }

    location /admin {
        try_files $uri @proxy_api;
    }

    location @proxy_api {
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass   http://server:8000;
    }

    location /django_static/ {
        autoindex on;
        alias /app/backend/django_static/;
    }
        
}

The server will formard all / and /admin requests to the @proxy_api, which is our gunicorn serving Django. The /django_static/ requests will be served from /app/backend/django_static/ - it is a path where static files will be collected by Django.

Create `docker-compose`

We have Dockerfile for server and worker ready. The Nginx configuration is waiting for requests. Let’s add a docker-compose.yml to our project:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
version: '2'

services:
    nginx:
        restart: always
        image: nginx:1.23-alpine
        ports:
            - 80:80
        volumes:
            - ./docker/nginx/default.conf:/etc/nginx/conf.d/default.conf
            - static_volume:/app/backend/django_static
    server:
        restart: unless-stopped
        build:
            context: .
            dockerfile: ./docker/backend/Dockerfile
        entrypoint: /app/docker/backend/server-entrypoint.sh
        volumes:
            - static_volume:/app/backend/django_static
        expose:
            - 8000     
        environment:
            DEBUG: "True"
            CELERY_BROKER_URL: "redis://redis:6379/0"
            CELERY_RESULT_BACKEND: "redis://redis:6379/0"
            DJANGO_DB: postgresql
            POSTGRES_HOST: db
            POSTGRES_NAME: postgres
            POSTGRES_USER: postgres
            POSTGRES_PASSWORD: postgres
            POSTGRES_PORT: 5432
    worker:
        restart: unless-stopped
        build:
            context: .
            dockerfile: ./docker/backend/Dockerfile
        entrypoint: /app/docker/backend/worker-entrypoint.sh
        volumes:
            - static_volume:/app/backend/django_static
        environment:
            DEBUG: "True"
            CELERY_BROKER_URL: "redis://redis:6379/0"
            CELERY_RESULT_BACKEND: "redis://redis:6379/0"
            DJANGO_DB: postgresql
            POSTGRES_HOST: db
            POSTGRES_NAME: postgres
            POSTGRES_USER: postgres
            POSTGRES_PASSWORD: postgres
            POSTGRES_PORT: 5432
        depends_on:
            - server
            - redis
    redis:
        restart: unless-stopped
        image: redis:7.0.5-alpine 
        expose:
            - 6379
    db:
        image: postgres:13.0-alpine
        restart: unless-stopped
        volumes:
            - postgres_data:/var/lib/postgresql/data/
        environment:
            POSTGRES_DB: postgres
            POSTGRES_USER: postgres
            POSTGRES_PASSWORD: postgres
        expose:
            - 5432
    
volumes:
    static_volume: {}
    postgres_data: {}

lines 4-11 - it is a definition of nginx container. It is using standard nginx:1.23-alpine image. We copy configuration file in line 10. We add static_volume at line 11. The same static_volume will be mounted for server container. All static files will go there. There is port 80 available outside the docker, it can be reached from external world.
lines 12-31 - it is our Django server. It is created using /docker/backend/Dockerfile (line 16). It has static_volume mounted at line line 19. It exposes port 8000 - it can be reached only internally in docker-compose. We have envrionment variables set in lines 22-31. The container will execute server-netrypoint.sh script (line 17).
lines 32-52 - it is a worker container. It doesn’t expose any ports. It depends on redis and server containers (lines 50-52).
lines 53-57 - it is a Redis container. It exposes internally port 6379 (can’t be reached from outside of docker-compose).
lines 58-68 - it is Postgres database. It exposes internally port 5432. It has environment variables in lines 63-66 to initialize database at start. All database files are stored in postgres_data volume.
lines 70-72 - definition of volumes used in the docker-compose.

We are almost ready to start using the docker-compose. We need to update Django configuration to read envrionment variables. Please edit backend/backend/settings.py file:

# new import 
import os

#
# rest of the code ...
#

# set variables
SECRET_KEY = os.environ.get(
    "SECRET_KEY", "django-insecure-6hdy-)5o6k6it_6x%s#u0#guc3(au!=v%%qb674(upu6rrht7b"
)

DEBUG = os.environ.get("DEBUG", True)

ALLOWED_HOSTS = ["127.0.0.1", "0.0.0.0"]

if os.environ.get("ALLOWED_HOSTS") is not None:
    try:
        ALLOWED_HOSTS += os.environ.get("ALLOWED_HOSTS").split(",")
    except Exception as e:
        print("Cant set ALLOWED_HOSTS, using default instead")


#
# rest of the code ...
#

# set database, it can be set to SQLite or Postgres

DB_SQLITE = "sqlite"
DB_POSTGRESQL = "postgresql"

DATABASES_ALL = {
    DB_SQLITE: {
        "ENGINE": "django.db.backends.sqlite3",
        "NAME": BASE_DIR / "db.sqlite3",
    },
    DB_POSTGRESQL: {
        "ENGINE": "django.db.backends.postgresql",
        "HOST": os.environ.get("POSTGRES_HOST", "localhost"),
        "NAME": os.environ.get("POSTGRES_NAME", "postgres"),
        "USER": os.environ.get("POSTGRES_USER", "postgres"),
        "PASSWORD": os.environ.get("POSTGRES_PASSWORD", "postgres"),
        "PORT": int(os.environ.get("POSTGRES_PORT", "5432")),
    },
}

DATABASES = {"default": DATABASES_ALL[os.environ.get("DJANGO_DB", DB_SQLITE)]}

#
# rest of the code ...
#

# set static URL address and path where to store static files
STATIC_URL = "/django_static/"
STATIC_ROOT = BASE_DIR / "django_static"

#
# rest of the code ...
#

# celery broker and result
CELERY_BROKER_URL = os.environ.get("BROKER_URL", "redis://localhost:6379/0")
CELERY_RESULT_BACKEND = os.environ.get("RESULT_BACKEND", "redis://localhost:6379/0")

The project structure at this stage should look like:

├── backend
│   ├── assignments
│   │   ├── admin.py
│   │   ├── apps.py
│   │   ├── __init__.py
│   │   ├── migrations
│   │   │   ├── 0001_initial.py
│   │   │   └── __init__.py
│   │   ├── models.py
│   │   ├── serializers.py
│   │   ├── tasks.py
│   │   ├── tests.py
│   │   ├── urls.py
│   │   └── views.py
│   ├── backend
│   │   ├── asgi.py
│   │   ├── celery.py
│   │   ├── __init__.py
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   ├── db.sqlite3
│   └── manage.py
├── docker
│   ├── backend
│   │   ├── Dockerfile
│   │   ├── server-entrypoint.sh
│   │   └── worker-entrypoint.sh
│   └── nginx
│       └── default.conf
├── docker-compose.yml
├── LICENSE
├── README.md
└── requirements.txt

`docker-compose` commands

We are ready to build our docker-compose:

# run in main project directory

sudo docker-compose build

After build you can run all containers with:

sudo docker-compose up

I’m using Ctrl+C to stop containers.

When deploying to the production I’m using:

sudo docker-compose up --build -d

The above one command is building all contaners and running them in detached mode. I can close the SSH connection to the VPS machine and the service will run. The command to stop the docker-compose:

sudo docker-compose down

One more command that is useful. It can be used to login to the running container:

sudo docker exec -it <container_name> bash

The docker-compose is running at 0.0.0.0. Just enter this address in your web broswer to play with your web application.

Summary

We created a simple web application with Django and Celery. We used Postgres database and Redis to have connectivity between Django and Celery. The project was put to docker containers thanks to docker-compose. All code is available at our GitHub repository. Please create GitHub issue there if you have problems or need help. We will try to help you!

You have a lot information from today article. You can start deploying your own application.

If you are looking for more advanced topics (for example, deploying with Let’s encrypt) please subscribe to our newsletter and check our React and Django course on how to build SaaS web application from scratch.

Dynamically update periodic tasks in Celery and Django

2022-10-17T00:00:00+00:00

Celery is a popular distributed tasks queue. It is often used with Django framework to process heavy computational tasks in the background. You can add a single task to the queue or define periodic tasks. Periodic tasks are automatically scheduled for execution based on set time. The most common approach is to define the periodic tasks before the Celery worker is started. For example, it can be daily cleaning of the database. What if we would like to define periodic tasks dynamically? Recently, I’ve been working on a web app for uptime monitoring. The service continuously monitors a server and send email notification when the server is down. In the web app, the user can add a server address to be monitored and select the interval between requests to the server. How to dynamically add periodic tasks in Celery? I want to describe my approach in this article.

Overview

The uptime monitoring idea is simple. The user defines the server address, for example https://github.com and interval. Every time period, a request is made to the server. We store in the database the information about response time and status. There can be monitored multiple servers, each at a different interval. Moreover, the user can add or delete monitored addresses at any time. Here is a sequence diagram of my approach.

Start Django project

In this article, I will create a Django application with simple monitoring functionality to show you my approach for dynamic periodic tasks in Celery and Django. All code for this article is available in public GitHub repository.

Please set up a new Python virtual environment:

virtualenv dynenv --python=python3.8

source dynenv/bin/activate

We need to install the required packages:

django for web app development,
djangorestframework, markdown, django-filter for Django Rest Framework,
celery, django-celery-beat, gevent, sqlalchemy for background tasks processing,
requests to make HTTP requests.

Please install the following packages:

pip install django djangorestframework markdown django-filter celery django-celery-beat gevent sqlalchemy requests

It is a good practice to add required packages into the requirements.txt file.

We need to initialize the Django project with the django-admin command line tool:

django-admin startproject backend

Please change the directory to the backend and create a new app:

python manage.py startup monitors

We need to setup the Django to use a newly generated monitors app. Please update the INSTALLED_APPS in the backend/settings.py file:

# the rest of the code ...

INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    # 3rd party
    "rest_framework",
    "django_celery_beat",
    # apps
    "monitors",
]

# the rest of the code ...

We added to the INSTALLED_APPS:

rest_framework - package for faster REST API development,
django_celery_beat - package that provides PeriodicTask model,
monitors - our new package.

Monitor database model

We will need two database models:

Monitor - a model for storing information about the monitored address and time interval between checks,
MonitorRequest - model to keep response time and status.

The backend/monitors/models.py file content:

from django.db import models
from django_celery_beat.models import PeriodicTask


class Monitor(models.Model):

    # monitored endpoint
    endpoint = models.CharField(max_length=1024, blank=False)

    # interval in seconds
    # enpoint will be checked every specified interval time period
    interval = models.IntegerField(blank=False)

    task = models.OneToOneField(
        PeriodicTask, null=True, blank=True, on_delete=models.SET_NULL
    )

    created_at = models.DateTimeField(auto_now_add=True)


class MonitorRequest(models.Model):

    # endpoint response time in miliseconds
    response_time = models.IntegerField(blank=False)

    response_status = models.IntegerField(blank=False)

    monitor = models.ForeignKey(Monitor, on_delete=models.CASCADE)

    created_at = models.DateTimeField(auto_now_add=True)

Please notice that Monitor has one-to-one relationship with PeriodicTask.

task = models.OneToOneField(
    PeriodicTask, null=True, blank=True, on_delete=models.SET_NULL
)

The PeriodicTask will be used to inform Celery about task execution and its frequency.

When the end-user adds a new server address for monitoring the Monitor object will be created in the database. The server address will be stored in the endpoint field. The interval field is in seconds.

Each request to the server will be saved in the database as a MonitorRequest object. We will store response time (in milliseconds) and status.

We will need to add serializers, views and URLs to have REST API available for Monitor and MonitorRequest.

Please add a new file serializers.py in the backend/monitors directory:

from rest_framework import serializers

from monitors.models import Monitor, MonitorRequest


class MonitorSerializer(serializers.ModelSerializer):
    class Meta:
        model = Monitor
        read_only_fields = ("id", "created_at")
        fields = (
            "id",
            "created_at",
            "endpoint",
            "interval",
        )


class MonitorRequestSerializer(serializers.ModelSerializer):
    monitor_endpoint = serializers.SerializerMethodField()

    def get_monitor_endpoint(self, obj):
        return obj.monitor.endpoint

    class Meta:
        model = MonitorRequest
        read_only_fields = ("id", "created_at")
        fields = (
            "id",
            "created_at",
            "response_time",
            "response_status",
            "monitor_endpoint",
        )

The serializers will just return available fields. The next step is to edit backend/monitors/views.py:

import json

from django.db import transaction
from django.shortcuts import render
from django_celery_beat.models import IntervalSchedule, PeriodicTask
from rest_framework import viewsets
from rest_framework.exceptions import APIException

from monitors.models import Monitor, MonitorRequest
from monitors.serializers import MonitorRequestSerializer, MonitorSerializer


class MonitorViewSet(viewsets.ModelViewSet):

    serializer_class = MonitorSerializer
    queryset = Monitor.objects.all()

    def perform_create(self, serializer):
        try:
            with transaction.atomic():

                instance = serializer.save()
                schedule, created = IntervalSchedule.objects.get_or_create(
                    every=instance.interval,
                    period=IntervalSchedule.SECONDS,
                )

                task = PeriodicTask.objects.create(
                    interval=schedule,
                    name=f"Monitor: {instance.endpoint}",
                    task="monitors.tasks.task_monitor",
                    kwargs=json.dumps(
                        {
                            "monitor_id": instance.id,
                        }
                    ),
                )
                instance.task = task
                instance.save()

        except Exception as e:
            raise APIException(str(e))

    def perform_destroy(self, instance):
        if instance.task is not None:
            instance.task.delete()
        return super().perform_destroy(instance)


class MonitorRequestViewSet(viewsets.ModelViewSet):

    serializer_class = MonitorRequestSerializer
    queryset = MonitorRequest.objects.all()

We have two views. The MonitorRequestViewSet derives from ModelViewSet and doesn’t overwrite any functions. It is simple CRUD for MonitorRequest objects.

The MonitorViewSet overwrites perform_create(self, serializer) and perform_destroy(self, instance). During monitor creation, a PeriodicTask instance is created. The PeriodicTask instance requires an IntervalSchedule object. The IntervalSchedule defines the time period between every task execution.

with transaction.atomic():

    instance = serializer.save()
    
    # create `IntervalSchedule` obejct
    schedule, created = IntervalSchedule.objects.get_or_create(
        every=instance.interval,
        period=IntervalSchedule.SECONDS,
    )

    # create `PeriodicTask` object
    task = PeriodicTask.objects.create(
        interval=schedule,
        name=f"Monitor: {instance.endpoint}",
        task="monitors.tasks.task_monitor",
        kwargs=json.dumps(
            {
                "monitor_id": instance.id,
            }
        ),
    )
    # save task in monitor object
    instance.task = task
    instance.save()

During the PeriodicTask object creation, we pass the task function signature monitors.tasks.task_monitor and define kwargs. We will implement the task_monitor in a moment.

The last step is to define urls.py in the backend/monitors:

from django.urls import re_path
from rest_framework.routers import DefaultRouter

from monitors.views import MonitorRequestViewSet, MonitorViewSet

router = DefaultRouter()
router.register(r"monitors", MonitorViewSet)
router.register(r"requests", MonitorRequestViewSet)
monitors_urlpatterns = router.urls

We used DefaultRouter from Django Rest Framework. A basic viewer for REST API for monitors and requests will be generated by DRF.

We need to add monitors_urlpatterns in the backend/backend/urls.py to make them available in the Django application.

from django.contrib import admin
from django.urls import path

from monitors.urls import monitors_urlpatterns

urlpatterns = [
    path("admin/", admin.site.urls),
]

urlpatterns += monitors_urlpatterns

Please make migrations and apply them:

# please run in the backend directory

python manage.py makemigrations

python manage.py migrate

Celery configuration

We need to configure the Celery framework. Please add a new file celery.py in the backend/backend directory:

import os
import sys

from celery import Celery

CURRENT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, CURRENT_DIR)
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "backend.settings")

app = Celery("backend")

app.config_from_object("django.conf:settings", namespace="CELERY")

app.autodiscover_tasks()

In the backend/backend/settings.py please add the Celery configuration:

# the rest of the code ...

# celery broker and results in sqlite
CELERY_BROKER_URL = "sqla+sqlite:///celery.sqlite"
CELERY_RESULT_BACKEND = "db+sqlite:///celery.sqlite"

We will use SQLite as a broker and results backend.

It is an example project showing how to use PeriodicTask, thus broker and results backend performance is out of the scope of this article.

We have the Celery configuration completed.

Background task

Please add the tasks.py file in the backend/monitors directory:

from datetime import datetime, timedelta
from decimal import Decimal

import requests
from celery import shared_task

from monitors.models import Monitor, MonitorRequest


@shared_task(bind=True)
def task_monitor(self, monitor_id):

    try:

        monitor = Monitor.objects.get(pk=monitor_id)

        response = requests.get(monitor.endpoint, timeout=60)

        MonitorRequest.objects.create(
            response_time=int(response.elapsed.total_seconds() * 1000),
            response_status=response.status_code,
            monitor=monitor,
        )

    except Exception as e:
        print(str(e), type(e))

The task_monitor(self, monitor_id) function has @shared_task decorator. It accepts monitor_id as an argument - it is passed when PeriodicTask is created as a kwargs field.

The task_monitor sends a GET request to the monitored server and saves response time and status. It is a simplified version of task_monitor used in my uptime monitoring service.

Run Django and Celery

We are ready to play with our web application. You will need three terminals. Please start the Django development server in the first one:

python manage.py runserver

In the second terminal, please start the Celery worker:

celery -A backend worker --loglevel=info -P gevent --concurrency 1 -E

In the third terminal, please start Celery beat:

celery -A backend beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler --max-interval 10

Celery beat service uses DatabaseScheduler from django-celery-beat package. The beat service checks scheduled tasks from the database. Tasks defined with PeriodicTask are persistent. Tasks will be available even after the Celery worker and beat restart.

Please open your (favorite) web browser and enter the address http://127.0.0.1:8000. You should see the REST API viewer automatically generated by DRF:

If you have problems or need help, please create a GitHub issue. We will try to help you! You won’t be alone.

Please open the monitors API at http://127.0.0.1:8000/monitors/1/. The list of monitors should be empty; let’s add the first monitor. Please fill out the form and click the POST button.

Please wait some time to have some results, and you should see in the Celery beat terminal:

[2022-10-17 12:05:59,887: INFO/MainProcess] Scheduler: Sending due task Monitor: https://github.com (monitors.tasks.task_monitor)
[2022-10-17 12:06:59,887: INFO/MainProcess] Scheduler: Sending due task Monitor: https://github.com (monitors.tasks.task_monitor)
[2022-10-17 12:07:59,909: INFO/MainProcess] Scheduler: Sending due task Monitor: https://github.com (monitors.tasks.task_monitor)
[2022-10-17 12:08:59,909: INFO/MainProcess] Scheduler: Sending due task Monitor: https://github.com (monitors.tasks.task_monitor)
[2022-10-17 12:09:59,914: INFO/MainProcess] Scheduler: Sending due task Monitor: https://github.com (monitors.tasks.task_monitor)

Example output for Celery worker terminal:

[2022-10-17 12:08:00,644: INFO/MainProcess] Task monitors.tasks.task_monitor[e8ea92d4-e834-4bdf-8f4a-af6ee0601b55] received
[2022-10-17 12:08:01,304: INFO/MainProcess] Task monitors.tasks.task_monitor[e8ea92d4-e834-4bdf-8f4a-af6ee0601b55] succeeded in 0.65664959300193s: None
[2022-10-17 12:09:00,101: INFO/MainProcess] Task monitors.tasks.task_monitor[8c597c74-4917-4f3e-894f-a6ae117fc0f3] received
[2022-10-17 12:09:01,909: INFO/MainProcess] Task monitors.tasks.task_monitor[8c597c74-4917-4f3e-894f-a6ae117fc0f3] succeeded in 1.8063794959998631s: None
[2022-10-17 12:10:00,604: INFO/MainProcess] Task monitors.tasks.task_monitor[739f9227-4b63-409d-9d72-57720c2da5f0] received
[2022-10-17 12:10:00,873: INFO/MainProcess] Task monitors.tasks.task_monitor[739f9227-4b63-409d-9d72-57720c2da5f0] succeeded in 0.2661438309987716s: None

Please open requests API at http://127.0.0.1:8000/requests/ at the beginnig you will see only an empty list:

After some time, it will be filled with requests data:

You can stop the monitoring task by deleting the monitor. Please open the http://127.0.0.1:8000/monitors/1/ (where 1 is monitor ID) and click the Delete button.

You should see that no more requests are produced. The monitor and an associated PeriodicTask object have been removed. The code for this article is available in the GitHub repository.

To check periodic tasks, you can open the Django Admin Panel at http://127.0.0.1:8000/admin.

Summary

Celery is a great task queue. You can dynamically create or delete periodic tasks with the django-celery-beat package without Celery restart. What is more, the PeriodicTask object allows you to dynamically change the interval values and pause the task (it was not described in this article). These features were very helpful for me while implementing the uptime monitoring service.

Email Verification View in React for DRF backend

2022-10-06T00:00:00+00:00

After registration, a user will get an email with a verification link. Why is there such a procedure? We would like to have users with valid emails only to be able to contact them. (If we don’t care about emails, we could use the username for login)

When a user clicks the verification link, then the website for email verification will be opened. The token from the link will be used to send a POST request to the server. The verification status will be displayed in the website.

This article is part of SaaSitive paid course.

Email Verification View

Below is the email with a verification link:

Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: [example.com] Please Confirm Your E-mail Address
From: webmaster@localhost
To: [email protected]
Date: Thu, 25 Aug 2022 13:53:15 -0000
Message-ID: <166143559536.50748.16793504971950475831@komp2>

Hello from example.com!

You're receiving this e-mail because user pepek has given your e-mail address to register an account on example.com.

To confirm this is correct, go to http://127.0.0.1:8000/verify-email/MQ:1oRDIJ:EliIs1rQ4mAr_jch8LuC-T98gIoKBhirwSMlygiINkU/

Thank you for using example.com!
example.com
-------------------------------------------------------------------------------

The verification link:

http://127.0.0.1:8000/verify-email/MQ:1oRDIJ:EliIs1rQ4mAr_jch8LuC-T98gIoKBhirwSMlygiINkU/
/-----domain--------/--view-url--/--------------token-----------------------------------/

We will change the domain later, let’s take the link and paste it into the web browser with domain localhost:3000:

http://localhost:3000/verify-email/MQ:1oRDIJ:EliIs1rQ4mAr_jch8LuC-T98gIoKBhirwSMlygiINkU/

We should see an empty page.

Let’s add a new view VerifyEmailView.tsx in the frontend/src/views directory:

export default function VerifyEmailView() {
  return (
    <div>
      <div className="container">
        <div className="row">
          <div className="col-md-10 offset-md-1">
            <h1>Verify Email</h1>
          </div>
        </div>
      </div>
    </div>
  );
}

This view will be displayed for the verification link. Let’s add a new Route in frontend/src/AppRoutes.txs file:

import VerifyEmailView from "./views/VerifyEmailView";

// the rest of the code ...

<Route path="/" element={<WebLayout />}>
  
  <Route path="/verify-email/:key" element={<VerifyEmailView />} />
  
  {/* the rest of the code ... */}
  
</Route>

// the rest of the code ...

Please note that the path is "/verify-email/:key". The :key is for obtaining a token from the URL address.

When we enter the verification link we will see only Verify Email header. We need to perform a POST request with the token in the view. We will need a new variable in the auth store to keep information about email verification status.

Update `authSlice`

Let’s update the frontend/src/slices/authSlice.ts:

// the rest of the code ...

const initialState = {
    usernameRegisterError: "",
    emailRegisterError: "",
    password1RegisterError: "",
    verifyEmailStatus: "unknown", // new variable in the store
};

const authSlice = createSlice({
    name: 'auth',
    initialState,
    reducers: {
        setUsernameRegisterError(state, action: PayloadAction<string>) {
            state.usernameRegisterError = action.payload;
        },
        setEmailRegisterError(state, action: PayloadAction<string>) {
            state.emailRegisterError = action.payload;
        },
        setPassword1RegisterError(state, action: PayloadAction<string>) {
            state.password1RegisterError = action.payload;
        },
        //
        // new function to set variable value
        //
        setVerifyEmailStatus(state, action: PayloadAction<string>) {
            state.verifyEmailStatus = action.payload;
        },
    },
});

export default authSlice.reducer;

export const {
    setUsernameRegisterError,
    setEmailRegisterError,
    setPassword1RegisterError,
    //
    setVerifyEmailStatus, // export new function
    //
} = authSlice.actions;

// ...

// new getter function
export const getVerifyEmailStatus = (state: RootState) => state.auth.verifyEmailStatus;


// the rest of the code ...

Updates in the authSlice:

we added a new variable in the initialState with an unknown value,
we added a new function setVerifyEmailStatus to set status,
we added a new function getVerifyEmailStatus to get email verification status from the store.

The store is ready. The next step is to write a function to do POST request with a verification token.

POST request with a verification token

Please add a new function at the end of the authSlice.ts file:

export const verifyEmail =
    (key: string) =>
        async (dispatch: TypedDispatch) => {
            try {
                // set status to started 
                dispatch(setVerifyEmailStatus("started"));

                // send POST request
                const url = "/api/auth/register/verify-email/";
                await axios.post(url, { key });

                // set verify email status to ok
                dispatch(setVerifyEmailStatus("ok"));
            } catch (error) {
                // set status to error
                dispatch(setVerifyEmailStatus("error"));
            }
        };

The verifyEmail function gets the key as the argument. The key value is a verification token. It is sent to the backend to the /api/auth/register/verify-email/ endpoint. The verifyEmailStatus is set to "ok" for successful response and "error" otherwise.

Update `VerifyEmailView.tsx`

We will use the verifyEmailStatus variable to display proper messages for our users.

The frontend/src/views/VerifyEmailView.tsx:

import { useEffect } from "react";
import { useSelector } from "react-redux";
import { useNavigate, useParams } from "react-router-dom";
import { getVerifyEmailStatus, verifyEmail } from "../slices/authSlice";
import { useAppDispatch } from "../store";

export default function VerifyEmailView() {
  let { key } = useParams(); // get key (token) from URL
  let navigate = useNavigate(); // create navigate variable 
  const dispatch = useAppDispatch();
  const emailVerifyStatus = useSelector(getVerifyEmailStatus); //get emailVerifyStatus

  // after view load send POST request
  useEffect(() => {
    if (key) {
      dispatch(verifyEmail(key));
    }
  }, [dispatch, key]);

  return (
    <div>
      <div className="container">
        <div className="row">
          <div className="col-md-10 offset-md-1">
            <h1>Verify Email</h1>

            {(emailVerifyStatus === "unknown" ||
              emailVerifyStatus === "error") && (
              <p>
                We can't verify your email. Please try to register again or
                contact us by email [email protected]
              </p>
            )}
            {emailVerifyStatus === "started" && (
              <p>Email verification started, please wait a while ...</p>
            )}
            {emailVerifyStatus === "ok" && (
              <p>
                Successfull email verification🎉 Please login to start
                monitoring!
                <br />
                <button
                  className="btn btn-lg btn-primary my-2"
                  onClick={() => navigate("/login")}
                >
                  Login
                </button>
              </p>
            )}
          </div>
        </div>
      </div>
    </div>
  );
}

We will use useParams from react-router-dom to get the token value from the URL address.

let { key } = useParams(); // get key (token) from URL

We will get the emailVerifyStatus from the store:

const emailVerifyStatus = useSelector(getVerifyEmailStatus); //get emailVerifyStatus

The POST request to the server will be sent after the VerifyEmailView component is loaded. We will use the useEffect React’s hook for this:

useEffect(() => {
    if (key) {
      dispatch(verifyEmail(key));
    }
  }, [dispatch, key]);

The useEffect hook is called only once after loading the component. It dispatches the verifyEmail(key).

We use the emailVerifyStatus value to display proper information for the user.

{(emailVerifyStatus === "unknown" ||
    emailVerifyStatus === "error") && (
    <p>
    We can't verify your email. Please try to register again or
    contact us by email [email protected]
    </p>
)}
{emailVerifyStatus === "started" && (
    <p>Email verification started, please wait a while ...</p>
)}
{emailVerifyStatus === "ok" && (
    <p>
    Successfull email verification🎉 Please login to start
    monitoring!
    <br />
    <button
        className="btn btn-lg btn-primary my-2"
        onClick={() => navigate("/login")}
    >
        Login
    </button>
    </p>
)}

The login button is displayed after successful verification. The user will be redirected to log in view after clicking it.

Summary

We created a view for email verification. It displays information about verification status for the user and makes a POST request with a token value from the URL. We will add toasts and BlockUI in the next post (in the course).

This article is part of SaaSitive paid course.

Django Rest Framework Register New User with Email Verification

2022-10-06T00:00:00+00:00

In this step, we will look closely at how the new user registration process looks and write a unit test for registration.

This article is part of SaaSitive paid course.

Below is a sequence diagram of what the registration looks like:

We will focus only on the backend part. The registration is available at /api/auth/register/ endpoint. You can manually register a new user by filling out the form on http://127.0.0.1:8000/api/auth/register/. After the successful user creation, the server will send an email to the user. Right now, the email sending is not configured. The server will print the email in the terminal where the development server is running. There is a verification link sent in the verification email. We need to provide a custom URL address for it. You can read more about it in dj-rest-auth documentation.

Let’s edit the backend/accounts/urls.py file:

from django.conf.urls import include
from django.urls import path, re_path
from django.views.generic.base import TemplateView

accounts_urlpatterns = [
    path("api/auth/", include("dj_rest_auth.urls")),
    path("api/auth/register/", include("dj_rest_auth.registration.urls")),
    # path to set verify email in the frontend
    # fronted will do POST request to server with key
    # this is empty view, just to make reverse works
    re_path(
        r"^verify-email/(?P<key>[-:\w]+)/$",
        TemplateView.as_view(),
        name="account_confirm_email",
    ),
]

I’ve added a line:

re_path(
    r"^verify-email/(?P<key>[-:\w]+)/$",
    TemplateView.as_view(),
    name="account_confirm_email",
),

It will allow the dj-rest-auth package to construct a verification link in the email in the form like:

server-address/verify-email/some-key-here

Please remember to add imports with re_path and TemplateView.

The address /verify-email/ will be available in the frontend. The verification view will be displayed when the user clicks the verification link. This view will parse the verification key from the URL address and send it in the POST request to the server (endpoint /api/auth/register/verify-email/).

Unit test

Let’s create a unit test that will check how registration is working. I like writing unit tests for the backend because it helps me create frontend code. From unit tests, I see precisely how the backend works, what status codes are returned, and how response data looks.

Please add the following code in the backend/accounts/tests.py:

from django.core import mail
from rest_framework import status
from rest_framework.test import APITestCase

class AccountsTestCase(APITestCase):

    register_url = "/api/auth/register/"
    verify_email_url = "/api/auth/register/verify-email/"
    login_url = "/api/auth/login/"

    def test_register(self):

        # register data
        data = {
            "email": "[email protected]",
            "password1": "verysecret",
            "password2": "verysecret",
        }
        # send POST request to "/api/auth/register/"
        response = self.client.post(self.register_url, data)
        # check the response status and data
        self.assertEqual(response.status_code, status.HTTP_201_CREATED)
        self.assertEqual(response.json()["detail"], "Verification e-mail sent.")

Let’s run the code and then dive deep into the details. Please execute the following command to run tests:

python manage.py test accounts

You should get the output like below:

Found 1 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.162s

OK
Destroying test database for alias 'default'...

It is important to remember that the django framework for each unit test starts with an empty database.

What has just happened? We created a class AccountsTestCase that derives from APITestCase (DRF testing class). In this test class, there are defined three attributes:

register_url = "/api/auth/register/"
verify_email_url = "/api/auth/register/verify-email/"
login_url = "/api/auth/login/"

There are endpoints that will be used in the test. I don’t like to use reverse() method for obtaining the URL address in the django framework because for frontend implementation, I will need complete addresses.

Our test class has one unit test: test_register().

Every unit test needs to start with test_ string.

The test_register() method:

    def test_register(self):

        # register data
        data = {
            "email": "[email protected]",
            "password1": "verysecret",
            "password2": "verysecret",
        }
        # send POST request to "/api/auth/register/"
        response = self.client.post(self.register_url, data)
        # check the response status and data
        self.assertEqual(response.status_code, status.HTTP_201_CREATED)
        self.assertEqual(response.json()["detail"], "Verification e-mail sent.")

What is the unit test doing:

It prepares test data for registration.
It sends the POST request with test data to register a new user.
It checks the server response. The server responds with HTTP 201 CREATED status and the message Verification e-mail sent..

You can print the server response in the test by adding at the end:

print(response.json())

and running tests again: python manage.py test accounts.

Let’s try to log in before the email verification is done. Please update the test_register() method:

    def test_register(self):

        # register data
        data = {
            "email": "[email protected]",
            "password1": "verysecret",
            "password2": "verysecret",
        }
        # send POST request to "/api/auth/register/"
        response = self.client.post(self.register_url, data)
        # check the response status and data
        self.assertEqual(response.status_code, status.HTTP_201_CREATED)
        self.assertEqual(response.json()["detail"], "Verification e-mail sent.")
        
        # try to login - should fail, because email is not verified
        login_data = {
            "email": data["email"],
            "password": data["password1"],
        }
        response = self.client.post(self.login_url, login_data)
        self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
        self.assertTrue(
            "E-mail is not verified." in response.json()["non_field_errors"]
        )

Test email

The next step is to verify the email. The django tests run with a built-in email service. We can access emails that will be sent during tests with django.core.mail package.

Please update the test_register():

    def test_register(self):

        # register data
        data = {
            "email": "[email protected]",
            "password1": "verysecret",
            "password2": "verysecret",
        }
        # send POST request to "/api/auth/register/"
        response = self.client.post(self.register_url, data)
        # check the response status and data
        self.assertEqual(response.status_code, status.HTTP_201_CREATED)
        self.assertEqual(response.json()["detail"], "Verification e-mail sent.")
        
        # try to login - should fail, because email is not verified
        login_data = {
            "email": data["email"],
            "password": data["password1"],
        }
        response = self.client.post(self.login_url, login_data)
        self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
        self.assertTrue(
            "E-mail is not verified." in response.json()["non_field_errors"]
        )

        # expected one email to be send
        # parse email to get token
        self.assertEqual(len(mail.outbox), 1)
        email_lines = mail.outbox[0].body.splitlines()
        activation_line = [l for l in email_lines if "verify-email" in l][0]
        activation_link = activation_line.split("go to ")[1]
        activation_key = activation_link.split("/")[4]

The mail.outbox returns the list of sent emails. There should be 1 email sent:

self.assertEqual(len(mail.outbox), 1)

To access the email body, we will use the variable mail.outbox[0].body. The following lines parse the email to get the verification link:

email_lines = mail.outbox[0].body.splitlines()
activation_line = [l for l in email_lines if "verify-email" in l][0]
activation_link = activation_line.split("go to ")[1]
activation_key = activation_link.split("/")[4]

Please add the following code add the end of the test to check what the email looks like:

print(mail.outbox[0].body)
print(activation_key)

After running tests (python manage.py test accounts) you should get output like:

Found 1 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
Hello from example.com!

You're receiving this e-mail because user user2 has given your e-mail address to register an account on example.com.

To confirm this is correct, go to http://testserver/verify-email/MQ:1oEB8H:L5qGQGcPdxS8C9VoJNgBK07mIBvvaT73AYCfpaFHipc/

Thank you for using example.com!
example.com
MQ:1oEB8H:L5qGQGcPdxS8C9VoJNgBK07mIBvvaT73AYCfpaFHipc
.
----------------------------------------------------------------------
Ran 1 test in 0.296s

OK
Destroying test database for alias 'default'...

Don’t worry that there is example.com in the email. We will set the proper domain after production deployment.

The MQ:1oEB8H:L5qGQGcPdxS8C9VoJNgBK07mIBvvaT73AYCfpaFHipc is a verification key. Let’s send it in a POST request:

# please add at the end of test_register() method
        response = self.client.post(self.verify_email_url, {"key": activation_key})
        self.assertEqual(response.status_code, status.HTTP_200_OK)
        self.assertEqual(response.json()["detail"], "ok")

After successful verification, let’s try to log in:

# please add at the end of test_register() method
        response = self.client.post(self.login_url, login_data)
        self.assertEqual(response.status_code, status.HTTP_200_OK)
        self.assertTrue("key" in response.json())

The full unit test from file backend/accounts/tests.py:

from django.core import mail
from rest_framework import status
from rest_framework.test import APITestCase

class AccountsTestCase(APITestCase):

    register_url = "/api/auth/register/"
    verify_email_url = "/api/auth/register/verify-email/"
    login_url = "/api/auth/login/"

    def test_register(self):

        # register data
        data = {
            "email": "[email protected]",
            "password1": "verysecret",
            "password2": "verysecret",
        }
        # send POST request to "/api/auth/register/"
        response = self.client.post(self.register_url, data)
        # check the response status and data
        self.assertEqual(response.status_code, status.HTTP_201_CREATED)
        self.assertEqual(response.json()["detail"], "Verification e-mail sent.")
        
        # try to login - should fail, because email is not verified
        login_data = {
            "email": data["email"],
            "password": data["password1"],
        }
        response = self.client.post(self.login_url, login_data)
        self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
        self.assertTrue(
            "E-mail is not verified." in response.json()["non_field_errors"]
        )

        # expected one email to be send
        # parse email to get token
        self.assertEqual(len(mail.outbox), 1)
        email_lines = mail.outbox[0].body.splitlines()
        activation_line = [l for l in email_lines if "verify-email" in l][0]
        activation_link = activation_line.split("go to ")[1]
        activation_key = activation_link.split("/")[4]

        response = self.client.post(self.verify_email_url, {"key": activation_key})
        self.assertEqual(response.status_code, status.HTTP_200_OK)
        self.assertEqual(response.json()["detail"], "ok")

        # lets login after verification to get token key
        response = self.client.post(self.login_url, login_data)
        self.assertEqual(response.status_code, status.HTTP_200_OK)
        self.assertTrue("key" in response.json())

Please run the test; it should execute without errors.

Summary

Great! We have the registration process ready. The REST API is waiting for the frontend. Let’s wait a while with frontend creation. We will need to extend a User model to keep information about subscription. In the next step of the course, I will show you how to add UserProfile.

This article is part of SaaSitive paid course.

Django Rest Framework Email Verification

2020-12-23T00:00:00+00:00

Email verification is an important part of the SaaS application. We will contact the user by email in many cases: for a password reset, announcement of new features, or for sending the invoice. During registration, a user provides the email address. We need to check if the email belongs to the user and that there are no typos/errors in it. This can be easily done by automatically sending the verification email with an activation link. Such a link contains the unique token assigned to the user. After opening the activation link in the web browser, the request is sent to the web application (Django Rest Framework). The web server compares the token from the activation URL with the token stored in the database. If they are the same, the email address is verified.

In this tutorial you will learn:

how to set email as a mandatory field during registration,
how to set login with email and password,
how to send a verification email,
how to re-send a verification email,
write test cases for email verification flow,
the Django Rest Framework and Djoser packages will be used.

The next posts will describe the user interface and production email setup. Fill out the form to be notified about future posts.

This article is a part of a series of articles on how to build SaaS from scratch with Django and React. I will use the code from the previous article: Docker-Compose for Django and React with Nginx reverse-proxy and Let’s encrypt certificate.

Email required in registration

The email field is available in User model in Django, but it is not mandatory. To set email as the required field and use it for login (email + password), we need to do the below steps.

Please update the models.py file in accounts application. We will update the email field in the database to make it required.

# /backend/server/apps/accounts/models.py
from django.contrib.auth.models import User

User._meta.get_field('email')._unique = True
User._meta.get_field('email').blank = False
User._meta.get_field('email').null = False

We need to update Django application settings.py:

# backend/server/server/settings.py

# ...
# configure Djoser
DJOSER = {
    "USER_ID_FIELD": "username",
    "LOGIN_FIELD": "email"
}

# ...

With the above changes, the email is required during the registration and login. Remember to make migrations and apply them to the database (we changed the database models).

Activation email sending in the Django

The next step will be to configure the activation email sending.

We need to update settings.py file:

# backend/server/server/settings.py

# ...
# configure Djoser
DJOSER = {
    "USER_ID_FIELD": "username",
    "LOGIN_FIELD": "email",
    "SEND_ACTIVATION_EMAIL": True,
    "ACTIVATION_URL": "activate/{uid}/{token}",
    'SERIALIZERS': {
        'token_create': 'apps.accounts.serializers.CustomTokenCreateSerializer',
    },
}

EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
SITE_NAME = "SaaSitive"

# ...

We added in the Djoser configuration:

enabled send activation email by "SEND_ACTIVATION_EMAIL": True;
setup the activation email link URL "ACTIVATION_URL": "activate/{uid}/{token}". Please notice that there are uid and token in the activation URL. Both are required to activate the account. The link is pointing to the URL address in our fronted (we will add React route for it). There is no such endpoint on the backend;
defined the custom token create serializer token_create - it will be needed to allow the user to log in even with an unverified email.

We set the EMAIL_BACKEND as the Django django.core.mail.backends.console.EmailBackend. This backend is simply displaying all emails in the console. It is useful only for development. (How to set up Django email backend for production will be described in future posts.) The nice thing about Django is the switchable email backend. For testing purposes Django will automatically switch the email backend to django.core.mail.backends.locmem.EmailBackend and give us easy access to the email outbox.

We also set the SITE_NAME = "SaaSitive". The site name variable will be used in the activation email. You can set here your application name.

An example of an activation email:

Subject: Account activation on SaaSitive

Body:

You're receiving this email because you need to finish the activation process on SaaSitive.

Please go to the following page to activate account:
http://testserver/activate/MQ/afc0m5-e3336fd5fa02874a588c9085d9bdf881

Thanks for using our site!

The SaaSitive Team

You can overwrite the text in the activation email by setting a different email template in the Djoser configuration.

Custom create a token serializer

The last thing is to add the custom create token serializer. Please add serializers.py file in the backend/server/apps/accounts directory. Our custom token will simply overwrite the TokenCreateSerializer from the Djoser package and will allow to obtain the auth_token for the user without an activated account.

# backend/server/apps/accounts/serializers.py

from django.contrib.auth import authenticate, get_user_model
from djoser.conf import settings
from djoser.serializers import TokenCreateSerializer

User = get_user_model()

class CustomTokenCreateSerializer(TokenCreateSerializer):

    def validate(self, attrs):
        password = attrs.get("password")
        params = {settings.LOGIN_FIELD: attrs.get(settings.LOGIN_FIELD)}
        self.user = authenticate(
            request=self.context.get("request"), **params, password=password
        )
        if not self.user:
            self.user = User.objects.filter(**params).first()
            if self.user and not self.user.check_password(password):
                self.fail("invalid_credentials")
        # We changed only below line
        if self.user: # and self.user.is_active: 
            return attrs
        self.fail("invalid_credentials")

Testing email verification flow

The email verification flow in our application is described below. Let’s write test cases to check them. We can also check this manually in the web browser by using DRF browsable API. Writing tests will take some time but have many advantages:

we have tests for user registration flow, so we are confident that all works well,
when developing a new feature, writing tests first might be faster than repeated manual tests.

In our tests, we will use the following endpoints:

endpoint to register the new user: /api/v1/users/ with POST request,
endpoint to verify the email: /api/v1/users/activation with POST request,
endpoint to resend verification email: /api/v1/users/resend_activation/ with POST request,
endpoint to login: /api/v1/token/login/ with POST request,
endpoint to get user details: /api/v1/users/ with GET request.

Let’s define the empty test in tests.py file in the backend/server/apps/accounts/ directory:

# backend/server/apps/accounts/tests.py

from django.core import mail
from rest_framework import status
from rest_framework.test import APITestCase

class EmailVerificationTest(APITestCase):

    def test_register_with_email_verification(self):
        pass

We are using APITestCase from the Django Rest Framework (see the docs for more details). To run this empty test, let’s run it in the backend/server directory:

./manage.py test apps

The expected output:

Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK
Destroying test database for alias 'default'...

The first test will check registration with email activation before login.

# backend/server/apps/accounts/tests.py

from django.core import mail
from rest_framework import status
from rest_framework.test import APITestCase

class EmailVerificationTest(APITestCase):

    # endpoints needed
    register_url = "/api/v1/users/"
    activate_url = "/api/v1/users/activation/"
    login_url = "/api/v1/token/login/"
    user_details_url = "/api/v1/users/"
    # user infofmation
    user_data = {
        "email": "[email protected]", 
        "username": "test_user", 
        "password": "verysecret"
    }
    login_data = {
        "email": "[email protected]", 
        "password": "verysecret"
    }

    def test_register_with_email_verification(self):
        # register the new user
        response = self.client.post(self.register_url, self.user_data, format="json")
        # expected response 
        self.assertEqual(response.status_code, status.HTTP_201_CREATED)
        # expected one email to be send
        self.assertEqual(len(mail.outbox), 1)
        
        # parse email to get uid and token
        email_lines = mail.outbox[0].body.splitlines()
        # you can print email to check it
        # print(mail.outbox[0].subject)
        # print(mail.outbox[0].body)
        activation_link = [l for l in email_lines if "/activate/" in l][0]
        uid, token = activation_link.split("/")[-2:]
        
        # verify email
        data = {"uid": uid, "token": token}
        response = self.client.post(self.activate_url, data, format="json")
        self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)

        # login to get the authentication token
        response = self.client.post(self.login_url, self.login_data, format="json")
        self.assertTrue("auth_token" in response.json())
        token = response.json()["auth_token"]

        # set token in the header
        self.client.credentials(HTTP_AUTHORIZATION='Token ' + token)
        # get user details
        response = self.client.get(self.user_details_url, format="json")
        self.assertEqual(response.status_code, status.HTTP_200_OK)
        self.assertEqual(len(response.json()), 1)
        self.assertEqual(response.json()[0]["email"], self.user_data["email"])
        self.assertEqual(response.json()[0]["username"], self.user_data["username"])

The steps of the first test:

create a new user,
parse the verification email,
activate the account by sending token and uid from the verification email,
login to get auth_token,
get user details.

When you run the test the expected output is below:

> ./manage.py test apps 

Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.330s

OK
Destroying test database for alias 'default'...

Let’s add the next test with login without email verification and re-sending the verification email.

# backend/server/apps/accounts/tests.py

# ... the rest of EmailVerificationTest code ...

    def test_register_resend_verification(self):
        # register the new user
        response = self.client.post(self.register_url, self.user_data, format="json")
        # expected response 
        self.assertEqual(response.status_code, status.HTTP_201_CREATED)
        # expected one email to be send
        self.assertEqual(len(mail.outbox), 1)

        # login to get the authentication token
        response = self.client.post(self.login_url, self.login_data, format="json")
        self.assertTrue("auth_token" in response.json())
        token = response.json()["auth_token"]

        # set token in the header
        self.client.credentials(HTTP_AUTHORIZATION='Token ' + token)
        # try to get user details
        response = self.client.get(self.user_details_url, format="json")
        self.assertEqual(response.status_code, status.HTTP_401_UNAUTHORIZED)

        # clear the auth_token in header
        self.client.credentials()
        # resend the verification email
        data = {"email": self.user_data["email"]}
        response = self.client.post(self.resend_verification_url, data, format="json")
        self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)
        
        # there should be two emails in the outbox
        self.assertEqual(len(mail.outbox), 2)

        # parse the last email
        email_lines = mail.outbox[1].body.splitlines()
        activation_link = [l for l in email_lines if "/activate/" in l][0]
        uid, token = activation_link.split("/")[-2:]
        
        # verify the email
        data = {"uid": uid, "token": token}
        response = self.client.post(self.activate_url, data, format="json")
        # email verified
        self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)

The steps of the second test:

create a new user,
log in the get auth_token,
try to get user details and expect for HTTP_401_UNAUTHORIZED,
resend the verification email,
parse the last email to the uid and token,
activate the account by verifying the email.

I will additionally add two tests:

to test the response for the wrong email address during re-sending the verification,
to test activation with wrong uid and token

# backend/server/apps/accounts/tests.py

# ... the rest of EmailVerificationTest code ...

    def test_resend_verification_wrong_email(self):
        # register the new user
        response = self.client.post(self.register_url, self.user_data, format="json")
        # expected response 
        self.assertEqual(response.status_code, status.HTTP_201_CREATED)
        
        # resend the verification email but with WRONG email
        data = {"email": self.user_data["email"]+"_this_email_is_wrong"}
        response = self.client.post(self.resend_verification_url, data, format="json")
        self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
        

    def test_activate_with_wrong_uid_token(self):
        # register the new user
        response = self.client.post(self.register_url, self.user_data, format="json")
        # expected response 
        self.assertEqual(response.status_code, status.HTTP_201_CREATED)
        
        # verify the email with wrong data
        data = {"uid": "wrong-uid", "token": "wrong-token"}
        response = self.client.post(self.activate_url, data, format="json")
        self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
    

The expected output:

> ./manage.py test apps 

Creating test database for alias 'default'...
System check identified no issues (0 silenced).
....
----------------------------------------------------------------------
Ran 4 tests in 0.963s

OK
Destroying test database for alias 'default'...

Commit your changes

Remember to commit all the code changes and add a new file:

git add apps/accounts/serializers.py 
git commit -am "DRF email verification"
git push

Summary

We wrote the backend for email verification with Django Rest Framework and Djoser.
The new functionality has been tested, so we are confident that it works as expected.
In the next post, we will write the user interface with React for email verification functionality.

The code for this tutorial is available at Github repository.

GDPR Overview

2020-12-21T00:00:00+00:00

Equip your SaaS with legal bases so that the use of your website is safe for users and you. If you search for different sites, the Privacy Policy and Terms of Service differ from each other. The same with the cookies. Each piece of information has varying meanings for the website, and it concludes crucial statements. There’s no unique example for all websites because of the specification, the idea of webites, purpose, kind and other features, collected data.

Usually, websites are available around the world, including Europe. Thus, you have to take care of your European users’ personal data. It is required by the law of the European Union, to be exact the Regulation (EU) 2016/679 (General Data Protection Regulation - GDPR). But it could seem challenging to differentiate such users by giving some of the privileges.

Here’s the GDPR’s source https://eur-lex.europa.eu/eli/reg/2016/679/oj

All these documents, amenities are nothing but SaaS and User Agreement where SaaS declares:

to support some service described in Terms of Service and treat Users personal data with respect to GDPR what is mentioned in Privacy Policy,

and User state that:

he’s going to use your website, service according to Terms of service, and give necessary personal data, but you must use them as intended and provide adequate protection.

There’s no direct dictation to implement the Personal Data Protection Policy (mentioned in art. 24 GDPR). Still, it would be transparent to collect all rules and ways of handling data in one document. It will include a defined, very narrow catalog of mandatory documents: a register of processing activities, a record of categories of processing activities, documentation of personal data breaches, documentation related to the conduct of a data protection impact assessment, and prior consultations with the supervisory authority. After this document is ready, you can start creating your idea and figuring out how your project will affect personal data. The important stuff is to figure out the best security measures to keep your users safe.

If you already prepared the Personal Data Protection Policy, you know that the next step (after creation) is to assess the risk for your users’ data by filling Data Protection Impact Assessment and Privacy by Design. Not every project needs to be checked by Data Protection Impact Assessment, each EU member state defines the catalog itself, but we recommend to do it for each project. Then you can start building your website.

Check the providers with whom you intend to conclude contracts, their privacy policy, the method of data protection, where the data is stored, whether in the EU or outside. Start to build Privacy Policy and Terms of Service. Check if all your documents and your SaaS contain key provisions and ensure that the principles of respecting users’ personal data are respected.

You can check the source https://ec.europa.eu/justice/smedataprotect/index_en.htm where you can find some necessary information about GDPR for small businesses.

The Personal Data Protection Policy

It will be the primary document that includes all GDPR requirements. Is it necessary? The provisions of the GDPR require the implementation of appropriate technical and organizational measures so that the data processing is compliant and can be demonstrated. It’s only an internal document which helps you to keep in order all requirements and duties. It is supposed to contain regulations about: processing of personal data, risk analysis, registers regarding the processing of personal data and records, proceedings in the event of a breach of personal data protection, the security of information systems.

This document can be the basis for your workers to know how to deal with personal data. In the beginning, give some basic definitions to explain key terminology. List the rights of people, describe the role of Personal Data Protection Officer, IT System Administrator, other persons authorized to process data, and Administrator’s responsibilities. In any case, the crucial role and responsibility lie with the Administrator. Indicate the purposes and legal grounds for data processing, as (art. 6 GDPR):

consent of the data subject,
the necessity to perform the contract to which the data subject is a party,
the need to fulfill the legal obligation incumbent on the Administrator,
the legitimate interest of the Administrator.

List how you collect data (direct contact by person, subscription, conclusion of a contract, hire an employee). What about the period of data processing and data retention? That’s important information you need to review. You can’t keep personal data too long. That’s the time limitations rule – you are obliged to store data in a form that allows the identification of persons to whom they relate, no longer than necessary to achieve the purpose of processing. Establish a time after which you remove all those data.

Are you transferring data outside the European Economic Area (it includes processors who do it on your premise)? Write about it. The European Union has strict rules about taking care of personal data, but it’s not a priority everywhere.

Under GDPR regulation, you should provide a register of personal data processing activities and a register of categories of processing activities if you:

hire more than 250 employees;
process data in a way that involves the risk of violating the rights or freedoms of data subjects;
process data more often than sporadically;
process information covering special categories of personal data;
process personal data relating to criminal convictions and offenses.

(Picture from https://ec.europa.eu/justice/smedataprotect/index_en.htm)

GDPR also requires to document or register personal data breaches.

Finally, let’s go to System security. Describe how you protect data by using technical measures. Are you doing a backup? Do you have your own servers or use those in the cloud, using an antivirus program? Do you anonymize data? Reviewing the program architecture is key to organizing and checking data security.

Here is some crucial thing which you should be aware of. You cannot describe the measures you implemented in every detail unless it could be a confidential attachment available only for a few people if needed!

Risk Analysis

Privacy by Design and Data Protection Impact Assessment. It’s mandatory to know where your users’ data goes to know the risk of processing. While creating a new project, you should implement presumption privacy by default. Every new project needs Privacy by Design evaluation. Some also require risk assessment by filling DPIA ( Data Protection Impact Assessment). It all helps to find vulnerabilities to support the best protection for data. DPIA is obligatory (which results from the GDPR and the European guidelines of the working party) whenever the processing is:

Evaluation or scoring.
Automated decision-making with legal or similarly significant effect.
Systematic monitoring.
Sensitive data or data of a highly personal nature.
Data processed on a large scale.
Matching or combining datasets.
Data concerning vulnerable data subjects.
Innovative use or applying new technological or organizational solutions.
Preventing data subjects from exercising a right or using a service or contract.

GDPR includes terminology without supporting definition. For example, high risk or large scale is relevant to describe.

High risk – a situation when DPIA is needed (in some cases, it must occur with other items to be mandatory for DPIA). EU member states will publish lists of processing types that require a DPIA in their jurisdiction, e.g., large-scale profiling, invisible processing, tracking, genetic data, biometrics, innovative technology, denial of service, data matching, targeting of children or other vulnerable individuals.

Innovative technology is processing involving innovation (as artificial intelligence, machine learning, and deep learning, market research involving neuro-measurement, internet of things applications) or the novel application of existing technologies. It’s all new developments in technological knowledge, including new ways of collecting and using data.

Large scale GDPR doesn’t define this but to decide if it’s large-scale processing or not, you should consider: the number of individuals concerned with the volume of data, the variety of data, the duration of the processing, and the geographical extent of the processing.

Vulnerable individuals - in this meaning, people who, because of their circumstances, are not aware of their data processing implications, or they can’t freely consent or reject that processing (e.g., children).

Invisible processing is a situation when you obtained data not directly from the individual, and you don’t in-form that person about processing his data. Processing is invisible because that person is unaware that you are collecting and using their personal data, even if you publish a privacy notice on your website. A DPIA is re-quired where this processing is combined with any of the criteria from the European guidelines.

All contained in this article information is just an essence form how GDPR permeates every professional activi-ty. Although the intention to introduce the GDPR resulted from the need to protect natural persons, they im-pose additional entrepreneurs’ obligations. Anyone offering services should meet the basic requirements and take the user’s will as a benchmark.

Note from the author: Even though it contains a lot of practical information about legal stuff, you cannot treat it as legal advice. Each case is different and requires an individual approach. The purpose of this article is to help you to understand the essence of its subject.

GDPR Meaning - The General Data Protection Regulation

2020-12-19T00:00:00+00:00

Earn users trust by respecting and securing their data. Since 2018 the European Union (EU) has imposed a new obligation on entrepreneurs to protect customers personal data. A duty for one another is a right, a tool to defend one’s rights. Each user should have the right to decide on providing information about himself, resign from using his data, delete his account. It’s a strong right that should be obeyed by everyone. Why? Because of substantial penalties, up to 20 mln euro! Be well prepared with Privacy Policy, Terms of Service, and GDPR requirements.

The General Data Protection Regulation is an EU law regulation on data protection and privacy in the European Union and the European Economic Area (EEA). It also regulates how to proceed when personal data are transferred outside the EU and EEA areas.

Personal data is any information relating to an identified or identifiable person, also known as the data subject. Example of personal data:

first name and last name,
address,
ID /Passport number,
Income,
cultural features,
IP address.

That law gives a strong right for people to manage their data. It protects us, physical people, before the abuse. It imposes an obligation on companies to ensure everyone the right to: get information about his data, obtain access to personal data, correct his data, erased data - to be forgotten, object to the processing of personal data for marketing purposes, request the restriction of the processing of personal data, data portability - receive personal data in a machine-readable format and send it to another controller, request that decisions based on automated processing concerning or significantly affecting the user and based on his data are made by natural persons, not only by computers and the right to be notified - if data has been a breach (a person should be informed within 72 hours of first having become aware of the violation). How to ensure that these rights are respected for SaaS users?

On the EU GDPR webpage (https://gdpr.eu/what-is-gdpr/), we can find this phrase:

“(GDPR) is the toughest privacy and security law in the world.”

And we fully agree. It is limited and requires small businesses to be fully committed to protecting user rights. On the other hand, there is a risk that profiling and making automated decisions on our behalf may limit our freedom.

The GDPR provides broad protection of the individual’s rights while imposing several new obligations on entrepreneurs. Until now, the entities, confirming their due diligence, have applied for ISO certificates in the field of data security. The current regulations impose the obligation to ensure the processed data’s safety - on all entities dealing with personal data. One of these obligations is appointing a Personal Data Protection Inspector (DPO) in certain cases and keeping detailed documentation describing data processing.

GDPR relates to companies based in the European Union, citizens and people lived in the EU, and those who want to offer services for them. If you’re outside of the EU, but your customers are from the EEA (The EEA covers more countries than the EU itself), it refers to you as well.

GDPR applies to every company, both sole proprietorships and companies - operating in the European Union, which processes personal data. It does not matter the nationality of the persons whose data is processed, where the processing takes place, or where the servers are located.

Examples of entities covered by the GDPR:

an entrepreneur with a headquarters outside the EU, but performing activities on its territory,
entities that offer their services to clients outside the Union but have their offices in the Union,
companies processing data via cloud computing - it does not matter where the servers are located,
an entrepreneur who does not have organizational units in the EU but offers EU citizens goods and services (e.g., an online store).

The GDPR may apply to entities (controllers and processors) that do not have an organizational unit in the EU, also in the scope of the obligation to appoint a personal data protection officer, when the processing activities carried out by them are related to:

offering goods or services to such data subjects in the EU, whether or not they are required to pay; or
monitoring their behavior as far as this behavior occurs in the EU.

Data Protection Officer, do you need it?

When your company is established in the EU, you are obliged to appoint a DPO if:

You are a public authority or body and have appointed a DPO (except if you are a court acting in our judicial capacity).
Your main activity is processing operations requiring regular and systematic monitoring of data subject on a large scale by their nature, scope, or purposes (e.g., you’re Google ;-)).

Here it is crucial to explain “large-scale” meaning. The GDPR doesn’t define the concept of “large-scale” data processing. However, it is recommended to consider the following factors in order to determine whether large-scale processing takes place: the number of data subjects (a specific number or percentage of a particular group of the population), the scope of personal data processed, the period for which the data is processed, geographical scope of personal data processing. Thus, describing the process with this term is quite relative. The concept of “regular and systematic monitoring” of data subjects is also not defined in the GDPR. However, “monitoring of data subjects’ behavior” is mentioned in Recital 2415 and includes all online tracking and profiling forms, including for advertising purposes behavioral.
Your main activity is the large-scale processing of special categories of personal data and personal data related to criminal convictions and offenses.

Now, what does it mean “special categories of personal data”? Data of a special category can be included revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, trade union membership, and the processing of genetic data, biometric data allowing the identification of a natural person.

As mentioned above, if your company is not an EU resident and your clients are EU residents, you should (as the data controller) have a representative registered in the EU. According to its competence, the DPO appointment should be notified to the relevant Member State’s supervisory authority.

The DPO should:

be aware of national and European data protection laws;
be familiar with the practices in the field of personal data protection;
have business and industry knowledge regarding the administrator’s activities;
know the data processing processes,
know the information systems and security measures used by the controller and its data protection needs;
demonstrate knowledge of the entity’s administrative procedures and operations.

The inspector is, therefore, to play a crucial role in supporting the “data protection culture” and help in the implementation of the necessary elements of the GDPR, i.e.:

rules for the processing of personal data;
the rights of data subjects;
data protection by design and data protection by default;
keep a register of processing activities;
processing security requirements;
reporting violations.

DPO suppose to be easily accessible as a point of contact for employees, individuals, and Information Commissioner’s Officer (ICO). His contact details should be published on your website. If you are not obliged to appoint a DPO, you can designate a person who will track the newest trends, UE recommendations and keep a hand on your company’s personal data security.

Note from the author:

Even though it contains a lot of practical information about legal stuff, you cannot treat it as legal advice. Each case is different and requires an individual approach. The purpose of this article is to help you to understand the essence of its subject.

Visual Identity for SaaS

2020-11-12T00:00:00+00:00

Are you going to start your business as a SaaS? Have you got an idea and now you want to settle up your website, but you’re not a graphic designer? Start here by creating your Visual Identity Book.

Why it is so useful? It will help you to smooth your website, make your SaaS recognizable, will be handy for preparing presentations, advertising on the Internet, or social media e.g. Facebook, Instagram, Twitter. Would it be comfortable to keep all branding graphics in one place? For sure, but is it not a waste of time?

Too many questions above. Now let’s make some answers. Visual identification is the basic tool for creating the company image on the market. You may view thousands of websites and some stayed in your mind. That means the graphic designers done a good job. There are some rules by which you can create a consistent look for a brand. Basic visual identity consists of features like: logo, colors, font, icons, and pictures and all these elements give a consistent look to the website. Collect them in one place called Visual Identity Book. This book will save you a lot of time, and your co-workers will know how to properly engage in building a website and strengthen your brand recognition. All this will ensure that the prepared content (and tutorials, advertisements, graphics, charts) will inspire trust, positive perception of users, and the message will be uniform.

You can achieve effective visual identification on your website with these steps:

Creative work

Some creative work to do. Try to imagine your service with a miniature or with one tool, thing, object. Scrawl a few pages to find that one you will identify (including an abstract spot) or… look for some inspirations on the Internet. The best way is to prepare your logo in vector so you can use it in many materials, advertisement, website. You just can scale it, vector graphics need less space for storage. It will make website loading faster. Search similar projects, compare or try logo generators e.g. (but for most of them you have to pay). An easy way is to use a free, unusual font and give the name of your business. An example is saasitive.com, where we used two different fonts:

Colors

Think about colors. For sure You will do some labels, templates, differ parts of your website, icons, buttons. Use a color palette to chose a few of them and to be sure it won’t be too radical or too mismatched. If you do not feel confident in choosing colors, use ready-made color palettes (take a look at Coolors where you can inspire). The aesthetic selection of matched colors to the website and its content can create a good impression and a sense of polished appearance. When mixing colors, designers use the 60-30-10 rule, which means that 60% fills up main color, 30% the second and 10% the third color, used mainly to emphasize and highlight important elements. This is only a suggestion, but it can prove useful if you want a colorful page while still being readable and clear.

For our website we chose:

Psychology of the colors has a big meaning. The most preferable color by woman and man is blue while disliked color is brown and orange. Men prefer saturated, darker colors, they also accept gray tones. Women like lighter and less saturated colors, gray is not their favorite shade.

Take a look at the graph below it shows the meaning of colors. As you can notice each color gives a variety of impression and can be addressed to a different group of people (from https://www.firstdesign.eu/the-meaning-of-colors-in-design ):

Font

Find a font that suits your website. Pick at most 3 fonts. You can use only one font in different sizes and types (bold, italic). If you need special signs check it out whether font provides them. Font color is also important. Usually, we use black but for example, headings can be in one of your chosen colors suitable for the logo.

Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Ww Xx Yy Zz 1234567890!@#$%^&*()<><>?|\,./

Icons

Choose icons that can describe features of your SAAS. The best is to use a wide set of uniform style icon where you have a lot of variety to choose from. If you choose colored icons remember about the colors of your brand. You can check Uxwing , aiconica, Iconfinder, or Flaticon. Remember that each icon has its license to use so read it to know how to use it for free. Sometimes it’s better to pay (the cost is not that big) to have a clear conscience and use a whole set of icons. Some platforms don’t require attribution. Examples of icons monochromatic:

Pictures

Pictures used on the website should correspond with the content on the website and icons (if you chose monochromatic icons pictures can bring some pleasant impression). There’s a lot of stock of photos you can choose from. You have to find out what will arouse the curiosity of the user and will be a sneak peek of your article, tutorial, essence feature you describe, or action you encourage users to do. You can check Undraw where you can get nice vector pics and change color so it will suit your website. Example from SaaSitive:

The devil is in the details.

Prepare a favicon. Don’t forget about it, it’s a little stamp on your website placed in the browser. Use the first letter from your brand name, logo, or a sign which will be memorable. That’s a little detail but necessary! If your users have a few pages opened in your browser they will find your webpage with this tiny sign. It needs to be square 16x16 pixels, but you can start from 64x64, and after it’s ready then resize it.

You can just try to use generators like Formito.

Want to share your website on social media? You better prepare a metaimage. It consists of an image with a title, a short description, and the domain name. It should have 1200x630 px. You can check how your graphic will look at Metatags. Let your users share articles, tutorials from your website by giving a link with an expressive picture. It could looks like this:

To be accurate prepare footer to e-mails which will be sent automatically. Use the chosen font and logo so users remember your visual identity. Include the address of your website, e-mail address, and other important information.

All adverts, banners and graphics you can do with well prepared elements above.

Summary

Preparing visual identification is a process in which you create a clever brand image. Using all hints above will help your SAAS to stand out of the competition and give a consistency in communication. Besides your website will give strong impression and build trust of the users and attract them.

Leading an internet business mainly based on advertisement, social media, GitHub, StackOverflow, and others. To advertise you need ready visual identification. Logo starts to be recognizable if you use it to promote, advertise your SaaS. First of all, you need to gather this information in one place to find it quickly. Create a folder and put all items listed below in it.

The minimum what your Visual Identity Book needs:

company symbols - logo, logotype, decorative symbols (if it’s possible to try to make it also in black-white colors, in many combinations vertical, horizontal, and in few different sizes also keep the most used logo in different formats e.g. pdf, jpg, SVG);
colors (describe colors in many options: RGB, CMYK and Hex Code, HTML - to find it use e.g. Rapidtables );
company typography - font type and size;
icons or pictures you will dress features, functions, or descriptions, or the content.

The rest you will compile with elements upstream.

You can check Visual Identity Book of SaaSitive in our repository. Let us know if it was helpful.

How to generate Django Secret Key?

2020-11-10T00:00:00+00:00

Have you ever pushed to the repository a Django project with SECRET_KEY? Ups, it happens to me very often. Don’t worry. This can be easily fixed.

The SECRET_KEY is used in Django for cryptographic signing. It is used to generate tokens and hashes. If somebody will have your SECRET_KEY he can recreate your tokens.

Storing SECRET_KEY in the repository code is not secure. It should be removed from the code and loaded from environment variables (or some configuration). You can use for example python-decouple to separate configuration variables from the code.

Once SECRET_KEY was committed into the code repository, it needs to be generated again. Luckily, we can use Django get_random_secret_key() function to generate new SECRET_KEY.

from django.core.management.utils import get_random_secret_key
# print new random secret key
print(get_random_secret_key())

This code can be run in the terminal as a command:

python -c 'from django.core.management.utils import get_random_secret_key; \
            print(get_random_secret_key())'

If you have new SECRET_KEY then you can use python-decouple.

# in your settings.py file 

from decouple import config

SECRET_KEY = config("SECRET_KEY")

The SECRET_KEY can be then set as environment variable or can be saved in .env file (which is not tracked in the code repository).

Docker-Compose for Django and React with Nginx reverse-proxy and Let’s encrypt certificate

2020-10-30T00:00:00+00:00

The most exciting moment of the web application development is a deployment. Your app is going live! It can also be nerve-wracking moment. Unfortunately. There are many options, many variables and configurations. It is easy to miss something … In this article, I will show you how to pack Django and React application into containers and deploy them with docker-compose. The presented approach can be reused on any Cloud Provider (AWS, DigitalOcean, Linode, GCP, Heroku) - you just need a Virtual Private Server (VPS).

In this article:

We will create two docker-compose configuration files. One for development (easier version) and one for production (with SSL certificate from Let’s Encrypt).
The React static files will be served by nginx.
The Django static files (from admin and DRF browsable API) will be served by nginx.
The nginx will be reverse-proxy to the Django server (gunicorn).
In the production, we will add certbot to renew the certificate. To issue a certificate we will use a bash script. You need to have a domain to issue the certificate .

We will be using code from the previous article: CRUD in Django Rest Framework and React (code with tag v6).

Create Dockerfile for Django and Nginx

Let’s start by adding a new directory docker in the main directory of the project. Your project structure should look like below:

.
├── backend
├── docker
├── frontend
├── LICENSE
└── README.md

In the docker directory please add backend directory with two files:

docker/backend/Dockerfile - it will define how to build Django container,
docker/backend/wsgi-entrypoint.sh - it will be entrypoint for Django container. It will apply migrations, collect static files and run WSGI server with gunicorn.

Django Dockerfile

The Dockerfile to build Django container:

# docker/backend/Dockerfile

FROM python:3.8.3-alpine

WORKDIR /app
ADD ./backend/requirements.txt /app/backend/


RUN pip install --upgrade pip
RUN pip install gunicorn
RUN pip install -r backend/requirements.txt

ADD ./docker /app/docker
ADD ./backend /app/backend

In this Dockerfile the python:3.8.3-alpine is used as base image. Let’s decode this image name and tag. It contains python in version 3.8.3, running in the Linux distribution Alpine which is lightweight. The purpose of using lightweight Linux is to have small image which will result in faster builds. You can of course use different base image. For purpose of this tutorial the python:3.8.3-alpine is sufficient.

We create /app directory in the container and copy /backend/requirements.txt. Then, gunicorn and all needed packages are installed. The backend source code is copied at the end of the container build. It is on purpose. It makes building faster. When you change something in the code (without changing the requirements.txt file), only the last line of the Dockerfile will be executed and the rest will be read from cache (of course if available).

The Dockerfile is not executing any command, we will use wsgi-entrypoint.sh script for this purpose.

Docker entrypoint

Please add a new file docker/backend/wsgi-entrypoint.sh and add execute rights to the file (it is a bash script):

# add execute rights
chmod +x docker/backend/wsgi-entrypoint.sh

The docker/backend/wsgi-entrypoint.sh file:

#!/bin/sh

until cd /app/backend/server
do
    echo "Waiting for server volume..."
done

until ./manage.py migrate
do
    echo "Waiting for db to be ready..."
    sleep 2
done

./manage.py collectstatic --noinput

gunicorn server.wsgi --bind 0.0.0.0:8000 --workers 4 --threads 4

#####################################################################################
# Options to DEBUG Django server
# Optional commands to replace abouve gunicorn command

# Option 1:
# run gunicorn with debug log level
# gunicorn server.wsgi --bind 0.0.0.0:8000 --workers 1 --threads 1 --log-level debug

# Option 2:
# run development server
# DEBUG=True ./manage.py runserver 0.0.0.0:8000

The above script is waiting till serever volume and database is ready. It runs migration on database and collect static files. It runs gunicorn server with IP address 0.0.0.0 and (docker default IP) and port 8000.

The gunicorn is running 4 workers with 4 threads each. You can set different numbers here. It depends on your machine. There are some heuristic rules that set number of workers as 4 * CPU cores (see the gunicorn docs). If you are deploying application to machine with 1 CPU core, then you can use 4 workers. (Remember, it is just heuristic not an exact rule). Then you can specify how many threads will be running in each worker. In our case, there are 4 threads (gunicorn docs). This means that we can process 4 * 4 = 16 concurrent requests.

1st Note: We are using SQLite for our development - it doesn’t support concurrency. We will need to replace SQLite with advanced database engine like PostgreSQL. I will replace database at the end of this tutorial to make final deployment. I think it is good to deploy application. We will practice deployment and code updates in the production. That’s why we are here.

2nd Note: One more thing, I added other options to run Django server in wsgi-entrypoint.sh which are commmented. Why? You might need them for debugging. The first option add logging to the console with debug level and run gunicorn with one worker and single thread. The second option runs Django development server with DEBUG=True. It needs setting environment variable and load it in settings.py with python-decouple (for example), I will write about this in the future post.

Django static files

Before going further, let’s update STATIC_URL variable in the backend/server/server/settings.py:

# backend/server/server/settings.py

MEDIA_URL = '/media/'
STATIC_URL = '/django_static/' 
STATIC_ROOT = BASE_DIR / 'django_static'

We overwrite STATIC_URL with a new value. All static files will be served with /django_static/ in the URL. We add two new variables:

MEDIA_URL - it is URL for serving files uploaded to Django application, we will use it in future posts.
STATIC_ROOT - it is a directory where static files from Django application will be stored after running collectstatic command. The nginx will point to this path.

Nginx Dockerfile and Configuration

Please add nginx directory in the docker directory. There will be added three files:

docker/nginx/Dockerfile - the instructions how to build nginx container image,
docker/nginx/development/default.conf - the configuration file for nginx server used in the development,
docker/nginx/production/default.conf - the configuration file for the production.

The Dockerfile for nginx container will use two stage build.

At first stage, we build the React static files.
At second stage, the static files are copied and nginx server starts. We copy React static files into /usr/share/nginx/html directory.

The docker/nginx/Dockerfile file:

# The first stage
# Build React static files
FROM node:13.12.0-alpine as build

WORKDIR /app/frontend
COPY ./frontend/package.json ./
COPY ./frontend/package-lock.json ./
RUN npm ci --silent
COPY ./frontend/ ./
RUN npm run build

# The second stage
# Copy React static files and start nginx
FROM nginx:stable-alpine
COPY --from=build /app/frontend/build /usr/share/nginx/html
CMD ["nginx", "-g", "daemon off;"]

At the second stage, we copy only the final product of the first stage: static files. The node_modules (they can take a lot of space) with dependency files are not in the final image. Such two stage-build makes the size of the final image smaller.

Let’s add nginx configuration for development version of docker-compose (without SSL certificate):

server {
    listen 80;
    server_name _;
    server_tokens off;
    client_max_body_size 20M;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
        try_files $uri $uri/ /index.html;
    }

    location /api {
        try_files $uri @proxy_api;
    }
    location /admin {
        try_files $uri @proxy_api;
    }

    location @proxy_api {
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Url-Scheme $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass   http://backend:8000;
    }

    location /django_static/ {
        autoindex on;
        alias /app/backend/server/django_static/;
    }
}

What do you feel when you see nginx configuration file?

No troubles! The nginx is here to help us. It will do what we will ask in the configuration.

server {
    listen 80;
    server_name _;
    server_tokens off;
    client_max_body_size 20M;

At the begining of the configuration we define a server (server keyword). It will listen on port 80 (it is a default HTTP port). There is no name assigned to the server (server_name _;). We switch off option to show server version on error pages (server_token off; see docs). The last setting is to set maximum request size (see docs) (client_max_body_size 20M). It means that requests larger than 20MB will result in error with HTTP 413 (Request Entity Too Large).

We have five location blocks (location docs) in the config which specify configuration for each URL (routing for requests).

The first location / defines what to do if the request comes from / (main domain or IP). The index.html file from /usr/share/nginx/html will be served as a response. It is our React static index.html file.

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
        try_files $uri $uri/ /index.html;
    }

The second and third locations location /api and location /admin redirect requests to location @proxy_api. It is a reverse-proxy:

    location /api {
        try_files $uri @proxy_api;
    }
    location /admin {
        try_files $uri @proxy_api;
    }

What we have in fourth location @proxy_api? It is our Django application served with gunicorn. The nginx forwards all requests with /api and /admin in the URL to http://backend:8000 which is the address of the Django application in the docker-compose:

    location @proxy_api {
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass   http://backend:8000;
    }

The last location /django_static/ serves static files from the Django application (created with collectstatic command):

    location /django_static/ {
        autoindex on;
        alias /app/backend/server/django_static/;
    }

That’s all. We have configuration file for nginx server.

Docker-compose for Django, Nginx and React

Our docker-compose will run two containers:

the nginx container that will run nginx server on port 80 (default HTTP port). The nginx container has two volumes mounted, one with django_static and one with configuration file (from docker/nginx/development directory). The nginx container depends on backend container, which means that nginx will start after backend (Django app).
the backend will run container with Django application. It runs the /app/docker/backend/wsgi-entrypoint.sh script that starts the gunicorn server. The backend has django_static volume with static files.

We will save our docker-compose in the docker-compose-dev.yml file (in main project directory):

version: '2'

services:
    nginx: 
        restart: unless-stopped
        build:
            context: .
            dockerfile: ./docker/nginx/Dockerfile
        ports:
            - 80:80
        volumes:
            - static_volume:/app/backend/server/django_static
            - ./docker/nginx/development:/etc/nginx/conf.d
        depends_on: 
            - backend
    backend:
        restart: unless-stopped
        build:
            context: .
            dockerfile: ./docker/backend/Dockerfile
        volumes:
            
        entrypoint: /app/docker/backend/wsgi-entrypoint.sh
        volumes:
            - static_volume:/app/backend/server/django_static
        expose:
            - 8000        

volumes:
    static_volume: {}

There are few useful commands for dealing with docker-compose.

Build containers

docker-compose -f docker-compose-dev.yml build

Please notice that we are using -f docker-compose-dev.yml - it is to point custom yml file. By default docker-compose is reading yml configuration file from docker-compose.yml (we keep default name for production setting).

Run containers

docker-compose -f docker-compose-dev.yml up

Stop containers

docker-compose -f docker-compose-dev.yml down

You can also use Ctrl+C to stop containers if not running in the background.

Build and run containers

docker-compose -f docker-compose-dev.yml up --build

Please build containers and run them with last command. You should see logs from backend and nginx in the terminal.

OK, let’s go into browser and go to http://0.0.0.0 address. You should see the Home view. Let’s try to login:

After login attempt you should see toast with error: “Network Error” and some errors in the console.

We need to update our code. We need to set axios configuration to point to correct address of server in the frontend. In the backend, we need to add 0.0.0.0 to ALLOWED_HOSTS (in settings.py file).

Update frontend/src/App.js code.

// frontend/src/App.js
// remove this line
//axios.defaults.baseURL = "http://localhost:8000";

// new code
if (window.location.origin === "http://localhost:3000") {
  axios.defaults.baseURL = "http://127.0.0.1:8000";
} else {
  axios.defaults.baseURL = window.location.origin;
}

In the frontend, we set a logic to point the correct address of the server:

in the case of development the axios will call server at http://127.0.0.1:8000,
in the production, the axios will call server at the same location origin as frontend. Frontend will call in fact nginx and it will redirect request to the Django.

In the backend/server/server/settings.py we need to update ALLOWED_HOSTS variable:

# in backend/server/server/settings.py

# ...
ALLOWED_HOSTS = ['0.0.0.0']
# ...

After changes we need stop containers (Ctrl+C) and then rebuild docker containers and run them again.

docker-compose -f docker-compose-dev.yml up --build

Now, if you try to login you should be successful! What is more, you can use this docker-compose and run it on VPS in the cloud and make it available to others. BUT, it will be unsecure! It is using only HTTP. We need to add SSL certificate to make it secure. For this we will create next docker-compose configuration file for production. We also add production nginx configuration file.

Production docker-compose

Let’s add directory production in the docker/nginx/ with default.conf file. It will be a configuration of nginx server to be used in the production:

server {
    listen 80;
    server_name boilerplate.saasitive.com;
    server_tokens off;

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl;
    server_name boilerplate.saasitive.com;
    server_tokens off;

    ssl_certificate /etc/letsencrypt/live/boilerplate.saasitive.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/boilerplate.saasitive.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    client_max_body_size 20M;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
        try_files $uri $uri/ /index.html;
    }

    location /api {
        try_files $uri @proxy_api;
    }
    location /admin {
        try_files $uri @proxy_api;
    }

    location @proxy_api {
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Url-Scheme $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass   http://backend:8000;
    }

    location /django_static/ {
        autoindex on;
        alias /app/backend/server/django_static/;
    }
}

It is similar to the development configuration. It has two server definitions. The first one:

server {
    listen 80;
    server_name boilerplate.saasitive.com;
    server_tokens off;

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

The above definition listen on port 80 and redirects all requests to HTTPS (return 301 https://$host$request_uri;). In this part there is also a server_name filled to boilerplate.saasitive.com. I’m planning to run the application from this tutorial on boilerplate.saasitive.com domain. You should use here your own domain.. The location /.well-known/acme-challenge/ is used by cerbot for issuing the certificate.

Let’s look closer at second server in the configuration:

server {
    listen 443 ssl;
    server_name boilerplate.saasitive.com;
    server_tokens off;

    ssl_certificate /etc/letsencrypt/live/boilerplate.saasitive.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/boilerplate.saasitive.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    client_max_body_size 20M;

    #...

It is listening on 443 port which is default for HTTPS. The server_name is set to domain. There are added paths with certificate. You should rename them to your domain:

    ssl_certificate /etc/letsencrypt/live/---your-domain.com---/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/---your-domain.com---/privkey.pem;
    

The rest of the configuration is very similar to development configuration, except that we need to set additional headers:

    location @proxy_api {
        proxy_set_header X-Forwarded-Proto https; # additional 
        proxy_set_header X-Url-Scheme $scheme; # additional 
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # additional
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass   http://backend:8000;
    }

Let’s add production docker-compose. The docker-compose.yml file in the main project directory:

version: '2'

services:
    nginx: 
        restart: unless-stopped
        build:
            context: .
            dockerfile: ./docker/nginx/Dockerfile
        ports:
            - 80:80
            - 443:443
        volumes:
            - static_volume:/app/backend/server/django_static
            - ./docker/nginx/production:/etc/nginx/conf.d
            - ./docker/nginx/certbot/conf:/etc/letsencrypt
            - ./docker/nginx/certbot/www:/var/www/certbot
        depends_on: 
            - backend
    certbot:
        image: certbot/certbot
        restart: unless-stopped
        volumes:
            - ./docker/nginx/certbot/conf:/etc/letsencrypt
            - ./docker/nginx/certbot/www:/var/www/certbot
        entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"       
    backend:
        restart: unless-stopped
        build:
            context: .
            dockerfile: ./docker/backend/Dockerfile
        entrypoint: /app/docker/backend/wsgi-entrypoint.sh
        volumes:
            - static_volume:/app/backend/server/django_static
        expose:
            - 8000        

volumes:
    static_volume: {}

The nginx server is listenning on two ports: 80 (HTTP) and 443 (HTTPS). There are added additional volumes in nginx with certificate. There is a new container certbot that is responsible for certificate renewal. There are no changes in the backend container.

OK, to be ready to run docker-compose we need to get the certificate from Let’s Encrypt. I will use for it a bash script init-letsencrypt.sh. The script is from:

article Nginx and Let’s Encrypt with Docker in Less Than 5 Minutes,
the Github repository with script: link.

The init-letsencrypt.sh file:

#!/bin/bash

if ! [ -x "$(command -v docker-compose)" ]; then
  echo 'Error: docker-compose is not installed.' >&2
  exit 1
fi

domains=(boilerplate.saasitive.com www.boilerplate.saasitive.com)
rsa_key_size=4096
data_path="./docker/nginx/certbot"
email="" # Adding a valid address is strongly recommended
staging=1 # Set to 1 if you're testing your setup to avoid hitting request limits

if [ -d "$data_path" ]; then
  read -p "Existing data found for $domains. Continue and replace existing certificate? (y/N) " decision
  if [ "$decision" != "Y" ] && [ "$decision" != "y" ]; then
    exit
  fi
fi


if [ ! -e "$data_path/conf/options-ssl-nginx.conf" ] || [ ! -e "$data_path/conf/ssl-dhparams.pem" ]; then
  echo "### Downloading recommended TLS parameters ..."
  mkdir -p "$data_path/conf"
  curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf > "$data_path/conf/options-ssl-nginx.conf"
  curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot/certbot/ssl-dhparams.pem > "$data_path/conf/ssl-dhparams.pem"
  echo
fi

echo "### Creating dummy certificate for $domains ..."
path="/etc/letsencrypt/live/$domains"
mkdir -p "$data_path/conf/live/$domains"
docker-compose run --rm --entrypoint "\
  openssl req -x509 -nodes -newkey rsa:1024 -days 1\
    -keyout '$path/privkey.pem' \
    -out '$path/fullchain.pem' \
    -subj '/CN=localhost'" certbot
echo


echo "### Starting nginx ..."
docker-compose up --force-recreate -d nginx
echo

echo "### Deleting dummy certificate for $domains ..."
docker-compose run --rm --entrypoint "\
  rm -Rf /etc/letsencrypt/live/$domains && \
  rm -Rf /etc/letsencrypt/archive/$domains && \
  rm -Rf /etc/letsencrypt/renewal/$domains.conf" certbot
echo


echo "### Requesting Let's Encrypt certificate for $domains ..."
#Join $domains to -d args
domain_args=""
for domain in "${domains[@]}"; do
  domain_args="$domain_args -d $domain"
done

# Select appropriate email arg
case "$email" in
  "") email_arg="--register-unsafely-without-email" ;;
  *) email_arg="--email $email" ;;
esac

# Enable staging mode if needed
if [ $staging != "0" ]; then staging_arg="--staging"; fi

docker-compose run --rm --entrypoint "\
  certbot certonly --webroot -w /var/www/certbot \
    $staging_arg \
    $email_arg \
    $domain_args \
    --rsa-key-size $rsa_key_size \
    --agree-tos \
    --force-renewal" certbot
echo

echo "### Reloading nginx ..."
docker-compose exec nginx nginx -s reload

You need to set two variables in the script:

domains=(boilerplate.saasitive.com www.boilerplate.saasitive.com) you should enter your domain here,
staging=1 - is set to test the configuration first with Let’s encrypt staging environment! It is important to not set staging=0 before you are 100% sure that your configuration is correct. If you are sure, that all is set correctly, set staging=0. This is because there are limited number of retries to issue the certificate and you don’t want to wait till they are reseted (once a week). To read more about this, check Let’s encrypt rate limits docs.

We will use this script first with staging=1. We will test if web application is running correctly. If yes then we will issue production certificate and run the application with it.

Note: You need to have your own domain in this tutorial. The Let’s Encrypt certificates doesn’t work with public IPs (link to forum topic).

Deploy to VPS

I will use AWS to deploy the project. The application will be deployed to boilerplate.saasitive.com address.

We need to do below steps to deploy application:

start EC2 instance,
configure DNS,
copy code to EC2 instance,
issue certificate,
start docker-compose.

Start EC2 instance

First thing is to start EC2 instance. I’m using t2.micro instance (cant use t2.nano because of too low memory to build containers). Please launch a new EC2 instance. I’m using Ubuntu Server 18.04 LTS as the image (the last image in the screenshot below):

After clicking select, please go with t2.micro instance type (the cost of running is 8.35$ per month). Click Next: Configure Instance Details. In configuration please click Next and stop on Step 6: Configure Security Group. Please add there rules to accept HTTP and HTTPS traffic (like in the image below):

Click Review and Launch and then Launch. You should get the *.pem file for secure connection to the instance. You can use existing key pair or generate the new one. My file is boilerplate.pem. After a few seconds you should see your instance running. Congratulations, you have a VPS running! Now, we will configure domain to point to our VPS.

Configure DNS

In the instances view, please click on your instance and you should see Public IPv4 address. Please copy this address.

Let’s go to Route 53 service in AWS to configure the DNS. I have there Hosted zone configured for saasitive.com domain. I will add there two type A records. Please click on Create record. Then select Simple routing. Click Next and Define simple record. Please give the record name and route traffic to VPS IP, like in the image below:

I want to have routing for:

boilerplate.saasitive.com
www.boilerplate.saasitive.com

That’s why I’ve added two records, one for boilerplate and one for www.boilerplate. Both pointing to the same IP.

If you don’t want to run application on subdomain, just point main domain to VPS IP.

Copy code to VPS

We need to have our application code in the VPS to run it. There are many ways to do it. I’m using rsync command. Here is how.

Please select the instance (in EC2 instances list) and click Connect at the top of the website. There should be example how to connect to instance with SSH:

ssh -i "boilerplate.pem" [email protected]

Please remember to:

give correct permissions to *.pem file: chmod 400 boilerplate.pem,
please replace the name of *.pem file to yours.

After ssh, you should be logged in the VPS. Let’s install docker and create the app directory:

sudo snap install docker
mkdir app

In the new console, please navigate to the project directory and rsync it with directory in VPS:

# run locally in project main directory
rsync -avz -progress -e "ssh -i boilerplate.pem" --exclude backend/venv --exclude frontend/node_modules . [email protected]:/home/ubuntu/app/

The above command sync the current directory (there is a . dot in the command!) with home/ubuntu/app/ directory in the VPS.

That’s all! If you go to terminal with SSH connection to VPS and run ls command in app directory, you should see all your project files.

Issue certificate

Let’s go to VPS SSH connection and go into app directory. In this directory, please first build docker-compose containers:

# run in VPS!
cd app
sudo docker-compose build

First, we will issue certificate from Let’s Encrypt staging environment.

# run in VPS! in app dir
sudo ./init-letsencrypt.sh 

If successful, please stop all running containers and run the docker-compose in the background:

# run in VPS! in app dir
sudo docker-compose down
sudo docker-compose up --detach

You should be able to navigate to your website. Please go to web browser and enter your domain (or subdomain). In my case, I will go to https://boilerplate.saasitive.com. You should see warning message (depending on your web browser) that will tell you that connection is insecure:

To see the website, you need to go into advanced options and accept the risk. After that, you should see the application:

We need to issue production-ready certificate. Let’s go to VPS and stop containers:

# run in VPS! in app dir
# stop containers
sudo docker-compose down

You need to edit init-letsencrypt.sh file and set staging=0. Then run the script:

# run in VPS! in app dir
sudo ./init-letsencrypt.sh

Once again, stop containers and run them in the background:

# run in VPS! in app dir
sudo docker-compose down
sudo docker-compose up --detach

Let’s refresh the web browser:

Great, the application is running with HTTPS! What is more, if you enter URL with http:// you will be redirected to https://! Nice!

Is it working correctly? Please try to login or signup …

Ups, there is some error …

Edit ALLOWED_HOSTS

One more thing, we need to add boilerplate.saasitive.com and www.boilerplate.saasitive.com to the ALLOWED_HOSTS in Django settings.py file. And, there is a DEBUG flag in settings.py. Please set it to False.

# backend/server/server/settings.py

# ...
DEBUG=False

ALLOWED_HOSTS = ['0.0.0.0', 'boilerplate.saasitive.com', 'www.boilerplate.saasitive.com']
# ...

Save the file and sync the code on VPS.

# sync code
rsync -avz -progress -e "ssh -i boilerplate.pem" --exclude backend/venv --exclude frontend/node_modules . [email protected]:/home/ubuntu/app/

On VPS machine, please stop docker, build it and re-run in the background:

# run in VPS! in app dir

# stop containers
sudo docker-compose down

# build containers
sudo docker-compose build

# run containers
sudo docker-compose up --detach

Please try to login after update:

You can try to create new users and login for example from your phone web browser. The application will be working!!! :)

Commit changes to the repository code

Please remember to commit changes to the repository:

# in the main project dir

git add docker-compose*
git add init-letsencrypt.sh
git add docker

git commit -am "add docker-compose"
git push

Summary

We’ve created two docker-compose configurations, one for development and one for production.
The production docker-compose has SSL certificate from Let’s Encrypt.
We’ve deployed application to AWS EC2 instance with docker-compose.
There are still many things to be added in the project.
We need to better handle configuration variables and SECRET_KEY in the Django code. We will add this in the future post with python-decouple package.
We need to switch database engine to PostgreSQL.
After every update in the application, we will need to run a set of commands. This can be boring and error-prone. That’s why we will need to add Continous Integration (CI) to our project. This can be added in many different ways, I will try to figure out the simplest approach (usually, I go with bash scripts). I’m open to your suggestions.
It is important to remember that if EC2 machine is restarted, then new IP address is assigned. So there is a need to reconfigure DNS again. This can be prevented with static IP address. But, you need to pay for it additionally. I’m not setting the static IP. I’m planning to write custom blue/green deployment script (in the future post).

What’s next?

In the next article we will extend user model.

React and Django Tutorial

Docker compose with Django 4, Celery, Redis and Postgres

Example project

Setup environment

Start Django project

Assignment database model

Configure Celery

Running locally

Create Dockerfile

Entrypoint scripts

Nginx configuration

Create docker-compose

docker-compose commands

Summary

Dynamically update periodic tasks in Celery and Django

Overview

Start Django project

Monitor database model

Celery configuration

Background task

Run Django and Celery

Summary

Email Verification View in React for DRF backend

Email Verification View

Update authSlice

POST request with a verification token

Update VerifyEmailView.tsx

Summary

Django Rest Framework Register New User with Email Verification

Unit test

Check login before the email verification

Test email

Summary

Django Rest Framework Email Verification

Email required in registration

Activation email sending in the Django

Custom create a token serializer

Testing email verification flow

Commit your changes

Summary

GDPR Overview

Where to start with GDPR

The Personal Data Protection Policy

GDPR Meaning - The General Data Protection Regulation

GDPR meaning

Who the GDPR applies to?

Data Protection Officer, do you need it?

Visual Identity for SaaS

Creative work

Colors

Font

Icons

Pictures

The devil is in the details.

Summary

How to generate Django Secret Key?

Docker-Compose for Django and React with Nginx reverse-proxy and Let’s encrypt certificate

Create Dockerfile for Django and Nginx

Django Dockerfile

Docker entrypoint

Django static files

Nginx Dockerfile and Configuration

Docker-compose for Django, Nginx and React

Build containers

Run containers

Stop containers

Build and run containers

Production docker-compose

Deploy to VPS

Start EC2 instance

Configure DNS

Copy code to VPS

Issue certificate

Edit ALLOWED_HOSTS

Commit changes to the repository code

Summary

What’s next?

Create `docker-compose`

`docker-compose` commands

Update `authSlice`

Update `VerifyEmailView.tsx`