Deploying a full Django stack with Docker-Compose

This is a tutorial developed in March 2016 for informative purposes. It will not be updated for future versions of the technology or the dependencies used in it.

In this tutorial, we will see how we can deploy a full stack (Django web app, with PostgreSQL and Redis) using Docker Compose.

First, make sure you have a docker setup ready, or follow this documentation, depending on your distribution, here we will suppose the host and development machine is a Debian 8.

You will also need Docker Compose.

We will deploy a total of 5 Docker containers :

• 1 Front, that contains the application code, if you don’t know Django, this will give you a little introduction to this awesome web framework.
• 1 Reverse proxy, that will handle the static assets and forward the dynamic ones to the Django WSGI process.
• 1 PostgreSQL database
• 1 Redis database
• 1 Data instance, that will contain the PostgreSQL data files, so we can freely rebuild and upgrade the PostgreSQL instance without impact on the data.

Here is a schema of the final docker platform :

We will also create an automatic nightly backup of the database.

To start, we create a new git repository what will contain everything (Just for ease, this is not mandatory), let’s place inside the following directories :

• nginx : that will contain the nginx Dockerfile and the vhost definition
• postgres : That will contain the PostgreSQL initialization script
• web : That will contain our application code and the Dockerfile for the front
• scripts : That will contains all our scripts (The backup script here)

The basics

First, we will create an env file ad the root directory of the project that will contain environment variables shared by many systems and scripts:

DB_NAME=myproject_web
DB_USER=myproject_web
DB_PASS=shoov3Phezaimahsh7eb2Tii4ohkah8k
DB_SERVICE=postgres
DB_PORT=5432

Then, we create the docker-compose.yml file with all of the following content, let me explain each host definition:

web:
  restart: always
  build: ./web/
  expose:
    - "8000"
  links:
    - postgres:postgres
    - redis:redis
  env_file: env
  volumes:
    - ./web:/data/web
  command: /usr/bin/gunicorn mydjango.wsgi:application -w 2 -b :8000

• Restart : This container should always be up, and it will restart if it crashes
• Build : We have to build this image using a Dockerfile before running it, this specifies the directory where is located the Dockerfile
• Expose : We expose the port 8000 to linked machines (that will be used by the NGINX container)
• Links : We need to have access to the postgres instance using the postgres name (This create a postgres entry in the /etc/hosts files that point to the postgres instance IP), idem for the redis.
• Env_file : This container will load all the environment variables from the env file.
• Volumes : we specify the different mountpoints we want on this instance
• Command : What command run when starting the container ? Here we start the wsgi process.

nginx:
  restart: always
  build: ./nginx/
  ports:
    - "80:80"
  volumes_from:
    - web
  links:
    - web:web

postgres:
  restart: always
  image: postgres:latest
  volumes_from:
    - data
  volumes:
    - ./postgres/docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
    - ./backups/postgresql:/backup
  env_file:
    - env
  expose:
    - "5432"

• Here instead of the build option, we have the image option that targets an existing image on the Docker registry
• we also use volumes_from to load all the volumes of another container. We use it on the Nginx container to load the static directory from the application and serve it, and on the PostgreSQL container to load the persistent tablespace that is in the data container.

redis:
  restart: always
  image: redis:latest
  expose:
    - "6379"

data:
  restart: always
  image: alpine
  volumes:
    - /var/lib/postgresql
  command: "true"

• We create a standard redis container, using the latest version of the official image and exposing the redis port.
• On the data container, we use true (That does nothing and just keep the container running) as a command as we just want this container to hold the PostgreSQL tablespace.
• Using a data container is the recommended way when we want to manage data persistence, using this, we don’t risk any accidental deletion during, for example, an upgrade of the PostgreSQL container (That will delete all the data from its container)

The PostgreSQL container

• On the data container, we use true as command as we just want this container to hold the PostgreSQL tablespace.

First, we will configure the PostgreSQL container to initialize itself. The Postgres image loads by default all the scripts in the /docker-entrypoint-initdb.d directory, let’s create a simple script that will create a user and a database using the information from the env file :

> cat postgres/docker-entrypoint-initdb.d/myproject_web.sh
#!/bin/env bash
psql -U postgres -c "CREATE USER $DB_USER PASSWORD '$DB_PASS'"
psql -U postgres -c "CREATE DATABASE $DB_NAME OWNER $DB_USER"
> chmod a+rx postgres/docker-entrypoint-initdb.d/myproject_web.sh

The reverse proxy container

We will configure the nginx container, the recommended way when using this tutum/nginx is to use a Dockerfile, let’s create a simple one:

> cat nginx/Dockerfile
FROM tutum/nginx

RUN rm /etc/nginx/sites-enabled/default
ADD sites-enabled/ /etc/nginx/sites-enabled

And we just have to create a file in sites-enabled/ to serve the static files and forward the rest to the application:

> cat nginx/sites-enabled/django
server {

    listen 80;
    server_name not.configured.example.com;
    charset utf-8;

    location /static {
        alias /data/web/mydjango/static;
    }

    location / {
        proxy_pass http://web:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

}

The web application container

The easy way is to just use an official Python image which is based on Debian, but the resulting image will be quite big (mine was like 900MB).

We want to create a very light front image so we can easily and rapidly upgrade it when we do code change and we want to reduce as maximum the attack surface from the outside. For this, we will base our image on Alpine Linux that is specialised into this and very common in the Docker world.

Then, let’s create our custom Alpine for the Django application, we will first run an interactive session to create the project and then we will create the Dockerfile.

But first, fill the web/requirements.txt with all the Python modules we need:

> cat web/requirements.txt
Django==1.8.5
redis==2.10.3
django-redis==4.3.0
gunicorn==19.3.0
psycopg2==2.6

Then, we can start:

> docker run -ti --rm -v /root/myproject/web/:/data/web alpine:latest sh
/ # cd data/web/
/data/web # apk add --update python3 python3-dev postgresql-client postgresql-dev build-base gettext vim
/data/web # pip3 install --upgrade pip
/data/web # pip3 install -r requirements.txt
/data/web # django-admin startproject mydjango
/data/web # mkdir mydjango/static

We’ll create an instance to create our Django application. Note that we do this to not have to install Python and it’s dependencies on our host system. We’ll start an Alpine instance, and interactively :

> docker run -ti --rm -v /root/myproject/web/:/data/web alpine:latest sh
/ # cd data/web/
/data/web # apk add --update python3 python3-dev postgresql-client postgresql-dev build-base gettext vim
/data/web # pip3 install --upgrade pip
/data/web # pip3 install -r requirements.txt
/data/web # django-admin startproject mydjango
/data/web # mkdir mydjango/static

Let’s populate the configuration file (mydjango/settings.py) with parameters usable with the information Docker provides in the containers, remove all the content between the “DATABASE” part and the “INTERNATIONALIZATION” and replace it with this :

if 'DB_NAME' in os.environ:
    # Running the Docker image
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'NAME': os.environ['DB_NAME'],
            'USER': os.environ['DB_USER'],
            'PASSWORD': os.environ['DB_PASS'],
            'HOST': os.environ['DB_SERVICE'],
            'PORT': os.environ['DB_PORT']
        }
    }
else:
    # Building the Docker image
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.sqlite3',
            'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
        }
    }

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://redis:6379/0",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
        }
    }
}

SESSION_ENGINE = "django.contrib.sessions.backends.cache"
SESSION_CACHE_ALIAS = "default"

Once done, just leave the container with exit (the –rm flag when creating the instance will destroy it when leaving, the application is saved in the mounted volume)

Here we load the Docker database if we find the environment variable provided by the env file, and use all those information to connect, if not, it’s because we are building the image, we don’t want this to block some commands (Like if you want to compile the gettext translation of your website)

For the redis container, we just target to the redis DNS that will be present once we have deployed it using docker-compose.

Now we have our little django application, let’s put all that we need in the Dockerfile of the web container:

FROM alpine

# Initialize
RUN mkdir -p /data/web
WORKDIR /data/web
COPY requirements.txt /data/web/

# Setup
RUN apk update
RUN apk upgrade
RUN apk add --update python3 python3-dev postgresql-client postgresql-dev build-base gettext
RUN pip3 install --upgrade pip
RUN pip3 install -r requirements.txt

# Clean
RUN apk del -r python3-dev postgresql

# Prepare
COPY . /data/web/
RUN mkdir -p mydjango/static/admin

As you see, here we also clean the packages we will not need after we installed the Python modules (For compiling the PostgreSQL module for example)

Let the magic happen

Take a coffee (depending of your connection speed) and build everything

> docker-compose build
> docker-compose up -d

Then you can check your containers with this :

> docker-compose ps
        Name                      Command               State         Ports
----------------------------------------------------------------------------------
myproject_data_1       true                             Up
myproject_nginx_1      /usr/sbin/nginx                  Up      0.0.0.0:80->80/tcp
myproject_postgres_1   /docker-entrypoint.sh postgres   Up      5432/tcp
myproject_redis_1      /entrypoint.sh redis-server      Up      6379/tcp
myproject_web_1        /usr/bin/gunicorn mydjango ...   Up      8000/tcp

> docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                          PORTS                NAMES
d7162329302d        myproject_nginx     "/usr/sbin/nginx"        2 minutes ago       Up 2 minutes                    0.0.0.0:80->80/tcp   myproject_nginx_1
402c2ca47789        myproject_web       "/usr/bin/gunicorn my"   2 minutes ago       Up 2 minutes                    8000/tcp             myproject_web_1
2c92e1fa0021        postgres:latest     "/docker-entrypoint.s"   8 minutes ago       Up 2 minutes                    5432/tcp             myproject_postgres_1
ad58f3138339        alpine              "true"                   9 minutes ago       Restarting (0) 58 seconds ago                        myproject_data_1
29ece917fcbb        redis:latest        "/entrypoint.sh redis"   9 minutes ago       Up 2 minutes                    6379/tcp             myproject_redis_1

The “restarting” on the data container is normal as it does not have persistent process.

If everything is fine, go to your public IP and then you should see the default Django page (That should only work if it can connect successfully to the database and load the cache engine):

The backups

Then if everything went fine, we can setup backups for the PostgreSQL database, here we just created a small script to do it every night and upload it to S3 (on a bucket with versioning enabled and a lifecycle policy)

/scripts/do_backup.py

import os
import socket
import boto3

BUCKET = "myproject-backups"
S3_DIRECTORY = socket.gethostname()
DB_BACKUP_FILE = "myproject_postgres_1.sql"
DB_BACKUP_PATH = "/tmp/{filename}".format(filename=DB_BACKUP_FILE)
DB_S3_DIRECTORY = "{directory}/postgresql".format(directory=S3_DIRECTORY)

s3 = boto3.resource('s3')

os.system("docker exec -u postgres myproject_postgres_1 pg_dumpall > {path}".format(path=DB_BACKUP_PATH))

backup = open(DB_BACKUP_PATH, "rb")
s3.Object(BUCKET, "{directory}/{filename}".format(directory=DB_S3_DIRECTORY, filename=DB_BACKUP_FILE)).put(Body=backup)

Don’t forget to install the boto3 PIP module, the AWS cli and to configure your credentials with aws configure

Then, just configure a cron job to schedule it when you want.

The end

You now have a fully working docker compose environment 🙂 When you do a modification, you just need to run a new docker-compose build; docker-compose up -d and everything will be automatically updated.

TAGS: django, docker