While this post refers to deployment of a Flask web application, it should work similarly for a Django web application. If that’s not the case, then I’ll update the post accordingly.

This should be another long post, and also the end of the trilogy about setting up a server for that new shiny webapp you’ve been working on.

1. Installing Python 3.6 inside a Ubuntu 16.04LTS VPS
2. Managing Virtual Environments on Python

At the end of this post, you’ll have…

• …your web application running on an Ubuntu VPS,
• with Gunicorn acting as the WSGI HTTP server for Flask,
• which will work as a systemd service that starts itself automatically,
• with Nginx acting as a reverse proxy…
• …serving only HTTPS requests (and redirecting HTTP to HTTPS),
• using free TLS certificates that renew by themselves!

Wew. That’s a lot of stuff, and we’ll cover all of it in this post. Later we may split it in several, more detailed posts.

Now, as a cowardly disclaimer, if your application is actually business critical, it needs to be easily updated (i.e. several instances behind a load balancer), or it requires careful to serve a LOT of requests, then this post WILL fall short.

This is more of a recount of what I did to have my boring file transfer site up and running.

Requisites

You have an Ubuntu 16.04LTS VPS with your desired version of Python installed (either globally, or using pyenv) and that have some way of creating virtual environments. Both things are covered in the previous posts of this trilogy.

I’ll use Python 3.6 installed globally from a PPA, and pipenv to create the virtualenvs.

Setting up your firewall with UFW

We’ll only allow SSH, and HTTP/S through. HTTP for getting our first TLS cert with certbot, and then for allowing the redirects to HTTPS.

# ufw allow 22
# ufw allow 80
# ufw allow 443
# ufw enable
# ufw status

Of course, if you use a different port for ssh, then you should allow that instead of 22.

Installing your application and Gunicorn

First, you should decide where you want to place your application in the server. For this example I’ll use the application I’ll just use that file transfer site I spoke about earlier, and I’ll place it in /opt/filenigma.

Please note that I previously had a user called max that I use for remote administration. If you do this as root then you can omit the chown.

$ cd /opt/
# mkdir filenigma
# chown max:max filenigma
$ git clone https://github.com/0x1cc/filenigma
$ cd filenigma

I use pipenv with the PIPENV_VENV_IN_PROJECT environment variable set to 1, to create the virtualenv in /opt/filenigma/.venv/. I now create the env and install requirements.

Of course, if you already had a virtualenv that you want to use, then activate it instead and install your application inside.

$ pipenv install --python=python3.6
$ pipenv install gunicorn

(The following 3 commands are specific to the example project, filenigma.)
I install Filenigma as a package, and then run a command to create the sqlite DB it uses.

$ pipenv shell
(filenigma) $ pip install --editable .
(filenigma) $ FLASK_APP=filenigma flask initdb

After that, we can finally test our application with Gunicorn.

(filenigma) # gunicorn filenigma:app -b :7777

With that we should be able to access our application from the public IP of our server, at port 7777. We may need to temporarily allow that port in, though: ufw allow 7777.

Running our webapp as a systemd service

Now, while technically we could skip this step and go setup Nginx right now, I’m not a big fan of having to start my sites manually when the server restarts. Or having to hunt for the process that represents my site to see if it’s running (vs. systemctl status filenigma).

User setup

We aren’t going to run our site as root, so we have at least two options:

1. Create a user specifically for running our application, and nothing else.
2. Just use our non-root user that we use for logging-in remotely.

Isn’t that much harder to create a dedicated user, so we’ll do that. I’ll call it filenigma so later it’s obvious that’s related with the application of the same name.

# adduser filenigma --shell=/bin/false --no-create-home --disabled-password
...Feel free to just spam enter to use the defaults...

# usermod -aG filenigma max
# chown -R filenigma:filenigma /opt/filenigma
# chmod 2775 /opt/filenigma

Now, let’s study that step by step.

# adduser filenigma --shell=/bin/false --no-create-home --disabled-password

We had a user that has no home directory, without a password and without a default shell. We aren’t going to use this user ourselves. It exists for the only purpose of running Gunicorn and to act as a container of permissions.

# usermod -aG filenigma max

We add our non-root administrative user to the group of the new “user”, so we can modify its owned files without having to sudo-all-the-things.

# chown -R filenigma:filenigma /opt/filenigma

We change the owner and group of the application directory and contents to those of our new user. Otherwise it will be unable to update anything inside.

# chmod 2775 /opt/filenigma

Now, this requires a little more elaboration. We are changing the permissions of the directory so both the owner and [filenigma] group members can read and write content inside it. The execute bit is so we can cd into the directory.

What’s not so common though, is the 2, as chmod masks usually have only 3 digits. This 2 sets the setgid bit on, which causes that anything created in any subdirectory to have its group set to that of the closest parent directory with a setgid bit set.

We do this so anything we create inside the application directory will have its group set to filenigma, which will make it accessible and modifiable by our filenigma user without further changes. Not doing this will require us copious amounts of chowning after any change.

If you cannot modify files inside the application directory, despite being in the same group as the files, it may be because of the group write bit that by default is off in Ubuntu 16.04.

chmod g+w -R /opt/filenigma would solve the issue for now, and setting a umask of 002 in ~/.profile would set that by default when creating new files they are writable by group members.

• chmod flags
• chmod numeric modes
• chmod symbolic modes - permissions

Service setup

Create a file in /etc/systemd/system with the desired service name and a extension of .service. Example: filenigma.service.

[Unit]
Description=Filenigma
After=network.target

[Service]
PIDFile=/tmp/filenigma.pid
User=filenigma
Group=filenigma
WorkingDirectory=/opt/filenigma
ExecStart=/opt/filenigma/.venv/bin/gunicorn --pid /tmp/filenigma.pid filenigma:app
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

This file is by the most part, self-explanatory, and what’s most interesting is the ExecStart directive. Here we can change the options passed to the Gunicorn process (e.g. number of workers).

Then start the service, and enable it so auto-starts on reboot.

# systemctl start filenigma
# systemctl status filenigma
# systemctl enable filenigma

If there are problems, they’ll be listed after running systemctl status.

Finally, you may have notices, but we aren’t binding Gunicorn to :7777 anymore, but rather the default localhost:8000, so our application is no longer accesible from the Internet. But that will change soon.

Setting up Nginx

Nginx is, among other things, a load balancer, and a web server. It comes in two versions: the Open Source version, and Nginx Plus.

We care about the OSS version, which can be installed in Ubuntu by doing:

# apt-get install nginx -y

This will install nginx, start up the nginx.service and setup a default site (Welcome to Nginx!) which can be visited at the external IP of your server.

Nginx has quite a bit of functionality, but fortunately what’s relevant to us regards the Web Server mostly.

• Nginx Beginners Guide
• Nginx Basic Funcionality
• Nginx Web Server

There’s no point on repeating what’s in there, so there’s a short summary:

In /etc/nginx/ there is the nginx.conf configuration file. By default in the Ubuntu 16.04 it includes the content of any file in /etc/nginx/conf.d/ and those existing in /etc/nginx/sites-enabled/.

The sites-enabled directory contains symbolic links to files in the /etc/nginx/sites-available directory, which by default only contains one file: the default file. While we can use that as a starting point, here’s something more specific for what we are doing now:

server {
    server_name filenigma.sinenie.cl;
    client_max_body_size 256m;

    location /static/ {
        root /opt/filenigma/filenigma/;
    }

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://127.0.0.1:8000;
    }
}

What it means is that any request coming to filenigma.sinenie.cl will be handled by this virtual server. Of those requests:

• Those starting with /static/ will simply be served as static files (e.g. a request for filenigma.sinenie.cl/static/app.css will get the file /opt/filenigma/filenigma/static/app.css).
• Any other request will be proxied to our Gunicorn server running in port 8000.

This file must be located at the aforementioned sites-available directory, with any name, but I’ll call it filenigma for clarity.

Then in the sites-enabled directory you delete the default symlink, and create a new symlink by running:

ln -s /etc/nginx/sites-available/filenigma /etc/nginx/sites-enabled/

You can reload the nginx configuration by running nginx -s reload. After that, you should be able to access the site by its specified server name.

HTTPS with LetsEncrypt and Certbot

At this point we have a nicely working server, but we can do better.

To protect the integrity of the data transmitted between our users and our server, we will get and install a free TLS certificate from LetsEncrypt. The process is mostly automated, this including the installation and the renewal of the certs.

# add-apt-repository ppa:certbot/certbot -y
# apt-get update
# apt-get install python-certbot-nginx -y

That will install Certbot on our machine. To get a new certificate:

certbot --nginx -d filenigma.sinenie.cl

We can request a certificate valid for multiple domains if the append more -d (site) arguments. That’s useful when we want something valid for www.mydomain.tld and mydomain.tld at the same time.

Anyways, executing the above command will ask you for a email to where send warnings if for some reason your cert cannot be auto-renewed. After that, and other questions, certbot will perform challenges to validate ownership of the domains to include in the cert.

If everything goes well, you should see something like the following:

Obtaining a new certificate
Performing the following challenges:
http-01 challenge for filenigma.sinenie.cl
Waiting for verification...
Cleaning up challenges
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/filenigma.com

And immediately after, you will have the option to set HTTP to HTTPS redirects automatically. I strongly recommend that unless you have a good reason to not do so, you select the second option.

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
-------------------------------------------------------------------------------
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/filenigma

After this, accessing the site either via https://filenigma.sinenie.cl or just filenigma.sinenie.cl will redirect to the HTTPS version of it.

Finally, we can test the automatic cert renewal by running:

# certbot renew --dry-run

Wrapping-up

With this last post in the series, we covered the basics of deploying a Python Web application. That doesn’t mean that we cut corners, as the thing runs under its own user, as a service, in an isolated virtual environment and serves all of its requests using HTTPS.

What this isn’t, is horizontally scalable. While doing administrative tasks in just one server is one thing, doing it on dozens of them running the same thing is at best a PITA, pretty much always a risk; risk of forgetting about upgrading one server… and now we have several, potentially conflicting, versions of the same program running in different servers.

Later we’ll cover about more DevOpsie ways of doing this same thing, but for now, this is more than enough for my smaller applications running on virtual potatoes around the world.