I WENT THROUGH 7 PROXIES
GOOD LUCK
— Anonymous

In a previous post we briefly mentioned the usefulness of local reverse proxies in the context of an ASP.NET Core application: we expose the battle-hardened Nginx server to the Internet, and place Kestrel behind it.

In this post we’ll explore why and how we can setup additional proxies over multiple physical hosts, and more specifically, over the Internet.

Why?

There are three main reasons:

1. Load balancing
2. Server redundancy
3. Hiding your server’s IP address

The first and second point are obvious: load balancing with a single machine is an oxymoron, more so regarding server redundancy.

The third point relates mostly to mitigating DDoS attacks, but also to privacy if you’re self-hosting something on a personal IP (e.g. the one provided by the ISP); free reverse DNS lookup services are a serious threat here.

DDoS mitigation

The basic advice for DDoS mitigation is “don’t expose your webserver’s IP directly”.

By doing so, you can add additional mitigation before it if you want; you could add multiple custom DDoS protection proxies, or maybe pay for commercial offerings from Cloudflare, Fastly, or alternative providers.

Network-level solutions are available too, the good ol’ null route doesn’t have to mean that your services can’t communicate between themselves if you kept a separate address for public traffic, and “private” traffic.

For example, here Client makes requests over the Internet to example.com, which acts only as a reverse proxy and forwards requests to teemo.example.net. At the same time, other of our servers makes requests directly to teemo.example.net.

If example.com got hammered by a DDoS attack we can’t handle, we can always just turn it off, which would not disrupt interactions between garen and teemo.

Privacy concerns when self-hosting

Self-hosting Internet exposed services is less common than what it once was, due to cheap availability of VPS instances, and because many ISP companies continue to [ab]use CG-NAT instead of upgrading their stuff to IPv6.

If you do self-host, however, then the usual situation is the following:

The example.com name is bound to our x.x.x.x public address, which is the same address that’s logged by any Internet connected application (e.g. websites) we visit.

But what’s the problem with that?

[max@home ~]$ whois sinenie.cl
[Querying whois.nic.cl]
[whois.nic.cl]
%%
%% This is the NIC Chile Whois server (whois.nic.cl).
%%
%% Rights restricted by copyright.
%% See https://www.nic.cl/normativa/politica-publicacion-de-datos-cl.pdf
%%

Domain name: sinenie.cl
Registrant name: Cristian Ojeda Flores
Registrant organisation: 
Registrar name: NIC Chile
Registrar URL: https://www.nic.cl
Creation date: 2017-01-07 02:15:21 CLST
Expiration date: 2022-01-07 02:15:21 CLST
Name server: ns1.digitalocean.com
Name server: ns2.digitalocean.com
Name server: ns3.digitalocean.com

%%
%% For communication with domain contacts please use website.
%% See https://www.nic.cl/registry/Whois.do?d=sinenie.cl
%%

I don’t feel comfortable shitposting like this.

We can blame this on the complete disregard for privacy from both NIC.cl and SERVEL; it’s incredibly easy to doxx someone in matter of seconds if we don’t do something about what’s shown in Figure 2.

A possible solution would look like this:

How?

We’ll explore what I did to setup a reverse proxy for both HTTPS and SSH traffic for the GitLab instance hosted at git.sinenie.cl.

Frontend Server Setup

The frontend server is the server or servers that act as our remote reverse proxies. They forward requests to the backend.

It’s basically the same server block as a local proxy, except that the proxy_pass directive points not to localhost, but rather to the backend’s HTTPS address:

server {
        listen 443 ssl;
        server_name git.sinenie.cl;
        client_max_body_size 0;

        ssl_certificate /etc/letsencrypt/live/git.sinenie.cl/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/git.sinenie.cl/privkey.pem;

        location / {
                proxy_set_header Connection keep-alive;
                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_pass https://asdf.noip.com/gitlab/;
        }
} 

With regards to SSH, it’s very easy to do since Nginx 1.9 because of the Stream module.

The stream configuration block must be placed at the same level as the http configuration block, usually at the nginx.conf for Ubuntu-like Nginx installs.

# === /etc/nginx/nginx.conf ===

# Misc. Nginx settings here...

http {
    # HTTP settings here...
}

# SSH reverse proxy
stream {
        upstream ssh {
                server asdf.noip.com:50515;
        }
        server {
                listen 9001;
                proxy_pass ssh;
        }
}

Any TCP request that comes to the port 9001 is proxied to backend’s port 50515.

Backend Server Setup

The backend server is the server or servers that actually perform the work required to respond to a client request.

The backend server block looks like this:

server {
        listen 443 ssl;
        server_name asdf.noip.com;
        allow y.y.y.y;
        deny all;

        ssl_certificate /etc/letsencrypt/live/asdf.noip.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/asdf.noip.com/privkey.pem;

        client_max_body_size 0;
        underscores_in_headers on;

        location /gitlab/ {
                proxy_pass http://localhost:5000/;

                # Websocket connection
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection upgrade;

                # Forwarded headers
                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
}

There’s one, albeit VERY IMPORTANT detail: both the location and the proxy_pass value end with a slash. This way the server block will match any path that begins with /gitlab/, but remove that segment before passing the path to proxy_pass.

Example:

1. Request arrives at frontend for https://git.sinenie.cl/users/sign_in
2. Frontend makes a request for https://asdf.noip.com/gitlab/users/sign_in
3. Backend receives the request and makes a request for http://localhost:5000/users/sign_in
4. GitLab generates a response, which is passed back to each proxy and eventually is passed back to the user’s client (usually a web browser).

Why are easily missable slashes so important? ¯_(ツ)_/¯

GitLab has an embedded SSH server; all you need to do is to edit the configuration files to configure the listening port, and open such port in the firewall.

Reasoning for using location instead of alternatives

Choosing to use location, instead of having multiple server blocks, each with its own server_name directive wasn’t a random choice. I considered the following three alternatives:

1. Use location directives so I could use a single A record and forward a single port in my router.
2. Use multiple A records, e.g. asdfapp1.noip.com, asdfapp2.noip.com, etc.
3. Use multiple ports, e.g. asdf.noip.com:50844, asdf.noip.com:56709, etc.

And that’s basically it. Location won because it works, and it’s simpler.

Preventing unauthorized requests to the backend server

It’s possible, and recommended even, to secure requests by using client certificates as described in this guide:

https://docs.nginx.com/nginx/admin-guide/security-controls/securing-http-traffic-upstream/

In my case, I have my server behind a NAT, and I’ve setup my router settings to only redirect requests to it if they come from the frontend server’s IP address; I know it’s vulnerable to targeted attacks using IP spoofing, but it’s good enough for now.

Summary

In this post we’ve explored why, and how we can setup a remote reverse proxy for HTTPS and SSH by using Nginx. Usually, we do this in order to run our applications in multiple servers, or to hide our servers’ IP addresses.

The configuration itself is really simple, and very similar to what we need for a local reverse proxy. Just be careful to secure your backend server somehow!

In a future post we’ll explore how to use Content Delivery Networks (CDN) to take this to another level, and improve our application’s response times globally.