Deploy to your test server with git hooks

Configure git

First, we will configure git. You can use a global configuration by editing /etc/gitconfig (users can still change it for all their repos in ~/.gitconfig or per each repo in .git/config).

Note that if you intend to do other things than git deploys with your server, the global configuration may not be the best choice.

Here is the git configuration we'll use

[core]
    hooksPath = /etc/git/hooks

[receive]
    denyCurrentBranch = updateInstead

You can check it by running: git config receive.denyCurrentBranch and git config core.hooksPath.

You must adapt the path to where the git hooks will be. If you choose a per repo configuration, you don't need to specify it and it will default to .git/hooks

By default git won't allow you to push to a non bare repo (ie a repo in which you can access files easily) because it is dangerous. But in our case, it is required, otherwise, we won't be able to push anything while making the site work. That's the point of the ``denyCurrentBranch = updateInstead`` option. See the documentation for more details.

Now we can create the directory where the global hooks will lie: mkdir -p /etc/git/hooks. We will add scripts here later.

Prepare the server repo

1. Clone the remote repo on your dev machine (where you want) so you can push from it.

2. Clone the remote repo on the server. You could also push directly from your dev machine to it but I find it easier this way (and you still have to create a repo on the server). Do it:

   1. Create the folder /var/www/bl.test/jujens.bl.test (the location may seem odd right now, but it will have its importance later): # mkdir -p /var/www/bl.test/jujens.bl.test
   2. Set user and group: # chown jujens:nginx /var/www/bl.test/jujens.bl.test
   3. Set the proper permissions: # chmod u=rwX,g=rXs,o=- /var/www/bl.test/jujens.bl.test (this will give all permission the current user, read and "execute" permissions to the group and nothing to others. The s is there to set the "sticky bit". Thanks to it, all file created inside the directory will belong to the group you specify.
   4. Clone: # git clone https://gitlab.com/Jenselme/hello-flask.git /var/www/bl.test/jujens.bl.test

3. Add the test remote on your dev repo (so you can push to the test server): $ git remote add test jujens@bl.test:/var/www/bl.test/jujens.bl.test

4. Check everything is configure properly with a $ git push test It should output:

   The authenticity of host 'bl.test (192.168.122.45)' can't be established.
   ECDSA key fingerprint is SHA256:nb/a1ypBTTgahruVf92GzLNGTAdrrDi2afIWfXaFAa0.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added 'bl.test' (ECDSA) to the list of known hosts.
   Everything up-to-date
   

Basic setup of the application server

First, we will prepare our venv (or whatever else you need in your language) with the proper dependencies. This section must be adapted if you are not using Python. Run on the server:

1. Create the venv: $ python3 -m venv .venv

2. Activate it: $ source .venv/bin/activate

3. Install the dependencies: $ pip install -r requirements.txt

4. Start the application server: $ gunicorn hello:app

5. Check it is running with $ curl localhost:8000 You should see the following output

   Hello World!
   

In a true server, you may want to use --forwarded-allow-ips to limit who can connect to guincorn. See: gunicorn's documentation

Now that it's done, we can add to the post-receive hook (/etc/git/hooks/post-receive) the following line to update our venv each time code is pushed:

# We are again in .git by default.
( cd .. && source .venv/bin/activate && pip install -r requirements.txt )

Basic nginx configuration

Create nginx configuration file in /etc/nginx/conf.d/bl.test.conf (or /etc/nginx/site-enabled/bl.test.conf depending on your system, I'll use /etc/nginx/conf.d for the rest of this tutorial) with this content:

server {
    listen 80;
    server_name jujens.bl.test;
    # Put it to the folder that will contains all the repos.
    root /var/www/bl.test/;

    # Prevent access to source files.
    location ~ .*\.pyc? {
        return 404;
    }

    # Prevent access to git
    location ~ .git {
        return 404;
    }

    # Relay everything else to the application server.
    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://127.0.0.1:8000;
    }
}

Test the configuration with nginx -t and if everything is fine, reload nginx: systemctl reload nginx. On your development machine, run curl jujens.bl.test to check everything is fine. You should see something like:

Hello World!

Configuring the application server

We will rely on systemd to start and restart the application servers. First, create a file named /etc/systemd/system/gunicorn@.service. It is a service template: it is meant to run multiple instances of a service based on a name (jujens.bl.test for instance). The name will be accessible as a variable (%i) in the template so we can alter the behavior of the service based on it.

Put this content in the file (adapted from http://docs.gunicorn.org/en/stable/deploy.html#systemd):

[Unit]
Description=gunicorn daemon for %i
After=network.target

[Service]
PIDFile=/var/run/gunicorn/%i.pid
User=nginx
Group=nginx
Environment="PATH=/var/www/bl.test/%i/.venv/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"
RuntimeDirectory=gunicorn
WorkingDirectory=/var/www/bl.test/%i
ExecStart=/var/www/bl.test/%i/.venv/bin/gunicorn \
        --pid /var/run/gunicorn/%i.pid   \
        --bind unix:/var/run/gunicorn/%i.sock \
        hello:app
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
PrivateTmp=true
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

I won't go into the details here (archlinux has a great doc for that). It is just how we create new unit in systemd. A unit being used to run a service. We use a description and dependencies in the Unit section. We then define the service:

• PIDFile is used by systemd to know whether the service is working.
• We define as who the service will run.
• We update the path so it can find our venv.
• We defined the working directory to where the app of any given user will live thanks to %i
• We defined how we want to start the service and how it can be reloaded and restarted.

I didn't use the --reload option of guincorn. That is because we may need to update the dependencies of our project before restarting it.

You may want to use the --worker X option to run multiple worker and thus allow your app to handle more traffic. In a test server it may not be needed though.

Now we need to "tell systemd" the file has changed with systemctl daemon-reload. We must also prepare the directories required by gunicorn to run:

1. Where the PID and socket files will be: mkdir -p /var/run/gunicorn
2. Correct permissions: chmod u=rwX,g=rX,o=- /var/run/gunicorn/

We can now start and enable the server by using the template and specifying the name of the instance like this: guincorn@DOMAIN. Or in my case: systemctl start gunicorn@jujens.bl.test and systemctl enable gunicorn@jujens.bl.test

The trailing .service in the service name is optional.

Use journalctl -u gunicorn@INTANCE to view the logs. For instance: journalctl -u gunicorn@jujens.bl.test

You don't actually need to create the /var/run/gunicorn directory. systemd will create it based on the value of RuntimeDirectory and set the permissions based on the values of User and Group.

On Ubuntu, /var/run/ is mounted on a tmpfs so the directory will disappear. It also seems that when you will launch multiple instances, the folder will be deleted and recreated making its usage as described in this tutorial impossible (guincorn cannot start by putting its PID and Socket directly in /var/run because of a permission issue). I suggest you use a runtime directory per user with RuntimeDirectory=gunicorn-%i or create a /var/myrun/ folder and adapt the configuration or dig into how Ubuntu does things.

Configuring nginx

We need to update the location / section of our nginx configuration to use the proper UNIX socket:

location / {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_redirect off;
    proxy_pass http://unix:/var/run/gunicorn/$host.sock;
}

We also need to update the server name into server_name *.bl.test;.

nginx will respond with a 502 Bad Gateway if there is no socket for that domain (either because it doesn't exist or because gunicorn is down).

server {
    listen 80;
    server_name *.bl.test;
    root /var/www/bl.test;

    # Prevent access to source files.
    location ~ .*\.pyc? {
        return 404;
    }

    # Prevent access to git.
    location ~ .git {
        return 404;
    }

    # If you request /page, we will check if /$host/page exists. If so we serve it.
    # Otherwise we pass the request to gunicorn for processing.
    # For instance, if I request jujens.bl.test/requirements.txt, nginx will check whether
    # /var/www/bl.test/jujens.bl.test/requirements.txt exists and serve it to me.
    # /var/www/bl.test comes from the server root and jujens.bl.test from the $host variable.
    location / {
        try_files /$host/$uri @gunicorn;
    }

    # Relay everything else to the django web server.
    location @gunicorn {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://unix:/var/run/gunicorn/$host.sock;
    }
}

Testing deployment

We are almost ready to test the deployment of our app. We still need two things. First, we must configure sudo to allow the user to restart gunicorn (because git hooks are run as the currently logged in user, jujens in my case, they may not be allowed to run systemctl restart by default). To do that:

1. Run visudo to edit the sudoers file (visudo will check the syntax of the file before exiting. You can use another editor by setting the EDITOR environment variable).
2. Add at the bottom of the file: %deploy ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart gunicorn@* This will allow all users in the deploy group to restart any instance of gunicorn. I think it is the easiest way to achieve our goal. Be more restrictive if you need to.

We can now add to our post-receive hook (/etc/git/hooks/post-receive):

# We move to the folder that contains our code.
# We extract the domain (and guincorn instance name) from the folder
# by keeping only the name of the current directory of the full path.
# Finally, we restart the proper instance of gunicorn.
( cd ..; unit_name=$(pwd); unit_name=${unit_name##*/}; sudo systemctl restart "gunicorn@${unit_name}" )

Now if you deploy a new commit, gunicorn will restart and if you do curl http://jujens.bl.test you will be able to see you modification.

Full configurations

• nginx configuration:

   1 server {
   2     listen 80;
   3     server_name *.bl.test;
   4     root /var/www/bl.test;
   5 
   6     auth_basic "Test server";
   7     auth_basic_user_file /etc/nginx/bl.test.passwd;
   8 
   9     # Prevent access to pyc and py files.
  10     location ~ .*\.pyc? {
  11         return 404;
  12     }
  13 
  14     location ~ .git {
  15        return 404;
  16     }
  17 
  18     location ~ ^/.env$ {
  19         return 404;
  20     }
  21 
  22     location / {
  23         try_files /$host/$uri @gunicorn;
  24     }
  25 
  26     # Relay everything else to the django web server.
  27     location @gunicorn {
  28         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  29         proxy_set_header Host $http_host;
  30         proxy_set_header X-Forwarded-Proto $scheme;
  31         proxy_redirect off;
  32         proxy_pass http://unix:/var/run/gunicorn/$host.sock;
  33     }
  34 }
  

• gunicorn configuration

   1 [Unit]
   2 Description=gunicorn daemon
   3 After=network.target
   4 
   5 [Service]
   6 PIDFile=/var/run/gunicorn/%i.pid
   7 User=nginx
   8 Group=nginx
   9 EnvironmentFile=/var/www/bl.test/%i/.env
  10 Environment="PATH=/var/www/bl.test/%i/.venv/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"
  11 RuntimeDirectory=gunicorn
  12 WorkingDirectory=/var/www/bl.test/%i
  13 ExecStart=/var/www/bl.test/%i/.venv/bin/gunicorn \
  14           --pid /var/run/gunicorn/%i.pid   \
  15           --bind unix:/var/run/gunicorn/%i.sock \
  16           hello:app
  17 ExecReload=/bin/kill -s HUP $MAINPID
  18 ExecStop=/bin/kill -s TERM $MAINPID
  19 PrivateTmp=true
  20 Restart=on-failure
  21 RestartSec=5s
  22 
  23 [Install]
  24 WantedBy=multi-user.target
  

• git hooks:

  • pre-receive

    1 #!/usr/bin/env bash
    2 
    3 # We launch the command in a subshell to avoid impacting the true environment with our unset.
    4 # We unset GIT_DIR because it points to the wrong directory.
    5 # We are in the .git directory when this hook is run, so we need to move one step up to run git commands.
    6 # We then force a checkout of the repo to discard any changes.
    7 ( unset GIT_DIR && cd .. && git checkout -f )
    

  • post-receive

     1 #!/usr/bin/env bash
     2 
     3 # Parameters are passed to stdin, we need to use read to get them. Only ref matters for what we do.
     4 read oldrev newrev ref
     5 # By default, ref contains refs/heads/branch-name. We use bash substitution mechanism to remove "refs/heads/".
     6 branch_name=${ref/refs\/heads\//}
     7 
     8 # Once again, force a checkout to the branch.
     9 ( unset GIT_DIR && cd .. && git checkout -f $branch_name )
    10 
    11 # You can add additionnal steps to complete checkout on the current branch (install deps, build JS/CSS files, …)
    12 # For instance:
    13 ( cd .. && source .venv/bin/activate && pip install -r requirements.txt )
    14 
    15 # We move to the folder that contains our code.
    16 # We extract the domain (and guincorn instance name) from the folder
    17 # by keeping only the name of the current directory of the full path.
    18 # Finally, we restart the proper instance of gunicorn.
    19 ( cd ..; unit_name=$(pwd); unit_name=${unit_name##*/}; sudo systemctl restart "gunicorn@${unit_name}" )
    20 # You could simply this line by using the $USER variable and creating BASE_DOMAIN="bl.test" into ``sudo systemcl restart "gunicorn@${USER}.${BASE_DOMAIN}"``