Sweet Tooth Deep Dive

The fist task creates a user. It uses the variable clojure_uberjar_webapp_app_user, which is defined in defaults/main.yml. It sticks with the "name things consistently" approach and defaults to clojure_uberjar_webapp_app_name. This user will become the owner of many of the files uploaded by sweet-tooth-clojure.clojure-uberjar-webapp-app. The intention is to add a little extra bit of security above making root the owner of everything, though I’m honestly not sure if I did a great job of that.

The next task, Create project directory, create’s the directory that will hold the application jar file. It also holds the directory created by the next task, Create config directory; this directory holds files used to configure the application, as opposed to e.g. configuring nginx. It’ll store files that set the environment variables for:

Now let’s look at the sweet-tooth-clojure.clojure-uberjar-webapp-app role.

For the next task, we actually install openjdk-8 and a few other packages:

This introduces another new key, with_items. When a task has the with_items key, it means run this task’s module for every element in the with_items sequence, assigning the element to the item variable; it’s a weird yaml-based looping construct. In this case, the module is apt and we’re using it to install each of the listed packages. openjdk-8-jre is the only tool that we absolutely need; the rest are useful for debugging. wget lets use easily download URLs, vim is a lightweight text editor, and curl is great for interacting with HTTP resources.

Notice that one of the arguments to the apt module is state=installed. You’re declaring the end state that you want the server to be in, as opposed to writing imperative code to take the actions which will result in the end state.

Check existence of local env file has a few new keys: local_action, register, ignore_errors, and become.

To understand this task, it helps to know that its purpose is to check whether there is a file on your local filesystem (the app env file) which should be used to set environment variables that will be read as configuration by your application. This task stores the result of its check in a variable, and the very next task, Copy app env file, checks that variable before executing.

Now let’s look at each of the arguments to the Check existence of local env file task. local_action is the module we’re using, and it’s unsurprisingly used to run commands on your local machine. Its arguments are stat and path="{{ clojure_uberjar_webapp_app_env_local_path }}", and the result is that the task checks the status of the file at clojure_uberjar_webapp_app_env_local_path.

The register key tells the task where to store the result: the variable app_env_local_file. This is one way that Ansible lets you communicate among tasks: the result of one task is registered in a global variable that can then be read by other tasks. You’ll see in a second that app_env_local_file is read by Copy app env file.

Next, we have to set the ignore_errors key to True because Ansible’s default behavior is to throw an exception and stop applying the playbook if the path given to stat doesn’t exist. become is set to False because we’re running this task locally, and we don’t want to escalate privileges.

Finally, this task has the configure tag. I haven’t had occassion to use this tag explicitly, but hey, it doesn’t hurt, and it kind of serves as documentation.

Once the task Check existence of local env file has finished, Ansible executes the task Copy app env file. This task obviously copies a file from your local machine to the remote machine, and its larger purpose in deploying and running a Clojure application is to give you a way to set environment variables for each environment (dev, staging, prod, etc). For example, you could set different Google Analytics tracking ids for dev, staging, and prod. Let’s look at from two perspectives: how the task is defined, and the role the task plays in setting up a functioning server.

The task has a couple new keys, when and template:

when defines the conditions for when the current task should run, in this case when app_env_local_file.stat.exists is truthy. The value app_env_local_file.stat.exists was set by the previous task.

We’re giving file two arguments, src and dest. It copies the file located at src on your local machine to dest on the remote machines. Easy peasy!

Now let’s look at the task from the perspective of the role it plays in running a Clojure application. To fully understand this, we’ll need to jump back and forth between the sweet-tooth-clojure.clojure-uberjar-webapp-app files and the character sheet example files.

In the task’s defintion, shown in the snippet above, the value of src is clojure_uberjar_webapp_app_env_local_path. By default, that value is files/env, but in our character sheet example we override it so that we can use a different file for each environment. Let’s walk through the whole process to see how that happens.

Let’s say that you run a playbook by executing the commands

You’re trying to deploy your application to your dev environment, and you want to use your dev environment variables. When you call ./deploy dev, the shell script runs this command:

This says, apply the sweet-tooth.yml playbook with the inventory at inventories/dev. inventories/dev defines the dev inventory with files structured in a way that Ansible understands. (If you need a refresher on this, read Chapter 2).

If you look at character-sheet-example/infrastructure/ansible/inventories/dev/group_vars/webservers, you’ll see this:

So that’s how we override the default value of clojure_uberjar_webapp_app_env_local_path to point to the dev file. Here’s a look at files/env/dev.sh (located in character-sheet-example/infrastructure/ansible):

All the file does is export environment variables; in this case, we’re setting a Google Analytics id for the dev environment.

This file gets uploaded to clojure_uberjar_webapp_app_env_path, which defaults to /var/www/localhost/config/.app for the dev inventory. Later on we’ll look at your remote machine handles the file /var/www/localhost/config/.app so that your application can read its environment variables.

One final note: in my own projects, I set git to ignore these environment files. My environment files contain slightly more sensitive information like API keys which shouldn’t get stored in version control. There are more secure ways to achieve the same result (I’ve heard that Ansible Vault is a good solution) but for small hobby sites it works OK. Please yell at me if this is a terrible idea!

The next task uploads a one-line file that exports the HTTP_SERVER_PORT environment variable:

This introduces a new key, template, which tells ansible to use the template the module. The template module works similarly to the file module: it copies the file from src on the local machine to dest on the destination machine. It differs from the file module in that the file getting copied is processed by the Jinja2 template system. Let’s look at the file’s contents to see why we might want to do this:

(You can find this file under templates/http-env.j2 in the clojure-uberjar-webapp-app repo.)

This file contains the familiar double-braces that we’ve been using in our tasks to interpolate variables. We want to do this so that we can run multiple Clojure applications on the same machine. For example, we could run a staging server on port 9000, and a production server on port 9010.

Later on, you’ll see how this file gets read so that its environment variable is available to your Clojure application.

We use Ubuntu’s upstart service to start and stop our Clojure application. From the upstart home page: upstart handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running.

To use upstart, we need to upload a file that tells upstart how to start the Clojure application server, and that file’s template is located at templates/app-upstart.conf.j2:

The start on and stop on bits are out of scope for this guide, so let’s skip those.

The line respawn means restart this application if it dies for some reason. Very important if you write code as buggy as I do!

The script…​ end script block tells upstart how to start your application. Within that block, we set environment variables with

set -a, is what ensures that your app can read the vars. You can read a little more about what set -` does in this StackExchange thread. The next line sources environment variables from a file that combines every configuration file that you’ve uploaded. (If you’re wondering what that period does at the beginning of the line, here’s a good explanation.) The configuration file’s contents will look something like this:

After we set environment variables, there’s this line: {{ clojure_uberjar_webapp_app_command }}, which will actually kick off the java process that runs your application. By default, that variable expands to something like:

We pass the Clojure application one argument, server because I coded that application so that it will start an HTTP server if the first argument is server. You can see this in character sheet example’s source code, in the file src/backend/character_sheet/core.clj:

Ignore final; it just does some error handling. The main (ah ha ha!) thing to note is that the -main function’s first argument, cmd, corresponds to the first command line argument. The -main function switches on cmd and evaluates the appropriate expression. If the argument is "server", it executes a function that starts a server.

The last bit of the config file specifies that the Clojure application should send standard out and standard error to /var/log/localhost/localhost.log.

This upstart file gets copied to /etc/init/{{ clojure_uberjar_webapp_app_service_name }}.conf on the remote machine because that’s where upstart expects to find its config files.

This task creates a directory to store the application’s log file and ensures that the application has permission to write to the file.

This task copies the uberjar from your local machine to the remote machine. By defaul, it looks for the file at files/app.jar on your local.

This is the first task that’s tagged deploy. You might remember that we first provision our machines before deploying to them; tasks tagged deploy don’t get run when we provision our machines, because it’s possible that the programs that deploy tasks depend on haven’t been installed. For example, if we ran all the deploy tasks while provisioning, then Ansible would try to install the Clojure app’s Datomic schemas, but Datomic wouldn’t have been installed yet.

This task uses the assemble module to concatenate all the files in the config directory into one file. The combined file gets sourced by the upstart script, as described in the Upload web app upstart config file section.

This task copies a shell script that you can use to do a quick sanity test on your app. Here’s the shell script’s template:

The here’s how it renders for the dev deployment of the character sheet app (it gets saved to /var/www/localhost/check.sh):

The script gets called in the next task, Run check. The script sources files that set environment variables, and then runs your uberjar with one argument: deploy/check. You need to set up your application to respond to that argument. If you look at character-sheet-example/src/backend/character_sheet/core.clj, you’ll see what gets run:

If youlook at the -main function, you’ll see that it takes one argument, cmd, and a list optional arguments, args. cmd is the first argument sent to the program on the command line, so with our check.sh script the argument is deploy/check and the value of cmd is "deploy/check". The -main function checks the value of cmd, and you can see in the final line of the snippet that it executes (final (config/full)). The config/full function validates that all required environment variables are set, and throws an exception if any are missing.

Thus, the check.sh script ensures that all environment variables are set. If the script fails, then Ansible stops applying tasks, which is good! If it didn’t stop, it would try to restart your application server with a bad build, and your site would be unavailable or you’d get weird errors.

The last thing to note about the Run check task is that it sends three notifications:

Notifications deserve their own section, so imma type out three equals signs because that what asciidoc uses to designate a new section heading:

You can find this defined in the clojure-uberjar-webapp-datomic-free repo in the file handlers/main.yml.

The Install schemas task runs your application, passing it one argument: db/install-schemas. It’s up to your application to handle it correctly. You can check out character-sheet-example/src/backend/character_sheet/core.clj again to see what it does. I won’t go into the details here; basically, it ensures that all the schemas you’ve defined exist in the Datomic database.

This task runs at the end of your deployment. It tells upstart to restart your application. Sadly, there’s usually some downtime when you restart, but it’s still good enough for me! If you come up with a clover way to do zero-downtime deployments that don’t eat up all your machine’s memory, please let me know!

Not much new here: this just installs nginx and openssl. openssl is needed if you want to serve sites using ssl. Crazy how that works out.

nginx’s default site configuration doesn’t provide anything useful, and it sometimes makes it harder to debug issues. Kill it!

We replace nginx’s default base config with one that I like better. Explaining what each of the config settings does it out of scope.

This task is kind of insane. Let’s look at the code:

That src key look cuh-ray-zay. The idea is that it’s deciding whether to use the app-nginx-no-ssl.conf.j2 template, or the app-nginx-ssl.conf.j2 template. It uses Jinja’s weirdo fake programming language to check whether the variable clojure_uberjar_webapp_nginx_use_ssl has been set to a truthy value, and uses that to pick which config template to use.

Let’s look at the no-ssl config file that’s been rendered and saved on the vagrant VM at /etc/nginx/sites-available/localhost.conf. If you’ve never looked at an nginx config file, it might be daunting; just stick with me and it’ll make sense.

To make sense of this file, it helps to take a step back and consider nginx’s role in handling HTTP requests. nginx’s job is to return some response for an HTTP request. Config files tell nginx how to handle HTTP requests.

For example, you might have a single nginx server for two domains, ilovehats.com and iloveheads.com. Your nginx configuration files might tell nginx to serve content from /var/www/ilovehats.com for one domain, and to server content from /var/www/iloveheads.com for the other.

In our case, we need to tell nginx to forward requests to our application server, or the upstream server. You define the upstream at ➊, giving it the name localhost_upstream. The next line, server 127.0.0.1:3000 tells nginx what address and port to use when it forwards requests. The Clojure application is running on the same machine and listening to port 3000, hence server 127.0.0.1:3000.

server_name localhost , at ➋, tells nginx to use this server configuration when the HTTP host is localhost. For my Community Picks site, this reads server_name www.communitypicks.com.

proxy_pass http://localhost_upstream, at ➌, tells nginx to handle requests by forwarding them to the server named localhost_upstream - the server that’s defined at ➊.

If you’re browsing the vagrant example, overall flow is:

The rest of the file sets some useful headers and sets log file locations.

The last task for this role simple creates a symlink at /etc/nginx/sites-enabled/localhost.conf pointing to /etc/nginx/sites-available/localhost.conf. This follows an nginx convention; the intention is to make it easy for you to disable a site by deleting its symlink in /etc/nginx/sites-enabled and the re-enable it just by re-symlinking.