Part 4. How to Version, Build, and Test Your Code

If you don’t have it already installed, follow these docs to install Git on your computer. Next, let Git know your name and email:

Create a new, empty folder on your computer just for experimenting with Git. For example, you could create a folder called git-practice within the system temp folder:

Within the git-practice folder, write some text to a file called example.txt:

It would be nice to have some version control for this file. With Git, that’s easy. You can turn any folder into a Git repo by running git init:

The contents of your folder should now look something like this:

You should see your original example.txt file, plus a new .git folder. This .git folder is where Git will record all the information about your branches, commits, revisions, and so on.

At any time, you can run the git status command to see the status of your repo:

The status command is something you’ll run often, as it gives you several useful pieces of information: what branch you’re on (main is the default branch when you create a new Git repo); any commits you’ve made; and any changes that haven’t been committed yet. To commit your changes, you first need to add the file(s) you want to commit to the staging area using git add:

Try running git status one more time:

Now you can see that example.txt is in the staging area, ready to be committed. To commit the changes, use the git commit command, passing in a description of the commit via the -m flag:

Git is now storing example.txt in its commit log. You can see this by running git log:

For each commit in the log, you’ll see the commit ID, author, date, and commit message. Take special note of the commit ID: each commit has a different ID that you can use to uniquely identify that commit, and many Git commands take a commit ID as an argument. Under the hood, a commit ID is calculated by taking the SHA-1 hash of the contents of the commit, all the commit metadata (author, date, and so on), and the ID of the previous commit.^[24] Commit IDs are 40 characters long, but in most commands, you can use just the first 7 characters, as that will be unique enough to identify commits in all but the largest repos.

Let’s make another commit. First, make a change to example.txt:

This adds a second line of text to example.txt. Run git status once again:

Now Git is telling you that it sees changes locally, but these changes have not been added to the staging area. To see what these changes are, run git diff:

You should use git diff frequently to check what changes you’ve made before committing them. If the changes look good, use git add to stage the changes and git commit to commit them:

Now, try git log once more, this time adding the --oneline flag to get more concise output:

Now you can see both of your commits, including their commit messages. This commit log is very powerful:

So far, all of your commits have been on the default branch (main), but in real-world usage, you typically do work across multiple branches, which is the focus of the next section.

Let’s practice creating and merging Git branches. To create a new branch and switch to it, use the git checkout command with the -b flag:

To see if it worked, you can use git status, as always:

You can also use the git branch command at any time to see what branches are available and which one you’re on:

Any changes you commit now will go into the testing branch. To try this out, modify example.txt once again:

Next, stage and commit your changes:

If you use git log, you’ll see your three commits:

But that third commit is only in the testing branch. You can see this by using git checkout to switch back to the main branch:

Check the contents of example.txt: it’ll have only two lines. And if you run git log, it’ll have only two commits. So each branch gives you a copy of all the files in the repo, and you can modify them in that branch, in isolation, as much as you want, without affecting any other branches.

Of course, working forever in isolation doesn’t usually make sense. You eventually will want to merge your work back together. One way to do that with Git is to run git merge to merge the contents of the testing branch into the main branch:

You can see that Git was able to merge all the changes automatically, as there were no conflicts between the main and testing branches. If you now check the contents of example.txt, it should have three lines in it, and if you run git log, you’ll see three commits.

Now that you know how to run Git locally, on your own computer, you’ve already learned something useful: if nothing else, you now have a way to store revisions of your work that’s much more effective than essay-v2.doc, essay-v3.doc, etc. But to see the full power of Git, you’ll want to use it with other developers, which is the focus of the next section.

The single most important best practice with version control is: use it.

It’s easy, it’s cheap or free, and the benefits for software engineering are massive. If you’re writing code, always store that code in version control. No excuses.

Commit messages are important. When you’re trying to figure out what caused a bug or an outage, or staring at a confusing piece of code, git log and git blame can be your best friends—but only if the commit messages are well written. Good commit messages consist of two ingredients:

Here’s an example of such a commit message:

Just adding these two simple ingredients, summary and context, will make your commit messages already considerably better. If you’re able to follow these two rules consistently, you can level up your commit messages even further by following the instructions in How to Write a Good Commit Message. And for especially large projects and teams, you can also consider adopting Conventional Commits as a way to enforce even more structure in your commit messages.

One of the keys to being a more effective programmer is learning how to take a large problem and break it down into small, manageable parts. As it turns out, one of the keys to using version control more effectively is learning to break down large changes into small, frequent commits.

In fact, there are two ideals to aim for:

Atomic commits and atomic PRs are ideals, and sometimes you’ll fall short of them, but it’s always a good idea to strive towards these ideals, as they give you a number of benefits:

Having your code reviewed by someone else is a highly effective way to catch bugs, reducing defect rates by as much as 50-80%.^[25] Code reviews are also an efficient mechanism to spread knowledge, culture, training, and a sense of ownership throughout the team.

There are different ways of doing code reviews:

Whatever process you pick for code reviews, you should define your code review guidelines up front, so you have a process that is consistent and repeatable across the entire team: that is, everyone knows what sorts of things to look for (e.g., automated tests), what sorts of things not to look for (e.g., code formatting, which should be automated), and how to communicate feedback effectively. For an example, have a look at Google’s Code Review Guidelines.

For many companies these days, the code you write is your most important asset—your secret sauce. Moreover, it’s also a highly sensitive asset: if someone can slip some malicious code into your codebase, it can be devastating, as that will bypass most other security protections you have in place.

Therefore, you should consider enabling the following security measures to protect your code:

Now that you know how to use a version control system to help everyone on your team work on the same code, the next step is ensure that you’re all working on that code in the same way: that is, building the code the same way, running the code the same way, using the same dependencies, and so on. This is where build system can come in handy, which is the focus of the next section.

To create a unit test, you need a unit of code. Let’s create a basic one just for learning in a new file called reverse.js with the contents shown in Example 74:

This code contains two functions:

Let me ask the same question again: how do you know this code actually works? I suppose you could stare at the code for a while, simulate what it does in your head, and declare it as either working or not. But if you’ve ever written code, you’ve written buggy code, so you know that relying on the simulator in your head isn’t enough. You’ve got to test the code.

One option is to test the code manually. For example, you could fire up a REPL (read-eval-print-loop), which is an interactive shell where you can manually execute code. For Node.js, you can get a REPL by running the node executable with no arguments:

You can now type code into this REPL, hit ENTER, and Node.js will execute the and show you the result:

You could use this to test the reverseWords function as follows:

(When you’re done experimenting with the REPL, use Ctrl+D to exit).

Hey, that seems to work! But this is hardly a thorough test, and doing this manually every time is time-consuming and error-prone. So let’s capture the steps you did in a REPL in an automated test.

First, you need to install some testing libraries: you’ll use Jest as a testing framework and, a little later on, SuperTest as a library for testing HTTP apps. Use npm install to add these dependencies, but this time, use the --save-dev flag to save them as dev dependencies; this way, they’ll be available during development (where you run tests), but won’t be packaged into your app for production deployments (where you don’t run tests):

This will update package.json with a new devDependencies section, as shown in Example 75:

Next, update the test command in package.json to run Jest, as shown in Example 76:

Now you can start writing tests. Create a file called reverse.test.js with the contents shown in Example 77:

Here’s how this code works:

Use npm test to run this test:

The test passed! And it took all of 0.192 seconds, which is a whole lot faster than manually typing in a REPL.

Add a second test for reverseWords as shown in Example 78:

Re-run npm test:

Oops. Looks like the code has a bug after all: it doesn’t handle trailing whitespace correctly. A powerful benefit of automated testing is that you can create tests for all the various corner cases and find out if your code really does what you expect. The fix for this issue shown in Example 79:

How can you be sure that this fix works? Re-run the tests!

This is a good example of the typical way you write code when you have a good suite of automated test to lean on: you make a change, you re-run the tests, you make another change, you rerun the tests again, and so on, adding new tests as necessary. With each iteration, your test suite gradually improves, you build more and more confidence in your code, and you can go faster and faster.

Rapid feedback loops are a big part of the DevOps methodology, and a big part of being more productive as a programmer. And this not only helps you fix the bug with whitespace faster, but now you also have a regression test in place that will prevent that bug from coming back. This is a massive boost to productivity that often gets overlooked.

A useful feature available with some automated testing tools is the ability to measure code coverage: that is, what percent of your code got executed by your tests. If your tests are only executing 10% of your code, then how much confidence can you have that your code really works, and that there aren’t bugs hiding in the other 90%? To have Jest measure code coverage, add the --coverage flag to the test command in package.json, as shown in Example 80:

Run the tests one more time:

You now get a table at the end of the output that shows each file and what percentage of the code was executed during the automated tests. You should also see a new coverage folder that contains even more information, including some HTML reports within coverage/lcov-report//fundamentals-of-devops. Try opening this file in a browser:

You should see a page that looks similar to Figure 38:

As you can see, you’re only testing about two-thirds of your code. To get more details into what you missed, click on reverse.js, and you should see something similar to Figure 39:

This page shows you your own code: the green "2x" and "1x" indicate those parts of the code were executed during testing twice or once, respectively. The highlighted code in red tells you code that wasn’t executed at all: of course, there are no tests for the reverseCharacters function!

Now that code coverage has helped you see where your tests are lacking, head into reverse.test.js, and add a new unit test for reverseCharacters, as shown in Example 81:

Run the tests one more time:

And there you go, all tests are passing, and code coverage is now at 100%.

Now that you have a feel for unit testing, let’s go back to the original problem of testing app.js, and add an end-to-end test for the app: that is, a test that makes an HTTP request to the app, and checks the response. To do that, you first need to split out the part of app.js that listens on a port into a separate file called server.js. That way, you’ll be able to run multiple copies of your app concurrently during automated testing without getting errors due to trying to listen on the same port.

First, update app.js to solely configure the Express app, and to export it, as shown in Example 82:

Next, create a new file called server.js that imports the code from app.js and has it listen on a port, as shown in Example 83:

Make sure to update the start command in package.json to now use server.js instead of app.js, as shown in Example 84:

Now, finally, you can add a test for the app in a new file called app.test.js, as shown in Example 85:

Here’s how this test works:

Re-run the tests one more time:

Excellent! All your tests are passing, you have 100% code coverage, and all of this runs in just 0.345 seconds.

As an example, let’s use Terrascan as a static analysis tool for OpenTofu. First, create a config file for Terrascan called terrascan.toml, with the contents shown in Example 86:

This file supports a number of configuration settings for Terrascan, but the only one you need for now is to configure Terrascan to only show high severity errors. The lower severity ones aren’t relevant for our simple example here, but you may wish to look at them for production code.

Next, install Terrascan, and run it in the lambda-sample folder as follows:

The preceding command uses the following flags:

When you run Terrascan, it should identify one policy violation:

Terrascan has found that the environment variables you’re passing to the Lambda function are not encrypted. If those environment variables contained any secrets, this would be a serious issue, so it’s great to have that flagged for us. That said, in this simple example, you’re not passing any secrets around, so you can tell Terrascan to skip this check by adding its Rule ID to the skip-rules section of your Terrascan configuration file as shown in Example 87:

If you re-run Terrascan, it’ll still show the policy violation, but this time, it won’t cause the check to fail.

Static analysis is good, but a unit test is even better, as it actually executes your code and checks its functionality, which allows you to catch a much wider variety of errors.

As an example, let’s add a unit test that uses OpenTofu’s built-in test command. Note that these tests will deploy real resources into your AWS account, so in some sense, they are closer to integration tests, but because you’re going to be testing just a single unit—a single module—I’ll stick with the term unit test to disambiguate from tests that test the integration of multiple modules together.

First, you need to add a small helper module for testing HTTP endpoints. The blog post series’s sample code repo includes a module called test-endpoint in the ch4/tofu/modules/test-endpoint folder that can make an HTTP request to an endpoint you specify. Normally, you could use this module directly from GitHub, but currently, the test command has a limitation where it can only use local modules, so git clone the sample code repo (if you haven’t already) to the devops-book folder and make a copy of the test-endpoint module:

Next, in the lambda-sample module, create a file called deploy.tftest.hcl, with the contents shown in Example 88:

This automated test is written in the same language (HCL) as the rest of OpenTofu code. You put these tests into files with the .tftest.hcl extension, and they primarily consist of run blocks that OpenTofu executes sequentially:

To run this test, authenticate to AWS as described in Authenticating to AWS on the command line, and use the tofu test command:

OpenTofu will run apply, deploy your real resources, and then, at the end of the test, run destroy to clean everything up again. These tests take several minutes to run, so be patient. But when they are done, you can have reasonable confidence that your module deploys infrastructure that really works.

One question that comes up often is, which testing approach should you use? Unit tests? Integration tests? E2E tests?

The answer is: a mix of all of them! Each type of test has different types of errors they can catch, plus different strengths and weaknesses, as summarized in Table 8:

The only way to be confident your code works as expected is to combine multiple types of tests. That doesn’t mean that you use all the different types of tests in equal proportion. In most cases, you want the proportion of tests to follow the test pyramid shown in Figure 40:

The idea of the test pyramid is that, as you go up the pyramid, the cost and complexity of writing the tests, the brittleness of the tests, and the runtime of the tests all increase. Therefore, you typically want to do as much testing as you can with the types of tests that are fast to write, fast to run, and stable, which means the majority of your testing (the base of the pyramid), should leverage the compiler, static analysis, plan tests, policy tests, and unit tests. You then have a smaller number of integration tests (the middle of the pyramid), and an even smaller number of very high-value end-to-end and performance tests (the top of the pyramid).

Moreover, you don’t have to add all the different types of tests at once. Instead, pick the ones that give you the best bang for your buck and add those first. Almost any testing is better than none. If all you can add for now is static analysis or unit tests, then that’s still better than no tests. Or maybe the highest value in your use case is a single end-to-end test that verifies your system really works: that’s a perfectly fine starting point, too. Start with something, anything that gives you value, and then add more and more tests incrementally.

Another question that comes up often is, what should you test? There are some testing purists out there who believe that every line of code must be tested and that your tests must achieve 100% code coverage. I am not one of them.

There is no doubt that tests offer a huge amount of value. But they also have a cost. Writing tests, maintaining them, and ensuring they run quickly and reliably can take a lot of time. In many cases, this overhead is worth it. In some cases, it’s not. In practice, I’ve found that deciding if you should invest in adding automated tests for a certain part of your code comes down to continuously evaluating your testing strategy and making trade-offs between the following factors:

One thing that impacts both the cost of writing tests and the benefits you get from those tests is when you write those tests: trying to add tests several years after you write the code tends to be much more expensive and not as beneficial; adding tests a day after you write the code tends to be cheaper and have more benefits; and as it turns out, adding tests before you write the code may be the lowest cost and most beneficial option of all, as discussed in the next section.

The mere act of trying to write tests for your code will force you to take a step back and ask some important questions: How do I structure the code so I can test it? What dependencies do I have? What are the common use cases? What are the corner cases?

If you find that your code is hard to test, it’s almost always a sign that it needs to be refactored for other reasons, too. For example, if the code uses lots of mutable state and side effects, it will not only be hard to test but also hard to reuse and reason about the code. If the code is difficult to test because it has many complex interactions with its dependencies, then it’s likely that the code is too tightly coupled and will be difficult to change. If the code is hard to test because there are too many use cases to cover, then that’s a sign that the code is doing too much and needs to be broken up.

In other words, tests not only help you write correct code, they also provide feedback that leads to a better design. You get the most benefit from this feedback if you write the test code before you write the implementation code. This is known as test-driven development (TDD).

The TDD cycle is:

One of the unexpected consequences of using TDD is that the design of your code emerges as a result of a repeated test-code-test cycle. Without TDD, you often end up shipping the first design you can think of; with TDD, the very act of trying to figure out how to write tests forces you to iterate on your design, and you often ship something more effective.

In fact, TDD affects not only the design of small parts of your code, but also the design of your entire system. Some TDD practitioners use TDD only for writing unit tests up front, but you can also use TDD with integration tests, end-to-end tests, and performance tests. Thinking about integration tests up front forces you to consider how the different parts of your system will communicate with one another. Thinking about end-to-end tests up front forces you to think through how to deploy everything. Thinking about performance tests up front forces you to think about where your bottlenecks are and what metrics you need to gather to identify them.

Being forced to think through all of this up front, simply because you try to write tests first, can have a profoundly positive impact on the final design. If you want to see some concrete examples of how TDD leads to emerging design, see Growing Object-Oriented Software, Guided by Tests by Steve Freeman and Nat Pryce (Addison-Wesley Professional), as well as my previous book, Hello, Startup: A Programmer’s Guide to Building Products, Technologies, and Teams (O’Reilly).

It’s also worth mentioning that writing tests up front increases the chances that you’ll have thorough test coverage, because it forces you to write code incrementally. Tests can be tedious to write, so it’s easier if you only have to do a few at a time: write a few tests and then a little bit of implementation, then a few more tests, then a little more implementation, and so on. It’s less effective to spend weeks writing thousands of lines of implementation code and then try to go back and do a marathon session of writing test cases for all of it. You’ll get bored and end up with far less code coverage. You’ll also miss many corner cases because you’ll forget the nuances of implementation code you wrote weeks ago.

Note that not all types of coding are a good fit for TDD. For example, if you’re doing exploratory coding, where you don’t yet know exactly what you’re building, and you’re just exploring the problem space by coding and messing with data, TDD might not be a good fit. If you don’t know what result you’re looking for, you can’t write a test for it. In that case, writing tests shortly after you write the code is still valuable.

Also, if you’re working on an existing codebase that doesn’t have any tests (a legacy codebase), then you might feel like TDD doesn’t apply, as the development is already done. However, you can still use TDD for any changes you make to the codebase. It’s very similar to the standard TDD process, but with a couple steps at the front:

This way, you incrementally build out the test coverage for the codebase, specifically in those parts of the code that you’re modifying. Each test you add for the existing functionality gives you a way to prove to yourself that you truly understand how that code works, which can make it less scary to start making changes; and each test you add for new functionality gives you all the standard benefits of TDD. And when you’re done, you leave the codebase with at least a few tests in place, which will make it less scary—make it feel less like legacy code—for the next developer.

TDD is also useful for bug fixing. If you found a bug in production, that means there was no test that caught the problem, so a good way to start on the fix is, as a first step, to write a failing test that reproduces the bug (test-driven bug fixing). That proves you understand the cause of the bug and, after you fix it, leaves you with a regression test that will prevent the bug from coming back.