Structuring Microservices in Go

Project Layout

The Go community has, by-and-large, agreed on some broad standards for how to structure Go projects based on a set of common historical and emerging project layout patterns. These best practices for project layout are described in detail here.

Here’s how I typically structure a Go microservice, based on these best practices:

1.env
2go.mod
3go.sum
4README.md
5Makefile
6Dockerfile
7/bin
8/scripts
9/cmd
10    /myapp
11        main.go
12    /anotherapp
13        main.go
14/pkg
15/internal
16    env.go
17    /app
18        /myapp
19            index.go
20        /anotherapp
21            index.go
22    /pkg
23    /db
24/test
25    /testdata
26    /mocks
27    /pkg

`.env`

I highly recommend using an env file to set your service environment variables locally (just make sure you’ve added it to your .gitignore so that it’s not checked into version control). It has the format:

1VAR1=value1
2VAR2=value2

I use the following bash snippet to load the environment in my current shell session before starting the service. This script usually lives in ./scripts/env.sh and can be sourced from the Makefile (more on that below). Note that this snippet supports commenting out lines in the .env using a # character.

1#!/bin/bash
2
3if [ -f "$1" ]; then
4    export $(egrep -v '^#' $1 | xargs)
5fi

`Makefile`

The ubiquitous Makefile has been here since the dinosaurs roamed the earth. Best to keep it as simple as possible. If your targets start growing too large, stick them in a bash script and call that from your target instead.

At minimum, I typically implement some form of the following targets:

build To build the application binaries.
test To run unit/integration tests.
run To execute an application binary.
watch To run the application with file watching (usually using reflex with go run).
docker To build a container image.

For commands that require it, I typically load the environment by sourcing the env script before execution:

1env=.env
2
3.PHONY: run
4run: build
5    @source ./scripts/env.sh $(env) && ./bin/myapp

Then make run will execute the application binary with the environment loaded from .env. The env file can be overridden with e.g. make run env=.env2.

I’ll include more Makefile tips and tricks in a later post.

`/cmd`

This structure gives you the flexibility to create multiple application entrypoints, building separate binaries for each of them. For example, given a build Make target like:

1cmd=myapp
2
3.PHONY: build
4build:
5    @go build -o ./bin/$(cmd) ./cmd/$(cmd)/main.go

…you can then build the default myapp binary with make build, or the anotherapp binary with make build cmd=anotherapp.

In addition to working well in a monorepo context, this pattern can also allow you to build lightweight binaries for specific tasks, without having to build the larger codebase (as would be required if using subcommands in the main application binary like ./bin/myapp subcommand).

`/pkg`

Mimicking the $GOPATH package folder, code that lives in here is public. It is intended for reuse by external projects. You are exposing an API that may be depended on by others and as such have a responsability to correctly follow semantic versioning.

`/internal`

This is code that is for internal use by your application only. The bulk of your application logic will reside here. The subfolder structure inside /internal is similar to that of the top level.

The application entrypoint(s) typically live in the app subfolder, e.g. ./app/myapp.
Code internally shared by your application lives in ./pkg.
I typically put SQL schema definitions & migrations in a ./db folder, and leverage tools like packr to package them in the application binary. More on that in another post.
I typically put application configuration logic in an env.go and leverage Viper. More on that here.

`/test`

Don’t forget this folder.

Not sure if quality is really good or testing was really bad

Testing is a broad and important topic, which I’ll certainly cover in future posts. For now, a high-level description will suffice:

Fixtures and ground-truth live in ./testdata.
Mocks, generated with e.g. mockgen live in ./mocks.
Common code that is only ever required for testing might live in packages inside ./pkg. One example of this is sql-mock custom matching logic.

Note that test files themselves do not typically live here (with the possible exception of end-to-end tests). Instead, test files are colocated with the code under test using the _test suffix in the filename, as described in the Go documentation.

Conclusion

Application structure is important. A good structure makes it easier for developers to find their way around unfamiliar code. Following established conventions is perhaps most important for this very reason.

That said, many smaller projects do not require this level of organisation. Start off simple, and add structure as your code base grows.

How do you structure your Go apps? Hit me up on Twitter!