greatcodeNavigate back to the homepage

Integration Testing in Go

Mike Christensen
August 27th, 2020 · 3 min read

Why Write Integration Tests?

While unit testing ensures that individual units of code work correctly in isolation, the goal of integration testing is to ensure that the application behaviour is correct when those units are wired together.

Note that integration tests concern the integration of units of code within a single application. This is distinct from system or end-to-end tests, in which the integration of the entire software product, with all its constituent applications and services, is under test.

Testing Pyramid

Integration tests typically take longer to write and to run than unit tests. As such, a general rule of thumb is to write integration tests for the code paths that return a successful result. There are much fewer such paths than ones which lead to error, and so the numerous error paths should be covered by your unit tests.

How to Write Integration Tests

Like unit tests, your integration tests should live alongside the file that contains the (entrypoint to) the code under test. To distinguish from unit tests, I typically use a *_integration_test.go suffix.

The entrypoint should be as high-level in the application as is reasonable. For example, if you have implemented a gRPC handler, your test should live alongside the handler and invoke it directly.

To Mock or Not?

If your application depends on external services, these will need to either be made available for the application to talk to or they will need to be mocked. How do you decide?

Mocking, of course, entails some work. You need to describe what the mocked service should return for every input that is expected - and in an integration test context, that could be a lot of inputs. As such, spinning up an actual copy of the service can make a lot of sense.

However, say your app, service A, depends on the API exposed by service B. You decide to spin up a copy of service B for your integration tests. But what if service B depends on service C? Which in turn needs service D? Pretty soon you’re testing your entire stack, and the integration has become an end-to-end test!

A general rule of thumb is to only spin up dependencies that are themselves standalone, isolated services. Additionally, only spin up dependencies if doing so is cheaper than developing the mock. Examples include databases or caches. Finally, third-party APIs should also be mocked in order to avoid dependency on the stability of services outside of your control (this is the jurisdiction of end-to-end testing), as well as to avoid unnecessary costs in the cases where the APIs are paid-for according to usage.

Integration Test

Testing Against a SQL Database

It’s a good idea to create a separate database for each of your integration tests. This will allow your tests to execute in parallel without causing data conflicts. Of course, the separate databases can and should all exist in the same Postgres (or other provider) instance, to avoid running multiple copies of your database software.

Spinning Up the Database

First, spin up an instance of your database. I often have a Makefile target that looks something like:

6.PHONY: integ_deps_pg
7integ_deps_pg: export POSTGRES_ADDR=$(integ_pg_addr)
8integ_deps_pg: export POSTGRES_PORT=$(integ_pg_port)
9integ_deps_pg: export POSTGRES_USER=$(integ_pg_user)
10integ_deps_pg: export POSTGRES_PASSWORD=$(integ_pg_password)
11integ_deps_pg: export POSTGRES_DATABASE=$(integ_pg_db)
13 @echo "Spinning up database..."
14 @-docker kill integ_pg && docker rm integ_pg
15 @-docker volume create pg_dummy_data
16 @-docker run -d --shm-size=2048MB -p $(POSTGRES_PORT):5432 --name integ_pg -e POSTGRES_PASSWORD=$POSTGRES_PASSWORD -v $(pg_dummy_data):/var/lib/postgresql/data postgres:12-alpine
17 @sleep 3
18 @docker exec -it --user postgres integ_pg psql -U postgres -c "CREATE USER ${POSTGRES_USER} WITH PASSWORD '${POSTGRES_PASSWORD}' CREATEDB;"
19 @docker exec -it --user postgres integ_pg psql -c "ALTER USER ${POSTGRES_USER} SUPERUSER;"
20 @docker exec -it --user postgres integ_pg psql -c "CREATE DATABASE ${POSTGRES_DATABASE} OWNER ${POSTGRES_USER};"

The initial ‘main’ database, here called dummy, can then be used to bootstrap creation of a new database for each test, which gets torn down after the test has completed:

1// testutil/db.go
3var migrationsMu sync.Mutex
4var mainDBClient *sql.DB
6func init() {
7 config, err := pgx.ParseConfig(
8 driver.PSQLBuildQueryString(
9 viper.GetString(internal.PostgresUserVar),
10 viper.GetString(internal.PostgresPasswordVar),
11 viper.GetString(internal.PostgresDatabaseVar),
12 viper.GetString(internal.PostgresHostVar),
13 viper.GetInt(internal.PostgresPortVar),
14 "disable",
15 ) + " client_encoding=UTF8",
16 )
17 if err != nil {
18 panic(err)
19 }
20 mainDBClient = stdlib.OpenDB(*config)
23// NewTestDBClient returns a connection to a test database with the given name. It
24// drops the database automatically when the test is finished.
25func NewTestDBClient(t testing.TB, name, owner, pw, host string, port int) *sql.DB {
26 t.Helper()
27 migrationsMu.Lock()
28 defer migrationsMu.Unlock()
30 // use the main database to bootstrap test databases
31 name = strings.ToLower(name)
32 _, err := mainDBClient.Exec(fmt.Sprintf("CREATE USER %s WITH PASSWORD '%s' CREATEDB; ALTER USER %s SUPERUSER;", owner, pw, owner))
33 if err != nil {
34 // only allow an error indicating that the role already exists
35 require.Contains(t, err.Error(), fmt.Sprintf(`ERROR: role "%s" already exists (SQLSTATE 42710)`, owner))
36 }
37 _, err = mainDBClient.Exec(fmt.Sprintf("CREATE DATABASE %s OWNER %s;", name, owner))
38 require.NoError(t, err)
40 // connect to new database
41 config, err := pgx.ParseConfig(
42 driver.PSQLBuildQueryString(
43 owner,
44 pw,
45 name,
46 host,
47 port,
48 "disable",
49 ) + " client_encoding=UTF8",
50 )
51 require.NoError(t, err)
52 dbClient := stdlib.OpenDB(*config)
54 // teardown databases when done
55 t.Cleanup(func() {
56 migrationsMu.Lock()
57 defer migrationsMu.Unlock()
58 ctx, cancel := context.WithTimeout(context.Background(), time.Second*20)
59 defer cancel()
60 dbClient.Close()
61 _, err = mainDBClient.ExecContext(ctx, "REVOKE CONNECT ON DATABASE "+name+" FROM public; ")
62 require.NoError(t, err)
63 _, err = mainDBClient.ExecContext(ctx, `
64 SELECT pg_terminate_backend( FROM pg_stat_activity
65 WHERE pg_stat_activity.datname = $1;
66 `, name)
67 require.NoError(t, err)
68 _, err = mainDBClient.ExecContext(ctx, "DROP DATABASE "+name)
69 require.NoError(t, err)
70 })
71 return dbClient

Creating the Schema

To create your database schema, you can use a schema dump of your database created with e.g. pg_dump. If you’re following this migration strategy, your schema is available in ./internal/db/schema.sql.

1// CreateDBSchema creates the schema.
2func CreateDBSchema(t *testing.T, dbClient *sql.DB) {
3 sql := db.GetSQL() // a packr *Box
4 schemaSQL, err := sql.FindString("schema.sql")
5 _, err = dbClient.Exec(schemaSQL)
6 require.NoError(t, err)
7 _, err = dbClient.Exec("SET search_path TO public;")
8 require.NoError(t, err)

Loading Data

If your test requires your database to be prepopulated with specific data, the best way to do this is to load some test fixtures. These are simple SQL files that insert that data you need. Given a fixture like:

1-- ./test/testdata/fixtures/foo.sql
2INSERT INTO foo (bar, baz) VALUES ("bar1", "baz1");
4INSERT INTO foo (bar, baz) VALUES ("bar2", "baz2");
6INSERT INTO foo (bar, baz) VALUES ("bar3", "baz3");

You can load this data into your test database with a function like:

1// LoadFixture executes the SQL in the given file against the given database.
2func LoadFixture(t *testing.T, db *sql.DB, filename string) {
3 t.Helper()
4 file, err := ioutil.ReadFile(path.Join(viper.GetString(internal.TestDataDirVar), "fixtures", filename))
5 require.NoError(t, err)
6 requests := strings.Split(string(file), ";\n")
7 for _, request := range requests {
8 _, err := db.Exec(request)
9 require.NoError(t, err, filename)
10 }

Putting this all together, your integration test setup might look something like:

1func TestFoo(t *testing.T) {
2 t.Parallel()
3 ctx := context.Background()
4 name := strings.Join(strings.Split(t.Name(), "/"), "_")
5 dbClient := testutil.NewDBClient(t,
6 name,
7 viper.GetString(internal.PostgresUserVar),
8 viper.GetString(internal.PostgresPasswordVar),
9 viper.GetString(internal.PostgresHostVar),
10 viper.GetInt(internal.PostgresPortVar),
11 )
12 testutil.CreateDBSchema(t, dbClient)
13 testutil.LoadFixture(t, dbClient, "foo.sql")
15 // Now do your test!

Testing Against AWS

Often, your application may rely on various services made available by your cloud provider. One option in this case is to mock the library you use to interact with the cloud provider.

If you’re using AWS, another option I’ve successfully used in the past is LocalStack. LocalStack lets you spin up a local copy of the AWS stack with which your application integration tests can interact.

The easiest way to get started with LocalStack is using Docker. First, copy the Docker Compose file into your third_party folder. Then you can spin up LocalStack using docker-compose and use the AWS CLI to perform any setup required (like creating S3 buckets):

1.PHONY: integ_deps_localstack
3 @-docker kill localstack_main
4 @TMPDIR=/private$(TMPDIR) docker-compose -f ./third_party/localstack/docker-compose-localstack.yml up -d
5 @echo "Waiting 20s for LocalStack..." && sleep 20
6 @eval $(aws ecr get-login --no-include-email)
7 @source ./scripts/ $(env) && AWS_ACCESS_KEY_ID=dummy AWS_SECRET_ACCESS_KEY=dummy aws --endpoint-url=http://localhost:4566 s3 mb s3://bucket-foo


Writing integration tests is an essential part of ensuring your application components work correctly together. Done correctly, they will give you confidence that future changes do not break existing functionality and that your application is robust, stable and correct.

Do you have other integration test tips and tricks? Hit me up on Twitter!

More articles from Christensen Codes

Unit testing in Go

How to test pieces of code in isolation.

August 26th, 2020 · 2 min read

Dependency Injection in Go

How to write modular, maintainable and testable Go applications.

August 25th, 2020 · 2 min read
© 2020–2022 Christensen Codes
Link to $ to $ to $