Configuring Go Microservices with Viper and Cobra

Store Config in the Environment

Factor 3 of a Twelve Factor App declares the following rule: Store config in the environment. It’s a golden rule. Config management is one of those things that starts off simple but can become a tangled mess with time. Even if you take the sensible-seeming approach of storing config as JSON/YAML, rather than as in-code constants (eek!), you’ll still run into headaches eventually:

Config files accidentally get checked into version control
Config files become difficult to correctly construct and document
Config files are scattered across multiple locations
Config files become overly complex and nested, making them brittle and difficult to modify
Config files produce a combinatorial explosion of variations, e.g. mike-staging-v2-local-tunnel.json.

Ouch. Anyone familiar with Kubernetes is familiar with this pain.

In highly complex and configurable applications like k8s, config files may be the only real option. However in most cases, and for microservice apps in particular, the far better approach is to store config as environment variables. Each env var should be single-purposed and granular. Additionally they should be fully orthogonal to one another; there should be no overlap of responsibility, as that is a recipe for confusion. This has several benefits:

Env vars are easy to change between deploys
Env vars are easy to set up and configure in cloud environments
Env vars are easy to document and construct
Env vars are easy to manage independently between deploys
Env vars are language- and OS-agnostic

Sold?

Define Env Vars with Viper 🐍

Viper is a fantastic configuration solution for Go applications, used by many high-profile projects including Hugo and Docker Notary. It supports multiple configuration strategies, including reading from env vars.

I typically define the env vars that make up my application configuration in a top-level env.go inside the /internal folder.

The first thing to do is define the env vars required by your application:

1const (
2    EnvVar         = "env"
3    LogLevelVar    = "log_level"
4    
5    HealthPortVar = "health_port"
6    PortVar       = "port"
7
8    PostgresHostVar     = "postgres_host"
9    PostgresPortVar     = "postgres_port"
10    PostgresUserVar     = "postgres_user"
11    PostgresPasswordVar = "postgres_password"
12    PostgresDatabaseVar = "postgres_database"
13
14    // ... etc
15)

Note that the Viper library will capitalise these strings when they are later used to identify environment variables.

Next, we can define the variables in which the values of these variables can be stored:

1var (
2    Env         string
3    LogLevel    string
4
5    HealthPort int
6    Port       int
7
8    PostgresHost     string
9    PostgresPort     int
10    PostgresUser     string
11    PostgresPassword string
12    PostgresDatabase string
13    
14    // ... etc
15)

Now we need to register these variables with Viper, setting some default values. It’s usually best to use sensible values for a local environment as your defaults:

1func init() {
2    wd, _ := os.Getwd()
3    viper.AutomaticEnv()
4    viper.SetDefault(EnvVar, "local")
5    viper.SetDefault(LogLevelVar, "debug")
6    
7    viper.SetDefault(HealthPortVar, 8080)
8    viper.SetDefault(PortVar, 50054)
9
10    viper.SetDefault(PostgresHostVar, "localhost")
11    viper.SetDefault(PostgresPortVar, 5432)
12    viper.SetDefault(PostgresUserVar, "dummy")
13    viper.SetDefault(PostgresPasswordVar, "dummy")
14    viper.SetDefault(PostgresDatabaseVar, "dummy")
15
16    // ... etc
17}

It’s a good idea to define a validation function that returns an error if any of the supplied values are badly formed. It should have the following signature:

1func ValidateEnv() error {
2    // perform environment validation
3}

Finally, you can read the value of an env var from code like:

1pgHost := viper.GetString(internal.PostgresHostVar)

Define Commands with Cobra 🐍

Cobra is an excellent library for building CLI applications, used by industry-leading applications like Kubernetes and GitHub CLI. It allows us to easily define and manage application commands, subcommands, flags and more. What’s more, it integrates tightly with Viper.

Usually, your application entrypoint lives in /cmd/<appname>/main.go. Here’s how you might define your root Cobra command and validate the environment on startup:

1var (
2    logger logrus.FieldLogger = logrus.StandardLogger()
3    
4    rootCmd = &cobra.Command{
5        Use:   "myapp",
6        Short: "My Application",
7        PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
8            logger = logrus.StandardLogger()
9            return internal.ValidateEnv()
10        },
11        RunE: func(cmd *cobra.Command, args []string) error {
12            // start the application here...
13        },
14    }
15)

For maximum flexibility, it’s nice if we can override env var values via flags that we pass to our application. For example, if we wish to run locally on a different port, we can simply run ./bin/myapp --port=1234. To set this up with Cobra we can bind our Viper env vars to Cobra flags with something like:

1func init() {
2    rootCmd.PersistentFlags().StringVarP(&internal.Env, internal.EnvVar, "e", viper.GetString(internal.EnvVar), "Environment")
3    viper.BindPFlag(internal.EnvVar, rootCmd.PersistentFlags().Lookup(internal.EnvVar))
4    rootCmd.PersistentFlags().StringVarP(&internal.LogLevel, internal.LogLevelVar, "l", viper.GetString(internal.LogLevelVar), "Log level")
5    viper.BindPFlag(internal.LogLevelVar, rootCmd.PersistentFlags().Lookup(internal.LogLevelVar))
6    
7    rootCmd.PersistentFlags().IntVar(&internal.HealthPort, internal.HealthPortVar, viper.GetInt(internal.HealthPortVar), "Health check port")
8    viper.BindPFlag(internal.HealthPortVar, rootCmd.PersistentFlags().Lookup(internal.HealthPortVar))
9    rootCmd.PersistentFlags().IntVar(&internal.Port, internal.PortVar, viper.GetInt(internal.PortVar), "gRPC server port")
10    viper.BindPFlag(internal.PortVar, rootCmd.PersistentFlags().Lookup(internal.PortVar))
11
12    rootCmd.PersistentFlags().StringVar(&internal.PostgresHost, internal.PostgresHostVar, viper.GetString(internal.PostgresHostVar), "Postgres host")
13    viper.BindPFlag(internal.PostgresHostVar, rootCmd.PersistentFlags().Lookup(internal.PostgresHostVar))
14    rootCmd.PersistentFlags().IntVar(&internal.PostgresPort, internal.PostgresPortVar, viper.GetInt(internal.PostgresPortVar), "Postgres port")
15    viper.BindPFlag(internal.PostgresPortVar, rootCmd.PersistentFlags().Lookup(internal.PostgresPortVar))
16    rootCmd.PersistentFlags().StringVar(&internal.PostgresUser, internal.PostgresUserVar, viper.GetString(internal.PostgresUserVar), "Postgres user")
17    viper.BindPFlag(internal.PostgresUserVar, rootCmd.PersistentFlags().Lookup(internal.PostgresUserVar))
18    rootCmd.PersistentFlags().StringVar(&internal.PostgresPassword, internal.PostgresPasswordVar, viper.GetString(internal.PostgresPasswordVar), "Postgres password")
19    viper.BindPFlag(internal.PostgresPasswordVar, rootCmd.PersistentFlags().Lookup(internal.PostgresPasswordVar))
20    rootCmd.PersistentFlags().StringVar(&internal.PostgresDatabase, internal.PostgresDatabaseVar, viper.GetString(internal.PostgresDatabaseVar), "Postgres database")
21    viper.BindPFlag(internal.PostgresDatabaseVar, rootCmd.PersistentFlags().Lookup(internal.PostgresDatabaseVar))
22
23    // ... etc
24}

For more details, check out the Cobra docs.

Finally, don’t forget to execute your root command:

1func main() {
2    if err := rootCmd.Execute(); err != nil {
3        logger.Fatal(err)
4    }
5}

Using a `.env`

In a local environment (only!), it can be handy to have a single file that defines the combination of env vars you need to run the application. I typically do this with a .env in the project root (which is included in the .gitignore!) which looks something like:

1ENV=local
2LOG_LEVEL=debug
3HEALTH_PORT=8080
4PORT=50054
5POSTGRES_HOST=localhost
6POSTGRES_PORT=5432
7POSTGRES_USER=dummy
8POSTGRES_PASSWORD=dummy
9POSTGRES_DATABASE=dummy

Additionally, I include a bash script in /scripts/env.sh that will export those variables in my current session. Note that this script supports commenting out lines in the .env using a # character:

1#!/bin/bash
2
3if [ -f "$1" ]; then
4    export $(egrep -v '^#' $1 | xargs)
5fi

Then I typically load the environment by sourcing the env script before executing the command:

1env=.env
2
3.PHONY: run
4run: build
5    @source ./scripts/env.sh $(env) && ./bin/myapp

which can be invoked with a simple make run. 🥳

Conclusion

Application configuration is something that every microservice developer will need to consider. Fortunately, these fantastic tools make config a breeze in Go. Put the structure in place, then forget about it and start writing your app! 😎

How do you manage config in your Go apps? Hit me up on Twitter!

Configuring Go Microservices with Viper and Cobra

Store Config in the Environment

Define Env Vars with Viper 🐍

Define Commands with Cobra 🐍

Using a `.env`

Conclusion

More articles from Christensen Codes

Structuring Microservices in Go

The Complex World of Music Licensing and Royalties, Simplified @ Blokur

Store Config in the Environment

Define Env Vars with Viper 🐍

Define Commands with Cobra 🐍

Using a .env

Conclusion

More articles from Christensen Codes

Structuring Microservices in Go

The Complex World of Music Licensing and Royalties, Simplified @ Blokur

Using a `.env`