Configuring Go Applications with Viper and Cobra

tags: #go #golang

Store Config in the Environment

Factor 3 of a Twelve Factor App declares the following rule: Store config in the environment. It’s a golden rule. Config management is one of those things that starts off simple but can become a tangled mess with time. Even if you take the sensible-seeming approach of storing config as JSON/YAML, rather than as in-code constants (eek!), you’ll still run into headaches eventually:

Config files accidentally get checked into version control
Config files become difficult to correctly construct and document
Config files are scattered across multiple locations
Config files become overly complex and nested, making them brittle and difficult to modify
Config files produce a combinatorial explosion of variations, e.g. mike-staging-v2-local-tunnel.json.

Ouch. Anyone familiar with Kubernetes is familiar with this pain.

In highly complex and configurable applications like k8s, config files may be the only real option. However in most cases, and for microservice apps in particular, the far better approach is to store config as environment variables. Each env var should be single-purposed and granular. Additionally they should be fully orthogonal to one another; there should be no overlap of responsibility, as that is a recipe for confusion. This has several benefits:

Env vars are easy to change between deploys
Env vars are easy to set up and configure in cloud environments
Env vars are easy to document and construct
Env vars are easy to manage independently between deploys
Env vars are language- and OS-agnostic

Sold?

Define Env Vars with Viper 🐍

Viper is a fantastic configuration solution for Go applications, used by many high-profile projects including Hugo and Docker Notary. It supports multiple configuration strategies, including reading from env vars.

I typically define the env vars that make up my application configuration in a top-level env.go inside the /internal folder.

The first thing to do is define the env vars required by your application:

const (
	EnvVar         = "env"
	LogLevelVar    = "log_level"
	
	HealthPortVar = "health_port"
	PortVar       = "port"

	PostgresHostVar     = "postgres_host"
	PostgresPortVar     = "postgres_port"
	PostgresUserVar     = "postgres_user"
	PostgresPasswordVar = "postgres_password"
	PostgresDatabaseVar = "postgres_database"

	// ... etc
)

Note that the Viper library will capitalise these strings when they are later used to identify environment variables.

Next, we can define the variables in which the values of these variables can be stored:

var (
	Env         string
	LogLevel    string

	HealthPort int
	Port       int

	PostgresHost     string
	PostgresPort     int
	PostgresUser     string
	PostgresPassword string
    PostgresDatabase string
    
    // ... etc
)

Now we need to register these variables with Viper, setting some default values. It’s usually best to use sensible values for a local environment as your defaults:

func init() {
	wd, _ := os.Getwd()
	viper.AutomaticEnv()
	viper.SetDefault(EnvVar, "local")
	viper.SetDefault(LogLevelVar, "debug")
    
    viper.SetDefault(HealthPortVar, 8080)
	viper.SetDefault(PortVar, 50054)

    viper.SetDefault(PostgresHostVar, "localhost")
	viper.SetDefault(PostgresPortVar, 5432)
	viper.SetDefault(PostgresUserVar, "dummy")
	viper.SetDefault(PostgresPasswordVar, "dummy")
	viper.SetDefault(PostgresDatabaseVar, "dummy")

	// ... etc
}

It’s a good idea to define a validation function that returns an error if any of the supplied values are badly formed. It should have the following signature:

func ValidateEnv() error {
    // perform environment validation
}

Finally, you can read the value of an env var from code like:

pgHost := viper.GetString(internal.PostgresHostVar)

Define Commands with Cobra 🐍

Cobra is an excellent library for building CLI applications, used by industry-leading applications like Kubernetes and GitHub CLI. It allows us to easily define and manage application commands, subcommands, flags and more. What’s more, it integrates tightly with Viper.

Usually, your application entrypoint lives in /cmd/<appname>/main.go. Here’s how you might define your root Cobra command and validate the environment on startup:

var (
    logger logrus.FieldLogger = logrus.StandardLogger()
    
    rootCmd = &cobra.Command{
		Use:   "myapp",
		Short: "My Application",
		PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
			logger = logrus.StandardLogger()
			return internal.ValidateEnv()
		},
		RunE: func(cmd *cobra.Command, args []string) error {
			// start the application here...
		},
	}
)

For maximum flexibility, it’s nice if we can override env var values via flags that we pass to our application. For example, if we wish to run locally on a different port, we can simply run ./bin/myapp --port=1234. To set this up with Cobra we can bind our Viper env vars to Cobra flags with something like:

func init() {
	rootCmd.PersistentFlags().StringVarP(&internal.Env, internal.EnvVar, "e", viper.GetString(internal.EnvVar), "Environment")
	viper.BindPFlag(internal.EnvVar, rootCmd.PersistentFlags().Lookup(internal.EnvVar))
	rootCmd.PersistentFlags().StringVarP(&internal.LogLevel, internal.LogLevelVar, "l", viper.GetString(internal.LogLevelVar), "Log level")
	viper.BindPFlag(internal.LogLevelVar, rootCmd.PersistentFlags().Lookup(internal.LogLevelVar))
	
	rootCmd.PersistentFlags().IntVar(&internal.HealthPort, internal.HealthPortVar, viper.GetInt(internal.HealthPortVar), "Health check port")
	viper.BindPFlag(internal.HealthPortVar, rootCmd.PersistentFlags().Lookup(internal.HealthPortVar))
	rootCmd.PersistentFlags().IntVar(&internal.Port, internal.PortVar, viper.GetInt(internal.PortVar), "gRPC server port")
	viper.BindPFlag(internal.PortVar, rootCmd.PersistentFlags().Lookup(internal.PortVar))

    rootCmd.PersistentFlags().StringVar(&internal.PostgresHost, internal.PostgresHostVar, viper.GetString(internal.PostgresHostVar), "Postgres host")
	viper.BindPFlag(internal.PostgresHostVar, rootCmd.PersistentFlags().Lookup(internal.PostgresHostVar))
	rootCmd.PersistentFlags().IntVar(&internal.PostgresPort, internal.PostgresPortVar, viper.GetInt(internal.PostgresPortVar), "Postgres port")
	viper.BindPFlag(internal.PostgresPortVar, rootCmd.PersistentFlags().Lookup(internal.PostgresPortVar))
	rootCmd.PersistentFlags().StringVar(&internal.PostgresUser, internal.PostgresUserVar, viper.GetString(internal.PostgresUserVar), "Postgres user")
	viper.BindPFlag(internal.PostgresUserVar, rootCmd.PersistentFlags().Lookup(internal.PostgresUserVar))
	rootCmd.PersistentFlags().StringVar(&internal.PostgresPassword, internal.PostgresPasswordVar, viper.GetString(internal.PostgresPasswordVar), "Postgres password")
	viper.BindPFlag(internal.PostgresPasswordVar, rootCmd.PersistentFlags().Lookup(internal.PostgresPasswordVar))
	rootCmd.PersistentFlags().StringVar(&internal.PostgresDatabase, internal.PostgresDatabaseVar, viper.GetString(internal.PostgresDatabaseVar), "Postgres database")
	viper.BindPFlag(internal.PostgresDatabaseVar, rootCmd.PersistentFlags().Lookup(internal.PostgresDatabaseVar))

    // ... etc
}

For more details, check out the Cobra docs.

Finally, don’t forget to execute your root command:

func main() {
	if err := rootCmd.Execute(); err != nil {
		logger.Fatal(err)
	}
}

Using a `.env`

In a local environment (only!), it can be handy to have a single file that defines the combination of env vars you need to run the application. I typically do this with a .env in the project root (which is included in the .gitignore!) which looks something like:

ENV=local
LOG_LEVEL=debug
HEALTH_PORT=8080
PORT=50054
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_USER=dummy
POSTGRES_PASSWORD=dummy
POSTGRES_DATABASE=dummy

Additionally, I include a bash script in /scripts/env.sh that will export those variables in my current session. Note that this script supports commenting out lines in the .env using a # character:

#!/bin/bash

if [ -f "$1" ]; then
    export $(egrep -v '^#' $1 | xargs)
fi

Then I typically load the environment by sourcing the env script before executing the command:

env=.env

.PHONY: run
run: build
	@source ./scripts/env.sh $(env) && ./bin/myapp

which can be invoked with a simple make run. 🥳

Conclusion

Application configuration is something that every microservice developer will need to consider. Fortunately, these fantastic tools make config a breeze in Go. Put the structure in place, then forget about it and start writing your app! 😎