envee, reading configuration data from environment variables or files made easy

At c.technology we love containers. Containers are great because they allow stable and reproducible systems. We use containers everywhere: for development, in our continuous integration and continuous development pipelines, as well as on our test and production servers. Thus, our containers needs to be adaptable to these different environments.

A convenient way to configure software packed in a container is using environment variables, which is also proposed by the Twelve-Factor App methodology for building software-as-a-service applications. However, storing sensitive data as environment variables is not advised. E.g. badly configured software may inadvertently leak environment variables and thus sensitive information such as passwords or other credentials. To mitigate this risk, platforms such as Docker or Kubernetes offer the ability to mount sensitive information, in this context often referred to as secrets, as files into containers.

This has implications for how we implement our software. In our development environment, where no sensitive information is present, the sole use of environment variables to configure our software is very convenient. Meanwhile, the same software should support the use of files to store secrets for our test and production environments.

To facilitate this task, we created a small library called envee. To showcase its functionality, let's look a the following task of configuring a database connection.

The following code snippet is fairly standard to read the configuration variables necessary to configure a database from the environment. However, it is not yet possible to read data also from files.

import os

POSTGRES_HOST = os.environ["POSTGRES_HOST"]
POSTGRES_PORT = int(os.environ.get("POSTGRES_PORT", 5432))
POSTGRES_DB = os.environ["POSTGRES_DB"]
POSTGRES_USER = os.environ["POSTGRES_USER"]
POSTGRES_PASSWORD = os.environ["POSTGRES_PASSWORD"]

Let's now extend this code snippet to read the password alternatively from a file:

import os

POSTGRES_HOST = os.environ["POSTGRES_HOST"]
POSTGRES_PORT = int(os.environ.get("POSTGRES_PORT", 5432))
POSTGRES_DB = os.environ["POSTGRES_DB"]
POSTGRES_USER = os.environ["POSTGRES_USER"]
postgres_password_file_path = "/run/secret/postgres_password"
if os.path.exists(postgres_password_file_path):
    with open(postgres_password_file_path) as f:
        POSTGRES_PASSWORD = f.read().strip()
else:
    POSTGRES_PASSWORD = os.environ["POSTGRES_PASSWORD"]

We can now retrieve the POSTGRES_PASSWORD variable either from the environment or a file. But only this variable. And the code looks already complicated! What if we want to read all variables from either files or environment variables?

Let's now look at how the same task looks like with envee:

import envee

@envee.environment
class Environment:
    POSTGRES_HOST: str
    POSTGRES_PORT: int = 5432
    POSTGRES_DB: str
    POSTGRES_USER: str
    POSTGRES_PASSWORD: str

env = envee.read(Environment)

For each field of the with the @envee.environment decorated class, envee will look if a file exists in the /run/secrets directory. This is the standard directory where Docker stores its secrets. If no corresponding file is found, envee will try to read the data from an environment variable if it exists. For each field, envee will look for a file with the lowercase field name or uppercase environment variable name. In the case of POSTGRES_HOST, this would be either the file /run/secrets/postgres_host or the environment variable POSTGRES_HOST.

Of course, this opinionated behavior can be overridden, if needed:

@envee.environment
class Environment:
    postgres_host: str = envee.field(
       file_path="/path/to/POSTGRES_HOST.txt",
       env_name="postgres_host"
    )

Reading variables from files is not the only feature of envee. As previously shown, it is possible to define default values by providing default values for the class fields. Primitive types, such as bool, float or int are automatically converted. Optional variables can be defined as well by defining the type of the class field as Optional. More complex transformations are possible by passing a custom transformation function to envee.field:

@envee.environment
class Environment:
    timestamp: datetime.datetime = envee.field(
        conversion_func=lambda x: datetime.datetime.fromisoformat(x)
    )

If you are interested in trying it out yourself, we released envee as open source on Github (https://github.com/c-technology/envee) and the Python Package Index (https://pypi.org/project/envee).