Experiment: multi-environment configuration in Python #161
Conversation
app/src/__main__.py
Outdated
| all_configs = src.config.load_all() | ||
| logger.info("loaded all", extra={"all_configs": all_configs}) |
There was a problem hiding this comment.
This code should move to a unit test, where it checks that each one is an instance of RootConfig.
app/src/config/default.py
Outdated
| def default_config(): | ||
| return RootConfig( | ||
| app=AppConfig(), | ||
| database=PostgresDBConfig(), | ||
| logging=LoggingConfig( | ||
| format="json", | ||
| level="INFO", | ||
| enable_audit=True, | ||
| ), | ||
| ) |
There was a problem hiding this comment.
Would this require any process that runs to be able to load all environment variables? For example, if I have a config for connecting to a separate service, and it's only used in a few select processes, would I put that here in the default config? Let's assume that I don't want to set defaults in the config and want it to error if any fields aren't set.
app/src/config/env/local.py
Outdated
| # | ||
| # Configuration for local development environments. | ||
| # | ||
| # This file only contains overrides (differences) from the defaults in default.py. | ||
| # | ||
|
|
||
| import pydantic.types | ||
|
|
||
| from .. import default | ||
|
|
||
| config = default.default_config() | ||
|
|
||
| config.database.password = pydantic.types.SecretStr("secret123") | ||
| config.database.hide_sql_parameter_logs = False | ||
| config.logging.format = "human_readable" | ||
| config.logging.enable_audit = False |
There was a problem hiding this comment.
Would there be a way to have this check for a file named local_overrides.py and use that? I'm thinking about local testing where I want to tinker with environment variables, but not check them in - especially when working with passwords or keys to access services. We could add this local_overrides.py file to the .gitignore.
Adding something like this to the end of this file:
if os.path.exists("local_overrides.py"):
importlib.import_module("local_overrides") # whatever the right path/params are
| @@ -2,7 +2,7 @@ | |||
|
|
|||
|
|
|||
| class AppConfig(PydanticBaseEnvConfig): | |||
| environment: str | |||
| environment: str = "unknown" | |||
There was a problem hiding this comment.
Maybe environment: Optional[str] = None? That might be annoying with the linter though
| continue | ||
|
|
||
| env_var_name = field.field_info.extra.get("env", prefix + name) | ||
| for key in (env_var_name, env_var_name.lower(), env_var_name.upper()): |
There was a problem hiding this comment.
Might be nice to be stricter here and require a certain casing, avoid letting someone accidentally set the environment variable in lowercase and then not understand why it's not being overridden if it's also set in uppercase elsewhere.
| class PydanticBaseEnvConfig(BaseSettings): | ||
| class Config: | ||
| env_file = env_file | ||
| validate_assignment = True |
There was a problem hiding this comment.
Could require a prefix option here too to provide some namespacing for the various configs, e.g., have the PostgresConfig environment variables all start with a PG_DB_ prefix.
There was a problem hiding this comment.
(The implementation on PFML does this)
Ticket
Changes
src.configmodule that defines config for each environment entirely in Python.Context for reviewers
Start with default.py and local.py to understand how this works.
Goals of this approach:
Testing
Note: some tests are still broken.