ETL + Data warehouse

[lorenyu]

No description provided.

[acouch]

Noting we are working on a data platform that includes Metabase as a BI tool and Postgres as a warehouse with a Python transformation service at grants.gov. For the BI tool we evaluatede a couple of options. Metabase might be a nice choice for the platform since Quicksight is so poor as an AWS option (Looker is much better on GC) and the fact it is open source makes it easy to spin up locally and could easily be adopted for new teams. Jypter notebooks might be a nice option for more dev-centered reporting needs. Apache Superset is also a high quality open source option that has a less user-friendly UI than Metabase but is pretty on par feature-wise.

We chose postgres over Redshift b/c the performance gains for Redshift don't appear to kick in until higher volumes of data than we are expecting in the near future. Postgres is also a known entity that is also easy to spin up locally. The whole toolchain is captured in a lightweight docker-compose file which is a nice outcome / productivity boost.

[coilysiren]

I wrote both the ADRs listed above, so feel free to ask me for any more context as needed!