Apache Airflow is the most popular Data Workflow Orchestrator with 120+ Python distributions. This talk explains how recent Python packaging tools like uv and prek have simplified managing a massive monorepo, and covers an innovative approach to implementing "statically linked" libraries in Python.
Apache Airflow is the most popular Data Workflow Orchestrator - developed under the Apache Software Foundation umbrella. We have 120+ Python distributions in our repo, and we often release ~ 100 of them every two weeks.
This presentation explains how recent Python packaging tools like uv and prek have simplified managing a massive monorepo, allowing us to eliminate thousands of lines of custom code. Additionally, the talk covers an innovative approach to implementing "statically linked" libraries in Python, enabling the use of different versions of the same library within a single system.