Apache Airflow Charm


#1

Lots of asks about an Airflow charm.
Thought I would start the conversation about what a good path forward may be.


#2

I think there is a fair bit of space to figure out what would Airflow as-a-charm actually going to function. Is the intent that you would just deploy it as infrastructure, or is the intent that you would deploy the infrastructure as charms and have them register themselves to Airflow.

I’m curious what the actual requests are.


#3

Our primary use case is to have airflow run spark-submit based pipelines.


#4

On the face of it, it would make the most sense to me if other services that are provisioned as charms (databases, file systems, …) could somehow register themselves with Airflow. Airflow wants (at least) read access to as many services as possible.

One of the tricky things to work out is how secrets should be managed. Airflow wants to have control over secrets itself, but I think that a Juju deployment would like everything in code.