Airflow On Kubernetes




Airflow on Kubernetes

16 May 2019


Added 01-Jan-1970

Due to popular demand, Daniel Imberman, the committer of Kubernetes Executor and the guy who played a vital role in integrating K8s with Airflow, is going to talk from the United States on how you too can use Airflow with K8s following best practices.

This time it is a Webinar - so Airflow'ers over the world are welcome to join us.

Talk #1: Airflow on Kubernetes: A different kind of operator
Speaker: Daniel Imberman, Senior Software Engineer at Bloomberg & Committer of Apache Airflow project

Airflow offers a wide range of native operators for services ranging from Spark and HBase to Google Cloud Platform (GCP) and Amazon Web Services (AWS). Until recently, the Airflow user experience has been hindered by the need to launch and maintain statically-sized Celery-based Airflow clusters. These clusters were both expensive (over and under-utilization) and complex (multiple points of failure).

To address these issues, we developed and published a native Kubernetes Operator and Kubernetes Executor for Apache Airflow. These products allow one-step Airflow deployments, dynamic allocation of Airflow worker pods, full power over run-time environments, and per-task resource management.

The link of the Webinar would be emailed & posted here on the day of the webinar (will try to post a day before - if it is possible !).