23 January 2019
Talk #1: Let it Whirl - Make Airflow Development & Testing Fun again
Speaker: Bas Beelen, Data Juggler at GoDataDriven
Developing Airflow DAGs can be hard. What if there's only one Airflow instance running both production and work-in-progress DAGs? What if you have restricted access to Airflow due to security policies? What if you are extracting part of your logic to a separate Python module, hence working simultaneously in two projects? How do you prevent from going crazy and how do you make Airflow development fun again?
He & his colleague Kris Geusebroek struggled with such situations the past six months at ING Wholesale Banking Advanced Analytics. To make Airflow DAG development fun and productive again, they invested time in locally developing and running these DAGs. Since they are probably not the only ones experiencing these difficulties, he would like to share with us the approach they took.
Talk #2: Introduction to Google Cloud Composer & Demo
Speaker: David Sabater, Cloud Customer Engineer, Big Data & ML at Google
David will give a brief introduction to Cloud Composer, the Apache Airflow managed service in Google Cloud Platform. We will then run a demo showing how to transfer data between the US and EU in a scalable way.