Loading…
Attending this event?
Seattle, WA
December 10–13, 2018
Click Here for More Information & Registration
View Venue Map
View analytic
Wednesday, December 12 • 3:40pm - 4:15pm
Airflow on Kubernetes: Dynamic Workflows Simplified - Daniel Imberman, Bloomberg & Barni Seetharaman, Google

Sign up or log in to save this to your schedule and see who's attending!

Log in to leave feedback.
Apache Airflow is an open source workflow orchestration engine that allows users to write Directed Acyclic Graph (DAG)-based workflows using a simple Python library. Airflow offers a wide range of native operators for services ranging from Spark and HBase to Google Cloud Platform (GCP) and Amazon Web Services (AWS). Until recently, the Airflow user experience has been hindered by the need to launch and maintain statically-sized Celery-based Airflow clusters. These clusters were both expensive (over and under-utilization) and complex (multiple points of failure).

To address these issues, we developed and published a native Kubernetes Operator and Kubernetes Executor for Apache Airflow. These products allow one-step Airflow deployments, dynamic allocation of Airflow worker pods, full power over run-time environments, and per-task resource management.

Speakers
avatar for Daniel Imberman

Daniel Imberman

Senior Software Engineer, Bloomberg
Daniel Imberman is a Senior Software Engineer at Bloomberg, where he specializes in Data Science Infrastructure and Cloud Computing. Daniel received his BS/MS from UC Santa Barbara in 2015 with a concentration in distributed machine learning. | | Daniel was the lead contributor... Read More →
BS

Barni Seetharaman

Senior SWE, Google
Bharanidharan(Barni) Seetharaman is a Software engineer at Google where he works on the open source Kubernetes project. Prior to Google, he worked as Networking, Cloud platform and Distributed Systems engineer at companies including CISCO, Juniper, Alcatel Lucent. He has experience... Read More →



Wednesday December 12, 2018 3:40pm - 4:15pm
Tahoma 1/2 @ TCC The Conference Center (TCC) - Separate from WSCC building
  • Skill Level Any