Containerizing Data Workflow

Schedule:

6:00 - Doors & Food
6:30 - Talk 1
7:15 - Talk 2
7:45 - Wrap & Chat

Talk 1: Pros and cons of containerizing data workflows (and how to have the best of both worlds)
Speaker: Tian Xie, Data Engineer @ Enigma
Abstract:
At Enigma, we run over one hundred workflows to ingest public data into our system. Running so many workflows also means managing dependencies and deployment for each of those workflows. Over time, we have iterated over several solutions to this problem and this is our story. Spoiler: docker is involved, but (*plot twist*) it only leads to another set of problems in the 2nd act.
Bio:
Tian Xie has been working in the NYC tech start-up scene for the last eight years on consumer video rendering, on-demand shipping, and now data engineering at Enigma Technologies.

Talk 2: TBD