Let's build a Service Oriented Data Pipeline
Like many software projects, data pipelines built by Business Intelligence
teams often start out as quickly built monoliths, but over time they can be
made simpler and more maintainable by splitting them into a series of tightly
defined services. In this talk we'll look at the challenges we've faced
scaling Data Analytics at Hootsuite, then move into a live coding session,
where we'll stitch together a data pipeline as a series of Scala apps, deployed
to AWS Lambda, connected using Airbnb's open source Airflow tool.
Yasha is a Software Developer on the Data Lab team at Hootsuite, where he helps
the Product and Business teams make data driven, customer centric decisions.
Originally an Oceanographer working on carbon flux modelling, he got into
Software Development through scientific computing, and has become especially
passionate about building stable systems to process and analyze large volumes