Keep up to date with every new upload!

Join free & follow O'Reilly Radar Podcast - O'Rei
Share
  • 2 years ago
Building systems for massive scale data applications

Building systems for massive scale data applications

The O’Reilly Data Show podcast: Tyler Akidau on the evolution of systems for bounded and unbounded data processing.In this episode of the O'Reilly Data Show, I sat down with Tyler Akidau one of the lead engineers in Google's streaming and Dataflow technologies. He recently wrote an extremely popular article that provided a framework for how to think about bounded and unbounded data processing (a follow-up article is due out soon). We talked about the evolution of stream processing, the challenges of building systems that scale to massive data sets, and the recent surge in interest in all things real time:On the need for MillWheel: A new stream processing engine

At the time [that MillWheel was built], there was, as far as I know, literally nothing externally that could handle the scale that we needed to handle. A lot of the existing streaming systems didn't focus on out-of-order processing, which was a big deal for us internally. Also we really wanted to hit a strong focus on consisten

Comments