How this Github project can take your local Prefect workflows to the next level by provisioning AWS & Prefect Cloud infrastructure and the important role Github Actions plays as part of it. TL;DR: This project sets up a dataflow management system powered by Prefect and AWS. Its deployment has been…

How Kinesis Analytics for SQL fits into our Data Pipeline and what challenges we faced with it — This is the second and final part of my series about designing and building an event generation engine. …

Our challenges introducing real-time analytics over flight data to deliver valuable insights — At Spidertracks we work to help make aviation safer by providing real-time aircraft tracking as well as transforming flight data into valuable insights. …

A Workflow consists of steps, configured to respect a predefined order and accomplish a specific business objective. They vary from something as simple as defining an IT request process in a small company to complex data transformations aimed to deliver key business insights. Leaving complexity aside, some characteristics are common…

This is Part 3 of this blog series and we are now going to make use of the architecture described on Parts 1 and 2 to stream database events to Kafka and consume them through KSQL. All steps described on this post can be reproduced by deploying my github project…

This is part 2 of my blog series about building a Kafka playground on AWS. If you missed part 1 please check this. Now that we know the Kafka components and the AWS services of choice let’s look into a graphical representation of this architecture and explain how it works. VPC …

As part of my recent studies I decided to explore Apache Kafka. Kafka is an open-source streaming platform used to collect/analyse high volumes of data, and Its ecosystem is composed by multiple components. These components are deployed separately to the core of Kafka and while this decoupled architecture has clear…

Trade Me, like many companies around the globe, is leveraging the capabilities of the public cloud. The countless services and the cost-saving opportunities are only a few advantages the cloud brings to businesses. This was a collaborative project between Business Intelligence and Data Engineering and this blog post - written…