How this Github project can take your local Prefect workflows to the next level by provisioning AWS & Prefect Cloud infrastructure and the important role Github Actions plays as part of it.

TL;DR: This project sets up a dataflow management system powered by Prefect and AWS. Its deployment has been…


How Kinesis Analytics for SQL fits into our Data Pipeline and what challenges we faced with it

This is the second and final part of my series about designing and building an event generation engine.


Our challenges introducing real-time analytics over flight data to deliver valuable insights

At Spidertracks we work to help make aviation safer by providing real-time aircraft tracking as well as transforming flight data into valuable insights. …


A Workflow consists of steps, configured to respect a predefined order and accomplish a specific business objective. They vary from something as simple as defining an IT request process in a small company to complex data transformations aimed to deliver key business insights.

Leaving complexity aside, some characteristics are common…


This is Part 3 of this blog series and we are now going to make use of the architecture described on Parts 1 and 2 to stream database events to Kafka and consume them through KSQL.

All steps described on this post can be reproduced by deploying my github project


This is part 2 of my blog series about building a Kafka playground on AWS. If you missed part 1 please check this.

Now that we know the Kafka components and the AWS services of choice let’s look into a graphical representation of this architecture and explain how it works.

VPC


As part of my recent studies I decided to explore Apache Kafka. Kafka is an open-source streaming platform used to collect/analyse high volumes of data, and Its ecosystem is composed by multiple components. …


Trade Me, like many companies around the globe, is leveraging the capabilities of the public cloud. The countless services and the cost-saving opportunities are only a few advantages the cloud brings to businesses.

This was a collaborative project between Business Intelligence and Data Engineering and this blog post - written…


One of the benefits of running in the cloud is that everything can be translated into code. Going for infrastructure as code early on in your project is a smart call because although using the cloud provider´s GUI to explore services can be helpful, relying on such manual interactions to…


Hi ! I am Maikel.

The year is 2019, technology has taken over the world and writing a blog post is something straightforward that anyone can do. While part of this is true because here I am, writing my first blog post, technology hasn’t completely taken over the world. …

Maikel Penz

Data Engineering Team Lead @ Plexure | AWS Certified Solution Architect | LinkedIn: https://www.linkedin.com/in/maikel-alexsander-penz/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store