Building a Kafka playground on AWS — Part 2: The AWS Architecture

AWS Architecture for Kafka Deployment

VPC

AWS MSK Cluster

Configuration

Limitations

Cluster configuration list

AWS ECS Cluster

Configuration

  • Task Definition: A task definition describes how a container should launch, memory and CPU required, the docker image, environment variables, etc.
ECS Task Definition
  • Service: Controls the state of running tasks. A Service points to a single Task Definition and it controls how many containers with the related task definition must be running at once, replaces failed tasks and exposes containers to external services, like a Load Balancer, for example.
ECS Services
ECS Container Instances

Container Communication

  • Communicating containers with Kafka (MSK)
List of MSK Brokers
KSQL Task Definition referencing list of Kafka brokers
  • How to make containers talk to each other
ECS Service pointing to a Route 53 endpoint
ECS Task Definition
  • How to make containers accessible through the browser
Schema Registry UI ports
Load Balancer Listener
ECS Service link to Load Balancer
Accessing Schema Registry UI

--

--

--

Data Engineering Team Lead @ Plexure | AWS Certified Solution Architect | LinkedIn: https://www.linkedin.com/in/maikel-alexsander-penz/

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Ingesting data from Google Cloud Storage into Python and Pushing into BigQuery using GCP (functions…

Difference between Web1.0 , 2.0, 3.0

CS373 Spring 2022: Nisha Ramesh

Flask Configuration

Separate function handler modules when using Python Socketio

Parcility — Web-based jailbreak repository

How Math and Computational Geometry Influence Digital Art

Shared Protobuf Library for Java and Golang

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Maikel Penz

Maikel Penz

Data Engineering Team Lead @ Plexure | AWS Certified Solution Architect | LinkedIn: https://www.linkedin.com/in/maikel-alexsander-penz/

More from Medium

The What, Why, and How of Implementing Microservices Architecture on AWS

Microservices

When We used ActiveMQ over kafka

When Elastic Search masquerades as a database…

Redis as a Polyglot Data Persistence Layer