When I work with Apache Kafka, I like to use Conduktor Desktop, a graphical user interface for Apache Kafka, because it allows me to quickly configure secure connections and test APIs.įirst, we need to establish a connection into our VPC, for which you have several options.įor this post, I used the Client VPN methodology. In production, make sure you configure your security groups properly.Īfter you launch your MSK cluster, you can start working with IAM policies to manage Apache Kafka security. The IAM-based security mechanism runs on port 9098 on your Apache Kafka brokers, and consists of only one setting to enable in your cluster configuration.įor this post, I made the MSK security group as permissive as possible, to remove any constraints on security groups. When creating a MSK cluster, you can enable one of several security mechanisms. The following diagram illustrates our solution architecture.
#Conduktor gui apache kafka series how to
In this post, we explore how this new feature works in detail (setting up and creating a topic, producer, and consumer), and how to connect Conduktor, a graphical Apache Kafka desktop client, which allows us to quickly test our connectivity to Amazon MSK and ensure our first IAM administrator policy gets applied correctly. Amazon MSK took a step forward to make authN/Z easier by standardizing security management for MSK clusters and Apache Kafka using IAM. Amazon MSK began with support for mutual TLS authN/Z, and then offered SASL/SCRAM, which are standard Apache Kafka security options. Traditionally, Apache Kafka comes with its own ways of managing authentication and authorization. These new features complete the suite of existing security features for Amazon MSK such as Amazon VPC integration for private connectivity and network isolation, at-rest encryption via AWS Key Management Service (AWS KMS), and encryption in transit via TLS. This is a game-changer from a security perspective for AWS customers who use Apache Kafka: I recommend Amazon MSK customers use IAM Access Control unless they have a specific need for using mutual TLS or SASL/SCRAM authN/Z.Īs a cherry on the cake, IAM Access Control logs events related to Apache Kafka resource changes to Amazon CloudTrail, such as topic creation, adding partitions, and topic configuration modifications, which can be very helpful for adding an audit layer to your Apache Kafka clusters (something you could only obtain otherwise by parsing unstructured Apache Kafka logs). This eliminates the need for administrators to run an unfamiliar system to control access to Apache Kafka on Amazon MSK, and learn intricate details and specific commands to manage Apache Kafka access control lists (ACLs). This is a guest blog post by AWS Data Hero Stephane Maarek.ĪWS launched IAM Access Control for Amazon MSK, which is a security option offered at no additional cost that simplifies cluster authentication and Apache Kafka API authorization using AWS Identity and Access Management (IAM) roles or user policies to control access.
0 kommentar(er)
