dpkp/kafka-python

kafka-python is a Python library for connecting to Apache Kafka, a system that many companies use to move large amounts of data between different parts of their software in real time. Kafka works like a message bus: one part of an application publishes messages to a named channel called a topic, and other parts read from that topic. kafka-python gives Python applications a way to do both, without needing to use the official Java tooling that Kafka ships with. The library provides two main components. KafkaProducer sends messages to a Kafka topic. You point it at a Kafka server, call the send method with a topic name and a value, and the message is queued for delivery. The producer handles batching, compression, and retries in the background. KafkaConsumer reads messages from a topic. You can iterate over it like a list, and each item you get back contains the message content along with metadata like which topic it came from, which partition, and its position in that partition. For consumers, the library supports consumer groups, which is a feature that lets multiple instances of an application share the work of reading a topic. Kafka tracks which messages each group has already processed so that if one instance restarts, it picks up where it left off rather than re-reading everything. Compression is supported in several formats including gzip, LZ4, Snappy, and Zstandard, though some of those require installing additional Python packages. The library also supports transactions, which allow a producer to send a batch of messages that either all succeed or all get rolled back together. Installation is through pip. The library supports Kafka versions from 0.8 through 4.0 and works with current Python versions. Documentation is hosted on readthedocs.