segmentio/kafka-go

★ 8,549GoAudience · developerComplexity · 3/5Setup · moderate

Mindmap

mindmap
  root((kafka-go))
    What It Does
      Kafka client for Go
      Read and write messages
      No C dependencies
    APIs
      Low-level Conn
      High-level Reader
      High-level Writer
    Features
      Auto reconnect
      Offset tracking
      Graceful shutdown
    Requirements
      Go 1.15 or later
      Running Kafka server
      No cgo needed

mindmap root((kafka-go)) What It Does Kafka client for Go Read and write messages No C dependencies APIs Low-level Conn High-level Reader High-level Writer Features Auto reconnect Offset tracking Graceful shutdown Requirements Go 1.15 or later Running Kafka server No cgo needed

Click or tap to explore — scroll the page freely

Things people build with this

USE CASE 1

Connect a Go microservice to a Kafka topic to consume or publish messages without adding C build dependencies.

USE CASE 2

Use the high-level Reader to consume Kafka messages with automatic reconnection and offset tracking so you never skip or re-read messages.

USE CASE 3

Build a Go data pipeline that reads from one Kafka topic, processes messages, and writes results to another.

Tech stack

Getting it running

Difficulty · moderate Time to first run · 30min

Requires a running Kafka server to connect to, the library does not include Kafka itself.

No license information was provided in the description.

In plain English

kafka-go is a Go programming library for working with Kafka, a system used to pass large streams of messages between different parts of a software application. Think of Kafka as a high-throughput message queue: one part of your system sends messages in, other parts read them out. This library is the connector that Go programs use to talk to a Kafka server. The library was created by Segment because the existing Go options for Kafka had problems. The most popular one at the time had a confusing low-level API, was poorly documented, and did not support modern Go features. Another option required bundling a C library into your Go code, which complicates builds and deployment. kafka-go was written to avoid these issues by following the conventions of Go's standard library, making it feel familiar to Go developers. It offers two levels of control. The lower-level connection type lets you write and read messages directly, with full control over exactly how you connect and what you send. The higher-level reader and writer types handle more things automatically, including reconnecting if the connection drops and tracking where you left off in the message stream so you do not re-read or skip messages. The higher-level API also supports cancellation, meaning you can stop a read operation cleanly if your program needs to shut down. The library is tested against several versions of Kafka and requires Go 1.15 or later to use. It is written entirely in Go with no external C dependencies, which makes it straightforward to include in a Go project without special build steps. Segment built this for their own production infrastructure and released it publicly. It is a developer tool, not an end-user application.

Copy-paste prompts

Prompt 1

Using segmentio/kafka-go, write a Go consumer that reads messages from a topic called 'orders' and prints each one to stdout, with automatic reconnection on failure.

Prompt 2

Show me how to set up a kafka-go Writer to produce messages to Kafka from a Go service, including error handling and graceful shutdown on SIGTERM.

Prompt 3

I'm using kafka-go's Reader and need to stop consuming cleanly when my service receives SIGTERM. Show me how to cancel the read loop using a context.

Prompt 4

When should I use kafka-go's low-level Conn API versus the high-level Reader and Writer? Give me a concrete example of each.

Open on GitHub → Explain another repo

← segmentio on gitmyhub — every repo by this author, as a profile.

Verify against the repo before relying on details.