fluent/fluentd

★ 13,529RubyAudience · ops devopsComplexity · 3/5LicenseSetup · moderate

Mindmap

mindmap
  root((Fluentd))
    What It Does
      Collect log events
      Route to destinations
      Unify log pipeline
    Inputs
      App log files
      System events
      Custom plugins
    Outputs
      Databases
      Cloud storage
      Big-data systems
    Tech
      Ruby gem
      Config-driven
      CNCF project

mindmap root((Fluentd)) What It Does Collect log events Route to destinations Unify log pipeline Inputs App log files System events Custom plugins Outputs Databases Cloud storage Big-data systems Tech Ruby gem Config-driven CNCF project

Click or tap to explore — scroll the page freely

Things people build with this

USE CASE 1

Collect logs from all your services and machines in one place instead of each app writing to its own location in its own format.

USE CASE 2

Route log events to multiple destinations at once, such as a database, S3 bucket, and an analytics service, from a single config file.

USE CASE 3

Replace a patchwork of ad-hoc log scripts with one Fluentd instance that handles format conversion, tagging, and routing centrally.

Tech stack

RubyRubyGemsBundler

Getting it running

Difficulty · moderate Time to first run · 30min

Requires Ruby and Bundler, connecting to destinations like Elasticsearch or S3 requires installing additional output plugin gems.

Use freely in any project, including commercial, under the Apache License 2.0, as long as you include the license and copyright notices.

In plain English

Fluentd is an open-source tool for collecting logs. When software runs, it constantly produces small records of what is happening: a user signed in, a request failed, a job finished. These records are called events or logs, and in a real system they come from many different programs and machines at once. Fluentd's job is to gather all of those events from their various sources and then send them on to wherever you want them stored or analyzed. The README describes Fluentd as a way to unify your logging infrastructure. Instead of each program writing logs in its own format to its own location, Fluentd sits in the middle and acts as a single collection layer. It can read events coming in and write them out to many kinds of destinations, including plain files, traditional databases, NoSQL stores, cloud services, and big-data systems like Hadoop. The point is to have one consistent place that handles the flow of log data. Getting started is shown as a few short commands. You install Fluentd as a Ruby gem, generate a sample configuration, start the program pointed at that configuration, and then pipe a small test message in to confirm it is working. The behavior of Fluentd is driven by its configuration file, which is where you describe the sources to read from and the destinations to write to. The rest of the README is aimed at people who want to work on Fluentd itself. It lists the prerequisites, such as a recent version of Ruby and git, and explains how to install the project's dependencies with Bundler and how to run its test suite. It also points to the official website, documentation, community Slack, and discussion forum for more help. Fluentd is written in Ruby, is released under the Apache License version 2.0, and is a project hosted under the Cloud Native Computing Foundation. The README notes that a third-party security audit of the project was carried out.

Copy-paste prompts

Prompt 1

Write a Fluentd config that tails log files in /var/log/myapp/, tags each line with the service name and hostname, and forwards them to an Elasticsearch index.

Prompt 2

How do I install Fluentd as a Ruby gem, write a minimal config file, and pipe a test message through it to confirm it's collecting and outputting correctly?

Prompt 3

I want Fluentd to route application error logs to a Slack webhook and all logs to an S3 bucket simultaneously. Show me the config to do both from the same input source.

Prompt 4

How do I install Fluentd's test dependencies with Bundler and run its test suite locally after modifying the Ruby source?

Open on GitHub → Explain another repo

← fluent on gitmyhub — every repo by this author, as a profile.

Verify against the repo before relying on details.