intuitivedesigns/streamkernel-io

StreamKernel is a data pipeline runtime for the Java Virtual Machine (JVM) that handles moving and transforming data streams while enforcing policies and running AI models, all inside a single process. Instead of stitching together separate tools for policy checks, data transformations, AI inference, error routing, and delivery to multiple destinations, StreamKernel bundles those responsibilities into one configurable pipeline. The problem it solves: in most production data pipelines, teams end up writing "glue code" to connect message queues (like Kafka or Apache Pulsar) to storage destinations (like Delta Lake, Snowflake, or MongoDB), while also handling failed records, applying security policies, and tracking metrics. That glue is expensive to build and hard to audit. StreamKernel replaces it with a single process configured through a properties file, keeping source, policy, transformation, caching, inference, sink delivery, dead-letter queue routing, and metrics inside one runtime boundary. A notable feature is built-in AI enrichment: the pipeline can run machine learning models in-process using the ONNX format, integrated with the MLflow model registry. This allows a new model version to be promoted or rolled back without redeploying the pipeline, and each record can carry labels identifying which model version processed it, useful for regulated industries that require audit trails. The project is source-available: the core runtime is visible but private implementation details and model artifacts are withheld under a commercial license. It is written in Java and runs on Java 21. The full README is longer than what was provided.