irenee28/akm-clr-governance-layer

AKM-CLR is a governance and control layer that sits in front of shared LLM serving systems such as vLLM, deciding whether a request from a particular tenant or task should be allowed through, sent to a fallback, or blocked before it ever reaches the model. It is aimed at platforms that serve many different customers, tasks, retrieval augmented generation sources, or fine-tuned adapters from the same shared infrastructure, where a mistake could let one tenant see data or responses meant for another. The project provides tenant and task authorization, routing rules for domains and adapters, blocking of unsafe or cross-tenant requests before inference happens, safe fallback behavior, checks on model output, audit trace logs, and usage metrics, and it integrates with vLLM or any OpenAI-compatible backend. The author ran a series of controlled experiments to test the idea. These included a large simulated test with forty tenants, eighty adapters, and twenty domains, and a run of one thousand simulated attacks, both reporting zero measured leakage between tenants or adapters. A separate test placed the governance layer above a real vLLM server running the Qwen2.5 model, where authorized requests passed through while unsafe or unauthorized ones, including tenant spoofing and cross-tenant probing, were blocked before they reached the model. Later experiments tested packaging the project with Docker, Kubernetes manifests, a Helm chart skeleton, Prometheus metrics, and OpenTelemetry tracing, and a further test checked that a tenant is authorized to use a specific workspace, session, cache entry, or retrieved context object before a request goes to inference. The author describes this clearly as a validated prototype, not a production ready product, and is looking for a small number of design partners running multi-tenant LLM platforms to pilot it with. The README does not state a license.