gowtham-beep/ai-labs

This repository is a collection of self-directed weekend experiments about running large language model inference in production, treating the problem as a distributed systems challenge rather than just a matter of writing good prompts. The author's stated goal is to move past the common pattern of simply wrapping an API call and instead build and rigorously test the unglamorous plumbing that real AI infrastructure depends on, such as queues, controlling how many requests run at once, rate limiting, and circuit breakers that stop cascading failures. Each experiment, called a lab, is documented with real measured data rather than a polished summary, including the dead ends and bugs the author ran into along the way. At the time of this writing there is one completed lab, an LLM inference gateway that separates the queue from the workers processing requests and puts a limit on how many requests run simultaneously. That lab found that adding this kind of concurrency control improved the throughput of batches of requests by roughly four point seven times, while making the response time for any single individual request worse. The author frames these labs as a bridge between traditional backend engineering skills and modern AI operations work, mapping to the kind of system design and coding interview questions that come up when discussing infrastructure meant to handle slow, unreliable, or rate limited dependencies like LLM providers. The tech stack draws on well established production tools: Node.js and TypeScript for the core code, Fastify as the API framework, BullMQ and Redis for the queue, Docker, NGINX, and PM2 for running things in production, and Groq along with locally run Ollama models as the actual LLM providers being tested against.