lee101/omniserve

Omniserve is a server that runs many different AI models on a single GPU at once. Instead of setting up one endpoint per model, like some hosting services do, or running everything with no server discipline, like some open source image tools do, omniserve tries to combine both: it can serve image generators, video generators, LoRA adapters, and chat style language models from the same machine, loading and unloading them automatically based on demand. When a request comes in for a model that is not currently loaded, omniserve loads it, and when GPU memory runs low it removes the least recently used model to make room, checking first whether there is even enough space for the new one. Idle models are put to sleep, meaning their data moves out of GPU memory but can be woken back up in seconds, rather than being fully unloaded and reloaded from scratch. LoRA adapters, which are small add-on files that customize a model's behavior, can be attached to any request by name or by web link, and are only downloaded once across a group of machines. The server speaks the same API format as OpenAI's services, covering chat, text completion, image generation, audio speech, and audio transcription, so tools already built for that format should work with little change. Requests can be tagged with priority tiers such as paid, subscriber, free, or background, and the GPU is allocated to higher tiers first, with protections so a low priority request cannot bump a paying customer's model too soon. Setup is done with a Python package install and a command line tool, and the project can also be packaged as a Cog container for one click GPU deployment, including a ready made deploy button for the app.nz hosting service. The project includes tests that can run without a GPU. It is released under the Apache-2.0 license, which allows free use, modification, and commercial use as long as the license and copyright notice are kept.