bhat-sahab/llama-inference

This project is a toolkit for running large language models locally on your own computer, built specifically around one graphics card, the AMD Radeon RX 9070 XT, paired with an Intel i9 processor. It wraps a popular tool called llama.cpp, which is what actually runs the AI models, and adds a simple graphical launcher so you do not need to type long commands to get a local AI chat server working on your machine. The launcher lets you pick from a handful of preconfigured models of different sizes, choose which backend runs the calculations, either Vulkan, which needs no extra setup and works out of the box, or ROCm, AMD's own compute platform which requires installing an additional software kit but runs faster for some tasks. Once you pick a model and backend and click launch, a local web server starts up that you can open in a browser to chat with the model. The repository also documents detailed speed benchmarks comparing these two backends across several models, showing that Vulkan tends to generate text faster while ROCm tends to process incoming prompts faster, with the better option depending on the specific model being used. Beyond the graphical launcher, there is also a simple command line script for people who prefer typing commands directly. Part of the project also tracks the progress of a newer, more compressed model format called Q2_0 as it gets added to the main llama.cpp project across different hardware backends, noting which backends already support it and which are still waiting. Until that support lands everywhere, the project bundles a modified version of llama.cpp specifically to support that format on AMD hardware. This is a personal hardware focused setup rather than a general purpose tool, most useful to someone who already owns this specific AMD graphics card and wants a simple way to run and benchmark local AI models on it, or wants to compare its performance across backends.