xlite-dev/leetcuda

LeetCUDA is a collection of learning notes and working code examples for CUDA, a programming model used to run computations on NVIDIA graphics cards (GPUs). GPUs can process many operations in parallel, which makes them central to deep learning, scientific computing, and large-scale matrix calculations. Writing GPU code directly is considerably more complex than writing standard CPU code, and this repository is aimed at helping developers learn how to do it. The collection includes more than 200 CUDA kernel implementations, organized by difficulty from easy through progressively harder levels. A kernel is a function that runs on the GPU. The examples range from basic operations to advanced techniques like matrix multiplication using Tensor Cores, which are specialized circuits on modern NVIDIA GPUs designed specifically to accelerate the kind of math used in neural networks. Two major areas get extended treatment. The first is HGEMM, which stands for half-precision general matrix multiplication, a fundamental operation in training and running AI models. The implementations here reportedly reach 98 to 100 percent of the performance of NVIDIA's own cuBLAS library, which is the standard reference for GPU-accelerated linear algebra. The second area is Flash Attention, an algorithm that makes the attention mechanism in transformer models (the architecture behind most modern language models) faster and more memory-efficient. The repository also links to more than 100 blog posts covering related GPU programming topics. PyTorch, a widely used Python library for machine learning, appears throughout the examples as a companion tool. The intended audience is developers who already have programming experience and want to learn GPU kernel development from first principles. The full README is longer than what was shown.