shandaai/alayarenderer

AlayaRenderer is an AI-powered rendering system for games and virtual worlds, built around the idea of using video AI models to understand and re-style game footage. It consists of two connected tools working together: an Inverse Renderer and a Game Editing model. The Inverse Renderer takes a regular video clip from a game and breaks it apart into the underlying visual data that game engines use, things like the surface color without lighting (called albedo), surface angles (normals), how far objects are from the camera (depth), and material properties like roughness and metalness. These separate layers are called G-buffers. The Game Editing model then takes those G-buffer layers plus a text description and generates a new version of the video with a completely different look, for example transforming a sunny city scene into a snowy winter version. The research behind this project also released a large dataset to train these models: over 4 million video frames captured at 720p from two major commercial games (Cyberpunk 2077 and Black Myth: Wukong), with all six visual layers recorded simultaneously. Clips average 8 minutes each, covering diverse weather and lighting conditions. The code is written in Python and requires a GPU, the underlying AI models are available for download from Hugging Face. An interactive demo for the Game Editing part is publicly available online.