ashawkey/stable-dreamfusion

Stable-Dreamfusion is a Python tool for generating 3D objects from text descriptions or images. You type a prompt like "a hamburger" and the system produces a 3D model you can view and export as a mesh file. The project implements the DreamFusion research paper using Stable Diffusion, a publicly available image-generation model, as the guidance engine, since the model referenced in the original paper is not publicly released. The authors note upfront that this is a work in progress and results do not yet match the quality shown in the paper. The 3D generation works by repeatedly asking a 2D image model to evaluate and refine a 3D representation called a NeRF (neural radiance field). The NeRF stores a 3D scene as a mathematical function and renders it from any angle. Stable-Dreamfusion uses an accelerated NeRF variant called Instant-NGP that can render at around 10 frames per second on a GPU with 16 GB of memory. Once training is done, you can export the result as a standard 3D mesh with textures. You run training from the command line with a text prompt. After training finishes, you run the same script in test mode to export a video rotating around the object or to save the mesh file. The tool also supports image-to-3D: if you start from an existing photo, you can pass it in instead of a text prompt, though this requires downloading an additional pretrained model called Zero-1-to-3. A simple graphical interface is included as well. The main requirements are a recent NVIDIA GPU (tested on a V100 with 16 GB of VRAM), a working CUDA installation, and Python with PyTorch. Setup involves installing Python packages and optionally compiling custom CUDA extensions. Google Colab notebooks are provided so you can try it without a local GPU.