kris70lesgo/heartheworld

HearTheWorld is an accessibility application that helps people who are deaf or hard of hearing notice important sounds in their environment. It runs in a web browser, listens to the microphone, and sends visual alerts and push notifications when significant sounds are detected, so users who cannot hear an alarm, a knock, or a crying baby are notified through their screen even when they have switched to another tab. The application watches for specific high-priority sounds including baby crying, knocking, doorbells, thunderstorms, shouting, screaming, sirens, alarms, fire alarms, smoke detectors, glass breaking, gunshots, explosions, vehicle horns, and impacts. Lower-priority sounds like normal speech or background noise appear in the app's history view but do not trigger push notifications by default. The browser captures microphone audio and streams it over a WebSocket connection to a backend server. The backend classifies the audio using YAMNet, a pre-trained sound classification model, runs a speech transcription model called faster-whisper to generate live captions, and uses a locally running language model through LM Studio to rewrite alerts into plain language. When an important sound is detected, a OneSignal push notification is dispatched to the browser. The frontend is built with Next.js and TypeScript, and the backend uses FastAPI in Python. Both run locally during development. For deployment, the frontend suits Vercel and the backend suits services that support long-running WebSockets and Python machine learning dependencies. Browser background monitoring works only while the page tab stays open.