explaingit

akinia0315/sky-with-you

Analysis updated 2026-05-18

★ 18PythonAudience · developerComplexity · 4/5LicenseSetup · hard

TLDR

An automation tool that gives an AI companion a physical presence in the PC game Sky: Children of the Light, letting it chat, react, and hold hands with you in real time.

Mindmap

mindmap
  root((sky-with-you))
    What it does
      AI companion body
      Reads chat panel
      Reacts to hand holds
    Tech stack
      Python
      OCR
      Arduino HID
    Use cases
      Automate companion actions
      Study perception plus LLM setup
    Audience
      Sky game players
      Hobby automation builders
    Setup
      Windows PC
      Arduino board
      OpenRouter key

Code map

Detail Auto

An interactive map of this repo's files and how they connect — its source is parsed live in your browser. Click Visualize to build it.

filefunction / class

What do people build with it?

USE CASE 1

Give an AI controlled companion a body that reacts to your Sky: Children of the Light gameplay in real time.

USE CASE 2

Automate routine companion actions like lighting candles, bowing, and accepting teleport invites while you play.

USE CASE 3

Study a working example of combining screen perception, an LLM, and hardware keyboard injection.

What is it built with?

PythonOpenCVOCRArduino HIDMCPOpenRouter

How does it compare?

akinia0315/sky-with-you1038lab/comfyui-agnes-aiadityaarsharma/wordpress-malware-removal
Stars181818
LanguagePythonPythonPython
Setup difficultyhardeasymoderate
Complexity4/52/53/5
Audiencedeveloperdeveloperdeveloper

Figures from each repo's GitHub metadata at analysis time.

How do you get it running?

Difficulty · hard Time to first run · 1h+

Needs a Windows PC running the game plus an Arduino compatible board flashed with custom firmware and an OpenRouter API key.

MIT license: free to use, modify, and distribute for any purpose, including commercial use, as long as you keep the copyright notice.

In plain English

This project lets an AI companion play as a physical presence inside Sky: Children of the Light, a PC game. It watches the game's chat panel, replies to messages, lights and receives candles, bows, accepts teleport invitations, holds your hand when you reach out, and can say a line like come on, let's go home before guiding you back to your home area. The system runs entirely on the same Windows machine as the game and is built in three layers. A perception layer takes screenshots and combines visual matching with template recognition and text reading to build a shared picture of what is happening on screen, including which panels are open, whether a popup is showing, and what scene the game is in. A scheduling layer reads messages, asks a language model to write a reply based on a persona file the user sets up, and decides what action to take next, always working from that shared picture rather than the live screen. An execution layer sends the actual keyboard presses using an external Arduino board that plugs in over USB, so the game reads them as real hardware key presses instead of software simulated ones, since the game blocks software key simulation. This layer also handles Chinese text input through the clipboard, takes screenshots, and checks that the game window is in focus before pressing anything. Setting it up requires a Windows PC running the game, a small Arduino compatible board flashed with the included firmware, and an OpenRouter API key for the language model. You edit a persona file to describe who your AI companion is, then run two Python scripts in separate terminals. The project also documents several safety details worked out through trial and error, such as refusing to send key presses when the game window is not focused, and ignoring the automation's own messages when they get read back by the text recognition step so it does not talk to itself. The README notes this only reads the screen and sends keyboard input. It does not modify the game or read its memory, and automating gameplay this way may violate the game's terms of service, so use is at your own risk. The project builds on an earlier keyboard injection tool credited in the README. It is released under the MIT license.

Copy-paste prompts

Prompt 1
Help me write a persona.txt file for my Sky: Children of the Light AI companion using this project's template.
Prompt 2
Walk me through flashing the Arduino Leonardo with the sky_keyboard_v2.ino firmware from this repo.
Prompt 3
Explain how the panel_detector.py perception layer combines OCR and template matching to build WorldState.
Prompt 4
Show me how to set the OPENROUTER_API_KEY environment variable so sky-loop-v7.py can call the language model.

Frequently asked questions

What is sky-with-you?

An automation tool that gives an AI companion a physical presence in the PC game Sky: Children of the Light, letting it chat, react, and hold hands with you in real time.

What language is sky-with-you written in?

Mainly Python. The stack also includes Python, OpenCV, OCR.

What license does sky-with-you use?

MIT license: free to use, modify, and distribute for any purpose, including commercial use, as long as you keep the copyright notice.

How hard is sky-with-you to set up?

Setup difficulty is rated hard, with roughly 1h+ to a first successful run.

Who is sky-with-you for?

Mainly developer.

Open on GitHub → Explain another repo

This repo across BitVibe Labs

Verify against the repo before relying on details.

Alternatives & similar repos