minjie05/knowbase_ai

KnowBase_AI, also titled ShopKeeper Brain, is a Python backend that lets you build a question-and-answer system on top of your own product documents. The core idea is that you upload PDF or Markdown files describing your products, the system processes and stores them, and customers (or staff) can then ask questions in plain language and get answers pulled from those documents rather than from a generic AI. The document import pipeline runs in eight sequential steps. It parses PDFs into structured text using a tool called MinerU, which handles tables, formulas, and multi-column layouts. It then splits the text into chunks, identifies which product each chunk belongs to, and creates two types of numeric representations (dense vectors that capture meaning and sparse vectors that capture keywords) using a locally hosted model called BGE-M3. Those representations go into a vector database called Milvus. At the same time, the system uses a language model to extract the entities and relationships in each chunk and stores them in Neo4j, a graph database, so structural connections between product concepts are preserved. When a user asks a question, the system takes several parallel retrieval paths at once: a conventional vector search against Milvus, a second search that first generates a hypothetical answer and then searches on that (a technique called HyDE that often works better for complex questions), and a graph-based search against Neo4j. A web search via a local MCP tool can also contribute results. The results from all paths are combined using a ranking method called RRF, then re-ranked by a cross-encoder model for accuracy, and finally passed to a large language model that produces the response. The infrastructure runs locally: Milvus, Neo4j, MongoDB (for conversation history), and MinIO (for file storage) can each be started with a single Docker command. The language model calls go to Tongyi Qianwen (Alibaba DashScope) by default, though any OpenAI-compatible endpoint works. The FastAPI backend exposes the import and query pipelines as API endpoints, and LangGraph orchestrates the multi-step flows.