syntaxdsamurai/openlibrary

This is a small data pipeline project written in Python that pulls book information from Open Library, a free public book database, and saves it locally for analysis. The author built it as part of a personal data engineering learning path, and it deliberately avoids using pandas, a popular data manipulation library, sticking to plain Python instead. When you run the program, it asks you to type in a search topic, such as cricket or machine learning, how many pages of search results to fetch, and a name for the output file. It then calls the Open Library search API, automatically works through the requested number of result pages, and cleans up the raw response so only useful fields remain: book title, author names, the year a book was first published, available languages, and how many editions exist, which the project uses as a rough measure of a book's popularity. Missing fields are handled without crashing the program. The cleaned results are saved in two formats. CSV is a simple spreadsheet style file that opens in almost any tool, useful for quickly sharing data. Parquet is a format used in real data engineering pipelines because it stores data by column instead of by row, which makes large analytical queries faster, compresses the file automatically, and keeps track of each column's data type so nothing has to be guessed later. The codebase is split into three files: one that handles the API calls and paging, one that cleans and saves the data, and one that acts as the entry point and prints a summary once the run finishes. It uses Python 3.14, the requests library for calling the API, and pyarrow for reading and writing Parquet files. The author describes this as the third project in a longer personal roadmap of data engineering practice projects.