elipcs/grading-is-easy-feedback-is-hard

This repository is a research artifact that goes along with an academic paper called Grading Is Easy, Feedback Is Hard. The paper studies whether large language models can act as helpful assistants for teachers grading a university programming assignment, and this code lets anyone reproduce the paper's results and figures. The assignment being studied is an object oriented programming project called FilmNow, and the researchers collected 84 anonymized Java student submissions. They compared two AI models, GPT 5.5 and Gemini 3.1 Pro, on two questions. The first is how closely each model's grades match an expert human grader's grades, compared against a teaching assistant's grades as a middle reference point. The second is how accurately each model can spot the same specific issues in student code that expert graders identified. Running the reproduction does not require any AI service accounts or API keys for the default path. Instead, all the grading results the models already produced are bundled with the repository, so the analysis scripts just recompute statistics and regenerate charts from that saved data. You need Python 3.10 or newer, about 250 megabytes of free disk space, and a normal computer with no special hardware required. After setting up a virtual environment and installing the listed dependencies, a single command rebuilds the paper's benchmark tables, stability checks across repeated grading runs, and all figures. There is also an optional mode that actually calls the OpenAI and Gemini APIs again to redo the grading from scratch, which does need API keys and may produce slightly different numbers than the original paper. Student identities are protected throughout: submissions are labeled only with anonymous codes, and no real repositories, name mappings, or API traces are included. The repository also documents where to find the paper's figures and tables for anyone wanting to inspect the results more closely.