tristan0318/fraudbench

FraudBench is the official code and data release for an academic research paper about detecting fake photos used to fraudulently claim refunds. The problem it studies is this: someone buys a product online, then uses AI image editing tools to fake damage on the product photo and submits it as evidence to get a refund without actually returning anything broken. FraudBench tests whether AI models can tell real damaged product photos apart from these AI faked ones. The benchmark includes 822 real customer review samples and nearly 8,000 images spanning 29 different product and service categories, covering online shopping, food delivery, and travel bookings. The fake evidence images were created from genuine undamaged reference photos using six different state of the art image editing and generation tools, so the fakes closely resemble the kind of manipulated evidence a real scammer might produce. The paper evaluates 11 multimodal AI models that can process both text and images, 4 specialized fraud detection tools, and human reviewers, comparing how well each one spots the fakes across five different evaluation angles. Researchers can run the benchmark under six different conditions, such as showing the model a single image alone, showing it alongside the customer's written review, or showing several images either all at once or one at a time in a longer conversation. Two additional ablation studies test how sensitive results are to the exact wording of the prompt, and what happens when a review text is deliberately mismatched with images from a different product category. All model calls go through vendor APIs like Alibaba's DashScope, xAI, Gemini, and OpenAI, using API keys set as environment variables, so no local GPU is needed to reproduce the experiments. A small local web app built with Flask lets human evaluators view images and record their own fraud judgments for comparison against the AI models. The authors are explicit that this benchmark is meant only for academic research into detecting this kind of fraud and building better safeguards, not for helping anyone actually commit refund fraud. Setup requires Python 3.10 or newer and a handful of Python libraries installed via pip.