theofpa/hadoop-on-gce

This repository lets you quickly set up a big data cluster on Google Cloud in a hands-off way. Instead of manually creating virtual machines and installing software, you run a single command and the scripts handle everything, spinning up 10 machines, installing Hadoop and related tools, and configuring them to work together. The project wraps up Cloudera's Hadoop distribution (called CDH) so you can get a production-ready data processing cluster running on Google's cloud infrastructure. When you first run the build command, it creates virtual machines on Google Cloud, installs the necessary software on each one, and sets them up to communicate with each other. You can customize what services you want (like Hive, HBase, or Spark) during the setup process, and basic settings like machine size are stored in a configuration file you can edit beforehand. Once your cluster is live, you can pause it without losing your work or spending money on idle machines, the disks are saved so you can restart everything in seconds later. When you're truly done, a cleanup command removes all the machines and associated resources. The README notes this is still a work in progress, designed right now for clusters where all machines are identical, though the author plans to support mixed machine types in the future. This is most useful for data engineers or researchers who need a temporary cluster for processing large datasets or running Hadoop jobs without managing infrastructure manually. Instead of spending hours configuring servers, you get a functioning cluster in minutes. It's less hands-on than learning Hadoop from scratch, but still requires you to know broadly what you're doing with distributed data processing.