Mini MapReduce
A distributed text-processing demo that simulates how large-scale systems like Hadoop or Spark work - but entirely inside my browser and Node backend.
It splits input text into "shards," maps them in parallel to count words, and then reduces those partial counts into one final result - all with live progress tracking and real-time output visualization.
Maximum file size: 3.5MB
Maximum text size: 3.5MB
How MapReduce works in real life
Imagine you own a busy pizza restaurant, and one night, you get a huge order for 10,000 pizzas - all with different toppings. There's no way a single chef could make them all in time, right? So, you decide to divide the work.
You take that giant list of 10,000 orders and split it into smaller chunks. You hand each chunk to one of your 10 chefs, so everyone works in parallel.
Each chef keeps a mini report of what they made. This is the Map phase - each worker processes a piece separately.
Once all chefs finish, you collect all their reports. You shuffle all the data together by pizza type:
You go through each pile and combine the partial results:
That's the Reduce phase - merging partial data into the final output.
If one chef made all 10,000 pizzas alone, it would take days. But by splitting the job among 10 chefs, everyone finishes at roughly the same time - results come back way faster.
💡 That's distributed systems: divide, process in parallel, and combine the results!
My Mini MapReduce app works just like this pizza shop - but instead of chefs, you have workers. And instead of pizzas, they handle text data.
The input text (like shakespeare.txt) is split into shards (small chunks).
Each worker counts the words in its shard separately - just like each chef makes their pizzas independently.
The app combines all those partial word counts into a final total for every word - just like the manager totaling up all the pizza types.
You see exactly how many times each word appears, with graphs, live progress, and downloadable output - all in real time! 📊
🌍 Why It Matters
This is how Google, Amazon, and tech giants handle massive datasets:
Google splits the web across thousands of servers to index it faster
Netflix splits recommendation data across machines to calculate user preferences
Banks process transactions in parallel to handle millions per second
🚀 My app is a miniature version of that, showing distributed computing - right inside a browser - with real-time visualization!
This project simulates true distributed computation, using event-driven APIs and async workers, wrapped in a clean UI that feels like a data science dashboard.
You can:
- Upload or type any text
- Choose how many shards (parallel workers) to divide the job into
- Watch each shard map its chunk of data into partial word counts
- See the reducer combine those into a final aggregated result
- View the entire pipeline's progress, timings, and word-frequency visualizations
- Simulates true distributed computation with event-driven APIs and async workers
⚙️ Core Architecture
🛠️ Tech Stack
🧾 Credits
How MapReduce works in real life
Imagine you own a busy pizza restaurant, and one night, you get a huge order for 10,000 pizzas - all with different toppings. There's no way a single chef could make them all in time, right? So, you decide to divide the work.
You take that giant list of 10,000 orders and split it into smaller chunks. You hand each chunk to one of your 10 chefs, so everyone works in parallel.
Each chef keeps a mini report of what they made. This is the Map phase - each worker processes a piece separately.
Once all chefs finish, you collect all their reports. You shuffle all the data together by pizza type:
You go through each pile and combine the partial results:
That's the Reduce phase - merging partial data into the final output.
If one chef made all 10,000 pizzas alone, it would take days. But by splitting the job among 10 chefs, everyone finishes at roughly the same time - results come back way faster.
💡 That's distributed systems: divide, process in parallel, and combine the results!
My Mini MapReduce app works just like this pizza shop - but instead of chefs, you have workers. And instead of pizzas, they handle text data.
The input text (like shakespeare.txt) is split into shards (small chunks).
Each worker counts the words in its shard separately - just like each chef makes their pizzas independently.
The app combines all those partial word counts into a final total for every word - just like the manager totaling up all the pizza types.
You see exactly how many times each word appears, with graphs, live progress, and downloadable output - all in real time! 📊
🌍 Why It Matters
This is how Google, Amazon, and tech giants handle massive datasets:
Google splits the web across thousands of servers to index it faster
Netflix splits recommendation data across machines to calculate user preferences
Banks process transactions in parallel to handle millions per second
🚀 My app is a miniature version of that, showing distributed computing - right inside a browser - with real-time visualization!
This project simulates true distributed computation, using event-driven APIs and async workers, wrapped in a clean UI that feels like a data science dashboard.
You can:
- Upload or type any text
- Choose how many shards (parallel workers) to divide the job into
- Watch each shard map its chunk of data into partial word counts
- See the reducer combine those into a final aggregated result
- View the entire pipeline's progress, timings, and word-frequency visualizations
- Simulates true distributed computation with event-driven APIs and async workers