Map Reducing the Royals with Mongo

MongoDb is a real lifesaver when it comes to improving developer productivity in web applications, however, that’s only a small part of the power in MongoDb. To do a lot of the deep down data mining, we need to learn to use Map/Reduce to massage our data. Please note, some of this functionality can be accomplished using Mongo’s Aggregate functions, however, I’ve intentionally avoided it, as there are limitations with using aggregates on sharded environments, and I expect most of my Mongo apps will need to be sharded.

Since we just finished the 2012 All-Star Game here in Kansas City, a baseball statistics example seems appropriate.

Setting up your environment
You’ll need to have console access to a mongodb database. To set up mongo on your computer, see the Quick Start.

Loading some sample data
Lets create some realistic baseball stats. I’ll start with the real roster for the Kansas City Royals. However, instead of using their real stats, we’ll generate some random numbers using javascript’s Math object. For example, we know that the best players in the league will get 200 hits, the worst players get none. Math.floor(Math.random()*200) will give us a random number between 0 and 200. We’ll make sure that the number of hits never exceeds the number of At-Bats, and we’ll keep the number of Home Runs capped at 50 (rather generous for the Royals).

To add a single player, we can run the following javascript:

db.Player.save({ 
  number : 47, 	
  name: 'Nathan Adcock', 
  hits : Math.floor(Math.random()*200), 
  ab: Math.floor(Math.random()*300)+200, 
  bb:Math.floor(Math.random()*50)+5, 
  hr: Math.floor(Math.random()*50)
});

Grab the script for the whole roster here, and run it in your mongo console.

Counting Home Runs
Confirm that you’ve got the data loaded. Your stats for Billy Butler will vary (my Billy Butler kind of sucks), but you should always have 43 players.

>db.Player.count();
43
>db.Player.find({name: 'Billy Butler'});
{
  "_id" : ObjectId("50021639b5145ef5327c66b2"),
  "number" : 16,
  "name" : "Billy Butler",
  "hits" : 66,
  "ab" : 386,
  "bb" : 5,
  "hr" : 5
}

We now know how many home runs Billy Butler hit this season, but let us say we want to find the number of home runs that the combined Royals roster hit this season.

var map = function() {
	emit( { }, { hr: this.hr} );
};

var reduce = function(key, values) {
	var sum = 0;
	values.forEach(function(doc) {
    	sum += doc.hr;
  	});
	return sum;
};


homeRuns = db.runCommand( {
     mapreduce: 'Player',
     map: map,
     reduce: reduce,
     out: 'totalHomers',
     verbose: true
} );

db[homeRuns.result].find();

A more complex example

Cool huh? Lets take a slightly more complicated case. We’d like to take all players with more than 250 AB, and group them by batting average.

var map = function() {
	var ba = this.hits /this.ab;
	if (ba < .250) {
		key = '< .250';
	}
	if (ba > .250 && ba < .300) {
		key = '.250 -> .300';
	}
	if (ba > .300) {
		key = '> .300';
	}
	emit(key, { count : 1});
};

var reduce = function(key, values) {
	var sum = 0;
	values.forEach(function(doc) {
	  sum += doc.count;
	});
  	return sum;
};

ba = db.runCommand( {
    mapreduce: 'Player',
    map: map,
    reduce: reduce,
    query: {"ab": {$gt: 250}},
    out: 'battingAverages',
    verbose: true
} );

db[ba.result].find();

Source Code

These examples are pretty simple, but we can still take away a few lessons:

  • Do the heavy lifting in the map function. These are the tasks that get executed in parallel across your shards. For example, by pushing the batting average calculation, and the categorization into the map function, we ensure a fast runtime across a large dataset.
  • Make use of the query arg for the map/reduce command. By filtering out the undesireable data, we save mapping operations and reduce the load on the db

Credits:
Thanks to several bloggers who helped me understand this concept:

Full source code is available on github.

Advertisements
Tagged , ,

2 thoughts on “Map Reducing the Royals with Mongo

  1. Ben Kittrell says:

    Very cool, where is the Javascript running? Server side? I love the lag line on your blog BTW.

    • adamnengland says:

      With MongoDB, the client application issues all of its commands as javascript, which is executed on the server-side. The map/reduce operations are distributed between the servers in your configuration. I need to do a little research,and see if Mongo understands coffeescript.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: