This week Basho, the company behind the open source NoSQL database Riak, released a beta of Riak Pipe. Basho community manager Mark Philips describes Pipe as “more or less a rewrite of our existing MapReduce framework. It builds on the lessons we learned in the initial (and still in use) version of MapReduce.” You can find the code in GitHub.
Basho Senior Software Engineer Bryan Fink elaborates in the official announcement:
Essentially Riak Pipe allows you to specify work in the form of a chain of function pairs. One function of that pair describes how to produce output from input, and the other describes where in the cluster an input should be processed. Riak Pipe handles the details of ferrying data between workers by building atop Riak Core’s distribution power.
Riak Pipe will eventually power Riak’s MapReduce system and expand its MapReduce capabilities. You can get an early look at a Riak branch that integrates Pipe here.
You can learn much more about how Pipe works in the Readme file, and you can learn more about how Riak is being used in the real world here.
