I'm certainly not suggesting that you throw away your existing subscriptions, particularly if you're a vendor or solution provider. Put some effort into those relationships, and they'll pay themselves back several times over. But there's something to be said for the little guy, and there are hundreds of smaller analysis firms that can provide you with the kind of service and support you need to make informed decisions on a daily basis.

There's no way to provide a comprehensive list of analysts or coverage areas in small firms, but I've chosen five analysts who exemplify the kind of breadth in business model, coverage areas and perspective you can find when you look beyond the Big Three. Full disclosure: I've worked with some of these people before, but don't hold that against them.

Billy Pidgeon
M2 Research
Coverage Area: Gaming

The gaming industry is a tough nut to crack. It's an art, a business and a unique exercise in supply-chain economics. Plenty of analysts cover financials ("300,000 units shipped!") and tech ("11 million polygons!"), but most leave the games themselves to the press.

M2's Billy Pidgeon understands all three worlds. While he's spent the last dozen years at various research houses, Pidgeon will always be a gamer at heart. He's produced more than 20 games, including major releases such as 1997's Turok: Dinosaur Hunter. This street cred gives him access to insights and talent that more buttoned-up analysts might miss. If you're looking for one-on-one practical advice about the gaming market from someone who's been there but also gets the big picture, check him out.

The Guys at RedMonk
RedMonk
Coverage Area: Multiple (Tech-Related)

If you're a Firefly fan, think of RedMonk as the BrownCoats of the analyst world. If you're not, their motto should tell you what you need to know. "Analysis by the people, for the people" says it all. I would have chosen just one of their four analysts, but that would have violated their whole "community" vibe.

RedMonk tips its hat to the open-source world it covers by giving away its research, believing that an open discussion provides the greatest benefit to everyone, including their paying customers. They make their money from consulting services that start at a flat $5,000 per year, increasing with the size of your company or your consulting demands. For your money, you get access to very astute technical minds focused on helping vendors produce tools that developers will actually want to use. As the business model might suggest, it's a very populist approach in which the end user, IT manager, or systems analyst is a lot more important than the CIO, which is dramatically different than the coverage aims of most larger firms. If you're a software developer, $5,000 a year is a very small price to pay for a contrarian perspective.

David Schatsky
Green Research
Coverage Area: Sustainability

Sustainability is no longer just hip; it's an essential (and sometimes mandated) part of doing business, sitting on a growing pile of hard science. It's a big industry, so hundreds of consultancies have bolted on an "eco-" to get your business. It's tough to weed out the pretenders.

David Schatsky has a background in technology, policy and finance. He also spent nearly 10 years at JupiterResearch as a Research Director and President (yet more disclosure: He was also my boss there for a while), so he understands the analyst gig. But what sets him apart from the rest of the eco-kids is his understanding that he shouldn't do it alone. When he founded Green Research, Schatsky brought in David Meyers, an environmental heavyweight, to build out the company's real-world expertise and complement his research experience, and they've further rounded out their expertise with associated content providers. The result is a small, personalized shop that should be able to address most of your environmental concerns directly, but has the connections to pull in other experts where needed.

Tony Byrne
Real Story Group
Coverage Area: Content Management

Real Story Group doesn't work with vendors they cover. At all. No consultations, white papers, or appearances at vendor events - nothing that could possibly influence their coverage. This independence irritates the industry and helps their clients (anyone working with content or knowledge management) trust what they read. While RSG has a number of top-notch analysts (Theresa Regli deserves a shout-out, particularly regarding international content management issues), the man behind the business model is Tony Byrne, the company's founder.

RSG's Evaluation Reports are their most popular deliverable, largely because of their Consumer Reports-style comparison charts. They aren't cheap (running around $2,500 per report), but they can save you tens or hundreds of thousands during your evaluation process and give you the answers you need to ask the right questions of your vendors. Byrne is convinced that RSG's objectivity and laser focus will convince most one-off purchasers to stick around as clients for further research, as well as advisory services to help manage the tools and content with the software you've bought. So far, so good.

Laurie Orlov
Aging in Place Technology Watch
Coverage Area: Seniors, Health Technology

Seniors are our fastest-growing demographic segment, and the technology required to help them age is of tremendous social and financial importance. So it's strange that until fairly recently, most major research firms treated the category like an afterthought. Laurie Orlov is one of the few experts in that space, and the foremost authority in the study of using technology to remain in the home as you age. In fact, she kind of created it.

Jeff Makowka, AARP's Senior Strategic Advisor, Thought Leadership, explains her impact: "She's a real visionary. She took her past life (as a Forrester analyst) and overlapped it with a caregiving experience and basically thought up the category. Solutions already existed, but she defined and legitimized Aging in Place Technology."

Like every boutique analyst, Orlov's journey is unique, and probably impossible at one of the largest firms. Small firms will never give you the coverage of the Big Three, and can't shout your voice as loudly to the world, but they do a great job of filling the gaps if you're willing to do some searching.

Have you had experiences with small research firms? Let us know who you've used and how it worked.

Lead image courtesy of Shutterstock.

Joshua McKenty's still-young career is, compared to those of other capitalist executives, surprisingly replete. He's led development teams for the Netscape browser, and is intimately familiar with Netscape's successors at Mozilla. His next stroke of luck was with the space program, helping to create and then lead one of the world's most successful cloud computing projects, NASA Nebula. His work with NASA spawned the open source community's most successful - and perhaps most important - project in the last few years, the OpenStack cloud operating system - and he sits on that project's governing body. In-between jobs, he just happened to pioneer an earthquake modeling system for the World Bank.

What McKenty's learned from a life that may not be one-third of the way over yet, are the lessons you'd expect to tell your grandkids. Now he's the CEO of Piston Cloud, the first commercial vendor dedicated to OpenStack. Already, he has a boatload of life lessons for the open source development community at large, and he's not about to wait for grandkids to come along to start sharing them. With a notable degree of eagerness and enthusiasm, Josh McKenty shared his insights with RWW.

Open source vs. customer focus

McKenty's story begins with Netscape. His time there began with the somewhat confused period late in the company's existence as a division of AOL. Immediately he learned that "open" comes in many shades and colors.

He calls the late period of Netscape 8 and 9 "probably the most complicated and nuanced open source environment you can imagine. Netscape was released as the Mozilla code base, and so everyone thinks of Firefox being to the benefit of Netscape. But the code is a tri-license that allows Netscape to close it and develop it for proprietary [purposes] afterwards. Then you have the fact that Firefox, which everybody thinks of as being a Mozilla project, was actually a fork of Mozilla by a sole individual that Mozilla then reverse-forked back into their organization and turned into a $300 million-per-year business, which they did not share."

It's ownership and licensing schemes such as this, McKenty says, that make personal politics more prominent in open source projects than philosophies and governance models. He believes one of the Google Chrome project's greatest strengths comes from the insistence by the Chromium development team - to which Google appears to be adhering - that the code base remain "fully open-sourced, in the open."

It was at about this time when Mozilla began mitigating what the community described as "the tooltip bug" (typically with an amalgam of punctuation attached), and what the organization officially recorded as dozens of related bugs (just one culmination was recorded here).

"It was a bug that was recorded and watched and re-reported and duplicated 460 times over 7 years," McKenty relates. "And the response from the Mozilla developer community was always the same: 'If you really cared, you would learn how to code and fix it yourself.'

"It has always been the worst part of many open source projects, but I think Mozilla's more guilty of it than anyone else: the attitude that the developers tend to develop over time that they are the important users of the product," the Piston Cloud CEO continues. "And this has never really been true. That's what's interesting about most open source software: The developers who are really deeply engaged in building it may have started out solving their own pain. But if they're really successful, there are usually two or three orders of magnitude more people using it than actually building it. So the disconnect between what it is and what it needs to become, gets larger and larger. This is why the open source projects that really survive in the long term figure out how to build that bridge back to the end user's requirements. And those are often commercial entities."

Stage fright

After Netscape, McKenty went on to be a software architect and business developer for Flock, Inc., whose product was a socially-oriented Web browser built on the Mozilla code base. While there, he tells us, "we would hire new developers turn them loose, and say, 'Every commit that you make is going to be looked at by other people in the world.' That's a terrifying experience, especially if you're a young coder or you're new to the code base... It has nothing to do with the philosophy of open source. It has to do with a sense of embarrassment or nervousness to have your daily commits be scrutinized by folks that maybe you think of as being more experienced than you."

McKenty points to the very project he oversees now - OpenStack - as one example where important components are not produced in the open: for example, support for IPv6 contributed by NTT Data of Japan. "[Of] the 260-odd folks who are actively contributing code, as opposed to design features or documentation or localization or whatever else... about a third of them are not building in the open. They all design in the open; that's the community requirement. But then they go home and they code for a couple of months until they have something that they can show, and then they make a big code drop, and then we attack it."

To space and beyond

Some of the earliest Mercury astronauts credit their good fortune with having been at the right place at the right time. By virtue of being teamed with just the right group of consultants in 2009, Joshua McKenty found himself as a lead architect on a project launched at NASA. Called Nebula, it was America's most significant government-funded computer research facility since the heyday of the supercomputer.

On paper, McKenty worked with a firm called Anso Labs. He soon learned that doing any kind of government project involved a separation of what you see in reality from what's on paper.

"Despite whatever retrospective history has been applied to it, [Anso] was simply, at the time, a maneuver to deal with NASA internal politics around contracting," McKenty tells RWW. "The Nebula Project itself was built inside a larger IT contract that employs a couple of hundred people, and we were two or three subcontractors down from NASA proper at the time. So we were just trying to guarantee that we could keep our developer team together working on Nebula, while NASA went through a re-compete on the contract."

Anso made it possible for Nebula to continue for as long as it did, keeping the project open and sharing its benefits with other developers through OpenStack. But that was only for so long. Eventually Nebula was de-funded, and the operation was folded back into NASA's existing supercomputing projects. There, McKenty says, outright opponents to cloud computing have taken full advantage of the opportunity to do next to nothing with Nebula.

Meanwhile, Anso Labs was acquired by cloud service provider RackSpace. While some sources played the acquisition as though it were a conspiracy, McKenty tells us that RackSpace's motives are purely genuine: to see OpenStack succeed, and to continue what their own people, essentially, started.
"I think what Rackspace has done with the contribution of Swift to OpenStack is really unprecedented in their market segment," says Piston Cloud CEO Joshua McKenty. "But RackSpace does not want to be a software company, and they don't really even want to be an open source software company. They want to be a fanatical support company; that's who they are, that's their DNA. Now they have this core open source project... around which they can provide their service.

"But to my mind, OpenStack isn't finished," he continues. "We haven't finished changing the world yet, and I wanted to stay focused on building out OpenStack to what it really needs to be."

Spinout from the Nebula

While many of McKenty's personal best friends went with Anso to RackSpace, he stayed out to pursue his own dream. Exactly what that was hadn't quite formed yet, even as late as the fall of 2010. He took a six-month sabbatical, during which he found time for that little earthquake modeling project for the World Bank. He joined a working group on the role of governments and organizations in building technology infrastructure, along with a handful of friends and associates - Vint Cerf, Sergey Brin, and Vivek Kundra.

"None of those environments actually gave me the same opportunity to understand the requirements that I've gotten out of being the CEO of a startup," McKenty tells us. "The reason to be Piston Cloud right now - aside from the fact that it's going to be the best business ever - [is] to be in the room with the people who are going to use the product, and actually talk to them about what they need. It's very hard to get in the room with the folks who will be most impacted by the technology without being a vendor."

Despite the most honest and thoughtful community outreach, says McKenty, a non-commercial or semi-commercial open source project simply does not garner the level of confidence from its customers that a commercial project does. Almost immediately upon building Piston Cloud, McKenty found himself in direct communication with existing OpenStack users who had never been in contact with anyone representing OpenStack in the past. Being a CEO makes one more accessible than being a committee.

One example: A disaster risk reduction facility is headquartered in Indonesia, but has OpenStack-based cloud services in Australia and customers in Australia. It wanted to expand its cloud to Jakarta, and needed Piston Cloud's help. It's the type of customer contact Anso Labs would have only dreamed about, and now it's coming directly to the source.

"Nobody in the OpenStack community even knew that the Australian government was evaluating it, let alone had pushed it into production," remarks McKenty. "In one phone call from an inbound request, I ended up on the phone with the folks in Australia who had been operating OpenStack at scale, and was able to talk to them about exactly what their needs were, and what they wanted to see in the next release. And there was no way they were going to make it to Boston in person to talk about this.

"The property of being a vendor, and the property of being in the press, as odd as that sounds, gives us this amazing opportunity to really understand what people need," he continues. "The commercial side of this has an amazingly crystalizing effect on people's priorities of what they want built. We are an open source company, and every open source company lives or dies by their ability to balance their interaction with the community with their interaction with their customers... Can we present ourselves in ways that are understandable to those two different communities without ever becoming two-faced, or engaging in a dichotomy? It has to be, honestly, this is who we are, in all cases."

What Daniel D. McCracken managed to accomplish as early as 1957 was to give the knowledgeable layperson a foundation for understanding business processes in terms of procedural mathematics. As a young author decades ago, I studied McCracken's methods and I attempted to take his lessons to heart. In some of my first books on Visual Basic, I was inspired by McCracken to demonstrate a relatively simple concept using a substantively more complex tool: I demonstrated program control using sort algorithms.

McCracken did this from the very beginning, and he may have mystified his audience as much as I did mine. But he performed a necessary task, and he was probably the first to do it. He showed how a complete program that performs a complex function is developed from the core out. Although the concept was probably born in the first IBM laboratories to put FORTRAN to use on a daily basis, McCracken was the first to demonstrate a concept borrowed from artistry: the ability to see the complete product holistically, build working models instead of partial fragments and keep it working with each implementation.

McCracken knew exactly what he was doing - the man reasoned on all levels at once, as evidenced by his later treatises on public policy and theology. In this excerpt from his 1975 paper, "How to Teach Structured COBOL to Beginners," McCracken reiterates the principle he had already, by that time, been putting to use in his classroom and his published work for two decades. What he said, and how he said it, has not in the nearly four decades since.

Structured programming, for the purposes of this paper, may be defined as a style of programming in which only three logic control elements are used, namely, sequence, selection (IFTHENELSE), and iteration (DOWHILE). The scope of control of each selection and iteration element is displayed by consistent indentation. The use of only three logic elements applies both to coding, using whatever logic elements the chosen language provides, and to program design, which is done using either structured flowcharts or, preferably, some form of pseudo-code. A complete program design is achieved in a series of approximations, beginning with a simpler problem from which the desired design can be developed, in a process commonly described as stepwise refinement.

The goal of structured programming is the production of programs that as clearly as possible display their structure, i.e., the interrelationship of their parts. This clarity is a primary benefit of the restriction to only a few logic elements, which leads to programs that can be read in a "top-down" fashion, that is, without skipping around through the program. The purpose and function of any program statement can generally be understood by looking at only a few other statements, all physically close by, which is very seldom true of programs written by conventional methods. The main drawback of GO TO statements is that they tend to destroy this locality of context.

The design and coding stages of programming may or may not be shorter when structured programming methods are used, but the check-out and maintenance phases are generally much faster and easier to manage since programs are easier to understand. As a result overall programmer productivity tends to increase, sometimes by a dramatic factor.

There is a missing characteristic to what is said about computing, but nothing at all missing about what is meant by the finest of its practitioners. Daniel D. McCracken gave the world six wonderful decades of brilliant explanation, most of which today is buried in barely accessible texts and vastly dispersed libraries. Most of it is treated as obsolete. A great deal of it, however, is about as obsolete and inapplicable to the current age as the three brilliant, illustrative paragraphs cited above.

I've gone to several of these final presentations in the past (here is a report from 2009) and I continue to be impressed with what the students come up with. Sure, there are the usual mishaps: code that doesn't compile, or last-minute hacks to add one more feature or tweak a particular icon to display properly. But the class is a great arena for preparing these senior computer science majors with what they are going to have to face in the real world.

The students have to propose an idea for their app, look around on the App Store to see what is currently available, put together a data and coding plan and then write the code. Often they have to access particular Web services and public APIs for their app, and these interfaces can and do change over the course of the course, of course. (Sorry.) One of the teams was trying to access information from the video streaming site Justin.tv, and lamented the lack of any programmatic connection. Another was wrestling with a badly formed series of RSS feeds. How many computer science grads could even think about these things, let alone debate these issues? It brought a smile to my face.

Over the six semesters that the instructor, Todd Sproull, has taught the class, he has had close to 150 students, and mostly male (last night was 100% guys). There is usually a waiting list, as there are only so many computers to go around. "The students have always been bright, but it seems that more of them are using tools from other courses and companies (Google and Facebook APIs as examples) to create more compelling apps," he said. Some of the class band together and program as a team. This creates a certain wow factor at the final presentation - "speaking to the power of teamwork," as Sproull mentioned. Others go it alone.

Some of the graduates have gone on to get great jobs: one student interviewed at Apple, and as a result of demoing their iPhone project, got hired. It helped that they were doing active debugging of the app during the interview. So much for asking silly questions like how many manhole covers it would take to pave over a baseball field and such. Almost all of these students go on to work in the industry, no surprise. Recruiters and others: take note and come take a look at these fertile fields in the future.

There are five apps from all of the classes that have been actually published on the App Store, with a few more awaiting campus approval. One of them is a dandy. It is called WU Map, and has the entire campus map right on your iPhone, as you can see in the screenshot. This is a little thing, but for those freshmen and people like me that aren't that familiar with the campus, it sure beats having to print out a map every time we have to visit. According to Sproull, several of these apps have at least made back their $100 listing fees paid by the students.

Sproull has done a better job preparing his students to understand Web data sources and how to get around some of the quirks of the Apple simulator, too. The summer class seemed to have more fun than in previous semesters, maybe it was just the concentrated amount of time that the students had for class together. One of the students had an app that would allow the user to note his favorite beers, geolocated to the bar that it was consumed. He found a database of more than 8,000 beers, and we joked that we could see that hands-on research really paid off with his app.

What surprised me was that only one of the apps presented was for iPads. This was an app that will be finished soon for the business school magazine. This was very polished, indeed looking better than many mag apps that I have seen from professional editorial operations. It used an RSS feed to grab the various articles from the magazine's Web site. The rest of the projects were focused on the iPhone.

The biggest hurdle these days has nothing to do with the quality of the code, however. It is about the ownership and approval of intellectual property of the student-created apps. The university is still cogitating on exactly how this transpires. There is some concern about a student intentionally (or even unintentionally) distributing a malicious app, and how the school will approve apps that get distributed on the App Stores.

Nevertheless, it was an enlightening evening, and I wish all these students well with their apps.

So how exactly is Python programming useful in creative writing? Parrish's course doesn't deal with artificial intelligence, or attempts at creating narratives or creating interactive hypertext or anything like that. It covers, for lack of a better term, procedural poetry. Typically, a student takes a starting set of text, writes a Python program to modify that text and then interprets the results.

Parrish cited non-electronic procedural poetry experiments as inspirations for the course. For example, he talked about Raymond Queneau's Cent mille milliards de poèmes, a book in which the text has been cut into strips that can be re-arranged to create nearly endless configurations:

Photo by Thomas Guest. More photos of the book on Flickr.

Parrish also mentioned Ted Berrigan's Sonnets and David Melnick's PCOET. Parrish didn't mention them in his talk, but the course website also mentions Brion Gysin and William S. Burroughs' work with the cut-up technique.

Using these works as a springboard, Parrish teaches his students UNIX commands for working with text, Python text processing techniques (such as ranging from basic string manipulation to n-gram analysis) and regular expressions to help them create their own procedural texts. He says he chose Python because it's easy to use and has a lot of tools for working with text. Using computers students can process more text and do so more quickly than the physical methods used by the experimenters of the past.

Parrish says pacing is one of the most difficult issues faced in the class. Beginner programmers always feel the class moves too quickly, while experienced programmers find that it moves too slow.

Parrish's focus is clearly in creative writing and helping students explore text in new ways. But it's a really interesting experiment in helping art or humanities students learn to program (see our interview with Douglas Rushkoff (who is also a teacher at ITP on why everyone should learn to program). A similar model could also be used to inject the humanities into programming and engineering education.

Where to Find More

Many of the lessons can be found here and the code examples are in Github.

Unfortunately, it omits such influential languages as Lisp, ALGOL 60 and Smalltalk. But including every important language ever would make for a pretty long infographic.

The popularity stats at the end are apparently sourced from the oft criticized TIOBE. You can find the full sized graphic and bibliography here.

In comment at Lambda the Ultimate, Eich wrote: "History has reason and rhyme as well as chance, it is not all and only random. For my part, there was little 'arbitrary' in what I did, including the mistakes -- some of those weirdly recapitulated early LISP mistakes."

Eich's comments were in response to comments like this one: "If Brendan Eich chose SmallTalk for the Netscape browser, that's probably what you'd be gushing about today."

And this one: "It's just dumb luck and path dependence. If Netscape had put scheme into Navigator we'd be using that instead."

Eich writes at Hacker News:

"Subtle chains of cause and effect were at play among people involved, going back years to Silicon Graphics (Netscape drew from UIUC and SGI, plus montulli from Kansas, and jwz). Also going back through the living history of programming languages. SICP and some of the Sussman & Steele 'Lambda the ...' papers made a big impression on me years before, although I did not understand their full meaning then.

"Remember, I was recruited to 'do Scheme', which felt like bait and switch in light of the Java deal brewing by the time I joined Netscape. My interest in languages such as Self informed a subversive agenda re: the dumbed down mission to make 'Java's kid brother', to have objects without classes. Likewise with first-class functions, which were inspired by Scheme but quite different in JS, especially JS 1.0.

"Apart from the 'look like Java' mandate, and 'object-based' as a talking point, I had little direction. Only a couple of top people at Netscape and Sun really grokked the benefit of a dynamic language for tying together components, but they were top people (marca, Rick Schell [VP Eng Netscape], Bill Joy).

"Rather than dumb luck, I think a more meaningful interpretation is that I was a piece of an evolving system, exploring one particular path in a damn hurry. That system contains people playing crucial parts. Academic, business, and personal philosophical and friendship agendas all transmitted an analogue of genes: ideas and concrete inventions from functional programming and Smalltalk-related languages."

Eich also tells the story of how JavaScript came to be here.

In short, the decision to use JavaScript instead of a language like Smalltalk or Scheme was far from arbitrary and arose from specific circumstances.

Also, although the lack of a better alternative has certainly ensure JavaScript's popularity, neither Netscape nor JavaScript's success was ensured from the beginning. Other browsers, such as Mosiac and Cello existed. I had all three on my computer around the time JavaScript was being developed, and I still knew people who still preferred Lynx to the graphical Web.

And as Eich points out, not everyone saw the value in a scripting language in addition to Java. Had JavaScript not been good enough, it could have been discontinued or replaced.

InfoQ is running an interview with Twitter engineer Evan Weaver who explains more about the shift.

Here are a few interesting points:

• The first class languages at Twitter are JavaScript, Ruby, Scala and Java. Soemtimes C is used as well.
• The usage of Ruby is shrinking at Twitter as JavaScript takes over the front-end and JVM-based languages take over the back-end.
• In general, developers at Twitter from a Ruby background prefer Scala, and those with a C/C++ background prefer Java.
• The search team uses Lucene and is experienced in Java. Java is more convenient for them than Scala or Ruby.
• Twitter uses a library called Finagle for building asynchronous RPC servers and clients in Java, Scala or any JVM langauge.
• The move to Scala and Java at Twitter is driven more by a need for encapsulation than for performance and scalability and much of the existing Ruby code is quite workable for the time being.
• Static typing has been a productivity boon as Twitter search has moved towards a service oriented architecture.

The interview also goes into more specific technical reasons for preferring Scala to Rails, such as better vertical integration. Weaver also talks about Twitter's overall architecture, which was described in the talk we covered here.

Ars Technica has an interesting retrospective on the BeOS file system, BFS, which is now in use in both Haiku and SkyOS. BFS had many forward looking features, including 64-bit data structures, journaling and metadata support.

The article includes a handy glossary of storage terms and interviews with a Be engineer responsible for BFS and Haiku developer Axel Dörfler. The Be engineer asked not to be named "to comply with the wishes of his current employer."

There's a good chance that employer is Google), which employees several former Be engineers.

Several ex-Be employees went to work for Danger after the company told to Palm. Some of them moved on to Android, which was co-founded by Danger co-founder Andy Rubin and acquired by Google. Others stayed on at Palm, but ended up joining Google after PalmSource (which was spun out of Palm) was acquired by Access.

According to Wikipedia Dominic Giampaolo, one of the main developers of BFS and author of Practical File System Design with the Be File System (PDF), has been working on file systems for Apple since 2002. One of the other key developers, Cyril Meurillon, is now a consultant.

At the moment, there are 351 stories.

Here are a few:

"by finding artistic problems which made absorbing the knowledge easy." - Lilli Thompson

"because my first computer would only sit with a blinking cursor if I didn't program it." Ashley Feniello

"because my TRS-80 didn't come with any video games, so I had to write my own."
- Mykle Hansen

"by writing a BASIC game on a manual typewriter, and being driven by my mom to the local Radio Shack where I typed it in to the TRS-80 I coveted."- Matt Hillman

"in college, when I realized I could have a big impact on the world by building software with a focus on humanity." - Vanessa Hurst

"because I believed (and still do) that it's one of the best skills to have in order to change people's lives for the better." Alyssa Daw

"when I wanted a piano and got an Atari -- programmed the keys to make sounds." Katie A. Siek

"because I wanted to make interactive electronic literature." - Zuzana Husarova

"because I wanted to build cool things. Now that's my job." - Alex Gaynor

"because I like learning languages -- computer ones included!" - Angelica Lim

"after graduating with a Philosophy degree." - Thomas Saunders

"by being a designer who needed a programmer and couldn't find one!" - Catherine Hicks

BackType captures online conversations, everything from tweets to blog comments to checkins and Facebook interactions. Its business is aimed at helping marketers and others understand those conversations by measuring them in a lot of ways, which means processing a massive amount of data.

To give you an idea of the scale of its task, it has about 25 terabytes of compressed binary data on its servers, holding over 100 billion individual records. Its API serves 400 requests per second on average, and it has 60 EC2 servers around at all times, scaling up to 150 for peak loads.

Christopher Golda Michael Montano Nathan Marz

It has pulled this off with only seed funding and just three employees: Christopher Golda, Michael Montano and Nathan Marz. They're all engineers, so there's not even any sysadmins to take some of the load.

Coping with that volume of data with limited resources has forced them to be extremely creative. They've invented their own language, Cascalog, to make analysis easy, and their own database, ElephantDB, to simplify delivering the results of their analysis to users. They've even written a system to update traditional batch processing of massive data sets with new information in near real-time.

The backbone of BackType's pipeline is Amazon Web Services, using S3 for storage and EC2 for servers. It leverages technologies such as Clojure, Python, Hadoop, Cassandra and Thrift to process this data in batch and real-time.

The start of the pipeline is a group of machines that ingest data from the Twitter firehose, Facebook API and millions of sites and other social media services. The first interesting feature of the architecture is that it actually has two different pipelines, one the traditional batch layer that takes hours to produce results, and a "speed layer" that reflects new changes immediately.

Captured data is fed into the batch layer through processes on each machine called collectors. These append new data to a local file, which is then copied over to S3 periodically. This raw data is then put through a process they call shredding, which organizes it in two different ways. First, data units are stored with others of the same type. For example the content of tweets or blog comments would be stored together and separate from the names of their authors. Second, the same data is sliced by time, so everything within a single day will be stored together.

Why do they do this? The organization of the data enables them to run more efficient queries only against the relevant data. When they have a job that requires analyzing Twitter retweets for example, they can just pull out the content, sender and time for each message, and ignore all other metadata. This process is made a lot easier thanks to their use of Thrift for the data storage. Everything in their system is described by a graph-like Thrift schema, which controls the folder hierarchy the data is stored into, and automagically creates the Java/Python/etc code for serialization.

Cascalog is one of their secret weapons, a Clojure-based query language for Hadoop that makes it simple for them to analyze their data in new ways. Inspired by the venerable Datalog, and built on top of Cascading, it allows you to write queries in Clojure and define even complex operations in simple code. Unlike alternatives like Pig or Hive, it's written within a general-purpose language, so there's no need for separate user-defined functions, but it's still a highly-structured way of defining queries.

Its power has enabled them to quickly add features like domain-level statistics and per-user influence scores with just a couple of screens of code. It's spread beyond BackType and has an active user community including companies like eHarmony, PBworks and Metamarkets.

The final part of the batch processing puzzle is how to get the results of your analysis to the final user. They experimented with writing out the data to a Cassandra cluster, but ran into performance issues. What they ended up creating instead was a system they call ElephantDB. It takes all the data from a batch job, splits it up into shards, each of which is written out to disk as BerkeleyDB-format files. After that they fire up an ElephantDB cluster to serve the shards. Unlike many traditional databases, it's read-only, so to update data served from the batch layer you create a new set of shards.

So that's how the heavy processing is done, but what about instant updates? The speed layer exists to compensate for the high latency of the batch layer. It is completely transient and because the batch layer is constantly running it only needs to worry about new data. The speed layer can often make aggressive trade-offs for performance because the batch layer will later extract deep insights and run tougher computations. It takes the data that came in after the last batch processing job and applies fast running algorithms.

Because the Hadoop processing is run once or twice a day, the fast layer only has to keep track of a few hours of data to produce its results. The smaller volume makes it easy to use database technologies like MySQL, Tokyo Tyrant and Cassandra in the speed layer. Crawlers put new data on Gearman queues and workers process and write to a database. When the API is called, a thin layer of code queries both the speed layer database and the batch ElephantDB system, and merges the information from both to produce the final output that's shown to the outside world.

BackType isn't the only startup to split its processing using this combination of speed and batch layers; Hunch does something similar for its user recommendations. The trouble is that nobody has found an approach that is as elegant or generally applicable as MapReduce for real-time processing of continuous streams of data.

Instead, Marz and Montano have been working on a new framework based on their own experiences. The technology managing the streaming processing and guaranteeing reliability of messages is called Storm, and though it can run a variety of languages, they've designed one especially for it called Thunderlog, based on Cascalog.

Though they are not yet ready for release, Storm and Thunderlog are being actively developed and will soon replace their more hand-coded speed layer. The system will incorporate many of the tips they picked up building their first system. For instance, to avoid concurrency issues without paying a performance penalty, you can group events by key so that possibly conflicting changes happen on the same machine in a serial fashion.

At the end of the tour of their technology, I was left very impressed by how much they have accomplished with so few engineers. Instead of the firefighting and housekeeping burden I'd expect from such a complex system, they seem to spend most of their time focused on applications that solve customer problems.

The secret is their ability to automate the routine tasks with tools like Cascalog, ElephantDB and Thunderlog. Writing those allows them to spend their limited time on writing new applications that offer direct value to their users, without having to wrestle with screenfuls of boilerplate code first. They are on the lookout for new team members, and say they've only stayed so small because they are so committed to only hiring the very best. If you're interested in working on the cutting edge of big data processing, drop them an email at jobs@backtype.com.