One thing the recession has done is fuel innovation by imposing financial constraints. The constraints have led to some dramatic market changes, in particular related to the rise of Lucene/Solr, the open-source search technology.
As a result, it feels like we are the beginning of something. It’s like we are witnessing the emergence of a new search. I tweeted that today about Lucene Revolution, the conference I am attending here in Boston. It is reminiscent of 2003, when blogs gave rise to RSS, a format designed to help us make sense of the online content that had started to flow like never before.
As RSS helped us make sense of information in 2003 so is Lucene/Solr helping make sense of data today. Big data is here to stay. It is increasingly fueling Web apps and is leading to new innovations in search, primarily in the open-source market. You see it here at Lucene Revolution. Twitter, LinkedIn, Salesforce.com and a host of others are discussing how the Web apps they have developed need Hadoop and Lucene/Solr to function properly for the apps to do what they want them to do.
RSS in many ways served as the precursor to a new breed of syndication technologies and the modern Web app, which gave rise to the activity stream. It’s a river of news, a term coined by Dave Winer, one of the Web’s pioneering developers . Twitter came out of this wave, be it indirectly, through the creations by Evan Williams and his team. Williams started Pyra Labs, which gave rise to Blogger. After selling to Google, Williams became captivated by the next incarnation of RSS. That was podcasting, which never really took off in a big way but perhaps more importantly served as a catalyst for a new wave of innovation in media delivery. Odeo flopped but Twitter rose from its ashes.
Twitter represented a new way to get news. It harnessed APIs like no other service had before. Developers embraced it. It marked a new wave, the app movement, which in turn has fueled the fast rise of social technologies. Text, pictures, videos are at the heart of these apps. The more we create, the more reason we give to developers to build new apps.
In the midst of this, the financial markets collapsed and with it a spiraling economy. Suddenly, it made even more sense to develop your own apps.
And here we are today. Budgets are still thin but the pipes are fat with content. So much so that we need new tools to make sense of it. What is coming from that is the start of something pretty significant. It’s the new search and it is helping us make sense of things.
Sometimes it feels like 2003.
