Social Media “Glue” and Gnip’s Co-opetition with FriendFeed

We believe that enabling web technologies are going through a similar development cycle as enterprise application integration technology did 10+ years ago. Companies are creating tools, applications and platforms to enable more productive and automated uses of resources that have become ubiquitous parts of the online ecosystem. We think about these enabling technologies as the glue that will increasingly hold together that ecosystem.

Seth Levine, Foundry Group

Venture capital firm Foundry Group, which includes partner Brad Feld, described an important investment theme for their firm. Titled Theme: Glue, the thesis is that the growing number of web services and content-generating sites are causing increased complexity, and that there is a need for an infrastructure to handle all this.

This can seem a bit dry (“I know this back end plumbing stuff is boring to most of you”, as Michael Arrington says), but its relevance is can be considered in the context of:

• Twitter limiting API requests to manage its server load
• FriendFeed’s success (so far) in putting real-time updates in the hands of its users

Foundry’s thesis extends beyond server-load management. But its initial investment in Gnip starts on that part of the “Glue” story.

The Problem Gnip Solves

The rise of user generated content has made this problem particularly acute. We’re creating so much content, all over the place. Flickr, Del.icio.us, Digg, YouTube, Twitter, WordPress.com, Google Reader, etc. I mean, there’s a lot of stuff!

It turns out a lot of other sites want to consume this stuff – FriendFeed, Plaxo Pulse, Strands, SocialMedian and many, many other sites. And the direction for production and consumption of all this content is only going one way –> up.

The problem that arises is the way consuming services, such as FriendFeed, have had to find out if you’ve got a new tweet, blog post, Digg, etc. The consuming services need to ping every individual user’s account on some site, such as Flickr,  with the query, “got anything new?” For most people, the answer is no. But that query is the cause of a lot of traffic and latency. Imagine all these new web services pinging en masse all the UGC sites. It can be quite a load to handle. In Twitter’s case, it was too much to handle.

Gnip addresses this issue, standing between UGC sites and consuming sites:

The UGC sites (aka producers) push updates for their various users to Gnip. These updates are either change notifications, or full content for each user. So Gnip becomes the reference layer for anything occurring for a UGC site’s users.

The consuming sites then look to Gnip as a “single throat to choke” in terms of updates. Gnip handles the updates, and gets them out to consuming sites in real-time. Gnip also removes the burden on consuming sites to write and maintain polling scripts for all the various UGC sites of users.

The idea is a good one, as it offloads a lot of the burden for both producers and consumers. Of course, it shines a light on Gnip’s scalability and uptime stats.

Initial consuming sites using Gnip include:

• Plaxo Pulse
• Strands
• MyBlogLog
• Soup.io
• Eventvue

Gnip is off to a nice start. But what about FriendFeed?

FriendFeed’s Ex-Googlers Roll Their Own

FriendFeed is one of those consuming sites. But they’ve not signed on for Gnip so far. Not surprising, considering their Google background. Lots of good knowledge about scalability to be learned from Google.

Rather than sign on to Gnip’s service, Friendfeed has proposed the simple update protocol (SUP). What’s SUP?

SUP (Simple Update Protocol) is a simple and compact “ping feed” that web services can produce in order to alert the consumers of their feeds when a feed has been updated.

The idea is that the UGC sites provide a single point for posting notifications of new user activities. Rather than the consuming sites running the “got anything new?” query for every single user on their platform, they go to a single place to see what’s new. They have a list of the user IDs they want to check, which they run against the SUP location. Much more efficient.

Which does sound a little like Gnip, doesn’t it? Here’s a Q&A between Marshall Kirkpatrick of ReadWriteWeb and FriendFeed co-founder Paul Buchheit:

RWW: [Where is this relative to] Gnip? (See our coverage of Gnip, a startup that appears to be aiming to do what SUP will do and more.)

Buchheit: We’re talking with several companies about supporting SUP, but aren’t ready to announce anything.

On the TechCrunch post about Gnip 2.0, commenter Nikolay Kolev writes:

Even if I like Gnip’s concept a lot at this moment, I think it’s just a temporary solution of the real problem. It solves deficiencies of the vast majority of the data producers nowadays, yes, but if more implement XMPP PubSub, FriendFeed SUP and other similar technologies, there will be less incentives for data consumers to make their business rely on a single provider that supposedly aggregates and replicates all of the Web…

On FriendFeed, user Dani Radu writes this in response to Gnip:

pretty interesting, I mean making all this data handling drop dead simple is great – but this means they want to cache and route the direction of interest. Own the process and sell it. Again, perfectly sane – but I’d rather go for the SUP (simple update protocol) which in a way – if adopted widely – does the same but keeps the handling free (as free as the services are anyway) We shall see what future brings tho…

The Gnip vs. SUP question came up on Hacker News, which included this exchange with FriendFeed’s Paul Buchheit:

ryanwaggoner: Isn’t this what Gnip is doing, except that Gnip’s solution is readily available to anyone who wants it? In fact, I believe Gnip uses XMPP to push notifications to data consumers, which seems even more efficient. Am I missing something?

paul: No, Gnip is a complementary service and will likely consume SUP. SUP is intended to make it easier for feed publishers to expose information about which feeds have been updated. Without this information, Gnip can’t know when feeds have updated except by polling all of them. SUP allows them to poll a single URL instead.

ryanwaggoner: Got it. So this is designed to be the piece that allows publishers to easily integrate with intermediate services like Gnip, or with aggregation services like FF, SocialThing, etc.

paul: Exactly

Paul’s right, but the earlier comments are also right. Gnip may want to get updates for its UGC producing sites using SUP. But there’s truth to the idea that if producers offer SUP, some of the value proposition of Gnip is eroded.

Gnip: More Than Real-Time Updates

But Gnip appears to provide a range of services above and beyond simple update notifications. My guess is that those extra services Gnip provides above and beyond providing a single place to get notifications will be their secret sauce.

On the Gnip blog, product head Shane Pearson writes the four use cases on which Gnip is focused:

1. Eliminate the need for developers to write dozens of pollers for UGC sites, all of which must be maintained and updated
2. Target business specific applications that need this data. There may be interesting functional or vertical application that SUP won’t cover.
3. Offload the overhead on UGC producers’ sites (which sounds like SUP). But beyond that, create an alternative channel for their content, provide analytics on the data consumed through Gnip ,and add filters an d target endpoints.
4. Use Gnip as a source of market research and brand analysis for what consumers are saying about companies.

So what you see here is that the developer world sees SUP competition in the infrastructure part of Gnip. Gnip is looking beyond the developer world in terms of where it delivers value.

I wouldn’t be surprised if other companies enter the mix. “Glue” is an early, interesting space right now.

*****

See this post on FriendFeed: http://friendfeed.com/search?q=%22Social+Media+%E2%80%9CGlue%E2%80%9D+and+Gnip%E2%80%99s+Co-opetition+with+FriendFeed%22&who=everyone