Feeding Frenzy with SimpleFeed

OR, how to add a proper social activity feed to your ruby-based application in fifteen minutes.

For the Impatient

I am very excited to announce the official release of the open source Ruby library called Simple Feed, released as a ruby gem.

I am the primary developer on this project, and my name is Konstantin Gredeskoul — which you probably already knew. This library would not have been possible without the generosity and sponsorship of Simbi.com.

Simple Feed is running live in production, and is powering three separate social feeds on Simbi: the global feed, followers feed, and "own" feed — events related the feed owner's actions:

The software is distributed under the MIT License and is available today. It's written in ruby and does not depend on Ruby on Rails. Therefore it can power applications running on Rack, Sinatra and using alternative web frameworks. Having said that, at Simbi we are using it with Rails, and I may release a small Rails adapter for SimpleFeed in the future.

The short-list of its features is:

  • You can define any number of feeds per Ruby VM
  • SimpleFeed stores pure strings associated with a floating point number (typically – time), and so does not assume any particular format of the data
  • The data is stored in pluggable backends, with Redis and Hash providers supplied.
    • New providers can be easily built and used with the same API.
    • Using Twemproxy, the backend can be transparently sharded to support millions of users.
    • Redis provider is optimized for multi-user batch operations, such as posting an event to many users at once using Redis pipeline.
  • Rich API allowing both single-user (i.e., for reading the feed) and multi-user (i.e., for writing to feed) batch operations
  • Powerful DSL
  • Near 100% test coverage :)

For information on how to use, install and other technical details I refer you to README or the more detailed discussion below.

Feeds 101

The word "feed" has a bit of a mythical origin, but what it describes is rather well understood at this point. A related term Data Feed is defined by the Wikipedia as a ... mechanism for users to receive updated data from data sources.

Wikipedia then goes into describing various types of feeds, such as the RSS or News Feeds, The Web feeds, and even Product Feeds — all of which I had to deal with extensively in my career.

This is why I feel that I have a particular responsibility to explain them to folks that may not fully get the concept, in a way that just makes sense. And there is no better place to start than to clarify that....

A feed is just a list. An ordered list of things, and typically — similar looking things.

— Really? — You might exclaim.

— Yup. It's a list of things, but it does have some special properties.

You see, it's meant to be consumed (or read, loaded, imported) by the computers, not humans. At least this is how the word "feed" entered into the technical jargon in the late 90s.

If I scribble you a shopping list on a piece of paper, — that's just a shopping list.

But if I enter this list into an Excel, or into a database, then suddenly this list turns into a form that an average modern day computer can read. It can then "do some awesome stuff" with this list. Like — in the case of a product feed — it could search for products on the Internet, and compare prices, or verify product availability, etc. In fact, I built an app that did just that.

But what's true, is that only the web geeks like myself (and, perhaps, some very progressive teenagers) are going shopping with a shopping feed. Most of us are still shopping with a simple list.

And so with the understanding that a feed is just a list of things that is intended for a computer to ingest, we can move forward, and talk about Social Feeds, which are, ironically, intended for humans.

In the world of World Wide Web, and in particular, in the world of the Social Web, when we mention the word "feed" when we talk about a time-ordered list of events that have occurred. We might even call this an "Activity Feed." And it makes sense: you the see activity that matters to you and the connections you created on the social network.

Importantly, as the new activity occurs on the site, it is "fed" to you via this feed. You could say that we are all being fed information that comes through a feed(-ing tube?[ -ED.]).

Feed This

We see Activity Feeds everywhere these days: social networks such as FaceBook, Twitter, GitHub, LinkedIn — they all have them.

It's common sense — in addition to tailoring information to your interests, modern applications show you a highly personalized view into what is happening over time (including right now), and people love it. We all love it, and I love it too.

In fact, it bothers me when I signup or register for a new application only to realize that I can't easily see what's going on: what are my friends doing, what I am doing, what else is going on?

It feels awfully quiet on a social site without an activity feed.

And the number of apps that don't offer a social activity feed is still staggeringly large.

And so why is that?

Perhaps the reason is that it's not that easy to build this feature from scratch — the activity feed is a moderately difficult software project. It's hard, because:

  • The feed collects many different types of events from many users, and often has complicated rules as to what is shown to the user, and when.
  • The feed must be updated nearly in real time, so as new events happen, they should immediately show up in user's feeds.
  • the feed must be shown to each user quickly, or people get bored and leave. How many times have you quit an app that was stuck "spinning its wheels"?

The last two requirements — being near real time, and being fast to render — are in a conflict with each other. This is why it took Twitter so long to get this right at a massive scale. The renown "Bieber" problem which occurs when Justin Bieber tweets to his tens of millions of followers, generates enormous activity on Twitter's backend.

Luckily, most social sites aren't Twitter.

Feed Simple

So why do I know so much about these feeds, you might ask?

Well, funny you should ask. You see, I built more than a few feeds for various social networks I worked for, and they are all remarkably still live and in production.

My previous employer — Wanelo.com — uses activity feeds for both the "Magic" and "My Feed" features. In Wanelo's case, the feeds are populated with products, and the software powering these feeds did experience the "Bieber Problem," except not with Justin Bieber, but with the very popular store Urban Outfitters, that amassed over 3M followers on that social network. Whenever the store would post a new product, three million users must see that product in their feeds. That's no simple feat.

The company I work for now — Simbi — is a social network for a symbiotic economy: where people can barter services with each other. The social feed was a natural fit, if not a real necessity.

And it was Simbi that kindly supported the development of the open source library called Simple Feed, which, belatedly, I am stoked to announce to the world. And even though it has already been mentioned on the Peter Cooper's Ruby Weekly (thanks, Peter!), I always thought that it's better to be late than never.

SimpleFeed is a pure Ruby library that uses Redis as a storage mechanism, and deals with storing pure strings in the feed, associated with the timestamp (or to be precise, with any floating point number that defaults to time).

Because SimpleFeed only stores string data type using Redis, it is left to the developer that integrates the library to decide how exactly to serialize their data in the feed.

Feeding The Frenzy

If you are a site owner, and you've always wanted to add a social feed, or a news feed to your site today — there are several pretty good options.

A couple of them are written in Ruby. Unfortunately, many of the options are either Rails-specific, or they are rather extensive in what they offer and how much code and functionality they load into your Ruby VM when used.

That's not to say these options aren't good, and perhaps if you were to consider building one, I'd recommend that you reviewed them all.

Here are the ones I found:

  • GetStream.io is a commercial solution with clients in many languages, as well as the REST API. This option is free under 3M updates per months, and so if you are someone with a very low site traffic, this might be one of the easiest options, as it does not require any backend programming. That said, you must be comfortable relying on a 3rd party for uptime and reliability.

  • activity_feed — is an open source Ruby gem that's probably the closest one to SimpleFeed, but unfortunately it lacks some pretty basic features: for example, it assumes you want to define one and only one activity feed in your application: you configure your feed at the global level. In SimpleFeed you can configure as many specialized feeds as you like. Also, it offers some strong opinions about what is stored in the feed, i.e., the data structure of the feed item. SimpleFeed does no such thing, and stores pure strings, leaving it up to you to decide how to serialize your data.

  • public_activity — is another open source ruby gem that's quite a bit out of date.

Having discussed other options for completeness, I encourage you to give SimpleFeed a proper test-drive. Perhaps then you may find that it fills the sweet spot, by being both rich in functionality required to implement an activity feed, and yet simple enough that it takes a short time and a small effort to integrate with it.

And, of course, I will always be here to help. For any problems — please create a GitHub issue, and I will address it as soon as I can. Better yet — submit a pull request!

Happy Feeding!

— Konstantin.

References