3 posts about rss

Relatively Simple RSS Aggregation

Mar 20 2006

I recently posted about my need for a simpler RSS aggregator.

I worked through the massive list of existing RSS aggregators at wikipedia and couldn't reallly find something that did exactly what I wanted and worked.

I needed something that was simple, cached hourly, displayed various encodings well and worked with RSS and Atom formats. Most importantly, I wanted something I could install on my own server, and it needed to be community oriented (not designed for a single reader).

Most of the existing solutions on the web are hosted by a third party, and are limited in the number of feeds that they can aggregate.

A robust, simple option for a thrid-party hosted aggregator is Feed2JS, but if you want a bunch of feeds on your site you'll have to use a bit of javascript for each one. Multiple feeds get ugly fast, and if their server goes down, your scripts go down.

The best hosted solution I found was Gregarius, which has a lovely community. I think that gregarious will take the cake sometime this year as aggregating catches on. For now, however, the plugins are pretty limited — you are locked into a personal reader with "Read/Unread" tagging. (Which makes Gregarious a great replacement for a newsreader like Bloglines, but it's not good for a community aggregator where lots of people visit.)

If I were a better person I'd just write a Gregarius plugin, but instead I've just reworked some of Feed2JS's code (which is in turn dependent on the Magpie RSS parser.)

You can download it here.

It's a index.php file with an "admin" folder. Drop it on your server in an appropriate directory and you should have a page that displays 15 biofuels blogs. NOTE: if you don't know what php is or don't have ftp access to your server, just use Feed2JS and insert their javascript into your page.

Otherwise edit the php file with the rss or atom feeds you want to include. It will probably need a little CSS to spice things up. You can send comments or questions about the script to chris(at)nonprofitdesign.org. I'll release another version sometime soon with better documentation and a prettier front end.

An example of the styled script in action is here (at the Piedmont Biofuels website).

UPDATE: That page now uses simplepie, which kicks ass.

Combining RSS feeds and Displaying them on Your Page with Javascript and PHP

Feb 24 2006

Last night I was trying to do something that I thought would be pretty simple: display a bunch of recent weblog posts on one page.

There is a great online community of folks in the biofuels blogosphere, and this page would give a quick summary of their myriad, nerdy, wonderful events and research.

So the goal is to have the title of a weblog, followed by the most recent posts, each with the date posted and a bit of the post body. The entire web page might be called "biofuels digest," with a total of perhaps 30 weblogs. Often on the web you will see "blogrolls" that list lots of blogs, but these are usually just links to the blogs (there isn't a post excerpt) and they are almost always either hardcoded html or javascript-included from a third party like Bloglines (see my own blogroll on the front page).

I'd had experience with building this type of page last year, when I just wanted to have an "aggregator" page of all my most loved online reading. I ended up just slapping things around with Magpie RSS (an excellent open source PHP class), and it worked fine. Not slick, but fine.

I could have easily used a number of services that are available online for displaying other people's rss on your own page, without all the mussing with PHP. (Feedburner or Feed Digest are services that I'd recommend for doing this type of thing, if you want to go that route.) But who wants to mess with a bunch of javascript calls to someone else's server? And you get stuck with limits on the number of feeds you can run. And the there's the annoying "powered by ..." sticker at the bottom. And you'd have to use a third-party RSS splicer to combine all of your feeds.

So forget all that, because this isn't just a wonky personal project — it will hopefully end up being part of the excellent Piedmont Biofuels website — so it needs to be quick and hosted on the server.

So last night I opened up the latest installation of MagpieRSS and installed it on my server, created all of the necessary php for each of the blogs, and I ended up with a decent document. The major problems with this first version (using just the Magpie class) is the inconsistent treatment of the posts — some appear and some don't — and the improper encoding of the blogs. (I went 'round and 'round with the encoding. It's a common problem, but I couldn't get those damn posts clean.) Probably a few days in the Magpie listserv archives at Sourceforge would clear all of this up ... but the archives are exceptionally annoying, the Magpie blog is down, and the first version was still surprisingly slow anyway, even with the cache working.

So I found another solution, Alan Levine's Feed to JS, which is built on Magpie. This is an excellent free (and libre) service that has both hosted version and downloadable script. (It relies on the magpie class, but uses javascript to display the results., giving the added benefit of having an administrator's page that simplifies some of the options for display (such as the number of posts), and it yields much more compliant utf8 encoding (no more bloody diamond question marks in place of fancy quotes).

The downside: there is currently no way to "splice" all of the feeds together before running them through the javascript, so you end up calling the js file for each feed you parse. I felt sure this would make it to slow to be usable, but I think (hope) I was wrong, even with 25 blogs on the same page. I trimmed each of the blogs to display only 3 posts anyway, so at least the compiled filesize is real slim.

(Actually, while I'm writing all this out, I should bother to mention that there are a hell of a lot of 3rd-party RSS splicers/combiners ... but, again, they're all third party, and they seem to go extinct quickly: e.g. the defunct rollup.org. Most of these also have ads, are not free, or have limitations on the number of feeds, like feed digest. I was surprised and disappointed that I couldn't find something to install on the server that would take care of this — somebody please let me know if there's something reliable out there. This would allow me to combine all the posts and just run the JS once.)

So, from a programmer's view, this is a little inelegant, but the result is really consistent, and it still comes in at 8.5 seconds on 56K. The (minimally styled) latest version is here.

What The Hell Is RSS?

May 5 2005

There's a good discussion going on now at TechSoup about how you can use technology to increase capacity of your nonprofit.

Their latest focus is on RSS, which is one of the more amazing technical innovations of modern communications. If you don't know what RSS is, you're not in the loop.

Most notably, you are missing two things. 1.) The ability to prrocess inhuman amounts of information on a regular basis. RSS is used by many sites to create "digests" of (a.k.a. "syndicate") their content. News sites, for example, offer these RSS "feeds" to sections of their newspaper.

It's a lot like having a TiVO for your favorite sites.

Once you subscribe to one, it appears in your RSS reader of choice. In the end, it's a lot like having a TiVO for your favorite sites — every time you sit down and you want to read, you have a list of updated articles in front of you. No wasting your life poking around the less glamorous parts of a newspaper page.

2.) If you have a website that is updated regularly, you can create a RSS feed so that others can subscribe to your page. This may seem a little superfluous if you don't even know what a RSS feed is. "Who would use it?" Plenty of folks will, especially over the next year or two. So start writing some engaging articles for your site.

Here's a bit from the Tech Soup conversation:

"If you're like most people who use the Internet, chances are you often come across new and interesting sites, but then completely forget to visit them again. Or likely you spend too much time visiting the same sites looking for new information, only to be disappointed. In addition, your e-mail inbox is probably flooded with messages you barely have time to read, including subscriptions to newsletters that tell you about new content available on still more Web sites. ... Wouldn't it be nice if there were an easy way to have all this information come to you and go to them in a way that was easy to manage, timely, and put the reader in control? "

Read the article (and check out TechSoup forums, which are among the best of their kind on the web.) TechSoup - Articles: Using the Internet - RSS for Nonprofits: