inkBlots has moved to TechBrew.net

February 8th, 2007

Dear inkBlots readers,

I have joined forces with several other technology writers to form a not-for-profit site called TechBrew.net. We provide articles, how-to’s, and free or open source software to benefit everyone. Our main focus is software and technology, but you’ll find generally geeky and interesting things along the way.

I will still write about RSS and Atom, of course, but my hope is that you’ll benefit from a broader spectrum of content and contributers. Please check us out and join the discussion. See you there!

Piping Feeds

February 8th, 2007

Looks like I was a day too soon with Filtering Feeds… Yahoo!’s new entry into the mashup marketplace with Yahoo! Pipes has some amazing potential.

By potential, I mean that it is probably cool once they get it working… I’m guessing they didn’t anticipate the load they’re hit with this morning. I’ve seen more of this than I care to count:

Yahoo! Pipes

Once they get their servers massaged back into shape, everybody and their brother will be piping away. Hat tip to Yahoo! if they pull this off.

Filtering Feeds

February 7th, 2007

As mashups and feed-centric apps abound, the need to filter feeds is becoming more prevalent. The idea behind a feed filter is that you may only want to see items from an RSS-based publication on a certain topic, keyword, or tag. This is an essential aspect of the “newsmastering” domain.

ReFilter Reconsidered

ReFilter is an online feed filter service that lets you filter an RSS feed based on key words. It is a pretty handy offering, but in my recent coverage of Kitchen Sink I noted that the use of ReFilter as “man-in-the-middle” might be troublesome under load. It just doesn’t lend a sense of permanency.

So, I spoke with Kitchen Sink developer Marjolein Hoekstra today and she said the developer of ReFilter has asked her not to use it, because of the potentially high volume that could go through his servers. He does provide his source code if other people want to host it, however.

MySyndicaat URL-based Filters

Kitchen Sink is already using MySyndicaat to combine all of the marketing feeds. I learned from Marjolein that MySyndicaat allows for filtering right on the feed URL, by keyword and date ranges, in fact. I knew you could do this sort of thing when setting up a feedbot, but I didn’t know you could do “post filters” on the feedbot feed itself. Accordingly, Marjolein has dropped the use of ReFilter and is now using the URL feed filtering directly from MySyndicaat.

Here’s how it works:

Once you have a feedbot set up in MySyndicaat, the filtering syntax on any given feedbot RSS is simple. For example, this feed URL is for Top US news:
http://mysyndicaat.com/myfeed/feed/MySyndicaat_Top%20News%20-%20US

Add “?query=Bush”, and you now can get only US news RSS items that mention the keyword “Bush“. Add “&daterange=2007-02-04%20to%202007-02-05″ to only see items about Bush on February 4th and 5th.

Cache in Hand

MySyndicaat uses a 15 minute cache to aid in performance on feed retrieval, but I wasn’t sure whether adding query parameters would bypass their cache. Checking in with Giovanni Guardalben at KipCast, he assured me that query results are also cached. Accordingly, whoever does the query first will get a slower result than any repeated requests.

I’ve already taken advantage of this on my JetStream project to provide tag-specific feeds… it works perfectly. (Example) This is really useful functionality, and I fully expect to see it used in a lot more places in the future.

Boutique Blog Search via RSS/OPML Mashup

February 5th, 2007

Thanks to Giovanni Guardalben for the tip this weekend:

Marjolein Hoekstra and Todd And have used a bunch of free tools to create a OPML/RSS mashup called Kitchen Sink. They have NewsGator, FeedBurner, MySyndicaat, and Grazr in the mix to create a boutique search engine and feed source for a select group of blogs. (In this case, that group is from the list of Todd’s 150 Top Marketing Blogs. If marketing isn’t your thing, you could do the same approach with any OPML file.)

How it Works

Here’s the basic flow: OPML reading list of feeds from Todd’s Newsgator Online account. The OPML is then pulled into MySyndicaat to pull together the actual content of all of the RSS and Atom feeds into a river of news. The massive combined RSS feed that results is then prettied up in FeedBurner.

Kitchen SinkFrom there they used the very-beta GrazrScript (and a lot of help from Mike at Grazr) to create a GUI that lets you explore the data from the feed, using ReFilter to filter content based on keywords.

Kitchen Sink also lets you subscribe to a custom feed of your keyword searches… you’ll only get items from the blog list that have your keywords.

Wish List

I would like to see the Kitchen Sink widget hooked directly to MySyndicaat (using ReFilter in the GUI). That would cover the bare minimum of technologies needed and still deliver the desired end state: Choose/find your sources in MySyndicaat, and let them do the work of pulling the feeds and storing the master content. Use GrazrScript for the front end, and ReFilter for on-demand filtering of the master content.

Trouble Spots

1. I don’t know how well ReFilter will hold up under a heavy load, but I’m betting that if your user base is in the hundreds or low thousands, it should be fine. Hopefully they have a good bandwidth usage plan, (or a good cache scheme) since they have to re-read the feed every time.

2. The lynch pin, of course, is that the Grazr guys probably aren’t going to be able to do custom work for everybody who asks. :) My guess is development of GrazrScript will need to mature for awhile before this kind of approach is widely possible for the geek masses.

Update February 6, 2007

I chatted with Marjolein this morning about Kitchen Sink, congratulating her on making TechMeme for so long yesterday. Marjolein noted that although she has “less hair” because of various issues she had to work through in GrazrScript, the Grazr team didn’t need to contribute much. She did note that they readily squashed any bugs she found in GS, so that aspect of their support was invaluable. (Mike at Grazr also commented below on his involvement, and believes that GS is more mature than I state above.)

Radio Userland and RSS vs. Atom

January 31st, 2007

fightWhen does it become a bad idea to not handle the competition’s format?

Dave Winer, the creator of RSS 2.0 and Radio Userland’s RSS Aggregator, chides a friend for switching from RSS to Atom:

I was subscribing to your feed, generally reading all your updates, and now I see the feed moved…. I saw that your new feed isn’t RSS, to which I ask — why?? Do you want to lose subscribers?

The position he’s taking, of course, is that there is no practical reason to use Atom (if there were, his own Radio Userland software would support it). Now he’s put himself out on a ledge where he has to ask people to stay on RSS, rather than just support Atom.

Let’s face it, support for both formats isn’t just commonplace, it is expected. To whit:

James Roberson notes:

Dave Winer must be the only guy on the planet with a newsreader that can’t handle Atom.

The comments on Roberson’s post are growing, but here are a few telling tidbits:

Rogers Cadenhead: I’m not aware of any newsreader outside of Radio UserLand and Manila that doesn’t support Atom 1.0

Asbjørn Ulsberg: The only reason Dave can’t read Atom feeds is because his software doesn’t support it. And the reason it doesn’t is politically based, not technically.

Elliotte Rusty Harold: Sooner or later they’ll upgrade when more and more of the feeds they want to aggregate are no longer available in RSS.

Dave, I think it’s time to rethink your position. If your goal is to provide software that is useful to people, why not handle the Atom format gracefully? Heck, even Microsoft Word can read other document formats. That doesn’t mean Redmond is admitting defeat, it just means good software is flexible, and there might be something other than .doc out there.

Like it or not, Atom 1.0 exclusivity is growing. Why not give yourself and your users a chance to read the content, even if you don’t like the package it comes in?