I have joined forces with several other technology writers to form a not-for-profit site called TechBrew.net. We provide articles, how-to’s, and free or open source software to benefit everyone. Our main focus is software and technology, but you’ll find generally geeky and interesting things along the way.

I will still write about RSS and Atom, of course, but my hope is that you’ll benefit from a broader spectrum of content and contributers. Please check us out and join the discussion. See you there!

By potential, I mean that it is probably cool once they get it working… I’m guessing they didn’t anticipate the load they’re hit with this morning. I’ve seen more of this than I care to count:

Once they get their servers massaged back into shape, everybody and their brother will be piping away. Hat tip to Yahoo! if they pull this off.

ReFilter Reconsidered

ReFilter is an online feed filter service that lets you filter an RSS feed based on key words. It is a pretty handy offering, but in my recent coverage of Kitchen Sink I noted that the use of ReFilter as “man-in-the-middle” might be troublesome under load. It just doesn’t lend a sense of permanency.

So, I spoke with Kitchen Sink developer Marjolein Hoekstra today and she said the developer of ReFilter has asked her not to use it, because of the potentially high volume that could go through his servers. He does provide his source code if other people want to host it, however.

MySyndicaat URL-based Filters

Kitchen Sink is already using MySyndicaat to combine all of the marketing feeds. I learned from Marjolein that MySyndicaat allows for filtering right on the feed URL, by keyword and date ranges, in fact. I knew you could do this sort of thing when setting up a feedbot, but I didn’t know you could do “post filters” on the feedbot feed itself. Accordingly, Marjolein has dropped the use of ReFilter and is now using the URL feed filtering directly from MySyndicaat.

Here’s how it works:

Once you have a feedbot set up in MySyndicaat, the filtering syntax on any given feedbot RSS is simple. For example, this feed URL is for Top US news:
http://mysyndicaat.com/myfeed/feed/MySyndicaat_Top%20News%20-%20US

Add “?query=Bush”, and you now can get only US news RSS items that mention the keyword “Bush“. Add “&daterange=2007-02-04%20to%202007-02-05″ to only see items about Bush on February 4th and 5th.

Cache in Hand

MySyndicaat uses a 15 minute cache to aid in performance on feed retrieval, but I wasn’t sure whether adding query parameters would bypass their cache. Checking in with Giovanni Guardalben at KipCast, he assured me that query results are also cached. Accordingly, whoever does the query first will get a slower result than any repeated requests.

I’ve already taken advantage of this on my JetStream project to provide tag-specific feeds… it works perfectly. (Example) This is really useful functionality, and I fully expect to see it used in a lot more places in the future.

Marjolein Hoekstra and Todd And have used a bunch of free tools to create a OPML/RSS mashup called Kitchen Sink. They have NewsGator, FeedBurner, MySyndicaat, and Grazr in the mix to create a boutique search engine and feed source for a select group of blogs. (In this case, that group is from the list of Todd’s 150 Top Marketing Blogs. If marketing isn’t your thing, you could do the same approach with any OPML file.)

How it Works

Here’s the basic flow: OPML reading list of feeds from Todd’s Newsgator Online account. The OPML is then pulled into MySyndicaat to pull together the actual content of all of the RSS and Atom feeds into a river of news. The massive combined RSS feed that results is then prettied up in FeedBurner.

From there they used the very-beta GrazrScript (and a lot of help from Mike at Grazr) to create a GUI that lets you explore the data from the feed, using ReFilter to filter content based on keywords.

Kitchen Sink also lets you subscribe to a custom feed of your keyword searches… you’ll only get items from the blog list that have your keywords.

Wish List

I would like to see the Kitchen Sink widget hooked directly to MySyndicaat (using ReFilter in the GUI). That would cover the bare minimum of technologies needed and still deliver the desired end state: Choose/find your sources in MySyndicaat, and let them do the work of pulling the feeds and storing the master content. Use GrazrScript for the front end, and ReFilter for on-demand filtering of the master content.

Trouble Spots

1. I don’t know how well ReFilter will hold up under a heavy load, but I’m betting that if your user base is in the hundreds or low thousands, it should be fine. Hopefully they have a good bandwidth usage plan, (or a good cache scheme) since they have to re-read the feed every time.

2. The lynch pin, of course, is that the Grazr guys probably aren’t going to be able to do custom work for everybody who asks. My guess is development of GrazrScript will need to mature for awhile before this kind of approach is widely possible for the geek masses.

Update February 6, 2007

I chatted with Marjolein this morning about Kitchen Sink, congratulating her on making TechMeme for so long yesterday. Marjolein noted that although she has “less hair” because of various issues she had to work through in GrazrScript, the Grazr team didn’t need to contribute much. She did note that they readily squashed any bugs she found in GS, so that aspect of their support was invaluable. (Mike at Grazr also commented below on his involvement, and believes that GS is more mature than I state above.)

Dave Winer, the creator of RSS 2.0 and Radio Userland’s RSS Aggregator, chides a friend for switching from RSS to Atom:

I was subscribing to your feed, generally reading all your updates, and now I see the feed moved…. I saw that your new feed isn’t RSS, to which I ask — why?? Do you want to lose subscribers?

The position he’s taking, of course, is that there is no practical reason to use Atom (if there were, his own Radio Userland software would support it). Now he’s put himself out on a ledge where he has to ask people to stay on RSS, rather than just support Atom.

Let’s face it, support for both formats isn’t just commonplace, it is expected. To whit:

James Roberson notes:

Dave Winer must be the only guy on the planet with a newsreader that can’t handle Atom.

The comments on Roberson’s post are growing, but here are a few telling tidbits:

Rogers Cadenhead: I’m not aware of any newsreader outside of Radio UserLand and Manila that doesn’t support Atom 1.0

Asbjørn Ulsberg: The only reason Dave can’t read Atom feeds is because his software doesn’t support it. And the reason it doesn’t is politically based, not technically.

Elliotte Rusty Harold: Sooner or later they’ll upgrade when more and more of the feeds they want to aggregate are no longer available in RSS.

Dave, I think it’s time to rethink your position. If your goal is to provide software that is useful to people, why not handle the Atom format gracefully? Heck, even Microsoft Word can read other document formats. That doesn’t mean Redmond is admitting defeat, it just means good software is flexible, and there might be something other than .doc out there.

Like it or not, Atom 1.0 exclusivity is growing. Why not give yourself and your users a chance to read the content, even if you don’t like the package it comes in?

My oldest son Caleb, now nearly 7, has been diagnosed with Sensory Processing Disorder (also called Sensory Integration Dysfunction.) This is a Non-Verbal Learning Disability that is neurological in origin. In short: His brain isn’t handling all of the sensory information his body is providing. Or what it does get, it processes incorrectly.

This disorder is typically treated with occupational therapy and - in school - an Individualized Education Plan (IEP). The lives of kids with this disorder can be improved with learned coping mechanisms, but he’ll never see, hear, or feel the world the way most people do. I cannot describe the heartbreak I feel as a father, knowing what lies ahead for my son.

He’ll always be the wierd kid. The “trouble maker.” Studies show he’ll always zig when others zag. He may cope, but he’ll never thrive, never adjust, never fit in.

Maybe.

We have recently been exploring a neurotechnology treatment that might help Caleb’s brain figure out how to process all of the sensory inputs we take for granted. It is expensive, and it has no guarantees. But we’re going to give it a try.

I’ll be writing about the treatment as we go through it, quite possibly in a venue larger than inkBlots.

Stay tuned. When the first article goes public, I’ll link here.

Update: February 10, 2007. Read the series that starts today on Wired.com

Here’s an example feed: http://www.saletastic.com/rss/userrss/test.rss

The RSS 2.0 feed has a couple of minor validation problems (as of this writing), but all-in-all a nice way to go. Since you don’t know until you sign up, here’s an exhaustive list of the merchants who currently provide coupons through the service:

Sometimes old ideas can still be good ones. Now where’s the coupon for those nose hair trimmers?

Dear BlockBuster Online,

Your survey company RightNow Technologies doesn’t do you justice. If they were the the lowest bidder on providing surveys for you, it shows.

I got this in my email yesterday:

Dear Mark, Your opinions and feedback can make a difference! As a valued customer of BLOCKBUSTER Online, your input will have a direct impact on how we continue to improve our service to meet your entertainment needs. Please take a moment out of your busy day to share your thoughts with us in an important research survey we are conducting.

Sure, BlockBuster, you’ve been doing some nice stuff lately, especially with your “Total Access” program, so I’m willing to give a little feedback.

Following the link, I’m taken aback. The web pages are circa-1998 ugly, not at all what I would expect fropm Blockbuster Online proper. Checking the URL, I see that the survey is actually administered by RightNow Technologies. Oh well, how bad could it be?

About half-way through, the survey wants me to fill in several dozen fields, each blank and requiring a number. Hint to survey company: Default to zero on fields likely to be zero. This is where things get unbearable. Every time I enter I a number, I get a Javascript alert box telling me I need to use a number. “I am using a number!” This happens about 20 more times. So I view the source of the web page, and spot the problem:

onKeyPress="if (event.keyCode < 48 || event.keyCode > 57) {event.returnValue = false; alert('Please enter numbers only.');}";

The check on event.keyCode is proprietary to IE’s DOM, and doesn’t exist in FireFox, which is what I use. If you’re using FF, (and not in a RSS reader) here is how it acts:

Yeah, painful (unless you’re in IE.) Not at all cross-browser. Handling onkeypress events across browsers is pretty well documented, however. Or as a friend of mine says, “it isn’t rocket surgery.” Here is just one example of how to make it work in both IE and netscape/mozilla. After 20 more of the spurious alert boxes, I gave up and dashed off a note to RNT:

From: Mark Woodman
Subject: Survey is too painful to continue in Firefox
Message: All your number validation rules (”how many movies…”) are broken in
Firefox. It’s too much of a hassle to complete the survey.

Shortly thereafter, they responded with this gem:

From: Jean
Subject: Response (Blockbuster Online)
Message: I’m sorry to hear you’re having trouble with our web site, Mark. Internet Explorer is the preferred browser we recommend and is what our tech team uses to test with. Other browsers such as Mozilla/Firefox, AOL/Netscape, Safari, and so forth may not be completely compatible. If you have security or privacy software running such as Norton/McAfee/AVG, then that is very likely the reason why you’re having trouble. If you aren’t sure how to allow our cookies via your software, then please contact the developer for those instructions.

I don’t know what’s worse:
a) An online survey company that doesn’t want to do cross-browser checking in javascript in more than IE.
b) An online survey company that blames their own bad javascript on antivirus software or cookies.

I would love to know what percentage of Blockbuster users have Firefox/Opera/Safari. Whoever contracted this firm at Blockbuster Online should be more than a little embarassed by this.

Update Jan 16, 2007

The ironies continue to pile up. I sent a message via Blockbuster’s main customer support web form with a link to this article, but their customer support service appears to also be farmed out to RightNow Technologies. As if that weren’t bad enough, the support reps can’t get online from their own trouble ticket system:

From: Ryan
Subject: Response (Blockbuster Online)
Message: Hello Mark, Thanks for contacting BLOCKBUSTER Online Customer Care. Please update me with your concern because I just cannot click on the link you have given. Our system won’t read it.

I’m guessing this isn’t going to put me on RNT’s “favorite blogger” list. Any bets on whether Blockbuster HQ ever sees this thread?

“A content syndication platform, such as a web content syndication platform, manages, organizes and makes available for consumption content that is acquired from the Internet. In at least some embodiments, the platform can acquire and organize web content, and make such content available for consumption by many different types of applications. These applications may or may not necessarily understand the particular syndication format. An application program interface (API) exposes an object model which allows applications and users to easily accomplish many different tasks such as creating, reading, updating, deleting feeds and the like.”

That describes tons of applications both corporate and open source. Any programmatic API like ROME comes to mind. Did you create the RSS Platform in Windows Vista before all of them? Please do tell, I’m all ears.
(More objections by Dave Johnson here. CNET coverage here.)

One point that Mike made is that non-technical people look at feeds like a microwave… they don’t want to know about the neutrons and the power configuration. They want to know which button to push to make popcorn. The real power of the RSS/Atom story isn’t in how it works, but in what it does for you.

This goes back to Greg Reinacker’s comment in 2005 that “RSS is plumbing.” The specification merits of RSS or Atom will only win over the geekerati. Everybody else just wants to know where the popcorn button is.

So how do we make RSS as easy as microwave popcorn? Personally, I don’t see this happening until every email client - desktop and online - has free, robust support for RSS feeds. The biggest barrier I see in my workplace is resistance to “yet another application to monitor.” If feed items show up with their email, suddenly it isn’t so bad. Unfortunately, few email client feed readers are both good AND free. If GMail ever integrated Google Reader, that would be a killer, killer app.

Now where’s that butter?