Kurt McKee

I've started working to migrate my online presence back to kurtmckee.org; it's been two-and-a-half years since I self-hosted my blog, and in that time I've fragmented my identity across LiveJournal, Blogger, Tumblr, and a smattering of other services including YouTube and Flickr.

Thus far I've upgraded my Wordpress install, dropped a number of unused database tables, and cleaned up some inconsistencies in the comments metadata. I still need to convert the crufty latin1 database encoding to utf8, import all of my LiveJournal and Blogger entries and comments, and configure pretty URLs. I've yet to decide how I want to handle photos and tumblelog entries; for now I expect that I'll continue using Flickr and Tumblr (unless someone endorses an outstanding Wordpress plugin).

The best outcome so far has been the rapid increase of my knowledge of SQL (and, as a direct result of my experimentation, the discovery that I have good database backups). I can construct a select or update statement fairly competently now. Oh, and drop table, if I forswear tab completion!

Stupid tab completion...

Last night my mom fell in the icy darkness outside of our house and injured her wrist. To schedule an X-ray this morning she called Physicians and Surgeons, located at 900 Porter Avenue here in Norman, OK. She is currently uninsured.

When she called, she was told that they didn't know how much an X-ray would cost; the only person in the office who decides how much an X-ray costs an uninsured patient wasn't in, and wouldn't be back until the 17th. She then offered to pay the same cost as an insured patient and cover the difference when it was determined, but was told they didn't know how much an X-ray would cost for an insured patient, either: that same person determines that was well. In the end she was told to call back in a week when their person was back in-office and could determine how much an X-ray would cost.

Let me repeat that: Physicians and Surgeons employs a Goddamn oracle to confer directly with Heaven, apparently through a complicated but surely systematic reading of conch shell ridges and oak wood rings that would drive mere mortals instantaneously to hysterics. I assume my mom called just after the last uninsured person asked for an X-ray of their wrist, the result of the reading having overcome their oracle and driven her to self-immolation.

Clearly they expect a full recovery by the 17th.

I'm pleased to announce the release of listparser v0.13! This is an important bugfix release, and I recommend everyone upgrade immediately.

Bug fixes

The last release of listparser contained an infinite loop bug in the Injector code. Large documents that contained undeclared character references would trigger the bug, which occurred because the cache never got cleared, and each call to read() would return the same cached content over and over and over. This is fixed, and indeed the Injector code has been significantly simplified.

Additionally, I fixed a long-standing intermittent bug in the unit tests. Back when I released version 0.10, I mentioned that I was occasionally seeing a test fail; after a little sleuthing I discovered that the unit tests were sometimes starting to run before the server thread was ready to accept localhost URL requests. This bug is now fixed.

Unit testing

I've overhauled and modularized the unit testing code; most significantly, very few of the test files are retrieved from the server thread anymore, which has noticeably sped up the test suite. Additionally, a number of tests now call the function they're meant to test (such as _mkfile() and _rfc822()), rather than pushing everything through parse(). Finally, more tests were added, bringing the code coverage very close to 100% again.

Get it!

With all of these changes, I think that this may be the most stable listparser release ever. You can download listparser from PyPI and copy listparser.py into your project, or you can install it easily using easy_install:

$ easy_install listparser

Have fun!

listparser is a Python library that parses subscription lists (also called reading lists) and returns all of the feeds, subscription lists, and "opportunity" URLs that it finds. It supports OPML, RDF+FOAF, and the iGoogle exported settings format.

[ homepage | download | repository | documentation | bugs ]

Today, Cox Communications fouled up my internet connection and broke one of my listparser tests. I was surprised to find that one of my tests abruptly started failing today without a change to the code. Within minutes I discovered that Cox is resolving DNS queries in a way that breaks internet specs. See, when I request http://badurl.com.INVALID, now I get a webpage back with search results and ads.

I called up Cox and spoke with Doug. He began to tell me that I could opt out, and I informed him that I was aware that I could opt out; I was expressly calling to file a complaint so that Cox would know my displeasure. I told Doug that I understood the benefit to people who type gogle.com or goggle.com or google.con: they'll end up at google.com using this service, but I'm furious that .invalid, .test, and .example also redirect.

It was at this point that Doug argued with me, that many internet service providers were already doing the same thing. I argued back that up until now Cox hasn't poisoned DNS queries. Then Doug betrayed that he wasn't listening to me at all by arguing that people who type gogle.com or goggle.com or google.con...

I cut him off and told him that I had just said the same thing, that I understood the reasoning behind it, and that I wanted only to file a complaint so wouldn't he like to file that complaint for me or redirect me to someone who could?

Doug said he would make a note in the ticket and thanked me for choosing Cox.

Listen here, Cox, I appreciate the service you're providing (it's an absolute godsend for some people I know). I appreciate that I have the ability to opt out. However, I share an internet connection, and as I told Doug, I would like the reserved top-level domains (.invalid, .example, .localhost, and .test) to remain reserved so that I could leave the service enabled to the benefit of my parents while still being able to write my software to spec.

Updated

After attempting to opt out of the service and being unable to do so in Firefox for Linux and Internet Explorer 8 for Windows, I decided to look at Cox' operating system-specific opt out directions. Turns out it's just as easy as modifying your DNS server addresses:

New and busted

68.105.28.11
68.105.29.11

Old hotness (no DNS poisoning or anything!)

68.105.28.13
68.105.29.13

For the past six months I've been actively working to make listparser the single best subscription list parser available, and I've had a blast working toward that goal. I've spent most of my time improving the library, but my unit test code has begun to resemble a shanty town; the number of unit tests has grown, certainly, but they've obviously been dumped off in lptest.py.

Today I've been spending a good deal of time turning that shanty town into a respectable village, using a bunch of function factories to spit out prefab TestCase functions. In doing so, I've reduced the amount of time it takes to run the unit tests, and I've even caught another crasher bug! (I consider any uncaught exception to be a crasher bug.)

So why put in the effort to improve the unit tests? (And here I'll drop the shanty town metaphor, lest I say something about how the unit tests do the same job regardless of whether they're comfortably organized or not.) The reason is simple: lptest.py serves as a basis for writing unit tests for anything else I work on. Remember back when I released an improved syndication plugin for rawdog? I used lptest.py as a template for its unit tests, which meant that all of the lptest.py baggage carried over to it. Not cool.

I expect to push out another release of listparser in the very near future, particularly because I found an infinite loop bug in the 0.12 release.

...I mean, I don't *think* I'm an idiot, but my code seems to have a lot of problems with corner cases...

While driving to a client

Me: You've got the "Church of Jesus Christ, Scientist", and the "Church of Jesus Christ, Cowboy". They don't differ significantly in theology, they just emphasize different aspects of the Godhood.
Him: Right, and this one emphasizes the Jesus Christ that wrangles steer.
Me: And the Holy Spirit is the sheepdog to whom Jesus is yelling "Hep! Hep! Haw!"

I'm pleased to announce that listparser v0.12 is now available for download!

In this release I ironed out several crasher bugs on Jython 2.5.1 and Python 3.0. I additionally realized that there was a bug in the HTTP User-Agent code and fixed that (developers weren't able to override the global USER_AGENT variable). There were other minor changes, but the crasher and behavioral bugs are the big news for this release.

listparser is a Python library that parses subscription lists (also called reading lists) and returns all of the feeds, subscription lists, and "opportunity" URLs that it finds. It supports OPML, RDF+FOAF, and the iGoogle exported settings format.

[ homepage | download | repository | documentation | bugs ]

I'm proud to announce the release of listparser v0.11!

I've made two simple changes. First, listparser now runs on Jython 2.5.1. Second, parse() accepts local filenames. Up until now it's been necessary to use a URL notation like file:///home/smith/opml.xml, but no more! I'm pretty sure that it will run fine on Windows, but please test it and report back any errors.

listparser is a Python library that parses subscription lists (also called reading lists) and returns all of the feeds, subscription lists, and "opportunity" URLs that it finds. It supports OPML, RDF+FOAF, and the iGoogle exported settings format.

[ homepage | download | repository | documentation | bugs ]

While out shopping five days before Christmas; the cashier had just called for the next customer in line.

Her: Did you cut ahead of that guy?
Me: No, he was on his cell phone waiting for his wife and gestured to go ahead.
Her: Okay. I guess I won't crucify you, then.
Me: Yeah, save that for Easter.
Her: Well, I am half Jewish.

It's been over two months since the last listparser release, but believe me: the wait has been worth it!

Python 3 support

listparser is now 100% Python 3 compatible! I can prove it, too: it passes all of the 159 unit tests! This is in addition to the existing support for Python 2.4, 2.5, and 2.6. In order to convert listparser to Python 3 format, simply run the following command:

$ 2to3 -w listparser.py lptest.py

After a little churning, 2to3 will write out the necessary changes, and you can run the unit tests by typing:

$ python3 lptest.py

(On that note: I noticed a weird issue where Python 3 would only occasionally fail a single test the first time it was run. Re-running the test suite, however, makes the problem go away. I have no idea what's causing that...)

Support for undeclared character references

I've known for a long time that listparser would eventually need to handle undeclared character references to be robust, so I finally took the time to whip up DOCTYPE injection code. The basic idea is that if listparser encounters an undeclared character reference (such as æ) it will inject the necessary declarations and re-parse the document. Thus, æ would be correctly transformed to æ. Sweet.

Crasher bugs, et cetera

I identified several places where listparser might crash and patched those problems right up. Additionally, thanks to coverage and cProfile, I identified several places that hadn't been tested thoroughly, and even improved some very stupid code that made listparser take over five minutes to parse the Planet KDE FOAF file (which, by the way, listparser can now parse just fine).

Cheeseshop support

listparser has really grown over the past six months, and I figured it was time to make it available at the Python Package Index. Assuming you have easy_install on your system, you can now install listparser by typing:

$ easy_install listparser

easy_install will automatically go out and get the latest version for you. By the way, I discovered that the Python Package Index can host documentation, so may I recommend that you go check out the listparser documentation? It looks great!

listparser is a Python library that parses subscription lists (also called reading lists) and returns all of the feeds, subscription lists, and "opportunity" URLs that it finds. It supports OPML, RDF+FOAF, and the iGoogle exported settings format.

[ homepage | download | repository | documentation | bugs ]

This evening while driving home from work I happened to see a woman walking along the side of the highway. She turned briefly to oncoming traffic and threw her arm and thumb out wildly, obviously trying to catch the passing traffic's attention. For some reason I felt compelled to stop the car, noting sardonically that if it had been a lean, aged, and bearded man with a large backpack on I wouldn't have stopped.

When she got in the car I asked where she was going and was shocked to hear that her destination was thirty minutes away in the opposite direction. She began crying and thanked me over and over for stopping and told me that she had been walking for an hour-and-a-half, wishing that a passing car would just hit her. I told her to give me a second to get turned around, and she cautiously corrected me that the destination was ahead. Moments later she realized that she had been walking for an hour-and-a-half in the wrong direction and cried all the more.

She told me that four men broke into her home and murdered her husband, and that she was a witness. She and her kids had had to move out of their home, but shortly after the subpoena came to her temporary home, someone came and blew up her car (she said that the fire department told her that a bomb was planted on the car). She and her kids were now living wherever they could. She had no family to turn to, and without a car she wasn't able to get to work consistently. She had gotten disoriented earlier in the afternoon while walking back from looking for a used car. She continued crying and coughing, and apologized over and over.

As we approached her destination, I considered that I had no cash, but had the ability to buy her family dinner. I pulled into a drivethru, and she asked me to order two small burgers and some french fries. I joked that that wasn't enough food to constitute a dinner for me. When she replied that she and her kids didn't eat much, I smiled and said I don't believe you. I think you're just trying to be careful on my behalf. Turning back to the window I blurted out Make that three burgers, and pulled forward.

I spent $15 on my own lunch today. Why oh why didn't I blurt out Triple that order! Why oh why didn't I blurt out something that would have turned that order into a meal?

This evening I gave a woman a warm car to ride home in, an opportunity to voice her troubles and weep for her family anonymously, and I delivered her directly to her children. It cost me only time, and I feel miserable because I shorted her and her two children a filling meal.

While waiting on printer drivers to install:

Me: Do you use teeth whiteners?
Her: No. I just use a pre-brushing rinse.
Me: A what?
Her: It's made by Listerine. It's not Listerine, though. That's bad for you. It dries out your gums.
Me: Well you could always pop in a Burt's Bees Gum Balm.

While talking with a doctor at one of the sites I work at, she told this story:

During my residency, a prescription pad was stolen from the office. A few days later I got a call from a pharmacist. "Now, I know you didn't write this prescription, but I just had to call and tell you about it. Someone just gave me a prescription for 'mophine...one pound'."

If the name of the food product takes up less space on the packaging than the list of things that it doesn't have in it, I don't want it.

I've finally "finished" my work on the rawdog rss plugin (which should realistically be called "syndication.py" or something similar). You can view the results at the rawdog-rss github repository. The plugin was originally authored by Jonathan Riddell, and was later updated by Adam Sampson. While I may leave the git repository online, it should not be considered the new plugin homepage. Now, with that out of the way, the big changes I made:

1. There's now a unit test framework built up around the plugin.
2. The plugin outputs well-formed XML (in RSS, OPML, and FOAF formats).
3. FOAF output is now more better.

I also added a hack: after noticing that the Planet KDE configuration file included HTML references instead of Unicode characters (i.e. "á" instead of just "á"), I decided to make the plugin do what was intended, not what was actually written.

The only other thing worth noting is that you can't run the test suite without modifying the rawdog source code. Comments, questions, and criticisms are welcome. Bug reports should be filed at github.

While perusing Adam Sampson's website (the author of rawdog), I noticed that he maintains a list of ideas he's had. I think that's a great idea, so I'm going to take note of something that occurred to me a few moments ago.

I often glance at Planet Gnome (PGO). I'm already subscribed to many of its syndicated authors, but every so often someone who hasn't posted in a while shows up on its front page, and I find someone new to subscribe to. But therein lies the problem: I'm wasting significant time maintaining a separate list of feeds. PGO has a blogroll in two formats, but my current feed reader doesn't give me any tools for filtering out the authors and content that are irrelevant to me.

Linus Torvalds once noted that If you can do something really fast, really well, people start using it differently (skip to 50:38). He was talking about merging in git, but I think it applies to feed reading as well.

Earlier this week I found an author who was posting about a bit of software he had written. I was discouraged from subscribing to his blog because he used tags and categories so disjointedly that I knew that by subscribing to a tag- or category-specific feed I would either be inundated with unrelated posts, or I would miss relevant content that I really wanted to see. And that's when it hit me (again) that my feed reader doesn't (and all feed readers I've ever used don't) give me the tools to insta-subscribe, and insta-filter, and insta-unubscribe (heaven forbid) when it's clear that the content doesn't interest me any longer.

If and when I write my own feed reader, I hope I remember to make it incredibly easy to subscribe to feeds, but brain-dead easy to filter content on my terms.

After previously noting that listparser chokes on invalid XML (as it should, but eventually won't), I've taken it upon myself to patch the software that was creating invalid XML in the first place. I found that Planet KDE's FOAF blogroll was being created by a plugin for rawdog.

I wrote to both the plugin author and the rawdog author and got the go-ahead to update the plugin, so I downloaded the latest version of the plugin and set about creating unit tests. Right off the bat I discovered a bug in rawdog: it was reading -- but ignoring -- my custom configuration files! The solution is to add a single line near the bottom of rawdoglib/rawdog.py:

  elif o in ("-c", "--config"):
      try:
          config.load(a)
      except ConfigError, err:
          print >>sys.stderr, "In " + a + ":"
          print >>sys.stderr, err
          return 1
+     rawdog.sync_from_config(config)

With that fixed, I don't think it'll be long before the plugin will be outputting valid FOAF and OPML files.

I'm very pleased to announce that listparser v0.9 is now available! There are some big changes in this release.

RDF+FOAF

listparser now supports RDF+FOAF, an XML-based subscription list format that the vast majority of Planet-based websites produce. If you like reading what the Pidgin developers are up to, keep in mind that Pidgin Planet has a FOAF blogroll. Or if you're an RDF junkie, Planet RDF has a FOAF blogroll. So do Planet Gnome, Planet Ubuntu, and Planet Mozilla.

Adding support for another format was exciting, but it prompted me to finally fix an issue that had been bothering me.

Opportunities

A while back I discovered that Wordpress-based blogs can output an OPML version of the links in the blog sidebar. What I noticed, however, was that most blog owners only put in the linked blog's homepage, and not its feed URL.

While implementing RDF+FOAF support I discovered that the same problem could exist in that format (see Planet Freedesktop's FOAF blogroll for an example). Thus, I decided to save those homepage URLs.

The way I see it, software can either choose to beat users over the head for problems they neither understand nor have any way to fix, or it can do its best to handle the problems gracefully. By storing those homepage URLs, feed readers that use listparser have the opportunity to run feed and subscription list autodiscovery software against those homepage URLs. Obviously they don't have to, but its an opportunity for the feed reader to help the user, rather than hinder them. Its for that reason that I called the set of homepage URLs "opportunities".

distutils support

At long last, listparser can be easily installed using the standard Python distutils. Rather than just copying listparser.py somewhere in your path, you can now just install it like any other piece of Python software:

$ python setup.py install

It's just as easy as that.

In closing

listparser has become much more mature; there is exactly one more significant shortcoming that I'm aware needs to be addressed (you can look at Planet KDE's broken FOAF blogroll to see exactly what listparser -- like Firefox -- will not handle). However, I think I'm going to take a break from listparser and recharge my batteries.

listparser is a Python library that parses subscription lists (also called reading lists) and returns all of the feeds and subscription lists that it finds. It currently supports OPML, RDF+FOAF, and the iGoogle exported settings format.

[listparser homepage]
[listparser downloads]

While working on September 11th, I had a number of computer boxes to open and tear down for a lady. I whipped out my trusty box cutter, made short work of the boxes, and commented to the lady how useful the tool was. She turned her eyes down and raised her eyebrows while shaking her head, saying I just don't know if that word will ever be the same after 9-11.

I was surprised; her tone didn't suggest that she was reprimanding me for saying "box cutter"; it just seemed like she was hollowly acting out a part. I replied that Well, box cutters can be useful no matter what day of the year it is.

I just think that, I don't know, maybe some words should be treated differently on the anniversary of 9-11.

As the conversation continued I became irritated with her. There aren't merit badges for NEVER FORGETting, but this lady seemed determined to demonstrate that she could be more plaintive than anyone else. While honoring the event doesn't bother me, putting on a show about it does, and that was the impression she gave me.

listparser has had some magical code in it for a while, and today I removed the two most obvious magics.

If you've ever looked at the code and have seen the expect-related code, you've probably said "wat". Yes, the code was absurd, but it's now gone. If you don't know what I'm talking about, just appreciate that the stage has been set for support for another subscription list format.

The other big change (and the one I'm most pleased with) is the replacement of the tag and category code. When I first started writing it, list comprehensions seemed like an obvious solution. Unfortunately, as I discovered corner cases I continued to solve the problem with more list comprehensions. Eventually I ended up with this ugly mess:

def or_strip(x, y):
    return x.strip() or y.strip()
tags = [x.strip() for x in attrs[(None, 'category')].split(',') if x.strip() and '/' not in x]
cats = (x.strip() for x in attrs[(None, 'category')].split(',') if '/' in x)
cats = (x.split('/') for x in cats if reduce(or_strip, x.split('/')))
cats = (xlist for xlist in cats if reduce(or_strip, xlist))
cats = [[y.strip() for y in xlist if y.strip()] for xlist in cats]

Splits and strips and lists, oh my! All told, I replaced 15 lines of code with 16 lines but gained shorter line lengths and more maintainable code.

I'm looking forward to releasing the next version of listparser.