From the hip

The Overwhelming Loss of Self

2008-09-08T22:14:00.002-04:00

I suppose it had never occurred to me before. I mean, people died when I was younger but I guess I never really put two and two together. Maybe nobody does. I suppose it actually had much more to do with the type of relationship I had with those who died. I'm not talking about the quality of the relationship; no, this has more to do with socio-geography than it does with long walks on the beach. The first people that I remember dying were not people that I spent a lot of time with, either physically or in my head. In fact, I probably would have learned this earlier had I spent more time with some or all of them. I heard something on the radio today that I liked quite a bit: when you lose a parent, you lose the past; when you lose a child, you lose the future; when you lose a spouse, you lose the present. Losing the past is something terrible, but as children or young adults, we are pretty used to losing the past; we welcome it in most cases. Even so, we still lose that part of ourselves that existed only in the intersection of our lives with the other person's. Losing a child or a spouse (or sibling or friend) is even more terrible in the sense that you lose not only the part of you that existed in conjunction with that person, but also all the moments that were to come. Horrifically, it is the death of potential. Even though your potential may be realised eventually, the dance with that person is over. Insofar as that part of you is defined by its relation to the dance, you die too. To all those who have died but are still alive, I'm sorry for not understanding sooner.

How to add trusted keys to apt

2008-07-14T19:44:00.000-04:00

The aptitude manual has a little section on how add trusted keys to apt.

The list of keys that apt will trust is stored in the keyring file /etc/apt/trusted.gpg. Once you have the GPG key, you can add it to this file by executing the command gpg --no-default-keyring --keyring /etc/apt/trusted.gpg --import newkey.asc . aptitude will then trust any archive that is signed with the key contained in newkey.asc.

Qemu networking goodness

2008-07-09T13:32:00.002-04:00

It took forever but I finally got all of my Qemu networking working. My setup uses VDE, a TUN/TAP device to connect to my LAN, dnsmasq to give my QEMU hosts IP addresses and handle DNS requests, and IP Masquerading because apparently my wifi card can't spoof MAC addresses.

There follows a (probably incomplete) description of the setup

Setting up the TAP device

First thing to do is create the TAP device we will use to connect the VDE network to the LAN.

sudo modprobe tun
sudo chmod 666 /dev/net/tun     # I'm all alone on this box so...
sudo tunctl                     # This should create a device called tap0
sudo ifconfig tap0 10.0.0.1 up  # This is the IP for the VDE network

We're all done with TAP stuff.

Set up IP Masqerading through the TAP device

Now we need to make sure that traffic coming through the TAP device gets sent out over the LAN.

sudo su -c "echo 1 > /proc/sys/net/ipv4/ip_forward"       # Enable IP forwarding
sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE -v          # wlan0 is my wifi card

OK, that's it for IP Masquerading.

dnsmasq for DNS requests and DHCP

This one was easy; just install the package and modify the conf where it says #interface= to say interface=tap0 (without the comment mark and substituting whatever you got back from tunctl above.

VDE setup

First we'll create a virtual switch

sudo vde -s /tmp/switch1

Then give everybody access to the VDE

sudo chmod -R a+rwx /tmp/vde.ctl

OK, that's it for VDE.

Qemu hosts

This was a trickier bit. The stumbling block for me was that if you specify a MAC address for the host (I'm using Debian Etch as the guest OS), a new eth device is created. So make sure you specify the right MAC address from the start. If you screw up, you can always edit /etc/udev/rules.d/z25_persistent-net.rules and remove the extra eth devices. The reason this happens is that udev figures out that you added a new card (because of the new MAC) and so it configures another device. There's probably a more elegant way around this, I just haven't figured it out yet.

Boot your Qemu host

After all of the above, booting should go smoothly; remember to specify -net nic,macaddr=XX:XX:XX:XX:XX:XX -net vde. If you don't get an IP automatically, just run dhclient ethX on the guest and you should be set.

Frustrated by UML

2008-06-29T02:49:00.001-04:00

I recently have been experimenting with User Mode Linux. My goal is to be able to run some of my tests on my local machine without needing to setup a whole EC2 instance.

At the same time I am evaluating the possibility of using UML on my instance to separate out the various services.

The main problem I am having at the moment is with the networking. Although I am able to connect to the host machine, I seem to be unable to create a route to the rest of the network.

They were once so shiny...

2008-06-11T18:33:00.005-04:00

I was just looking at this picture of my pipes. I had forgotten how shiny they were when I first got them. Now the brass has started to develop a patina so it is quite a bit duller and a little splotchy in places.

The patina is actually desirable (for a lot of pipers) as it makes the pipes look antique. We would probably all play 100-year-old sets if we could.

The wood (which is cocobolo - a type of rosewood) has also darkened considerably since I got the set. The maple in the mainstock hasn't changed much though, nor have the mounts. There is a slight discoloration of the chanter mounts since I keep it in a red leather case I made that has tinted the wood somewhat.

The three things that make us great: culture, culture and culture

2008-04-03T15:30:00.002-04:00

My last post was in response to George Dinwiddie's "What would you like your software developers to learn" question. Here is a follow-up exchange that I wanted to share.

George wrote back to me asking:

Are your developers learning more how to take a long-term view? What steps are you currently taking to help them learn? What steps do you think you should take, but aren't yet, for some reason?

The rest of this post is my answer.

I think that one of the key roles that manager/architect types need to play is that of strategist. As one of my senior developers put it recently: "I know about tactics, what I need is strategy." An ability to predict the outcome of various actions is, in my experience, an indispensable quality in those who are responsible for guiding the overall direction of a development effort.

Some developers care about learning how to do this and some don't. Not everybody on the team needs to be a superstar, and there is usually no shortage of trash to be taken out by those who want to do that (or who lack the chops to do much else). On the other hand, those developers that do want to grow should be encouraged and given the appropriate tools.

On my team, I have a pretty heterogenous group where about 30% of them are interested in learning this stuff. They come from a variety of backgrounds and so some are starting from further back than others. I find that a good strategy for winnowing the grain from the chaff is to offer some of the basic tenets to everybody and see who picks up on it.

We have a 1.5 hour workshop once a week where we go over various parts of our codebase, learn about new techniques, have different developers do presentations on code they have written or had to maintain, etc. Generally I follow up on those workshops with additional attention to those whose eyes do not glaze over when we start talking about certain things.

We also have a very strong team culture that encourages developers to watch each others' backs and make sure that the "social contract" we have all agreed to is upheld. Most of the questions that I listed in my initial answer are things that my guys and gals check each other on all the time. It's a bit of a game: who can find the mistake in the other person's work. This, of course, is all done in a spirit of cooperation and mutual acceptance of constructive criticism (critical components in any team).

I am fortunate to have the leeway to do pretty much whatever I want to improve the current performance of my team. In that respect, I rarely find that there are things I want to do but cannot. This is not the same in every environment, of course, but I select the companies I work for specifically keeping in mind the fact that the most successful companies are the ones that pick the right people, put them in the right place, and get out of the way while they do their job.

At the end of the day I would say that the single most important factor in turning competent developers into excellent ones is culture. I try to lead by example and I expect my seniors to do the same. Every senior on my team had the role and responsibilities long before they had the title. Every junior on my team is capable of training up new juniors (and even some more intermediate-level developers) to be effective within two iterations (four weeks). The culture on our team is one of constant improvement and the juniors frequently teach the seniors lessons (which the seniors take in stride). With the right culture, everything else just seems logical. Without it, you are constantly swimming against the current.

From competence to excellence: what developers should know

2008-04-03T11:30:00.002-04:00

One of the things that most developers have a lot of trouble with is a long-term view. I think it is critical that there be a very good understanding of the impact of current decisions and design choices on the future of the project and the company. A lot of developers (and in my experience this is only worse with contractors and consultants) have the impression that once the code is written, the job is done. They frequently omit details such as sustainability of the overall system and how easy it will be to make future changes.

One good indicator of a developer's current level of understanding of this concept is to look at the following:

Coding Practices

Does the developer write idiomatic code that would be easy for any other developer to understand given a basic comprehension of the development environment? Are the interfaces to their objects clean, minimal and easily understood? Are the techniques used to modularize their code sparingly and appropriately applied? Do they consistently strive for high coherence and low coupling in the codebase? Do they understand the value of a properly written test suite?

Source Management

Does the developer understand more than the basics of their source control tool? Do they know how to do branching and merging properly? Do they leave clear, concise comments for each commit? Common errors when using SCM tools include incorrect spelling for changeset names (which makes searching for a particular changeset difficult), bundling of unrelated changes into a single changeset (which makes reverting only one of the changes difficult/impossible), linking changesets to the incorrect development task (which makes it difficult to assert that a particular task has been completed successfully) and fumbling with branch and merge (which causes problems when a developer needs to work on some separate task - how do I fork the codebase? how do I re-integrate my changes? etc.)

Development Environment

Is the development environment properly documented and easy to re-build? Can each developer work on any other developer's machine or do they need their "own setup"? Does the development style employed by the developer depend on a specific tool that may not be supported in the future (i.e. web developers who do everything in DreamWeaver, Java developers who are lost without Eclipse's auto-complete and refactoring tools, etc.)? Is the project neatly organized such that it's physical layout (on the filesystem) is easy to understand/explain?

Developers who have a good grasp of the above concepts also usually have a more complete understanding of the lifecycle of a software product (as a product and not as a project). By looking ahead, and by applying certain principles now, we can often have a huge impact on the future maintainability of a system. Maybe I'm just lucky to work with guys and gals that have the basics down pat but I would say that the above is what separates the "software developers" from the "coders". Obviously, there are a number of more basic items that come first but a long-term view of the evolution of a codebase is something that lifts developers from competence to excellence.

Asterisk moved into EC2 cloud

2008-03-10T19:35:00.001-04:00

I moved the Asterisk server into the EC2 cloud today. It was pretty quick.

Running it off my home box was no longer an option since we have people testing the service these days and I want to make sure there are no outages. I still need to setup a test server so we can test out new configurations before putting them live. I had previously hacked up a little configuration script that shuffles stuff around and performs a couple of emerges. I used the "eminent" AMI that is based on a simple Gentoo install. I plan on putting together my own clean base image at some point in the future but right now this one is good enough. I also plan on building a nice overlay that I can use with portage to update my boxen quickly and efficiently.

FWIW, I figured out how to get Asterisk working from behind a NAT. It was as simple as setting the externip variable in the sip.conf file.

Using encrypted SSL keys and certs in twill

2008-01-24T16:34:00.001-05:00

Recent versions of mechanize support SSL connections and so, by extension, does twill. Recently I have been trying to open an HTTPS connection to a site that requires both a certificate and a client key. To try this at home, you will need a version of openssl.

The key and cert came to me packaged in a PKCS#12 format file. The first thing I needed to do was unpack the key and the cert like so:

openssl pkcs12 -clcerts -nokeys -in cert.p12 -out cert.pem
openssl pkcs12 -nocerts -in cert.p12 -out key.pem

After doing this, I plugged in the filenames of the cert and key in my twill code. Note that you need to access the underlying mechanize.Browser object when setting the client certificate:

import twill
from twill.commands import *

host = 'mysecurehost.com:443'
b = twill.get_browser()
b._browser.add_client_certificate(host, 'key.pem', 'cert.pem')

go('https://mysecurehost.com/secured_url.html')
show()

The problem you run into when doing this is that the SSL libraries prompt the user to enter the password for encrypted keys at runtime. This makes automating the interaction tricky at best. I tried fumbling with the PyOpenSSL library but it seems that setting a callback for the passphrase retrieval does not actually work. The set method call returns but the callback is never called during key decryption.

My (hackish) solution was to remove the encryption on the key before using it to connect. You can do this by re-exporting the key from the PKCS#12 file:

openssl pkcs12 -nodes -nocerts -in cert.p12 -out unencrypted_key.pem

Now if you use unencrypted_key.pem in the twill code above, you will be able to connect without providing a password for the key.

Centralized version control blues

2007-09-21T09:52:00.000-04:00

I am in the process of migrating a SVN repository to Darcs. A couple of the changesets in the SVN repo make baby Jesus cry. They create conflicts such that checking them out in order fails spectacularly and you need to go hands with the working directory to clean everything up before moving on. All of this highlights a rather interesting (though totally predictable) issue that plagues centralized version control.

In Darcs, if a changeset is no good, you can just refuse to pull it. Doing so excludes that patch from your repo and no harm is done (unless, of course, a subsequent patch depends on it). In SVN, given that everybody checks into the same repo, by the time you realise that a patch is going to trip you up, it's already in the bloody database. Now, SVN provides no tools for deleting a specific revision (although I hacked together something that I'll try to toss out there someday) so you can't exclude the changesets that put the DB in an inconsistent state. All of this means that I have been wrestling for the past week and a half trying to get this stupid repository to ignore the "corrupted" changesets. Arrrrgh!

I just called...

2007-07-30T13:08:00.001-04:00

Two weeks ago I purchased my first wake-up call. It was late at night and I had a particularly important meeting the next morning that I really couldn't afford to miss. So I slapped "wake-up call" into Google and hit the first site that popped up. If I'm not mistaken, it cost me $1.20 USD for that call.

The second time I wanted a call I did a little more research and turned up a site that offered calls at $0.75 USD with a "snooze package" of three snoozes for an additional $1.20 USD or something. I used this site several times trying to work out how the different options work.

Then (as usual) I got frustrated by the fact that I was essentially paying to use somebody else's script when I could write one myself. I spent and evening getting cozy with the Skype API and finally (after quite a bit of hair-pulling) figured out how to get SkypeOut to call my home phone and play a sound file (in WAV format) to me.

This weekend I put the finishing touches on my new wake-up call script. I can now set a date/time and have my computer call me to either remind me of something or to wake me up. I could also do other, wacked-out things like have my computer monitor an inbox for an important email while I'm at the country and then have it call me and read (via Text To Speech) the email's contents to me over the phone. This morning I used the scheduling to wake up and it worked like a charm!

The only problem now is that Skype charges me $0.024 CDN per minute for each call as well as a $0.05 CDN connection fee for SkypeOut. Still, it's much better than the $0.75 USD which was the cheapest I had found before (that is, without a subscription).

Performance Tuning - applying a function to a list

2007-06-15T19:39:00.001-04:00

>>> import timeit
>>> timeit.Timer('map(f, range(10))', 'f=lambda x: str(6+x)').timeit()
15.467374484745967
>>> timeit.Timer('[f(x) for x in range(10)]', 'f=lambda x: str(6+x)').timeit()
16.062227741235269
>>> timeit.Timer('for x in range(10): f(x)', 'f=lambda x: str(6+x)').timeit()
14.686095299821623

And so, we can see that map is still faster than list comprehensions and the for loop beats them both. If you don't need the return value of that function, don't create the list: friends don't let friends create unnecessary objects. On the other hand, if performance is critical and you need the return values, you should prefer map over a list comprehension.

Picking teeny-tiny cherries

2007-06-15T15:58:00.001-04:00

I would love to use Bazaar or Mercurial as a DVCS; just the fact that they are written in Python (and therefore eminently hackable as far as I'm concerned) is worth making the move. I just can't get past the fact that Darcs (my current favourite) provides support for hunk cherry-picking. This is a killer feature. It's great to be able to cherry-pick files into patches, but Darcs actually allows me to package up my hunks that relate to different changes into different patches. This gives me much more control over the way I decide to describe the modifications I am making to my project. When I can only operate at a higher (coarser) level of granularity (like Mercurial or, God forbid, SVN) I get stuck fiddling around removing hunks from files before committing the patches.

Another project to take up my time

2007-06-05T00:41:00.001-04:00

I've just tossed out the first version of a small web-stack for Python called Khepri - a clever bit of word-play on the name "Apophis". There are a million others out there but I'm very picky. This one is mine and I like the way it is so far. Obviously, it is far from being really usable but if you are a developer, you can judge for yourself. Check out the darcs repo.

The cat came back the very next day

2007-05-06T18:33:00.001-04:00

Well, not quite, but he is back. Tygger finally came home on Friday night. I came back from an evening of recording stuff at Justin's and called out to him out of habit. I heard somebody yelling at me from the neighbour's porch and got angry for a moment at Jack for looking so much like Tygger. I headed towards him to pat him anyway and he fled. It was then that I noticed that he wasn't wearing his bell. I followed him for a bit then gave up when he continued to elude me. As I headed upstairs, I thought once again "wouldn't it be stupid..." So I picked up the flashlight and went back outside. I crooned and cajoled for about ten minutes before he finally bolted past me and into the house. He was super-thin and even once he was inside I had to open his mouth to check for the broken tooth that confirmed his identity. He still smells a bit off but that's nothing a couple of baths won't cure. I guess that for now everything is right in my little world!

Tygger still missing

2007-04-25T10:03:00.001-04:00

Our little guy is still missing and although everybody seems to have a story about how so-and-so's cat took 18 days to come home and their cousin's dog once crossed all the continents to get back to his owners I feel less and less confident every day that he will return. Marquis is all depressed and doesn't play; he just wanders around looking for Tygger and trying to get attention from us since he has nobody to play with. I certainly hope that he is huddled under a porch not twenty yards from our front door but I fear he is wounded somewhere or that he has been "taken in" by some well-meaning people who have discovered what a wonderful cat he is.

Linking into the weeb

2007-04-23T09:51:00.001-04:00

So Justin spent the weekend setting up a bunch of accounts for himself at blogger and del.icio.us. I figured he needs a bit of link love since he can't manage to get himself onto the first page of Google's results. Now, I'm not sure that Google still indexes blog posts the way they used to but you can check out Justin's blog or go directly to his website.

Missing

2007-04-23T08:35:00.001-04:00

Our cat, Tygger, got out the window Saturday night while I was sleeping on the couch in front of the TV. We spent the day yesterday looking for him and calling his name but to no avail. Marquis is starting to feel a bit lonesome. We put up posters this morning around the neighbourhood but I'm not really all that optimistic. We worry that he may have tried to go back to our place on Stratford. The tricky bit is that the wife of the guy who is renting from us isn't supposed to know that we had a cat living there before they moved in. Anyway, we'll go and put up more posters this evening. The apartment seems very empty without our little guy snoozing away on his blanket on the couch.

2nd April Menu

2007-04-18T08:57:00.001-04:00

April is cool with frequent showers, hail and just a little wet snow. But spring is just around the corner. You can smell earth in the humid, expectant air. Lightness to match the season coming is in order, but these past weeks have been quite chilly, so a little warmth would not go amiss. Thus I propose to you the 2nd April menu with a hint of the exuberance of youth and just enough heat to drive winter's final tendrils back into the darkness. All dishes listed are low calorie unless indicated (with a *).

Entrées & Sidedishes

Cheesy Broccoli Bake

Oriental Broccoli Salad (soy, honey and sesame)

Broccoli with Herbed Breadcrumbs

Roasted Zucchini with Fresh Thyme

Asparagus Tips with Roasted Red Peppers

Orange Ginger Asparagus

Spicy Nutmeg Carrots

Ruby Carrots (cranberry)

Carrot and Coriander Soup

Cheesy Chicken Chowder

Main Courses

Zesty Chicken Sauté

Apricot Glazed Chicken Breasts

Sesame Chicken Salad

Skillet Shepherd's Pie

Bell Pepper Stir-Fry (on white rice)

Risotto Napoletana (sun-dried tomatoes, salami and parmesan)

Broiled Sesame Salmon

Oven-Baked Tandoori Salmon

Desserts

Raspberry Sorbet

Fruit Crisp*

Last know good configuration

2007-03-28T10:30:00.001-04:00

Darcs has a feature (that I have never really tried) that allows you to locate the last working version of your source. I am currently shlepping through changesets in SVN applying them one by one in order to figure out which one breaks a series of tests that depend on our mocking library. It seems like it would be so simple to just repeatedly run the tests, stepping back through revisions until the tests pass.

Twittering tools

2007-03-28T08:04:00.001-04:00

Last night I installed Twitterbar, an extension for Firefox that adds the ability to post to Twitter straight from the address bar. In conjunction with the Jabber extension, JabBar, this provides a really nice little interface for using Twitter. I have a small, single line rolling feed from Jabber and since Twitter updates me via IM I have a mix of lines from any conversations I'm currently having with my contacts and any updates from people I'm following on Twitter. Now, I also have a way of easily posting twitters without changing the current window I'm using. Previously, I would have to open up the conversation panel with the Twitter IM bot, now I just fire and forget.

A reality check for old party lines

2007-03-27T08:53:00.001-04:00

Last night Quebeckers overwhelmingly told both the PLQ and the PQ that they no longer want to focus discussions along sovereignty/federalism lines. The ADQ was boosted into a position as the official opposition and I think that, this morning, people are looking for meaning in the moves from yesterday.

It is important to remember that a fairly low proportion of the population should be voting "strategically". The reason for this is simple: unless the results in your riding are really a forgone conclusion (and last night should serve as a reminder that nothing is as certain as we think), you may end up splitting the vote against your party. To have the foresight required to strategically align one's vote so that an overall message is communicated is equally problematic: in trying to send a message to my party by issuing a protest vote I may actually participate in their defeat (given the extremely thin margins between candidates in some ridings...cough...Sherbrooke...cough).

A common error is to ascribe some sort of intelligence or design to the results of an election. Phrases like "the voters have spoken" or "Quebeckers are sending a strong message" are not false, but they only hint at the underlying truth: the message delivered is a composite one. To boil it down to any one factor or issue would be to imply some sort of external collusion on behalf of the voting population. To look only at the votes that counted towards a win is equally treacherous, especially when one notes that most ridings had at least two if not three candidates neck and neck all evening long. Indeed, one of the more cogent reflections of the evening remarked that in addition to winning 41 seats, the ADQ managed to come in second in over 40 ridings. Wow. Think of what could happen if they managed to attract a couple of extra votes! Dumont was absolutely correct in saying "on connait la prochaine etape; je vous y donne rendez-vous."

All in all, I am pleased with the results of this election; a lot more pleased than I have been in recent years. I was disappointed to not see any Green Party members garner seats but noted with satisfaction that QS did not do much better. An unfocused and inexperienced campaign may have hurt these two parties - certainly QS did not exactly shine in a post-election speech last night where all they could muster was some vague ranting about proportional representation (a point that should be conceded but that hardly belongs exclusively to small struggling political parties).

I am happy that Quebeckers have decided to keep a leader that has experience. I am happy that they have placed a young, dynamic and energetic "watch dog" opposite Mr. Charest at the National Assembly. I am happy that the rise of the ADQ shows that a party can amount to something even if it is not one of the province's "natural" parties. I am happy that the election did not get polarized along "nationalist/federalist" lines. I am happy with our decision to have a minority government: in the future I think we'll see a lot more of these as we move from strongly polarised discussions to shades of grey. The only time we should have a majority government is when everybody agrees and the talent and experience exist to support that arrangement. Mr. Dumont is not ready for power but the last 12 years have certainly shaped his character; I look forward to seeing him push for a government of his own 2 years from now.

Mostly, I am happy with our commitment to the future of our province: by providing for an experienced guiding hand and a powerful young movement, we have the best of both worlds. By smacking down the PQ we have said, in no uncertain terms, "your platform is not for us this year, you will play king-maker and nothing else; when we are ready to discuss sovereignty again we have no doubt you will be waiting."

Term "visible minority" considered harmful

2007-03-08T19:58:00.000-05:00

A UN committee released a report today on Canada's efforts to combat racism. The report praises Canada on some aspects of it's anti-racism programs but says that the term "visible minorities" may not be "in accordance with the aims and objectives of the [International Convention on the Elimination of All Forms of Racial Discrimination]". The convention says that "distinction based on race, colour, descent, or national or ethnic origin is discriminatory".

Great. Now we can no longer tell the difference between a black man and a Ukrainian because to make the distinction between them (that is, to employ another definition of the word, to discriminate) based on the colour of one's skin and the other's nationality is "discriminatory". Um yeah. Aren't we pushing this a little too far? I mean, how far am I from being an "individual with low amounts of skin pigmentation from a city near the coast with parents and in posession of a penis"? Come on! Sometimes we need to differentiate between two people or two groups.

Now, I understand that penalizing one group or another based on these distinctions is what is really at issue here. That doesn't stop me from being a bit frustrated that there are actually people somewhere in the world paid to come up with this shit, write it down and hand it out to other people who are paid to read it. We could be solving "real" problems but instead we are haggling over what to call somebody who is black. How to talk about a deaf person without mentioning the fact that they can't hear.

Talking about people's differences or referring to them in some way is not discriminatory. There is only a problem when those differences cause you to judge based on those differences instead of based on personal merit. Now go home, make friends with a black deaf retarded gay woman and call it a day.

Version control through the ages

2006-12-02T17:37:00.000-05:00

It just occurred to me that over the past six or seven years I have gone from using CVS to track all my code and projects to using Subversion and from there to using Darcs. In general I find that each one has really been an improvement over the previous tool. I miss certain Subversion features in Darcs (specifically svn:externals) and get a bit confused sometimes when dealing with multiple repositories but the extra functionality is well worth the effort to learn.

Quebec: a question of definition

2006-11-23T12:48:00.000-05:00

Prime Minister Stephen Harper has said he will introduce a motion to recognize Quebec as a "nation within a united Canada". This was immediately pounced upon by Gilles Duceppe who demanded that the motion be reformulated in order to recognize a "nation that is currently within Canada".

The unfortunate thing is that we are spending our time debating the wrong thing. That a certain socio-ethnic group of people exists in Canada is not a question. That the people in that group have inherited a certain culture and certain beliefs is not at issue. The real problem is not that nobody "recognizes" Quebec. The real problem is the divide between those who believe that Quebec should have a certain level of sovereignty and those who disagree.

It all comes down to control; and a lot of separatists/sovereignists feel that Quebec should have full and complete control over everything that goes on inside its borders; including immigration/emigration and any other kind of import/export. This is not a question of culture, nor is it a question of inclusion/exclusion: this is a question of power. We can see it when people suggest sending delegates from Quebec to international assemblies; as if we really wanted to debate the north american missile shield with the "ambassador" of the state of New Mexico. Countries exist for a reason. They are a grouping of geographically and socio-economically like-minded people who get together for the same reasons as any other community forms. They provide a tool for dealing with other groups of people and, as such, are very useful.

In the case of Canada, the country also provides a way to re-distribute wealth to attenuate the issues encountered by one or another constituent population going through hard times. When the farmers of western Canada need help, the fishers of eastern Canada provide it (albeit indirectly). With regards to this particular issue, Quebec has historically felt hard done-by because of its own richness and other provinces relative poverty.

Meta-communities also provide the opportunity to set certain guidelines and enforce them at a higher level. We can see some of the effects of this by looking at organizations like the U.N. where a number of participants agree to live by the rules established by all. Difficulties arise however when participants in such a scheme attempt to assert their independence or sovereignty. A perfect example is when the U.S. decided unilaterally to invade Iraq against the recommendations of the U.N. security council. Unfortunately, the sanctions available to the U.N. is very limited. It comes down to "might makes right" and in Quebec's case we can expect more of the same: because of its socio-economic place in the country, Quebec stands to gain a significant amount of "bullying power" by declaring its independence (or at least having it recognized). The simple fact that a political party exists at the federal level to represent and defend the concerns of a single province is proof enough of the fact that Quebeckers are prepared to do what it takes in order to guarantee that their concerns become matters of national importance.

Do I believe that Quebec is distinct? Absolutely! I was born here and will always be a Quebecker in my heart. Do I think that this "distinctness" (as opposed to the distinctness of Ontarians or Nova Scotians) merits the declaration of the independence of the province from the "rule" of the rest of the dominion? No. Absolutely not. So let's stop debating our uniqueness and start debating what's really at stake here: whether or not Quebeckers (and to a certain extent the rest of Canada) are entitled to a greater amount of "self-determination" than other provinces; I mean, at some level, each city could declare itself sovereign and we could start all over again building counties from cities, provinces from counties and (gasp) countries from provinces. If the only goal here is to get more power in parliament I say "let it be." You already have a whole national-level political party to yourself. Isn't that enough bullying for one nation? If not, then let's get the Mohawks and the Haida and the Inuit in there too so that they can represent their people.

pkg_resources.VersionConflict

2006-11-21T11:45:00.000-05:00

This exception is raised when you update the version of a setuptools package without deleting the egg-info folder. The test command ends up expecting that the version is whatever is listed from the previous build whereas the value has changed in the meantime. Just run setup.py clean --all from your "root" directory to get rid of the error.

Some long-awaited updates

2006-11-12T18:37:00.000-05:00

I took some time this weekend to revamp a lot of the infrastructure around this site. I really would like to start documenting my progress in pipe-making as well as some of the other things I'm currently juggling.

I have set up my own little proprietary engine to pipe the contents of different RSS feeds of mine onto the front page of my site. This should allow me to collect the various things I'm up to into a single spot.

Packaging Python

2006-03-20T12:14:00.000-05:00

Ian Bicking writes about packaging Python libraries. He says simply that his advice is: "you shouldn't".

I really can't say that I agree on this one. While I do see the benefits of packaging libraries with the applications they go with, I have to say that after years of working in Java, where JARs are distributed and need to be included in an application-specific classpath, I really find the Python site-packages to be a breath of fresh air.

Users don't care about you

2006-03-10T20:05:00.000-05:00

Jeff Atwood has posted an entry titled "Users don't care about you". I will use this soapbox to pick at a semantic nit and explain why not instead of trying to explain how to get them to care.

The reason is pretty simple (as all good reasons are in hindsight): they are users. There are no "users" reading my blog. There never will be. And this is really the crux of the whole revival-of-the-cult-of-personality-but-on-the-web thing. Unless your audience feels they have a personal relationship with you, they won't read your blog. I read all of my friends' blogs (if you are my friend and I don't read your blog, please send me a link to it). Why? Because even if the crap they write is boring, pompous, puerile, silly, humiliating, myopic and self-centered it's familiar crap. There's something about the odour that reminds me of that person. It's probably why you're reading this right now.

The majority of people writing blogs are doing it for their personal acquaintances. The "long tail" of blogdom is the really real world. You all know where I work, what I do, what my hobbies are and so on. Because you care about me. If you didn't, you wouldn't be reading this. What causes so many broken hearts (and wasted electrons) is that a huge number of bloggers sort of expect that they will clickety-click over to blogspot.com, tappity-tap to fill in the fields in the registration form, select a template, dash off an article or two on what they had for dinner and skyrocket to Number One On Google. Everybody wants to be the next Scoble (or whatever). Don't worry, the podcasters all want to be Adam Curry (much to Dave Winer's dismay); the real-estate agents all want to be Donald Trump; the investors all want to be Warren Buffet; the programmers all want to be Bill Gates or Linus Torvalds (depending on whether they want to be rich and famous or just plain famous).

Of course everybody wants to be just like those heroes and anti-heroes that epitomize and characterize their environments. That's normal. It's OK to wanna be cool. And part of being cool is being you. I know, I know, it's trite (oops, I missed that adjective up there in the rant about crap) but it's so true: who is Bill Gates like? Who did Michael Jordan copy? Do they have heroes? Sure! But they didn't rush out to try to be carbon copies of whatever they thought was the best thing since sliced bread.

I just watched a video of Seth Godin presenting at Google. In it he talks about making a remarkable product as being the first step in a marketing campaign. Of course, we're talking about blogs (or are we?) but the same principle applies: if you want people to make remarks and tell their friends to go and read your blog, you need to be remarkable. It's sort of like getting a girl to go with you to the dance in high school. You need to be remarkable if you want people to notice you. Truism after truism.

I would like to segue briefly here into a comment about attention. It's not enough to get attention, you need to get the kind of attention you want. I was talking to a friend recently and he mentioned that (verbally at least) I am able to hold people's attention for a lot longer than most. I credit this to the incredibly competitive environment at the dinner table in the house where I grew up. I am the eldest of three sons and if you are the eldest of your siblings, you know what it is to need to fight for that spotlight. I employ a range of techniques to keep people's attention and none of them involve brutally assaulting my listeners verbally or physically. I do things like only pausing for breath at a place in my sentence where I know that everybody wants to hear the end; raising my voice just enough to be heard and then lowering it again as attention focuses on me, rewarding my listeners by not continuing to bludgeon them with a loud, crass delivery of my idea(s); shifting my physical position to be in more people's line of sight or so that there are less obstructions between me and my listeners - every person in the vicinity not listening to me is not only not a part of my audience but may even begin to sabotage me by luring other listeners into separate conversations. What I'm getting at here is that throwing big temper tantrums and gesticulating wildly will make people sidle quietly away to talk to me until you are left with only the most spineless of people standing around you desperately trying to finish their drinks so they can escape to the punch table before you start another long-winded story. Or maybe that's me...

OK, so what's the bottom line here? Well, if you really want people to read your crap (I know I said I wouldn't explain how just why but this is my blog so...) they need to really know you. I remember when I first got into podcasting and I listened a lot to this one guy who was rude, crude and whose podcast was really just him blabbing around a couple of pieces of "underground" hip-hop and techno music. I fast-forwarded through the music. What I really liked about this guy was:

How raw he was: there was nothing polished about his performance (lots of "ums" and "ahs" and plenty of swearing), his podcasts (plenty of snap, whistle and pop) or his distribution (a simple RSS feed without much around it)
He came from Montreal (yay locals!)
He talked about the shit going on in his life and made me feel his pain

I identified, ultimately, with how crappy this guy was - there was something so human about it. Adam Curry, in contrast, with his silky voice (run through 18 compressors) and his trailers (recorded by adoring fans and mixed on donated software from companies hoping for a plug), seemed more of a confection than anything else.

So, at the end of the day I just want to read a blog, written by you, to me, for us and our friends. If you can slip in some stuff that only we know (it's like a club) or that I don't know yet ('cause I still like to learn - even when I'm killing time reading my friends' blogs) then that's a real bonus. But really, please, just be you and I'll be me, and hopefully, because we're friends, we'll actually have something to say to each other.

Update: After posting this I realized how huge that first item in the list is. This is both because I am being clever (see Ma, no hands) and because it is really the most important. I like it raw.

Pivot points

2006-02-13T01:25:00.000-05:00

I just completed a webservice that calculates pivot points for major FX pairs. Try it out with USDJPY. The data backing it is downloaded and cached from Forexite Ltd. You can swap the symbol in the URL for different sets of pivots. You can also append .txt or .html at the end of the URL to specify the output format - the default is HTML.

Foreign Exchange

2006-02-12T19:24:00.000-05:00

Over the Christmas holidays I discovered forex trading. This is the practice of buying and selling large amouints of foreign and domestic currencies and speculating on changes in the price. I have spent much of the last several weeks reading books on the subject of technical analysis and accumulating a small hoard of resources to help me understand how to trade. I have opened a demo account with Alpari, a russian forex site (don't worry Mom, they don't have my money yet) and am attempting to learn how to play with the big boys.

I have started a blog to track my trading progress. You can watch my progress as I battle my way through a $1.5 trillion market.

An email address for every site

2006-01-06T18:02:00.000-05:00

I wrote this advice to a friend on the topic of alternate email addresses used to sign up at various sites:

By the way, a cool trick that I picked up somewhere is to register an account at http://www.cjb.net. You pick a subdomain of cjb.net (like joeblow.cjb.net) and then they redirect any email that goes to @joeblow.cjb.net to an email address of your choosing. So what I do is I have ilowe.cjb.net and every time I sign up somewhere, I give an address that encodes the name of the site I am registering on (for example yahoo-groups@ilowe.cjb.net). This has two benefits: first, I don't have to give out my real email address; second, if somebody starts spamming me using one of the addresses, I just add a filter to my mail client to kill that particular address and continue along my merry way. That way I don't have dozens of sites pointing to one "junk" email address that I need to log into every once in a while to clean out.

Subversion with Python 2.4

2005-12-30T14:52:00.000-05:00

Buried deep in the comments on one of Simon Brunner's posts I found a reference to Eugene Lazutkin's setup where he hacked the Python SVN bindings to work with Python 2.4. I am pleased to report that his hack worked beautifully for me and I am now the proud owner of a Trac installation running on Subversion v1.2.3 and Python 2.4.

The trick is to download the Python 2.3 bindings in ZIP format, then unzip them and drop them into your site-packages folder. Once this is done, fire up a hex editor and edit each of the DLLs that are installed in the libsvn package and replace all instances of the string python23.dll with the string python24.dll. That's all there is to it.

YAXL v0.0.16 released

2005-12-10T17:33:00.000-05:00

I just released version 0.0.16 of YAXL, tightening up QName manipulation and overall documentation. Check out the website.

Update: the source in the website looks horrible. I'm working on fixing it.
Update (20051219): the source in the website is fixed.

typecheck v0.2.0 released

2005-11-22T00:19:00.000-05:00

I just released version 0.2.0 of the typecheck module to the Python Package Index. The module's website has been updated to reflect the new version number and link to the newest build but has not yet been modified to document the changed interface after the merge with Collin's code. He tagged the 0.2.0 version yesterday so we still need to catch up on a couple of minor documentation points.

The new version of the module maintains 90% compatibility with my old API - the only change is that you can no longer elect to not typecheck a parameter; a feature that I hope we will be able to bring back in a subsequent release.

YAXL v0.0.15 released

2005-11-16T21:34:00.000-05:00

I just released a maintenance version of YAXL. This release contains a couple of passing test cases that had been waiting for release for a little while. I also fixed an embarrassing bug in the Python Egg distro that screwed up the builtin help.

YAXL's Subversion repository is now public. You can browse version 0.0.15's tag or just snoop around in the trunk to find the bleeding-edge stuff.

New code updates for typecheck

2005-11-16T20:35:00.000-05:00

I will be merging the code in my typechecking module with Collin Winter's codebase. The new subversion repository for the continued development of this module is at http://www.ilowe.net/software/typecheck/svn.

We should be releasing a new version shortly with a whole whack of other/new features that Collin has implemented such as return-type checking and logical operators on types (and and or). Look for version 0.2.0 in your local Cheeseshop RSS Feed.

Email tacks

2005-10-19T10:14:00.000-04:00

Rob Bray suggests a different tack to take for spam prevention/reduction.

Rob blogged about his resistance to the penny tax for emails that Tim Bray suggested in a recent blog post. In this installment, however, he changes his mind and instead comes up with a brilliant solution: only un-answered emails cost you anything. Most of us write emails to people and receive answers from those same people; spammers on the other hand send emails to thousands of people and only a few (idiots) reply. If we could make it really costly to send un-answered emails and cheap to send answered emails we might have the beginnings of a really nice, low-tech solution.

This suggestion of Rob's reminds me of a socio-economic system called Stone Society described by Peter Merel. The system involves creating an artificial resource that is then exchanged and manipulated by participants in order to allow decision-making to proceed.

In the world that Rob describes, you would exchange tokens freely with people with whom you have a back-and-forth. Spammers would simply send you tokens which you could accumulate. In other words, spam would be beneficial to you even if you didn't want to receive it.

If the tokens here were indeed pennies you would actually get paid to receive spam. You could still have email filters to make sure you don't have to read it. This plan is all about raising the barrier to entry for spammers.

Hacking the iPod nano

2005-10-18T03:13:00.000-04:00

I hacked together a little Python script that scrapes Atom feeds and transforms them to iPod "contacts" allowing me to read blog posts from my nano. It uses html2text right now but a lot of entries with embedded HTML come out looking pretty weird. I'll post the script once I clean up the text generation.

I managed to get a couple of the vCalendar features working and found out that even though Apple says you need iSync or iCal to use the nano to add TODO items, you can just add standard vTodo objects to a vCalendar stream and it works just fine.

Also, the rdate property is what controls repeating tasks/events - not any of the other ones that purportedly serve the same purpose.

Now it's time to go to bed...

String/XML representations in YAXL

2005-10-16T18:08:00.000-04:00

Damian Cugley asks

Could you eliminate the need for the text member variable by using str(elt) instead? Or would that interfere with using str to return the XML representation for the whole element?

In YAXL v0.0.14 I've split the representation of an element between str and repr so that the former would be a shortcut to return the text property and the latter would return the XML representation of the element.

I'm pleased with the separation since now if you evaluate an element at the interactive prompt, it returns the XML for the element but if you print or interpolate the element its string value will be used. There is also a nice parallel with the way nodes are handled in XPath.

YAXL elements retain the text property in order to continue to support the oodles of screaming fans who have already shipped billion-dollar projects based on my little library.

Thanks for the suggestion Damian!

Fall cleaning

2005-10-16T10:46:00.000-04:00

Justin and I cleaned out some of the more cobwebby parts of mystic yesterday. We wanted to install Python 2.4 with subversion, trac and apache 2. It took a fair bit of wrangling (i.e. building from source) to get everything working together after which Justin realised that we hadn't upgraded to Sarge (duh!). He rebuilt everything using the latest packages and it looks OK so far.

The transition from Apache 1.3 to 2.x was pretty smooth with the SSL stuff at the high end weighing in at about 20-30 minutes to figure out. Although the new sites-available/sites-enabled organization is nicely architected, it does make for some pretty painful contortions when performing wholesale changes on a group of virtual hosts. For example, we started off testing the new installation of Apache 2.x on port 81 (which required that we split out the vhost declarations into files of their own) and then proceeded to switch it to port 80 (which ended up requiring a quick cat/sed/re-direct combination). Without a good knowledge of shell scripts or some scripting language this would have been really painful.

The other rough item on our plate was permissions when using the Berkeley DB version of a subversion repository. My advice: don't go there. Just stick to FSFS and you should be OK. This being said, I had originally created the repository in FSFS but had subsequently dumped it and loaded it onto mystic. It seems that I did something wrong because it loaded in Berkeley DB mode. All that is fixed now and taken care of.

We also tossed in Apache subversion support for good measure so now we can setup anonymous browsing and reading of our SVN repositories.

iPod nano in da house

2005-10-15T00:38:00.000-04:00

Vem and I finally both got our iPod nanos! What a cool gadget. These things are too slick to be real!

My one small gripe is that we ordered a bunch of stuff from Apple and they shipped it all separately with the predictable result that the iPod armband for Vem's gym sessions arrived first (wow, look - an armband) followed by her iPod, followed several days and a trip to the local Fedex detention centre for un-claimed packages later by mine own.

Rumors about the nano being scratch-prone seem to have some truth to them as I can already (after a fay and a half) detect small scoring on the exterior plastic. Of course, we ordered protective "skins" but since Apple hasn't shipped them yet (and nobody in their right mind would leave an un-opened nano sitting on the shelf for three weeks) we may find that our nanos are not as pristine as they could be. Ah well...

Still, the photo feature is pretty sweet (useless but sweet) although it would be nice to be able to set a photo that displays automatically when you power on (sort of like Vem's digital camera), the rating feature is very cool (I was just ranting to Vem the other day about needing one of these). iTunes is not all it could be but I haven't started playing around with the iPod libraries out there to see what I can hack up.

Anyways, overall we love our nanos and would buy them again: 9.8/10

YAXL v0.0.12 released

2005-10-10T18:51:00.000-04:00

Over the weekend I've managed to work in a number of bugfixes, add some more XPath support, add sequence-style access for children and implement a whole slew of namespace features. The latest source version includes 78 unit-tests that cover all this fancy functionality.

I'm still debating the question of whether or not to allow access to children via property names (like xmltramp). The problem is that you end up having to mangle either child names or "real" property names. Also, you need to have a mangled way for dealing with node-sets. YAXL curretly supports XPath even in XML fragments so theoretically you should use that, but it's hard not to like root.title as opposed to root('title'). More complex queries, however, reveal the power of XPath: compare [x for x in root.head.children if x.localname == 'style'] to root('head//style') to retrieve all style elements of an HTML document. Or even root('//p') to retrieve all the paragraphs.

Ideally, YAXL would support both methods but I still need to come to terms with all the mangling required.

Expat doesn't support namespace prefixes

2005-10-07T09:54:00.000-04:00

This via "Crest" who sent in some nice stack traces generated by YAXL. I wrote in a fix yesterday to handle a missing QName but I didn't realise that expat would throw a SAXException. As a result, nobody using expat can use the parse function until I add a try/except for that exception.

SAX Parser Funny Stuff

2005-10-06T22:51:00.000-04:00

It turns out that the startElementNS method on a SAX ContentHandler is not necessarily passed the QName of the element. The name is passed as a URI/element name tuple, but the QName is not mandatory unless the feature_namespace_prefixes feature is enabled. The following code shows how to do this:


from xml.sax.handler import feature_namespaces, feature_namespace_prefixes
from xml.sax import make_parser

contentHandler = MyContentHandler()

parser = make_parser()

# Perform namespace processing
parser.setFeature(feature_namespaces, True)

# Report original prefixed names (i.e. qname is passed to startElementNS)
parser.setFeature(feature_namespace_prefixes, True)

parser.setContentHandler(contentHandler)

parser.parse(inputSource)

I was made aware of this issue when YAXL failed to parse XML source properly on somebody else's machine; the elements were being named "None". That tip led to this fix. Merci Marc!

YAXL Reviewed

2005-10-06T02:53:00.000-04:00

Jeremy Jones blogs about YAXL (already!). I agree with Jeremy that ElementTree is pretty much the best XML tool there is out there for Python. I also agree that the children attribute should not contain descendants. It never really did but since YAXL Elements are represented as XML fragments, it may have seemed that way.

Here's what an interactive session looks like:


>>> import yaxl
>>> x = yaxl.Element('x')
>>> x.append('y').append('z').append('w')
<w />
>>> x.children
[<y><z><w /></z></y>]

It may seem from the output that the z and w elements are returned as part of children but they are just part of the XML fragment that y outputs.

In the latest release (0.0.6) you can also "call" the element with an XPath of child::* and get back a list of immediate children.

YAXL gets XPath

2005-10-05T10:56:00.000-04:00

I added some basic XPath support to YAXL this morning. It now supports both abbreviated and unabbreviated XPath queries on Element objects. Currently only attribute-value and node/node-set selections are supported. Also, only the following axes are allowed in queries: self, parent, ancestor, ancestor-or-self, child, descendant, descendant-or-self

Especially cool (IMO) is the fact that YAXL supports two methods of performing XPath queries: you can call select on an instance of Element or you can call an instance of Element and supply the XPath as the only parameter.

Something new, something old

2005-10-04T09:20:00.000-04:00

I just launched a new site at http://www.ilowe.net. I will be retiring the schmeez.org site (the permanent redirects are already in place for the 3 people that visit regularly) for many reasons (not least of which is having to spell it every time I tell somebody where my "homepage" is).

I also released the first version of YAXL, a Python module for reading, writing and manipulating XML. It will form the basis for an Atom library I am currently writing.

New version of typecheck module

2005-09-29T03:11:00.000-04:00

I've released version 0.1.4 of my typechecking module for python. This version contains a fix that allows you to use doctest tests within typechecked functions.

This doesn't work out of the box since the doctest module looks for tests by recursing through a set of objects and looking for docstrings. If it finds a function, it checks that the functions func_globals object is the same object as the function's module's __dict__ property. This is not true of any decorated function where the decorator is defined outside the module the function is defined in.

Anyway, the new typecheck.doctest module allows you to bypass the check that doctest usually uses and replaces it with a much more liberal one that just verifies that a function's __module__ property is the same as the name of the module in which the function is defined.

All locked up and no place to go

2005-09-13T10:20:00.000-04:00

Dave Pollard asks

What does it take to make someone so dissatisfied, so unable to bear the life they lead, that they can walk away and start a new, better life, a different way, elsewhere? And what does it take before we realize that the prisons in which each of us live, societally and metaphorically, are prisons without locks, just waiting for us to let ourselves out?

Personally what keeps me here are the links I have with other people. I have frequently said that if I had none of these I would quickly enroll myself in a monastery somewhere for 10 years. In the same way that Scott Peck (A Road Less Travelled) says that everybody could benefit from psychotherapy I believe that everybody could benefit from some solitude and meditation.

I pick 10 years since I figure the schedule goes something like this:

Spend the first 3-4 years getting used to my new surroundings, learning to be "disconnected" from all the things I am accustomed to.

The next 2-3 years are spent learning how to be alone with myself and how to think when I'm alone. I currently feel a need to bounce ideas off others and this would not be possible in an ideal circumstance.

The final years are spent in quiet contemplation of myself, the world and the divine. Hopefully I can then choose whether or not to "go back" with some sort of idea both of what I am leaving and of where I am going.

I used to have nightmares where somebody was pressing me for an answer I didn't have and all I could muster was "Wait! Wait!" over and over. The frustration of not being able to stand back, take a deep breath, and see things the way they are is a constant enemy of tranquility and clear vision.

I suppose it's possible to become dissatisfied with bits of your life and work to change them, but the same obstacles rear their ugly heads every time: fear, inertia and peer pressure in its various forms.

It's all about the user experience

2005-09-10T12:48:00.000-04:00

Kudos to ArgoUML for having a Java WebStart version of their software. Launching this over the web is so easy it makes you wonder why you bother actually "installing" anything.

What's good for the goose

2005-08-29T10:42:00.000-04:00

I discovered last night that Microsoft's web servers display an error for the URL http://www.microsoft.com/.net. Now, I may be a bit demanding but don't you think that a company like that should support tacking ".net", "office", "visualstudio", etc. to their base URL?

As a side note, using any additional path element that starts with a "." causes the same error. Requesting http://www.microsoft.com/visualstudio and others provide a reasonable error page that at least allows you to search on the site.

Why do we treat email differently than a phone call?

2005-08-10T12:52:00.000-04:00

Jason at 37signals.com asks "Why do we treat email differently than a phone call?"

I think the answer is a lot simpler than the dozens of commenters imply: we treat email differently because text can be manipulated in ways that audio content cannot.

I could record all the phone calls I take or make but what would be the point? I couldn't go back and search through them, it's impossible to "scan through" a conversation to see the interesting/important bits, etc.

Although it's true that you could do all of this with transcripts of phone calls the fact remains that transcribing those calls is a Herculean task; either you need a dedicated secretary or you need to be very understanding of the mistakes made by your voice recognition software. All of this doesn't even get into the difficulty of either the secretary or the software dealing with accents, voice modulation and so on.

A lot of the meaning in a conversation (even on the phone) comes from things other than the actual words used. With email, the entirety of the conversation is captured. You see what you saw when the email first arrived.

Bottom line, we treat phone calls and email differently because the two media (voice and text) are divergent in so many aspects.

Ignorance and judgement

2005-08-01T09:58:00.000-04:00

David Toub blogs about snobbery, elitism and "atty-tood". Sadly, I think most elitism and snobbery comes from simple ignorance. Most people lose at least some part of their superiority when they are (properly) exposed to the thing that they deride.

In the world of programmers nowhere is this more prevalent than in the language/editor/platform wars. A lot of these wars are started in various fora by somebody who has a certain degree of knowledge about the language/editor/platform whose virtues they extoll (enough to know which features to laud at least) but little knowledge of the competing technologies.

Although editors and platforms may indeed provide different features, languages, like music, can say anything to those who want to listen. Sure they have different flavours; that's why I prefer Python to Ruby (down in the back row). But at the end of the day they are all capable of moving the machine to do the same things (notwithstanding the fact that some languages cannot do certain things for architectural reasons).

Those who refuse to acknowledge certain basic qualities of these forms of expression (music, languages, fashion, painting, sculpture) do so out of ignorance and myopia. And prejudice is never far behind ignorance and judgement.

Weekend of the WOD

2005-07-23T18:51:00.000-04:00

It's been a busy weekend so far. Today I cleaned up the logging code in the WOD and added a setuptools script for building eggs and so forth. I moved a huge amount of code around and consolidated a number of dangling modules at the wodfs.* level into the wodfs package.

I also threw together a wodfs.SimpleSystem class that can be easily used whenever you want to hack with the WOD programmatically (i.e. not through the filesystem). I also shifted the code from wodfs.server so you can call wodfs.start(port=9000) to get a self-documenting XMLRPC server backed by the WOD.

I then shuffled the website around and moved everything to wodfs.org. The old address now issues a permanent redirect to the new one.

Type-checking module for Python

2005-07-23T02:09:00.000-04:00

I just published version 0.1.0 of typecheck, a type-checking module for Python. It is registered at the Python Package Index and everything.

I wrote the setup scripts in about fifteen minutes and tweaked them for an hour or so using setuptools which I am now convinced kicks some pretty major ass. It was really easy to package everything up and upload it to PyPI. setuptools can even do special tricks like allowing you to deploy a "development" version of your package that you can edit from your checkout directory but that still gets included in sys.path - really neat stuff.

Pattern recognition

2005-07-21T13:09:00.000-04:00

Jeff Atwood describes the practice of Just Try Again where coders sometimes re-execute code that fails to see if it "does it again".

I think that one of the benefits of running code multiple times without modification is that it allows us to see patterns. It's a bit like trying to pick out the lyrics in a song: listening to the song once is not enough. In fact, even several times may not be enough to allow you to isolate the different ways the singer has of pronouncing certain words. Depending on which words are grouped with which others the pronounciation may change. This is all very similar to the way that different components in a software system act together to increase the overall complexity of the task of isolating an error.

So when I re-run broken code without changes what I'm really doing is looking for patterns in different parts of the code on each run. Doing so allows me to break down the run into a set of behaviours which I can then analyse to figure out what is out of place.

Universalism and a machine existance

2005-07-21T11:28:00.000-04:00

In his article WWW and UU and I Tim Berners-Lee says:

The whole spread of the Web happened not because of a decision and a mandate from any authority, but because a whole bunch of people across the 'Net picked it up and brought up Web clients and servers, it actually happened.

This really captures (for me) the way we should be living our lives: not as subjects to an external authority but by working together towards a "good" that can benefit us. Our own internal authority should allow us to interoperate and co-exist much better than machines. And yet it seems they have the upper hand.

Myofilms Launch

2005-07-19T20:41:00.000-04:00

I just launched the Myofilms Blog with Olivier. He promises to write and keep in touch.

Python gets CPAN

2005-07-14T15:45:00.000-04:00

Phillip J. Eby has put together a set of extensions to distutils called setuptools. One of the included scripts, easy_install.py is a shot at duplicating CPAN/apt-get and other package management tools.

setuptools also works with Python Eggs, a package format that extends the distutils format.

Although I haven't tried to package anything as an Egg yet I have tried the setuptools package (which incidentally comes with a bootstrap script that downloads and installs the package automatically). Everything ran straight out of the box and I was able to install and upgrade several packages already installed on my system very easily. Great work! This is something Python has sorely needed for a while.

Sometimes I feel so alone

2005-06-14T15:16:00.000-04:00

All I really want is to use Python to connect to an MS SQL 2000 database without using a DSN or ODBC. Is it really that complicated? Is there really nobody else that has ever wanted this before (and moved beyond complaining on mailing lists)?

Different strokes

2005-06-10T18:27:00.000-04:00

As I was randomly flipping through pages on Ward's Wiki filled with code examples comparing Ruby and Python I felt the urge to fire off this message into cyberspace: don't ever think it impresses me that you can write code in your language that I can write in mine. It should be obvious to anybody with an internet connection and a propensity for coding that any two turing-complete languages are mathematically identical. I obviously like my language and you like yours. It is just plain asinine to think that by showing me you can write code that does the same thing as my code you can convince me that your language is better. We all use the languages we use because a) we were born to it (i.e. it's all we know) or b) we like it best. Either way, it's not impressive to see the same algorithm implemented in a different language alongside a claim of superiority. 'Nuff said.

What to do, what to do

2005-06-07T16:25:00.000-04:00

What do you do when you ask somebody "can you guys handle XMLRPC" and they respond "yeah, sure, XMLRPC, LMNOP, whatever you like"?

AJAX encapsulation and some gooey musings

2005-05-30T11:08:00.000-04:00

Jon Udell blogs about AJAX encapsulation with TIBCO General Interface. This is one seriously cool product and I'm now officially itching to get my hands on it.

On the other hand it makes me wonder why we persist in trying to rebuild the functionality of the OS (windowing systems and so on) inside the browser. More and more it seems to me that we need to have a complete re-write of the OS windowing system to take advantage of all the things we have learned about GUIs in the past little while.

Imagine if the "windowing" software for the OS was actually just a giant browser. You'd get automatic "Active Desktop". You'd be able to open frames and dialogs and file choosers and all the other crunchy goodness that we have painstakingly baked into the browser. You'd also stop worrying about whether your application's super-duper feature was going to work on MacOS, Windows and Linux since all three could run this "super-browser" as the base object in the windowing system.

There's probably a really good reason (other than the phenomenal effort required) to not do this... but I can't think of it.

Interviewed by Richard Jones

2005-05-28T11:58:00.000-04:00

I asked Richard Jones to interview me.

I'm sure that my answers (found below) are a bit longer than most people's attention spans. To alleviate my guilt, please follow these instructions and further the meme:

Leave me a comment saying, "Interview me."

I will respond by asking you five questions. I get to pick the questions.

You will update your weblog with the answers to the questions.

You will include this explanation and an offer to interview someone else in the same post.

When others comment asking to be interviewed, you will ask them five questions.

1. Why do you write a weblog?

I write to try to capture my thoughts. I suppose that I don't really care about other people reading them but on some level I like the fact that they do. There is an ego trip associated with putting your thoughts out there and having other people read them. At the same time, I get to use readers as a sounding board: anybody who disagrees with what I write is welcome to comment and hopefully give me some insight that I hadn't previously seen.

I decided a while ago that I didn't want to have the kind of blog that goes on and on about what flavour of ice cream I ate this morning or which of my friends I will be seeing tonight. I think that in the future I will come back and read my thoughts from this year and last year and try to see what changes took place in my head and when (and maybe even why). Being able to see what I was thinking when certain events took place is kind of cool. I don't want to look back and see a list of what I was doing - I want to see what I was thinking about.

Finally, I blog because I'm fundamentally a yapper. I have an answer to every question (even though it might not be a great answer) and I sometimes feel the need to get that answer off my chest even if nobody is listening. That's why I sometimes blog about morality or society. My friends laugh at me (kindly) because I always have something to say about everything so I figure that some of that should end up here.

2. What's the coolest online community enabling system right now?

I think that wikis hold the #1 spot for me. They allow users to collaborate, chat and create together which is something that most other systems lack for one reason or another. A lot of features from other systems could be folded into wikis but even the basic (simplest) wiki system is far more powerful than most other systems.

Blogs are cool but they don't really create tight communities. They create clouds of people that read each other's blogs and leave comments for each other but they remain fundamentally un-collaborative publishing mechanisms - a way for me to tell you what I think/feel/do.

IRC has more immediate interaction but loses completely on the asynchronous level. I know, I know, there are bots that will tell you when the last time foxy6969 was in the channel but it's not the same. Wikis are always now.

Usenet has asynchronicity but is more like mailing lists. In fact, let me just lump mailing lists in here as well (even though there are access restrictions to most mailing lists that don't exist for most newsgroups) and say that both allow you to post articles (like a blog) on a given topic. I would say that Usenet fosters more community spirit than blogs but still loses to wikis.

Online games (like MMORPGs and so on) also foster great community spirit but are very limited in their focus. Even though most wikis have a theme, almost all have a very heterogenous collection of pages.

Tagging services like Flickr and del.icio.us don't create communities. They expose existing interest groups. You and I may tag the same resource the same way but have nothing else to say to each other. Also, tagging is not conversation (no matter what the pundits say) and is an incredibly low-bandwidth medium of communication. The lack of consistent commenting means that tagging services are more like personality quizzes than they are like a community blackboard.

So wikis is my final answer.

3. What on Earth is WOD?

The WODFS is a Write-Once Distributed FileSystem. I wrote it in the winter of last year with my friend Justin and we put a lot of work into our "baby". The system is based on the design of archival storage systems and distributed filesystems and its main goal is to allow users to simulate an infinite, indestructible hard-drive.

The WOD allows you to keep saving data forever to a network of computers that store only the unique parts of your data. This means that each "chunk" of data is only ever stored once (conceptually). Each node in the network caches the most frequently used blocks so that you can achieve a disk access speed that is reasonable.

What's really cool is that since the WOD is write-once, you never actually "lose" any data. Even deleted data can be easily retrieved.

The system includes a set of extensions to the filesystem that allow you to create special folders and access files that contain data from the past. For example, if you have a folder called myprojects where you put all your code, you can open (either by creating it in Windows or by cd'ing to it in UNIX) a folder called myprojects@2004-01-01 and you will see that folder's contents exactly as they were on the first of January, 2004. This works with any folder (even the root) in your filesystem. Individual files track every single saved modification and allow you to browse through the history of the file using dates (as explained above) or by appending @ and a negative number representing the number of steps backwards to take.

I'm working hard right now to finish the code to hook the system into the Windows explorer but right now things run only on UNIX. Performance is not as good as we would like but the whole thing is written in un-optimized Python so that is to be expected.

4. What's your favourite Python hack?

I think that right now, my favourite hack has to be the soft reference module that we developed for the WOD. We hooked into the GC routines of the Python runtime to manage a set of objects that gets collected only when the memory they use is needed.

Close runners-up include hacks that involve meta-classes or decorators.

I'm not a big fan of cramming as much code as possible into one line. I like to see elegance and consistency. By consistency I mean that even hacks should be done in the spirit of the language used.

I've spent the last years coding in Java and so I really like hacks that show flexible (sometimes even functional) capabilities of languages. I like to see languages pushed to do things that they were not necessarily meant to do in clean and cogent ways.

I've always been a big fan of LISP and Scheme so I like to see hacks that modify language structure or build "small languages" or that modify a set of entities (objects or functions or whatever) transparently and orthogonally. That's why I like playing with decorators and meta-classes.

5. Blogs vs. Usenet, fight to the death. Who wins?

Blogs win. Honestly I think that blogs and usenet are pretty similar and if bloggers had some more tools (other than Trackback and comments) to allow a threaded style of linking between their posts (like metadata that allowed a tool to generate lists of related posts) then blogs would be a lot like Usenet but without the crappy flamefests and trolling and religious wars.

Blogs mean I get to choose whose writings I read. I like that choice. There's no spam and in general most blogs are consistent enough that I know after a little while whether or not to keep reading the blog.

Usenet can go any way. There are often spam messages posted to newsgroups. There are trolls who spend their time wasting other people's lives. Also, newsgroup organization is more centralized than blog management (says the guy who blogs on blogspot.com).

All in all, blogs are more flexible and allow more personal choice and freedom. They get my leftist vote.

Tag maintenance

2005-05-26T15:08:00.000-04:00

Clay Shirky blogs about the Dynamic growth of Tag Clouds. He put together a script that showed how the cloud of tags associated to a URL on del.icio.us grows over time.

An interesting aspect (mentioned later in his post) is that some of the top tags for a resource only percolate to the top later in the tagging process. The example he uses is the tag "ajax" being applied to the "original Adaptive Path article". The "ajax" tag wasn't used until after more than 1/3 of taggers had already tagged the resource.

This begs the questions: how can users who are unaware of a certain term ever going to use that term as a tag; and how will that term ever be added to the list of tags used by those who have already tagged the resource?

To be clear: I tagged the original article. I did not however use the tag "ajax" since at the time the term meant nothing to me. It had not yet caught on as a global term for the practice of using javascript to communicate with a server behind the scenes. Today, I might very well attempt to find that article using the "ajax" tag. Of course, this wouldn't work since I never used that tag. It would be nice if I could search my tagged resources using others' tags when I don't have the tag I'm searching for. Obviously my own tags should have priority but if I search my resources for the "ajax" tag, the resources that I have tagged and that others have tagged as "ajax" should show up in my search.

This points to a general maintenance problem with tagging (and I would venture to guess with classification schemes overall): how is the metadata maintained cheaply. Currently, tagging a resource into del.icio.us takes me less than 30 seconds. It would be a real pain if I had to review my resources all the time and add new tags.

Maybe another solution (other than the one listed above) would be to produce a feed of resources that I have tagged and an interface that would guess which tags from others would be most useful to me.

It feels like we are moving beyond tagging to searching. So is the issue the technology behind tagging/classification or behind searching?

Sparklines

2005-05-19T18:19:00.000-04:00

I discovered a fantastic page containing a draft chapter from Edward Tufte's new book via Pensieri di un lunatico minori.

Digging deeper as is my wont I found a whole whack of cool articles about sparklines and their implementations.

http://bitworking.org/news/Sparklines_in_data_URIs_in_Python
http://www.ietf.org/rfc/rfc2397
http://www.bissantz.de/sparklines/
http://agiletesting.blogspot.com/2005/04/sparkplot-creating-sparklines-with.html
http://dealmeida.net/en/Projects/PyTextile/sparklines.html
http://www.waler.com/flint10.exe

[Update: fixed link to Pensieri di un lunatico minori]

WODFS storage architecture

2005-05-19T18:17:00.000-04:00

An off-the-cuff description of the WOD's storage system

OK, I admit it. We shamelessly ripped off the idea for how to break up and store data from the Venti filesystem papers.

So the basic idea is that each file is broken up into a number of blocks. Each block is 1024 bytes in size (for now, more later on variable-sized blocks) and its fingerprint is the SHA-1 hash of the data in the block.

A given block can be either a data block or a pointer block. Pointer blocks have pointers to other pointer blocks or to data blocks. Each file is conceptually a tree of pointer blocks that end up pointing to leaf data blocks. Because has as its fingerprint the hash of the data in the block, each block can be interpreted as either a data block or a pointer block. This is not an issue to begin with but as the system fills with data, we project that this will result in considerable savings in terms of space since the data and the data structure are expressed in the same way.

In addition, each file is uniquely identified by a fingerprint which is the hash of its contents: the leaf data blocks are referenced by the pointer blocks all the way up to a root block for the file; that block's fingerprint is the file's fingerprint. Similarly, the entire filesystem is represented by a single root fingerprint. This root fingerprint is stored between sessions and is, in fact, the only item necessary to rebuild the entire system from scratch.

For the moment the type of a block is decided by a flag in the pointer block that points to it. This is a bit fragile since there is a limited amount of space for flags and thus the scheme could become unwieldy at some point in the future.

Since blocks are context-neutral they can be shared by different instances of the WOD running on the same machine.

The blocks are stored on the disk contiguously without any specific order (this sucks because retrieval is via random-access only). To speed up block retrieval, an index maps fingerprints to offsets in the data "file". This index can be rebuilt from the data "file" at any time and so it is not crucial to the functioning of the system. Theoretically the index could be stored within the system itself and loaded after the system is initialized. In this way the bootstrap routines would be a bit slower but once the index was loaded the whole system would speed up.

Right now the big hit for performance is in what we call the chainer. This part of the system takes a contiguous stream of bytes and breaks it up into a tree of blocks. It would be really nice to figure out a way to derive the "chained fingerprint" (i.e. the root fingerprint for the processed file) from the initial data in order to be able to skip the chaining phase when it is not required. Already the system discards write hits for blocks that are pre-extant in the data "file"; it would be nice to bring that up a level so that entire files could be discarded if their contents were known (this could be trivially done by adding the original hash to the root block since the original hash of all files with the same root block would be identical).

Moral objections to medical procedures

2005-05-19T14:30:00.000-04:00

Medical decisions should not be coloured by a doctor's personal beliefs

David Toub blogs about "compulsory abortion training" for OB/GYNs. He mentions that at Yale it is not compulsory for students to learn abortion procedures although they did have to learn how to handle post-abortion patients.

It sickens me to think that there are doctors in the world that may be (by hook or by crook) imposing their own moral views on their patients.

The idea that I would have to change doctors because mine refuses to perform a certain procedure on moral grounds seems to me like a direct violation of the Hippocratic Oath which in theory involves providing the best care possible for a sick patient. It must be said that the original oath contained a clause that specifically forbade abortion; but that clause is absent from most modern versions of the oath.

Should a Jewish doctor be allowed to refuse to perform a porcine heart implant on the grounds that they do not believe that a pig's heart should be in a human body? Should any doctor refuse to perform a blood transfusion because they believe that their blood is their own and that to accept another's blood is to import impurities?

Science and scientists have always been on the razor edge of this knife: can I refuse to employ my knowledge because of a known outcome that I personally find reprehensible?

In many cases the answer is "yes". Engineers can decide to not build bridges that they believe will cause environmental damage, atomic physicists can refuse to build a better bomb, etc. But doctors, even more so than other scientists, are in the tricky business of directly saving lives.

To say that a doctor should be able to pick and choose which conditions he treats would be like saying that a policeman could pick and choose whom he protects; or like saying that a fireman could decide to not put out a fire based on what business took place in the building; or like a teacher being allowed to pick and choose which of his students he teaches.

When your profession places you in a position of responsability for the well-being of somebody else I think that your decisions for the treatment of that person (or that person's property in the case of the fireman) should be made based on the person's system of beliefs and not on your own. It's one thing for a doctor to advise against a certain operation (although the morality of the thing becomes very grey indeed) it's another entirely to relegate the patient to a lesser (or even just another) doctor just to satisfy one's own sense of moral values.

When somebody depends on you, putting them at risk to satisfy your own subjective reality can hardly be said to be the moral high ground.

Serendipity in RSS-land

2005-05-17T15:16:00.000-04:00

In Jon Aquino's post Tip: Random-number generator for Firefox, he lists some of the things that he uses the random number for. He makes a choice between emailing a random contact or reading a page out of any one of a number of different sources. It would be kind of cool to have an RSS feed with a random page from a number of sources that would provide a little serendipity.

On a slightly related note, sometimes it's nice to let a couple of days go by without reading a given feed. It let's the juicy bits pile up so you can read them all at once.

Bitten by indentation

2005-05-11T14:54:00.000-04:00

Python should raise more appropriate exception when the indentation in a class definition is not consistent.

Python allows you to use either spaces or tabs to indent your code; but you can't use both. You can't mix and match. Now this is not a big deal (although it does sort of violate the "there is one way to do things" rule that Python seems to adhere to fairly consistently) but it does mean that if you cut and paste some code from another file you could be lining yourself up for problems.

The thing is that Python, for some reason, doesn't complain when your indentation is not consistent within the definition of a class; instead, it gives an error that has nothing to do with the actual problem. For example, the following code:


1 class t:
2 [tab]def f():
3 [tab][tab]x = 5
4 [space][space]print x
5
6 [tab]def x():
7 [tab][tab]pass
8
9 t()

Will raise a NameError and say that 'x' is not defined on line 4. I don't understand why this doesn't raise an IndentationException the same way it would outside the class definition.

RSS Reading in Firefox

2005-05-10T11:34:00.000-04:00

I just installed the Firefox Bookmark Synchronizer. I have had Sage installed for a while but now I can actually have my feeds synchronized between home and the office. Next on the list is getting my browser running off my USB stick so I can literally take my browser anywhere I go.

The Whisper Campaign

2005-05-09T11:23:00.000-04:00

Dave writes that he has had a change in philosophy but I'm not sure this is really a change but maybe more of a refinement of previous ideas: there are certain types of praise that are more suited to being given in private.

I think the key factor may be intimacy; if I have intimate praise for somebody, sharing it publically may work against me more often than not. Public praise should focus on shared goals whereas private praise should focus on personal and/or intimate aspects of the relationship. When you tell somebody that they are "amazing" you are referring to some aspect of your relationship with that person that causes you to be amazed. This is not something that needs to be bandied about or else it loses it's subtlety.

In social gatherings my girlfriend and I often spend a lot of time apart, flitting from one group of people to the next. We always manage to catch the other's eye however and deliver a small, subtle gesture of caring even across a crowded room.

I think private praise gives a feeling of belonging and closeness that is not acheived via public praise.

Exception-based Switch-Case and the flavour of Python

2005-04-28T10:35:00.000-04:00

There's been a lot of discussion around switch-case flow-control in Python lately. With this recipe, Zoran Isailovski weighs in with what I find is a nice, clean, pythonic way of handling switch statements. He references another recipe by Brian Beck that handles the switch with a for loop (very cool).

I notice a comment on Zoran's recipe that references the C2 wiki's page on not using exceptions for flow control. Now I spend most of my time switching between Java and Python and it's true that I would never dream of using exceptions to control the flow in my Java application, but it's different in Python. In standard Python, iterators raise an exception to stop iteration and this is a pattern that is fairly well-recognized in the community.

All of which just points to the differences in flavour between Python and Java.

Lexical Analysis, Python-style

2005-04-28T10:27:00.000-04:00

I've always wanted to be able to do lexical analysis Python-style and I've always had the feeling that it should be done with regexps instead of character by character. Now Jason Diamond and Frederik Lundh have presented recipes for doing just that. I'll have to try out these techniques and see if a re-write of the parser for Founder makes things cleaner.

Embedding data in HTML documents with the "data" URL scheme

2005-04-27T16:28:00.000-04:00

The "data" URL scheme allows you to embed data directly in an HTML document. By encoding the data in Base64 you can even embed binary data straight in the page. The following example illustrates this by embedding the source of an image directly in the image tag itself.


<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAFAAAAAMCAIAA

AB6P612AAAAmklEQVR42uWWSw6AIAxE7VXg/keqV8FPE12AdUYgamDREFomfVBIJaU0jTSEA

xbZbI8zaqXs6OwuUdVbkRDjamfVY4IEUzHILmQ4OubygHPOzwIjGV4CO5y4bkNgSpAApjiR/

CgGZwVxEcD2ulnO/06gT6uylp5dbE1W5SoIoQFwZaL9gAvkgwCfygb8cqeV9wn9ehuTH621X

AArWx8EprTSmAAAAABJRU5ErkJggg==">

The code above produces the following graphic

How well do you know Python -- Name mangling

2005-04-18T14:48:00.000-04:00

In his article Spyced: How well do you know Python, part 1, Jonathan Ellis points out the danger of using exec. This principle shows up in inheritance too if you're not careful (or you come from Java):


>>> class A:

...     def __init__(self):

...             self.__x = 5

...     def getx(self):

...             return self.__x

...

>>> class B(A):

...     def getx(self):

...             return self.__x + 5

...

>>> b = B()

>>> b.getx()

Traceback (most recent call last):

  File "", line 1, in ?

  File "", line 3, in getx

AttributeError: B instance has no attribute '_B__x'

Duck Typing dilemmas - A strawman for "pure" Python interfaces

2005-04-15T15:01:00.000-04:00

The Duck Typing debate is less about type-checking and more about expectations

Cedric Beust blogs about The Perils of Duck Typing and he's not alone. Many people have been talking about dynamic and static typing for quite a while now and it's very close to being an Emacs vs. vi argument.

I don't think the problem here is really the absence of static typing but more the absence of "good" design or reasonable expectation management. Let me give an example (so you can make fun of me if I'm wrong).

In Python, a number of methods and functions take "file-like objects" as parameters. Now some of these methods actually want all the methods that files implement and others are content with just having read and readline. This interface is obviously a little hard to code to since in order to implement it you need to read the specific requirements of the method/function you're calling.

Now the only thing wrong with this whole setup is that you really have no idea what methods to implement and as Cedric points out, you may omit the implementation of a method that is not called directly from the function you're calling. So here's my strawman proposal to those naysayers who demand more static typing: just implement an empty base class that throws NotImplementedException from your methods. Note that in Python this class wouldn't even need to be bundled with the base distribution since Duck Typing works whether you like it or not.

In the function below it doesn't matter what type the fileobject parameter is. It will work fine with a file from the open or file functions as well as with any other iterable object.


def line_count(fileobject):
    count = 0

    for i in fileobject:
        count += 1

    return count

Now let's assume that we want to make it explicit that you need to pass an iterable object. Most pythonistas will tell you to re-write the function like this.


def line_count(iterable_object):
    count = 0

    for i in iterable_object:
        count += 1

    return count

Now that's nice and clear. We can tell (as long as we have two brain cells to rub together) from the name of the argument what the type should be. Of course then the whole discussion takes a nasty swerve towards the "Hungarian notation debate" side of things.

Putting the notion of encoding the type in the argument name aside we can still provide a nice way to show what type is expected. All we have to do is write our function like this.


class Iterable(object):
    def next(self):
        raise NotImplementedException

    def __iter__(self):
        return self

def line_count(itobj):
    """Pass me an instance of Iterable, please."""

    count = 0

    for i in itobj:
        count += 1

    return count

Now, because Python supports multiple inheritance, we can just use the Iterable "interface" as a mixin. It doesn't really matter whether we use it or not because the methods that it defines are clearly described and listed in the class definition. In addition, the function will still accept non-Iterable objects that adhere to the "Iterator types" description in the Python Library Reference. It's just that designing like this makes it simpler on the client to use the library you've developed. Note also that any class that implements the "Iterator types" set of methods could also just mix in the Iterable class for the hell of it.

I know that this seems like a lot of flailing for nothing (since there is still no type-checking until runtime at which point it still happens using Duck Typing) but the point here is that I think the Duck Typing argument is a design argument and not a dynamic vs. static typing argument. You can implement Java-like interfaces in a dynamic language, they just won't be checked at compile-time; but maybe that's OK. The real goal is to: a) make sure that the implementor knows all the methods they need to support and b) make sure that you are not passing the wrong instance by mistake to your function. I think that by adhering to a few simple design guidelines both of those goals are pretty easy to accomplish.

NTP on Windows - Keeping up with the time

2005-04-15T09:25:00.000-04:00

You can easily keep your local clock synchronized on recent versions of Windows

The NTP (Network Time Protocol) allows you to synchronize your local time to the time maintained by a set of servers. All those servers are fed (eventually, somewhere up the chain) by a clock of incredible precision (no, not Big Ben). UNIX users have had this feature since its creation and now it's time (no pun intended) for Windows users to have their day in the sun.

Just go to your start menu, select "Run..." and type "cmd" into the little text box. Click OK and type this line into the resulting command line:


net time /setsntp:pool.ntp.org

Your clock should now remain synchronized and up-to-date until the end of time.

Version 0.1.0 of WODFS Released

2005-04-08T18:49:00.000-04:00

We finally released a version of the WOD. It still requires a tremendous amount of effort for others to get it to run (you need to install a bunch of dependencies) but we wanted to "release early, release often". The next release should contain a couple of improvements such as Windows support, limited WebDAV support, XMLRPC support and a re-structuring of some of the filesystem level code.

Podcasting meets blogging

2005-04-06T10:34:00.000-04:00

Jon Aquino has started playing with an automatic blog to podcast script that looks pretty cool.

One box where two will do

2005-04-05T10:10:00.000-04:00

By adding a keyword search for Google in Firefox you can remove the need for the extra Google search box

All you have to do is go to http://www.google.com and right-click on the form. Select "Add a keyword for this Search" and fill in the boxes in the popup. You can now use the keyword you just created in the address bar instead of using the right-hand-side Google search box.

Reading between the lines

2005-04-02T14:36:00.000-05:00

Reality can never be communicated, only experienced.

Words can never cause another person to experience anything. Only the receiver can perform the act of experiencing. We speak in order to draw smaller and smaller circles around the experience we are trying to provoke in the other person. When the circles are as small as we can go, we have a reasonable assurance that the other person has experienced the same thing that we have.

Writing (or saying) something using a lot of words allows us to refine the various possible meanings of the communication to a point where we not only have been able to express ourselves (i.e. we feel like we have accurately represented the experience) but also have been understood (i.e. we feel that the recipient has accurately reproduced the experience for themselves). This is easy. The way to do it is to start talking (or writing) and not stop until the other person is able to communicate back that they have experienced the same thing.

When we read religious or spiritual texts we often are confronted with the fact that the text is much shorter than what we really need in order to be able to reproduce the author's experience. This is simply because it would be impossible to expect an author to be able to write a text that could be understood by anybody at any time in any situation.

Writers of these texts therefore resort to a form of compression that allows them to accomplish two goals.

The first goal is to express themselves in a form that can be understood. It is not important how long it takes for the understanding to occur and indeed many of the authors (of these texts) that are read today have been dead for some time. The notion that it is irrelevant how much time and effort are required to understand the text is an interesting one since it underlines the fact that it is the understanding of the communication that is the ultimate goal.

The second goal is to allow the communication of the concept to be passed on even by those who do not understand it. In order for this to be possible it is important for the concept to be expressed simply. If the simplicity of the expression is great enough then even the greatest fools of the earth will propagate the message until such time as someone with the capacity to understand it can hear it.

So we have two conflicting goals: understanding and simplicity. How then are we to formulate a concept that is inherently complicated and deep in a way that is simple? Recall that simple in this context refers to the complexity of the message, not the complexity of the concept.

The Tao Te Ching expresses complicated concepts in simple verse. The Bible expresses the truth in the words of Jesus via parables. A highschool student could learn and recite each of these stories with little difficulty. However without the key to what is locked inside, they remain opaque and confusing.

The key is to "explode" the concepts hidden within each story; to tease each sentence apart and to push our understanding of each aspect of the work to it's limit. We need to read between the lines. Unfortunately, there are not very many lines so there appears to be a huge number of possible interpretations for each of these esoteric texts. The trick then is to draw as many lines as we can so that the space between them becomes smaller and smaller until, one day, perhaps, we will experience reality.

The saga of USENET: The insanity continues

2005-03-30T16:46:00.000-05:00

This is beautiful. A thread about acquaintance spam spiraled off into a furious bout of invective and harsh words. It amazes me that people who should know better don't. Now it may be true that somebody "started it", but at least they had the tact and intelligence to back off and not involve themselves in the comment war that ensued.

I just found it hilarious that a supposedly high signal/noise ratio medium (blogging/RSS/contemporary "collaboration" tools) can be reduced to a tangled USENET thread in less time than it takes to click on the "BlogThis" bookmarklet in your toolbar.

Python typing part II

2005-03-26T13:09:00.000-05:00

Again, I still don't understand why we really need this type checking stuff. It seems to me that since we are adding type checking that will be done at runtime anyway the whole thing is a bit of a moot point. I may be missing something critical here but AFAIK the BDFL is not considering modifying the compiler/interpreter to handle type-safety. Why not just have a library with decorators that allows you to do type checking?

I've hacked together a strawman typechecking decorator that I would appreciate comments and flames on. I'm just trying to get a grip on what the issue with keeping the language small and consistent is.

Static typing and Interfaces in Python

2005-03-25T10:34:00.000-05:00

OK. So here's the obligatory weigh-in on static typing in Python. I don't see why we need to introduce a new syntax fior something that could be handled very cleanly with decorators (and I'm sure many people have argued this point).

I also choke on the whole "Interface" thing. We don't need those either. Just create a class with a bunch of raise 'Not Implemented' statements for all the method implementations. Anybody subclassing the class would be forced to implement the methods. We could even have a meta-class for it that checks that all the methods are implemented - again, no need for extended syntax.

One of the things I like most about python is its consistency: things usually work the same way. Let's keep it nice, simple and predictable.

On a slightly related note I see that the BDFL is considering wiping out map, filter and reduce. I applaud this since the list comp syntax neatly handles these functions' roles.

Claiming my blog at blogshares

2005-03-11T09:24:00.000-05:00

So I joined Blogshares to see what the fuss is about.

Reading the fine print

2005-03-09T15:31:00.000-05:00

Reading the printer-friendly version of articles removes distractions.

To my great chagrin a number of signal-heavy sites now have ads scattered all over their articles, breaking up the flow and generally getting in the way. Not to mention the Flash ads that whirl and blink or the DHTML ads that seem to want to take over the entire screen. I'm looking at you IBM, and ITworld.com and cnet.com and all the others that for years have had great technical/business/whatever-you-want articles but who now waste my time and visual bandwidth with their crappy, non-targetted ads.

No more.

From now on I will be reading only the "printer-friendly" version of these articles. This version contains no ads (well, there may be some but they are usually at the top or bottom) and in general has a better layout since they have removed the ugly navigation that these providers have managed to plaster all over 2/3 of my screen.

On a related note, check out Greasemonkey - a firefox extension that allows you to run custom DHTML for each site you visit. By default this extension ships with a script that disables Google ads. When I say disables, I actually mean hides since the original GET to the server actually returns the HTML page with the ads built in but Greasemonkey strips it out on the client side. So your favourite blogger is still reaping the benefits of the ad bar but you don't have to suffer the indignity of giving up screen real-estate.

This is so wrong!

2005-03-09T14:47:00.000-05:00

Now I'm not usually a namby-pamby "love the animals" kind of guy; and I've eaten (and enjoyed) my fair share of rabbits, but this is just too much.

Introducing the main idea

2005-03-09T09:59:00.000-05:00

Start each article/post with a single phrase that briefly describes the idea being communicated.

I've been reading Dave Pollard's excellent weblog for a little while now and I've noticed that at the beginning of each article he writes "The Idea" followed by a description of what he wants to communicate in the article.

This is a great tool for focussing the contents of an article and it also allows readers to read the full story with the basic idea in mind. It provides a sort of scaffolding for the rest of the article so that additional ideas can glom onto the original one to create a (hopefully) coherent whole.

Automatic filing with del.icio.us

2005-03-07T10:35:00.000-05:00

From Otaku, Cedric's weblog: Automatic filing with del.icio.us I made a couple of modifications to allow the launch to happen in another window. Drag this bookmarklet to your toolbar to apply the toread tag to a page.

Cedric also made a delete from del.icio.us bookmarklet.

Online medical records - strawman

2005-03-02T11:37:00.000-05:00

David Toub writes that the three main obstacles to having online medical records for patients are

Concerns about security (getting better, but reports of major academic institutions being hacked don't help. Regardless, EMRs are still inherently more secure than paper records)

Cost (big issue-a small practice just doesn't have $20k to blow on a new system)

Startup: How to input thousands of paper-based records into an EMR fast and inexpensively

So let's take a look at these items one by one.

Security concerns

It is normal, I think, in this day and age, to worry about the accessibility of your data online. With so many scandals and so much spam and hacking it is easy to become overly anxious about exposing your information to a potentially hostile environment. The answer to this is a liberal application of strong encryption. I'm not talking about weak SSL with it's pansy 128 bit keys (many implementations of which have failed to demonstrate that there is no redundancy in the key bits), I'm talking about public key cryptography combined with some sort of DRM tool that would allow doctor's to access a patient's information once, in the controlled context of a visit.

I recently discovered a feature in PGP that allows you to encrypt a text document to a built-in viewer that is distributed with PGP and that in addition to requiring you to enter the appropriate passphrase for the key also displays the text in a non-copyable, tempest technology resistant window. Something like this could be used so that a patient would be able to authorize a given physician to access their data once (and only that one time). The physician would have full access to the patient's file for the duration of the consultation but afterwards they would be unable (even for their own purposes) to retrieve information other than

The fact that the given patient visited them
The treatment that they provided
Perhaps (but not necessarily) a general description of the problems that the patient presented with

Obviously, it would be important to allow physicians to update the patient's records in such a way that the next time the patient visited the physician would be able to view their own notes on the previous treatment.

The key point here is that it is definately possible to build a system where the patient controls all access to their information. This would also limit physicians' liability in certain situations. For example, staff from a doctor's office would be categorically unable to share any details about patients other than their name (or other identifier) and the fact that they visited. It would even be possible to visit a doctor anonymously. As long as the physician has access to your records, they don't really need to know who you are. This would encourage people to use these records for things that could be traced back to them disfavourably such as abortions, HIV/AIDS treatments and so on.

Cost of a new system

In order for the system described above to really work it would need to be de-centralized and managed by each patient individually. Any attempt to centralize the system would probably open the door for abuse (if for no other reason than administrators of the central system could track usage statistics in a non-anonymous way).

I don't think that physicians would need to pay anything at all to use the system (other than the fees for a computer and an internet connection).

We have seen from systems like blogging, FOAF, bookmarks and so on that privately owned and managed resources can still be shared profitably.

Cost of inputting older records

I don't think this is really a big deal either. The fact of the matter is that right now (most) medical records are maintained by each healthcare institution individually. When a patient goes for X-Rays or for blood tests or any other tests, the testing body needs to send the results of the tests back to the physician. How easy is it really for a given physician to find out everything about the patient that is presenting? Not very, I'll wager. This is mainly due to poor organization and not any malice on anybody's part.

There isn't really a need to input all prior records at once. It would (or should anyway) be enough to input them as they become pertinent to the current activities of the patient.

For example, let's say that a patient had a colonoscopy at the request of their physician last year. Whatever condition prompted the test was resolved weeks after the test. The results of that test (and even to a certain extent the fact that it was performed) is of no import the next year when the patient presents with a broken wrist. In that situation, the physician would simply enter the new information without worrying about the lack of the results of the colonoscopy (never mind the fact that the patient should be able to selectively allow the physician to only access the parts of their record that are pertinent).

If on the other hand the same patient presented with bloody stool and the physician wanted to order a fecal occult blood test they should be (perhaps verbally) made aware of the colonoscopy the previous year. At that point the physician would order the results from the lab (or get them from the patient). So far I think this is how things work today. The difference is that once the test results are obtained the physician would enter the new information into the system.

In this way the system would be built up incrementally over time and would not require a huge amount of up-front investment.

The future flavour of webservices

2005-03-01T15:38:00.000-05:00

You've probably heard about webservices: they're those things that use SOAP, XML-RPC and a whole alphabet soup of technologies better left untouched. Another acronym to know is REST. It is the red-headed stepchild of webservice - the technology that IBM, Microsoft, BEA and all the others don't want you to know about. Why? Because it exists already. Your browser already uses REST to browse the web. Certain tools (like WebDAV) use REST to store and modify resources on the web. And where would we be without POSTs from forms?

SOAP and XML-RPC are both suites that allow you to exchange XML-encoded messages with other hosts on the network. Let's say you want to find all the top scoring players in the NHL. You prepare an XML document that describes your request and you POST it to a specific host that you know will actually respond. The URL you post to is called the "endpoint" and theoretically represents a resource or process. The host returns another (specially prepared) XML document that (once you un-wrap it) contains the list of players.

In REST things are much simpler. All you do is GET the URL. No fancy documents or anything. GET the URL and the server returns the list.

What is interesting however is the difference in viewpoint between what are now being called the WS-* camp (the SOAP and XML-RPC guys) and the RESTafarians.

The RESTafarians are very focussed on the semantic nature of URLs, often talking about the relationship between URLs and the resource representations they refer to. WS-* guys often talk about how the latest envelope spec can be enhanced.

In case you're wondering, I'm pretty much sold on RESTafarianism although I do like XML-RPC since it has a much simpler design/interface than SOAP.

Here is a very cogent introduction to the principles and philosophies of REST: http://naeblis.cx/rtomayko/2004/12/12/rest-to-my-wife

Here is the home of XML-RPC: http://www.xmlrpc.com

Here is the home of the SOAP working group: http://www.w3c.org/2000/xp/Group/

Security in the WOD's networking layer

2005-02-26T15:05:00.000-05:00

When you use the WOD, you are connected to the global network of WOD users. You immediately begin exchanging data packets with all your peers on the network to increase redundancy for your packets and to ensure that your data is stored permanently. In addition, clients may request specific packets at any time - requests that must be replied to if the network is supposed to work properly.

In order to guarantee a certain level of privacy for each WOD user, we use a packet-level encryption scheme. This scheme needs to allow the WOD's similarity-based capabilities to continue working and therefore encrypting a given cleartext packet must produce the same ciphertext for all clients that perform the encryption. This is of critical importance since the benefits gained from the self-similarity of data in the network are lost immediately if there are two possible ciphertext results for the same cleartext.

To circumvent this issue, each packet is encrypted with it's own MD5 hash. This guarantees that every client encrypting a given packet will encrypt it exactly the same way. Pointer blocks/packets maintain both the fingerprint of the packet (the SHA-1 hash of the cleartext) as well as the password of the packet (the MD5 hash of the cleartext). The security of MD5 is not really an issue here since collisions in the hashing function do not significantly reduce the strength of the generated password.

Packets are encrypted using AES - a symmetric encryption algorithm. This is because the key for a given encrypted block is determined by the block's data itself. Generating public/private keypairs might enhance security but would make things a lot more complicated to implement.

This encryption scheme is not proof against chosen plaintext attacks: if an attacker has a file and wants to prove that you have the file too they are able to regardless of the encryption. This does mean however that only people with the same files as you can prove you have those files. Users would be well advised to employ third-party (higher level) encryption for their files if additional security is needed at this level. The rationale for not implementing per-user security is that, for the most part, as long as I own the same file as you, I don't care if you see that I own it.

ISBN Linker Bookmarklet

2005-02-22T11:26:00.000-05:00

Drag and drop the [ISBN Linker] into your toolbar and click it to transform ISBN numbers in the current page into links to Amazon.com.

The WOD's network design

2005-02-17T18:50:00.000-05:00

Each system on the network is a node and each node is connected to zero or more peers.

Nodes

When a node is connected to less peers than its threshold, it broadcasts a packet onto a multicast address with the port on which it would like to receive data. It also listens on the multicast address and picks up advertisements for other nodes until it has reached its threshold. Once the threshold is reached the node stops advertising itself and stops listening on the multicast IP. If the number of connected peers ever drops below the threshold, the node will begin to advertise itself and listen again for other peers.

Peers

Once a packet is received from a prospective peer, the node examines the requested port and then connects to the peer on that port. Packets sent by any peer are gathered together and sent into a central pipe.

The central pipe

The central pipe of packets coming in from the network is watched by a series of functions that have registered to be notified of incoming packets. Each watcher may operate on a packet before it is passed on to the next watcher.

By default, a watcher is set to pick up request packets and handle them. If the cache contains the fingerprint requested, the data for the fingerprint is packaged up and the node connects straight back to the requestor to deliver the data. If the cache does not contain the fingerprint, the request is rebroadcast to a single peer. In this way requests move across the graph of nodes and are replied back to their sources once they can be fulfilled.

This means that the requesting of a fingerprint on the network acts a little like a future (we should probably have explicit support for that).

Another default watcher receives data packets and processes them.

Data Packet Processing

Each data packet is picked off the network and examined; if the TTL (a stupid name but I'll explain it in a second) is 1 then the data is stored on the node and not re-transmitted. If the TTL is greater than 1 then the node stores the data and re-transmits the packet after decrementing its TTL. In this way data blocks get stored on several nodes in the network to provide redundancy. Each node will be stored by a number of nodes equal to the TTL set by the original emitter. So if a packet is emitted with a TTL of 5, it is guaranteed to be stored on 5 nodes other than the original before the packet is removed from the network and no longer broadcast.

SHA-1 Broken

2005-02-16T09:57:00.000-05:00

From Schneier on Security: SHA-1 Broken:

SHA-1 has been broken. Not a reduced-round version. Not a simplified version. The real thing.

The research team of Xiaoyun Wang, Yiqun Lisa Yin, and Hongbo Yu (mostly from Shandong University in China) have been quietly circulating a paper describing their results:

collisions in the the full SHA-1 in 2**69 hash operations, much less than the brute-force attack of 2**80 operations based on the hash length.
collisions in SHA-0 in 2**39 operations.
collisions in 58-round SHA-1 in 2**33 operations.

This attack builds on previous attacks on SHA-0 and SHA-1, and is a major, major cryptanalytic result. It pretty much puts a bullet into SHA-1 as a hash function for digital signatures (although it doesn't affect applications such as HMAC where collisions aren't important).

This is pretty annoying since the WOD uses SHA-1 to calculate the fingerprints of the blocks in the system. We pretty much depend on the fact that the SHA-1 algorithm requires 2⁸⁰ comparisons in order to generate a data block whose fingerprint collides with another block's. The reduction in the overall namespace for fingerprints is a real pain. I'm already starting to look for another option that will fit in 160 bits and will have the original unrestricted namespace.

It also occurs to me at this point that we will need to build in some support for changing the hash algorithm used for fingerprints to allow for eventual upgrades to the hashing algorithms we are using. Maybe we need to add a couple of bits to a fingerprint that we can use to specify the algorithm used to generate the fingerprint. Of course, this would add a certain amount of overhead to the whole operation since we would have to decide which algorithm to use each time. We also run into the problem of figuring out how/when to change the algorithm, in other words, how do we figure out that a given algorithm is generating collisions now?

Tip for URLs in Podcasts

2005-02-04T13:34:00.000-05:00

Try using TinyURL to create URLs for all the links you want to give out in the program.

It would be nice to have a service similar to TniyURL that would produce mnemonic (pronounceable) URLs as well as returning the same "tiny" URL for a given URL every time it is entered.

who's important? who cares?

2005-02-03T09:56:00.000-05:00

David Toub asks who's important? who cares?

I think that people always want to categorise things; it's part of our basic pattern recognitory mindset. When we try to classify however we often get caught up in our own constructs. I'm talking here about hierarchical classification versus categorization versus any other method of classifying. A debate is raging right now about "tagging" given the proliferation of services such as flickr and del.icio.us. The debate centers around the idea that tagging provides a flat namespace that gets corrupted/polluted when the same tag can have multiple semantic values. Tim Bray gives the example of military "drills" versus oil "drills". Because the system that we invented (tagging in this case) is built in a certain way it constrains us to act (and even think sometimes) in a way that is congruent with it's physical design.

I was talking to a friend the other day and he mentioned that a guy we have know for years and years (and never particularly liked) happened to be around and start beatboxing (vocal percussion where you make the sound of drums and cymbals with your mouth). It turns out that the guy was quite good at it. So my friend asked the guy for his number and then felt all bad that he had known the guy for years and never asked.

Each person presents a number of facets to the world and some of them may be interesting to you and others won't be. I don't think it's fair to beat yourself up just because you didn't like someone before discovering a certain facet of theirs. I'm sure that had I met Beethoven or Mozart when they were kids I wouldn't have hung out with them much: they would have been too much into music and I would have wanted to talk about and do other things.

Our faults are what make us unique. If we all were perfect we would all be identical (assuming there is a single definition of perfect). The difference between a master and an apprentice is that the mast selects which faults to show and which to hide which gives their practice a unique flavour that is hard or impossible to duplicate. The apprentice spends so much time fussing with the tools and are not able to choose which of their faults to hide which makes their practice unpolished and rough.

So to categorize people is to either deny that there is anything more to them than the single aspect which we are considering as a basis for categorization or to say that we don't care about the other aspects of the person. Now, when we are doing studies on fertility rates by geographical area it is obvious that we don't care about professions, but if we are looking at ranking people there are often a lot more aspects involved in the ranking than simple seniority or even a simple qualitative assessment of skills.

Tweak Firefox to work faster

2005-01-31T16:16:00.000-05:00

Go to about:config and set network.http.pipelining to true and network.http.pipelining.maxrequests from 4 to 10.