Using encrypted SSL keys and certs in twill

Recent versions of mechanize support SSL connections and so, by extension, does twill. Recently I have been trying to open an HTTPS connection to a site that requires both a certificate and a client key. To try this at home, you will need a version of openssl.

The key and cert came to me packaged in a PKCS#12 format file. The first thing I needed to do was unpack the key and the cert like so:

openssl pkcs12 -clcerts -nokeys -in cert.p12 -out cert.pem
openssl pkcs12 -nocerts -in cert.p12 -out key.pem

After doing this, I plugged in the filenames of the cert and key in my twill code. Note that you need to access the underlying mechanize.Browser object when setting the client certificate:

import twill
from twill.commands import *

host = 'mysecurehost.com:443'
b = twill.get_browser()
b._browser.add_client_certificate(host, 'key.pem', 'cert.pem')

go('https://mysecurehost.com/secured_url.html')
show()

The problem you run into when doing this is that the SSL libraries prompt the user to enter the password for encrypted keys at runtime. This makes automating the interaction tricky at best. I tried fumbling with the PyOpenSSL library but it seems that setting a callback for the passphrase retrieval does not actually work. The set method call returns but the callback is never called during key decryption.

My (hackish) solution was to remove the encryption on the key before using it to connect. You can do this by re-exporting the key from the PKCS#12 file:

openssl pkcs12 -nodes -nocerts -in cert.p12 -out unencrypted_key.pem

Now if you use unencrypted_key.pem in the twill code above, you will be able to connect without providing a password for the key.

Centralized version control blues

I am in the process of migrating a SVN repository to Darcs. A couple of the changesets in the SVN repo make baby Jesus cry. They create conflicts such that checking them out in order fails spectacularly and you need to go hands with the working directory to clean everything up before moving on. All of this highlights a rather interesting (though totally predictable) issue that plagues centralized version control.

In Darcs, if a changeset is no good, you can just refuse to pull it. Doing so excludes that patch from your repo and no harm is done (unless, of course, a subsequent patch depends on it). In SVN, given that everybody checks into the same repo, by the time you realise that a patch is going to trip you up, it's already in the bloody database. Now, SVN provides no tools for deleting a specific revision (although I hacked together something that I'll try to toss out there someday) so you can't exclude the changesets that put the DB in an inconsistent state. All of this means that I have been wrestling for the past week and a half trying to get this stupid repository to ignore the "corrupted" changesets. Arrrrgh!

I just called...

Two weeks ago I purchased my first wake-up call. It was late at night and I had a particularly important meeting the next morning that I really couldn't afford to miss. So I slapped "wake-up call" into Google and hit the first site that popped up. If I'm not mistaken, it cost me $1.20 USD for that call.

The second time I wanted a call I did a little more research and turned up a site that offered calls at $0.75 USD with a "snooze package" of three snoozes for an additional $1.20 USD or something. I used this site several times trying to work out how the different options work.

Then (as usual) I got frustrated by the fact that I was essentially paying to use somebody else's script when I could write one myself. I spent and evening getting cozy with the Skype API and finally (after quite a bit of hair-pulling) figured out how to get SkypeOut to call my home phone and play a sound file (in WAV format) to me.

This weekend I put the finishing touches on my new wake-up call script. I can now set a date/time and have my computer call me to either remind me of something or to wake me up. I could also do other, wacked-out things like have my computer monitor an inbox for an important email while I'm at the country and then have it call me and read (via Text To Speech) the email's contents to me over the phone. This morning I used the scheduling to wake up and it worked like a charm!

The only problem now is that Skype charges me $0.024 CDN per minute for each call as well as a $0.05 CDN connection fee for SkypeOut. Still, it's much better than the $0.75 USD which was the cheapest I had found before (that is, without a subscription).

Performance Tuning - applying a function to a list

>>> import timeit
>>> timeit.Timer('map(f, range(10))', 'f=lambda x: str(6+x)').timeit()
15.467374484745967
>>> timeit.Timer('[f(x) for x in range(10)]', 'f=lambda x: str(6+x)').timeit()
16.062227741235269
>>> timeit.Timer('for x in range(10): f(x)', 'f=lambda x: str(6+x)').timeit()
14.686095299821623

And so, we can see that map is still faster than list comprehensions and the for loop beats them both. If you don't need the return value of that function, don't create the list: friends don't let friends create unnecessary objects. On the other hand, if performance is critical and you need the return values, you should prefer map over a list comprehension.

Picking teeny-tiny cherries

I would love to use Bazaar or Mercurial as a DVCS; just the fact that they are written in Python (and therefore eminently hackable as far as I'm concerned) is worth making the move. I just can't get past the fact that Darcs (my current favourite) provides support for hunk cherry-picking. This is a killer feature. It's great to be able to cherry-pick files into patches, but Darcs actually allows me to package up my hunks that relate to different changes into different patches. This gives me much more control over the way I decide to describe the modifications I am making to my project. When I can only operate at a higher (coarser) level of granularity (like Mercurial or, God forbid, SVN) I get stuck fiddling around removing hunks from files before committing the patches.

Another project to take up my time

I've just tossed out the first version of a small web-stack for Python called Khepri - a clever bit of word-play on the name "Apophis". There are a million others out there but I'm very picky. This one is mine and I like the way it is so far. Obviously, it is far from being really usable but if you are a developer, you can judge for yourself. Check out the darcs repo.

The cat came back the very next day

Well, not quite, but he is back. Tygger finally came home on Friday night. I came back from an evening of recording stuff at Justin's and called out to him out of habit. I heard somebody yelling at me from the neighbour's porch and got angry for a moment at Jack for looking so much like Tygger. I headed towards him to pat him anyway and he fled. It was then that I noticed that he wasn't wearing his bell. I followed him for a bit then gave up when he continued to elude me. As I headed upstairs, I thought once again "wouldn't it be stupid..." So I picked up the flashlight and went back outside. I crooned and cajoled for about ten minutes before he finally bolted past me and into the house. He was super-thin and even once he was inside I had to open his mouth to check for the broken tooth that confirmed his identity. He still smells a bit off but that's nothing a couple of baths won't cure. I guess that for now everything is right in my little world!

Tygger still missing

Our little guy is still missing and although everybody seems to have a story about how so-and-so's cat took 18 days to come home and their cousin's dog once crossed all the continents to get back to his owners I feel less and less confident every day that he will return. Marquis is all depressed and doesn't play; he just wanders around looking for Tygger and trying to get attention from us since he has nobody to play with. I certainly hope that he is huddled under a porch not twenty yards from our front door but I fear he is wounded somewhere or that he has been "taken in" by some well-meaning people who have discovered what a wonderful cat he is.

Linking into the weeb

So Justin spent the weekend setting up a bunch of accounts for himself at blogger and del.icio.us. I figured he needs a bit of link love since he can't manage to get himself onto the first page of Google's results. Now, I'm not sure that Google still indexes blog posts the way they used to but you can check out Justin's blog or go directly to his website.

Missing

Our cat, Tygger, got out the window Saturday night while I was sleeping on the couch in front of the TV. We spent the day yesterday looking for him and calling his name but to no avail. Marquis is starting to feel a bit lonesome. We put up posters this morning around the neighbourhood but I'm not really all that optimistic. We worry that he may have tried to go back to our place on Stratford. The tricky bit is that the wife of the guy who is renting from us isn't supposed to know that we had a cat living there before they moved in. Anyway, we'll go and put up more posters this evening. The apartment seems very empty without our little guy snoozing away on his blanket on the couch.

2nd April Menu

April is cool with frequent showers, hail and just a little wet snow. But spring is just around the corner. You can smell earth in the humid, expectant air. Lightness to match the season coming is in order, but these past weeks have been quite chilly, so a little warmth would not go amiss. Thus I propose to you the 2nd April menu with a hint of the exuberance of youth and just enough heat to drive winter's final tendrils back into the darkness. All dishes listed are low calorie unless indicated (with a *).



Entrées & Sidedishes



Cheesy Broccoli Bake

Oriental Broccoli Salad (soy, honey and sesame)

Broccoli with Herbed Breadcrumbs



Roasted Zucchini with Fresh Thyme



Asparagus Tips with Roasted Red Peppers

Orange Ginger Asparagus



Spicy Nutmeg Carrots

Ruby Carrots (cranberry)

Carrot and Coriander Soup



Cheesy Chicken Chowder



Main Courses



Zesty Chicken Sauté

Apricot Glazed Chicken Breasts

Sesame Chicken Salad



Skillet Shepherd's Pie



Bell Pepper Stir-Fry (on white rice)

Risotto Napoletana (sun-dried tomatoes, salami and parmesan)



Broiled Sesame Salmon

Oven-Baked Tandoori Salmon



Desserts



Raspberry Sorbet

Fruit Crisp*

Last know good configuration

Darcs has a feature (that I have never really tried) that allows you to locate the last working version of your source. I am currently shlepping through changesets in SVN applying them one by one in order to figure out which one breaks a series of tests that depend on our mocking library. It seems like it would be so simple to just repeatedly run the tests, stepping back through revisions until the tests pass.

Twittering tools

Last night I installed Twitterbar, an extension for Firefox that adds the ability to post to Twitter straight from the address bar. In conjunction with the Jabber extension, JabBar, this provides a really nice little interface for using Twitter. I have a small, single line rolling feed from Jabber and since Twitter updates me via IM I have a mix of lines from any conversations I'm currently having with my contacts and any updates from people I'm following on Twitter. Now, I also have a way of easily posting twitters without changing the current window I'm using. Previously, I would have to open up the conversation panel with the Twitter IM bot, now I just fire and forget.

A reality check for old party lines

Last night Quebeckers overwhelmingly told both the PLQ and the PQ that they no longer want to focus discussions along sovereignty/federalism lines. The ADQ was boosted into a position as the official opposition and I think that, this morning, people are looking for meaning in the moves from yesterday.



It is important to remember that a fairly low proportion of the population should be voting "strategically". The reason for this is simple: unless the results in your riding are really a forgone conclusion (and last night should serve as a reminder that nothing is as certain as we think), you may end up splitting the vote against your party. To have the foresight required to strategically align one's vote so that an overall message is communicated is equally problematic: in trying to send a message to my party by issuing a protest vote I may actually participate in their defeat (given the extremely thin margins between candidates in some ridings...cough...Sherbrooke...cough).



A common error is to ascribe some sort of intelligence or design to the results of an election. Phrases like "the voters have spoken" or "Quebeckers are sending a strong message" are not false, but they only hint at the underlying truth: the message delivered is a composite one. To boil it down to any one factor or issue would be to imply some sort of external collusion on behalf of the voting population. To look only at the votes that counted towards a win is equally treacherous, especially when one notes that most ridings had at least two if not three candidates neck and neck all evening long. Indeed, one of the more cogent reflections of the evening remarked that in addition to winning 41 seats, the ADQ managed to come in second in over 40 ridings. Wow. Think of what could happen if they managed to attract a couple of extra votes! Dumont was absolutely correct in saying "on connait la prochaine etape; je vous y donne rendez-vous."



All in all, I am pleased with the results of this election; a lot more pleased than I have been in recent years. I was disappointed to not see any Green Party members garner seats but noted with satisfaction that QS did not do much better. An unfocused and inexperienced campaign may have hurt these two parties - certainly QS did not exactly shine in a post-election speech last night where all they could muster was some vague ranting about proportional representation (a point that should be conceded but that hardly belongs exclusively to small struggling political parties).



I am happy that Quebeckers have decided to keep a leader that has experience. I am happy that they have placed a young, dynamic and energetic "watch dog" opposite Mr. Charest at the National Assembly. I am happy that the rise of the ADQ shows that a party can amount to something even if it is not one of the province's "natural" parties. I am happy that the election did not get polarized along "nationalist/federalist" lines. I am happy with our decision to have a minority government: in the future I think we'll see a lot more of these as we move from strongly polarised discussions to shades of grey. The only time we should have a majority government is when everybody agrees and the talent and experience exist to support that arrangement. Mr. Dumont is not ready for power but the last 12 years have certainly shaped his character; I look forward to seeing him push for a government of his own 2 years from now.



Mostly, I am happy with our commitment to the future of our province: by providing for an experienced guiding hand and a powerful young movement, we have the best of both worlds. By smacking down the PQ we have said, in no uncertain terms, "your platform is not for us this year, you will play king-maker and nothing else; when we are ready to discuss sovereignty again we have no doubt you will be waiting."

Term "visible minority" considered harmful

A UN committee released a report today on Canada's efforts to combat racism. The report praises Canada on some aspects of it's anti-racism programs but says that the term "visible minorities" may not be "in accordance with the aims and objectives of the [International Convention on the Elimination of All Forms of Racial Discrimination]". The convention says that "distinction based on race, colour, descent, or national or ethnic origin is discriminatory".

Great. Now we can no longer tell the difference between a black man and a Ukrainian because to make the distinction between them (that is, to employ another definition of the word, to discriminate) based on the colour of one's skin and the other's nationality is "discriminatory". Um yeah. Aren't we pushing this a little too far? I mean, how far am I from being an "individual with low amounts of skin pigmentation from a city near the coast with parents and in posession of a penis"? Come on! Sometimes we need to differentiate between two people or two groups.

Now, I understand that penalizing one group or another based on these distinctions is what is really at issue here. That doesn't stop me from being a bit frustrated that there are actually people somewhere in the world paid to come up with this shit, write it down and hand it out to other people who are paid to read it. We could be solving "real" problems but instead we are haggling over what to call somebody who is black. How to talk about a deaf person without mentioning the fact that they can't hear.

Talking about people's differences or referring to them in some way is not discriminatory. There is only a problem when those differences cause you to judge based on those differences instead of based on personal merit. Now go home, make friends with a black deaf retarded gay woman and call it a day.

Version control through the ages

It just occurred to me that over the past six or seven years I have gone from using CVS to track all my code and projects to using Subversion and from there to using Darcs. In general I find that each one has really been an improvement over the previous tool. I miss certain Subversion features in Darcs (specifically svn:externals) and get a bit confused sometimes when dealing with multiple repositories but the extra functionality is well worth the effort to learn.

Quebec: a question of definition

Prime Minister Stephen Harper has said he will introduce a motion to recognize Quebec as a "nation within a united Canada". This was immediately pounced upon by Gilles Duceppe who demanded that the motion be reformulated in order to recognize a "nation that is currently within Canada".

The unfortunate thing is that we are spending our time debating the wrong thing. That a certain socio-ethnic group of people exists in Canada is not a question. That the people in that group have inherited a certain culture and certain beliefs is not at issue. The real problem is not that nobody "recognizes" Quebec. The real problem is the divide between those who believe that Quebec should have a certain level of sovereignty and those who disagree.

It all comes down to control; and a lot of separatists/sovereignists feel that Quebec should have full and complete control over everything that goes on inside its borders; including immigration/emigration and any other kind of import/export. This is not a question of culture, nor is it a question of inclusion/exclusion: this is a question of power. We can see it when people suggest sending delegates from Quebec to international assemblies; as if we really wanted to debate the north american missile shield with the "ambassador" of the state of New Mexico. Countries exist for a reason. They are a grouping of geographically and socio-economically like-minded people who get together for the same reasons as any other community forms. They provide a tool for dealing with other groups of people and, as such, are very useful.

In the case of Canada, the country also provides a way to re-distribute wealth to attenuate the issues encountered by one or another constituent population going through hard times. When the farmers of western Canada need help, the fishers of eastern Canada provide it (albeit indirectly). With regards to this particular issue, Quebec has historically felt hard done-by because of its own richness and other provinces relative poverty.

Meta-communities also provide the opportunity to set certain guidelines and enforce them at a higher level. We can see some of the effects of this by looking at organizations like the U.N. where a number of participants agree to live by the rules established by all. Difficulties arise however when participants in such a scheme attempt to assert their independence or sovereignty. A perfect example is when the U.S. decided unilaterally to invade Iraq against the recommendations of the U.N. security council. Unfortunately, the sanctions available to the U.N. is very limited. It comes down to "might makes right" and in Quebec's case we can expect more of the same: because of its socio-economic place in the country, Quebec stands to gain a significant amount of "bullying power" by declaring its independence (or at least having it recognized). The simple fact that a political party exists at the federal level to represent and defend the concerns of a single province is proof enough of the fact that Quebeckers are prepared to do what it takes in order to guarantee that their concerns become matters of national importance.

Do I believe that Quebec is distinct? Absolutely! I was born here and will always be a Quebecker in my heart. Do I think that this "distinctness" (as opposed to the distinctness of Ontarians or Nova Scotians) merits the declaration of the independence of the province from the "rule" of the rest of the dominion? No. Absolutely not. So let's stop debating our uniqueness and start debating what's really at stake here: whether or not Quebeckers (and to a certain extent the rest of Canada) are entitled to a greater amount of "self-determination" than other provinces; I mean, at some level, each city could declare itself sovereign and we could start all over again building counties from cities, provinces from counties and (gasp) countries from provinces. If the only goal here is to get more power in parliament I say "let it be." You already have a whole national-level political party to yourself. Isn't that enough bullying for one nation? If not, then let's get the Mohawks and the Haida and the Inuit in there too so that they can represent their people.

pkg_resources.VersionConflict

This exception is raised when you update the version of a setuptools package without deleting the egg-info folder. The test command ends up expecting that the version is whatever is listed from the previous build whereas the value has changed in the meantime. Just run setup.py clean --all from your "root" directory to get rid of the error.

Some long-awaited updates

I took some time this weekend to revamp a lot of the infrastructure around this site. I really would like to start documenting my progress in pipe-making as well as some of the other things I'm currently juggling.

I have set up my own little proprietary engine to pipe the contents of different RSS feeds of mine onto the front page of my site. This should allow me to collect the various things I'm up to into a single spot.

Packaging Python

Ian Bicking writes about packaging Python libraries. He says simply that his advice is: "you shouldn't".

I really can't say that I agree on this one. While I do see the benefits of packaging libraries with the applications they go with, I have to say that after years of working in Java, where JARs are distributed and need to be included in an application-specific classpath, I really find the Python site-packages to be a breath of fresh air.