The Overwhelming Loss of Self

I suppose it had never occurred to me before. I mean, people died when I was younger but I guess I never really put two and two together. Maybe nobody does. I suppose it actually had much more to do with the type of relationship I had with those who died. I'm not talking about the quality of the relationship; no, this has more to do with socio-geography than it does with long walks on the beach. The first people that I remember dying were not people that I spent a lot of time with, either physically or in my head. In fact, I probably would have learned this earlier had I spent more time with some or all of them. I heard something on the radio today that I liked quite a bit: when you lose a parent, you lose the past; when you lose a child, you lose the future; when you lose a spouse, you lose the present. Losing the past is something terrible, but as children or young adults, we are pretty used to losing the past; we welcome it in most cases. Even so, we still lose that part of ourselves that existed only in the intersection of our lives with the other person's. Losing a child or a spouse (or sibling or friend) is even more terrible in the sense that you lose not only the part of you that existed in conjunction with that person, but also all the moments that were to come. Horrifically, it is the death of potential. Even though your potential may be realised eventually, the dance with that person is over. Insofar as that part of you is defined by its relation to the dance, you die too. To all those who have died but are still alive, I'm sorry for not understanding sooner.

How to add trusted keys to apt

The aptitude manual has a little section on how add trusted keys to apt.

The list of keys that apt will trust is stored in the keyring file /etc/apt/trusted.gpg. Once you have the GPG key, you can add it to this file by executing the command gpg --no-default-keyring --keyring /etc/apt/trusted.gpg --import newkey.asc . aptitude will then trust any archive that is signed with the key contained in newkey.asc.

Qemu networking goodness

It took forever but I finally got all of my Qemu networking working. My setup uses VDE, a TUN/TAP device to connect to my LAN, dnsmasq to give my QEMU hosts IP addresses and handle DNS requests, and IP Masquerading because apparently my wifi card can't spoof MAC addresses.

There follows a (probably incomplete) description of the setup

Setting up the TAP device

First thing to do is create the TAP device we will use to connect the VDE network to the LAN.

sudo modprobe tun
sudo chmod 666 /dev/net/tun     # I'm all alone on this box so...
sudo tunctl                     # This should create a device called tap0
sudo ifconfig tap0 10.0.0.1 up  # This is the IP for the VDE network

We're all done with TAP stuff.

Set up IP Masqerading through the TAP device

Now we need to make sure that traffic coming through the TAP device gets sent out over the LAN.

sudo su -c "echo 1 > /proc/sys/net/ipv4/ip_forward"       # Enable IP forwarding
sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE -v          # wlan0 is my wifi card

OK, that's it for IP Masquerading.

dnsmasq for DNS requests and DHCP

This one was easy; just install the package and modify the conf where it says #interface= to say interface=tap0 (without the comment mark and substituting whatever you got back from tunctl above.

VDE setup

First we'll create a virtual switch

sudo vde -s /tmp/switch1

Then give everybody access to the VDE

sudo chmod -R a+rwx /tmp/vde.ctl

OK, that's it for VDE.

Qemu hosts

This was a trickier bit. The stumbling block for me was that if you specify a MAC address for the host (I'm using Debian Etch as the guest OS), a new eth device is created. So make sure you specify the right MAC address from the start. If you screw up, you can always edit /etc/udev/rules.d/z25_persistent-net.rules and remove the extra eth devices. The reason this happens is that udev figures out that you added a new card (because of the new MAC) and so it configures another device. There's probably a more elegant way around this, I just haven't figured it out yet.

Boot your Qemu host

After all of the above, booting should go smoothly; remember to specify -net nic,macaddr=XX:XX:XX:XX:XX:XX -net vde. If you don't get an IP automatically, just run dhclient ethX on the guest and you should be set.

Frustrated by UML

I recently have been experimenting with User Mode Linux. My goal is to be able to run some of my tests on my local machine without needing to setup a whole EC2 instance.

At the same time I am evaluating the possibility of using UML on my instance to separate out the various services.

The main problem I am having at the moment is with the networking. Although I am able to connect to the host machine, I seem to be unable to create a route to the rest of the network.

They were once so shiny...

I was just looking at this picture of my pipes. I had forgotten how shiny they were when I first got them. Now the brass has started to develop a patina so it is quite a bit duller and a little splotchy in places.

The patina is actually desirable (for a lot of pipers) as it makes the pipes look antique. We would probably all play 100-year-old sets if we could.

The wood (which is cocobolo - a type of rosewood) has also darkened considerably since I got the set. The maple in the mainstock hasn't changed much though, nor have the mounts. There is a slight discoloration of the chanter mounts since I keep it in a red leather case I made that has tinted the wood somewhat.

The three things that make us great: culture, culture and culture

My last post was in response to George Dinwiddie's "What would you like your software developers to learn" question. Here is a follow-up exchange that I wanted to share.

George wrote back to me asking:

Are your developers learning more how to take a long-term view? What steps are you currently taking to help them learn? What steps do you think you should take, but aren't yet, for some reason?

The rest of this post is my answer.

I think that one of the key roles that manager/architect types need to play is that of strategist. As one of my senior developers put it recently: "I know about tactics, what I need is strategy." An ability to predict the outcome of various actions is, in my experience, an indispensable quality in those who are responsible for guiding the overall direction of a development effort.

Some developers care about learning how to do this and some don't. Not everybody on the team needs to be a superstar, and there is usually no shortage of trash to be taken out by those who want to do that (or who lack the chops to do much else). On the other hand, those developers that do want to grow should be encouraged and given the appropriate tools.

On my team, I have a pretty heterogenous group where about 30% of them are interested in learning this stuff. They come from a variety of backgrounds and so some are starting from further back than others. I find that a good strategy for winnowing the grain from the chaff is to offer some of the basic tenets to everybody and see who picks up on it.

We have a 1.5 hour workshop once a week where we go over various parts of our codebase, learn about new techniques, have different developers do presentations on code they have written or had to maintain, etc. Generally I follow up on those workshops with additional attention to those whose eyes do not glaze over when we start talking about certain things.

We also have a very strong team culture that encourages developers to watch each others' backs and make sure that the "social contract" we have all agreed to is upheld. Most of the questions that I listed in my initial answer are things that my guys and gals check each other on all the time. It's a bit of a game: who can find the mistake in the other person's work. This, of course, is all done in a spirit of cooperation and mutual acceptance of constructive criticism (critical components in any team).

I am fortunate to have the leeway to do pretty much whatever I want to improve the current performance of my team. In that respect, I rarely find that there are things I want to do but cannot. This is not the same in every environment, of course, but I select the companies I work for specifically keeping in mind the fact that the most successful companies are the ones that pick the right people, put them in the right place, and get out of the way while they do their job.

At the end of the day I would say that the single most important factor in turning competent developers into excellent ones is culture. I try to lead by example and I expect my seniors to do the same. Every senior on my team had the role and responsibilities long before they had the title. Every junior on my team is capable of training up new juniors (and even some more intermediate-level developers) to be effective within two iterations (four weeks). The culture on our team is one of constant improvement and the juniors frequently teach the seniors lessons (which the seniors take in stride). With the right culture, everything else just seems logical. Without it, you are constantly swimming against the current.

From competence to excellence: what developers should know

One of the things that most developers have a lot of trouble with is a long-term view. I think it is critical that there be a very good understanding of the impact of current decisions and design choices on the future of the project and the company. A lot of developers (and in my experience this is only worse with contractors and consultants) have the impression that once the code is written, the job is done. They frequently omit details such as sustainability of the overall system and how easy it will be to make future changes.

One good indicator of a developer's current level of understanding of this concept is to look at the following:

Coding Practices

Does the developer write idiomatic code that would be easy for any other developer to understand given a basic comprehension of the development environment? Are the interfaces to their objects clean, minimal and easily understood? Are the techniques used to modularize their code sparingly and appropriately applied? Do they consistently strive for high coherence and low coupling in the codebase? Do they understand the value of a properly written test suite?

Source Management

Does the developer understand more than the basics of their source control tool? Do they know how to do branching and merging properly? Do they leave clear, concise comments for each commit? Common errors when using SCM tools include incorrect spelling for changeset names (which makes searching for a particular changeset difficult), bundling of unrelated changes into a single changeset (which makes reverting only one of the changes difficult/impossible), linking changesets to the incorrect development task (which makes it difficult to assert that a particular task has been completed successfully) and fumbling with branch and merge (which causes problems when a developer needs to work on some separate task - how do I fork the codebase? how do I re-integrate my changes? etc.)

Development Environment

Is the development environment properly documented and easy to re-build? Can each developer work on any other developer's machine or do they need their "own setup"? Does the development style employed by the developer depend on a specific tool that may not be supported in the future (i.e. web developers who do everything in DreamWeaver, Java developers who are lost without Eclipse's auto-complete and refactoring tools, etc.)? Is the project neatly organized such that it's physical layout (on the filesystem) is easy to understand/explain?

Developers who have a good grasp of the above concepts also usually have a more complete understanding of the lifecycle of a software product (as a product and not as a project). By looking ahead, and by applying certain principles now, we can often have a huge impact on the future maintainability of a system. Maybe I'm just lucky to work with guys and gals that have the basics down pat but I would say that the above is what separates the "software developers" from the "coders". Obviously, there are a number of more basic items that come first but a long-term view of the evolution of a codebase is something that lifts developers from competence to excellence.

Asterisk moved into EC2 cloud

I moved the Asterisk server into the EC2 cloud today. It was pretty quick.

Running it off my home box was no longer an option since we have people testing the service these days and I want to make sure there are no outages. I still need to setup a test server so we can test out new configurations before putting them live. I had previously hacked up a little configuration script that shuffles stuff around and performs a couple of emerges. I used the "eminent" AMI that is based on a simple Gentoo install. I plan on putting together my own clean base image at some point in the future but right now this one is good enough. I also plan on building a nice overlay that I can use with portage to update my boxen quickly and efficiently.

FWIW, I figured out how to get Asterisk working from behind a NAT. It was as simple as setting the externip variable in the sip.conf file.

Using encrypted SSL keys and certs in twill

Recent versions of mechanize support SSL connections and so, by extension, does twill. Recently I have been trying to open an HTTPS connection to a site that requires both a certificate and a client key. To try this at home, you will need a version of openssl.

The key and cert came to me packaged in a PKCS#12 format file. The first thing I needed to do was unpack the key and the cert like so:

openssl pkcs12 -clcerts -nokeys -in cert.p12 -out cert.pem
openssl pkcs12 -nocerts -in cert.p12 -out key.pem

After doing this, I plugged in the filenames of the cert and key in my twill code. Note that you need to access the underlying mechanize.Browser object when setting the client certificate:

import twill
from twill.commands import *

host = 'mysecurehost.com:443'
b = twill.get_browser()
b._browser.add_client_certificate(host, 'key.pem', 'cert.pem')

go('https://mysecurehost.com/secured_url.html')
show()

The problem you run into when doing this is that the SSL libraries prompt the user to enter the password for encrypted keys at runtime. This makes automating the interaction tricky at best. I tried fumbling with the PyOpenSSL library but it seems that setting a callback for the passphrase retrieval does not actually work. The set method call returns but the callback is never called during key decryption.

My (hackish) solution was to remove the encryption on the key before using it to connect. You can do this by re-exporting the key from the PKCS#12 file:

openssl pkcs12 -nodes -nocerts -in cert.p12 -out unencrypted_key.pem

Now if you use unencrypted_key.pem in the twill code above, you will be able to connect without providing a password for the key.