RssGeni

Kyle Brandt, a system administrator, asks Should Developers have Access to Production?

A question that comes up again and again in web development companies is:

"Should the developers have access to the production environment, and if they do, to what extent?"

My view on this is that as a whole they should have limited access to production. A little disclaimer before I attempt to justify this view is that this standpoint is in no way based on the perceived quality or attitude of the developers -- so please don't take it this way.

This is a tricky one for me to answer, because, well, I'm a developer. More specifically, I'm one of the developers Kyle is referring to. How do I know that? Because Kyle works for our company, Stack Overflow Internet Services Incorporated©®™. And Kyle is a great system administrator. How do I know that? Two reasons:

1. He's one of the top Server Fault users.
2. He had the audacity to write about this issue on the Server Fault blog.

From my perspective, the whole point of the company is to talk about what we're doing. Getting things done is important, of course, but we have to stop occasionally to write up what we're doing, how we're doing it, and why we're even doing it in the first place -- including all our doubts and misgivings and concerns. If we don't, we're cheating ourselves, and you guys, out of something much deeper. Yes, writing about what we're doing and explaining it to the community helps us focus. It lets our peers give us feedback. But most importantly of all, it lets anyone have the opportunity to learn from our many, many mistakes … and who knows, perhaps even the occasional success.

That's basically the entire philosophy behind our Stack Exchange Q&A network, too. Let's all talk about this stuff in public, so that we can teach each other how to get better at whatever the heck it is we love to do.

(Sometimes I get the feeling this idea makes my co-founder nervous, which I continually struggle to understand. If we don't walk the walk, why are we even doing this? But I digress.)

The saga of System Administrators versus Programmers is not a new one; I don't think I've ever worked at any company where these two factions weren't continually battling with each other in some form. It's truly an epic struggle, but to understand it, you have to appreciate that both System Administrators and Programmers have different, and perhaps complementary, supernatural powers.

Programmers are like vampires. They're frequently up all night, paler than death itself, and generally afraid of being exposed to daylight. Oh yes, and they tend think of themselves (or at least their code) as immortal.

System Administrators are like werewolves. They may look outwardly ordinary, but are incredibly strong, mostly invulnerable to stuff that would kill regular people -- and prone to strange transformations during a moon "outage".

Let me be very clear that just as Kyle respects programmers, I have a deep respect for system administrators:

Although there is certainly some crossover, we believe that the programming community and the IT/sysadmin community are different beasts. Just because you're a hotshot programmer doesn't mean you have mastered networking and server configuration. And I've met a few sysadmins who could script circles around my code. That's why Server Fault gets its own domain, user profiles, and reputation system.

Different "beasts" indeed.

Anyway, if you're looking for a one size fits all answer to the question of how much access programmers should have to production environments, I'm sorry, I can't give you one. Every company is different, every team is different. I know, it's a sucky answer, but it depends.

However, as anyone who has watched the latest season of True Blood (or, God help us all, the Twilight Eclipse movie) can attest, there are ways for vampires and werewolves to work together. In a healthy team, everyone feels their abilities are being used and not squandered.

On our team, we're all fair-to-middling sysadmins. But there are a million things to do, and having a professional sysadmin means we can focus on the programming while the networking, hardware, and operational stuff gets a whole lot more TLC and far better (read: non-hacky) processes put in place. We're happy to refocus our efforts on what we're expert at, and let Kyle put his skills to work in areas that he's expert at. Now, that said, we don't want to cede full access to the production servers -- but there's a happy middle ground where our access becomes infrequent and minor over time, except in the hopefully rare event of an all hands on deck emergency.

The art of managing vampires and werewolves, I think, is to ensure that they spend their time not fighting amongst themselves, but instead, using those supernatural powers together to achieve a common goal they could not otherwise. In my experience, when programmers and system administrators fight, it's because they're bored. You haven't given them a sufficiently daunting task, one that requires the full combined use of their unique skills to achieve.

Remember, it's not vampires versus werewolves. It's vampires and werewolves.

Like any self-respecting geek, I'm mostly an indoor enthusiast.

But on those unfortunate occasions when I am compelled -- for reasons entirely beyond my control -- to leave the house, I do so fully armed with my crucial utility belt items. Yes, you heard me, I transform from the geeky Bruce Wayne to the gosh-darned Batman!

At least, that's how I like to think of it.

I've been talking about this every-day carry stuff for quite a while now. The 2010 edition of my personal utility belt is mostly subtle tweaks, but I daresay it's the best one yet.

The art of every-day carry must go on. What you see here is the contents of my pocket:

1. Patriot 32 GB USB flash drive
   
   Now you can have a whole freakin' hard drive worth of files in your pocket. Just in case, you know, you have an emergency need to upload a virus to an alien mothership, or something. Beware the many cheap, slow USB flash drives out there; this one is a real gem. It's inexpensive, and per my measurements, about as fast as they get. This is important because the larger the flash drive, the more important speed becomes. Hard to believe I've gone from carrying a 512 megabyte flash drive in 2005 to a 32 gigabyte flashdrive in 2010.
   
   
2. Leatherman Squirt P4
   
   Ounce for ounce, nothing beats the utility of the Leatherman Squirt. This time I opted for the plier (P) version instead of the scissors (S), and after seeing how much more generally useful the pliers are, I am now a little ashamed to admit I ever carried the wussy scissors version. Pliers all the way, baby. And yes, that is a Pulp Fiction joke you see on it.
   
   
3. Fenix mini AAA LED flashlight model LD01R2
   
   Since I've been carrying them in 2005, the average LED flashlight has gone from bright, to very bright, to amazingly bright, to ridiculously blinding laser-like bright. It's scary how bright these fancy milled aluminum AAA LED flashlights get now. What I like about this one is that it lets you trade off stupid-brightness for something practical, like greater runtime: you can twist the top to switch levels: 9 Lumens for 11 hours, 28 Lumens for 3.5 hours, or 85 Lumens for 1 hour.
   
   
4. Small Nite Ize s-biner
   
   These little nite-ize carabiners are awesome for quick attachment and detachment of your EDC items, but I'll warn you: resist the urge to put everything you carry on a carabiner, because if you do, the weight and "jangliness" goes up a lot -- and this way lies madness. Consider how many items you actually remove from your keychain regularly. For me, the only item I frequently removed to work with was the Squirt, so that's the only one I put on a carabiner.

Rest assured, everything here is carefully selected with the appropriate levels of monomaniacal attention to detail. For this weight and size, I don't think you can do better. (And don't think I've forgotten about optimizing my wallet, either. Oh no. Quite the contrary.)

However, I have to add a special category this year for the other must-have EDC utility belt item: the smartphone. What self-respecting superhero would leave the house these days without their smartphone? I'm not religious about it, but I use and rather like the iPhone 4, and I'm continually amazed how many things it does that I used to carry separate items for:

• cell phone (obviously)
• "Nintendo DSwhatever" for portable gaming
• GPS
• point and shoot digital camera
• near-desktop quality mobile web browser and email client
• mp3 player with speakers
• audio and hi-def video recorder
• DVD player
• ebook reader
• watch, alarm
• emergency flashlight (via front facing LED flash control)
• scanner
• level and ruler

Smartphones really are the ultimate gadget. The list of functions is already enormous, and I'm sure I'm leaving out a few other things that you can do with a modern smartphone.

In a pinch, I could conceivably drop the AAA LED flashlight and the USB flash drive from my EDC kit and substitute the smartphone. Not exactly, mind you, but it's getting closer every year. At this rate, Apple could introduce a flip-out blade on the iPhone 7 and reduce my entire EDC kit to one item.

Anyway, that's what's on my utility belt in 2010. What's on yours?

On a recent airplane flight, I happened to catch the movie Groundhog Day. Again.

If you aren't familiar with this classic film, the premise is simple: Bill Murray, somehow, gets stuck reliving the same day over and over.

It's been at least 5 years since I've seen Groundhog Day. I don't know if it's my advanced age, or what, but it really struck me on this particular viewing: this is no comedy. There's a veneer of broad comedy, yes, but lurking just under that veneer is a deep, dark existential conundrum.

It might be amusing to relive the same day a few times, maybe even a few dozen times. But an entire year of the same day -- an entire decade of the same day -- everything happening in precisely, exactly the same way? My back of the envelope calculation easily ran to a decade. But I was wrong. The director, Harold Ramis thinks it was actually 30 or 40 years.

I think the 10-year estimate is too short. It takes at least 10 years to get good at anything, and alloting for the down time and misguided years [Phil] spent, it had to be more like 30 or 40 years [spent reliving the same day].

We only see bits and pieces of the full experience in the movie, but this time my mind began filling in the gaps. Repeating the same day for decades plays to our secret collective fear that our lives are irrelevant and ultimately pointless. None of our actions -- even suicide, in endless grisly permutations -- ever change anything. What's the point? Why bother? How many of us are trapped in here, and how can we escape?

This is some dark, scary stuff when you really think about it.

You want a prediction about the weather, you're asking the wrong Phil.

I'll give you a winter prediction.
It's gonna be cold,
it's gonna be gray,
and it's gonna last you for the rest of your life.

Comedy, my ass. I wanted to cry.

But there is a way out: redemption through repetition. If you have to watch Groundhog Day a few times to appreciate it, you're not alone. Indeed, that seems to be the whole point. Just ask Roger Ebert:

"Groundhog Day" is a film that finds its note and purpose so precisely that its genius may not be immediately noticeable. It unfolds so inevitably, is so entertaining, so apparently effortless, that you have to stand back and slap yourself before you see how good it really is.

Certainly I underrated it in my original review; I enjoyed it so easily that I was seduced into cheerful moderation. But there are a few films, and this is one of them, that burrow into our memories and become reference points. When you find yourself needing the phrase This is like "Groundhog Day" to explain how you feel, a movie has accomplished something.

There's something delightfully Ouroboros about the epiphanies and layered revelations in repeated viewings of a movie that is itself about (nearly) endless repetition.

Which, naturally, brings me to A/B testing. That's what Phil spends most of those thirty years doing. He spends it pursuing a woman, technically, but it's how he does it that is interesting:

Rita: This whole day has just been one long setup.

Phil: It hasn't.

Rita: And I hate fudge!

Phil: [making a mental list] No white chocolate. No fudge.

Rita: What are you doing? Are you making some kind of list? Did you call my friends and ask what I like and what I don't like? Is this what love is for you?

Phil: This is real. This is love.

Rita: Stop saying that! You must be crazy.

Phil doesn't just go on one date with Rita, he goes on thousands of dates. During each date, he makes note of what she likes and responds to, and drops everything she doesn't. At the end he arrives at -- quite literally -- the perfect date. Everything that happens is the most ideal, most desirable version of all possible outcomes on that date on that particular day. Such are the luxuries afforded to a man repeating the same day forever.

This is the purest form of A/B testing imaginable. Given two choices, pick the one that "wins", and keep repeating this ad infinitum until you arrive at the ultimate, most scientifically desirable choice. Your marketing weasels would probably collapse in an ecstatic, religious fervor if they could achieve anything even remotely close to the level of perfect A/B testing depicted in Groundhog Day.

But at the end of this perfect date, something impossible happens: Rita rejects Phil.

Phil wasn't making these choices because he honestly believed in them. He was making these choices because he wanted a specific outcome -- winning over Rita -- and the experimental data told him which path he should take. Although the date was technically perfect, it didn't ring true to Rita, and that made all the difference.

That's the problem with A/B testing. It's empty. It has no feeling, no empathy, and at worst, it's dishonest. As my friend Nathan Bowers said:

A/B testing is like sandpaper. You can use it to smooth out details, but you can't actually create anything with it.

The next time you reach for A/B testing tools, remember what happened to Phil. You can achieve a shallow local maximum with A/B testing -- but you'll never win hearts and minds. If you, or anyone on your team, is still having trouble figuring that out, well, the solution is simple.

Just watch Groundhog Day again.

Remember that Scene in Star Trek IV where Scotty tried to use a Mac Plus?

Using a mouse or keyboard to control a computer? Don't be silly. In the future, clearly there's only one way computers will be controlled: by speaking to them.

There's only one teeny-tiny problem with this magical future world of computers we control with our voices.

It doesn't work.

Despite ridiculous, order of magnitude increases in computing power over the last decade, we can't figure out how to get speech recognition accuracy above 80% -- when the baseline human voice transcription accuracy rate is anywhere from 96% to 98%!

In 2001 recognition accuracy topped out at 80%, far short of HAL-like levels of comprehension. Adding data or computing power made no difference. Researchers at Carnegie Mellon University checked again in 2006 and found the situation unchanged. With human discrimination as high as 98%, the unclosed gap left little basis for conversation. But sticking to a few topics, like numbers, helped. Saying “one” into the phone works about as well as pressing a button, approaching 100% accuracy. But loosen the vocabulary constraint and recognition begins to drift, turning to vertigo in the wide-open vastness of linguistic space.

As Robert Fortner explained in Rest in Peas: The Unrecognized Death of Speech Recognition, after all these years, we're desperately far away from any sort of universal speech recognition that's useful or practical.

Now, we do have to clarify that we're talking about universal recognition: saying anything to a computer, and having it reliably convert that into a valid, accurate text representation. When you constrain the voice input to a more limited vocabulary -- say, just numbers, or only the names that happen to be in your telephone's address book -- it's not unreasonable to expect a high level of accuracy. I tend to think of this as "voice control" rather than "voice recognition".

Still, I think we're avoiding the real question: is voice control, even hypothetically perfect voice control, more effective than the lower tech alternatives? In my experience, speech is one of the least effective, inefficient forms of communicating with other human beings. By that, I mean ...

• typical spoken communication tends to be off-the-cuff and ad-hoc. Unless you're extremely disciplined, on average you will be unclear, rambling, and excessively verbose.
• people tend to hear about half of what you say at any given time. If you're lucky.
• spoken communication puts a highly disproportionate burden on the listener. Compare the time it takes to process a voicemail versus the time it takes to read an email.

I am by no means against talking with my fellow human beings. I have a very deep respect for those rare few who are great communicators in the challenging medium of conversational speech. Though we've all been trained literally from birth how to use our voices to communicate, voice communication remains filled with pitfalls and misunderstandings. Even in the best of conditions.

So why in the world -- outside of a disability -- would I want to extend the creaky, rickety old bridge of voice communication to controlling my computer? Isn't there a better way?

Robert's post contains some examples in the comments from voice control enthusiasts:

in addition to extremely accurate voice dictation, there are those really cool commands, like being able to say something like "search Google for Balloon Boy" or something like that and having it automatically open up your browser and enter the search term -- something like this is accomplished many times faster than a human could do it. Or, being able to total up a column of numbers in Microsoft Excel by saying simply "total this column" and seeing the results in a blink of an eye, literally.

That's funny, because I just fired up the Google app on my iPhone, said "balloon boy" into it, and got .. a search for "blue boy". I am not making this up. As for the Excel example, total which column? Let's assume you've dealt with the tricky problem of selecting what column you're talking about with only your voice. (I'm sorry, was it D5? B5?) Wouldn't it be many times faster to click the toolbar icon with your mouse, or press the keyboard command equivalent, to sum the column -- rather than methodically and tediously saying the words "sum this column" out loud?

I'm also trying to imagine a room full of people controlling their computers or phones using their voices. It's difficult enough to get work done in today's chatty work environments without the added burden of a floor full of people saying "zoom ... enhance" to their computers all day long. Wouldn't we all end up hoarse and deaf?

Let's look at another practical example -- YouTube's automatic speech recognition feature. I clicked through to the first UC Berkeley video with this feature, clicked the CC (closed caption) icon, and immediately got .. this.

"Light exerts force on matter". But according to Google's automatic speech recognition, it's "like the search for some matter". Unsurprisingly, it does not get better from there. You'd be way more confused than educated if you had to learn this lecture from the automatic transcription.

Back when Joel Spolsky and I had a podcast together, a helpful listener suggested using speech recognition to get a basic podcast transcript going. Everything I knew about voice recognition told me this wouldn't help, but harm. What's worse: transcribing everything by hand, from scratch -- or correcting every third or fourth word in an auto-generated machine transcript? Maybe it's just me, but the friction of the huge error rate inherent in the machine transcript seems far more intimidating than a blank slate human transcription. The humans may not be particularly efficient, but they all add value along the way -- collective human judgment can editorially improve the transcript, by removing all the duplication, repetition, and "ums" of a literal, by-the-book transcription.

In 2004, Mike Bliss composed a poem about voice recognition. He then read it to voice recognition software on his PC, and rewrote it as recognized.

The real punchline here is that Mike re-ran the experiment in 2008, and after 5 minutes of voice training, the voice recognition got all but 2 words of the original poem correct!

I suspect that's still not good enough in the face of the existing simpler alternatives. Remember handwriting recognition? It was all the rage in the era of the Apple Newton.

It wasn't as bad as Doonesbury made it out to be. I learned Palm's Graffiti handwriting recognition language and got fairly proficient with it. More than ten years later, you'd expect to see massively improved handwriting recognition of some sort in today's iPads and iPhones and iOthers, right? Well, maybe, if by "massively improved" you mean "nonexistent".

While it still surely has its niche uses, I personally don't miss handwriting recognition. Not even a little. And I can't help wondering if voice recognition will go the same way.

After we created Stack Overflow, some people were convinced we had built a marginally better mousetrap for asking and answering questions. The inevitable speculation began: can we use your engine to build a Q&A site about {topic}? Our answer was Stack Exchange. Pay us $129 a month (and up), and you too can create a hosted Q&A community on our engine -- for whatever topic you like!

Well, I have a confession to make: my heart was never in Stack Exchange. It was a parallel effort in a parallel universe only tangentially related to my own. There's a whole host of reasons why, but if I had to summarize it in a sentence, I'd say that money is poisonous to communities. That $129/month doesn't sound like much -- and it isn't -- but the commercial nature of the enterprise permeated and distorted everything from the get-go.

(fortunately, the model is changing with Stack Exchange 2.0, but that's a topic for another blog post.)

Yes, Stack Overflow Internet Services Incorporated©®™ is technically a business, even a venture capital backed business now -- but I didn't co-found it because I wanted to make money. I co-founded it because I wanted to build something cool that made the internet better. Yes, selfishly for myself, of course, but also in conjunction with all of my fellow programmers, because I know none of us is as dumb as all of us.

Nobody is participating in Stack Overflow to make money. We're participating in Stack Overflow because …

• We love programming
• We want to leave breadcrumb trails for other programmers to follow so they can avoid making the same dumb mistakes we did
• Teaching peers is one of the best ways to develop mastery
• We can follow our own interests wherever they lead
• We want to collectively build something great for the community with our tiny slices of effort

I don't care how much you pay me, you'll never be able to recreate the incredibly satisfying feeling I get when demonstrating mastery within my community of peers. That's what we do on Stack Overflow: have fun, while making the internet one infinitesimally tiny bit better every day.

So is it any wonder that some claim Stack Overflow is more satisfying than their real jobs? Not to me.

If this all seems like a bunch of communist hippie bullcrap to you, I understand. It's hard to explain. But there is quite a bit of science documenting these strange motivations. Let's start with Dan Pink's 2009 TED talk.

Dan's talk centers on the candle problem. Given the following three items …

1. A candle
2. A box of thumbtacks
3. A book of matches

… how can you attach the candle to the wall?

It's not a very interesting problem on its own -- that is, until you try to incentivize teams to solve it:

Now I want to tell you about an experiment using the candle problem by a scientist from Princeton named Sam Glucksberg. Here's what he did.

To the first group, he said, "I'm going to time you to establish norms, averages for how long it typically takes someone to solve this sort of problem."

To the second group, he said, "If you're in the top 25 percent of the fastest times you get five dollars. If you're the fastest of everyone we're testing here today you get 20 dollars." (This was many years ago. Adjusted for inflation, it's a decent sum of money for a few minutes of work.)

Question: How much faster did this group solve the problem?

Answer: It took them, on average, three and a half minutes longer. Three and a half minutes longer. Now this makes no sense, right? I mean, I'm an American. I believe in free markets. That's not how it's supposed to work. If you want people to perform better, you reward them. Give them bonuses, commissions, their own reality show. Incentivize them. That's how business works. But that's not happening here. You've got a monetary incentive designed to sharpen thinking and accelerate creativity -- and it does just the opposite. It dulls thinking and blocks creativity.

It turns out that traditional carrot-and-stick incentives are only useful for repetitive, mechanical tasks. The minute you have to do anything even slightly complex that requires even a little problem solving without a clear solution or rules -- those incentives not only don't work, they make things worse!

Pink eventually wrote a book about this, Drive: The Surprising Truth About What Motivates Us.

There's no need to read the book; this clever ten minute whiteboard animation will walk you through the main points. If you view only one video today, view this one.

The concept of intrinsic motivation may not be a new one, but I find that very few companies are brave enough to actually implement them.

I've tried mightily to live up to the ideals that Stack Overflow was founded on when building out my team. I don't care when you come to work or what your schedule is. I don't care where in the world you live (provided you have a great internet connection). I don't care how you do the work. I'm not going to micromanage you and assign you a queue of task items. There's no need.

If you want to build a ship, don't drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea.
– Antoine de Saint Exupéry

Because I know you yearn for the vast and endless sea, just like we do.

When I first chose my own adventure, I didn't know what working remotely from home was going to be like. I had never done it before. As programmers go, I'm fairly social. Which still means I'm a borderline sociopath by normal standards. All the same, I was worried that I'd go stir-crazy with no division between my work life and my home life.

Well, I haven't gone stir-crazy yet. I think. But in building Stack Overflow, I have learned a few things about what it means to work remotely -- at least when it comes to programming. Our current team encompasses 5 people, distributed all over the USA, along with the team in NYC.

My first mistake was attempting to program alone. I had weekly calls with my business partner, Joel Spolsky, which were quite productive in terms of figuring out what it was we were trying to do together -- but he wasn't writing code. I was coding alone. Really alone. One guy working all by yourself alone. This didn't work at all for me. I was unmoored, directionless, suffering from analysis paralysis, and barely able to get motivated enough to write even a few lines of code. I rapidly realized that I'd made a huge mistake in not having a coding buddy to work with.

That situation rectified itself soon enough, as I was fortunate enough to find one of my favorite old coding buddies was available. Even though Jarrod was in North Carolina and I was in California, the shared source code was the mutual glue that stuck us together, motivated us, and kept us moving forward. To be fair, we also had the considerable advantage of prior history, because we had worked together at a previous job. But the minimum bar to work remotely is to find someone who loves code as much as you do. It's … enough. Anything else on top of that -- old friendships, new friendships, a good working relationship -- is icing that makes working together all the sweeter. I eventually expanded the team in the same way by adding another old coding buddy, Geoff, who lives in Oregon. And again by adding Kevin, who I didn't know, but had built amazing stuff for us without even being asked to, from Texas. And again by adding Robert, in Florida, who I also didn't know, but spent so much time on every single part of our sites that I felt he had been running alongside our team the whole way, there all along.

The reason remote development worked for us, in retrospect, wasn't just shared love of code. I picked developers who I knew -- I had incontrovertible proof -- were amazing programmers. I'm not saying they're perfect, far from it, merely that they were top programmers by any metric you'd care to measure. That's why they were able to work remotely. Newbie programmers, or competent programmers who are phoning it in, are absolutely not going to have the moxie necessary to get things done remotely -- at least, not without a pointy haired manager, or grumpy old team lead, breathing down their neck. Don't even think about working remotely with anyone who doesn't freakin' bleed ones and zeros, and has a proven track record of getting things done.

While Joel certainly had a lot of high level input into what Stack Overflow eventually became, I only talked to him once a week, at best (these calls were the genesis of our weekly podcast series). I had a strong, clear vision of what I wanted Stack Overflow to be, and how I wanted it to work. Whenever there was a question about functionality or implementation, my team was able to rally around me and collectively make decisions we liked, and that I personally felt were in tune with this vision. And if you know me at all, you know I'm not shy about saying no, either. We were able to build exactly what we wanted, exactly how we wanted.

Bottom line, we were on a mission from God. And we still are.

So, there are a few basic ground rules for remote development, at least as I've seen it work:

• The minimum remote team size is two. Always have a buddy, even if your buddy is on another continent halfway across the world.
• Only grizzled veterans who absolutely love to code need apply for remote development positions. Mentoring of newbies or casual programmers simply doesn't work at all remotely.
• To be effective, remote teams need full autonomy and a leader (PM, if you will) who has a strong vision and the power to fully execute on that vision.

This is all well and good when you have a remote team size of three, as we did for the bulk of Stack Overflow development. And all in the same country. Now we need to grow the company, and I'd like to grow it in distributed fashion, by hiring other amazing developers from around the world, many of whom I have met through Stack Overflow itself.

But how do you scale remote development? Joel had some deep seated concerns about this, so I tapped one of my heroes, Miguel de Icaza -- who I'm proud to note is on our all-star board of advisors -- and he was generous enough to give us some personal advice based on his experience running the Mono project, which has dozens of developers distributed all over the world.

At the risk of summarizing mercilessly (and perhaps too much), I'll boil down Miguel's advice the best I can. There are three tools you'll need in place if you plan to grow a large-ish and still functional remote team:

1. Real time chat
   
   When your team member lives in Brazil, you can't exactly walk by his desk to ask him a quick question, or bug him about something in his recent checkin. Nope. You need a way to casually ping your fellow remote team members and get a response back quickly. This should be low friction and available to all remote developers at all times. IM, IRC, some web based tool, laser beams, smoke signals, carrier pigeon, two tin cans and a string: whatever. As long as everyone really uses it.
   

   We're currently experimenting with Campfire, but whatever floats your boat and you can get your team to consistently use, will work. Chat is the most essential and omnipresent form of communication you have when working remotely, so you need to make absolutely sure it's functioning before going any further.

2. Persistent mailing list
   
   Sure, your remote team may know the details of their project, but what about all the other work going on? How do they find out about that stuff or even know it exists in the first place? You need a virtual bulletin board: a place for announcements, weekly team reports, and meeting summaries. This is where a classic old-school mailing list comes in handy.

   We're using Google Groups and although it's old school in spades, it works plenty well for this. You can get the emails as they arrive, or view the archived list via the web interface. One word of caution, however. Every time you see something arrive in your inbox from the mailing list you better believe, in your heart of hearts, that it contains useful information. The minute the mailing list becomes just another "whenever I have time to read that stuff", noise engine, or distraction from work … you've let someone cry wolf too much, and ruined it. So be very careful. Noisy, argumentative, or useless things posted to the mailing list should be punishable by death. Or noogies.

3. Voice and video chat
   

   As much as I love ASCII, sometimes faceless ASCII characters just aren't enough to capture the full intentions and feelings of the human being behind them. When you find yourself sending kilobytes of ASCII back and forth, and still are unsatisfied that you're communicating, you should instill a reflexive habit of "going voice" on your team.

   Never underestimate the power of actually talking to another human being. I know, I know, the whole reason we got into this programming thing was to avoid talking to other people, but bear with me here. You can't be face to face on a remote team without flying 6 plus hours, and who the heck has that kind of time? I've got work I need to get done! Well, the next best thing to hopping on a plane is to fire up Skype and have a little voice chat. Easy peasy. All that human nuance which is totally lost in faceless ASCII characters (yes, even with our old pal *<:-)) will come roaring back if you regularly schedule voice chats. I recommend at least once a week at an absolute minimum; they don't have to be long meetings, but it sure helps in understanding the human being behind all those awesome checkins.
   

Nobody hates meetings and process claptrap more than I do, but there is a certain amount of process you'll need to keep a bunch of loosely connected remote teams and developers in sync.

1. Monday team status reports
   

   Every Monday, as in somebody's-got-a-case-of-the, each team should produce a brief, summarized rundown of:

   • What we did last week
   • What we're planning to do this week
   • Anything that is blocking us or we are concerned about

   This doesn't have to be (and in fact shouldn't be) a long report. The briefer the better, but do try to capture all the useful highlights. Mail this to the mailing list every Monday like clockwork. Now, how many "teams" you have is up to you; I don't think this needs to be done at the individual developer level, but you could.

2. Meeting minutes
   

   Any time you conduct what you would consider to be a "meeting" with someone else, take minutes! That is, write down what happened in bullet point form, so those remote team members who couldn't be there can benefit from -- or at least hear about -- whatever happened.

   Again, this doesn't have to be long, and if you find taking meeting minutes onerous then you're probably doing it wrong. A simple bulleted list of sentences should suffice. We don't need to know every little detail, just the big picture stuff: who was there? What topics were discussed? What decisions were made? What are the next steps?

Both of the above should, of course, be mailed out to the mailing list as they are completed so everyone can be notified. You do have a mailing list, right? Of course you do!

If this seems like a lot of jibba-jabba, well, that's because remote development is hard. It takes discipline to make it all work, certainly more discipline than piling a bunch of programmers into the same cubicle farm. But when you imagine what this kind of intellectual work -- not just programming, but anything where you're working in mostly thought-stuff -- will be like in ten, twenty, even thirty years … don't you think it will look a lot like what happens every day right now on Stack Overflow? That is, a programmer in Brazil helping a programmer in New Jersey solve a problem?

If I have learned anything from Stack Overflow it is that the world of programming is truly global. I am honored to meet these brilliant programmers from every corner of the world, even if only in a small way through a website. Nothing is more exciting for me than the prospect of adding international members to the Stack Overflow team. The development of Stack Overflow should be reflective of what Stack Overflow is: an international effort of like-minded -- and dare I say totally awesome -- programmers. I wish I could hire each and every one of you. OK, maybe I'm a little biased. But to me, that's how awesome the Stack Overflow community is.

I believe remote development represents the future of work. If we have to spend a little time figuring out how this stuff works, and maybe even make some mistakes along the way, it's worth it. As far as I'm concerned, the future is now. Why wait?

We're currently in the midst of a CSS Zen Garden type excerise on our family of Q&A websites, which I affectionately refer to as "the Trilogy":

• Server Fault
• Super User
• Stack Overflow
• Meta Stack Overflow

(In case you were wondering, yes, meta is the Star Wars Holiday Special.)

These sites all run the same core engine, but the logo, domain, and CSS "skin" that lies over the HTML skeleton is different in each case:

They are not terribly different looking, it's true, but we also want them to be recognizable as a family of sites.

We're working with two amazing designers, Jin Yang and Nathan Bowers, who are helping us whip the CSS and HTML into shape so they can produce a set of about 10 different Zen Garden designs. As new sites in our network get democracied into being, these designs will be used as a palette for the community to choose from. (And, later, the community will decide on a domain name and logo as well.)

Anyway, I bring this up not because my pokemans, let me show you them, but because I have to personally maintain four different CSS files. And that number is only going to get larger. Much larger. That scares me a little.

Most of all, what I've learned from this exercise in site theming is that CSS is kind of painful. I fully support CSS as a (mostly) functional user interface Model-View-Controller. But even if you have extreme HTML hygiene and Austrian levels of discipline, CSS has some serious limitations in practice.

Things in particular that bite us a lot:

• Vertical alignment is a giant, hacky PITA. (Tables work great for this though!)
• Lack of variables so we have to repeat colors all over the place.
• Lack of nesting so we have to repeat huge blocks of CSS all over the place.

In short, CSS violates the living crap out of the DRY principle. You are constantly and unavoidably repeating yourself.

That's why I'm so intrigued by two Ruby gems that attempt to directly address the deficiencies of CSS.

1. Less CSS

/* CSS */

#header {
  -moz-border-radius: 5;
  -webkit-border-radius: 5;
  border-radius: 5;
}

#footer {
  -moz-border-radius: 10;
  -webkit-border-radius: 10;
  border-radius: 10;
}

// LessCSS

.rounded_corners (@radius: 5px) {
  -moz-border-radius: @radius;
  -webkit-border-radius: @radius;
  border-radius: @radius;
}

#header {
  .rounded_corners;
}

#footer {
  .rounded_corners(10px);
}

2. SASS

/* CSS */

.content_navigation {
  border-color: #3bbfce;
  color: #2aaebd;
}

.border {
  padding: 8px;
  margin: 8px;
  border-color: #3bbfce;
}

// Sass

!blue = #3bbfce
!margin = 16px

.content_navigation
  border-color = !blue
  color = !blue - #111

.border
  padding = !margin / 2
  margin = !margin / 2
  border-color = !blue

As you can see, in both cases we're transmogrifying CSS into a bit more of a programming language, rather than the static set of layout rules it currently exists as. Behind the scenes, we're generating plain vanilla CSS using these little dynamic languages. This could be done at project build time, or even dynamically on every page load if you have a good caching strategy.

I'm not sure how many of these improvements CSS3 will bring, never mind when the bulk of browsers in the world will support it. But I definitely feel that the core changes identified in both Less CSS and SASS address very real pain points in practical CSS use. It's worth checking them out to understand why they exist, what they bring to the table, and how you could possibly adopt some of these strategies in your own CSS and your favorite programming language.

I have what I would charitably describe as a hate-hate relationship with email. I desperately try to avoid sending email, not just for myself, but also in the code I write.

Despite my misgivings, email is the cockroach of communication mediums: you just can't kill it. Email is the one method of online contact that almost everyone -- at least for that subset of "everyone" which includes people who can bear to touch a computer at all -- is guaranteed to have, and use. Yes, you can make a fairly compelling case that email is for old stupid people, but let's table that discussion for now.

So, reluctantly, we come to the issue of sending email through code. It's easy! Let's send some email through oh, I don't know, let's say ... Ruby, courtesy of some sample code I found while browsing the Ruby tag on Stack Overflow.

require 'net/smtp'

def send_email(to, subject = "", body = "")
    from = "my@email.com"
    body= "From: #{from}\r\nTo: #{to}\r\nSubject: #{subject}\r\n\r\n#{body}\r\n"

    Net::SMTP.start('192.168.10.213', 25, '192.168.0.218') do |smtp|
        smtp.send_message body, from, to
    end
end

send_email "foo@example.com", "title", "body goes here"

There's a bug in this code, though. Do you see it?

Just because you send an email doesn't mean it will arrive. Not by a long shot. Bear in mind this is email we're talking about. It was never designed to survive a bitter onslaught of criminals and spam, not to mention the explosive, exponential growth it has seen over the last twenty years. Email is a well that has been truly and thoroughly poisoned -- the digital equivalent of a superfund cleanup site. The ecosystem around email is a dank miasma of half-implemented, incompletely supported anti-spam hacks and workarounds.

Which means the odds of that random email your code just sent getting to its specific destination is .. spotty. At best.

If you want email your code sends to actually arrive in someone's AOL mailbox, to the dulcet tones of "You've Got Mail!", there are a few things you must do first. And most of them are only peripherally related to writing code.

1. Make sure the computer sending the email has a Reverse PTR record

What's a reverse PTR record? It's something your ISP has to configure for you -- a way of verifying that the email you send from a particular IP address actually belongs to the domain it is purportedly from.

Not every IP address has a corresponding PTR record. In fact, if you took a random sampling of addresses your firewall blocked because they were up to no good, you'd probably find most have no PTR record - a dig -x gets you no information. That's also apt to be true for mail spammers, or their PTR doesn't match up: if you do a dig -x on their IP you get a result, but if you look up that result you might not get the same IP you started with.

That's why PTR records have become important. Originally, PTR records were just intended as a convenience, and perhaps as a way to be neat and complete. There still are no requirements that you have a PTR record or that it be accurate, but because of the abuse of the internet by spammers, certain conventions have grown up. For example, you may not be able to send email to some sites if you don't have a valid PTR record, or if your pointer is "generic".

How do you get a PTR record? You might think that this is done by your domain registrar - after all, they point your domain to an IP address. Or you might think whoever handles your DNS would do this. But the PTR record isn't up to them, it's up to the ISP that "owns" the IP block it came from. They are the ones who need to create the PTR record.

A reverse PTR record is critical. How critical? Don't even bother reading any further until you've verified that your ISP has correctly configured the reverse PTR record for the server that will be sending email. It is absolutely the most common check done by mail servers these days. Fail the reverse PTR check, and I guarantee that a huge percentage of the emails you send will end up in the great bit bucket in the sky -- and not in the email inboxes you intended.

2. Configure DomainKeys Identified Mail in your DNS and code

What's DomainKeys Identified Mail? With DKIM, you "sign" every email you send with your private key, a key only you could possibly know. And this can be verified by attempting to decrypt the email using the public key stored in your public DNS records. It's really quite clever!

The first thing you need to do is generate some public-private key pairs (one for every domain you want to send email from) via OpenSSL. I used a win32 version I found. Issue these commands to produce the keys in the below files:

$ openssl genrsa -out rsa.private 1024
$ openssl rsa -in rsa.private -out rsa.public -pubout -outform PEM

These public and private keys are just big ol' Base64 encoded strings, so plop them in your code as configuration string resources that you can retrieve later.

Next, add some DNS records. You'll need two new TXT records.

1. _domainkey.example.com
   "o=~\; r=contact@example.com"
2. selector._domainkey.example.com
   "k=rsa\; p={public-key-base64-string-here}"

The first TXT DNS record is the global DomainKeys policy and contact email.

The second TXT DNS record is the public base64 key you generated earlier, as one giant unbroken string. Note that the "selector" part of this record can be anything you want; it's basically just a disambiguating string.

Almost done. One last thing -- we need to sign our emails before sending them. In any rational world this would be handled by an email library of some kind. We use Mailbee.NET which makes this fairly painless:

smtp.Message = dk.Sign(smtp.Message, 
               null, AppSettings.Email.DomainKeyPrivate, false, "selector");

3. Set up a SenderID record in your DNS

To be honest, SenderID is a bit of a "nice to have" compared to the above two. But if you've gone this far, you might as well go the distance. SenderID, while a little antiquated and kind of.. Microsoft/Hotmail centric.. doesn't take much additional effort.

SenderID isn't complicated. It's another TXT DNS record at the root of, say, example.com, which contains a specially formatted string documenting all the allowed IP addresses that mail can be expected to come from. Here's an example:

"v=spf1 a mx ip4:10.0.0.1 ip4:10.0.0.2 ~all"

You can use the Sender ID SPF Record Wizard to generate one of these for each domain you send email from.

That sucked. How do I know all this junk is working?

I agree, it sucked. Email sucks; what did you expect? I used two methods to verify that all the above was working:

1. Test emails sent to a GMail account.
   
   Use the "show original" menu on the arriving email to see the raw message content as seen by the email server. You want to verify that the headers definitely contain the following:

   Received-SPF: pass
   Authentication-Results: ... spf=pass ... dkim=pass
   

   If you see that, then the Reverse PTR and DKIM signing you set up is working. Google provides excellent diagnostic feedback in their email server headers, so if something isn't working, you can usually discover enough of a hint there to figure out why.

2. Test emails sent to the Port25 email verifier
   
   Port25 offers a really nifty public service -- you can send email to check-auth@verifier.port25.com and it will reply to the from: address with an extensive diagnostic! Here's an example summary result from a test email I just sent to it:

   SPF check:          pass
   DomainKeys check:   fail
   DKIM check:         pass
   Sender-ID check:    pass
   SpamAssassin check: ham
   

   You want to pass SPF, DKIM, and Sender-ID. Don't worry about the DomainKeys failure, as I believe it is spurious -- DKIM is the "newer" version of that same protocol.

Yes, the above three steps are quite a bit of work just to send a lousy email. But I don't send email lightly. By the time I've reached the point where I am forced to write code to send out email, I really, really want those damn emails to arrive. By any means necessary.

And for those who are the unfortunate recipients of these emails: my condolences.

As far as I'm concerned, you can never be too rich, too thin, or have too much screen space. By "screen", I mean not just large monitors, but multiple large monitors. I've been evangelizing multiple monitors since the dark days of Windows Millennium Edition:

• Multiple Monitors and Productivity
• Multiple LCDs
• Joining the Prestigious Three Monitor Club
• The Large Display Paradox
• LCD Monitor Arms

If you're a long time reader you're probably sick of hearing about this stuff by now, but something rather wonderful has happened since I last wrote about it:

If you're only using one monitor, you are cheating yourself out of potential productivity. Two monitors is a no-brainer. It's so fundamental that I included it as a part of the Programmer's Bill of Rights.

But you can do better.

As good as two monitors is, three monitors is even better. With three monitors, there's a "center" to focus on. And 50% more display area. While there's certainly a point of diminishing returns for additional monitors, I think three is the sweet spot. Even Edward Tufte, in the class I recently attended, explicitly mentioned multiple monitors. I don't care how large a single display can be; you can never have enough desktop space.

Normally, to achieve three monitors, you have to either:

1. Buy an exotic video card that has more than 2 monitor connections.
2. Install a second video card.

Fortunately, that is no longer true. I was excited to learn that the latest ATI video cards have gone from two to three video outputs. Which means you can now achieve triple monitors with a single video card upgrade! They call this "eyefinity", but it's really just shorthand for "raising the standard from two display outputs to three".

But, there is a (small) catch. The PC ecosystem is in the middle of shifting display output standards. For evidence of this, you need look no further than the back panel of one of these newfangled triple display capable ATI video cards:

It contains:

• two DVI outputs
• one HDMI output
• one DisplayPort output

I suspect part of this odd connector layout is due to space restrictions (DVI is awfully chunky), but I've always understood DisplayPort to be the new, improved DVI connector for computer monitors, and HDMI to be the new, improved s-video/component connector for televisions. Of course these worlds are blurring, as modern high-definition TVs make surprisingly effective computer monitors, too.

Anyway, since all my monitors have only DVI inputs, I wasn't sure what to do with the other output. So I asked on Super User. The helpful answers led me to discover that, as I suspected, the third output has to be DisplayPort. So to connect my third monitor, I needed to convert DisplayPort to DVI, and there are two ways:

1. a passive, analog DisplayPort to DVI conversion cable for ~$30 that supports up to 1920x1200
2. an active, digital DisplayPort to DVI converter for $110 that supports all resolutions

I ended up going with the active converter, which has mixed reviews, but it's worked well for me over the last few weeks.

Note that this adapter requires USB power, and given the spotty results others have had with it, some theorize that it needs quite a bit of juice to work reliably. I plugged it into my system's nearby rear USB ports which do tend to deliver more power (they're closer to the power supply, and have short cable paths). Now, I have gotten the occasional very momentary black screen with it, but nothing severe enough to be a problem or frequent enough to become a pattern. If you have DisplayPort compatible monitors, of course, this whole conversion conundrum is a complete non-issue. But DisplayPort is fairly new, and even my new-ish LCD monitors don't support it yet.

The cool thing about this upgrade, besides feeding my video card addiction, is that I was able to simplify my hardware configuration. That's always good. I went from two video cards to one, which means less power consumption, simpler system configuration, and fewer overall driver oddities. Basically, it makes triple monitors -- dare I say it -- almost a mainstream desktop configuration. How could I not be excited about that?

I was also hoping that Nvidia would follow ATI's lead here and make three display outputs the standard for all their new video cards, too, but sadly that's not the case. It turns out their new GTX 480 fails in other ways, in that it's basically the Pentium 4 of video cards -- generating ridiculous amounts of heat for very little performance gain. Based on those two facts, I am comfortable endorsing ATI wholeheartedly at this point. But, do be careful, because not all ATI cards support triple display outputs (aka "eyefinity"). These are the ones that I know do:

• Radeon HD 5670 (~$100)
• Radeon HD 5770 (~$150)
• Radeon HD 5830 (~$250)
• Radeon HD 5850 (~$320)
• Radeon HD 5870 (~$450)

Unless you're a gamer, there's no reason to care about anything other than the least expensive model here, which will handily crush any 2D or 3D desktop GUI acceleration needs you might have. As an addict, of course I bought the high end model and it absolutely did not disappoint -- more than doubling my framerates in the excellent game Battlefield: Bad Company 2 over the GTX 280 I had before.

I'm excited that a triple monitor setup is now, thanks to ATI, so easily attainable for desktop users -- as long as you're aware of the DisplayPort caveat I discussed above. I'd encourage anyone who is even remotely interested in the (many) productivity benefits of a triple monitor setup to seriously consider an ATI video card upgrade.

Writing code? That's the easy part. Getting your application in the hands of users, and creating applications that people actually want to use -- now that's the hard stuff.

I've been a long time fan of Krug's book Don't Make Me Think. Not just because it's a quick, easy read (and it is!) -- but because it's the most concise and most approachable book I've ever found to teach the fundamental importance of usability. As far as I'm concerned, if you want to help us make the software industry a saner place, the first step is getting Don't Make Me Think in the hands of as many of your coworkers as you can. If you don't have people that care about usability on your project, your project is doomed.

Beyond getting people over the hurdle of at least paging through the Krug book, and perhaps begrudgingly conceding that this usability stuff matters, the next challenge is figuring out how to integrate usability testing into your project. It's easy to say "Usability is Important!", but you have to walk the walk, too. I touched on some low friction ways to get started in Low-Fi Usability Testing. That rough outline is now available in handy, more complete book form -- Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems.

Don't worry, Krug's book is just as usable as his advice. It's yet another quick, easy read. Take it from the man himself:

• Usability testing is one of the best things people can do to improve Web sites (or almost anything they’re creating that people have to interact with).
• Since most organizations can’t afford to hire someone to do testing for them on a regular basis, everyone should learn to do it themselves. And …
• I could probably write a pretty good book explaining how to do it.

If you're wondering what the beginner's "how do I boil water?" recipe for software project usability is, stop reading this post and get a copy of Rocket Surgery Made Easy. Now.

One of the holy grails of usability testing is eyetracking -- measuring where people's eyes look as they use software and web pages. Yes, there are clever JavaScript tools that can measure where users move their pointers, but that's only a small part of the story. Where the eye wanders, the pointer may not, and vice-versa. But, who has the time and equipment necessary to conduct an actual eyetracking study? Almost nobody.

That's where Eyetracking Web Usability comes in.

Eyetracking Web Usability is chock full of incredibly detailed eyetracking data for dozens of websites. Even though you (probably) can't afford to do real eyetracking, you can certainly use this book as a reference. There is enough variety in UI and data that you can map the results, observations, and explanations found here to what your project is doing.

This particular book is rather eyetracking specific, but it's just the latest entry in a whole series on usability, and I recommend them all highly. These books are a fount of worthwhile data for anyone who works on software and cares about usability, from one of the most preeminent usability experts on the web.

Usability isn't really cheap or easy. It's an endless war, with innumerable battlegrounds, stretching all the way back to the dawn of computing. But these books, at least, are cheap and easy in the sense that they give you some basic training in fighting the good (usability) fight. That's the best I can do, and it's all I'd ask from anyone else I work with.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

We’ve been opening new Stack Exchanges left and right on a variety of topics. In almost every case, the Stack Exchange appears to duplicate the content of an existing community. For example, our WordPress answers site (now in beta) covers the exact same material as WordPress.org’s existing forums.

This is nothing new to us at Stack Overflow, which purported to cover the exact same material as hundreds (if not thousands) of other programming sites. There’s no rule that says that there needs to be exactly one Q&A website per topic.

There is, however, a compelling case for the Stack Exchange technology. WordPress.org’s forums don’t have voting, so you have to read through every answer and decide for yourself which one might solve your problem. They don’t have reputation, so there’s no way to see whether you’re getting an answer from someone who knows what they’re talking about. They don’t have wiki-style editing, so collaboration is impossible. You have to log on to ask or answer a question, so the burden of participation is higher. Stack Overflow is simply better than traditional forums, which is why it largely replaced proprietary forums. I remember hours of discussion with John Resig and the folks at jQuery who couldn’t decide whether to replace the jQuery Google Group with a forum or with a Stack Exchange. Ultimately it didn’t matter that much, because most of the jQuery Q&A activity happens on Stack Overflow anyway.

One day, the features that are standard on Stack Exchange will be copied everywhere. Until then, we’ll keep churning out new sites.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

Sometimes I think a pretty good business model would be to copy the applications that 37signals makes, but make them more complex. More features, more promises—generally, just more complicated.

Here’s the video from a talk I gave at the Business of Software conference last year:

I’ll be speaking again at this year’s conference in Boston, October 4th-6th.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

We launched three new Stack Exchange sites this week!

• Gaming
• Cooking
• Pro Webmasters

We’ll have three more for you next week, too.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

“We decided that individually-branded sites felt more authentic and trustworthy. We thought that letting every Stack Exchange site have its own domain name, visual identity, logo, and brand would help the community feel more coherent. After all, nobody wants to say that they live in Housing Block 2938TC. They want to live in Colonial Manor. Never mind the connotation of, well, colonies.”

Domain Names

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

Want to know how to export mail from Gmail? Or delete your Facebook account? Or send giant files via email?

Well, the new Web Applications Stack Exchange is for you. It’s a part of the Stack Exchange network, so it has the clean, elegant design that made Stack Overflow a phenomenal success.

The newest member of the Stack Exchange Network is the first one to go through the community site-creation process called Area 51. There are more great sites in the pipeline, but they have to demonstrate that they can reach critical mass or we won’t create them.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

Neil has posted a video of Don Norman (most famous for his book The Design of Everyday Things) speaking at the Business of Software conference last year in San Francisco.

“Imagine you’re on the first slide of your powerpoint presentation and want to move to the next slide. Your remote control has two buttons. They are unmarked, but one button points up and one button points down.

“Which button do you press?”

It turns out half the people press up, half the people press down, and everybody thinks their choice is obvious. It’s a great talk.

The early bird discount for the Business of Software 2010 (Boston, October 4-6) saves you $400, but it expires this week, so this is the right time to sign up for my favorite conference.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.