The vigilant’s hat

In the german language, there is a proverb that means “being alert” or “being on guard”. It’s called “auf der Hut sein” and would mean, if translated without context, “being on hat”. That doesn’t make sense, even to germans. But it’s actually directly explainable if you know that the german word “Hut” has two meanings. It most of the time means the hat you put on your head. But another form of it means “shelter”, “protection” or “guard”. It turns up in quite a few derived german words like “Obhut” (custody) or “Vorhut” (vanguard). So it isn’t so strange for germans to think of a hat when they need to stay alert and vigilant.

Vigilant developers

Being mindful and careful is a constant state of mind for every developer. The computer doesn’t accept even the slightest fuzziness of thought. But there is a moment when a developer really has to take care and be very very precautious: When you operate on a live server. These machines are the “real” thing, containing the deployed artifacts of the project and connecting to the real database. If you make an error on this machine, it will be visible. If you accidentally wipe some data, it’s time to put the backup recovery process to the ultimate test. And you really should have that backup! In fact, you should never operate on a live server directly, no matter what.

Learning from Continuous Delivery

One of the many insights in the book “Continuous Delivery” by Jez Humble and David Farley is that you should automate every step that needs to take place on a live server. There is an ever-growing list of tools that will help you with this task, but in its most basic form, you’ll have to script every remote action, test it thoroughly and only then upload it to the live server and execute it. This is the perfect state your deployment should be in. If it isn’t yet, you will probably be forced to work directly on the live server (or the real database) from time to time. And that’s when you need to be “auf der Hut“. And you can now measure your potential for improvement in the deployment process area in “hat time”.

cowboy hats in action

We ain’t no cowboys!

In our company, there is a rule for manual work on live servers: You have to wear a hat. We bought several designated cowboy hats for that task, so there’s no excuse. If you connect to a server that isn’t a throw-away test instance, you need to wear your hat to remind you that you’re responsible now. You are responsible for the herd (the data) and the ranch (the server). You are responsible for every click you make, every command you issue and every change you make. There might be a safety net to prevent lethal damage, but this isn’t a test. You should get it right this time. As long as you wear the “live server hat”, you should focus your attention on the tasks at hand and document every step you make.

Don’t ask, they’ll shoot!

But the hat has another effect that protects the wearer. If you want to ask your collegue something and he’s wearing a cowboy hat, think twice! Is it really important enough to disturb him during the most risky, most stressful times? Do you really need to shout out right now, when somebody concentrates on making no mistake? In broadcasting studios, there is a sign saying “on air”. In our company, there is a hat saying “on server”. And if you witness more and more collegues flocking around a terminal, all wearing cowboy hats and seeming concerned, prepare for a stampede – a problem on a live server, the most urgent type of problem that can arise for developers.

The habit of taking off the hat after a successful deployment is very comforting, too. You physically alter your state back to normal. You switch roles, not just wardrobe.

Why cowboy hats?

We are pretty sure that the same effects can be achieved with every type of hat you can think of. But for us, the cowboy hat combines ironic statement with visual coolness. And there is no better feeling after a long, hard day full of deployments than to gather around the campfire and put the spurs aside.

Booked in February

Ok, the title is a bit misleading – it’s a play of words(*). This entry is actually a book preview on the upcoming book “97 Things Every Programmer Should Know” from O’Reilly.

97 Things Of Wisdom

The “97 Things” series started out with “97 Things Every Software Architect Should Know” early last year. The book essentially is a collection of short articles on specific topics that should bother today’s software architect. You may classify as a software architect if you don’t just stir up source code but are also in charge to give the system a shape.

The articles are straight to the point and can be read within five minutes each. Don’t expect detailed textbook chapters of the topics, but they work extremely well as creative appetizers. And there are nearly a hundred appetizers from well-respected members of the software architect community in this book.

Just imagine you would meet all the authors for five minutes each on a conference and just ask them for an appealing thought. This book serves as the best replacement for it.

Wisdom continued

Soon after the first book, there was a second book in the series, “97 Things Every Project Manager Should Know”. I haven’t read this one yet, but it is on my must-read list for 2010.

And now, next month, there will be another book, this time for the fellow coder: “97 Things Every Programmer Should Know”. As usual, there are 97 selected articles with bits of wisdom from big community names. Kevlin Henney is the editor for this book (we featured him in our last blog harvest). You can take a sneak peek online in the 97TEPSK wiki, where the articles were fostered (and a second part is likely to emerge). But don’t forget to buy a paper copy that you can foist on your peers to inspire them, too.

Telling from the articles I’ve read so far, the book will be great. Please don’t expect detailed language specifics, lengthy code examples or fancy UML diagrams. But expect a whole bunch of great ideas that stem from real experiences of real programmers.

One percent of a book

What’s our relation to the new book? We’ve contributed an article to it! Even if we thereby only wrote approximately one percent of the book, it feels great and we consider ourselves honored.

The topic of our article is Extreme Feedback Devices (XFD): “Let Your Project Speak for Itself”. We gathered quite a lot of these devices over the years and ran a few experiments, so we thought we are qualified to write about it. And there it is, the first bit of our wisdom, printed in a book.

We will, of course, continue to publish our wisdom on this blog first. If you’ve followed us over the last years, the article comes as no real surprise. But I’m sure some other articles of the book will. Go buy it!

(*) Play of words in a language other than your native tongue are always dangerous. I hope this one worked out well.

Smell if it’s well

We at the Softwareschneiderei are constantly searching for ways to gather feedback from our projects. We get feedback from our customers and their users, but we also get feedback directly from the code, be it through test results or code analysis. A great way to make your code speak for itself is to provide it some Extreme Feedback Devices (XFD).

IntroduciIMG_0574_smellng the Smell-O-Mat

One thing we always wanted to have was “code smells” that really smell for themselves. When we ran across an ultrasonic humidifier that can produce room-wide smells by dispensing essential oils, we found the right device for this feedback. We bought two humidifiers and labeled them “good” and “evil”. The hardest part was to find a smell everybody relates to “evil”, but won’t distract you too much from your work. Whenever our code analysis finds a new real code smell, the “evil” humidifier is turned on for some minutes. If an existing code smell is fixed, we get the “good” smell.

The effects

We do not produce code smells all too often. But once in a while, it happens. And this incident can now be perceived throughout the day just by breathing. On the other hand, fixing old smells is a source of refreshing air. Whenever the office atmosphere needs replenishment, all you have to do is to fix some code smells in our large code base (they do get rare!). Of course, most junior developers just open a window for that.

We chose grapefruit being our “good” smell, so our work area tastes mostly limony now instead of just “developer’s thoughts”, a fragrance that yet has to bottled.

The technical solution

Technically, the integration of the two humidifiers with our reporting infrastructure was very easy. Every XFD is controlled by an IRC bot that understands certain commands suitable for the device and hangs around at our central IRC server. As an humidifier only understands “on” and “off”, it could be controlled just like the ONOZ! lamp. We connected the humidifiers to a remote controlled power supply, switched it on and let the bot control the supply.

Our reporting infrastructure forwards its results to an aggregation software that interprets the numbers and produces IRC commands for the device bots. All of this is done with a combination of website scraping (Hudson as our continuous integration server has a wonderful XML API) and IRC messaging.

The history of XFD so far

Over the last years, we gathered XFDs for almost every human sense. We have visual effects, audible feedback using speech synthesis and even bought an USB rocket launcher for forced feedback needs. With the Smell-O-Mat, we can now deal with smelling, too.
The last human sense we have to address is tasting. Plans for the “coffee salter” were impeded by our sense of humanity. We keep searching.

Read more about our Extreme Feedback Devices:

Make it visible: The Project Cockpit

We are a project shop with numerous customers booking software development projects as they see fit, so we always work on several projects concurrently in various sub-teams.

We always strive for a working experience that provides more productivity and delight. One major concept of achieving it is “make it visible”. This idea is perfectly described in the awesome book “Behind Closed Doors” by Johanna Rothman and Esther Derby from the Pragmatic Bookshelf. Lets see how we applied the concept to the task of managing our project load.

What is the Project Cockpit?

The Project Cockpit is a whiteboard with titled index cards and separated regions. If you glance at it, you might be reminded of a scrum board. In effect, it serves the same purpose: Tracking progress (of whole projects) and making it visible.

Here is a photo of our Project Cockpit (with actual project names obscured for obvious reasons):


How does it work?

In summary, each project gets a card and transitions through its lifecycle, from left to right on the cockpit.

The Project Cockpit consists of two main areas, “upcoming projects” and “current projects”. Both areas are separated into three stages eachs, denoting the usual steps of project placing and project realization.

Every project we are contacted for gets represented by an index card with some adhesive tape and a whiteboard magnet on its back. The project card enters the cockpit on the left (in the “future” or “inquiry” region) and moves to the right during its lifecycle. The y-axis of the chart denotes the “importance” of the project, with higher being more important.


In the “upcoming” area, projects are in acquisition phase and might drop out to the bottom, either into the “delay filing” or the “trash”. The former is used if a project was blocked, but is likely to make progress in the future. The latter is the special place we put projects that went awry. It’s a seldom action, but finally putting a project card there was always a relief.

The more natural (and successful) progress of a project card is the advance from the “upcoming” area to the “present” bar. The project is now appointed and might get a redefinition on importance. Soon, it will enter the right area of “current” projects and be worked on.

The right area of “current” projects is a direct indicator of our current workload. From here on, project cards move to the rightmost bar labeled “past” projects. Past projects are achievements to be proud of (until the card magnet is needed for a new project card).

If you want to, you can color code the project cards for their urgency or apply fancy numbers stating their volume.

What’s the benefit?

The Project Cockpit enables every member of our company to stay informed about the project situation. It’s a great place to agree upon the importance of new projects and keep long running acquisitions (the delay filing cases) in mind. The whiteboard acts as an information radiator, everybody participates in project and workload planning because it’s always present. Unlike simpler approaches to the task, our Project Cockpit includes project importance, urgency and volume without overly complicating the matter.

The whiteboard occupies a wall in our meeting room, so every customer visiting us gets a glance on it. As we use internal code names, most customers even don’t spot their own project, let alone associate the other ones. But its always clear to them in which occupancy condition we are, without a word said about it.

Ultimately, we get visibility of very crucial information from our Project Cockpit: When the left side is crowded, it’s a pleasure, when the right side is crowded, it’s a pressure ;-)

Spelling the feedback: The LED bar

Our fully automated project ecosystem provides us with feedback of very different type and granularity. We felt it was impossible to render every single notable event into its own extreme feedback device (XFD). Instead, we implemented an universal feedback source: the LED bar.


You know the LED bar already from a shop window of your town. It tells you about the latest special bargain, the opening hours of the shop or just something you didn’t want to know. But you’ve read it, because it is flashing and moving. You just can’t pass that shop window without noticing the text on the LED bar.

Our LED bar sells details to us. The most important issues are already handled by the ONOZ Lamp and the Audio feedback, as both are very intrusive. The LED bar is responsible to spell the news, rather than to tell it.

A very comforting news might be “All projects sane”, which happen to be our regular state. You might be told that you rendered “project X BROKEN”, but you already know this, as the ONOZ Lamp lit up and you were the one to check in directly before. It’s better to be informed that “project X sane” was the build’s outcome. After a while, the text returns to the regular state or blanks out.

Setting up the LED bar

We aren’t the only ones out there with a LED bar on the wall. Dirk Ziegelmeier for example installed his at the same time, but blogged much earlier about it. He even gives you detailed information about the communication protocol used by the device and a C# implementation for it. The lack of protocol documentation was a bugger for us, too. We reverse engineered it independently and confirm his information. We wrote a complete Java API for the device (in our case a LSB-100R), which we might open source on request. Just drop us a note if you are interested.

Basically, we wrote an IRC bot that understands commands given to it and transforms it into API calls. The API then deals with the low-level transformation and the device handshake. This way, software modules that want to display text on the LED bar from anywhere on the internal net only need to talk on IRC.

The idea of connecting an IRC channel and the led bar isn’t unique to us, either. The F-Secure Linux Team blogged about their setup, which is disturbingly equal to ours. Kudos to you guys for being cool, too.

Effects of the LED bar

The LED bar is the perfect place to indicate project news. Its non-intrusive if you hold back those “funny” displaying effects but versatile enough to provide more than simple binary (on/off) information. Its the central place to look up to if you want to know what’s the news.

We even found out that our company logo (created by Hannafaktur) is scalable down to 7×7 pixels, which exactly fits the LED bar in height:


Try this with your company’s logo!

Read more about our Extreme Feedback Devices:

Extreme Feedback Device (XFD): The ONOZ! Lamp

When two good ideas meet, there’s a chance for an even better idea to be born. This happened to us some time ago, when the ONOZ! Lamp came up.

(This is a free translation and revision of an earlier article written in german)

The first good idea

On April 1st 2004, Alberto Savioa published a blog entry about an idea of two lava lamps (green and red) displaying the current build state of a project. I was somewhat distracted that day, marrying my wife, so the idea came to us two years later. Mike Clark wrote his wonderful book “Pragmatic Project Automation” and included not only the idea of the lava lamps, but also detailed construction guidance.

The second good idea

One day, an email contained a little animated gif with two panic guys running around.

Investigation suggests Jonn Wood as the author. We thought the guys act exactly like us after a broken build (one of the worst things that may happen here), so the gif and the word “ONOZ” were integrated into our company culture.

The birth of another idea

After we read about the lava lamps, we wanted to own them, too. But only after inspiration from the animated gif, we were sure about our specific realisation. We merged the two states into one lamp (on/off instead of green/red) and did without the lava. A normal desk light would do the job now.
We even have a good justification for the omission of the green (lava) lamp:

  • it saves energy
  • no timer switch is needed for the nights/weekends
  • our team includes colorblinds

The last reason is a good one when you look at this simulation of colorblindness:
These are pictures of the original green and red lava lamps:

This is how it looks to a colorblind employee. These images were generated by Vischeck, a website trying to inform about colorblindness practically.

Not much of a difference. If you swap them around secretly, ten percent (the percentage of colorblinds in the male population) of your team will panic without reason.

The ONOZ! Lamp

With little investment, we build a system supervising the build state of all our projects. Every build process sends its result to a server that checks for failures. If a build failed, the lamp gets switched on over traditional X10 signals. We can’t overlook the sudden burst of luminance, we panic a bit and try to fix the build. The lamp turns off when all projects are back to normal.

The ONOZ! Lamp is just a lamp standing around, until something ugly happens. Then it turns into a glowing infernal of failure. We nearly failed to give it a correctly spelled name, too. We named it “ONOEZ! Lamp” first, which seems to be the only invalid spelling of this exclamation.

The effects

The ONOZ! Lamp works great. Its mere presence has a comforting effect, as long as it is off. Which is the case most of the time. When it fires, the effect is like an alarm stopping all work. And the operator giving the alarm is always alert and incorruptible: our continuous integration server.

Read more about our Extreme Feedback Devices:

Extreme Feedback Device (XFD): The Code Flow-O-Meter

This is a free translation and revision of an earlier article written in german.

Since March 2007, the Schneide uses another Extreme Feedback Device (XFD): the Code Flow-O-Meter.

So what is this thing?

We bought a portable fountain made of slate, filled in water (no additives) and connected the power supply cord with a X10 application module. Then we programmed a litte IRC Bot (using the ten-minutes-to-success java IRC library pircbot) that triggers the module. We were then able to control the fountain by speaking to it over IRC.

Afterwards, we piped the commit messages of our source repositories to a little script that determines the “impact” of the commit by measuring the amount of changes to the code. This sounds more sophisticated as it really is, the number of changed files was a good enough guess for it. This impact is related to a duration, the more impact, the longer the timespan. Now the script tells the IRC bot to activate the fountain for that amount of time.

This way, we have a direct but unobtrusive notification about what is going on in the repositories, as this is the most important location of our company (talking about the numerous safety nets we applied to it would require another blog post). Initially, we thought about playing an audio sample singing “alleluia”, too, but this became ridiculous soon.

But why do you want this notification?

One of the rules of agile (or good) programming says “commit early, commit often”. But as soon as every little commit gets examined by a continuous integration server, all automated tests and a large number of software quality metric tools, the liability to keep the changes local a little bit longer grows. “After all, it’s ready when it’s done, and this is soon enough to check in” was a common justification especially among the less experienced programmers. But that’s the best way to miss the early feedback. And early feedback is effective feedback.

So we installed our portable fountain as a counter-incentive against late commits and started a little game. The rule of the game is simple:
Keep the Flow-O-Meter running!
When you commit, the fountain flows. The size of your commit is not as important as the commit itself, so its better to commit often. If everyone does it, the fountain may flow the whole day.

The Code Flow-O-Meter may be regarded as a measurement device of “progress”. Things are in a state of flux as long as it is running.

And did it work?

Yes, definitly. The Code Flow-O-Meter has become our little pet. Everyone loves it because it’s friendly and comforting. Think of a tamagochi, but without the annoyances. You only need to change and commit something to feed it and get the reward, nothing more.

As an additional gain, everybody else loves it, too. When we show it to a customer, they first see an ordinary portable fountain. When we explain and demonstrate our working cycle (code, commit, review) to them, something magical happens. I tend to think they involuntary get the notion of something happening after the commit when the fountain comes to life. This may be the concept of a build server, otherwise being invisible and intangible, materializing in the fountain. When we continue to explain what happens after the build, like the ONOZ! lamp lighting up to indicate a failed build, it’s already clear to them that the process does not end with the waterflow. The Code Flow-O-Meter serves as a link between the developer’s local work and the build feedback arriving out of nowhere some minutes later.

Read more about our Extreme Feedback Devices: