On the usefulness of volatile memory

I have a strange hobby: Raising ants as pets. Not literally pets, because I don’t give them names and they probably don’t know I even exist, but as tended animals in a controlled and restricted environment, the formicarium. This doesn’t sound too strange if you know three additional things about me: I was fascinated by ants since kindergarten, I play dwarf fortress for fun and I’m interested in computers.

My main colony is a lasius niger nest with about 1500 individuals. Lasius niger or black garden ant is a very common and easy to raise ant that is small enough to not require too much space but big enough to be observed and tracked.

I could tell you hours and hours of ant facts, but let’s concentrate on the topic of this blog post: An ant colony can be perceived as one big organism. Each individual ant has a clear role in the hierarchy:

  • The queen ant is the sole egg layer. In an established colony, she won’t do anything else. She will be fed, cleaned and protected by her workers. She was born a queen and cannot be replaced. If she dies, the whole colony will slowly fade away because nobody produces new ants anymore. Worker ants don’t know if they still have a queen and don’t care anyway. If a queen dies, she will be fed to the remaining larvas as soon as her scent disappears.
  • The male ants are born, fed and kicked out of the nest, preferably when the young queens fly out to find a mate. They won’t do anything else and have a lifespan of days.
  • The worker ants are all sterile females and do all the work. Literally all the work. They are born workers and spend every moment of their lives contributing to the hive or just sitting around scrounging food. Other workers are too busy to judge them.

But keep in mind that ants only have short-range communication means (scent, antennae drumming and ground vibrations), so no single ant has complete overview about the situation. It wouldn’t have the brains to process that information anyway. Ants have limited capability to remember things but no long-time memory. It isn’t necessary for the single worker ant to store any information. The information is stored in the hive – literally.

If you abstract some details away, you can also perceive and describe an ant colony as a computer or at least as a massive parallel problem solver. The algorithms are ingrained in the worker ants and are executed ruthlessly. The problems are food, water, enemies, cleaning and nurture. If the environment (the problem space) is suitable for the algorithms, the colony will thrive. Else the colony will ultimately fail. An ant colony at the side of a busy road will lose many workers in sudden “enemy” attacks, while a colony in the middle of a meadow might find less insects killed by cars. Not a single ant in each colony is aware of these relations. But they have found a way to share information: They mark their position (and therefore their way) by special scent fluid. They use their scents to label the environment for other ants.

If you ever encountered an avid user of printed sticker labels, this is what ants do, too: They put a sticker label on everything they come in contact with. If you seem to be food, they label you as food and rush back to the hive, laying out a “food in this direction” lane. If you seem dead and inedible, they mark you as garbage and them or some other ant will pick you up and follow the garbage line (finding the garbage line often requires them to return near their hive so it seems they mistook the garbage for food). The garbage area is marked with its own scent, preferably at a cliff. My ants love to throw things down a cliff! If you didn’t get the relationship to dwarf fortress yet, it should be clear by now. There are countless more examples, but you get the idea. The environment is not only an area, it is the long-term memory of the colony. The colony’s “memory brain” is scent on dirt, stones and sticks. The ant colony has outsourced its complete knowledge about the surrounding into the surrounding itself.

The ants have invented something I would call an “inverse cloud”. The cloud in IT is a concept of data storage that exists mostly independent of physical location and provides access to that data from virtually anywhere. The ants’ inverse cloud is a concept of data storage that is tightly coupled to a physical location and provides access to that data only if you are in the immediate vincinity. If you remove a physical data storage part in the IT cloud, it gets replaced by other parts that contain mirrored copies of the data. The cloud never forgets. If you remove a physical data storage part in the ants’ inverse cloud, it is forgotten immediately. Ants accept their environment as it is right now and never look back.

Now think about what happens when it rains and all the scents are washed away.

After every rain, the ant colony enters a “new level”, a fresh environment to be discovered and labeled. They probably never grow old to rediscover the same cliff again and again. This is what happens to a computer when the power is lost. It loses its working memory. But keep in mind that the colony retains some memories: the hive is underground and often rain-proof by overlaying stones or plants. So the colony remembers that it is currently in the hive, because it always smells like hive. The colony remembers the queen chamber because it always smells like queen. The colony remembers the brood chambers because they always smell like teen spirit (SCNR).

In my formicarium, it never rains. The ants get enough water by drinking troughs, but the marked lanes are never erased. This leads to all sorts of silly situations that the ants don’t even recognize as such. For example, a strong “food here” scent lane exists long after the food is gone. So a lot of enthusiastic workers run around searching in the target area. And remember that the hive smell never fades? My ants have assimilated area after area as “hive” after enough ants have marked it. So now they react excessively to disturbances because they think they are defending the hive – and therefore the queen! – but are ant miles away from the colony. Even better, a lot of young ants that usually never leave the hive until they are older wander out into the open (“it’s still the hive, just less dark”) and panic as soon as they encounter something that shouldn’t be in a hive. The panic spreads by, you’ve guessed it already, alarm scent and soon hundreds of battle-ready ants are running around frantically without any one of them knowing why.

So, to speak in computer terms, the main memory never gets erased and the caches never flushed. A little error can spread like a wildfire (like the panic example above) and cause disadvantages like energy consumption without any real gain (no enemy to defend against) or a lot of delicate ants sitting around in plain sight of their predators. The whole system is fragile and erratic and probably wouldn’t survive in the wild. A good measure of rain would remove the odd memories and probably ease the ants because their hive, the area that needs to be defended at all costs, would shrink again.

I’ve raised neurotic ants in need of a cold shower.

How does this give us any insight into modern computing? Well, my train of thoughts is this: If modern systems are unable to forget, because memory is cheap and permanent, we might be prone to design software that acts neurotic and hyped up. The ability to forget, to really don’t remember at all, might be crucial in designing resilient parallel systems. There is the cost of losing valueable information, but the benefit of losing all results of errors seems to match it. So volatile memory might be a nuisance for us programmers, but it also provides a “blank slate” every time the system starts and is the reason for the most important question in IT: “Have you tried turning it off and on again?”

Systems that rely on “place oriented programming”  seem to have the need of regular reset phases where the working memory is cleared and the system goes into the next cycle fresh and rested. We might even call it sleep. And in case you wonder: The sleep of ants is an ongoing topic for research.

Disclaimer: I know that not all ants are as dumb as lasius niger. Some ants even teach each other facts about their environment. The wikipedia article mentions some wonderful examples. I had ant colonies with more complex ants and they were wonderful. But right now, as I’m typing this, there is a lasius niger worker that heaves a wasp husk part to the top corner of the formicarium, throws it down (did I mention they love throwing things?) and runs back down to heave it up again, probably because the corner is somehow marked as a garbage dump zone. It has repeated this process at least half a dozen times now. This is how a biological infinite loop looks like. Some ants even parallelize such a loop to exhaustion:

Learning about Class Literals after twenty years of Java

I’ve programmed in Java nearly every day for twenty years now. At the beginning of my computer science studies, I was introduced to Java 1.0.x and have since accompanied every version of Java. Our professor made us buy the Java Language Specification on paper (it was quite a large book even back then) and I occassionally read it like you would read an encyclopedia – wading through a lot of already known facts just to discover something surprising and interesting, no matter how small.

With the end of my studies came the end of random research in old books – new books had to be read and understood. It was no longer efficient enough to randomly spend time with something, everything needed to have a clear goal, an outcome that improved my current position. This made me very efficient and quite effective, but I only uncover surprising facts and finds now if work forces me to go there.

An odd customer request

Recently, my work required me to re-visit an old acquaintance in the Java API that I’ve never grew fond of: The Runtime.exec() method. One of my customer had an recurring hardware problem that could only be solved by rebooting the machine. My software could already detect the symptoms of the problem and notify the operator, but the next logical step was to enable the software to perform the reboot on its own. The customer was very aware of the risks for such a functionality – I consider it a “sabotage feature”, but asked for it anyway. Because the software is written in Java, the reboot should be written in Java, too. And because the target machines are exclusively running on Windows, it was a viable option to implement the feature for that specific platform. Which brings me to Runtime.exec().

A simple solution for the reboot functionality in Java on Windows looks like this:


Runtime.exec("shutdown /r");

With this solution, the user is informed of the imminent reboot and has some time to make a decision. In my case, the reboot needed to be performed as fast as possible to minimize the loss of sensor data. So the reboot command needs to be expanded by a parameter:


Runtime.exec("shutdown /r /t 0");

And this is when the command stops working and politely tells you that you messed up the command line by printing the usage information. Which, of course, you can only see if you drain the output stream of the Process instance that performs the command in the background:


final Process process = Runtime.exec("shutdown /r /t 0");
try (final Scanner output = new Scanner(process.getInputStream())) {
    while (output.hasNextLine()) {
        System.out.println(output.nextLine());
    }
}

The output draining is good practice anyway, because the Process will just stop once the buffer is filled up. Which you will never see in development, but definitely in production – in the middle of the night on a weekend when you are on vacaction.

Modern thinking

In Java 5 and improved in Java 7, the Runtime.exec() method got less attractive by the introduction of the ProcessBuilder, a class that improves the experience of creating a correct command line and a lot of other things. So let’s switch to the ProcessBuilder:


final ProcessBuilder builder = new ProcessBuilder(
        "shutdown",
        "/r",
        "/t 0");
final Process process = builder.start();

Didn’t change a thing. The shutdown command still informs us that we don’t have our command line under control. And that’s true: The whole API is notorious of not telling me what is really going on in the background. The ProcessBuilder could be nice and offer a method that returns a String as it is issued to the operating system, but all we got is the ProcessBuilder.command() method that gives us the same command line parts we gave it. The mystery begins with our call of ProcessBuilder.start(), because it delegates to a class called ProcessImpl, and more specific to the static method ProcessImpl.start().

In this method, Java calls the private constructor of ProcessImpl, that performs a lot of black magic on our command line parts and ultimately disappears in a native method called create() with the actual command line (called cmdstr) as the first parameter. That’s the information I was looking for! In newer Java versions (starting with Java 7), the cmdstr is built in a private static method of ProcessImpl: ProcessImpl.createCommandLine(). If I could write a test program that calls this method directly, I would be able to see the actual command line by myself.

Disclaimer: I’m not an advocate of light-hearted use of the reflection API of Java. But for one-off programs, it’s a very powerful tool that gets the job done.

So let’s write the code to retrieve our actual command line directly from the ProcessImpl.createCommandLine() method:


public static void main(final String[] args) throws Exception {
    final String[] cmd = {
            "shutdown.exe",
            "/r",
            "/t 0",
    };
    final String executablePath = new File(cmd[0]).getPath();

    final Class<?> impl = ClassLoader.getSystemClassLoader().loadClass("java.lang.ProcessImpl");
    final Method myMethod = impl.getDeclaredMethod(
            "createCommandLine",
            new Class[] {
                    ????, // <-- Damn, I don't have any clue what should go here.
                    String.class,
                    String[].class
            });
    myMethod.setAccessible(true);

    final Object result = myMethod.invoke(
            null,
            2,
            executablePath,
            cmd);
    System.out.println(result);
}

The discovery

You probably noticed the “????” entry in the code above. That’s the discovery part I want to tell you about. This is when I met Class Literals in the Java Language Specification in chapter 15.8.2 (go and look it up!). The signature of the createCommandLine method is:


private static String createCommandLine(
        int verificationType,
        final String executablePath,
        final String cmd[])

Note: I didn’t remove the final keyword of verificationType, it isn’t there in the original code for unknown reasons.
When I wrote the reflection code above, it occurred to me that I had never attempted to lookup a method that contains a primitive parameter – the int in this case. I didn’t think much about it and went with Integer.class, but that didn’t work. And then, my discovery started:


final Method myMethod = impl.getDeclaredMethod(
        "createCommandLine",
        new Class[] {
                int.class, // <-- Look what I can do!
                String.class,
                String[].class
        });

As stated in the Java Language Specification, every primitive type of Java conceptionally “has” a public static field named “class” that contains the Class object for this primitive. We can even type void.class and gain access to the Class object of void. This is clearly written in the language specification and required knowledge for every earnest usage of Java’s reflection capabilities, but I somehow evaded it for twenty years.

I love when moments like this happen. I always feel dumb and enlightened at the same time and assume that everybody around me knew this fact for years, it is just me that didn’t get the memo.

The solution

Oh, and before I forget it, the solution to the reboot command not working is the odd way in which Java adds quote characters to the command line. The output above is:


shutdown /r "/t 0"

The extra quotes around /t 0 make the shutdown command reject all parameters and print the usage text instead. A working, if not necessarily intuitive solution is to separate the /t parameter and its value in order to never have spaces in the parameters – this is what provokes Java to try to help you by quoting the whole parameter (and is considered a feature rather than a bug):


final String[] cmd = {
        "shutdown",
        "/r",
        "/t",
        "0",
};

This results in the command line I wanted from the start:


shutdown /r /t 0

And reboots the computer instantaneous. Try it!

Your story?

What’s your “damn, I must’ve missed the memo” moment in the programming language you know best?

The personal economics of programming languages

Recently, one of my students asked a good question about what programming languages I would recommend learning. His ideal language would be “syntactically ugly, but giving insights that are universal to programming”. My first reaction was to answer that he has just described Perl, but that was too easy of an answer. So I tried to define the basics about programming languages, starting with the personal economics.

Economics of programming languages

An organization that wants to produce a piece of software needs to answer a lot of questions like “what programming language will be best suited for the task?”. Often, these questions get diluted and rather sound like “what programming language should we stipulate for all our projects, now and forever?”. That’s when politics and economics overlap and intermingle. We can leave this problem for the organizations to solve themselves. But if we scale the question down to an individual programmer – you, what influences are there to find an answer to the question “what programming languages should I learn?”.

I try to answer with the concept of utility: Learn those languages that, over a reasonable time, yield the most “utility”. There are at least two types of utility in our profession: money and joy. You can learn a programming language because your job requires it (money) or because you are curious and/or dig its particularities (joy). Most of the time, a specific programming language contains a mixture of both utilities for you. How you rate those utilities is up to you and probably varies from situation to situation. If you start a private fun project, picking the boring mainstream language from work might get things get done faster, but when would you want the fun to be over sooner?

Let me give two extreme examples for this concept:

  • If you start to learn COBOL now, chances are high that you will achieve two things: You will be disgusted by the language and the existing codebase, but delighted by the salary and job security. COBOL is a high money-utility programming language. It ranks low in any survey or statistics about programming languages, but is widely used in big business today and tomorrow. You might refer to https://blog.hackerrank.com/the-inevitable-return-of-cobol/ for more information.
  • If you start to learn Esterel now, you might experience two things over time: an epiphany about how flawed our concept of time is in most programming languages and an existential crisis because your brain isn’t capable to wrap itself around most sourcecode. Whatever comes first will define your learning success. There are virtually no jobs that require Esterel (even if some might benefit from it) and you can only program and build so many bicycle computers in your spare time (this is a typical introduction project to Esterel). Esterel is a pure joy-utility programming language. You can claim to be proficient in synchronous programming afterwards, but nobody will know what that even is.

A third type of utility

But I think that there might be a third type of utility for personal learning choices based on economics: The stirrup iron utility. Knowledge of some programming languages isn’t useful from a money-driven viewpoint and may lack enjoyability, but it serves as a door-opener to more enjoyable or sellable languages. It serves as an interim utility because it doesn’t have value in itself, but serves as a multiplier for either the money or joy utility. To rate the value of this utility to your career, you need to be clear about your career goals, especially your anticipated skill portfolio.

Skill portfolio shapes

Modern recruitment differentiates between several skill portfolio shapes, most noteably the “I” and “T” shape:

  • Programmers with “I”-shaped skill portfolios are experts in one specific field of programming. They might, for example, be the best C# programmer you’ve ever met. But they flop around like a fish out of water once they need to use another programming language. They will choose their familiar tools for every problem that needs to be solved and will solve it fast if possible or
  • Programmers with “T”-shaped skill portfolios have knowledge across all fields of programming, albeit limited, and drilled down into one field specifically. Why they chose to master their field can mostly be explained with the money or joy utility. They probably gained their broad knowledge base by using stirrup irons.

If you happen to know what’s expected from you until your retirement (let’s say you chose to program in COBOL), the “I”-shape is a viable and efficient strategy to manage your skill portfolio. There is nothing wrong with this approach (as long as it works).

If you have a hunch that you don’t have the capability to invest in broad knowledge, the “I”-shaped skill portfolio is your logical choice. It takes a lot to be able to come to a self-assessment that shows your limitations. It’s a good thing to know your limits and build a career within them. A lot of programmers don’t know their limits and burn out, because not meeting the requirements produce a lot of stress (on both sides). Better be yourself than over-promise and under-deliver constantly.

The “T”-shape means that you need to invest your time wisely. And we are not talking “work time” only, but “life time”, because you’ll probably need to spend your spare time working on your portfolio, too. Becoming a “jack of all trades” programmer is an endeavour of at least ten years without any possibility to shortcut. You need to select your jobs in accordance to your learning strategy and always be receptive to opportunities. You need to improve your learning abilities. You need to do so much at once that I suggest you start by watching Cory House’s talk about “Becoming an Outlier”. He’s spot on with so many things.

Stirrup iron programming languages

There are some programming languages that can be seen as the archetypes of a whole class of languages. Most knowledge of these archetypes can be directly applied or transfered to each language in the class. It’s the language’s concepts that are the real benefit. If you understand the synchronous programming aspect in Esterel, you’ll recognize it straight away in languages like LabView or SIGNAL. It may even just be a part of the other language (like in many multi-paradigm programming languages), but it will be familiar to you.

So what are some stirrup iron languages?

That’s a tough question and I want to place it out there. Can you drop a comment and name the programming language that had the most peculiar influence on your knowledge? I would like to refer to the book Seven Languages In Seven Weeks from the Pragmatic Bookshelf. It covers Ruby, Io, Prolog, Scala, Erlang, Clojure and Haskell. Do you agree with that selection? I would like to hear from you.

There are some ideas about this topic already: The talk “The Future of Programming” from Bret Victor (if you don’t know this guy already, please watch his legendary “Inventing on Principle” too). Richard Astbury presents three “new” hot programming languages (with matching outfits) in his talk “The State of the Art”. And Robert C. Martin is sure to have found “The Last Programming Language”.

One thing is sure: We should train the next generations of programmers in those stirrup iron languages, so they can quickly grasp the language flavour of the year. This is mostly done already, of course, but the students inevitably complain about the “weird” choices. So we need to explain upfront the economics of programming languages.

And, in a lighter tone at the end, there is always the ongoing competition for the worst programming language ever.

Eliminate the Water Carrier

Some years ago, an old lady with more than hundred years of life experience im America was asked which technology changed her life the most. She didn’t hesitate to answer: running water. The ability to open the tap and have instant access to fresh water was the single most important technology in her life, even before electricity and all the household appliances it enables. Without running water, every household is forced to employ or pay a worker that does nothing else but to carry water from the source to the sink.

In today’s physical world, with physical goods, there is still a profession that relies on a specific aspect of physical objects: They won’t move from A to B without a carrier. The whole field of logistics and transportation would be obsolete in the instant that physical goods learn to move themselves. The water carrier lives on, in the form of a cardboard or palette carrier.

The three basic goods of IT are software, data and information. They all share a common trait: They can move without a human carrier. In the old days before the internet, software was distributed on physical objects like floppy disks (think of oddly shaped usb sticks) or CDs later. With the ubiquitous access to running data (often called the internet and mobile computing), we can draw our software straight from the tap. (And yes, I like the metaphor of the modem as an “information tap”). As the data throughput of our internet connections grew, it became feasible to move large amounts of data into “the cloud”. The paper boy that brings the newspaper early every morning is replaced by a virtual newspaper that updates every few seconds. The profession of a data carrier didn’t exist outside of very delicate data movements. And even them got replaced by strong cryptography.

Even information and knowledge, a classic carrier-bound good, is slowly replaced by books and pre-recorded online courses. The “wise man” (or woman) still exists, but his range was extended from his immediate geographical surrounding and his arbirtrary placement on the timeline to the whole world and all times after his publication. We don’t need to be physically present to attend a course anymore and we don’t need to synchronize our schedule with the lecturer. Knowledge and information is free to roam the planet.

With all this said and known, why are there still jobs and activities that resemble nothing more than the water carrier of our information age? Let me reiterate once more what a water carrier does: He takes something from position A and moves it to position B. In the ideal case, everything he picked up at A is delivered at B, in full and unchanged. We don’t want the carrier to lose part of the water underway and we surely don’t want him to tamper with our water.

As soon as you add something valueable to the payload (you augment it) while you carry it from A to B, you aren’t a water carrier anymore, you can be described in terms of your augmentation. But what if you add nothing? If you deliver the payload in the same condition as you picked it up? Then you are a water carrier. You don’t have a justification for your work in IT. Or you have one that I can’t see right now, then I’m eager to hear from you! Please leave a comment.

There is a classic movie that describes life and work in IT perfectly: Office Space. If you haven’t seen it yet, please put it on your watch list. I’m sure you can even draw it from your information tap. In the movie, a company with a generic IT name needs to “consolidate their staff” (as in lose some slackers). They hire some consultants that interview the whole crew. Each interview is hilarious in itself, but one is funny, tragic and suitable for our topic at hand, the water carrier:

The problem with Tom Smykowski (the guy trying to defend his job) is, that he’s probably better with people than most developers, but he still cannot sell his augmentations to the two consultants. They try to tie him down to a physical good that must be carried, but even Tom has to admit that somebody else covers the physical level. So he tries to sell his “good influence” on the process as the augmentation, but the consultants are too ignorant to recognize it. Needless to say, Tom loses his job.

Every time you just relay information without transforming it (like appending additional information or condensing it to its essence), you just carry water. Improve your environment by bypassing yourself. If you take yourself out of the communication queue, you will save time and effort and nobody has a disadvantage. You should only be part of a communication or work queue if you can augment the thing being passed through the queue. If you can’t specify your augmentation, perhaps somebody else behind you in the queue can give you hints about it. I would argue that being able to pinpoint one’s contribution to the result is the most important part of every workplace description. If you know your contribution, you can improve it. Otherwise, you may be carrying water without even knowing it.

Eliminate the middlemen in your work queues to improve efficiency. But be sure to keep anybody who contributes to the result. So, eliminate the water carriers.

Recap of the Schneide Dev Brunch 2017-04-09

brunch64-borderedLast sunday, we held another Schneide Dev Brunch, a regular brunch on the second sunday of every other (even) month, only that all attendees want to talk about software development and various other topics. This brunch was well-attended and opened the sunroof season for us. We even had to take turns on the sunny places because we didn’t want to catch a sunburn in April. As usual, the main theme was that if you bring a software-related topic along with your food, everyone has something to share. Because we were very invested in our topics, we established an agenda for the event. As usual, a lot of topics and chatter were exchanged. This recapitulation tries to highlight the main topics of the brunch, but cannot reiterate everything that was spoken. If you were there, you probably find this list inconclusive:

Online courses

Our first topic was an report on an ongoing online course, a so-called MOOC (Massive Open Online Course) on the topic “Software Design for Non-Designers”. It aims at bringing basic knowledge of UX and UI design to programmers, who frequently lack even the most fundamental principles of design (other than code design and even that is open for discussion). A great advantage of these MOOCs is that you can minimize your brutto time investment and therefor maximize your netto yield. You are not bound to a certain place, free from specific times (other than the interaction with other participants) and yet free to engage in a community of peers. The question that remains is how valueable the certificate will be. But the initial expectations are met: The specific course is very practical and requires moderate effort in reasonable periods.

One crucial aspect is the professionality of the presenting lecturer. In this MOOC, there are talk-oriented presenters and then there is Scott Klemmer. His lectures stand out because he writes on an invisible wall before him. The camera looks through the wall. What seems like nice CGI turns out to be a real glass pane. Mr. Klemmer puts down his note in mirror writing! Once you realize that, you cannot help it but be in awe.

There are a lot of MOOCs nowadays. Other courses that got mentioned cover the topic of machine learning https://www.coursera.org/learn/machine-learning and Getting Started with Redux (a famous Javascript framework) by Dan Abramov on Egghead: https://egghead.io/courses/getting-started-with-redux. Some courses even take place on Youtube, if you manage to avoid the comment sections, like the talks from Geoffrey Hinton about neuronal networks and machine learning. Mr. Hinton is part of the Google Brain team.

The critical part of each MOOC is the final examination. Some courses require online or even real-time tests, some online provide certificates for test results in a certain timespan. Usually, the training assignments are peer reviewed by other course participants.

We will probably see this type of knowledge transfer more often in the future.

Interesting websites

While we talked about a lot of topics at once, some websites and projects got mentioned. I include them here without full coverage of the topics that led to it:

  • jsfiddle: A website that provides a quick sketchboard for web technologies like Javascript, HTML and CSS. It’s like a repl for the web.
  • regex101: A website that provides a quick sketchboard (and debugger) for regular expressions in different languages. It’s like an online IDE for regular expressions.
  • codefights: A website that puts you in the fighting pit for developers. Prove your programming skills against competition all around the globe!
  • vimgolf: A website that lets you prove your proficiency in the only text editor that counts: vim. Every keystroke counts and a mouse cannot be found!

Some of these websites might be a lot more fun in a team, except the regex one. Don’t use regular expressions in a team project! It’s a violation of the sane developer’s rules of engagement.

Workplace conflicts

One participant reported about his latest insights in conflict management during work. He applied the concepts of warfare and the four steps of complex tasks to recent disputes and had tremenduous results. Even the introduction chapter of the Strategies of War book was enough to install new notions and terms into his planning and acting. He was astounded by the positive effects of his new portfolio.

The new terminology seems to be the essential part. European (or even western) adults don’t learn the terminology of conflict and therefore cannot process disputes on a rational level, only with emotions. You cannot plan or communicate with emotions, so you cannot plan your conflict behaviour. As soon as you have the language to describe the things you perceive, you can analyze them, reflect on them and plan for them. Making a solid plan (other than “go in and win somehow”) is the best preparation for an upcoming conflict. Words shape our world. I’ve seldomly seen it clearer than in this report.

Just for starters, there is a difference between a “friend” and an “ally”.

Project documentation

An open question to all participants was our handling of documentation efforts in a project, be it for the user, customer or following developer. We discussed it with this open scope and came up with some tools that I can repeat here:

  • The arc42 software architecture template can help to shape the documentation effort for future developers or current developers if they aren’t included in the architecture effort.
  • The user manual is often written in TEX. Developers are used to the tool by constant exposition during their academic studies.
  • One idea was to generate the requirements for the developers from the user manual, as in “user manual first” or “user manual driven development”.
  • The good old Markdown syntax is useable but has its limits in top-notch aesthetics.
  • We see some potential in ASCIIDoc, but it needs to improve further to play in the same league as other tools.
  • Several participants have tried to automate the process of taking screenshots of the software for usage in various documents. If you want to try this, be warned! There are many detail problems that need to be solved before your solution will be fully automatic and reliable. A good starting point for thoughts is the “handbook data set” that can reproduce the same screenshot content (like entries in lists, etc.) in a different software version.

In the outskirt area of this discussion, the worthwhile talk “Stop Refactoring!” by Nat Pryce was mentioned. He presents an interesting take on the old question of “good enough”.

Epilogue

As usual, the Dev Brunch contained a lot more chatter and talk than listed here. The number of attendees makes for an unique experience every time. We are looking forward to the next Dev Brunch at the Softwareschneiderei in June. We even have some topics already on the agenda (like a report about first-hand experiences with the programming language Rust). And as always, we are open for guests and future regulars. Just drop us a notice and we’ll invite you over next time.

Look at the automated tests to diagnose the project ailments

A cornerstone of modern software development is developer testing. That means that developers are the primary authors of automated test code. In theory, that is a good thing and might look like the quality assurance department is out of work soon. In practice, we as a profession tried for nearly twenty years to install a culture of developer testing in our work and still end up with software projects that feature no automated tests at all (Side note: JUnit 1.0 was released in February of 1998).

What we know about automated tests

One piece of common understanding about developer testing is the test pyramide. Let’s iterate quickly what we know about it. There are different kinds of automated tests and the test pyramide differentiates three of them:

  • Acceptance tests or UI tests are the heaviest type of automated test. They operate on the software from the outside, with the means of a real user and try to assert that real use cases are accomplishable.
  • Integration tests often use several parts of the system in a test scenario that asserts the correct collaboration of the parts. Integration tests may take some time to come to a conclusion and utilize real hardware like network or disks.
  • Unit tests tend to be small and quick and focus on a particular aspect of an “unit” like a class or entity aggregate. Their reach into the system should be short and might be forcefully restricted by employing mocks.

These three types, the A, I and U of automated tests, should come in different numbers. A good rule of thumb is that for every acceptance test, there might be up to one thousand unit tests. If you draw the quantities as areas, they appear in form of a pyramide. A small top of acceptance tests rests on a broader seating of integration tests that relies on a groundwork of many unit tests. A healthy test pyramide looks like this:

Take this picture as an orientation, not as an absolute scale. But be sure to count your different test types from time to time.

Outlining the tests

This is actually one of the first things I do when I get introduced to a new and unknown code base. This happens quite often when I do consulting work for existing development teams. Have a look at the automated tests, determine their type and count their numbers. If it resembles anything close to the test pyramide, you’ve got a chance. If the resulting shape looks different, you might find this blog entry useful:

The Tower

If you have a hard time finding any tests (because there are none) or you find only some half-assed attempts to produce a meaningful automated test suite, you look at a tower project. The tower is rather small in diameter, in the cases of absent tests it is nothing more than a thin vertical line (the “stick”). If you find a solid number of tests for every type, you’ve found a “block” project. Block projects usually don’t have a problem, but a history of test effort migration either from unit to acceptance tests or, more common, in the other direction. If you find a block, you are fine.

The tower, though, is a case of neglect. The project team might have started serious efforts to automated their tests, but got demotivated by intrinsic or extrinsic influences and abandoned the tests soon after their creation. Nobody has looked after them since and the only reason they still pass green is that they didn’t really test anything to begin with or only cover an area of the system that is as finished as it is boring. Topics like user management or utility classes are usually the first and only things that got tests in a tower scenario.

Don’t get me wrong, the tower indicates the absence of tests, but not the absence of willingness to write automated tests, unless the tower is really a stick. A team willing to invest in automated tests may only lack knowledge and coaching about the topic. Be sure to lead them bottom-up (unit tests first), though.

The Egg

If you’ve categorized and counted the tests and couldn’t find many acceptance or unit tests, you’ve found an egg. The egg consists of mostly integration tests that may lean into unit testing territory by asserting smallest bits of functionality here and there (often embedded in an overarching test storyline) or dip their toes into gui-based testing by asserting presentation-specific properties of widget objects. While they provide ample test coverage for the system, they also tie application logic and presentation details together and don’t help to separate domain code from the use cases.

The project team is probably proud of their test coverage and doesn’t see any value in differentiating the automated tests types, because “every test improves the situation”. The blindness to test types is the core problem that may be cured with training and coaching (I’ve found the ATRIP-rules to be particularly effective to distinguish integration and unit tests), but the symptoms, especially the lack of separation of concerns, have to be mitigated soon, too.

One way to start there is to break the tests down into their integration and their unit test parts. You can work from assertion to assertion and ask: is this necessary to ensure the current use case? If not, extract a new unit test focussed on only this one assertion.

As soon as you add a pedestal consisting of unit tests to your egg, you are on your best way to a healthy test pyramide.

The Ice Cream Cone

This is the most fearsome automated test outline in existence, even more dramatic than the stick. Usually, the project team is really enthusiastic about writing tests or at least follow order to do so, but they cannot test parts of the application in isolation. A really tragic case was a complex system that was so entangled with its database, through countless stored procedures that contributed to the application logic, that it was hopeless to think about tests without the database. And because every automated test had to start the whole system including the database, there was really no need to differentiate between application logic and presentation logic. It all became a gordic knot of dependencies that enforced the habit of writing elaborate automated GUI-based tests to test the smallest logic bits deep inside the core. It felt like eating single rice grains with overly long, flimsy wooden chopsticks that would break often.

The ice cream cone is problematic because the project team needs to realize that their effort was mislead and the tests are all telling the bitter truth: the system’s architecture isn’t fit for proper automated tests. It’s not the tests, it’s you (or your architecture)! Nobody wants to hear that and more so, nobody wants to untangle the mess (without the help of a proper safety net consisting of automated tests). Pinning tests are probably helpful in this scenario.

But you need to turn the test pyramide around or the project team will suffocate by the overly costly test tax while increasing technical debt.

Epilogue

Please keep in mind that it’s not a problem in itself that your project doesn’t have a normal test pyramide. It’s great that you have automated tests at all! But your current test type distribution might not be as effective as possible, might be more expensive than necessary and might be not the right automated test setup for your development goals.

What are your stories with automated test setups? Care to share it with us in the comments?

Recap of the Schneide Dev Brunch 2017-02-12

brunch64-borderedYesterday at sunday, we held another Schneide Dev Brunch, a regular brunch on the second sunday of every other (even) month, only that all attendees want to talk about software development and various other topics. This brunch was a little smaller in numbers of attendents, but very interesting nonetheless. As usual, the main theme was that if you bring a software-related topic along with your food, everyone has something to share. Because we were very invested in our topics, we established an agenda for the event. As usual, a lot of topics and chatter were exchanged. This recapitulation tries to highlight the main topics of the brunch, but cannot reiterate everything that was spoken. If you were there, you probably find this list inconclusive:

Household roboters

We started our brunch talk by mentioning the services our five years old roombas provide for us, especially keeping the floor free of any small things. The biggest effect when having an electronic pet like a roomba is that you learn to keep your things above ground, especially cords with expensive electronics on the other end. The continuous elimination of dust is just a positive bonus on top of your behaviour adjustment. You keep the floor tidy, the roomba just mercilessly enforces this rule.

Today, there are many alternatives to the original roomba and most have really nice features and abilities. So no matter what brand you buy, you’ll get a capable floor police.

Code Review priorities

In our recent dev brunches, we talked about code review tools and code review habits. This time, we talked about code review priorities and the sorry state we are still in with current tools. We worked out that its nearly useless to only show the diff with a few lines of surrounding code and expect a thorough review. Even the concept of “changed files” is rather distracting in an object-oriented language. But even the current tools are only as good as we make use of them.

The first priority of code reviews should be finding and eliminating bugs – “real” bugs that would have had surfaced in production otherwise and “hypothetical” bugs that could have shown up in production. This means that code review is in its core an activity for the user of the software. Only second priority is the understandibility of the source code. If the reviewer doesn’t understand the code, chances are high that nobody will, including the author in a few weeks or months. Cleaning up the code now mitigates the problem for the lowest possible cost because the “hurdle of understanding” isn’t raised yet. A code review should never work on the level of linters and should not address topics that can be checked by an automatic tool. Suggestions about refactorings should be kept to a minimum because they may serve no purpose if the code isn’t touched again. Refactor when the code is opened for the second edit, not on the first review. Review the code on the semantic level, not on the syntactic.

And keep in mind that code review tend to be used for conditioning remarks (“don’t do that”, “this is ugly”, “I don’t approve”, etc.). Try to avoid conditioning and strive to provide educational comments (“if you change this to that, then you’ll profit from this benefit”, “here’s a suggestion for a better approach and here’s why it is better”, etc.). But we also discussed that at this point, the code review remarks are probably better said in a pair programming session.

Code reviews are a powerful tool for development teams, but with power comes danger. Hopefully, we get adequate software tools to help us avoid the common traps soon.

Time management

Out of interest, we talked about some principles and practices to better manage one’s time.

The first thing to be aware of are the two fundamentally different schedules of management and development. The manager’s schedule is clocked in 30 minute intervals and driven by outside demand (meaning that a manager idles when not requested), while the maker’s schedule works with 4-hour blocks of uninterrupted, deeply concentrated work. You can probably see the problems that arise when somebody in a maker cycle is interrupted multiple times as if he was in a manager cycle. The first thing you can do is to announce your maker cycles (by clear “busy right now” indicators like headphones or a “do not disturb” sign) or announce your manager cycles much to the effect of consultation hours. Let your disturber know if he can disturb safely or if even the question causes damage.

Another important thing is to arrange your surrounding according to your schedule. Your schedule is so important that you should choose your service providers according to it. For example, if you work full time, look for hairdressers that work saturdays or dentists that offer appointments in the night. If you need to contact people for personal matters during work hours, allocate a specific timebox each day or at one day in the week and do it then. Announce this timebox to everybody who might want to contact you during work. This way, to can differentiate people that respect your (announced!) schedule and people that don’t. Depending on your rigorousness, you can cut the people that harm your schedule out of your life.

Work only with people who value your expectations (if reasonable). If you give a task to somebody, let’s say a craftsman, and state the deadline, you need to be sure the deadline is met without you checking or the craftsman will report back in time. Don’t give tasks to people who leave you in the lurch.

It all boils down to keep the control about your calendar. Whenever you give somebody else the opportunity to “conquer” a slice of your available time at their convenience, you increase your own inconvenience.

Karlsruhe C++ User Group

The year 2017 started with a new-founded C++ user group in karlsruhe that started with great events. David, the organisator of the user group is a regular attendee at our Dev Brunch and reported about his experiences with the boot process of the user group. He found a sponsor in the Clausmark GmbH and accompanied the monthly talks and programming events with a regular table that provides a similar format as our Dev Brunch, just in the night and not in the morning. We also talked about possible future content, and found code-centric “git guided live casts” a worthwhile format. Another format, the excellent code retreats are a great way to learn from others, but require a full day and not just two hours in the evening. The Game of Life kata is really fun, even when done repeatedly. Once you discover the solution in APL, you’ll want a special APL keyboard, too.

We are looking forward to hear great talks and meet cool people at the C++ user group Karlsruhe.

Sales knowledge

Our last topic in the bonus time (we were lenient with our scheduled time box, it’s sunday!) was about sales and the installation of sales knowledge and sales behaviour in a group of developers. We agreed that starting with Strategic Selling is a good choice because the process/framework is compatible with established developer culture and effective in its results. The resulting shift in the perception of occurrences is immediate and powerful. Strategic Selling is a rather old sales process that share some similarities with Solution Selling, another nerd-friendly process for complex business-to-business (B2B) sales.

Epilogue

As usual, the Dev Brunch contained a lot more chatter and talk than listed here. The number of attendees makes for an unique experience every time. We are looking forward to the next Dev Brunch at the Softwareschneiderei in April. We even have some topics already on the agenda (like a report about first-hand experiences with the programming language Rust and a discussion about the concept of provisioning). And as always, we are open for guests and future regulars. Just drop us a notice and we’ll invite you over next time.