Recap of the Schneide Dev Brunch 2018-02-11

brunch64-borderedOn Sunday, February 11th, we held another Schneide Dev Brunch, a regular brunch on the second sunday of every other (even) month, only that all attendees want to talk about software development and various other topics. This brunch was well-attended, with two new guests that seemed to feel comfortable after just a few minutes. The table provided just enough space for us. As usual, the main theme was that if you bring a software-related topic along with your food, everyone has something to share. Because we were a larger group, we discussed with an agenda. As usual, a lot of topics and chatter were exchanged. This recapitulation tries to highlight the main topics of the brunch, but cannot reiterate everything that was spoken. If you were there, you probably find this list inconclusive:

Asciidoctor

Our first topic was a presentation of the asciidoc syntax and the asciidoctor converter. The asciidoc syntax can be used to describe structured textual content in a concise manner with a few funny special characters. It looks like markdown at the first glance, but has the benefit of being fully standardized and extensible instead of one of several competing dialects.

The asciidoctor is an active rewrite of the first asciidoc converter. Given the right set of formatters, you can generate a PDF, a self-contained interactive HTML presentation and a static web page from one single source. This follows the “one true source, many derived artifacts”-approach that every software developer should know by heart (Don’t Repeat Yourself!).

Because setting up a productive asciidoctor environment is still some manual work, one of our attendees has published a github repository that automates the manual work as much as possible: asciidoc presentation.

If you need an alternative to markdown or even TeX/LaTeX, have a look at asciidoc. It seems specifically aimed at software developers and is probably already integrated in your favorite IDE (the integration in IntelliJ is seamless).

Yarn

We discussed the two extreme approaches to handle dependencies for your project. The first extreme is to only include links to other projects/repositories that need to be fetched manually or automatically. Most modern build tools orientate towards this approach, even if there are some disadvantages like the recent Go/Github disturbance.

The second extreme is to include everything that’s needed in your repository. For a Javascript project that means that you provide your own, probably out-dated version of leftpad and thousand other libraries. You need a way to deal with transitive dependencies and keep an eye on all the versions to mitigate the risk of long-fixed vulnerabilities.

The second extreme is extremely helpful if you don’t have internet access but want to develop.

A good compromise is the local offline mirror, something that build tools/dependency managers like maven have for over a decade. This local repository is filled with all the leftpads and apache-commons that your projects need. If you checkout a new project, remember to make the build tool download the dependencies to your local repository before you go offline.

for Javascript, this concept seems a bit foreign. Who would develop for the web without the web, anyways? Yarn seems to provide a working offline mirror functionality for npm packages, though. Perhaps it is worth a look.

Opt-Out explained with groceries

During out dependency management discussion, we also compared downloadable installers with malware droppers. But that’s not where our comparisons stopped. We also came up with a good metaphor for Opt-In vs. Opt-Out methods.

If you enter a grocery store and grab a shopping cart, only to find that it already contains two or three packages of sweets and some overpriced milk, you chose an Opt-Out store. Your responsibility is to return the goods to their aisle or to buy them.

You’re probably used to Opt-In type grocery stores.

Book review: Functional Programming in Java

We took a look at Pierre-Yves Saumont’s book “Functional Programming in Java”. This book is a little bit odd in that you shouldn’t read it, you are meant to program it. Or at least try to solve the numerous training exercises and riddles. This makes it hard to read the paper version of the book, because it’s a pick-two situation of keyboard, mouse and book on your desk.

The book explains real functional programming and not the functional additions of Java 8. It explains it on top of the JVM, using Java’s language constructs. But, you will learn it from the origins and develop abstractions like Function oder Supplier yourself. Imagine you had all compiler magic of Java 8 but no JDK classes to leverage it – this book tells you how to use it.

It’s a good book, but unique in its style. It grounds on exercises and your own understanding of the material. It isn’t spoon-fed, you have to work for it yourself. It didn’t chose any existing pure functional language, but plain Java for this. So you have no excuse about weird syntax or unfamiliar ecosystems. It’s boring old Java turned in an exciting new way.

And if you are lazy and don’t feel like writing your own functional groundwork toolkit, you might want to look at vavr, a functional programming library for Java.

Polyglot language idioms

We discussed the portability of language idioms and highlighted the Curiously Recurring Template Pattern (CRTP) from C++. Then we spent some time explaining and understanding the CRTP and finally comparing it to similar things like Java’s Enum<E extends Enum<E>>. It can get wicked complex fast with those constructs.

Laser printer identification

Since 2011, we know that every single page of a color laser printer can be individually identified and traced back to your printer. This is common knowledge as stated on Wikipedia, but it still was a surprise to some of us. Why do we need such tracking? On request of many goverments.

Spectre and Meltdown

We didn’t repeat the fresh common knowledge about the nearly universal CPU security vulnaribilities Meltdown and Spectre. But we noted that it got eerily quiet, as if everybody holds their breath and waits for the morning clock to wake them up.

Some rumors has it that the current prototypes of ARM and Intel CPUs are not vulnerable, as if the manufacturers changed their speculative code execution unit long before the exploits came to light. Maybe they circumvented the problem by pure luck?

We hope to hit snooze soon.

Planned obsolescence

We discussed the notion of planned obsolescence. Typical consumer products have a flaw or weakness that is bound to break soon after manufacturer guarantee is void. Or it is deliberately incorporated into the product like page counters, waste tanks with limited capacity or the infamous short-lived light bulb.

A good start on the topic is the documentary “buy it for the waste” or “Kaufen für die Müllhalde” on german.

Given the recent noise around Apple battery life, we are now in an era where planned obsolescence is sold like a feature. Twenty-five years ago, this was Science Fiction. The author of this blog entry remembers a science fiction story by Robert Sheckley (“Utopia mit kleinen Fehlern” or “A Ticket to Tranai” in english). The protagonist reaches a planet that seems to be perfect. It is so perfect that nothing breaks anymore. The industry is desperate and sees the protagonist as a genius when he invents “planned obsolescence” and “designed discomfort” as means to raise sales. The planet has several other flaws as well. The story and the whole book is worthwhile and right on topic.

Book review (again): Clean Architecture

At last, we spoke about Robert C. Martin’s (Uncle Bob’s) new book “Clean Architecture”. I’ve already published my book review on our blog, but added some impressions and context after thinking about the book for some more time. Summary: The book is good, even if nearly half the pages might qualify as filler material and there are only two main messages. If the announcement of Uncle Bob on the last page in the Appendix becomes true, you might want to skip his next book, though.

Another book review for the future might be the new Effective Java, 3rd edition.

Epilogue

As usual, the Dev Brunch contained a lot more chatter and talk than listed here. The number of attendees makes for an unique experience every time. We are looking forward to the next Dev Brunch at the Softwareschneiderei in April. We even have some topics still on the agenda (like a report about first-hand experiences with the programming language Rust). And as always, we are open for guests and future regulars. Just drop us a notice and we’ll invite you over next time.

Advertisements

Oversimplified C++ Project FAQ 2018

If you are starting a new C++ project, you’re faced with a few difficult decisions. C++ is not a ‘batteries-included’ language, so you need to pick a few technologies before you can start.
Yet worse, the answer to most of the pressing questions is often ‘it depends’ and changing one of the choices mid-project can be very expensive.
Therefore, I have compiled this list to give totally biased and oversimplified to the most important questions. If you want more nuanced answers, feel free to do your own research.
This is meant to be a somewhat amusing starting point.

FAQ

1. Which OS should I pick?

Linux

Rationale

Usually, not a choice you can make yourself – but if you do: dependency management is easier with a package manager, and it seems to be the most dominant OS in the C++ community. Hence you will get the best support and easiest access to technologies.

2. Which build system should I use?

CMake

Rationale

This is what everyone else is using, and those that are not are a real pain. For better or worse, the market is locked in. With target based properties in modern CMake, it’s not even that bad.

3. Which IDE should I choose?

Visual Studio 2017 on Windows, CLion everywhere else.

Rationale

CLion is getting more robust and feature rich with every release. Native CMake support and really cool refactoring capabilities finally make this a valid contender to Visual Studio’s crown. However, the VS debugger is still the best in the game, so VS still comes out on top on Windows – tho not by a huge margin.

4. Which Language version should you use?

C++14

Rationale

C++17 is not quite there yet with library, tool and platform support. Also, people do not really know how to use it well yet. C++14 builds on the now well-established C++11, which a few rather important “fixes” – and support is ubiquitous.

5. Which GUI toolkit should you use?

Qt

Rationale

No other toolkit comes close in maturity. Qt’s signal/slot system almost seamlessly integrates with C++11 lambdas, making the precompile step needed for SLOTs a non-issue. Barring the license costs for closed-source projects, there is really no reason not to use it.

6. Should you use Boost?

No

Rationale

Boost is a huge and clunky dependency that will explode your build times as soon as you even touch it. And it’s ‘viral’ enough that you can distinguish a Boost project from a non-Boost project. Boost.Optional, Boost.Variant and Boost.Filesystem prepare you for a smooth transition to C++17, but there are other more lightweight alternatives available.

Closing thoughts

There you have my totally biased opinion but hopefully entertaining. YMWV, but I think this is a good starting point if you don’t want to exeriment too much.

OPC-UA Performance and Bulk Reads

In a previous post on OPC on this blog I introduced some basics of OPC. Now we’ll take look at some performance characteristics of OPC-UA. Performance depends both on the used OPC server and the client, of course. But there are general tips to improve performance.

  • to get maximum performance use OPC without security

OPC message signing and encryption adds overhead. Turn off security for maximum performance if your use case allows to use OPC without security.

  • bulk reads increase performance

Bulk reads

A bulk read call reads multiple variables at once, which reduces communication overhead between client and server.

Here’s a code example using Eclipse Milo, an open-source OPC-UA stack implementation for the Java VM.

final String endpointUrl = "opc.tcp://localhost:53530/OPCUA/SimulationServer";
final EndpointDescription[] endpoints = UaTcpStackClient.getEndpoints(endpointUrl).get();
final OpcUaClientConfigBuilder config = new OpcUaClientConfigBuilder();
config.setEndpoint(endpoints[0]);

final OpcUaClient client = new OpcUaClient(config.build());
client.connect().get();

final List<NodeId> nodeIds = IntStream.rangeClosed(1, 50).mapToObj(i -> new NodeId(5, "Counter" + i)).collect(Collectors.toList());
final List<ReadValueId> readValueIds = nodeIds.stream().map(nodeId -> new ReadValueId(nodeId, AttributeId.Value.uid(), null, null)).collect(Collectors.toList());

// Bulk read call
final ReadResponse response = client.read(0, TimestampsToReturn.Both, readValueIds).get();
final DataValue[] results = response.getResults();
if (null != results) {
	final List<Integer> values = Arrays.stream(results).map(result -> (Integer) result.getValue().getValue()).collect(Collectors.toList());
	System.out.println(values.stream().map(String::valueOf).collect(Collectors.joining(",")));
}

client.disconnect().get();

The code performs a bulk read call on 50 integer variables (“Counter1” to “Counter50”). For performance tests you can put the bulk read call in a loop and measure the times. You should, however, connect to the server over the target network, not on localhost.

With a free (however not open-source) OPC UA simulation server by Prosys and Eclipse Milo for the client I measured times around 3.3 ms per bulk read of these 50 integer variables. I got similar results with the UA.NET stack by the OPC Foundation. Of course, you should do your own measurements with your target setup.

Keep also in mind that the preferred way to use OPC UA is not to constantly poll the values of all the variables. OPC UA allows you to monitor variables for changes and to get notified in case of a change, which is a more event-driven approach.

Gradle projects as Debian packages

Gradle is a great tool for setting up and building your Java projects. If you want to deliver them for Ubuntu or other debian-based distributions you should consider building .deb packages. Because of the quite steep learning curve of debian packaging I want to show you a step-by-step guide to get you up to speed.

Prerequisites

You have a project that can be built by gradle using gradle wrapper. In addition you have a debian-based system where you can install and use the packaging utilities used to create the package metadata and the final packages.

To prepare the debian system you have to install some packages:

sudo apt install dh-make debhelper javahelper

Generating packaging infrastructure

First we have to generate all the files necessary to build full fledged debian packages. Fortunately, there is a tool for that called dh_make. To correctly prefill the maintainer name and e-mail address we have to set 2 environment variables. Of course, you could change them later…

export DEBFULLNAME="John Doe"
export DEBEMAIL="john.doe@company.net"
cd $project_root
dh_make --native -p $project_name-$version

Choose “indep binary” (“i”) as type of package because Java is architecture indendepent. This will generate the debian directory containing all the files for creating .deb packages. You can safely ignore all of the files ending with .ex as they are examples features like manpage-generation, additional scripts pre- and post-installation and many other aspects.

We will concentrate on only two files that will allow us to build a nice basic package of our software:

  1. control
  2. rules

Adding metadata for our Java project

In the control file fill all the properties if relevant for your project. They will help your users understand what the package contains and whom to contact in case of problems. You should add the JRE to depends, e.g.:

Depends: openjdk-8-jre, ${misc:Depends}

If you have other dependencies that can be resolved by packages of the distribution add them there, too.

Define the rules for building our Java project

The most important file is the rules makefile which defines how our project is built and what the resulting package contents consist of. For this to work with gradle we use the javahelper dh_make extension and override some targets to tune the results. Key in all this is that the directory debian/$project_name/ contains a directory structure with all our files we want to install on the target machine. In our example we will put everything into the directory /opt/my_project.

#!/usr/bin/make -f
# -*- makefile -*-

# Uncomment this to turn on verbose mode.
#export DH_VERBOSE=1

%:
	dh $@ --with javahelper # use the javahelper extension

override_dh_auto_build:
	export GRADLE_USER_HOME="`pwd`/gradle"; \
	export GRADLE_OPTS="-Dorg.gradle.daemon=false -Xmx512m"; \
	./gradlew assemble; \
	./gradlew test

override_dh_auto_install:
	dh_auto_install
# here we can install additional files like an upstart configuration
	export UPSTART_TARGET_DIR=debian/my_project/etc/init/; \
	mkdir -p $${UPSTART_TARGET_DIR}; \
	install -m 644 debian/my_project.conf $${UPSTART_TARGET_DIR};

# additional install target of javahelper
override_jh_installlibs:
	LIB_DIR="debian/my_project/opt/my_project/lib"; \
	mkdir -p $${LIB_DIR}; \
	install lib/*.jar $${LIB_DIR}; \
	install build/libs/*.jar $${LIB_DIR};
	BIN_DIR="debian/my_project/opt/my_project/bin"; \
	mkdir -p $${BIN_DIR}; \
	install build/scripts/my_project_start_script.sh $${BIN_DIR}; \

Most of the above should be self-explanatory. Here some things that cost me some time and I found noteworthy:

  • Newer Gradle version use a lot memory and try to start a daemon which does not help you on your build slaves (if using a continous integration system)
  • The rules file is in GNU make syntax and executes each command separately. So you have to make sure everything is on “one line” if you want to access environment variables for example. This is achieved by \ as continuation character.
  • You have to escape the $ to use shell variables.

Summary

Debian packaging can be daunting at first but using and understanding the tools you can build new packages of your projects in a few minutes. I hope this guide helps you to find a starting point for your gradle-based projects.

For your last project ask yourself: What did the stakeholders learn

At the start of a new project we like to begin with a naive mind, a beginner’s mind. In it we try to avoid our assumptions and start with a blank slate. Our clients do not. They are expert in their respective domain and know a lot. It’s naturally that during the project we learn lot about them and their domain, their work and their daily struggles. We see how they work around the limitations of their tools and cope with software written more than 30 years ago.
But besides us learning something about the domain, the stakeholders learn something about their domain, too. Because to develop the domain, the use cases and the daily work, we have to know details and reasons. Why is this step before that? Is it optional? Are these all the formats which are allowed? How long is the text usually? Why is there an exception to the rule? How often does it happen?
Usually we ask questions which cover the most traveled path, the happy trail. But in order to understand we need to get to the edges as well. The dark edges. Sometimes the number of objects we deal with is so big, nobody has all the answers. Our work, even before we write the software, enables collaboration. People and different departments have to work together. We work with all of them. Our software helps them to reach their common goals. But before that we need to know. And in order to tell us that the stakeholders need to dig deeper in their respective domain. Sometimes we need to look at the history in their domain, their work history, the decisions other stakeholders made in the past. It’s like archeology without the shovels, well, most of the time :).
Luckily the people we work with enjoy getting to know more about their work. They are astonished what depth the details have. How much different types of things, where gaps are. It is not always easy to light up areas that were kept in the dark so long. That were done just the way they were done. No we come and ask sometimes uneasy questions. We need to know. We need to know exactly. We need to know deeply.
This curiosity is not for its own sake. Our clients can confirm that the new software is so much better than the old. Not technical, but most importantly more adapted to their daily work.

That’s what’s important.

The four archetypes of cloud users – part 1 of 2

In the occupational field of accounting, the strong trend towards cloud services is noticeable. Everything needs to be digital, and with digital, they mean online, and with online, they mean in the cloud. Every expense voucher needs to be scanned and uploaded, because in many cases, it can be booked automatically. In the new era of accounting, human intervention is only needed for special cases.

I see this as a good example of how digital online services can transform the world. Every step in the process would have technically been possible for the last twenty years, but only the cloud could unify the different participants enough so that a streamlined end-to-end process is marketable to the masses. And in this marketing ecstasy, the stakeholders that profit the most (the accountants) often forget that their benefits are just a part of the whole picture. In order to assess the perceived and actual benefits of all stakeholders, you at least need to apply an archetype to each participant.

The four archetypes

In my opinion, there are four different archetypes of cloud users. Let’s have a look at them and then assess the risks and potentials when selling a digital online service to them. I’ll list the archetypes in the order from biggest risk to biggest potential.

Archetype 1: The tinfoil hat

A person that could be identified as a “tinfoil hat” doesn’t need to be a conspiracy weirdo or paranoid maniac. In fact, the person probably has deep and broad knowledge about technology and examines new technologies in detail. The one distinctive feature of the tinfoil hats is that they take security, including IT security, very seriously. They don’t take security for granted, don’t trust asseverations and demand proof. You can’t convince a tinfoil hat by saying that the data transfer is “encrypted”, you need to specify the actual encryption algorithm. Using RC4 ciphers for the SSL protocol isn’t good enough for the tinfoil. You need at least proof that you understood the last sentence and took actions to mitigate the problem. Even then, the tinfoil will hesitate to give any data out of hands and often choose the cumbersome way in order to stay safe. “Better safe than sorry” is his everyday motto.

Tinfoil hats always search for scenarios that could compromise their data or infrastructure. They are paranoid by default and actively invest in security. “On premises” is the only way they deploy their own services, and “on premises” is how they prefer to keep their data.

Typical signs of a tinfoil hat archetype include:

  • self-hosted applications
  • physical servers
  • lack of (open) wireless network
  • physically separated networks
  • signed and encrypted e-mails

Trying to sell a cloud service to a tinfoil hat is like trying to sell a flight to an aviophobian (somebody with fear of flying). There is always another way to get from A to B, seemingly safer and more controllable. If you are selling cloud services, tinfoil hats are your worst nightmare. If you can convince a tinfoil hat, your product is probably made of fairy dust and employs lots of unicorns.

Archetype 2: The clipboard

Clipboard people are wary of new technologies, but assess them in the context of usability. They demand high security, but will compromise if the potential of the new technology far exceeds the risk. Other than the tinfoil hat, the clipboard sees his role as an enabler, but will not rest to increase the perceived or actual safety of the product. You can appease a clipboard by giving evidence of security audits from a third party. They will trust known authorities, because it means that they can always deflect blame in case of an accident to these authorities.

Clipboards run on checklists, safety protocols and recurring audits. They don’t try to avert every possibility of a security breach, but will examine each incident in detail and update their checklists. They don’t care about “on premises” or “off premises” as long as the service is reachable, safe enough and reliable. If a cloud service has an higher availability than the local counter-part, the clipboard will think about a migration.

Typical signs of a clipboard archetype include:

  • Virtual Private Networks (VPN)
  • Two-Factor Authentication
  • Token-Based Authentication
  • Strong Encryption

The clipboard will listen if you pitch your cloud service and can be enticed by the new or better capabilities. But in the very next sentence, he will ask about security and be insistent until you provide proof – first-hand or by credible third parties. You can convince a clipboard if your product is designed with safety in mind. As long as the safety is state-of-the-art, you’ll close the deal.

Outlook on the second part

In the second part of this blog entry, we will look at the remaining two archetypes, namely the “combination lock” and the “smartphone”. Stay tuned.

Did you identify with one of the archetypes? What are your most important aspects of cloud services? I would love to hear from you.

C++17: The two line visitor explained

If you have ever used an “idiomatic” C++ variant datatype like Boost.Variant or the new C++17 std::variant, you probably wished you could assemble a visitor to dispatch on the type by assembling a couple of lambda expressions like this:

auto my_visitor = visitor{
  [&](int value) { /* ... */ },
  [&](std::string const& value) { /* ... */ },
};

The code in question

While reading through the code for lager I stumbled upon a curious way to to make this happen. And it is just two lines of code! Wow, that is cool.

template<class... Ts> struct visitor: Ts... { using Ts::operator()...; };
template<class... Ts> visitor(Ts...) -> visitor<Ts...>;

A comment in the code indicated that the code was copied from cppreference.com where I quickly found the source on the page for std::visit, albeit with the different name “overloaded”. There were, however, no comments as to how this code worked.

Multiple inheritance to the rescue

Lambda expressions in C++ are just syntactic sugar for callables, pretty much like a struct with an operator(). As such, you can derive from them. which is what the first line does.
It uses variadic templates and multiple inheritance to assemble the types of the lambdas into one type. Without the content in the struct body, an instantiation with our example would be roughly equivalent to this:

struct int_visitor {
  void operator()(int value)
  {/* ... */}
};

struct string_visitor {
  void operator()(std::string const& value)
  {/* ... */}
};

struct visitor : int_visitor, string_visitor {
};

Using all of it

Now this cannot yet be called, as overload resolution (by design) does not work across different types. Hence the using in the structs body. It pulls the operator() implementations into the visitor type where overload resolution can work across all of them.
With it, our hypothetical instantiation becomes:

struct visitor : int_visitor, string_visitor {
  using int_visitor::operator();
  using string_visitor::operator();
};

Now an instance of that type can actually be called with both our types, which is what the interface for, e.g. std::visit demands.

Don’t go without a guide

The second line intruiged me. It looks a bit like a function declaration but that is not what it is. The fact that I had to ask in the (very helpful!) C++ slack made me realize that I did not keep up with the new features in C++17 as much as I would have liked.
This is, in fact, a class template argument deducation (CTAD) guide. It is a new feature in C++17 that allows you do deduce template arguments for a type based on constructor parameters. In a way, it supercedes the Object Generator idiom of old.
The syntax is really quite straight-forward. Given a list of constructor parameter types, resolve to a specific template instance based on those.

Constructing

The last piece of the puzzle is how the visitor gets initialized. The real advantage of using lambdas instead of writing the struct yourself is that you can capture variables from your context. Therefore, you cannot just default-initialize most lambdas – you need to transport its values, its bound context.
In our example, this uses another new C++17 feature: extended aggregate initialization. Aggregate initialization is how you initialized structs way back in C with curly-brackets. Previously, it was forbidden to do this with structs that have a base class. The C++17 extension now lifts this restriction, thus making it possible to initialize this visitor with curly brackets.

Edit 2018/04/16: The people on r/cpp rightfully pointed out that using the “other name” in the code snippet was confusing – so the visitor is now called “visitor”.