Exiling a legacy COM component

One of our long-standing Java applications has several dependencies to native libraries, which are called via the Java Native Interface (JNI). We usually avoid native library dependencies, but this application must interface with some hardware devices for which the vendors only provide access through native libraries.

From 32 bit to 64 bit

Until recently the application ran in a 32-bit Java VM and the native libraries were 32-bit DLLs as well. Then the time had come to update the application to the 64-bit world. We wanted the application to run in a 64-bit JVM, and 32-bit library code cannot run in a 64-bit process.

We were able to replace the 32-bit libraries with 64-bit libraries, all except one. This particular dependency is not just a native library, but a Windows COM component. We had developed a wrapper DLL, which connected the COM component via JNI to the Java application. While we do have the source code of the wrapper DLL, we don’t have the source code of the COM component, and the vendor does not provide a 64-bit version of the component.

32 bit

So we decided to exile this COM component to a separate process, which we refer to as a container process. The main application runs in a 64-bit JVM and communicates with this process via a simple HTTP API. The API calls from the application do not require a very low latency.

Initially we had planned to implement this container process as a C++ program. However, after a spike it turned out that there is a quicker way to both interface with a COM component and provide a simple self-hosted HTTP service on the Windows platform: The .NET framework provides excellent support for COM interoperability. Using a COM component in .NET is as simple as adding the component as a reference to the project, importing the namespace and instantiating the COM object like a regular .NET object. The event handling of the COM component, which requires quite some boilerplate code to set up in C++, gets automatically mapped to the C#/.NET event handling. The only reminder of the fact that you’re using a COM component is the amount of out and ref parameters.

For the HTTP API we chose the Nancy framework, with which we have had good experiences in previous projects. The architecture now looks like this:

64-bit

While it is a drawback that the container process now depends on the .NET runtime, it is outweighed by the benefit for us: The C#/.NET code for interfacing with the COM component is more maintainable than the previous JNI wrapper code was or a native implementation of the container process would have been.

Updating from Grails 2.3 to something newer

We are developing, running and maintaining moderately sized Grails web application with > 120 domain classes  since 2008 or Grails 1.0.3. The web application is still in production running on Grails 2.3.8. Just recently we wanted Java 8 support and the usual bugfixes and improvements you get by updating the framework. Since time and budget are very limited (as always…) we decided not to move to 3.x but only to the latest 2.x version. It seemed a safer and easier option and opened up the way to 3.x where many things changed completely.

Trying to go to 2.5.4

The upgrade procedure is generally well documented in Grails. That allowed us to upgrade from 1.0 to 1.3, from 1.3 to 2.2 and finally from 2.2 to 2.3. We skipped 2.0 because of too many problems we faced during the upgrade. As usual the major changes and tasks are mentioned in the upgrade guide. It started smoothly but we finally had to abort the upgrade process because we were bitten by https://github.com/grails/grails-data-mapping/issues/581 . We had not the time to dig fully into it and resolve the issue.

Trying to go to 2.4.5

Many of the changes and improvements and most notably a Groovy version supporting the Java 8 runtime are already available in Grails 2.4.5. So we gave it a shot hoping for fewer problems than with 2.5.4. Actually we got our application running in less than an hour but quite some of our unit, integration and functional tests failed. After finding some advice in http://stackoverflow.com/questions/16532631/grails-unit-test-mock-domain-with-assigned-id we changed our unit tests to use the @Mock() mixin instead of mockDomain() which works in 2.3 and is broken in 2.4.

When trying to fix our integration tests we saw that some of our HQL queries failed. Something was wrong navigating/querying multiple association levels so we finally gave up on this one, too.

Conclusion

Even though we managed to keep our Grails application alive for many years and several framework versions each upgrade carries a significant risk of breakage and requires quite some effort. This time we are stuck again and will have to invest more time to bring the application up-to-date again.

I would advise anyone already using or deciding for Grails as the web framework of choice to start with the latest and greatest release and to budget several person days for upgrades of medium sized projects. The devil is in the details…

Recap of the Schneide Dev Brunch 2016-04-10

brunch64-borderedLast sunday, we held another Schneide Dev Brunch, a regular brunch on the second sunday of every other (even) month, only that all attendees want to talk about software development and various other topics. In case you miss the recap article about the february brunch: It didn’t happen. We all took a break, but are on track again. So if you bring a software-related topic along with your food, everyone has something to share. We were quite a lot of developers this time, so we had enough stuff to talk about. As usual, a lot of topics and chatter were exchanged. This recapitulation tries to highlight the main topics of the brunch, but cannot reiterate everything that was spoken. If you were there, you probably find this list inconclusive:

Why software development conferences?

We began with a curious question: Why are there even conferences about software development? You can read most of the content for free on the internet and even watch the talks afterwards. So why attend one for a lot of money? We discussed the topic a bit and came up with an analysis:
There are (at least) four different interested groups in a conference:

  • The organizer or commercial host is mostly interested in a positive revenue. As long as there’s a possibility for some net gain, somebody will host a conference. The actual topic is a secondary matter for them (this might explain some of the weirder conferences out there, like the boring conference).
  • The developers that really attend a conference are a small subset of all developers. They all have their own personal motives to pay money and invest time and inconviences to be there in person. Some might rely on the quality filter of a conference board, some are looking forward to meet their peers in an annual ritual. There might be those that can learn best if somebody talk-feeds them the topic. Whatever reason, a lot of developers enjoy participating at conferences. If it happens to be paid by the employer and booked as worktime, who would not?
  • Then there are the speakers. They have the additional burden to convince a committee of their topic, prepare a talk of high quality and be able to perform on stage (something that is harder than it looks). The speakers seek reputation and credible proof of expertise. His resume will probably profit, too.
  • And at last, the companies that sponsor the conference, maintain a booth with big roll-ups and smiling employees and give their developers a chance to attend are in the game to represent, to recruit and build their brand. A lot of traditional marketing effort goes into trade fairs, so why not treat the developer market like any other and be present in the developer fairs?

We can conclude that software development conferences can provide value for every associated stakeholder. As long as this sentence holds true, conferences will be held.
The question didn’t came out of the blue: one of our attendees got accepted as a speaker on the Karlsruher Entwicklertag 2016 and wanted to learn about the different expectations he needs to address. He will give his talk on the next Dev Brunch to practice the flow and to pass the hardest critics. The topic: git internals. We are thrilled!

Stratagems and strategies

The next topic contained another talk, not at a conference, but in the context of a “general topics” series at a local university (the Duale Hochschule in Karlsruhe). The talk introduces the concept of the 36 stratagems and of modern strategies to the audience. We talked a bit about the concept itself and found that the list of logical fallacies is somewhat similar. We even found an application of the stratagems in local history (sorry, only german source found): The Bretten’s Hundle
The talk itself is this monday, so you’ll need to hurry if you want to attend.

Psychology of deception

As often during the dev brunch, one topic led to the other, and we soon talked about morale and ethics. The concept of micro-expressions to reveal the hidden agenda of others came up, as well as the TV series “lie to me” that is inspired by the work of Paul Ekman, a professor of psychology. There even is a commercial training program to improve your skill of “spotting the liar”.

Games with morale aspects

Well, we are nerds. While crime investigation is thrilling, there is the even more enthralling topic of games with psychological and moralistic aspects. We soon exchanged our experiences with games like “Haze” or “Spec Ops: The Line”. But it doesn’t stop at shooter games, you can have similar insights by playing “Papers, Please” (a strong favorite for one of our next Schneide game nights) or “This War Of Mine”. You can even try some multiplayer games specifically designed for social insights, like “The Ship: Murder Party”.
And if you haven’t got much time but still want to learn something about yourself, little games like “60 Seconds!” are a great start.
This topic lead to some ideas for upcoming Schneide game nights in 2016.

Book review: A tour of C++

One attendee of the brunch provided a summary of the book “A Tour of C++” from Bjarne Stroustrup, that recently got updated to the language possibilities of C++ 11. In his words, the book is a rather incomplete introduction to the language, with way too many aspects described in a way too short manner. It’s more of a reading list to really grasp the concepts, so it may serve as a source of inspiration. For example, the notion of “move semantics” was explained, but to discover the consequences is up to the developer. The part about template programming was well done and every chapter has a suitable list of advices in the tradition of “Effective XYZ” at the end. So it’s not a bad book, but too short to be satisfying. It’s like a tourist’s tour around C++ 11, so the title holds its promise.

The left-pad incident

When we finished the “official” agenda, the topic of the recent left-pad incident came up and left us laughing. We really live in glorious times when the happiness of the (Javascript) world depends on a few lines of code. Not that this couldn’t happen in any other ecosystem.

Epilogue

As usual, the Dev Brunch contained a lot more chatter and talk than listed here. The number of attendees makes for an unique experience every time. We are looking forward to the next Dev Brunch at the Softwareschneiderei. And as always, we are open for guests and future regulars. Just drop us a notice and we’ll invite you over next time.

Modern CMake with target_link_libraries

Dependency hell?

One thing that has eluded me in the past was how to efficiently manage dependencies of different components within one CMake project. I’d use the include_directories, add_definitions and add_compile_options command in the top-level or in mid-level CMakeLists.txt files just to get the whole thing to compile. Of course, this is all heavily order-dependent – so the build system breaks as soon as you make an ever so subtle change to the directory layout. I’ve seen projects tackle this problem in various ways – for example by defining specifically named variables for each library and using that for their clients. Other projects defined “interface” files for each library that could be included by other targets. All these homegrown solutions work, but they are rather clumsy and don’t work well when integrating libraries not written in that same convention.

target_link_libraries to the rescue!

It turns out there’s actually a pretty elegant solution built into CMake, which centers around target_link_libraries. But information on this is pretty scarce on the web. The ones that initially put me on the right track were The Ultimate Guide to Modern CMake and CMake – Introduction and best practices. Of course, it’s all in the CMake documentation, but mentioned implicitly at best.

The gist is this: Using target_link_libraries to link A to an internal target B will not only add the linker flags required to link to B, but also the definitions, include paths and other settings – even transitively – if they are configured that way.

To do this, you need to use target_include_directories and target_compile_definitions with the PUBLIC or INTERFACE keywords on your targets. There’s also the PRIVATE keyword that can be used to avoid adding the settings to all dependent targets.

A simple example

Here’s a small example of a library that uses Boost in its headers and therefore wishes to have its clients setup those directories as well:

set(TARGET_NAME cool_lib)

add_library(${TARGET_NAME} STATIC 
  cool_feature.cpp cool_feature.hpp)

target_include_directories(${TARGET_NAME}
  INTERFACE ${CMAKE_CURRENT_SOURCE_DIR})

target_include_directories(${TARGET_NAME} SYSTEM
  PUBLIC ${Boost_INCLUDE_DIR})

Now here’s a program that wants to use that:

set(TARGET_NAME cool_tool)

add_executable(cool_tool main.cpp)

target_link_libraries(cool_tool
  PRIVATE cool_lib)

cool_tool can just #include "cool_feature.hpp" without knowing exactly where it is located in the source tree or without having to worry about setting up the boost includes for itself! Pretty neat!

PRIVATE, PUBLIC and INTERFACE

Typically, you’d use the PRIVATE keyword for includes and definitions that are exclusively used in you implementation, i.e. your *.cpp and *.c files and internal headers. It’s good practice to favor PRIVATE to avoid “leaking” dependencies so they won’t stack up in the dependent libraries and bring down your compile times. The INTERFACE keyword is a bit more curious: For example, with definitions, you can use it to define your .dll interface differently for compilation and usage. For include directories, one common usage is to set the own source directory with INTERFACE if you keep your headers and source files in the same folder. The PUBLIC keyword is used when definitions and includes are relevant for the own and dependent libraries. It pretty much is the combination of PRIVATE and INTERFACE – whenever you’re temped to put something in both of those, put it in PUBLIC instead. It is probably the most common option.

The future!

I hope that all open-source libraries switch to this style sooner rather than later so you can easily include them in your build-trees. Just don’t use the old commands that add properties for all following targets like add_definitions, include_directories etc. and use the commands with the target_ prefix!

All .NET assemblies for one and one for all

Sometimes you have developed a simple utility tool that doesn’t need the directory structure of a full-blown application for resources and other configuration. However, this tool might have a couple of library dependencies. On the .NET platform this usually means that you have to distribute the .dll files for the libraries along with the executable (.exe) file of the tool.

Wouldn’t it be nice to distribute your tool only as a single .exe file, so that users don’t have to drag around a lot of files when they move the tool from one location to another?

In the C++ world you would use static linking to link library dependencies into the resulting executable. For the .NET platform Microsoft provides a command-line tool called ILMerge. It can merge multiple .NET assemblies into a single assembly:

ILMerge

You can either download ILMerge from Microsoft as an .msi package or install it as a NuGet package from the package manager console (accessible in Visual Studio under Tools: Library Package Manager):

PM> Install-Package ilmerge

The basic command line syntax of ILMerge is:

> ilmerge /out:filename <primary assembly> [...]

The primary assembly would be the original executable of your tool. It must be listed first, followed by the library assemblies (.dll files) to merge. Here’s an example, which represents the scenario from the diagram above:

> ilmerge /out:StandaloneApplication.exe Application.exe A.dll B.dll C.dll

Keep in mind that the resulting executable is still dependent on the existence of the .NET framework on the system, it’s not completely independent.

Graphical user interface

There’s also a graphical user interface for ILMerge available. It’s an open-source tool by a third-party developer and it’s called ILMerge-GUI, published on Microsoft’s CodePlex project hosting platform.

ILMerge-GUI

You simply drag and drop the assemblies to merge on the designated area, choose a name for the output assembly and click the “Merge!” button.

Requirements should not drive development

For more than a decade I develop software for engineers. The software they use is coined by vendor lock-ins, terrible interfaces and dowdy technology. In the last years I am struggling to find a better way to make tools that help them to get their jobs done. In a more pleasant and efficient way. My journey started with trying to make the internal quality of the software better through practices like clean code, TDD and the like. Maybe the software got more robust but the benefit on the user side of things was negligible.
So I turned to learning user experience design (UX). A vast land of new unknown concepts, theories and practices lays before me. So big that even practitioners in the field cannot agree what UX is.
The label isn’t important to me.

I want to make the work of engineers better.

To make the work better I have to learn more about the people and the work. The importance of domain knowledge is undisputed in the software development world. And collaboration is a pillar of modern software projects. But after reading and practicing several methods to uncover information about people and their work (read: user research) I found the traditional way of communicating inside a software project lacking.
Normally a software project starts with gathering requirements. Requirements describe what the software should do in an abstract way. For example: the software should calculate the balance of the user’s bank account. How did we found out that this is a requirement? We interviewed a stakeholder.
At first sight this might sound reasonable. Looking closer we find many problems with this approach.
First in the context of enterprise software where I work the user and the stakeholder are often different people. Even more alarming is that we talk about the wants of the solution. We have no indicator what problems does this requirement try to solve. Without knowing what we want to solve we are doomed. We are imaging an illusion.
Now even if the one who made all research and wrote the requirements knows the problems he wants the software to address the requirement is a very poor communicator. We need to spread the knowledge about the problems throughout the team. The agile approach to include roles (as a), actions (I want to) and outcomes (so that) in so called user stories is not much help.
In order to know what properties a solution must have we need to understand the user and his work. For this we need to see his context, the situation he is in. In different situations he might want to do different things. So we need to capture his motivation and of course his goals.

Jobs to be done – job stories

In the search for a better way to communicate what we find out during research I stumbled across the jobs to be done methodology. Coming originally from marketing several teams adapted this way of thinking to product development. I think Paul Adams mentioned and Alan Klement developed the idea of the job story and several others have had a great success implementing it.
What’s in a job story? A job story take the form: When (situation), I want to (motivation), so I can (expected outcome).
Now the first part captures one of the most important concepts of UX: the context.

The situation

Context is so much more than I naively imagined: it is not location and surroundings. Context or better the situation is what happened before in the environment, in the system and for the user. Developers might call it the state of these. For example: When I am about to buy a car, when I start a measurement, when I am leading the race, when my application crashed, …
The situation and what happened just before is crucial to get a sense of what the user is struggling with.

The motivation

The user wants to move from the current situation to his goal. Why? And what is important for him? What might hinder him to do so? The jobs to be done framework calls this forces. We have 4 forces:

  • push of the current situation: what is bad now
  • pull of the future: what is good then
  • habits: business as usual might hinder progress
  • anxiety: the fear of the new might bring hesitation

These forces help us to document needs of the user in the current situation. If we want to better support the user we need to care about him as a person, not just as a machine going from one state to the next. Forces help us to communicate needs. Again a property the current solutions of documenting requirements lack.

The outcome

What benefit does the user get from his actions? He has a goal. We need to know his goals in his current situation to help him get there. Beware: this is not just a feature of the system, it is an understanding the user gets, an accomplishment, a symbol of status. The benefit is the value the user gets.

Conclusion

Describing situations in which people struggle to get to the wanted outcomes and goals helps us relating to them. It helps us documenting findings and communicate them to other team members. When evaluating different solutions we can lean on the job stories to determine if they fit. Requirements and user stories are just too solution focused and miss important details to drive any software development effort which wants to help people doing a better job.

Timestamps make horrible identifiers

vetre / fotoliaNot long ago, I’ve struggled with a system that uses timestamps as entity identifiers. What can I say? Timestamps aren’t meant to identify anything else than a specific point in time. Don’t use them as entity identifiers, ever. If you want to know why, I invite you to read on. The blog post is written in Freytag’s dramatic structure for added effect.

Exposition

We’ve designed a system that runs on multiple instances that communicate in all sorts of way. A central archive instance stores all data related to measurements. The whole network revolves around the notion of measurement. Measurement data is the most precious data and all instances will either produce or consume data based on these measurements.

Most important for human operators is an instance that lets you view all existing measurement data. Let’s call it the viewer. The viewer displays an overview list of all measurements in a given context and lets the operator choose to view ever more details of any of them. To be able to provide the overview list as fast as possible, we added a cache that holds the information.

Rising action

This measurement list cache was the source of all kinds of peculiar behaviour of the system. Most, but not all measurement data was incomplete. The list cache entries were assembled from different sources that were available at different times, so it seemed that while one part of the data got written to the cache, another part couldn’t be written for whatever reasons. The operator could load detailed data of some few measurements, but the majority just produced an error message that the data couldn’t be found (despite it being present).
The most obviously broken functionality left the following trace in the log files (paraphrased):

- storing measurement at 2016-02-28T13:25:55.189+01:00 into the list cache
- measurement stored
[...]
- loading measurement at 2016-02-28T13:25:55.189+01:00 from the list cache
- error: measurement not found in list cache

So, the system is essentially telling me that it can’t load some data it just stored. As you can imagine, this may lead to some questions about the sanity of the database product underneath.

Climax

After some investigation and fruitless integration testing, it dawned me: The problem wasn’t timing or the database. All the bugs could be explained with only one circumstance: Measurements were ultimately identified by their timestamp, the moment the measurement was made. There’s also a location, type and some other information in the identifier for each measurement, but only the timestamp changes between two measurements in the same narrow context. And the timestamp was stored in different precisions, depending on the origin of the measurement identifier. Most identifiers were create at the measurement producing system instances (let’s call them measurers) and had millisecond precision. As soon as they got stored in the production database (but not our development database), they lost the milliseconds. And some of the most important measurement data got exported to third-party systems, using a minute-based precision. So we had one measurement identifier in the system, but with three different types, each mostly incompatible to each other.

Falling action

That’s why the log excerpt above never occurred in development, but in production: The measurement is stored in the database, the used identifier gets passed around in the software, but a query on the exact same identifier in the database yields no result because the timestamps now differ in the millisecond range. And the strange effects that sometimes, everything worked just fine? That’s when the milliseconds are zero by chance. Given that most actions in the system are scheduled and performed automatically exactly on the zero mark, the zero milliseconds case happened more often than it would in an even distribution.

Our system dealt with three types of measurement identifiers: Millisecond-precise identifiers produced by the measurers, second-precise identifiers used by the measurement list cache and minute-precise identifiers used (and sometimes fed back into the system) by the data export. These identifiers were incompatible even for the same measurement most of the time, but not always. In unit tests, the timestamps were made up and didn’t reveal the problem properly (who thinks about odd milliseconds when making up a timestamp?).

My solution was to pull this incompatibility up into the type system. Instead of one measurement identifier, there are now three measurement identifiers: MillisecondPreciseIdentifier, SecondPreciseIdentifier and MinutePreciseIdentifier. An identifier of higher precision can be converted to an identifier of lower precision, but not the other way around. Everytime a measurement identifier is created, it needs to explicitely state its precision of the timestamp. This made the compiler highlight the problematic usages clearly as type conflicts and therefore dealing with the problem much easier.

Revelation

Choosing a timestamp as a vital part of a (measurement) identifier was a mistake from the beginning. The greater problem was the omission of the timestamp’s precision. Timestamps perform more like floating-point numbers and less like integers, even if every timestamp can be represented by a long. As soon as I made the precision of each timestamp clear to the compiler, the bugs revealed themselves. The annoying difference between developer and production database would have been detected much sooner, because a millisecond-precise timestamp will now warn in the log files if its millisecond part is zero. As soon as this log entry is seen very often, its clear that something is wrong. The new datatypes not only serve as a clearer API contract definition tool, but also as a runtime sanity check.

If you don’t want to repeat this mistake, keep in mind that each timestamp, date or whatever time-related data type you use will inherently have a maximum precision. As soon as you mix different precisions into the same data type, you’re going to have a bad time. Explicitely state the required precision in your type system and your compiler will keep an eye on it, too.