Learning about Class Literals after twenty years of Java

I’ve programmed in Java nearly every day for twenty years now. At the beginning of my computer science studies, I was introduced to Java 1.0.x and have since accompanied every version of Java. Our professor made us buy the Java Language Specification on paper (it was quite a large book even back then) and I occassionally read it like you would read an encyclopedia – wading through a lot of already known facts just to discover something surprising and interesting, no matter how small.

With the end of my studies came the end of random research in old books – new books had to be read and understood. It was no longer efficient enough to randomly spend time with something, everything needed to have a clear goal, an outcome that improved my current position. This made me very efficient and quite effective, but I only uncover surprising facts and finds now if work forces me to go there.

An odd customer request

Recently, my work required me to re-visit an old acquaintance in the Java API that I’ve never grew fond of: The Runtime.exec() method. One of my customer had an recurring hardware problem that could only be solved by rebooting the machine. My software could already detect the symptoms of the problem and notify the operator, but the next logical step was to enable the software to perform the reboot on its own. The customer was very aware of the risks for such a functionality – I consider it a “sabotage feature”, but asked for it anyway. Because the software is written in Java, the reboot should be written in Java, too. And because the target machines are exclusively running on Windows, it was a viable option to implement the feature for that specific platform. Which brings me to Runtime.exec().

A simple solution for the reboot functionality in Java on Windows looks like this:


Runtime.exec("shutdown /r");

With this solution, the user is informed of the imminent reboot and has some time to make a decision. In my case, the reboot needed to be performed as fast as possible to minimize the loss of sensor data. So the reboot command needs to be expanded by a parameter:


Runtime.exec("shutdown /r /t 0");

And this is when the command stops working and politely tells you that you messed up the command line by printing the usage information. Which, of course, you can only see if you drain the output stream of the Process instance that performs the command in the background:


final Process process = Runtime.exec("shutdown /r /t 0");
try (final Scanner output = new Scanner(process.getInputStream())) {
    while (output.hasNextLine()) {
        System.out.println(output.nextLine());
    }
}

The output draining is good practice anyway, because the Process will just stop once the buffer is filled up. Which you will never see in development, but definitely in production – in the middle of the night on a weekend when you are on vacaction.

Modern thinking

In Java 5 and improved in Java 7, the Runtime.exec() method got less attractive by the introduction of the ProcessBuilder, a class that improves the experience of creating a correct command line and a lot of other things. So let’s switch to the ProcessBuilder:


final ProcessBuilder builder = new ProcessBuilder(
        "shutdown",
        "/r",
        "/t 0");
final Process process = builder.start();

Didn’t change a thing. The shutdown command still informs us that we don’t have our command line under control. And that’s true: The whole API is notorious of not telling me what is really going on in the background. The ProcessBuilder could be nice and offer a method that returns a String as it is issued to the operating system, but all we got is the ProcessBuilder.command() method that gives us the same command line parts we gave it. The mystery begins with our call of ProcessBuilder.start(), because it delegates to a class called ProcessImpl, and more specific to the static method ProcessImpl.start().

In this method, Java calls the private constructor of ProcessImpl, that performs a lot of black magic on our command line parts and ultimately disappears in a native method called create() with the actual command line (called cmdstr) as the first parameter. That’s the information I was looking for! In newer Java versions (starting with Java 7), the cmdstr is built in a private static method of ProcessImpl: ProcessImpl.createCommandLine(). If I could write a test program that calls this method directly, I would be able to see the actual command line by myself.

Disclaimer: I’m not an advocate of light-hearted use of the reflection API of Java. But for one-off programs, it’s a very powerful tool that gets the job done.

So let’s write the code to retrieve our actual command line directly from the ProcessImpl.createCommandLine() method:


public static void main(final String[] args) throws Exception {
    final String[] cmd = {
            "shutdown.exe",
            "/r",
            "/t 0",
    };
    final String executablePath = new File(cmd[0]).getPath();

    final Class<?> impl = ClassLoader.getSystemClassLoader().loadClass("java.lang.ProcessImpl");
    final Method myMethod = impl.getDeclaredMethod(
            "createCommandLine",
            new Class[] {
                    ????, // <-- Damn, I don't have any clue what should go here.
                    String.class,
                    String[].class
            });
    myMethod.setAccessible(true);

    final Object result = myMethod.invoke(
            null,
            2,
            executablePath,
            cmd);
    System.out.println(result);
}

The discovery

You probably noticed the “????” entry in the code above. That’s the discovery part I want to tell you about. This is when I met Class Literals in the Java Language Specification in chapter 15.8.2 (go and look it up!). The signature of the createCommandLine method is:


private static String createCommandLine(
        int verificationType,
        final String executablePath,
        final String cmd[])

Note: I didn’t remove the final keyword of verificationType, it isn’t there in the original code for unknown reasons.
When I wrote the reflection code above, it occurred to me that I had never attempted to lookup a method that contains a primitive parameter – the int in this case. I didn’t think much about it and went with Integer.class, but that didn’t work. And then, my discovery started:


final Method myMethod = impl.getDeclaredMethod(
        "createCommandLine",
        new Class[] {
                int.class, // <-- Look what I can do!
                String.class,
                String[].class
        });

As stated in the Java Language Specification, every primitive type of Java conceptionally “has” a public static field named “class” that contains the Class object for this primitive. We can even type void.class and gain access to the Class object of void. This is clearly written in the language specification and required knowledge for every earnest usage of Java’s reflection capabilities, but I somehow evaded it for twenty years.

I love when moments like this happen. I always feel dumb and enlightened at the same time and assume that everybody around me knew this fact for years, it is just me that didn’t get the memo.

The solution

Oh, and before I forget it, the solution to the reboot command not working is the odd way in which Java adds quote characters to the command line. The output above is:


shutdown /r "/t 0"

The extra quotes around /t 0 make the shutdown command reject all parameters and print the usage text instead. A working, if not necessarily intuitive solution is to separate the /t parameter and its value in order to never have spaces in the parameters – this is what provokes Java to try to help you by quoting the whole parameter (and is considered a feature rather than a bug):


final String[] cmd = {
        "shutdown",
        "/r",
        "/t",
        "0",
};

This results in the command line I wanted from the start:


shutdown /r /t 0

And reboots the computer instantaneous. Try it!

Your story?

What’s your “damn, I must’ve missed the memo” moment in the programming language you know best?

Modern developer Issue #2: RPM like deployment on Windows

Deployment is a crucial step in every development project. Without shipping no one would ever see our work (and we get no feedback if our work is good).

drawer

Often we fear deploying to production because of the effort involved and the errors we make. Questions like ‘what if we forget a step?’ or ‘what if the new version we install is buggy?’ buzz in our mind.

fears

Deployment needs to be a non-event, a habit. For this we need to automate every step besides the first one: clicking a button to start deployment.

deploy

On Linux we have wonderful tools for this but what if you are stuck with deploying to Windows?

brave

Fear not, brave developer! Even on Windows we can use a package manager to install and rollback buggy versions. Let me introduce you to chocolatey.

choco

Chocolatey (or choco in short) uses the common NuGet package format. Formerly developed for the .net platform we can use it for other platforms, too. In our following example we use a simple Java application which we install as a service and as a task.
Setting up we need a directory structure for the package like this:

folders

We need to create two files: one which specifies our package (my_project.nuspec) and one script which holds the deployment steps (chocolateyinstall.ps1). The specification file holds things like the package name, the package version (which can be overwritten when building the package), some pointers to project, source and license URLs. We can configure files and directories which will be copied to the package: in our example we use a directory containing our archives (aptly named archives) and a directory containing the installation steps (named tools). Here is a simple example:

<?xml version="1.0" encoding="utf-8"?>
<!-- Do not remove this test for UTF-8: if “Ω” doesn’t appear as greek uppercase omega letter enclosed in quotation marks, you should use an editor that supports UTF-8, not this one. -->
<package xmlns="http://schemas.microsoft.com/packaging/2015/06/nuspec.xsd">
  <metadata>
    <id>my_project</id>
    <title>My Project (Install)</title>
    <version>0.1</version>
    <authors>Me</authors>
    <owners>Me</owners>
    <summary></summary>
    <description>Just an example</description>
    <projectUrl>http://localhost/my_project</projectUrl>
    <packageSourceUrl>http://localhost/git</packageSourceUrl>
    <tags>example</tags>
    <copyright>My company</copyright>
    <licenseUrl>http://localhost/license</licenseUrl>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <releaseNotes></releaseNotes>
  </metadata>
  <files>
    <file src="tools\**" target="tools" />
    <file src="archives\**" target="archives" />
  </files>
</package>

This file tells choco how to build the packages and what to include. For the deployment process we need a script file written in Powershell.

powershell

A Powershell primer

Powershell is not as bad as you might think. Let’s take a look at some basic Powershell syntax.

Variables

Variables are started with a $ sign. As in many other languages ‘=’ is used for assignments.

$ErrorActionPreference = 'Stop'

Strings

Strings can be used with single (‘) and double quotes (“).

$serviceName = 'My Project'
$installDir = "c:\examples"

In double quoted strings we can interpolate by using a $ directly or with curly braces.

$packageDir = "$installDir\my_project"
$packageDir = "${installDir}\my_project"

For escaping double quotes inside a double quoting string we need back ticks (`)

"schtasks /end /f /tn `"${serviceName}`" "

Multiline strings are enclosed by @”

$cmdcontent = @"
cd /d ${packageDir}
java -jar ${packageName}.jar >> output.log 2>&1
"@

Method calls

Calling methods looks a mixture of command line calls with uppercase names.

Write-Host "Stopping and deleting current version of ${packageName}"
Get-Date -format yyyyddMMhhmm
Copy-Item $installFile $packageDir

Some helpful methods are:

  • Write-Host or echo: for writing to the console
  • Get-Date: getting the current time
  • Split-Path: returning the specified part of a path
  • Join-Path: concatenating a path with a specified part
  • Start-Sleep: pause n seconds
  • Start-ChocolateyProcessAsAdmin: starting an elevated command
  • Get-Service: retrieving a Windows service
  • Remove-Item: deleting a file or directory
  • Test-Path: testing for existence of a path
  • New-Item: creating a file or directory
  • Copy-Item: copying a file or directory
  • Set-Content: creating a file with the specified contents
  • Out-Null: swallowing output
  • Resolve-Path: display the path after resolving wildcards

The pipe (|) can be used to redirect output.

Conditions

Conditions can be evaluated with if:

if ($(Get-Service "$serviceName" -ErrorAction SilentlyContinue).Status -eq "Running") {
}

-eq is used for testing equality. -ne for difference.

Deploying with Powershell

For installing our package we need to create the target directories and copy our archives:

$packageName = 'myproject'
$installDir = "c:\examples"
$packageDir = "$installDir\my_project"

Write-Host "Making sure $installDir is in place"
if (!(Test-Path -path $installDir)) {New-Item $installDir -Type Directory  | Out-Null}

Write-Host "Making sure $packageDir is in place"
if (!(Test-Path -path $packageDir)) {New-Item $packageDir -Type Directory  | Out-Null}

Write-Host "Installing ${packageName} to ${packageDir}"
Copy-Item $installFile $packageDir

When reinstalling we first need to delete existing versions:

$installDir = "c:\examples"
$packageDir = "$installDir\my_project"

if (Test-Path -path $packageDir) {
  Remove-Item -recurse $(Join-Path $packageDir "\*") -exclude *.conf, *-bak*, *-old*
}

Now we get to the meat creating a Windows service.

$installDir = "c:\examples"
$packageName = 'myproject'
$serviceName = 'My Project'
$packageDir = "$installDir\my_project"
$cmdFile = "$packageDir\$packageName.cmd"

if (!(Test-Path ($cmdFile)))
{
    $cmdcontent = @"
cd /d ${packageDir}
java -jar ${packageName}.jar >> output.log 2>&1
"@
    echo "Dropping a ${packageName}.cmd file"
    Set-Content $cmdFile $cmdcontent -Encoding ASCII -Force
}

if (!(Get-Service "${serviceName}" -ErrorAction SilentlyContinue))
{
  echo "No ${serviceName} Service detected"
  echo "Installing ${serviceName} Service"
  Start-ChocolateyProcessAsAdmin "install `"${serviceName}`" ${cmdFile}" nssm
}

Start-ChocolateyProcessAsAdmin "set `"${serviceName}`" Start SERVICE_DEMAND_START" nssm

First we need to create a command (.cmd) file which starts our java application. Installing a service calling this command file is done via a helper called nssm. We set it to starting manual because we want to start and stop it periodically with the help of a task.

For enabling a reinstall we first stop an existing service.

$installDir = "c:\examples"
$serviceName = 'My Project'
$packageDir = "$installDir\my_project"

if (Test-Path -path $packageDir) {
  Write-Host $(Get-Service "$serviceName" -ErrorAction SilentlyContinue).Status

  if ($(Get-Service "$serviceName" -ErrorAction SilentlyContinue).Status -eq "Running") {
    Start-ChocolateyProcessAsAdmin "Stop-Service `"${serviceName}`""
    Start-Sleep 2
  }
}

Next we install a task with help of the build in schtasks command.

$serviceName = 'My Project'
$installDir = "c:\examples"
$packageDir = "$installDir\my_project"
$cmdFile = "$packageDir\$packageName.cmd"

echo "Installing ${serviceName} Task"
Start-ChocolateyProcessAsAdmin "schtasks /create /f /ru system /sc hourly /st 00:30 /tn `"${serviceName}`" /tr  `"$cmdFile`""

Stopping and deleting the task enables us to reinstall.

$packageName = 'myproject'
$serviceName = 'My Project'
$installDir = "c:\examples"
$packageDir = "$installDir\my_project"

if (Test-Path -path $packageDir) {
  Write-Host "Stopping and deleting current version of ${packageName}"
  Start-ChocolateyProcessAsAdmin "schtasks /delete /f /tn `"${serviceName}`" "
  Start-Sleep 2
  Start-ChocolateyProcessAsAdmin "schtasks /end /f /tn `"${serviceName}`" "
  Remove-Item -recurse $(Join-Path $packageDir "\*") -exclude *.conf, *-bak*, *-old*
}

tl;dr

Putting it all together looks like this:

$ErrorActionPreference = 'Stop'; # stop on all errors

$packageName = 'myproject'
$serviceName = 'My Project'
$installDir = "c:\examples"
$packageDir = "$installDir\my_project"
$cmdFile = "$packageDir\$packageName.cmd"
$currentDatetime = Get-Date -format yyyyddMMhhmm
$scriptDir = "$(Split-Path -parent $MyInvocation.MyCommand.Definition)"
$installFile = (Join-Path $scriptDir -ChildPath "..\archives\$packageName.jar") | Resolve-Path


if (Test-Path -path $packageDir) {
  Write-Host "Stopping and deleting current version of ${packageName}"
  Start-ChocolateyProcessAsAdmin "schtasks /delete /f /tn `"${serviceName}`" "
  Start-Sleep 2
  Start-ChocolateyProcessAsAdmin "schtasks /end /f /tn `"${serviceName}`" "
  Remove-Item -recurse $(Join-Path $packageDir "\*") -exclude *.conf, *-bak*, *-old*

  Write-Host $(Get-Service "$serviceName" -ErrorAction SilentlyContinue).Status

  if ($(Get-Service "$serviceName" -ErrorAction SilentlyContinue).Status -eq "Running") {
    Write-Host "Stopping and deleting current version of ${packageName}"
    Start-ChocolateyProcessAsAdmin "Stop-Service `"${serviceName}`""
    Start-Sleep 2
  }

  if ($(Get-Service "$serviceName"  -ErrorAction SilentlyContinue).Status -ne "Running") {
    Write-Host "Cleaning ${packageDir} directory"
    Remove-Item -recurse $(Join-Path $packageDir "\*") -exclude *.conf, *-bak*, *-old*
  }
}
 
Write-Host "Making sure $installDir is in place"
if (!(Test-Path -path $installDir)) {New-Item $installDir -Type Directory  | Out-Null}

Write-Host "Making sure $packageDir is in place"
if (!(Test-Path -path $packageDir)) {New-Item $packageDir -Type Directory  | Out-Null}

Write-Host "Installing ${packageName} to ${packageDir}"
Copy-Item $installFile $packageDir

if (!(Test-Path ($cmdFile)))
{
    $cmdcontent = @"
cd /d ${packageDir}
java -jar ${packageName}.jar >> output.log 2>&1
"@
    echo "Dropping a ${packageName}.cmd file"
    Set-Content $cmdFile $cmdcontent -Encoding ASCII -Force
}

if (!(Get-Service "${serviceName}" -ErrorAction SilentlyContinue))
{
  echo "No ${serviceName} Service detected"
  echo "Installing ${serviceName} Service"
  Start-ChocolateyProcessAsAdmin "install `"${serviceName}`" ${cmdFile}" nssm
}

Start-ChocolateyProcessAsAdmin "set `"${serviceName}`" Start SERVICE_DEMAND_START" nssm

echo "Installing ${serviceName} Task"
Start-ChocolateyProcessAsAdmin "schtasks /create /f /ru system /sc hourly /st 00:30 /tn `"${serviceName}`" /tr  `"$cmdFile`""

Finally

Now we just need to create the package in our build script. The package will be named my_project.version.nupkg.
On our build machine we need to install choco. On the target machine we need the following tools installed:
chocolatey and nssm (for service management). Now we can create the package with:

  choco pack --version=${version}

Copy it to the target machine and install the current version with:

choco install -f -y c:\\installations\\${archive.name} --version=${version}

Put these steps inside a build script and use your favourite contininuous integration platform and voila.
Done.

deploy

Ugly problems, ugly solutions?

One type of our projects is to integrate some devices into our customers infrastructure. The tasks then mostly consist of writing bridging code for third party libs of the hardware vendor. The most fun part is when the libs do not have some needed capability or feature.

The situation

In my case I was building a device driver with following requirements:

  • asynchronous execution of long running tasks.
  • ability to cancel long running tasks.
  • at any time it is asked for its current status, it has to provide it.

The device is accompanied by a DLL with a following interface(simplified):

  • doWork(), a blocking function that returns after a configurable amount of time that can range from milliseconds to hours.
  • abortWork(), is supposed to cancel the process triggered by doWork() and to make doWork() return earlier.

First impressions

I was able to fullfill two requirements pretty fast. The state ist more or less a simple getter and the doWork function was called in a separate thread. Just cancelling the execution didn’t work. More precisely it didn’t work as expected. In the time between a call to doWork() and the moment it returned, the process always used 100% of one CPU core. After that it always dropped to nearly zero. Now, what happened, when I called abortWork()? There were two things: doWork() returned, but the CPU utilization stayed the same for an indefinite amount of time. Or the call was ignored completely. Especially funny was the first case, where the API seemed to work until the process run out of cores and the system practically grinded to a halt.

The “Solution”

Banging my head against the desk didn’t help, so my first thought was to forget abortWork() and kill the thread myself. Microsoft provides a nice function called TerminateThread for that purpose. Everyone who looks at the documentation, will see that the list of side effects is quite impressive, memory leaks being the least bad ones. I couldn’t guarantee that the application would work afterwards, so I decided against it. What would be the alternative? Process shutdown. When you stop the process all blocked threads should be away. Being too soft and trying to unload the DLL is a bad idea – you have a deadlock when the DllMain waits for the worker thread to finish. My last attempt was to suicide the process!

Now I was able to abort a running task, but my app were no longer available all the time. Every attempt to get the current status between the start of a shutdown and a completed startup failed. So a semi-persistent storage containing the last status of a living application was needed. To achieve this, I created an application with the same interface as the real device driver and proxy that delegated all the requests to it, caching the status responses. That way the polling application still assumed that the last action were still running until the restarting app was fully available again.

In the end the solution consisted of two device drivers, one for caching the state and the other for doing the work. When cancelling the task was required, the latter device driver died and restarted itself again.

Final thoughts

I hope that there is a way to do this in a more elegant way and I just overlooked some facts. It is unbelievable that you can lose all control over your app by a simple call to a third party library and that the only escape is death.