Documentation for your project: what and how

July 28, 2014

Writing documentation is seldom fun for developers and much useless documentation is written. I want to provide some guidelines helping to focus your project documentation efforts on useful stuff instead of following a set of dogmatic rules to plainly fulfill requirements.

Code Documentation

Probably written many times before but nevertheless often neglected:

  • Avoid untouched documentation templates, e.g. // This is a getter for A. They only clutter the code hurting developers instead of providing value.
  • Do not document every class, method, file etc. blindly. Focus on all API classes et al. to be used by other (external) developers.
  • Do not document what the code does, it should speak for itself. Rather explain why a certain algorithm or data structure is used. Try to communicate design decisions.
  • Check comments everytime you touch documented code and update them if necessary. Outdated documentation hurts more than its worth so if docs exists keep them up-to-date.

Project Documentation

This kind of documentation usually provides more value than many javadoc/doxygen generated pages. Nowadays, many people use a wiki software for project documentation. I encourage you to use a powerful wiki like Confluence because it provides rich formatting options and templating allowing for visually pleasing and expressive documentation. As such it may be even printed (to PDF) and handed out to your customers.

  • Putting parts like Installation into the code repository and integrating them into the wiki often serves administrators, managers (visibility!) and developers. See my older post “centralized project documentation” for some other ideas.
  • Wikis allow for easy editing and document sharing and are version controlled. All this facilitates reviews and updates of the documents.
  • Document prerequisites and external dependencies explicitly. They may be hard to find in configuration files but are of good use to people running your project.
  • Improve  searches in the wiki by providing tags and other metadata to help your future me and others finding the information they are looking for.
  • Provide consistent examples or even templates for common documentation tasks to encourage others and help them getting their project documentation started.

Conclusion

Good documentation is a real asset and can provide much value if you keep your efforts focused on the important stuff. Complex workflows and draconic rules will hinder documentation efforts wheres open collaboration and valuable documentation will motivate bringing more of it into existence.


When UTF8 != UTF8

July 21, 2014

Problem

Recently I encountered a problem with umlauts in file names. I had to read names from a directory and find and update the appropriate entry in the database. So if I had a file named hund.pdf (Hund is German for dog) I had to find the corresponding record in the database and attach the file. Almost all files went smooth but the ones with umlauts failed all.

Certainly an encoding problem I thought. So I converted the string to UTF-8 before querying. Again the query returned an empty result set. So I read up on the various configuration options for JDBC, Oracle and Active Record (it is a JRuby on Rails based web app). I tried them all starting with nls_language and ending with temporary setting the locale. No luck.

Querying the database with a hard coded string containing umlauts worked. Both strings even printed on the console looked identically.

So last but not least I compared the string from the file name with a hard coded one: they weren’t equal. Looking at the bytes a strange character combination was revealed \204\136. What’s that? UTF8 calls this a combining diaeresis. What’s that? In UTF8 you can encode umlauts with their corresponding characters or use a combination of the character without an umlaut and the combining diaeresis. So ‘ä’ becomes ‘a\204\136′.

Solution

The solution is to normalize the string. In (J)Ruby you can achieve this in the following way:

string = string.mb_chars.normalize.to_s

And in Java this would be:

string = Normalizer.normalize(string, Normalizer.Form.NFKC)

Ruby uses NFKC (or kc for short) as a default and suggests this for databases and validations.

Lesson learned: So the next time you encounter encoding problems look twice it might be in the right encoding but with the wrong bytes.


The various ways of error handling

July 14, 2014

There are various approaches and philosophies regarding error handling in different programming languages. This article tries to give an overview.

Exceptions

Most of the current mainstream programming languages use exceptions for error handling. When an exception is raised (“thrown”), the call stack is being unwound until the exception is caught. The functions passed along the way through the call stack have to ensure that any opened resources are properly closed. This is usually done via finally blocks. If the exception is not caught the program terminates.

Exceptions come in two flavors: checked exceptions and unchecked exceptions. The handling of checked exceptions is enforced by the compiler. Checked exceptions are part of the function signatures. A function explicitly declares in its signature what exceptions can be thrown:

void f() throws A, B, C

The caller of a function has to either handle the exceptions (fully or partially) or let them pass through by re-declaring them in the throws clause of its own function signature.

Checked exceptions have the property that it’s hard to forget to handle them. However, proponents of unchecked exceptions argue that checked exceptions have two problems: versioning and scalability.

Once declared they are part of the interface and adding another exception will break all client code. Multiple exception types also tend to accumulate the more different subsystems are being aggregated. Proponents of unchecked exceptions prefer a catch-all clause further up the call stack. Some languages (e.g. Erlang) even follow a “let it crash” paradigm and simply respawn crashed processes. This approach is more viable in distributed systems than in user-facing applications.

Java is known for its checked exceptions. C#, C++, Scala and most dynamically typed languages decided to go with unchecked exceptions.

No exceptions

An alternative to exceptions is no exceptions. Exceptions overlay multiple different control flows, which makes it harder to reason about the control flow of a function. With exceptions functions can return at many other points than the explicit return points.

If an error is just a value that is returned by a function it can be handled by the usual control flow mechanisms of a language (like if and else) without the need of a special sub-language for error handling. These errors tend to be handled closer to the place of their occurrence rather than further up the call stack.

In such a language, which uses return values to flag errors, you’d better check all errors, otherwise you risk continuing with an incorrect, invalid or meaningless value. This can be enforced either by the compiler or via a lint tool.

There are different possibilities of how an error could be returned from a function:

In C a sentinel value in the range of the return type is often used to indicate an error, e.g. a negative value or zero. This is not a good solution, because it intermingles two things that do not belong together and it limits the range of valid return values. Another solution in C could be the use of an error output parameter. Prominent examples are NSError in Objective C or GError in GLib. This brings us to another possibility:

Some languages support multiple return values (e.g. Go) or tuple types (product types), which can act as multiple return values. One value can hold the actual result (e.g. number of bytes written), the other can indicate an error.

Multiple return types / product types are a simple solution, cover the necessary use cases and require little additional language support. A more sophisiticated and more restrictive solution are sum types, but they require a little bit more language support: instead of returning a value AND a possible error, a function returns either a value OR an error. This way the programmer is forced to check for an error by discriminating between the two cases. This is usually done via a feature called structural pattern matching (not to be confused with pattern matching on strings), either explicitly with a switch/case-like control structure or implicitly via convenience function. A popular example is Haskell’s Maybe monad or the Option or similarly named type in some other languages (e.g. Scala, Standard ML, OCaml).

 


Finally: return considered harmful

July 7, 2014

Ok, calm down already – I’m not going to take your return statement away. It’s a nice little statement that gets rarer and rarer,  mostly because many modern languages allow for implicit return values, e.g. in Scala, everything is an expression and the last expression of a method is its return value – except when it isn’t. You dont need to write an explicit return statement anymore, so our laziness takes over and we omit it.

And I’m not argueing that a finally block is a bad thing – quite the contrary. Even in the age of try-with-resources and autoclose closures, a well-crafted finally block is a beautiful piece of code at its right place.

But put the two together and you’ve got a recipe for desaster, at least in Java (try it in your language of choice with care). Let’s have a look at code:

public void doSomething() {
    try {
        throw new Error("drama!");
    } finally {
        // dispose resources here
    }
}

Ok, I could have spared you the drama of an Error and just thrown a RuntimeException (or a declared and checked Exception), but it’ll soon illustrate the effect all the better.

First, the code does exactly what it should: if you call the method, you’ll get to catch an Error and all resources that were used inside the method are cleaned up right before you catch it. And since the method has no return value, you don’t miss out there.

But what if I want to give one of those endangered return statements a new home and insert it into the finally block?

public void doSomething() {
    try {
        throw new Error("drama!");
    } finally {
        // dispose resources here
        return;
    }
}

Not much has changed. I’m still not missing any return value and the Error still gets thrown. The resources are cleaned up quite as thorough as before, but something important has changed: You won’t ever see the Error. You won’t catch it because it gets eaten by that return statement at the end of the finally block.

So in short: return statements are hungry little beasts that will eat your Throwables if you corral them inside a finally block.

No wonder nobody wants to have them in their code if they behave like that.

Oh, and rest assured that you will be warned: All modern IDEs will point you to the inevitable desaster waiting to happen, if only with a compiler warning.

But where do Throwables go when they got eaten? Good question. They don’t show up on any UncaughtExceptionHandler, they don’t even stay in memory. Probably, they just are digested, never to be seen again.

If you don’t regard return statements as a little more dangerous now, you probably also raise switch-statements for fun.


Grails Update from 2.2 to 2.3

June 30, 2014

An update in the minor version does not seem like a big step but this is one brought a lot of changes, so here a step by step guide which highlights some pitfalls.

First update the version of Grails in your application properties:

app.grails.version=2.3.8

The tomcat and hibernate plugins now have versions of their respective frameworks and not the version number of Grails:

plugins.tomcat=7.0.52.1
plugins.hibernate=3.6.10.13

Grails 2.3 has a new databinding mechanism. To use the old one, especially if you use custom property editors you have to add this option to your Config.groovy:

grails.databinding.useSpringBinder = true

But even with the old databinding something changed. The field id is not bound in command objects you need to bind id explicitly:

def action = { MyCommand command ->
  command.id = params['id']?.toLong()
}

Besides the databinding mechanism also the dependency resolving changed. But you can use the old ivy mechanism by including this in BuildConfig.groovy:

grails.project.dependency.resolver="ivy"

Nonetheless all dependencies must be declared in application.properties or BuildConfig.groovy. If you have a lib directory with local jars in your application you need to add this to your repositories as a local directory:

grails.project.dependency.resolution = {
    repositories {
        flatDir name:'myRepo', dirs:'lib'
    }
}

When you have all dependencies declared your application should start.

Tests

Grails 2.3 features a new test mode: forking. This causes some problems and is better to be deactivated in BuildConfig.groovy:

grails.project.fork = [
        test: false,
]

With the new version only JUnit4 style tests are supported. This means that you don’t extend GroovyTestCase or GrailsUnitTestCase. All rules must be public and non static. All tests methods need to be annotated with @Test. Set up methods are annotated with @Before and must be public. The tearDown methods must also be annotated with @After and be public. A bug in Grails prevents you from naming the set up and tear down methods freely: the names must be setUp and tearDown. All test methods must be public void, the old def declaration is not supported anymore. Now without extending GroovyTestCase you lose the assertion methods and need to add a static import:

import static groovy.util.GroovyTestCase.*

Unit Tests

All tests should be annotated with @TestMixin([GrailsUnitTestMixin]). If you need to mock domain classes you change mockDomain to @Mock:

class MyTest {
	public void testThis() {
      mockDomain(MyDomainClass, [mdc])
  }
}
@Mock([Proposal])
class MyTest {
	public void testThis() {
      mdc.save()
  }
}

Configuration is now already mocked and your properties can be added easily:

config.my.property.value.is='123'

Integration tests

As mentioned before setUp method naming has a bug: you have to name them setUp otherwise the changes to your database aren’t rollbacked.

Acceptance Tests with Selenium

You need to patch the Remote Control Plugin because of a ClassNotFoundException. Add an additional constructor to RemoteControl.groovy to support setting the classloader:

RemoteControl(ClassLoader loader) {
  super(new HttpTransport(getFunctionalTestReceiverAddress(), loader), loader)
} 

In your tests you call this new constructor with the classloader of your class:

new RemoteControl(getClass().classLoader)

Using a Groovy Mixin in your application does not work in your tests and need to be replaced with grails.util.Mixin. But this only works in one way: the target class can access the mixin but the mixin not the target class. For this to work you need to let your mixin implement MixinTargetAware and declare a field named target:

class MyMixin implements MixinTargetAware {
	def target
}

Subtle changes and pitfalls

If you have a classname with a Controller suffix and a corresponding test but which isn’t a Grails controller Grails nevertheless tries to mock the class in your unit tests. If you rename the test to something without controller everything works fine.

In our pre 2.3 project we had some select tags in our views and used fieldValue for the selection:

<g:select value="${fieldValue(bean: object, field: 'value')}">

But now the select tag uses equals which fails if the values aren’t Strings. Just use the unescaped value:

<g:select value="${object?.value}">

I hope this guide and hints help others to avoid the headaches when upgrading your Grails application.


Recap of the Schneide Dev Brunch 2014-06-22

June 23, 2014

brunch64-borderedYesterday, we held another Schneide Dev Brunch at last. The Dev Brunch is a regular brunch on a sunday, only that all attendees want to talk about software development and various other topics. If you bring a software-related topic along with your food, everyone has something to share. The brunch was smaller this time, but we held the last brunch only three weeks ago. We had bright sunny weather and used our roof garden, but hurdled in the shadows. There were lots of topics and chatter. As always, this recapitulation tries to highlight the main topics of the brunch, but cannot reiterate everything that was spoken. If you were there, you probably find this list inconclusive:

Student again – from full employee to university

One of our attendees worked as a full-time software developer in the recent years and decided to study again. He told us about the practical challenges of an employee turned student:

  • The bureaucracy at universities is highly developed and not on your side. It takes days to accomplish the tiniest step towards matriculation.
  • To listen again. In the developer world, two hours of highly concentrated programming is satisfying, but two hours of concentrated listening to somebody who tells the important stuff only once is very hard. You are allowed to doze off (just like in big meetings), but it won’t do you no good.
  • Higher-level mathematics. Suddenly, all that stuff about fourier transformation and matrices is very important again.
  • Running on a lower gear. It seems like heaven to replace a 40h work week with a 20h study week, but the irregular pace (one day no lectures, one day lectures around the clock, etc.) will take its toll.
  • Self-organization. Good developers are of course self-organized and know what to do: work on the most important issues in your issue tracker/todo list. But university will not write issues for you and you are the one to fill the todo list. We joked that a “master’s student JIRA” would actually be a good idea.

It was a very entertaining talk and we digressed lots of times. Let’s have a look at some artifacts we came across during our discussion:

  • There seems to be a growing influence from military concepts on management. On book was specifically mentioned: Turn the ship around” by David Marquet.
  • “Bad work, good work and great work”. It’s a marketing video, but contains a message nonetheless. One practical advice is to not include “bad work knowledge” in your curriculum vitae, even if you have expertise in it. This minimizes the risk that your next job will contain a lot of “bad work” again.

The current state of JavaFX

Last year, JavaFX was aggressively marketed by Oracle as the next big thing in desktop UI. The claims and promises seem to finally be fulfilled. The combination of Java 8 and the latest JavaFX is especially joy-bringing. The JavaFX core is included in Java 8 and brings a lot of little but essential improvements. You can layout and design your graphical interfaces with a WYSIWYG editor and store it in XML-based layout files. These layouts are loaded, combined with custom logic and bound to custom data sources. The styling is based on a slightly outdated CSS dialect, but very powerful and done right in comparison to styling in past toolkit like Swing or SWT.

The best thing about JavaFX is that much less GUI code is needed for more pleasant user experiences. The toolkit feels alive and the details tell that the developers care and eat their own dogfood. Integration in the Eclipse IDE is enhanced by the e(fx)clipse project.

Our attendee has hands-on experience with Swing, SWT/JFace and JavaFX. If pressed to choose the technology for a new project, he would choose JavaFX anytime now.

Efficiency killers

We also traded tales about the most efficient efficiency killers in software development that we actually observed or endured:

  • Taking away notebooks and desktop computers and replacing them with zero clients – for developers that really need their multi-cores and gigabytes.
  • Restricting every employee to one (and only one!) computer. If you happen to choose a notebook, you probably don’t need that extra monitor, do you?
  • Installing a “privilege management” software. Basically, this software works like a firewall against user inputs. You want to install a new printer driver? It will be cheaper and faster to carve your text in stone slabs.
  • Fragmenting the company networks. This is actually a very good idea. You can have a wild-west style network for developers and a “privilege managed” one for management. It gets complicated when you need to cross the canyons everytime to get work done. Just imagine that your repository is behind a firewall and you need a clearance every time you want to commit/push.

Introduction to warfare

The last main topic was an overview of a self-study on warfare. And because the typical software developer won’t move whole armies around, it concentrated more on the principles and strategies of “common” warfare, which includes everyday conflicts as well as campaigns for a certain goal (e.g. establishing a technology). Three books serve as stepping stones:

  • The art of war” by Sun Tzu. The ancient classic book about warfare. There are probably a dozen different translations of the original chinese text, but they will only serve as a starter. This book alone will probably don’t give you deep insights, but you will come back to it often when you venture deeper into the mindset of warriors and generals.
  • Die Kunst der List” by Harro von Senger. This german book describes the political and rhetorical battle moves that are often used in everyday life. These so-called strategems can be identified and parried if you know about them. Most of us can react to some strategems by learnt lessons, but it certainly helps to be keen about the rest of them, too.
  • The 33 Strategies of War” by Robert Greene. This book doesn’t mess around. It is part of the “immoral series” of books about topics that don’t get discussed in this clarity often. You’ll learn much about strategies, their actual application in history (not only on the battlefield, but on movie sets, offices and political stages) and how to counter them. The book has a lot of content and many ideas and concepts to think about. And it contains an exhaustive list of literature to continue reading.

Is TDD dead?

We continued our discussion about the debate around David Heinemeier Hansson’s frontal attack on the hype around Test Driven Development. There were some new insights and an improved understanding about the manyfold contents of the hangout discussions.

Epilogue

As usual, the Dev Brunch contained a lot more chatter and talk than listed here. The high number of attendees makes for an unique experience every time. We are looking forward to the next Dev Brunch at the Softwareschneiderei. And as always, we are open for guests and future regulars. Just drop us a notice and we’ll invite you over next time.


Using Rails with a legacy database schema – Part 2

June 16, 2014

Part one of this blog post mini-series showed how to override default table names and primary key names in ActiveRecord model classes, and how to define alias attributes for legacy column names.

This part will discuss some options for primary key definitions in the schema file, which are relevant for legacy schemas, as well as primary key value generation on Oracle databases.

Primary key schema definition

The database schema definition of a Rails application is usually provided in a file called schema.rb via a simple domain specific language.

The create_table method implicitly adds a primary key column with name id (of numeric type) by default.

create_table 'users' do |t|
  t.string 'name', limit: 20
  # ...
end

If the primary key has a different name you can easily specify it via the primary_key option:

create_table 'users', primary_key: 'user_key' do |t|
  t.string 'name', limit: 20
  # ...
end

But what if a table has a primary key of non-numeric type? The Rails schema DSL does not directly support this. But there’s a workaround: you can set the id option of create_table to false, declare the primary key column like an ordinary non-nullable column, and add the primary key constraint afterwards via execute.

create_table 'users', id: false do |t|
  t.string 'user_key', null: false
  t.string 'name', limit: 20
  # ...
end
execute 'ALTER TABLE user ADD PRIMARY KEY (user_key)'

Primary key value generation

On Oracle databases new primary key values are usually created via sequences. The Oracle adapter for ActiveRecord assumes sequence names in the form of table name + “_seq”.  You can override this default sequence name in a model class via the sequence_name property:

class User < ActiveRecord::Base
  self.sequence_name = 'user_sequence'
  # ...
end

Sometimes primary key values are auto-generated via triggers. In this case you need the Oracle Enhanced adapter, which is a superset of the original ActiveRecord Oracle adapter, but with additional support for working with legacy databases. Now you can set the sequence_name property to the value :autogenerated:

class User < ActiveRecord::Base
  self.sequence_name = :autogenerated
  # ...
end

This circumvents the default convention and tells the adapter to not include primary key values in generated INSERT statements.


Follow

Get every new post delivered to your Inbox.

Join 82 other followers