Grails and the query cache

Look at the following code:

class Node {
  Node parent
  String name
  Tree tree
}

Tree tree = new Tree()
Node root = new Node(name: 'Root', tree: tree)
root.save()
new Node(name: 'Child', parent: root, tree: tree).save()

What happens when I query all nodes by tree?

List allNodesOfTree = Node.findAllByTree(tree, [cache: true])

Of course you get 2 nodes, but what is the result of:

allNodesOfTree.contains(Node.get(rootId))

It should be true but it isn’t all the time. If you didn’t implement equals and hashCode you get an instance equals that is the same as ==.
Hibernate guarantees that you get the same instance out of a session for the same domain object. (Node.get(rootId) == Node.get(rootId))

But the query cache plays a crucial role here, it saves the ids of the result and calls Node.load(id). There is an important difference between Node.get and Node.load. Node.get always returns an instance of Node which is a real node not a proxy. For this it queries the session context and hits the database when necessary. Node.load on the other hand never hits the database. It returns a proxy and only when the session contains the domain object it returns a real domain object.

So allNodesOfTree returns

• two proxies when no element is in the session
• a proxy and a real object when you call Node.get(childId) beforehand
• two real objects when you call get on both elements first

Deactivating the query cache globally or for this query only, returns two real objects.

Upgrading your app to Grails 2.0.0? Better wait for 2.0.1

Grails 2.0.0 is a major step forward for this popular and productive, JVM-based web framework. It has many great new features that make you want to migrate existing projects to this new version.

So I branched our project and started the migration process. Everything went smoothly and I had only to fix some minor compilation problems to get our application running again. Soon the first runtime errors occured and approximately 30 out of over 70 acceptance tests failed. Some analysis showed three major issue categories causing the failures:

1. Saving domain objects with belongsTo() associations may fail with a NULL not allowed for column "AUTHOR_ID"; SQL statement: insert into book (id, version, author_id, name) values (null, ?, ?, ?) [90006-147] message due to grails issue GRAILS-8337. Setting the other direction of the association manually can act as a workaround:

   book.author.book = book

2. When using the MarkupBuilder with the img tag in your TabLibs, your images may disappear. This is due to a new img closure defined in ApplicationTagLib. The correct fix is using

   delegate.img

   in your MarkupBuilder closures. See GRAILS-8660 for more information.

3. Handling of null and the Groovy NullObject seems to be broken in some places. So we got org.codehaus.groovy.runtime.typehandling.GroovyCastException: Cannot cast object 'null' with class 'org.codehaus.groovy.runtime.NullObject' to class 'Note' using groovy collections’ find() and casting the result with as:

    Note myNote = notes?.find {it.title == aTitle} as Note

   Removing type information and the cast may act as a workaround. Unfortunately, we are not able to reproduce this issue in plain groovy and did not have time to extract a small grails example exhibiting the problem.

These bugs and some other changes may make you reconsider the migration of some bigger project at this point in time. Some of them are resolved already so 2.0.1 may be the release to wait for if you are planning a migration. We will keep an open eye on the next releases and try to switch to 2.0.x when our biggest show stoppers are resolved.

Even though I would advise against migrating bigger existing applications to Grails 2.0.0 I would start new projects on this – otherwise great – new platform release.

Grails 2.0.0 Update: Test Problems

Grails 2 has some nice improvements over the previous 1.3.x versions and we thought we give it a try. Upgrading our application and its 18 plugins went smooth (we already used the database migration plugin). The application started and ran without problems. The better console output and stacktraces are a welcomed improvement. So all in all a pleasant surprise!
So just running the tests for verification and we can commit to our upgrade branch. Boom!

junit.framework.AssertionFailedError:
No more calls to 'method' expected at this point. End of demands.

Looking at the failing unit test showed that we did not use any mock object for this method call. Running the test alone let it pass. Hhhmm seems like we hit GRAILS-8530. The problem even exists between unit and integration tests. So when you mock something in your unit test it is also mocked in the integration tests which are run after the unit tests.
Even mocking via Expando metaclass and the map notation did not work reliably. So upgrading for us is not viable at the moment.

HTTP Get: The problem with Percent Encoded Parameters

Encoding problems are common place in software development but sometimes you get them in unexpected places.
About the setup: we have a web application written in Grails (though the choice of framework here doesn’t really matter) running on Tomcat. A flash application sends a HTTP Get request to this web application.
As you might know parameters in Get request are encoded in the URL with the so called percent encoding for example: %20 for space. But how are they encoded? UTF8?
Looking at our tomcat configuration all Get parameters are decoded with UTF8. Great. But looking at the output of what the flash app sends us we see scrambled Umlauts. Hmmm clearly the flash app does not use UTF8. But wait! There’s another option in Tomcat for decoding Get parameters: look into the header and use the encoding specified there. A restart later nothing changed. So flash does not send its encoding in the HTTP header. Well, let’s take a look at the HTTP standard:

If a reserved character is found in a URI component and no delimiting role
is known for that character, then it must be interpreted as representing the
data octet corresponding to that character's encoding in US-ASCII.

Ah.. US-ASCII and what about non ASCII ones? Wikipedia states:

For a non-ASCII character, it is typically converted to its byte sequence
in UTF-8, and then each byte value is represented as above.

Typically? Not in our case, so we tried ISO-8859-1 and finally the umlauts are correct! But currency signs like the euro are again garbage. So which encoding is similar to Latin-1 but not quite the same?
Yes, guess what: cp1252, the Windows native encoding.
And we tested all this on a Mac?!

Grails Gems: Command Objects

Besides domain objects command objects are another way to get validation and data binding of parameters. But why (or when) should you use them?
First when you do not want to persist the data. Like validating parameters for a search query.
Second when you just want a subset of the parameters which has no corresponding domain object. For example for keeping malicious data away from your domain objects.
Third when you get a delta of the new data. When you just want to add to a list and do not want to check if you get a single or a multiple value for your a parameter.

Usage

Usually you put the class of the command in the same file as the controller you use them in. The command object is declared as a parameter of the action closure. You can even use multiple one:

class MyController {
  def action = { MyCommand myCommand, YourCommand yourCommand ->
    ...
  }
}

Grails automatically binds the request parameters to the commands you supply and validates them. Then you can just call command.hasErrors() to see if the validation succeeded.

Grails: Beware of the second level cache

Recently we were hunting a strange bug. Take the following domain model:

class Computer {
  Coder coder
}

class Coder {
  static hasMany = [projects:Project]
}

Querying the computer and iterating over the respective coder and projects sometimes resulted in strange number of projects: 1. Looking into the underlying database we quickly found out that the number of 1 was not correct. It got even more strange: getting the coder in question via Coder.get in the loop yielded the correct results. What was the problem?
After some code reading and debugging another query which was called after the first one but before accessing the coder in the loop gave some insight:

  Coder.withCriteria {
    projects {
      idEq(projectId)
    }
  }

This second query also queried the Coder but constrained the projects to a specific one. These coders were populated into the second level cache and when we called computer.coder the second level cache returned the before queried coder. But this coder had only one project!
Since we only needed the number of coders with this project we changed the second
query to using count, so no instances of Coder are returned and thus saved in the second level cache. Bug fixed.

GORM-Performance with Collections

The other day I was looking to improve the performance of specific parts of our Grails application. I quickly found the typical bottleneck in database centric Grails apps: Too many queries were executed because GORM hides away database queries by its built-in persistence methods for domain objects and the extremely nice dynamic finders. In search for improvements and places to use GORM/Hibernate caching I stumbled upon a very good and helpful presentation on GORM-performance in general and especially collection usage. Burt Beckwith presents some common problems and good patterns to overcome them in his SpringOne 2GX talk. I highly recommend having a thorough look at his presentation.

Nevertheless, I want to summarize his bottom line here: GORM does provide a nice abstraction from relational databases but this abstraction is leaky at times. So you have to know exactly how the stuff in your domain classes is mapped. Be especially careful it collections tend to become “large” because performance will suffer extremely. We already observed a significant performance degradation for some dozen elements; your mileage may vary. For many simple modifications on a collection all its elements have to be loaded from the database!
Instead of using hasMany/belongsTo just add a back reference to the domain object your object belongs to. With the collection you lose cascading delete and some GORM functionality but you can still use dynamic finders and put the functionality to manage associations yourself into respective classes. This may be a large gain in specific cases!

Grails: The good, the bad, the ugly

After 3 years of Grails development it is time to take a step back and look how well we went.
(Info: we made several Grails apps ranging from small (<15 domain classes) to medium sized (50-70 domain classes) using front ends like Flex/Flash and AJAX)

The good parts

Always start with praise. So I tell you what in my opinion was and is good about developing in Groovy and Grails.

Groovy is Java with sugar

The Groovy syntax and the type system are so close to Java, so that when you come from a Java background you feel right at home.

Standard web stack

If you are accustomed to standard technologies like Spring and Hibernate you see Grails as a vacation.

Sensible defaults aka Convention over configuration

Many of the configuration options are filled with sensible defaults.

Fast start

You get from 0 to 100 in almost no time.

The bad things

Things which are not easily avoided.

Bugs, bugs, bugs

Grails has many, many bugs, unfortunately even in such fundamental things such as data binding and validation. A comment from a previous blog post: “To me, developing with Grails always felt like walking on eggs.”

Regression

Some bugs sneak back in again or are even reopened. Note that this is not the same as bugs, bugs, bugs because fixed bugs should be secured by a test.

Leaky abstractions

You have to know the underlying technologies especially Hibernate and Spring to get a foot on the ground. The GORM layer inherits all the complexity from Hibernate.

Slow integration tests

The ramp up time is 45 s on a decent build/development machine and then the first test hasn’t even started.

Uses the Java way of solving problems

Got a problem? There’s a framework for that!

Abandoned or prototype like plugins

Take a look at the list of plugins like Autobase, Flex.

Problems with incremental compiling

Don’t know where the real cause is buried: but using IntelliJ for developing Grails projects results in comments like:
Not working? Have you cleaned, invalidated your caches, rebuilt your project, deleted the .grails directory?

The ugly things

Things which are easily avoided or just a minor issue.

Groovys use of == and equals

Inherited from Java and made even worse: compare two numbers or a String and a GString

Groovys definition for the boolean truth

0, [], “”, null, false are all false

Groovys use of the NullObject and the plus operator

Puzzler: what is null + null ?

Uses unsupported/discontinued technologies

Hibernates SchemaExport comes to mind.

Mix of technology and intention

hasMany, hasOne, belongsTo have not only an intention revealing function but also determine how cascading works and the schema is generated.

Summary, opinionated

Grails has deficits and is bug ridden. But this will be better in the future (hopefully).
When you compare Grails with standard web stacks in the Java world you can gain a lot from it.
So if you want to know if you should use Grails in your next project ask yourself:

• do you have or want to use Spring and Hibernate?
• can you live without static typing? (remember: with freedom comes responsibility)
• are you ready to work around or even fix an issue or bug?
• is Java your home?

If you can answer all those questions with Yes, then Grails is for you. But beware: no silver bullets!

Grails upgrade – lessons learned

Grails is a great framework for rapidly developing web applications on top the proven java/servlet platform. Especially smaller, short-lived projects can be a real breeze with all the scaffolding, GORM and convention-over-configuration built into grails.

We happen to use it for a quite complex web application project for almost 3 years now. Half a year ago we upgrade from grails 1.0.x to 1.3.4. That makes 3 major versions in one upgrade step and produced obviously a lot of work and many small bugs. I do not want to put the blame on the grails guys here, because most of the stuff was mentioned in the release notes and it was a big step we decided to take when the decision came to continue the project for several years to come.

Our upgrade policy changed due to that experience and we try to stay a lot more current to be able to adapt our software to framework changes more incrementally. Some weeks ago we upgraded from 1.3.4 to 1.3.7 and this experience was not pleasant. Even though we skimmed through the changelogs and release notes and thought the update should be uncritical for us grails behaviour changed in two aspects which broke things for us:

1. An API-change where GroovyPagesException was changed to GrailsTagException
2. Behavioural change where no application context and injections are available in functional tests anymore

Item 1 was easy to fix but you need really good testing to spot it before it slips into production. Such subtle API changes should not happen in micro-version updates as that can easily break parts of the system whithout you knowing because of groovy/grails’ dynamic nature. No compiler saves you here.

Item 2 produced some amount of work for us because we build a quite extensive acceptance test suite using services and domain objects to setup the initial environment for each test. Luckily, there is the grails remote control plugin which you can use for things like that.

Lessons learned

• You should have extensive automated test suites when developing a grails application over a longer period because things can break in unexpected ways without code changes on your side.
• Try to plan upgrades some time ahead of releases and dedicate time to scanning the release notes and actually performing the upgrade. It may take you significantly longer than the smooth upgrade procedure itself suggests.

The grails team seems to be increasingly aware of backwards compatibility but they still have some way to go. We hope and expect to see fewer unexpected breakages to occur in the future.

The Grails performance switch: flush.mode=commit

– Disclaimer –
This optimization requires more manual work and is error prone but isn’t this with most (big) performance improvements?
For it to really work you have to structure your code accordingly and flush explicitly.
–
Recently in our performance measurements of a medium sized Grails project we noticed a strange behavior: every time we executed the same query the time it took increased. It started with 40ms and every time it took 1 ms more. The query was simple like Child.findAllByParent(parent)
The first thought: indexes! We looked at the database (a postgresql db) and we had indexes on the parent column.
Next: maybe the session cache got too large. But session.flush() and session.clear() did not solve that problem.
Another post suggested using a HQL query. Changing to

Child.executeQuery("select new Child(c.name, c.parent) from Child c where parent=:parent", [parent: parent])

had no effect.
Finally after countless more attempts we tried:

session.setFlushMode(FlushMode.COMMIT)

And not even the query executed in constant time it was also 10x faster?!
Hmmm…why?
The default flush mode in Grails is set to AUTO
Which means that before every query made the session is flushed. Every query regardless of the classes effected. The problem is known for hibernate but after 4! years it is still unresolved.
So my question here is: why did Grails chose AUTO as default?