Grails: Beware of the second level cache

August 16, 2011

Recently we were hunting a strange bug. Take the following domain model:

class Computer {
  Coder coder
}

class Coder {
  static hasMany = [projects:Project]
}

Querying the computer and iterating over the respective coder and projects sometimes resulted in strange number of projects: 1. Looking into the underlying database we quickly found out that the number of 1 was not correct. It got even more strange: getting the coder in question via Coder.get in the loop yielded the correct results. What was the problem?
After some code reading and debugging another query which was called after the first one but before accessing the coder in the loop gave some insight:

  Coder.withCriteria {
    projects {
      idEq(projectId)
    }
  }

This second query also queried the Coder but constrained the projects to a specific one. These coders were populated into the second level cache and when we called computer.coder the second level cache returned the before queried coder. But this coder had only one project!
Since we only needed the number of coders with this project we changed the second
query to using count, so no instances of Coder are returned and thus saved in the second level cache. Bug fixed.


A big benefit of Convention over Configuration

July 26, 2011

Recently I joined a long running Grails project and had to complete some issues in a short time. After a quick introduction I was ready to dive in and instantly could chunk out code. Convention over Configuration helped a lot getting a fast start:

  • Which classes are persisted? Look into the domain folder.
  • What controller handles xyz? It is named after xyz.
  • What is the page that is served for URL u? Look into the corresponding folder and find the view named accordingly.

IMHO this is a benefit that many conventions are determined by the framework. If you know the framework, you know the layout of the project. The opposite holds also true: if the project departs from these “standards” you have to look closely into the configuration.
So think twice before you do something your way. Sometimes you have no choice like when you are using Oracle and have to cut all table and column names to 30 characters. But normally you should keep the defaults.


Grails: The good, the bad, the ugly

April 5, 2011

After 3 years of Grails development it is time to take a step back and look how well we went.
(Info: we made several Grails apps ranging from small (<15 domain classes) to medium sized (50-70 domain classes) using front ends like Flex/Flash and AJAX)

The good parts

Always start with praise. So I tell you what in my opinion was and is good about developing in Groovy and Grails.

Groovy is Java with sugar

The Groovy syntax and the type system are so close to Java, so that when you come from a Java background you feel right at home.

Standard web stack

If you are accustomed to standard technologies like Spring and Hibernate you see Grails as a vacation.

Sensible defaults aka Convention over configuration

Many of the configuration options are filled with sensible defaults.

Fast start

You get from 0 to 100 in almost no time.

The bad things

Things which are not easily avoided.

Bugs, bugs, bugs

Grails has many, many bugs, unfortunately even in such fundamental things such as data binding and validation. A comment from a previous blog post: “To me, developing with Grails always felt like walking on eggs.”

Regression

Some bugs sneak back in again or are even reopened. Note that this is not the same as bugs, bugs, bugs because fixed bugs should be secured by a test.

Leaky abstractions

You have to know the underlying technologies especially Hibernate and Spring to get a foot on the ground. The GORM layer inherits all the complexity from Hibernate.

Slow integration tests

The ramp up time is 45 s on a decent build/development machine and then the first test hasn’t even started.

Uses the Java way of solving problems

Got a problem? There’s a framework for that!

Abandoned or prototype like plugins

Take a look at the list of plugins like Autobase, Flex.

Problems with incremental compiling

Don’t know where the real cause is buried: but using IntelliJ for developing Grails projects results in comments like:
Not working? Have you cleaned, invalidated your caches, rebuilt your project, deleted the .grails directory?

The ugly things

Things which are easily avoided or just a minor issue.

Groovys use of == and equals

Inherited from Java and made even worse: compare two numbers or a String and a GString

Groovys definition for the boolean truth

0, [], "", null, false are all false

Groovys use of the NullObject and the plus operator

Puzzler: what is null + null ?

Uses unsupported/discontinued technologies

Hibernates SchemaExport comes to mind.

Mix of technology and intention

hasMany, hasOne, belongsTo have not only an intention revealing function but also determine how cascading works and the schema is generated.

Summary, opinionated

Grails has deficits and is bug ridden. But this will be better in the future (hopefully).
When you compare Grails with standard web stacks in the Java world you can gain a lot from it.
So if you want to know if you should use Grails in your next project ask yourself:

  • do you have or want to use Spring and Hibernate?
  • can you live without static typing? (remember: with freedom comes responsibility)
  • are you ready to work around or even fix an issue or bug?
  • is Java your home?

If you can answer all those questions with Yes, then Grails is for you. But beware: no silver bullets!


The Grails performance switch: flush.mode=commit

March 8, 2011

– Disclaimer –
This optimization requires more manual work and is error prone but isn’t this with most (big) performance improvements?
For it to really work you have to structure your code accordingly and flush explicitly.

Recently in our performance measurements of a medium sized Grails project we noticed a strange behavior: every time we executed the same query the time it took increased. It started with 40ms and every time it took 1 ms more. The query was simple like Child.findAllByParent(parent)
The first thought: indexes! We looked at the database (a postgresql db) and we had indexes on the parent column.
Next: maybe the session cache got too large. But session.flush() and session.clear() did not solve that problem.
Another post suggested using a HQL query. Changing to

Child.executeQuery("select new Child(c.name, c.parent) from Child c where parent=:parent", [parent: parent])

had no effect.
Finally after countless more attempts we tried:

session.setFlushMode(FlushMode.COMMIT)

And not even the query executed in constant time it was also 10x faster?!
Hmmm...why?
The default flush mode in Grails is set to AUTO
Which means that before every query made the session is flushed. Every query regardless of the classes effected. The problem is known for hibernate but after 4! years it is still unresolved.
So my question here is: why did Grails chose AUTO as default?


Open Source Love Day October 2010

November 10, 2010

On Friday two weeks ago, we held our Open Source Love Day for October 2010. This day was special in several ways. We strayed very far from the usual schedule for this day, there were several internal tasks that couldn’t be delayed and we introduced a “fun practice” event. But we eventually produced something valuable this day.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

The distractions

  • A regular project needed an urgent cost estimation by the whole team. This was the last opportunity because of an upcoming parental leave to have the team together for a long time.
  • Another regular project needed an urgent problem solved. This turned out to be so obscure that one of our developers had to be on-site. You can read about it in this blog entry now.
  • We received several shipments of office furniture and computer parts. They had to be checked and placed in.
  • We had a fun practice event. We discovered the online “game” typeracer and practiced our raw typing skill against each other for some time. Pro tip for beginners: don’t look at the highscores!

On this OSLD, we accomplished the following tasks:

  • A new version 1.8 of the cmakebuilder hudson plugin implements several feature requests. You can now choose to NOT clean your workspace before building and set different paths for the cmake installation for every job or node (hudson slaves). The latter option can be applied using an environment variable.

On this OSLD, we also tried the following tasks:

  • We keep an eye on Scala and its associated web framework Lift as a promising technology. One issue with Lift that bugs us is the use of “sun bastard format” properties for internationalization. We tried to teach Lift to accept UTF-8 encoded property files. After a lot of “downloading the internet” (you can always tell which project uses maven by their initial setup delay), we quickly implemented our own ResourceBundle.Control. But the Lift framework itself could not be built: “Error occurred during initialization of VM: Could not reserve enough space for object heap”. We ran out of time and will investigate in this issue on the next OSLD.
  • Grails is another web framework we use in projects. There are some bugs that really annoy us, and the OSLD is the perfect time to fix them. One of these bugs is GRAILS-6475, which we tried to reproduce with the latest code base. After writing a test case that would go green unexpectedly, we tried to provoke the error by setting up a sample project. The bug didn’t show up there, too. We left a comment in the issuetracker and ceased development.

What were our lessons learnt today?

  • You can’t tear off massive amounts of time from the OSLD and expect it to still be working. An OSLD doesn’t scale down apparently.
  • Most issues that can’t be done fail with the project’s build. The build process of a foreign project is the most crucial phase in your decision on commitment. If it fails, your participation in the project is at risk. We’ve seen many brittle, undocumented and incredible complex build processes now. And we can state one thing: It doesn’t stop with throwing maven at a project, you still have to “think the build”.

Retrospective of the OSLD

This OSLD was special in the amount of non-OSLD work done. The remaining efforts weren’t as successful as we wished. This has been an ongoing issue with our OSLD for the last months now and we are looking forward to adapt our workstyle to yield better results in the future. The distraction by typeracer was fun, though.


Developing Grails Apps – Some Dark Sides

October 4, 2010

Usually, I try to avoid it but this time a Disclaimer is in order: This is not a Grails rant. Most of the time developing Grails projects is fast and smooth. Using Grails brought many advantages for us. But there are also dark sides…

My main criticism is that Grails abstractions are more than leaky! In every list of examples for the definition of the term Leaky Abstraction Grails should be top. As soon as you leave the tutorial/scaffolding/helloworld level you have to know a lot about the underlying stack. And with Hibernate and Spring neither of the words small, easy and lightweight do apply.

GORM, too, is only easy to use at first sight. The very informative blog series about GORM gotchas should absolutely become part of the user guide or the refence docs.

And there are those times where it gets really unpleasant. This is e.g. when a bug does appear in your grails application running in a servlet container (packaged in a .war)  but does not appear when the application is started from within the IDE. Our last one of those was a naming conflict in a .gsp file. The controller handed a model like this to the .gsp:

...
return [fieldValue: 'THE_VALUE', ...]

The model entry 'fieldValue' was used in the .gsp to set the value of a combo box. Unfortunately, 'fieldValue' is also the name of a built-in Grails tag ...

Admittedly, 'fieldValue' was not the wisest choice of names and I would certainly expect to get scolded loudly by Grails for that - ideally with a nice descriptive exception. But what happend instead led to a loud scolding of Grails from us. And to some big question marks: What is the difference between executing Grails from the IDE and within a servlet container with respect to naming resolution? Why is there a difference, at all?

We had a hard time figuring out this one, not least because the error message was not very telling. And since this was not the first of those works-in-the-IDE-but-not-in-a-real-environment bugs there is always this slightly uneasy feeling...

As I said in the beginning, most of the time developing Grails applications is nice and shiny. I would not support their slogan, though. My personal search for the best web development tool is definitively not over.

How about your search?


GORM Gotchas: Validation and hasMany

September 20, 2010

The excellent GORM Gotchas Series inspired me to write about a gotcha I found recently.
You have a domain class Container which contains elements:

class Container {
  static hasMany = [elements:Element]

  List<Element> elements
}

and the element has a constraint:

class Element {
  static belongsTo = [container:Container]

  static constraints = {
    email(email: true)
  }

  String email
}

When you try to save a container with more than one element that fails validation, only the first error appears:

Container c = new Container()
c.addToElements(new Element(email: "a"))
c.addToElements(new Element(email: "b"))
c.save()
assertEquals(2, c.errors.allErrors.size()) // fails, only one error is recorded!

The solution described in the docs coined with In some situations (unusual situations)) is to use a custom validator in the container class:

class Container {

  static constraints = {
      elements(validator: { List val, Container obj ->
          val.each {Element element ->
            if(! element.validate()) {
              element.errors.allErrors.each { error->
                obj.errors.rejectValue(
                      'elements',
                      error.getCode(),
                      error.getArguments(),
                      error.getDefaultMessage()
                )
              }
            }
          }
          return true
      })
  }

  static hasMany = [elements:Element]

  List<Element> elements
}

Grails pitfall: Proxies and inheritance

July 5, 2010

Problem

You have a domain class hierarchy and want to use the classes in an association. You typically use the base class in a hasMany:

class Domain {
  static hasMany = [others:BaseClass]
}

Dependent on the actual class in the set others you want to call a different method:

public doSomething(domain) {
  domain.others.each {doThis(it)}
}

public doThis(ChildClassOne c) {}

public doThis(ChildClassTwo c) {}
...

Here the proxy mechanism of Hibernate (and Grails) causes a MethodMissingException to be thrown because the proxies are instances of the base class and not the actual child ones.
One way would be to activate eager fetching or you could use...

Solution: A dynamic visitor

Declare a visit method in the base class which takes a closure

def visit(Closure c) {
return c(this)
}

and make the dispatch in a method for the base class:

public doThis(BaseClass c) {
  return c.visit {doThis(it)}
}

Open Source Love Day March 2010

March 27, 2010

Yesterday, we held our first Open Source Love Day (OSLD) for this year. The last OSLD was at December 2009. Then, we reassigned a day in January and February each to perform our relocation to the new (and much bigger) office. But now we are back to regular duty and had the time to donate some work back to the Open Source ecosystem.

The Open Source Love Day

We introduced a monthly Open Source Love Day to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

Participate at our OSLD by using the features we’ve built today:

  • Grails still has some bugs. Instead of only complaining about them, we try to fix them. There is a bug with checkboxes and nested boolean properties that bugged us in a customer project. It’s filed unter GRAILS-3299 and has a proposed patch now.
  • In previous OSLDs, we produced the cmake hudson plugin. In the corresponding blog entry, comments with bug reports began to pile up. They addressed issues with hudson master/slave setups. So we implemented a hudson master/slave test environment, using VirtualBox virtual machines to perform as slaves. This setup quickly revealed the problems that were typical enough to devote a complete blog entry about this topic soon. Fixing the problems resulted in the new cmake hudson plugin version 1.2 to be released yesterday.
  • We are using the RXTX project to perform serial (RS232) communication in several projects. We are really glad the project exists, because the “official” communications API from Sun/Oracle is nothing but a mess. With RXTX, we only had a problem with emulated COM ports. Emulated COM ports exist when you use a USB->Serial or Ethernet->Serial converter, which is what our customer chose to do. If you unplug the converter during operation, the corresponding COM port disappears. This causes RXTX to crash, bringing the JVM down, too. We wrote a test application and used it with every converter we own (and we own quite a lot of them!). Then we began tracing the RXTX source code (at C code level), altering it to “only” throw an IOException when the virtual COM port disappears. The corresponding patch will be proposed to the RXTX project soon.
  • Another API we use a lot is the tiny winp project, written by Kohsuke Kawaguchi, the creator of hudson. We kill Windows processes with it, within a project that runs on Windows 2000, Windows XP and Windows 7. The latest Windows version seemed incompatible with winp, even the 32bit edition. We didn’t find the cause for this, but developed a workaround that will be proposed to the winp project soon.

What were our lessons learnt today?

  • If you face OutOfMemoryErrors on a 64bit Java6 JVM, try to switch back to a 32bit Java5 JVM. It helped us with our Grails bugfixing (during the test phase).
  • Hudson Master/Slave support for plugins isn’t particularly hard. It’s just that you need to be aware of the topic and replace some types like java.io.File. We gathered the same experience twice with our Crap4J plugin and the cmake plugin. It’s time to tell the world about it. Stay tuned!
  • The good old error return code is an error prone coding paradigm, because all too often, users of a function/method just forget to check the returned result. This was the case with a call to WaitForSingleObject in RXTX.
  • If you don’t understand an implementation well enough to fix the cause, you might at least be able to produce a workaround. It’ll work for you and provide guidance for the original author about where the bug might hide. This is why we count our winp efforts as success, too.
  • Your project either is mavenized or it isn’t. Everything in between is half-assed.

This OSLD was a bit short, as we had some guests in the evening, but nevertheless, it was fun. Well, to be precise, it was this special software engineer’s type of fun: The whole company was remarkably quiet most of the day, with everyone working totally focussed. We scratched our own itches, enhanced our customer projects and contributed to the open source community. A very good day!

Stay tuned if you want to know more about the specifics of the hudson plugin development or the to-be-proposed patches. We will publish them here.


Grails Web Application Security: XSS prevention

July 20, 2009

XSS (Cross Site Scripting) became a favored attack method in the last years. Several things are possible using an XSS vulnerability ranging from small annoyances to a complete desaster.
The XSS prevention cheat sheet states 6 rules to prevent XSS attacks. For a complete solution output encoding is needed in addition to input validation.
Here I take a further look on how to use the built in encoding methods in grails applications to prevent XSS.

Take 1: The global option

There exists a global option that specifies how all output is encoded when using ${}. See grails-app/conf/Config.groovy:

// The default codec used to encode data with ${}
grails.views.default.codec="html" // none, html, base64

So every input inside ${} is encoded but beware of the standard scaffolds where fieldValue is used inside ${}. Since fieldValue uses encoding you get a double escaped output - not a security problem, but the output is garbage.
This leaves the tags from the tag libraries to be reviewed for XSS vulnerability. The standard grails tags use all HTML encoding. If you use older versions than grails 1.1: beware of a bug in the renderErrors tag. Default encoding ${} does not help you when you use your custom tags. In this case you should nevertheless encode the output!
But problems arise with other tags like radioGroup like others found out.
So the global option does not result in much protection (only ${}), double escaping and problems with grails tags.

Take 2: Tainted strings

Other languages/frameworks (like Perl, Ruby, PHP,...) use a taint mode. There are some research works for Java.
Generally speaking in gsps three different outputs have to be escaped: ${}, <%%> and the ones from tags/taglibs. If a tainted String appears you can issue a warning and disallow or escape it. The problem in Java/Groovy is that Strings are value objects and since get copied in every operation so the tainted flag needs to be transferred, too. The same tainted flag must also be introduced for GStrings.
Since there isn't any implementation or plugin for groovy/grails yet, right now you have to take the classic route:

Take 3: Test suites and reviews

Having a decent test suite in e.g. Selenium and reviewing your code for XSS vulnerabilities is still the best option in your grails apps. Maybe the tainted flags can help you in the future to spot places which you didn't catch in a review.

P.S. A short overview for Java frameworks and their handling of XSS can be found here


Follow

Get every new post delivered to your Inbox.

Join 76 other followers