Developing Grails Apps – Some Dark Sides

Usually, I try to avoid it but this time a Disclaimer is in order: This is not a Grails rant. Most of the time developing Grails projects is fast and smooth. Using Grails brought many advantages for us. But there are also dark sides…

My main criticism is that Grails abstractions are more than leaky! In every list of examples for the definition of the term Leaky Abstraction Grails should be top. As soon as you leave the tutorial/scaffolding/helloworld level you have to know a lot about the underlying stack. And with Hibernate and Spring neither of the words small, easy and lightweight do apply.

GORM, too, is only easy to use at first sight. The very informative blog series about GORM gotchas should absolutely become part of the user guide or the refence docs.

And there are those times where it gets really unpleasant. This is e.g. when a bug does appear in your grails application running in a servlet container (packaged in a .war)  but does not appear when the application is started from within the IDE. Our last one of those was a naming conflict in a .gsp file. The controller handed a model like this to the .gsp:

...
return [fieldValue: 'THE_VALUE', ...]

The model entry ‘fieldValue’ was used in the .gsp to set the value of a combo box. Unfortunately, ‘fieldValue’ is also the name of a built-in Grails tag …

Admittedly, ‘fieldValue’ was not the wisest choice of names and I would certainly expect to get scolded loudly by Grails for that – ideally with a nice descriptive exception. But what happend instead led to a loud scolding of Grails from us. And to some big question marks: What is the difference between executing Grails from the IDE and within a servlet container with respect to naming resolution? Why is there a difference, at all?

We had a hard time figuring out this one, not least because the error message was not very telling. And since this was not the first of those works-in-the-IDE-but-not-in-a-real-environment bugs there is always this slightly uneasy feeling…

As I said in the beginning, most of the time developing Grails applications is nice and shiny. I would not support their slogan, though. My personal search for the best web development tool is definitively not over.

How about your search?

GORM Gotchas: Validation and hasMany

The excellent GORM Gotchas Series inspired me to write about a gotcha I found recently.
You have a domain class Container which contains elements:

class Container {
  static hasMany = [elements:Element]

  List<Element> elements
}

and the element has a constraint:

class Element {
  static belongsTo = [container:Container]

  static constraints = {
    email(email: true)
  }

  String email
}

When you try to save a container with more than one element that fails validation, only the first error appears:

Container c = new Container()
c.addToElements(new Element(email: "a"))
c.addToElements(new Element(email: "b"))
c.save()
assertEquals(2, c.errors.allErrors.size()) // fails, only one error is recorded!

The solution described in the docs coined with In some situations (unusual situations)) is to use a custom validator in the container class:

class Container {

  static constraints = {
      elements(validator: { List val, Container obj ->
          val.each {Element element ->
            if(! element.validate()) {
              element.errors.allErrors.each { error->
                obj.errors.rejectValue(
                      'elements',
                      error.getCode(),
                      error.getArguments(),
                      error.getDefaultMessage()
                )
              }
            }
          }
          return true
      })
  }

  static hasMany = [elements:Element]

  List<Element> elements
}

Grails pitfall: Proxies and inheritance

Problem

You have a domain class hierarchy and want to use the classes in an association. You typically use the base class in a hasMany:

class Domain {
  static hasMany = [others:BaseClass]
}

Dependent on the actual class in the set others you want to call a different method:

public doSomething(domain) {
  domain.others.each {doThis(it)}
}

public doThis(ChildClassOne c) {}

public doThis(ChildClassTwo c) {}
...

Here the proxy mechanism of Hibernate (and Grails) causes a MethodMissingException to be thrown because the proxies are instances of the base class and not the actual child ones.
One way would be to activate eager fetching or you could use…

Solution: A dynamic visitor

Declare a visit method in the base class which takes a closure

def visit(Closure c) {
return c(this)
}

and make the dispatch in a method for the base class:

public doThis(BaseClass c) {
  return c.visit {doThis(it)}
}

Open Source Love Day March 2010

Yesterday, we held our first Open Source Love Day (OSLD) for this year. The last OSLD was at December 2009. Then, we reassigned a day in January and February each to perform our relocation to the new (and much bigger) office. But now we are back to regular duty and had the time to donate some work back to the Open Source ecosystem.

The Open Source Love Day

We introduced a monthly Open Source Love Day to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

Participate at our OSLD by using the features we’ve built today:

• Grails still has some bugs. Instead of only complaining about them, we try to fix them. There is a bug with checkboxes and nested boolean properties that bugged us in a customer project. It’s filed unter GRAILS-3299 and has a proposed patch now.
• In previous OSLDs, we produced the cmake hudson plugin. In the corresponding blog entry, comments with bug reports began to pile up. They addressed issues with hudson master/slave setups. So we implemented a hudson master/slave test environment, using VirtualBox virtual machines to perform as slaves. This setup quickly revealed the problems that were typical enough to devote a complete blog entry about this topic soon. Fixing the problems resulted in the new cmake hudson plugin version 1.2 to be released yesterday.
• We are using the RXTX project to perform serial (RS232) communication in several projects. We are really glad the project exists, because the “official” communications API from Sun/Oracle is nothing but a mess. With RXTX, we only had a problem with emulated COM ports. Emulated COM ports exist when you use a USB->Serial or Ethernet->Serial converter, which is what our customer chose to do. If you unplug the converter during operation, the corresponding COM port disappears. This causes RXTX to crash, bringing the JVM down, too. We wrote a test application and used it with every converter we own (and we own quite a lot of them!). Then we began tracing the RXTX source code (at C code level), altering it to “only” throw an IOException when the virtual COM port disappears. The corresponding patch will be proposed to the RXTX project soon.
• Another API we use a lot is the tiny winp project, written by Kohsuke Kawaguchi, the creator of hudson. We kill Windows processes with it, within a project that runs on Windows 2000, Windows XP and Windows 7. The latest Windows version seemed incompatible with winp, even the 32bit edition. We didn’t find the cause for this, but developed a workaround that will be proposed to the winp project soon.

What were our lessons learnt today?

• If you face OutOfMemoryErrors on a 64bit Java6 JVM, try to switch back to a 32bit Java5 JVM. It helped us with our Grails bugfixing (during the test phase).
• Hudson Master/Slave support for plugins isn’t particularly hard. It’s just that you need to be aware of the topic and replace some types like java.io.File. We gathered the same experience twice with our Crap4J plugin and the cmake plugin. It’s time to tell the world about it. Stay tuned!
• The good old error return code is an error prone coding paradigm, because all too often, users of a function/method just forget to check the returned result. This was the case with a call to WaitForSingleObject in RXTX.
• If you don’t understand an implementation well enough to fix the cause, you might at least be able to produce a workaround. It’ll work for you and provide guidance for the original author about where the bug might hide. This is why we count our winp efforts as success, too.
• Your project either is mavenized or it isn’t. Everything in between is half-assed.

This OSLD was a bit short, as we had some guests in the evening, but nevertheless, it was fun. Well, to be precise, it was this special software engineer’s type of fun: The whole company was remarkably quiet most of the day, with everyone working totally focussed. We scratched our own itches, enhanced our customer projects and contributed to the open source community. A very good day!

Stay tuned if you want to know more about the specifics of the hudson plugin development or the to-be-proposed patches. We will publish them here.

Grails Web Application Security: XSS prevention

XSS (Cross Site Scripting) became a favored attack method in the last years. Several things are possible using an XSS vulnerability ranging from small annoyances to a complete desaster.
The XSS prevention cheat sheet states 6 rules to prevent XSS attacks. For a complete solution output encoding is needed in addition to input validation.
Here I take a further look on how to use the built in encoding methods in grails applications to prevent XSS.

Take 1: The global option

There exists a global option that specifies how all output is encoded when using ${}. See grails-app/conf/Config.groovy:

// The default codec used to encode data with ${}
grails.views.default.codec="html" // none, html, base64

So every input inside ${} is encoded but beware of the standard scaffolds where fieldValue is used inside ${}. Since fieldValue uses encoding you get a double escaped output – not a security problem, but the output is garbage.
This leaves the tags from the tag libraries to be reviewed for XSS vulnerability. The standard grails tags use all HTML encoding. If you use older versions than grails 1.1: beware of a bug in the renderErrors tag. Default encoding ${} does not help you when you use your custom tags. In this case you should nevertheless encode the output!
But problems arise with other tags like radioGroup like others found out.
So the global option does not result in much protection (only ${}), double escaping and problems with grails tags.

Take 2: Tainted strings

Other languages/frameworks (like Perl, Ruby, PHP,…) use a taint mode. There are some research works for Java.
Generally speaking in gsps three different outputs have to be escaped: ${}, <%%> and the ones from tags/taglibs. If a tainted String appears you can issue a warning and disallow or escape it. The problem in Java/Groovy is that Strings are value objects and since get copied in every operation so the tainted flag needs to be transferred, too. The same tainted flag must also be introduced for GStrings.
Since there isn’t any implementation or plugin for groovy/grails yet, right now you have to take the classic route:

Take 3: Test suites and reviews

Having a decent test suite in e.g. Selenium and reviewing your code for XSS vulnerabilities is still the best option in your grails apps. Maybe the tainted flags can help you in the future to spot places which you didn’t catch in a review.

P.S. A short overview for Java frameworks and their handling of XSS can be found here

Small gaps in the grails docs

Just for reference, if you come across one of the following problems:

Validation only datasource

Looking at the options of dbCreate in Datasource.groovy I only found 3 values: create-drop, create or update. But there is a fourth one: validate!
This one helps a lot when you use schema generation with Autobase or doing your schema updates external.

Redirect

Controller.redirect has two options for passing an id to the action id and params, but if you specify both which one will be used?

controller.redirect(id:1, params:[id:2])

Trying this out I found the id supersedes the params.id.

Update:
Thanks to Burt and Alvaro for their hints. I submitted a JIRA issue

A DSL for deploying grails apps

Everytime I deploy my grails app I do the same steps over and over again:

• get the latest build from our Hudson CI
• extract the war file from the CI archive
• scp the war to a gateway server
• scp the war to the target server
• run stop.sh to shutdown the jetty
• run update.sh to update the web app in the jetty webapps dir
• run start.sh to start the jetty

Reading the Productive Programmer I thought: “This should be automated”. Looking at the Rails world I found a tool named Capistrano which looked like a script library for deploying Rails apps. Using builders in groovy and JSch for SSH/scp I wrote a small script to do the tedious work using a self defined DSL for deploying grails apps:

Grapes grapes = new Grapes()
def script = grapes.script {
    set gateway: "gateway-server"
    set username: "schneide"
    set password: "************"
    set project: "my_ci_project"
    set ciType: "hudson"
    set target: "deploy_target.com"
    set ci_server: "hudson-schneide"
    set files: ["webapp.war"]

    task("deploy") {
        grab from: "ci"
        scp to: "target"
        ssh "stop.sh"
        ssh "update.sh"
        ssh "start.sh"
    }
}

script.tasks.deploy.execute()

This is far from being finished but a starting point and I think about open sourcing it. What do you think: may it help you? What are your experiences with deploying grails apps?

Deploying a Grails app on an Oracle DB

Running our new grails app on HSQL and a Postgresql everything went fine. But the production DB was decided to be an Oracle. And suddenly the app crashed several times. Here’s a list of what problems we encountered:

• ORA-00972: identifier is too long
• want to store a null value in a non null column

Oracle identifiers are limited to 30 characters. So we thought using a mapping for the table should do the trick. But grails uses the table names to construct the n:m relations and their id column names between the domain classes. Looking at the grails docs we found a joinTable mapping:

static mapping = {
    table 'PROP'
    tablePerHierarchy false
    instrumentInfos joinTable: [name:'PROP_INS', key:'id', column:'instrumentInfos_id']
}

This worked most of the time but in some cases grails just didn’t want to take our definitions. The problem was a bug in grails. The workaround we took was to shorten the domain classes names.
The second problem arose as we tried to store empty strings into the database. Oracle stores empty strings as null values which causes a constraint violation exception. The solution was to declare the string columns nullable or not nullable and not blank but you cannot use a not nullable and blank string with an Oracle DB.

Global error pages with Jetty and grails

We wanted to configure global error pages for our grails app. Using your favorite search engine you quickly find the following info.

At the bottom it says that in order to support global error pages you have to forward to a context because error pages can only be handled within contexts/webapps.
So I started adding a context to the contexts dir which looked like this:

<Configure class="org.mortbay.jetty.webapp.WebAppContext">
      <Set name="contextPath">/</Set>
      <Set name="war"><SystemProperty name="jetty.home" default="."/>/webapps/mywebapp.war</Set>
      <Set name="extractWAR">false</Set>
...

This caused an “IllegalArgumentException: name” at startup. The solution was to set extractWAR to true. JSPs or other resources (like GSPs) cannot be used inside a war when extractWAR is set to false. But this way got another pitfall: using localhost:8080/mywebapp won’t work. So why not just forward all requests from / to /mywebapp. Said and done:

<Set name="handler">
  <New id="Handlers" class="org.mortbay.jetty.handler.RewriteHandler">
    <Set name="rewriteRequestURI">false</Set>
    <Set name="rewritePathInfo">false</Set>
    <Set name="originalPathAttribute">requestedPath</Set>
    <Call name="addRewriteRule"><Arg>/mywebapp/*</Arg><Arg></Arg></Call>
    <Call name="addRewriteRule"><Arg>/*</Arg><Arg>/mywebapp</Arg></Call>
    <Set name="handler">
here the old handlers are inserted...

Now /mywebapp points to my webapp. / gives a 500 and other invalid urls give a 404.
To use your custom error pages inside a grails app just add the error codes you want to map inside the UrlMappings.groovy file:

class UrlMappings {
  ...
  static mappings = {
    "500"(view:'/error')
    "404"(view:'/error404')
  }
}

Using grails projects in Hudson

Being an agile software development company we use a continuous integration (CI) server like Hudson.
For our grails projects we wrote a simple ant target -call-grails to call the batch or the shell scripts:

    <condition property="grails" value="${grails.home}/bin/grails.bat">
        <os family="windows"/>
    </condition>
    <property name="grails" value="${grails.home}/bin/grails"/>

    <target name="-call-grails">
		<chmod file="${grails}" perm="u+x"/>
        <exec dir="${basedir}" executable="${grails}" failonerror="true">
            <arg value="${grails.task}"/>
            <arg value="${grails.file.path}"/>
            <env key="GRAILS_HOME" value="${grails.home}"/>
        </exec>
    </target>

Calling it is as easy as calling any ant target:

  <target name="war" description="--> Creates a WAR of a Grails application">
        <antcall target="-call-grails">
            <param name="grails.task" value="war"/>
            <param name="grails.file.path" value="${target.directory}/${artifact.name}"/>
        </antcall>
    </target>

One pitfall exists though, if your target takes no argument(s) after the task you have to use a different call:

	<target name="-call-grails-without-filepath">
		<chmod file="${grails}" perm="u+x"/>
        <exec dir="${basedir}" executable="${grails}" failonerror="true">
            <arg value="${grails.task}"/>
            <env key="GRAILS_HOME" value="${grails.home}"/>
        </exec>
    </target>