Followup: Selenium Flex API with Firefox 3 and Selenium IDE 1.0 Beta 2 now working

Because of a bug in the Selenium Flex API, it didn’t work with Firefox 3 and the Selenium IDE 1.0 beta 2.
To fix this bug add the following line:

    if (flashObj.wrappedJSObject) {
        flashObj = flashObj.wrappedJSObject;
    }

to ide-extensions.js in Selenium.prototype.callFlexMethod after

var flashObj = selenium.browserbot.findElement(this.flashObjectLocator);

The state of functional testing in Flex

Functional testing of UIs is an important and often neglected way of ensuring quality and prevent regression. The Flex world of functional tests seems at the very beginning. We evaluated some of the tools available and used the following criterias:

• OS independence
  the tool and the created test scripts should run on at least every platform the Flex SDK and the Flash platform are available
• Tool changes
  how much we need to change or adapt the tool to suit our needs
• Code pollution
  how much the actual code needs to be polluted to support this testing tool
• Capture/Replay
  the tool needs at least an option to capture and replay test scripts
• Additional License Costs
  if we need to pay additional (besides the tool) license costs for things like the FlexBuilder Pro

	OS ind.	Tool changes	Code pollution	Capture / Replay	Add. costs
Automation based tools	+	-	0	+	-
SeleniumFlex	+	0	+	0	+
FunFx	-	0	+	-	0
Fluorida	+	-	+	-	+

Automation based tools (like FlexMonkey, QTP and RIATest) use the Flex automation API and have additional costs for FlexBuilderPro (700$ per license). For custom components you have to add automation code to them (pollution) and introduce them and their events in FlexMonkeys event map (tool changes).
SeleniumFlex uses the JavascriptBridge (ExternalInterface) of the FlashPlayer and needs you to add the custom components and events to this external interface which resides in the tool/test code (therefore a 0 at tool changes). You can use the Selenium plugin for spy (the ids)/replay but the capture option isn’t working yet (0 for capture/replay).
FunFx also uses the ExternalInterface and is written in Ruby but runs only on Windows (- for OS independence) because it connects to the Flex application via Win32OLE. I found no capture/replay (-) and the website says you need FlexBuilder (I don’t know why therefore a 0 for license costs, we use IntelliJ IDEA for Flex development)
Fluorida seems to be at the beginning and there is very little documentation so it looks like to need an investment (- for tool changes). It has no capture/replay (-).

Conclusion

So our tool of choice is SeleniumFlex and we hope to get capture/replay working in the near future.
What experience have you made with functional testing in Flex? Which one do you use?

Easy code inspection using QDox

So, you’ve inspected your Java code in any possible way, using Findbugs, Checkstyle, PMD, Crap4J and many other tools. You know every number by heart and keep a sharp eye on its trend. But what about some simple questions you might ask yourself about your project, like:

• How many instance variables aren’t final?
• Are there any setXYZ()-methods without any parameter?
• Which classes have more than one constructor?

Each of this question isn’t of much relevance to the project, but its answer might be crucial in one specific situation.

Using QDox for throw-away tooling

QDox is a fine little project making steady progress in being a very intuitive Java code structure inspection API. It’s got a footprint of just one JAR (less than 200k) you need to add to your project and one class you need to remember as a starting point. Everything else can be learnt on the fly, using the code completion feature of your favorite IDE.

Let’s answer the first question of our list by printing out all the names of all instance variables that aren’t final. I’m assuming you call this class in your project’s root directory.

public class NonFinalFinder {
    public static void main(String[] args) {
         File sourceFolder = new File(".");
         JavaDocBuilder parser = new JavaDocBuilder();
         builder.addSourceTree(sourceFolder);
         JavaClass[] javaClasses = parser.getClasses();
         for (JavaClass javaClass : javaClasses) {
             JavaField[] fields = javaClass.getFields();
             for (JavaField javaField : fields) {
                 if (!javaField.isFinal()) {
                     System.out.println("Field "
                       + javaField.getName()
                       + " of class "
                       + javaClass.getFullyQualifiedName()
                       + " is not final.");
                }
            }
        }
    }
}

The QDox parser is called JavaDocBuilder for historical reasons. It takes a directory through addSourceTree() and parses all the java files it finds in there recursively. That’s all you need to program to gain access to your code structure.

In our example, we descend into the code hierarchy using the parser.getClasses() method. From the JavaClass objects, we retrieve their JavaFields and ask each one if it’s final, printing out its name otherwise.

Praising QDox

The code needed to answer our example question is seven lines in essence. Once you navigate through your code structure, the QDox API is self-explanatory. You only need to remember the first two lines of code to get started.

The QDox project had a long quiet period in the past while making the jump to the Java 5 language spec. Today, it’s a very active project published under the Apache 2.0 license. The developers add features nearly every day, making it a perfect choice for your next five-minute throw-away tool.

What’s your tool idea?

Tell me about your code specific aspect you always wanted to know. What would an implementation using QDox look like?

Structuring CppUnit Tests

While unit testing in Java is dominated by JUnit, C++ developers can choose between a variety of frameworks. See here for a comprehensive list. Here you can find a nice comparison of the biggest players in the game.

Being probably one of the oldest frameworks CppUnit sure has some usability issues but is still widely used. It is criticised mostly because you have to do a lot of boilerplate typing to add new tests. In the following I will not repeat how tests can be written in CppUnit as this is described already exhaustively (e.g. here or here). Instead I will concentrate on the task of how to structure CppUnit tests in bigger projects. “Bigger” in this case means at least a few architectually independent parts which are compiled independently, i.e. into different libraries.

Having independently compiled parts in your project means that you want to compile their unit tests independently, too. The goal is then to structure the tests so that they can easily be executed selectively during development time by the programmer and at the same time are easy to execute as a whole during CI time (CI meaning Continuous Integration, of course).

As C++ has no reflection or other meta programming elements like the Java Annotations, things like automatic test discovery and how to add new tests become a whole topic of its own. See the CppUnit cookbook for how to do that with CppUnit . In my projects I only use the TestFactoryRegistry approach because it provides the most automatics in this regard.

Let’s begin with a simplest setup, the Link-Time Trap (see example source code): Test runner and result reporter are setup in the “main” function that is compiled into an executable. The actual unit tests are compiled in separate libraries and are all linked to the executable that contains the main function. While this solution works well for small projects it does not scale. This is simply because every time you change something during the code-compile-test cycle the unit test executable has to be relinked, which can take a considerable amount of time the bigger the project gets. You fall into the Link Time Trap!

The solution I use in many projects is as follows: Like in the simple approach, there is one test main function which is compiled into a test executable. All unit tests are compiled into libraries according to their place in the system architecture. To avoid the Link-Time-Trap, they are not linked to the test executable but instead are automatically discovered and loaded during test execution.

1. Automatic Discovery

Applying a little convention-over-configuration all testing libraries end with the suffix “_tests.so”. The testing main function can then simply walk over the directory tree of the project and find all shared libraries that contain unit test classes.

2. Loading

If a “.._test.so” library has been found, it simply gets loaded using dlopen (under Linux). When the library is loaded the unit tests are automatically registered with the TestFactoryRegistry.

3. Execution

After all unit test libraries has been found and loaded text execution is the same as in the simple approach above.

Here my enhanced testmain.cpp (see example source code).

#include ...

using namespace boost::filesystem;
using namespace std;

void loadPlugins(const std::string& rootPath)
{
   directory_iterator end_itr;
   for (directory_iterator itr(rootPath); itr != end_itr; ++itr) {
      if (is_directory(*itr)) {
         string leaf = (*itr).leaf();
         if (leaf[0] != '.') {
            loadPlugins((*itr).string());
         }
         continue;
      }
      if (extension(*itr) == ".so") {
         string fileName = (*itr).string();
         if (fileName.find("_tests.so") != string::npos) {
            void * handle = dlopen (fileName.c_str(), RTLD_NOW | RTLD_GLOBAL);
            cout <<"Opening : " << fileName.c_str() << endl;
            if (!handle) {
               cout <<"Error: " << dlerror() << endl;
               exit (1);
            }
         }
      }
   }
}

int main ( int argc, char ** argv )
{
   string rootPath = "./";
   if (argc > 1) {
      rootPath = static_cast<const char*>(argv[1]);
   }
   cout << "Loading all test libs under " << rootPath << endl;
   string runArg = std::string ( "All Tests" );
   // get registry
   CppUnit::TestFactoryRegistry & registry =
      CppUnit::TestFactoryRegistry::getRegistry();

   loadPlugins(rootPath);
   // Create the event manager and test controller
   CppUnit::TestResult controller;

   // Add a listener that colllects test result
   CppUnit::TestResultCollector result;
   controller.addListener ( &result );
   CppUnit::TextUi::TestRunner *runner = new CppUnit::TextUi::TestRunner;
   std::ofstream xmlout ( "testresultout.xml" );
   CppUnit::XmlOutputter xmlOutputter ( &result, xmlout );
   CppUnit::TextOutputter consoleOutputter ( &result, std::cout );

   runner->addTest ( registry.makeTest() );
   runner->run ( controller, runArg.c_str() );
   xmlOutputter.write();
   consoleOutputter.write();
   return result.wasSuccessful() ? 0 : 1;
}

As you can see the loadPlugins function uses the Boost.Filesystem library to walk over the directory tree.

It also takes a rootPath argument which you can give as parameter when you call the test main executable. This solves our goal stated above. When you want to execute unit tests selectively during development you can give the path of the corresponding testing library as parameter. Like so:

./testmain path/to/specific/testing/library

In your CI environment on the other hand you can execute all tests at once by giving the root path of the project, or the path where all testing libraries have been installed to.

./testmain project/root

CMake Builder Plugin for Hudson

Update: Check out my post introducing the newest version of the plugin.

Today I’m pleased to announce the first version of the cmakebuilder plugin for Hudson. It can be used to build cmake based projects without having to write a shell script (see my previous blog post). Using the scratch-my-own-itch approach I started out implementing only those features that I needed for my cmake projects which are mostly Linux/g++ based so far.

Let’s do a quick walk through the configuration:

1. CMake Path:
If the cmake executable is not in your $PATH variable you can set its path in the global Hudson configuration page.

2. Build Configuration:

To use the cmake builder in your Free-style project, just add “CMake Build” to your build steps. The configuration is pretty straight forward. You just have to set some basic directories and the build type.

cmakebuilder demo config

The demo config above results in the following behavior (shell pseudocode):

if $WORKSPACE/build_dir does not exist
   mkdir $WORKSPACE/build_dir
end if

cd $WORKSPACE/build_dir
cmake $WORKSPACE/src -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=$WORKSPACE/install_dir
make
make install

That’s it. Feedback is very much appreciated!!

Originally the plan was to have the plugin downloadable from the hudson plugins site by now but I still have some publishing problems to overcome. So if you are interested, make sure to check out the plugins site again in a few days. I will also post an update here as soon as the plugin can be downloaded.

Update: After fixing some maven settings I was finally able to publish the plugin. Check it out!