Open Source Love Day October 2010

On Friday two weeks ago, we held our Open Source Love Day for October 2010. This day was special in several ways. We strayed very far from the usual schedule for this day, there were several internal tasks that couldn’t be delayed and we introduced a “fun practice” event. But we eventually produced something valuable this day.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

The distractions

  • A regular project needed an urgent cost estimation by the whole team. This was the last opportunity because of an upcoming parental leave to have the team together for a long time.
  • Another regular project needed an urgent problem solved. This turned out to be so obscure that one of our developers had to be on-site. You can read about it in this blog entry now.
  • We received several shipments of office furniture and computer parts. They had to be checked and placed in.
  • We had a fun practice event. We discovered the online “game” typeracer and practiced our raw typing skill against each other for some time. Pro tip for beginners: don’t look at the highscores!

On this OSLD, we accomplished the following tasks:

  • A new version 1.8 of the cmakebuilder hudson plugin implements several feature requests. You can now choose to NOT clean your workspace before building and set different paths for the cmake installation for every job or node (hudson slaves). The latter option can be applied using an environment variable.

On this OSLD, we also tried the following tasks:

  • We keep an eye on Scala and its associated web framework Lift as a promising technology. One issue with Lift that bugs us is the use of “sun bastard format” properties for internationalization. We tried to teach Lift to accept UTF-8 encoded property files. After a lot of “downloading the internet” (you can always tell which project uses maven by their initial setup delay), we quickly implemented our own ResourceBundle.Control. But the Lift framework itself could not be built: “Error occurred during initialization of VM: Could not reserve enough space for object heap”. We ran out of time and will investigate in this issue on the next OSLD.
  • Grails is another web framework we use in projects. There are some bugs that really annoy us, and the OSLD is the perfect time to fix them. One of these bugs is GRAILS-6475, which we tried to reproduce with the latest code base. After writing a test case that would go green unexpectedly, we tried to provoke the error by setting up a sample project. The bug didn’t show up there, too. We left a comment in the issuetracker and ceased development.

What were our lessons learnt today?

  • You can’t tear off massive amounts of time from the OSLD and expect it to still be working. An OSLD doesn’t scale down apparently.
  • Most issues that can’t be done fail with the project’s build. The build process of a foreign project is the most crucial phase in your decision on commitment. If it fails, your participation in the project is at risk. We’ve seen many brittle, undocumented and incredible complex build processes now. And we can state one thing: It doesn’t stop with throwing maven at a project, you still have to “think the build”.

Retrospective of the OSLD

This OSLD was special in the amount of non-OSLD work done. The remaining efforts weren’t as successful as we wished. This has been an ongoing issue with our OSLD for the last months now and we are looking forward to adapt our workstyle to yield better results in the future. The distraction by typeracer was fun, though.

Advertisements

Open Source Love Day September 2010

On wednesday last week, we held our Open Source Love Day for September 2010. Our day started with the usual Homepage Comittee meeting and very soon, we were up and working. This time, our success rate wasn’t as high as we wanted, mostly because we worked on internal tools that didn’t work out quite as well as expected. But, we managed to produce something valuable this day.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

On this OSLD, we accomplished the following tasks:

  • A new version 1.6.1 of the cmakebuilder hudson plugin was published. This version consisted of bugfixes only and right now, it still seems flawed. We are working on the issue, expect a new version 1.6.2 soon.
  • Our internal time-tracking tool got love at several issues. One issue required the use of triangle-shaping CSS, as described in this blog post from Jon Rohan. Our issue weren’t finished because Javascript code can rot into a big pile of crap really quick.
  • We managed to make a long hatched dream come true at this OSLD. As you might be aware, we are big fanboys of crap4j, a metric tool that associates test coverage with code complexity. Thus, we wrote the crap4j hudson plugin, release the CrapMap and use some internal improvements, too. The main disadvantage of crap4j is the strong dependency on a specific test coverage tool. Our goal was to use the test coverage data we already collect using Cobertura. We achieved this goal and got the whole thing working. It will be released in the next few weeks, with a detailed blog post here. Stay tuned for this new tool (it already has a name: Crapertura).

What were our lessons learnt today?

  • No matter how clever you are, Javascript outsmarts you every time it appears in superior numbers. Refactoring is the key here, but difficult and tedious to apply.
  • When you dissect a foreign API or code base, you just need to find the right grip. I cannot decribe it more precise right now, but this grip is all you need to open up the code. When playing around with the crap4j code base, as soon as we held the grip, everything else followed naturally. Perhaps “the grip” can be translated with “catching the author’s intent”. These are always magical moments.

Retrospective of the OSLD

This OSLD wasn’t as successful as we wished, partly because of missing manpower (honeymoon holiday!) and because of our inability to tame a Javascript code base. We have to work on our expertise here and we are glad that we’ve found out at an OSLD, not a time- and mission-critical customer project.

Open Source Love Day July 2010

This friday , we held our Open Source Love Day for July 2010.  We began with several internal meetings and discussion (like the Homepage Comittee meeting) and dived right in our work afterwards. Everybody had a little backlog of issues that we wanted to get done on this day. Nearly everybody succeeded (well, the author had a minor delay – read about it below). The day went by in a very fast pace, but it felt right.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

On this OSLD, we accomplished the following tasks:

  • There are really cool new features in the latest JUnit versions and Rules are one of them. What hurt our aesthetic sense was that the field that hold the Rule instance has to be public. Checkstyle was on our side, so we tweaked JUnit to allow all kinds of visibility. You can read about the change needed here: http://github.com/KentBeck/junit/issues#issue/31. The fix is almost trivial and will hopefully be incorporated in the next versions of JUnit, so we do not publish our altered version.
  • We constantly receive requests and remarks about our cmake plugin for Hudson. This lead to a new version of the plugin fixing two issues with matrix builds and custom build types. Head over to the plugin homepage and grab the new version 1.6. The issues were in detail:
    • The plugin can be used with matrix builds now
    • Custom build types can be defined now
  • RXTX is our choice for serial port communication with Java. We fixed some issues during the last few OSLDs, with one issue left for today: When you flush your stream while using a special type of usb-to-rs232 converter, you got an exception. The corresponding issue is #102 in the RXTX issue tracker. We proposed a patch that fixes the problem.
  • Another hudson plugin is our crap4j reporter. It lacked some love for months now and finally broke when used with the latest hudson versions. Fixing the problem was a lot harder than we thought, basically because the plugin needed adjustments to recent API changes and we couldn’t figure out exactly what adjustments are necessary. You might have a look at the developer mailing list thread for this question. Finally, we got it resolved (on sunday, with a sudden stroke of insight) and a new version 0.8 is published.
  • We use an internal time tracking tool for our projects. This tool isn’t specifically open source yet, but continues to grow in terms of features and usability. The work invested in this tool helps us to continue with the OSLD, so it’s beneficial work nonetheless.
  • During the last OSLD, we had plans for a new hudson plugin and even produced a prototype. This time, we looked around the hudson plugin zoo (it’s getting a bit difficult to keep track of all of them) for inspiration and found a wonderful piece of art: The Groovy Postbuild Plugin. Using this plugin with a small groovy script served our needs exactly. No need for a full-blown plugin when you can scratch your itch with a simple script. Thanks to Serban Iordache for his great work!

What were our lessons learnt today?

  • If you need to setup a fresh workspace for an open source project, consider to prepare it over the night before, or the download delay will kill your precious work time. There is nothing more frustrating than staring at a “downloading…” progress bar while being eager to start programming.
  • Always look around what others have done before. We wanted to build a full hudson plugin from scratch when all we needed was a little groovy script placed inside another plugin. Sweet!
  • Do not hesitate to privately fix open source issues that won’t get done in time for you. Just make sure to have a management process in place to track those changes and be able to re-apply them to future versions. More important though, be able to tell exactly when NOT to re-apply them because the original project has fixed the issue.

Retrospective of the OSLD

The OSLD went smooth and was productive. We tend to work on backlogs instead of searching for random issues now, but that’s just a sign that our approach has matured and we depend on the OSLD to get work done.

Last wednesday, we held our Open Source Love Day for June 2010. This one was productive despite the heat that had us sweating the whole day long (as a sidenote: it got even warmer the days afterwards). Some features were finished and will help at least us in our projects. We still look forward for the right way to release them. Another release was even more problematic, you will read about it below.The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

On this OSLD, we accomplished the following tasks:

Open Source Love Day June 2010

Last wednesday, we held our Open Source Love Day for June 2010. This one was productive despite the heat that had us sweating the whole day long (as a sidenote: it got even warmer the days afterwards). Some features were finished and will help at least us in our projects. We still look forward for the right way to release them. Another release was even more problematic, you will read about it below.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

On this OSLD, we accomplished the following tasks:

  • Launch4j is a java application launcher for Windows, handling all the stuff a startup script would do, too. At the last OSLD, we added the ability to restart the application in case of a crash or other unplanned exit. To utilize this feature for automatic update routines, we needed to add the additional feature of starting another command instead of the original one. If the program fills a special file with the command needed, Launch4j will execute it after the program’s exit. This patch builds onto the previous patch and we are still investigating how to publish this functionality without breaking backward compatibility. We are looking forward to release it on the next OSLD.
  • We use RXTX to perform the serial (RS232) communication on all our java projects. We worked on an issue with serial converters over the course of several OSLDs now and released the patch to the issue tracker of RXTX after a longterm stability test. See the reworked patch for issue #144. There is another issue with the flush() method that seems to affect not only virtual RS232 ports that we currently investigate. But we aren’t yet able to come up with a complete issue description or a fix, so this will be suspended until the next OSLD.
  • We have written the Campfire Hudson Plugin as part of previous OSLDs. When issues emerged, we got patches from the community here. Thank you guys! We included the changes to the code and prepared a new release, when maven failed. This is not an issue, except when it fails repeatedly and messes up the workspace and the repository. After a long time of helpless fiddling with the parameters, we decided to start over and increase the version number to 2.1 (instead of 1.2). All of a sudden, everything worked out fine. Maven is a mysterious beast.
  • The initial work for a New Hudson Plugin was made. One tradition of the OSLD has always been to scratch our own itches. While there are many useful hudson plugins, we have the immediate need for another one that doesn’t exist yet. Without going into details here (we save this for the next OSLD), we produced a proof of concept and a first iteration of the code. Stay tuned for details on the next OSLD.

What were our lessons learnt today?

  • If you don’t succeed with maven’s automatic processes, do not try to sort out things manually. You’ll just end up with a gigantic mess that won’t work either way. The best way to deal with maven failures is to revert everything and try again with different parameters.
  • The best approach to develop hudson plugins is to adapt the old “monkey see, monkey do” process. There are so many plugins already, chances are good your immediate question was already answered somewhere. Just check the found solution for accidental complexity. Sometimes, the first solution isn’t the easiest.
  • When dealing with the legacy Win32 API, combined knowledge scraping is king. We had discussions throughout the day that only consisted of little parts of recollections about knowledge that seemed long forgotten. But finally, we put the pieces together and solved the problem. It should be called teamthink, i guess.

Retrospective on the OSLD

The weather at this OSLD was way too hot to operate at normal speed. But we got some nice results and a cliffhanger for the next OSLD. We left soaked with sweat but happy that evening.

Open Source Love Day May 2010

Last friday, we held our Open Source Love Day for May 2010. After the excessive yak shaving of last month’s OSLD, this one was more productive. We got one feature right nearly instantly and several others throughout the day. To keep us from feeling depressed, we allocated dedicated “research time” to get to know a project before starting to develop on it. So one point on our current list of accomplishments is only research work that should be counted as “thought investment” for further improvements.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

On this OSLD, we accomplished the following tasks:

  • Netty is a java framework to perform network communication, transparently using different transport mechanisms. We use it in a customer project environment and it worked mostly well. But we noticed some issues that might have to do with the framework, so we decided to put it under development. But before hopping in right away, we invested some hours to really anatomize it. Our achievement isn’t noticeable yet, but might be in the future.
  • Launch4j is a java application launcher for Windows, handling all the stuff a startup script (the infamous start.bat) would do, too. One thing that’s easy to perform in a batch script is to restart the application if it went down with an error. We wanted this feature in Launch4j and we succeeded. Before releasing the changes to the project, we need some rethinking of our approach, as it is very special to our case, but if you are interested, we send you the patch on request.
  • RXTX is a project to perform serial (RS232) communication. Over the course of the last OSLDs, we worked on an issue that appears in combination with serial converters. See the OSLD posting for march 2010 for details. This issue seems to be completely fixed now. We will run some longterm stability tests with it and then release a patch to the issue tracker of RXTX, issue #144.
  • Our internal tool for business administration was greatly improved this time. The tool isn’t open sourced yet, but we are not afraid to publish it in the future. The new features will directly decrease our administrative overhead and give us more time to perform our main duty – developing software.

A sidenote regarding the yaks (see last month’s OSLD posting to understand this one)

Our monthly “Homepage Comittee” meeting went smooth this time. The setup pain of last month is now paying out and we got our homepage updated. It seems the yaks haven’t grown beards again yet.

What were our lessons learnt today?

  • Pure C code tends to produce endless code blocks. Simple refactorings like extract method/function take a long time and aren’t supported by the IDE. The sentence “there is no boolean!” left some of us impressed, too.
  • The C/C++ Eclipse workbench (in fact, a bunch of plugins for Eclipse) was a joke instead of a help. It didn’t exactly prevent us from doing our work, but that’s all it did. Launch4j came along with project files for the Bloodshed Dev-C++ IDE, which – besides its bloody name – seems to be up to its tasks. We decided to use Dev-C++ over Eclipse for development on Launch4j.
  • Hpricot is a very versatile HTML parser/manipulator. It is just pure joy to work with. (Ok, we knew that one before)

After the fruitless OSLD of april, this one was a great relief for us. The new restart feature for Launch4j worked out instantly and will solve many of our current immediate problems (and some of yours, hopefully). The solution for RXTX got us out of trouble, too. All in all, this was a successful OSLD.

Open Source Love Day April 2010

Today, we held our Open Source Love Day for April 2010. Well, to be honest, we tried to. We didn’t achieve that much to present publically and felt really depressed at the end of the day. At least, we can try to explain what went wrong. It all has to do with our own breed of yaks that needed to be shaved today. But first, lets explain the basics.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

Today, we reached out to perform the following tasks:

  • RXTX is a fine java library to access the (rather legacy) serial port to send and receive data over the RS232. On our last Open Source Love Day, we proposed a patch for a problem with disappearing emulated COM ports. This happens a lot if you use USB or ethernet converters and unplug them accidentally or because your intranet got shut down. The patch works well, but not perfectly. We noticed that the IOException we introduced gets thrown much too often. Today, we wanted to fix our fix. We only came to the conclusion that the problem might be a lot more complex than we initially thought.
  • Launch4j is another java library/tool we use to deliver production-ready software bundles. It’s a launcher that increases the user experience at the critical “first contact” for Windows users. The project is mature and useful, but out of development for nearly two years now. We want some features in it, so we thought we might build them today. But the yaks didn’t let us.
  • We have an internal tool for our business administration that desperately needs some hugs. So even if this isn’t strictly Open Source yet, we allocated some time to add a few features. But they outgrew their timebox, too.

At the end of the day, we didn’t accomplish anything ready to be released. That’s frustrating to us and has no value to anybody else. We promise to do better next time.

How the yaks got shaved

Every organization has a secret stable where yaks are raised just for the sake of raising yaks. Once a year or so, they need a shave, so you’ll find yourself shaving yaks before you know what’s happening. That’s what happened to us today.

The term “yak shaving” isn’t our invention. It describes the result of a process when in order to accomplish task A, you need to do task B first. Task B can’t be done before tasks C and D both are completed. Task D, you figure it already, is dependent on task E, which in turn… Soon and inevitably, you’ll find yourself shaving yaks because it is what needs to be done to accomplish your todo list.

This is our story: On every Open Source Love Day, we also gather to have a short “Homepage Comittee” meeting. This is our formal approval process to publish content on our website. One improvement we wanted to implement today was an automatic publishing process. It copies the content of our internal Content Management System to our webserver. Simple as that. This was the starting point of an odyssee.

To copy the content, we used wget. But wget in the version of our distribution had a bug (and missed a feature), so we needed to update wget… Soon afterwards, we found ourselves upgrading our main PostgreSQL database (which went incredibly smooth but took forever) and wrestling with certain dependencies of other parts of the system. We realized we are now officially shaving yaks when we got the error message:

You need to have ruby gems 1.3.2 installed. Your version is 1.3.1!

Remember we started with taking a snapshot of our CMS using wget. Clearly, separating our services to more machines would have helped a lot. But that would’ve taken more than one day to do. It’s on our wish list for the next Admin Day.

Let’s conclude this blog post with the insight that we need to timebox our work better when working for the OSLD. And our yaks are shaved and ready for the summertime.

Open Source Love Day March 2010

Yesterday, we held our first Open Source Love Day (OSLD) for this year. The last OSLD was at December 2009. Then, we reassigned a day in January and February each to perform our relocation to the new (and much bigger) office. But now we are back to regular duty and had the time to donate some work back to the Open Source ecosystem.

The Open Source Love Day

We introduced a monthly Open Source Love Day to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

Participate at our OSLD by using the features we’ve built today:

  • Grails still has some bugs. Instead of only complaining about them, we try to fix them. There is a bug with checkboxes and nested boolean properties that bugged us in a customer project. It’s filed unter GRAILS-3299 and has a proposed patch now.
  • In previous OSLDs, we produced the cmake hudson plugin. In the corresponding blog entry, comments with bug reports began to pile up. They addressed issues with hudson master/slave setups. So we implemented a hudson master/slave test environment, using VirtualBox virtual machines to perform as slaves. This setup quickly revealed the problems that were typical enough to devote a complete blog entry about this topic soon. Fixing the problems resulted in the new cmake hudson plugin version 1.2 to be released yesterday.
  • We are using the RXTX project to perform serial (RS232) communication in several projects. We are really glad the project exists, because the “official” communications API from Sun/Oracle is nothing but a mess. With RXTX, we only had a problem with emulated COM ports. Emulated COM ports exist when you use a USB->Serial or Ethernet->Serial converter, which is what our customer chose to do. If you unplug the converter during operation, the corresponding COM port disappears. This causes RXTX to crash, bringing the JVM down, too. We wrote a test application and used it with every converter we own (and we own quite a lot of them!). Then we began tracing the RXTX source code (at C code level), altering it to “only” throw an IOException when the virtual COM port disappears. The corresponding patch will be proposed to the RXTX project soon.
  • Another API we use a lot is the tiny winp project, written by Kohsuke Kawaguchi, the creator of hudson. We kill Windows processes with it, within a project that runs on Windows 2000, Windows XP and Windows 7. The latest Windows version seemed incompatible with winp, even the 32bit edition. We didn’t find the cause for this, but developed a workaround that will be proposed to the winp project soon.

What were our lessons learnt today?

  • If you face OutOfMemoryErrors on a 64bit Java6 JVM, try to switch back to a 32bit Java5 JVM. It helped us with our Grails bugfixing (during the test phase).
  • Hudson Master/Slave support for plugins isn’t particularly hard. It’s just that you need to be aware of the topic and replace some types like java.io.File. We gathered the same experience twice with our Crap4J plugin and the cmake plugin. It’s time to tell the world about it. Stay tuned!
  • The good old error return code is an error prone coding paradigm, because all too often, users of a function/method just forget to check the returned result. This was the case with a call to WaitForSingleObject in RXTX.
  • If you don’t understand an implementation well enough to fix the cause, you might at least be able to produce a workaround. It’ll work for you and provide guidance for the original author about where the bug might hide. This is why we count our winp efforts as success, too.
  • Your project either is mavenized or it isn’t. Everything in between is half-assed.

This OSLD was a bit short, as we had some guests in the evening, but nevertheless, it was fun. Well, to be precise, it was this special software engineer’s type of fun: The whole company was remarkably quiet most of the day, with everyone working totally focussed. We scratched our own itches, enhanced our customer projects and contributed to the open source community. A very good day!

Stay tuned if you want to know more about the specifics of the hudson plugin development or the to-be-proposed patches. We will publish them here.