Open Source Love Day November 2009

November 25, 2009

Today, we celebrated our third Open Source Love Day (OSLD). It was slightly degraded by a company-wide illness outbreak, but we tried to make the best out of the situation.

Open Source Love Days are our way to show our appreciation and care to the Open Source software ecosystem. Our work wouldn’t be as fun or just not possible without professional Open Source software. You can read more about our motivation and specifica in our first OSLD blog posting.

You can participate at our OSLD by using the features we’ve built today:

  • We think git is an enrichment to the SCM market, as is Eclipse to the IDE market. Improving the quality of Eclipse’s git plugin is the next logical step. The ability to diff the content of two revisions in EGit was committed today. As a bonus, the name of the committer shows up in the right manner, too. See the screenshots to get the idea:

When developing EGit, we were already using it to pull the sources. Unfortunately, the repository URL changed bigtimes since our checkout without us noticing. This got us into trouble trying to follow the contributor guide. The command line version of git isn’t that communicative yet. But after all, this is a great time to learn about the real world problems when using git. The EGit contributor guide itself is a fantastic way for a project to show initial appreciation to volunteer efforts. Thanks for caring, guys! If you are interested to review our changes yourself, fetch the patch.

  • Another part of today’s work was on the KDevelop project. We tried to fix some outstanding little features or bugs, whatever is on the list of KDevelop 4. But we spent our day fixing our development machines instead. The Ubuntu linux operating system (8.10) was way too old to get useful results and KDE needs to be up-to-date to develop KDevelop. Besides our sluggishness to keep our virtual machines on the bleeding edge, the checkout experience of KDevelop was rather sleek. What bothers us a bit is the ominous entanglement between KDevelop and KDE. It seems you can’t have one without the other and need to master both to make a stand.
  • As a third part, we wanted to contribute to the TANGO project (not the useful icon collection, but the useful control system). They migrated their main repository from CVS to SVN lately, but the migration seems unfinished still. At least, the migration effort lacks public documentation for the occasional contributor. That’s a real showstopper, because you never get beyond the very first step: setting up a working project. We won’t give up and email the project leads on this topic, but it didn’t fit into this OSLD.

What were our lessons learnt today?

  • Just having a possibility to view or download the source code doesn’t make an Open Source project. The key to success is the ability of complete strangers to hop in and perform useful work. Having terse, but accurate documentation helps a lot. The EGit contributor guide is a good example of a single document that makes the difference. If you own an open source project and want to attract occassional contributors (like us), write such a document and watch us (and others) drop you a patch. That said, we come to belief that the person that writes technical documentation for the developers is one of the most important roles on a project. Perhaps we join some projects in the future to fill that role.

To sum it up, this OSLD was limited from the beginning by developer availibility. With lacking documentation, we nearly grinded to a halt. We look forward to our next OSLD in December.


We are software tailors

November 23, 2009

Our company is called Softwareschneiderei (which is German for software tailoring). This name describes our intention to write bespoken software, software that fits people perfectly. Over time different additional metaphors from the tailor’s world came around: seams/tucks which describe places in software systems where cuts can be made and testing can be done. Tailoring is a craftsmanship so an apprenticeship model and the pride in our work exists.
This describes the mentoring and bespoken software development we do. But besides that we do a lot of bug fixing, improvement of existing software which was written by others and evaluation of other people’s code. Thanks to a piece from Jason Fried (thanx Jason!) those other parts fit perfectly into our vision as software tailors: we iron/press (fix bugs, improve the code), we trim and cut (remove bottlenecks and unwanted functionality or extend the software to use other systems) and we measure (analyze, inspect and evaluate systems).


Speed up your buildbox, Part III: Memory

November 16, 2009

© Friedberg - Fotolia.comIn the first and second part of our effort to speed up our buildbox, we replaced the harddisk with a RAM disk and swapped in a bigger CPU. This brought the build time down from 03:30 minutes to 02:00 minutes.

Boosting the memory

When we began the journey, we wanted to undercut the 02:00 minutes threshold. The last component that directly impacts performance of our box was the memory. We started out with 4 GB of DDR2-800 modules. To have a feeling for the effects, we upgraded to 4 GB of DDR2-1066 first and then added another 4 GB, resulting in 8 GB of RAM. We expected the performance gain to be small, but noticeable. The RAM disk, for example, is directly affected by memory speed.

As much, but faster

The first upgrade brought the first surprise: Upgrading from DDR2-800 to DDR2-1066 modules didn’t change anything. It’s not that the mainboard or CPU doesn’t support the faster RAM, it just seems to be fast enough, despite the data bus clock rate. Our build process still took 02:00 minutes, reproducible and without exception.

Filling all the banks

The mainboard can load up to 16 GB of RAM, but our budget just allowed to buy 8 GB of DDR2-1066 RAM. We installed it and ran the same 32 bit Ubuntu Linux as before. The build process took 02:00 minutes, which was expected now.

Changing to 64bit

We changed to boot harddisk, installed a 64 bit Ubuntu Linux and ran the build again. Still 02:00 minutes. The switch to 64 bit wasn’t a big deal with Java, but some of the included native libraries complained about the change. Recompiling them solved the issue.

Finally reaching the target

As a last measure, we increased the maximum memory of the build JVM to the biggest value it would accept. This was -Xmx2600m, a surplus of 600 MB to the original setting. This sped up the build process by five seconds, it took 01:55 minutes now.

Conclusion and perspective

We’ve reached our anticipated target of less than two minutes build time. We exceeded our original budget of 500 EUR, but bought some parts that finally weren’t used in the build box, but elsewhere. The two parts that made the whole difference were the CPU and some more memory to spend it on the RAM disk.

If you want to speed up your single build box, aim for the CPU/RAM combo and try to install a RAM disk to perform all the work on.

This leads me to the perspective of the next part of the series: If you plugged in the most expensive CPU and enormous amounts of RAM to speed up your buildbox, you still aren’t done. You should invest some time to look into distributed builds. Hudson as our continuous integration server provides nearly instant “build slave” support. With this feature, you can set up a whole build farm to further increase your build throughput.

Stay tuned for “Part IV: Beyond the box”


CMake Builder Plugin Reloaded

November 9, 2009

A few months ago I set out to build my first hudson plugin. It was an interesting, sometimes difficult journey which came to a good end with the CMake Builder Plugin, a build tool which can be used to build cmake projects with hudson. The feature set of this first version was somewhat limited since I applied the scratch-my-own-itch approach – which by the time meant only support for GNU Make under Linux.

As expected, it wasn’t long until feature requests and enhancement suggestions came up in the comments of my corresponding blog post. So in order to make the plugin more widely useable I used our second  Open Source Love Day to add some nice little features.

Update: I used our latest OSLD to make the plugin behave in master/slave setups. Check it out!

Let’s take a walk through the configuration of version 1.0 :

Path to cmake executable

1. As in the first version you have to set the path to the cmake executable if it’s not already in the current PATH.

2. The build configuration starts as in the first version with Source Directory, Build Directory and Install Directory.

CMake Builder Configuration Page

3. The Build Type can now be selected more conveniently by a combo box.

4. If Clean Build is checked, the Build Dir gets deleted on every build

Advanced Configuration Page

5. The advanced configuration part starts with Makefile Generator parameter which can be used to utilize the corresponding cmake feature.

6. The next two parameters Make Command and Install Command can be used if make tools other than GNU Make should be used

7. Parameter Preload Script can be used to point to a suitable cmake pre-load script file. This gets added to the cmake call as parameter of the -C switch.

8. Other CMake Arguments can be used to set arbitrary additional cmake parameters.

The cmake call will then be build like this:

/path/to/cmake  \
   -C </path/to/preload/script/if/given   \
   -G <Makefile Generator>  \
   -DCMAKE_INSTALL_PREFIX=<Install Dir> \
   -DCMAKE_BUILD_TYPE=<Build Type>  \
   <Other CMake Args>  \
   <Source Dir>

After that, the given Make and Install Commands are used to build and install the project.

With all these new configuration elements, the CMake Builder Plugin should now be applicable in nearly every project context. If it is still not useable in your particular setting, please let me know. Needless to say, feedback of any kind is always appreciated.


Blog harvest: Metaprogramming in Ruby,Hudson builds IPhone apps, Git workflow, Podcasting Equipment and Marketing

November 4, 2009

harvest64
Four blog posts:

  • Python decorators in Ruby – You can do amazing things in a language like Ruby or Lisp with a decent meta programming facility, here a language feature to annotate methods which needed a syntax change in Python is build inside of Ruby without any change to the language spec.
  • How to automate your IPhone app builds with Hudson – Another domain in which the popular CI Hudson helps: building your IPhone apps.
  • A Git workflow for agile teams – As distributed version control systems get more and more attention and are used by more teams you have to think about your utilisation of them.
  • Podcasting Equipment Guide – A bit offtopic but interesting nonetheless: if you want to do your own podcasts which equipment is right for you.

and a video:


A more elegant way to HTTP Requests in Java

November 2, 2009

The support for sending and processing HTTP requests was always very basic in the JDK. There are many, many frameworks out there for sending requests and handling or parsing the response. But IMHO two stand out: HTTPClient for sending and HTMLUnit for handling. And since HTMLUnit uses HTTPClient under the hood the two are a perfect match.

This is an example HTTP Post:

HttpClient client = new HttpClient();
PostMethod post = new PostMethod(url);
for (Entry param : params.entrySet()) {
    post.setParameter(param.key, param.value);
}
try {
    return client.executeMethod(post);
} finally {
    post.releaseConnection();
}

and HTTP Get:

WebClient webClient = new WebClient();
return (HtmlPage) webClient.getPage(url);

Accessing the returned HTML via XPath is also very straightforward:

List roomDivs=(List)page.getByXPath("//div[contains(@class, 'room')]");
for (HtmlElement div:roomDivs) {
  rooms.add(
    new Room(this,
      ((HtmlElement) div.getByXPath(".//h2/a").get(0)).getTextContent(),
      div.getId())
  );
}

One last issue remains: HTTPClient caches its cookies but HTMLUnit creates a HTTPClient on its own. But if you override HttpWebConnection and give it your HTTPClient everything works smoothly:

public class HttpClientBackedWebConnection extends HttpWebConnection {
  private HttpClient client;

  public HttpClientBackedWebConnection(WebClient webClient,
      HttpClient client) {
    super(webClient);
    this.client = client;
  }

  @Override
  protected HttpClient getHttpClient() {
    return client;
  }
}

Just set your custom webconnection on your webclient:

webClient.setWebConnection(
  new HttpClientBackedWebConnection(webClient, client)
);

A Campfire plugin for Hudson

October 26, 2009

Our last OSLD resulted in a new hudson plugin: a build notifier for Campfire. Right now it is pretty simple and just posts the last build result in a chat room:
Campfire-Notification

You can configure your account data and the room to join in the job config page under Post Build actions:
Hudson-Campfire
But there is more to come:

  • global config
  • SSL support
  • a link back to hudson

or what is your favorite missing feature?


Blog harvest, October II

October 25, 2009

harvest64A great way to stay up to date with current musings and hypes of our industry is to follow other people’s blogs. We do this regularly – everybody scans his RSS feeds and roams the internet. But to have a pool of shared knowledge, we pick our favorite recent blog articles and usually write an email titled “blog harvest” to the rest of the company.

Then, the idea came up to replace the internal email by a public blog post. So here it is, the first entry of a new category called “blog harvests”. You’ll read more harvests in the future. They will be categorized and tagged appropriate and have the harvest icon nearby.

Second Blog harvest for October 2009

There are four main blog entries I want to share:

  • 8 Signs your code sucks – Let’s assume we all read Martin Fowler’s classic “Refactoring” book, then these eight signs are a mere starter. But as the follow-up post indicates, it got quite a few people started and upset for the “comments are code smells” line. Well, we heartfully agree with the premise that comments are clutter and code should be the comment. /* TODO: Add a joke using comments here */
  • ORMs are a thing of the past – Another opinion that might get in the way of hibernate fanboys. We’ve had our share of hibernate “experiences”. It’s a useful tool if you know how to use it – and when not to. Replies followed instantly, here are two noteworthy ones by Scot Mcphee and by Jens Schauder.
  • The Case for Clojure – Clojure is functional programming on the Java VM (think LISP). Stay tuned for our own book review on this topic. You can argue that Clojure isn’t pure, though.
  • Bad Programmers Create Jobs – As is already is a controversy harvesting, lets add some more, written by Mohammad Azam. Side note: Half of our work was initially created by “bad” programmers, so I think Mohammad hit the nail on the head. And remember that you’ve produced legacy code today.

Then there is a bit of (future) knowledge you shouldn’t miss:

That’s it for now. My harvest format has changed for the blog, i’ll evolve it further in the next months, Thanks for your attention, stay tuned.


Follow-up to Schneide “Dev Brunch” October 2009

October 22, 2009

brunch64Last weekend, we held our October Dev Brunch in the rooms of our company. This posting is the follow-up, summarizing the topics and providing additional information.

The Dev Brunch

Let me start by introducing the concept of a “Dev Brunch” as we perform it. Once a month, we spend nearly half a day of the weekend by meeting and talking about topics related to software development. The meeting starts at perfect brunch time, everybody brings along some brunchable food and the party begins.

Everybody who attains the Dev Brunch has to prepare a topic to tell about. We set a limit of 15 minutes for the talk and unlimited time for questions and discussion. We elect a moderator, though, to bring us back on course when we disgress too much.

To prepare a topic isn’t hard work. No slides are required, no written handout or code examples. You just have to work up a topic to fit it into 15 minutes.

The October 2009 Dev Brunch

The topics of this session were:

  • Java’s upcoming Fork/Join framework - Java 5 brought the util.concurrent classes, Java 7 will bring the Fork/Join framework to further ease concurrency in Java.
  • The current status of JIT on mobile devices – the tagline was “why is my Android phone so slow?”. This talk even included slides!
  • Project estimation with planning poker – the talk gave away the secrets of planning poker and even more secrets of how to sell it to the management.
  • Pitfalls of unit testing Spring infected code – Developers often mix up the framework with its concepts. The example given was dependency injection (concept) vs. Spring (framework).
  • First impressions of Scala – Tales of a first contact with Scala from a Java developer.

Several talks included bonus material that will be provided in the comment section of this blog posting. Most material will be in german, as were the talks. But to ease our international readers: most links within the bonus material point to english articles.

Stage your own Dev Brunch

We cannot stress this enough – holding your own Dev Brunch isn’t complicated but very valuable. Just invite your mates and bring food. Once you started, you’ll attract other developers from your vicinity and get to know them in an informal manner.


Speed up your buildbox, Part II: Processor

October 19, 2009

© Friedberg - Fotolia.comIn the first part of our effort to speed up our buildbox, we replaced the spindle harddisk with a Solid State Disk (SSD) and finally, a RAM disk. This brought the build time down from 03:30 minutes to 02:50 minutes.

The Central Performance Unit

The next step on our journey to a faster buildbox was to replace the processor. Our initial processor was an Intel Core2 Duo E6750 with 2.67 GHz. To our pleasure, the processor socket, namely the LGA775 socket, is extremely versatile in supporting different processors. We had no problem in plugging in faster dual or even quad core processors, except upgrading the BIOS.

Taking the 3 GHz mark

The next processor to try out was an Intel Core2 Duo E8500 with 3.17 GHz operating frequency. The L2 cache went up from 4 MB to 6 MB.

The build time went down immediately from 02:50 minutes to 02:20 minutes. That’s nearly 20 percent less build time. And it’s perfectly linear with the CPU speed increase (also nearly 20 percent).

As a result: Investing in CPU clock power seems to pay off. The higher the frequency, the lower the build time.

Doubling the cores

Fortunately, the LGA775 socket supports quad core processors, too. We plugged in a Core2 Quad Q9550 with 2.8 GHz and ran the build again.

The result was astonishing: Despite the lower frequency, the build time dropped from 02:20 minutes to 02:00 minutes. We can’t really explain this one with basic math like the frequency coupling of the dual cores.

If your build is perfectly multithreaded, something javac isn’t, you’ll notice an even bigger speedup.

To sum it up: you can’t have enough GHz or processor cores when running a build.

Reviewing the result

We replaced the harddisk with RAM and upgraded the processor to meet the current performance threshold. This brought us from a starting 03:30 minutes build time to 02:00 minutes now. The CPU is the major player in this game, so upgrade it first.

Outlook on the third part

But what about the RAM? We really wanted to know what happens when we replace the RAM with bigger and faster one. Read more about this experiment in the third part of the series, coming soon.


Follow

Get every new post delivered to your Inbox.

Join 79 other followers