Skip to content

Using wget to verify on the fly page compression

I’ve written about wget before, but I just found a very cool use for it. I’m looking at ways to speed up a site, by stripping out whitespace. I found one servlet filter by googling around. However, that code has a license restriction for commercial use (you just have to pay for it).

A bit more looking and I found this fantastic article: Two Servlet Filters Every Web Application Should Have. Give it a read if you do java web development. The article provides a free set of filters, one of which compresses a servlet response if the browser can handle it. (I’ve touched on using gzip for such purposes before as well.)

I was seeing some fantastic decreases in page size. (The article states that you can’t tell, but FireFox’s Page Info command [ Tools / Page Info ] seemed to reflect the differences.) Basically, a 300% decrease in size: 50K to 5K, 130K to 20K. Pretty cool. Note that if your application is CPU bound, adding this filter will not help performance. But if you’re bandwidth bound, decreasing your average page size will help.

I couldn’t believe those numbers. I also wanted to make sure that clients who didn’t understand gzip could still see the pages. Out comes trusty wget. wget url pulls down the standard sized file. wget --header="accept-encoding: gzip" url pulls down a gzipped version that I can even ungzip and verify that nothing has changed.

The only issue I saw was that ‘ c ‘ is apparently rendered as the copyright symbol in uncompressed pages. For compressed pages, you need to use the standard: ©. Other than that, the compression was transparent. Way cool.

GWT Mortgage Calculator Conclusion

This past week has been a whirlwind, including a whole lot of learning on my part and three releases of the Colorado HomeFinder site. Just to rehash, I’ve built three versions of the software:

In general, from a java developer perspective, this is my take on the strengths and weaknesses of GWT.


  • Can use normal java dev/debug environment–in particular, being able to set breakpoints in what will be javascript was useful
  • Hides javascript cross browser messiness
  • Allows rich uis to be built for web using same paradigms as other windowing toolkits (event listeners, layouts, panels, etc). As some folks have said, it is Swing for the web.
  • Ability to create widget libraries–using module ‘inheritance’ you can easily leverage other folk’s work. See this list for a collection of GWT widgets.


  • Build integration–I really don’t understand why they haven’t wrapped in an ant task, though others have done it for them, and there is an -ant switch on projectCreator.cmd which generates a stub build.xml
  • Documentation–again, I’ve found the Google Groups to be tremendous, while the Google provided documentation could use some brushing up.
  • Some cross browser weirdnesses–the only serious one I saw was IE not responding to click events on one page I built. This was due to a table of width 100% next to the div where the GWT widgets were dynamically creating DOM elements. But it only showed up on IE.
  • Any ui built by GWT is not indexable by search engines. This necessarily limits what you can do with it–you can build web applications but not web sites.

How to receive craigslist searches via email

Updated 1/7/2008: Note that Squeet appears to be non functional at this time. still appears to work.  Please let me know (via the comments) if there are other sites that provide this valuable service.

craigslist is an online classified ad service, with everything from personals to real estate to bartering offered online. I’ve bought a table from Denver’s craigslist and I know a number of folks who have found roommates via craigslist.

If you have a need that isn’t available right now, you can subscribe to a search of section of craigslist. Suppose you’re looking for a used cruiser bike in Denver, you can search for cruisers and check out the current selection. If you don’t like what you see but don’t want to keep coming back, you can use the RSS feed link for the search, which is at the lower right corner. Put this link into your favorite RSS reader (this is a simple application that manages RSS feeds, which are essentially lists of links. I’d recommend Bloglines but there are many others out there) and you can be automatically apprised of any new cruisers which are posted.

(You find tons of stuff via RSS–stock quotes, job listings, paparazzi photos… The list is endless.)

If you don’t want to deal with yet another application, or you’re not always in your RSS reader (like me), you can set up an RSS to email gateway. That way, if your cruiser bike search is so urgent you don’t want to let a good deal get away, you receive notification of a new posting relatively quickly. If you want, you can even email it to your mobile phone.

The basic steps:

  1. Go to the Squeet signup page. Sign up for a free account. Don’t forget to verify it–they’ll send an email to the address you give them.
  2. Open up a new browser window and go to craigslist, choose the city/section you are interested in, and do a search. The example up above was ‘cruiser’, in the bike section of the Denver CL.
  3. Scroll down to the bottom of the search results and right click on the RSS link. Choose either ‘Copy Shortcut’ or ‘Copy Link Location’, depending on your browser.
  4. Switch back to the Squeet window, and click in the ‘FEED URL’ box. Paste in the link you just copied. Choose your notification time period–I’d recommend a frequency of ‘live’, since cruiser bikes in the Denver area tend to move pretty quick. Then click the subscribe button.

That’s it. Just wait for the emails to roll in and soon enough you’ll find the cruiser bike of your dreams. Just be aware that it’s not real time–I’ve seen lags from 30 minutes to 2 hours from post to email. Still, it’s a lot easier that clicking ‘Refresh’ on your browser all day and night.

Step #3: Add a tabbed interface and other enhancements

Previously, I described how to integrate a network call into the GWT mortgage calculator. (See Step #2, Step #2a and Step #2b.) Now I’m going to add an another, more complex mortgage calculator panel. This will add in more real world costs. Both the simple and advanced calculators will be available at the same time, using a tabbed interface. Additionally, I’ve added a servlet which will serve up correct JSON in hosted mode, which means that anyone who wants to download and play with this code will be able to do so without any software beyond the source, the GWT, and a modern version of Eclipse. See the new tabbed interface in action and download the source. (Please note that the interface online is not exactly the same as the interface you can build via the download–my client had me make a couple of changes.) I’ll also talk about how I integrated this calculator onto a property specific page.

Adding a tabbed interface was actually quite easy. I used a TabPanel. The only hiccup was that you cannot reuse widgets. I thought it’d be nice if the loan amount, for example, stayed the same between the tabs. However, if you place a widget in two different containers, the last one wins–the widget simply isn’t in the first container. I tested this with the VerticalPanel and the FlexTable. Instead, I had to create new widgets. I’m sure that I could have achieved synchronicity between widgets with a KeyboardListener, but the payoff didn’t seem worth the hassle. Other than that, this is a minor revision. I did some refactoring and pushed validation up to the button widgets.

The JSONServlet shows how to keep server and client source in the same tree, and also provides a simple way to toy with JSON client-server communication. I tested this and you should be able to simply untar the source, import the project into Eclipse, do a bit of GWT judo, and run the project. Steps (only tested for windows, sorry) to reproduce the final working product:

  • You will have to download the GWT since I can’t legally redistribute it.
  • Unzip it into, for example, c:\gwt
  • Download and untar the mortgage calculator source, say into c:\mtgcalc
  • Copy gwt-user.jar and gwt-dev-windows.jar into the lib directory, for example c:\mtgcalc\lib. (This lets you run the MortgageCalc-compile.cmd batch file to compile the app from the command line. If you just want to see the javascript in action, you can stop there and copy the resulting .html/.xml/.gif files to a host someplace and view in a browser.)
  • To set up the debugging, start Eclipse.
  • Import the project vai the File \ Import menu item.
  • When you’re ready to run the browser in hosted mode, edit the launcher in Eclipse. Choose the Run menu, then the Run… option.
  • Select the MortgageCalc application (under Java Application)
  • Choose the Classpath tab. Remove the gwt-user.jar and gwt-dev-windows.jar files.
  • Click under ‘User Entries’
  • Click the ‘Add External Jars’ button.
  • Navigate to where you unzipped GWT (for example, c:\gwt and select both gwt-user.jar and gwt-dev-windows.jar
  • Click ‘Open’
  • You should see the two jars with this icon next to them: . If you see this one instead: , that means you added them as jars, not as external jars, and you won’t be able to run in hosted mode. (Updated 6/21: You’ll get the error described here: [ERROR] The browser widget class could not be instantiated java.lang.UnsatisfiedLinkError: Unable to load required native library 'gwt-ll')
  • Click ‘Apply’ and then ‘Run’.

The next step is to integrate the calculator into an existing JSP and pull some information from that JSP page. Since this is tightly integrated into the Colorado HomeFinder site, code would probably not be useful, so it’s not provided. However, if you’d like to see the results, visit Colorado HomeFinder and search for a home, click through to a property and then click the ‘Mortgage Payment’ link. (Sorry this process is not streamlined, but any link I put to a specific property page on that site will expire in a few months.)

The ideal calculator gets listing price and other information from the server, and prepopulates the appropriate fields. However, the calculator needs to know what property it is being shown for; in other words, it needs to pick property specific data out of the page. There are a couple of possible methods: I could write a javascript variable to the page and use JSNI to turn it into a data structure that my compiled gwt code could read. Or I could put text into a hidden div and then use the Google DOM classes to get the data.

Since what I needed to retrieve was a few simple numeric values, I went with the DOM option. If I had a more complex data structure to communicate, using JSNI might have been a better option. I used getElementById to find the appropriate div and getInnerText to get the contents of that division. With those pieces, I could query back to the server and get the price, taxes, etc. which I would use to fill in the appropriate text boxes on the mortgage calculator.

And I believe that’s all for now. There are, of course, many places where Icould expand this calculator, but this is functional enough and we now have an understanding of the strengths and weaknesses of GWT. And that’s the subject of my next post.

Google offers geocoding

Google now offers geocoding services. Up to 50,000 addresses a day. I built a geocoding service from the Tiger/Line database in the past. Comparing its results with the Google geocoding results, and Google appears to be a bit better. I’ve been looking around the Google Maps API discussion group and the Google Maps API Blog and haven’t found any information on the data sources that the geocoding service uses or the various levels of precision available.

Step #2b: Updating the GWT Client to communicate with the Server

The next step is to update the client to communicate with the server side components. Again, if you’re not very patient, you can see the live client code at Colorado HomeFinder, and download the source.

I need to add a drop down, which will offer users the option of a 5/25 ARM rate, or a 30 year fixed rate. I need to have some way of decoding the JSON response from the server, and I need to update the interest rate text box with the results from the server.

Adding the drop down is fairly easy: Google provides a listbox. The listbox is an instance variable, rather than a local variable, and I also change the interest text box from a local variable to an instance variable. The reason: the change listener, which is a non static member class, has easier access to an instance variable than to a local one. Here’s the relevant code:

rateChoiceBox.addItem("5/25 ARM");
rateChoiceBox.addItem("30 Year Fixed");
final RatesResponseHandler rrh = new RatesResponseHandler();
HTTPRequest.asyncGet(RATES_URL, rrh);

rateChoiceBox.addChangeListener(new ChangeListener(){
public void onChange(Widget sender) {
HTTPRequest.asyncGet(RATES_URL, rrh);
private class RatesResponseHandler implements ResponseTextHandler {

public void onCompletion(String responseText) {
boolean keepGoing = true;
JSONObject jso = null;
try {
jso = JSONParser.parse(responseText);
} catch (JSONException je) {
keepGoing = false;
// not sure what to do
String rate = "";
// ... get the data, put it into the rate variable.

In the RatesResponseHandler, I try to parse the JSON response that has been asynchronously downloaded from the server, and if I get meaningful text, I set the interest rate in the textbox: MortgageCalc.this.interest.setText(rate); If the server is not available, we still have a sane default. One interesting item: when you run the code with the server off in development, GWT pops you into the MS script debugger, which at least lets you know something is wrong.

The JSON processing code has already been written, so it made sense to leverage the JSON example that Google kindly provided. To do this, I made a few changes. Luckily, all the example code is made available under the Apache 2.0 license. You can find json.jar in the download, should you want to use it for a project of your own.

First, I wanted to make sure that none of the behavior of the EntryPoint JSON class occurred–I wanted the subsidiary libraries, but not the potato demostration. Therefore, I eviscerated the JSON class, and gave it an empty onModuleLoad method. On reading the module documentation, it appears this wasn’t needed–I could have just eliminated the <entry-point> entry.

Then, I started building the JSON code library. To pull in a set of external widgets, you need to do a couple of things.

  1. Build the library. I did this by compiling the JSON source, from the samples directory: mkdir builddir && javac -d builddir -classpath `cygpath -wp $PWD/bin/:.:../../../gwt-windows-1.0.21/gwt-user.jar` src/com/google/gwt/sample/json/client/JSON*.java. (The cygpath stuff is there because I’m developing on cygwin.) One thing to remember–you need to copy the java files into that same build directory: cd builddir/com/google/gwt/sample/json/client && cp ../../../../../../src/com/google/gwt/sample/json/client/*.java .. (Apparently you need the .class files for Eclipse or any other IDE, and you need to .java files so the Google compiler can compile the java to javascript. More here.) Also, don’t forget the JSON.gwt.xml, which should go in builddir/com/google/gwt/sample/json.
  2. Jar up the code, and place it in the classpaths. To compile from the command line, I added it to the Mortgage-compile.cmd batch file. For Eclipse development, I added it to the project buildpath (instructions for doing so). And I also had to add the json.jar file to the MortgageCalc application, so I could debug in Eclipse. To do this, choose ‘Run’ from the top menu, choose the ‘Run…’ option, select ‘MortgageCalc’, click the ‘Classpath’ tab, click ‘User Entries’ and add the jar using the Add button.
  3. Inherit from the JSON module. As far as I can tell, this isn’t inheritance in the usual object oriented sense, it’s more like importing packages in java. I did this by adding this line to the MortgageCalc.gwt.xml: <inherits name=''/> (I did try to use a source path to refer to the JSON classes, rather than inherit from them, but got this error:

    [WARN] Non-canonical source package: ../../../google/gwt/sample/json/client/

    [ERROR] Unable to find type ‘com.cohomefinder.gwt.mortgagecalculator.client.MortgageCalc’
    Hint: Check the inheritance chain from your module; it may not be inheriting a required module or a module may not be adding its source path entries properly


After going through those steps, I could use the JSONParser and other JSON objects in my widget. Here’s the balance of the RateResponseHandler:

if (keepGoing && jso != null) {
int selected = MortgageCalc.this.rateChoiceBox.getSelectedIndex();
String [] keys = jso.getKeys();
if (keys.length == 2) { // expected
JSONValue jsvrate =
JSONString jsstr = null;
if ((jsstr = jsvrate.isString()) != null) {
String rate = "";
if ((rate = jsstr.toString()) !=
null) {


The JSON parsing code is rather verbose. I also wasn’t a fan of the return values from the is* methods; I was always told that any method prefixed with ‘is’ should return a boolean. But then I peeked into the JSONParser class and decided that I’d just use the free code
and stop complaining. Final comments:

  • The module inheritance system is a quite powerful, if slightly
    misnamed, method of reusing code. In fact, there is already at least one blog about GWT widgets.
  • Packaging up code for a module is tedious, and perfectly suited to an
    ant task.
  • HTTPRequest is entirely adequate for pulling down data.
  • JSON, combined with HTTPRequest, gives you some typing, without tying
    you into a custom servlet. If you want typing stricter or more granular
    than javascript
    and are using a java backend, then Google’s
    might be a better choice.

Step #2a Creating the Server side components

After choosing the means for transmitting the data, the next step is to build the server side code. For Expresso, that means creating a new controller and a new state. A controller is similar to a Struts action; one controller may have many states. Please be aware that if you want to play with the new client code, you can use almost any dynamic web aware language, or even a static HTML page, to return the appropriate JSON, which will look similar to this: {"fivearm":"5.00","thirtyfixed":"6.50"}.

Adding a new state to handle a request for mortgage interest data in Expresso is not too complicated, though it could definitely be easier. I needed to:

  1. Add the values to the database. Expresso has a setup table for generic configuration information that I’ll use.
  2. Create a class that extends DBController, and have this class access the database and create the JSON.
  3. Update the configuration file to map a URL to a class. This version of Expresso is based on Struts.
  4. Add a JSP for output.
  5. Update the security tables so that this state can be accessed by everyone.

Adding the values to the database is just an insert statement. For the thirty year fixed interest rate, the SQL looks like this: insert into SETUP values ('com.cohomefinder.CoHomeFinderSchema', 'MTG_30_Year_Fixed_Rate','30 Year Fixed Mortgage Interest Rate','6.50');. One of the benefits of using the setup table is that Expresso ships with an Administrative Web Interface which lets non technical users change ‘setup values’ with a browser.

The next step is to create a class to respond to our request. That class is the MortgageCalculator controller. This is a fairly simple class, which uses the JSON.simple Java library to create a correctly formatted JSON object. Note that the rates are sent back as Strings even though JSON can handle converting to numbers. The reason for this is that I wanted some kind of formatting control; if the class sent back ‘6.5’ for the thirty year fixed interest rate, I might want it formatted as ‘6.50’. Formatting is simpler on the server, where I have the entire Java API to use, including NumberFormatter.

After that, I needed to create an entry in the configuration file to map some URL to this class.

<action path=”/MtgCalc” type=”com.cohomefinder.controller.MortgageCalculator” name=”default” scope=”request” validate=”false”>
<forward name=”getRatesAsJSON” path=”/expresso/components/registration/jsp/register/mtgcalc.jsp”/>

As you can see, this is very similar to configuring a struts-config.xml entry.

The JSP was extremely simple. It imports the expresso tag libraries in the standard manner, and then write the property that the MortgageCalculator class puts in the response (on the last line of the Java file).

<expresso:IfElementExists name=”JSON” type=”Output”><bean:write property=”JSON”/></expresso:IfElementExists>

The last thing to do is update the controller table to allow anyone to access this class. Again, it’s a simple SQL statement: insert into controllersecurity values ('com.cohomefinder.controller.MortgageCalculator','Everybody','*');. If I wanted to, I could restrict this information to certain classes of users, but in this case, everyone should have access to it.

After these steps, I can hit http://localhost:8080/ and get back valid JSON containing values stored in the database.

The next step is to update the client to access the server data using HTTPRequest and to change the GUI accordingly.

Step #2: A Calculator which retrieves data from a Java server process

(Updated 6/14 with links to the two parts of this step)

The next step in building a real world mortgage calculator using GWT is to retrieve something simple from a back end. (See the problem’s introduction and Step #1, as well as the client code and the server code portions of the step described below.) It will be easiest to start with something simple. In this example, the client will pull two different mortgage interest rates (a 30 year fixed rate and a 5/25 ARM rate) from the Expresso back end. Since I am accessing a Java back end, there are two options.

  • Google’s services infrastructure. The benefit of this is fairly painless and transparent marshaling of Java value objects–that’s according to the documentation, I haven’t used it. The main downside is that I can’t use Expresso’s default servlet (with its authentication, logging, and caching) to handle the service request, because you must always extend Google’s RemoteServiceServlet.
  • Use the HTTPRequest
    class. This class essentially wraps the XMLHttpRequest object in a cross browser way, so if you’re familiar with that Javascript construct (which is at the heart of AJAX), the API shouldn’t be too shocking. This class limits the types of response available; no XML/DOM tree is passed back from a request, just text (in a String). The benefit of this method is that it’s very familiar to folks who’ve used XMLHttpRequest and is relatively simple. The main downside is that you’re limited to Strings as return values. There are, however, ways around that limit.

Based on the current requirements, it made more sense to use HTTPRequest than Google services. I could see using the Google services layer if I were doing some green field development in Java, or in a situation where such transparent marshaling saved a significant deal of work.

Of course, when you’re sending back text, you have a couple of options for encoding the data. I considered a custom encoding, but that’s not very scalable to large datasets, escaping and unescaping can be non trivial, and there are well known text transfer formats out there. Of those I know, XML and JSON are the primary ones. I went with JSON because Google kindly (almost) provides a JSON parsing library in their sample code. This parsing library is nice because it allows me to send back values as Strings, Arrays, Booleans, Numbers or Objectss and converts the JSON to the appropriate type. I say almost because I had to make a few changes to their code.

In my next post, I’ll look at the server side changes I made, including integrating the JSON.Simple Java library into Expresso.

Step #1: A Calculator with No Server Interaction

(Update 6/26: Here’s step #2, step #3 and the conclusion, with more source code.)

The first step is to build a simple javascript mortgage calculator, like many others out there, but using GWT. It seems that GWT is a great way to build complicated JavaScript UIs without much knowledge of JavaScript. (As an aside, I know this first solution is vastly overengineered, but eventually the problem set will require more of GWT.)

If you’re in a hurry, the GWT Mortgage Calculator is live and here is the source (for eclipse, for Windows; you’ll need to download the GWT and place the libraries in the lib subdirectory).

First, I found this article about calculating loan payments, which seemed straightforward enough. Then I downloaded the GWT and played around with some of the examples. I then created a project for eclipse, using the instructions in the GWT getting started guide.

A mortgage calculator is a pretty simple problem, really. Five labels and text boxes cover the needed inputs and outputs for interest rate, loan amount, term, number of payments and paymount amount. Also needed is a ‘calculate’ button and some way of conveying an error, should the user give invalid input.

I looked at using HorizontalPanel and VerticalPanel, but they looked horrible. I ended up using a FlexTable, which worked just fine.

I wanted to do some inline validation on the textboxs to make sure no one was entering anything other than numbers and a decimal point. Luckily, there is sample code for doing just this. This code worked just fine when I was running in my hosted environment, but when I deployed to a server and looked at the calculator in FireFox, I couldn’t type anything. The Google Web Toolkit group had the answer:

The problem is that GWT does not handle the the way that Mozilla/Firefox uses keyCode. For a key press event, evt.keyCode is only set for the non-character codes (Function keys, PageUp etc). For character keys the evt.charCode field is set. For key up/down events evt.charCode is never set, only evt.keyCode.

After disabiling this input validation, the calculator worked on FF.

I started out using a popup for error conditions, like non numeric user input. This examples is wrong–rather than setWidget() as the last line of the constructor, you need add(), otherwise you end up seeing an error like this:

[ERROR] Uncaught exception escaped
java.lang.RuntimeException: JavaScript method '@
threw an exception

(More information here.) I ended up using a dialog box instead, because I felt it looked better.

Now I had a working copy of the MortgageCalculator, and I just needed to integrate it into my client's website. For speed, we decided I'd just check the derivative javascript/html/xml into the site's source tree. Obviously, for the long term this is not a viable solution. A better way to do this would be to store the java files (and libraries, etc) in CVS and use some of the ant integration out there. Eventually we'll get there.

For ease of integration, I added a paragraph element to an existing JSP with a unique ID that the java class will reference and into which the java class will push the generated html (updated 6/13 to fix a broken url). This is just like the sample application.

In order to access the javascript when the including file and the generated javascript are not in the same directory, you need to not only modify the src attribute from <script language="javascript" xsrc="gwt.js" mce_src="gwt.js"></script> to <script language="javascript" xsrc="/absolute/path/to/gwt.js" mce_src="/absolute/path/to/gwt.js"></script> but also the meta tag. This tag changes from <meta name='gwt:module' content='com.cohomefinder.gwt.mortgagecalculator.MortgageCalc'> to <meta name='gwt:module' content='/absolute/path/to/com.cohomefinder.gwt.mortgagecalculator.MortgageCalc'>. This tag can actually be placed in the body rather than in the head.

I ran into another issue when integrating this code into the existing website. For some reason, in FireFox the error dialog box doesn't show up where it should, and instead is placed at the bottom of the screen. Here is the relevant section of code:

if (retVal != 0) {
ErrorDialog err = new ErrorDialog("v1 Please enter all data in n
umeric format.");
Widget slot1 = RootPanel.get("slot1");
int left = slot1.getAbsoluteLeft() + 10;
int top = slot1.getAbsoluteTop() + 10;
err.setPopupPosition(left, top);

and here's the error message from the Javascript Console:

Error: Error in parsing value for property 'left'.  Declaration dropped.
Source File: http://localhost:8080/Colorado-mortgage.htm Line: 0
Error: Error in parsing value for property 'top'.  Declaration dropped.
Source File: http://localhost:8080/Colorado-mortgage.htm Line: 0

This seemed to occur no matter which widget I was trying to get the absolute position of--the button, the enclosing table, or the enclosing paragraph element. This only occurred when integrating the calculator into a JSP page (not in the generic generated html), so it's likely it was a site specific bug. To work around it, I used a label rather than a dialog box.

Other random tidbits:

  • In general, I was less than impressed with the Google provided documentation, but very impressed with the GWT group.
  • It was very easy to quickly develop using Eclipse and the hosted web browser.
  • The HTML of the file you're developing to lives under the src directory (all the way down in the public folder).
  • All the javascript and html are compiled into the www directory.

Next, we're going to integrate with Expresso and pull some values from the database.