Skip to content

Useful Tools: piwik, a worthy web statistics package

I recently installed a open source web analytics tool called piwik.  (You can demo it at that site.) I found out about it via the sourceforge.net mailing list. It was the featured project for July 2009. It bills itself as an alternative to Google Analytics (GA) (actually, right now, the home page states “Piwik aims to be an open source alternative to Google Analytics.”) and I can see why it does so. The architecture is similar, JavaScript executing on every page and sending data to a server; the interface is similar as well, with lots of whizzy Web 2.0, JavaScript heavy features and detailed data.

I had to see been using the Wusage installation that came with my web hosting service. piwik was quite a step up from that, with richer graphs, results and UI. Plus, because it was JavaScript executing and I was assured that every visit was actual visit by an actual person. Since it’s hosted on my server, I control all the data, which was a sticking point for me considering using Google Analytics.

I recently upgraded to 0.4.2, which broke the dashboard, but I’ve been assured a fix is in SVN (Update Aug 4: They no longer plan to fix the bug, but there is a workaround in that thread.).  If you want to get the latest code, go here.  You can download 0.4.1, the last working version I know of, here. I’ll update this to point to the piwik website when they have a release up that works. For some reason they don’t have a release archive that I could find.

So what’s good about piwik?  Well compared to what, Google analytics, or other website analytics tools? This is a fundamental question, because if you are using GA just for the web stats piece, or are using some other static logfile analysis tool, piwik is well worth reviewing.

In comparison to Google Analytics

The downside is

  • you have to maintain another server/database, etc.  I imagine that someone will offer piwik via SAAS sometime soon, though I couldn’t find anyone doing that right now.
  • it’s a beta product and is not as mature as Google Analytics, as evidenced by the 0.4.2 issue above
  • some key GA features are missing (goals, funnels, etc).

In comparison to the other website analytics tools I’ve used, AWstats (which I’ve written about before and is open source) and wusage (not open source, but free with my hosting contract), piwik has

  • a slick user interface
  • JavaScript execution, so you know you’re getting a real browser instead of a bot (the javascript browser guarantee)
  • click outs easier to track
  • easier configuration
  • javascript widgets available

The downside is:

This is obviously not intended to be a full, detailed analysis of all the differences between these tools, but I think that piwik has a lot of promise.  They have a roadmap full of planned features but they definitely aren’t yet an alternative to Google Analytics for anyone who uses some of the more advanced features of that product. Funnels, the click overlay or goals, are all unsupported in piwik as of this version. In the forums, I saw several requests for such richer analysis tools, and in the roadmap I saw a goal tracking plugin as a blocker for version 1.0, so the team is aware of the lack.

When browsing around doing research for this post, I saw a post (sorry, couldn’t find it again) about how piwik features would be developed for smaller websites because it’s an open-source alternative, but I believe that the support of openX (an ad server company that I wrote about in the past), who is funding at least one of the developers, will prevent such feature capture.  In addition, I find that open source projects that have an existing project to model themselves on (like GA), tend to try to reach feature parity.  If piwik continues on its current valid path of replicating Google Analytics features, then I think it will live up to its aim.

If you’re simply using Google Analytics to see who referred traffic to your sites, or for which keywords search engines are showing your site, and you want something more open or more control of your data, piwik is a good fit.  If you use any other web stats tool, and want a slicker admin interface or the javascript browser guarantee, piwik is also worth a look.

Update, 7/31: A friend pointed out this broad survey of the current state of free (as in beer) web analytics options
[tags]piwik,the javascript browser guarantee,google analytics, piwik vs google analytics, web stats[/tags]

Dynamic Languages Useful Tools Web Applications | moore |

Solix website launch

I was able to help launch a revamped website on Monday.  Solix Biofuels is an algal oil producer moving from R&D into production. I’m not really a biology person, but I did enjoy reading about their technology.  To my layperson’s eye, algal energy sources seem much more sustainable than crop fuel sources.

It’s been a while since I’ve been the prime mover behind a deployment–there’s always a bit of nail biting when you finally reveal work to the world–but this one was fairly smooth.

[tags]solix,site launch, algal energy, oilgae[/tags]

Technology Technology and Society Web Applications | moore |

GWT and complex javascript overlays

I remember hearing about javascript overlays and the Google Web Toolkit at the 2008 Google I/O conference, so when I had a chance to use them for a complicated JSON dataset, I thought it would be a good fit.  I use JSON in a number of widgets around a client’s site, mostly because it is the easiest way to have a client be cross site compatible (using JSONP for server communication), the data ranges in size from 13K to 87K.

But as I looked for examples, I didn’t see a whole lot.  There is, of course, the canonical blog post about javascript overlays, but other than that, there’s not much out there.  I searched the gwt google group, but didn’t see anything about complex javascript overlays.

Perhaps I should define what I mean by complex javascript overlays.  I mean JSON objects that have embedded objects and arrays; something like this: var jsonData = [{"id":52,"display":{"desc":"TV","price":125},"store":[1,2,3,4,5]}, {"id":2,"display":{"desc":"VCR","price":25},"store":[1,2,5]}];

The key is to realize that just as you can return a string from a native method, like the canonical blog post does: public final native String getFirstName() /*-{ return this.FirstName; }-*/;, you can also return any other javascript object, including arrays (helpfully available as the JsArray class).  So, for the above JSON example, I have a Display class that looks like this:

package com.mooreds.complexoverlay.client;
import com.google.gwt.core.client.JavaScriptObject;

public class Display extends JavaScriptObject {
protected Display() {}
public final native String getDesc()/*-{ return this.desc;}-*/;
public final native int getPrice()/*-{ return this.price;}-*/;
}

Code for the array ("store") looks like this:

package com.mooreds.complexoverlay.client;

import com.google.gwt.core.client.JsArrayInteger;
public class StoreList extends JsArrayInteger{
protected StoreList() {}

}

Running code that shows you how to parse and display the entire json string is here (using GWT 1.6.4) is available in this tarball.  I haven't yet tried it, but I imagine that it would be pretty easy a JSONP call work in a similar manner to the HTML embedded JSON in this example.

Update June 29: Note that if you want to use any overlay types in collections, that JavaScriptObject, which you must inherit from, has final hashCode and equals methods.  The hashCode method "uses a monotonically increasing counter to assign a hash code to the underlying JavaScript object" and the equals method "returns true if the objects are JavaScript identical (triple-equals)".  Both of these may cause issues with collections such as maps.
[tags]gwt, javascript overlays, google web toolkit, is anyone still using technorati?[/tags]

GWT Java Programming | moore |

Sending (and receiving) more than a file with Flex FileReference

The Flex FileReference object makes it very easy to send a file to the server.  I had a situation where I wanted to send some additional data and also get back server output.  This is possible, but not entirely intuitive, so I wanted to document this for others.  In the process of making this work, this post and this post were very helpful to me.

To send more than a file, you use a URLVariables object, like you normally would.  The key is to realize that you also have to set the URLRequest.method to URLRequestMethod.POST, otherwise these variables get lost.  (Makes sense–no one sends files via GET, but it was not obvious to me.)

var request:URLRequest = new URLRequest(url);
var variables:URLVariables = new URLVariables();
variables.docname = docName.text;
request.data = variables;
request.method = URLRequestMethod.POST;
try {
fileRef.upload(request);
} catch (error:Error) {
trace("Unable to upload file.");
}

To get any response from the server (like a success message or filename), you have to attach a listener to the DataEvent.UPLOAD_COMPLETE_DATA event, like so:

fileRef.addEventListener(DataEvent.UPLOAD_COMPLETE_DATA, onUploadCompleteData);

...
private function onUploadCompleteData (event : DataEvent) : void {
var myData = new String(event.data);
// do something with your serverside data...
}

[tags]flex,file upload[/tags]

Programming RIA | moore |

GWT encryption options

I was looking for an encryption package for GWT.  A client had some mildly private information that they wanted to encrypt in transmission.  Now, of course, this is ultimately futile.  Anyone who wants to get at this information can, because we send the source code (however obfuscated) that decrypts the information to the client.  To borrow Corey Doctorow’s words, the attacker is also the recipient.  But, sometimes just making getting the information inconvenient is good enough.

I looked at a couple of encryption options.  There’s are a couple of nice javascript libraries that do encryption: Gibberish-AES and javascript.crypto.library, but they hav no GWT hooks (and javascript.crypto.library is released under the AGPL which has some unclear legal ramifications).

However, there is a project from about two years ago that does Triple DES encryption and is written in pure GWT.  It’s even called gwt-crypto.  Unfortunately, it hasn’t been maintained recently.  I was able to download the files, apply a fix (for issue #1) and move some files around in such a way that it works.

Here’s how you use it:

on the server:
TripleDesCipher cipher = new TripleDesCipher();
cipher.setKey(Constants.GWT_DES_KEY);
try {
enc = cipher.encrypt(String.valueOf(value));
} catch (DataLengthException e1) {
e1.printStackTrace();
} catch (IllegalStateException e1) {
e1.printStackTrace();
} catch (InvalidCipherTextException e1) {
e1.printStackTrace();
}

On the client, make sure you inherit the module:
<inherits name='com.googlecode.gwt.crypto.Crypto'/>
Then:

TripleDesCipher cipher = new TripleDesCipher();
cipher.setKey(Constants.GWT_DES_KEY);
String dec ="";
try {
dec = cipher.decrypt(enc);
} catch (DataLengthException e) {
e.printStackTrace();
} catch (IllegalStateException e) {
e.printStackTrace();
} catch (InvalidCipherTextException e) {
e.printStackTrace();
}

I just use a constant DES key, which is not all that secure.  I’m sure you could do something more secure like hashing the request time, filename and some other secret key, but you need to make sure that both the server and the client agree on the key, otherwise you’ll not be able to decrypt the info.

Update 6/13: I got permission from the project owner to update the google project, so I’ve done that.  You can download the new gwt-crypto jar there.

The modified gwt-crypto jar file is here.  I’m hoping the administrator will let me at least check in the changes I’ve made so that it works on GWT 1.5.3 (can’t speak for GWT 1.6).

[tags]gwt,tripledes encryption,hiding in plain sight[/tags]

GWT Technology and Society | moore |

Best Consulting Sales Pitch Ever

I heard this at a presentation about Mule (an enterprise service bus) at BJUG tonight.  The presenter, Rich Remington, at the end of the talk, put up a slide detailing a contest.  Anyone who emailed him with a use case that Mule might be able to help with won a chance at a free lunch and 1-2 hours of free consulting about the use case. You have to email him within 48 hours of the talk.
What a great idea!  Not only does he get to pick a use case that Mule can help with, he also gets a chance to pitch his services, and show his value to the client.  And he gets a list of possible clients–people who know about something about Mule and think they might have a problem Mule can solve.

The contest winner gets more than a free lunch out of it too–the chance to pick an expert’s brain for free for a couple of hours can be worth quite a bit.

Win-win, for sure.  What a great idea for anyone consulting in a specialized field!

[tags]mule,rsi[/tags]

BJUG Business Consulting | moore |

Body Shop Basics

Here is a great writeup on the basics of working for a consulting company (“body shops”).  These companies hire talent and then farm them out to other companies who have needs–either cost control needs or specific skill sets, or both.  This article talks you through some of the hoops around this process, what you can expect, and how to choose a consulting company to work for.
Scott also gives good reasons why people might want to be consultants, rather than employees:

A consultant is more free than a regular employee to circulate within his professional community and to take more jobs in more challenging environments. He can also get more relevant training. There’s no room in the consulting industry for wasting time — training must be good, and it must be for a good reason.

Some of the advice is a bit out of date (I doubt that “web development” would be considered a “new skill” anymore), but Computer Aid is apparently still around and the entire article is generally sound.  I’ve worked for a consulting company or two in my professional life, both as an salaried employee and as a hourly contractor.  I wish I’d read his advice then.

[tags]ask the headhunter[/tags]

Consulting | moore |

CleanPrint Installed on my blog

About six months ago, I met with some friends who I’d worked with in the past, at Format Dynamics.  They gave me a bit of software to install on my blog, and I installed it and forgot about it. But it is worth mentioning.
It is a wordpress plugin that interfaces with CleanPrint, which is a pretty cool piece of software (I’ve written about it in the past.)  Installation is as easy as typical WP plugin installation.  You can try it out by going to any individual wordpress page and looking for the ‘Print Blog’ button:

I printed my notes from the BDNT meetup, just as an example; you can download the PDF here: June 2009 New Tech Meetup Notes.pdf.  (PDF created using the excellent PDF995 adware.)

The other option for pretty printing, of course, is to create your own print.css.  I believe that the plan is to eventually be able to display ads on the printouts, with some revenue sharing agreement.

[tags]wordpress plugins,pretty printing,shout out[/tags]

Technology Web Applications | moore |

Flex textareas, newlines and posting to a unix server

I was working on a flex app and wanted to post the contents of a flex textarea to a php script on a unix server, among other fields.  This is easy to do with the URLRequest.  The php script on the server then parsed the form fields.  In particular, it split the textarea’s contents on the newline.

When I first tried this, the php script only found one record, no matter how big the textarea content.  After haivng the php script write the content to a file, it was clear why.  The flex app was sending the textarea with Windows line endings.  I’ve never experienced this particular issue before.  I tested a plain old HTML form, and somewhere the newlines are converted from Windows to unix (for more than you ever wanted to know about the humble newline, consider wikipedia.)

Regardless, the answer was to make sure that wordwrap was set to false, and run a regular expression on the content of the textarea before sending it.  From the comments on this blog post:

loremTEXT2 = loremTEXT.replace(/[\r\n]+/g, “\n”);

[tags]the humble newline, flex, textarea[/tags]

RIA Tips | moore |