Skip to content

imap proxy and horde

I’m implementing an intranet using the Horde suite of tools. This is written in PHP, and provides an amazing amount of out of the box, easily configured functionality. The most robust pieces are the mail client (incidentally, used by WestHost for webmail, and very scalable), the calendar, and the address book. There are a ton of other projects using the Horde framework, but most of them are in beta, and haven’t been officially released. Apparently these applications are pretty solid (at least, that’s what I get from reading the mail list) but I wanted to shy away from unreleased code. I am, however, anxiously awaiting the day that the new version is ready; as you can see from the demo site that it’s pretty sharp.

Anyway, I was very happy with the Horde framework. The only issue I had was that the mail application was very slow. I was connecting to a remote imap server, and PHP has no way to cache imap connections. Also, for some reason, the mail application reconnects to the imap server every time. However, someone on that same thread suggested using UP IMAP Proxy. This very slick C program was simple to compile and install on a BSD box, and sped up the connections noticeably. For instance, the authentication to the imap server (the only part of the application that I instrumented) went from 10 milliseconds to 1. It apparently caches the user name and password (as an MD5 hash) and only communicates with the imap server when it doesn’t have the information needed (for example, when you first visit, or when you’re requesting messages from your inbox). It does have some security concerns (look here and search for P_NEWLOG), but you can handle these at the network level. All in all, I’m very impressed with UP IMAP Proxy.

And, for that matter, I’m happy with Horde. I ended up having to write a small horde module, and while the framework doesn’t give you some things that I’m used to in the java world (no database pooling, no MVC pattern) it does give you a lot of other stuff (an object architecture to follow, single sign-on, logging). And I’m not aware of any framework in the java world that comes with so many applications ready to roll out. It’s all LGPL and, as I implied above, the released modules have a very coherent structure that makes it easy to add and subtract needed functionality.

Bravo Horde developers! Bravo imap proxy maintainer!

Technology | moore |

mod_alias to the rescue

Have you ever wanted to push all the traffic from one host to another? If you’re using apache, it’s easy. I needed to have all traffic from a http://www.foo.com site go to https://secure.foo.com. Originally, I was thinking of having a meta header redirect on the index.html page, and creating a custom 404 page that would also do a redirect.

Luckily, some folks corrected me, and showed me an easier way. mod_alias (ver 2.0) can do this easily, and as far as I can tell, transparently. I just put this line in the virtual server section for www.foo.com:

Redirect permanent / https://secure.foo.com/

Now, every request for any file from www.foo.com gets routed to secure.foo.com. And since they share the same docroot, this is exactly what I wanted.

To do this, make sure you have mod_alias available. It should be either compiled in (you can tell with httpd -l) or a shared library (on unix, usually called mod_alias.so). You have to make sure to load the shared library; see LoadModule and AddModule for more information.

Technology | moore |

My most popular posting

I don’t know why, but my post on yahoo mail problems is my most popular post thus far. I suspect it got picked up in google, or some other search engine, and is now serving as a place for folks to gripe about the free Yahoo! mail service. (Incidentally, I’m the second “Dan Moore” in google now! Meri has some interesting things to say about this intersection between the internet and real life.) This is interesting (and a bit amusing) to me for several reasons:

For one, there’s no helpful content on that posting for these folks problems. In fact, I don’t even use the free service from Yahoo (I pay extra for storage). And the posting concerns the short term problems a client of mine had with the new Yahoo mail interface, and how outsourcing exposes you to those types of risks. The comments are not germane to the posting.

Or should I say that the posting is not germane to the comments? As is ever the case on internet forums, this posting has been hijacked by people who want to complain and share possible fixes to a very real problem–they can’t get to their email (I’m cranky when I can’t to my email, after all). I don’t begrudge them the use of my site; this just reinforces what Clay Shirky wrote about social software–people will twist software until it does what they need it to do, and fighting that is a lost cause (and has been for 20 years).

And that’s not just true for software, but for technology in general. After all, I doubt anyone working on radar thought it would someday be used for re-heating leftovers, and I’m sure that Daguerre (the inventor of photography) would be shocked at some of the pictures I’ve taken at house parties.

Technology and Society | moore |

IM Everywhere

The last two companies I worked at used instant messaging (IM) extensively in their corporate environment. They sent meeting notifications over IM, they used IM to indicate their availability for interactions, and they used it for the quick questions that IM is so good at handling (“hey John, can you bounce the server?”). IM has no spam, and is good at generating immediate responses.

I’m a latecomer to IM. I’ve used talk in college, but in a work environment, email was usually enough. And, I have to confess, when I’m programming, it’s not easy for me to task switch. And IM demands that, in the same way that a phone call does. How many times have you been deep in a conversation with someone, only to have their phone ring? You know what happens next: they say “can I get that?” and reach for the phone. Whatever flow and connection you had is disrupted.

Now, obviously you can configure IM to be less intrusive than a phone call, and the first thing I did was switch off all sound notifications in my yahoo IM client. However, the entire point of IM is to disrupt what you’re doing–whether it’s by playing a sound or blinking or popping up a window, the attraction of IM is that it is immediate.

I’ve found that ninety percent of people would rather talk to a person than look something up for themselves. (I am one of those ninety percent.) There are a number of reasons. It’s easier to ask unstructured questions. People are more responsive, and can come up with answers that you wouldn’t think to find on your own. And it’s just plain reassuring to find out what someone else thinks–you can have a mini discussion about the issue. This last is especially important if you aren’t even sure what you’re trying to find.

IM is a great help for this kind of ad-hoc discussion. However, it’s another distraction at work. The real question is, do we need more distractions at work? Jakob Nielsen doesn’t think so (see number 6) and I agree.

However, IM is becoming ingrained in these corporations, and I don’t see anything standing in the way of further adoption. The original impetus to write this essay was the astonishment I felt at these two facts:

1. the widespread corporate use of IM

2. the paucity of corporate level control over IM

In all the time I was working at these companies, I saw many many IMs sent. But I only heard one mention of setting up a corporate IM server (someone mentioned that one of the infrastructure projects, long postponed, was to set up a jabber server). Now, I don’t pretend that any corporate secrets were being exchanged, at least no more than are sent every day via unencrypted email. But every corporation of a decent size has control over its email infrastructure. I was astonished that no similar move had taken place yet for IM. Perhaps because IM is a young technology, perhaps because it is being rolled out from the bottom up, perhaps because it’s not (always) a permanent medium.

For whatever reason, I think that you’re going to see more and more IM servers (even MS has an offering) being deployed as businesses (well, IT departments) realize that IM is being heavily used and is not being monitored at all. Perhaps this is analogous to the explosion of departments static HTML intranets that happened in the late 1990s, which only came to an end when the IT department realized what was happening, and moved to standardize what became an important business information resource.

Technology | moore |

Jalopy

I like javadoc. Heck, I like documentation. But I hate adding javadoc to my code. It’s tedious, and I can never remember all the tags. I don’t use an IDE so the formatting gets to me.

After attending a presentation at BJUG about software tools, I investigated jalopy and I liked what I found. Now, jalopy is more than just a javadoc comment inserter, but javadoc insertion was my primary use of the tool. It may be piss poor for code formatting and whatnot, but it was pretty good at inserting javadoc. I was using the ant plug-in and the instructions were simple and straight forward. It didn’t blow away any existing comments, and it didn’t munge any files, once I configured it correctly. And there are, make no mistake, lots of configuration options.

Jalopy has a slick Swing interface to set all these configuration options, and you can export your configuration to an XML file which can be referenced by others. This, along with the ant integration, make it a good choice for making sure that all code checked in by a team has similar code formatting.

However, I do have a few minor quibbles with this tool.

1. The default configuration of javadoc is busted. When you run it, it javadocs methods and classes just fine, but any fields are marked with “DOCUMENT ME!” when they should be commented out: “/** DOCUMENT ME! */”. This means that, with the default configuration, you can’t even run the formatter twice, since jalopy itself chokes on the uncommented “DOCUMENT ME!”.

2. The configuration file is not documented anywhere that I could find. I looked long and hard on the Internet, and only found one example of a jalopy configuration file here. And this is apparently just the default options exported to a file. I’ve put up a sample configuration file here which fixes problem #1. (This configuration is only for javadoc; it accepts all other defaults.)

3. The zip file that you download isn’t in its own directory. This means that when you unassumingly unzip it, it spews all over your current directory.

None of these are show stoppers, that’s for sure. If you’re looking for a free, open source java code formatting tool, jalopy is worth a close look.

Java Programming | moore |

Is the American tech worker obsolete?

I’ve been doing some thinking about offshoring recently. For those of you not in the IT industry, ‘offshoring’ is jargon for outsourcing of white collar jobs to developing countries like India and China. There’s been a lot of talk about this phenomenon: Salon (1 and 2), The Economist [sorry, it’s a pay article], InfoWorld, and CNET News.com have all commented recently. In addition to reading these articles and others like them, I’ve also talked to folks in the software industry, as well as friends who work in aerospace and packaged good manufacturing industries. And I’ve come to the conclusion that a certain amount of offshoring is inevitable, since the labor cost differential is so great.

However, like any other business fad (or any fad, for that matter), there are external costs that haven’t simply are not fully understood. Among these are

1. Loss of customers
The constant mantra of folks who are trying to tell IT workers how to adjust to losing their jobs is ‘retrain, retrain’. But retrain for what? What job isn’t offshorable? Nursing is the only one that comes to mind. If allvirtualizable (i.e. can be done without face time) service jobs where there is a price differential get pushed overseas, that’s millions of jobs lost. The American economy has been the engine for most of the world’s growth over the last 10 years, sucking up exports from other countries to the tune of billions of dollars. What happens when the consumers of the USA don’t spend money, either because they are out of a job, or afraid of losing their job soon? It takes a visionary like Henry Ford to realize that if you grow the amount of folks who can buy your products, you grow your business.

2. Loss of future business leaders
If you export big chunks of your IT department, perhaps keeping only senior folks who can help manage external projects, then you win in the short run. However, eventually those folks will retire (ask NASA). And if you haven’t brought any entry level talent in, where will you go to replace these needed folks?

3. Difficulty of managing far flung teams from different cultures
This is difficult in two ways. One is the logistical aspect. If you want a teleconference, you have to adjust either your schedule or theirs. And if the country is on the opposite side of the globe, possibly both folks need to be in the office at an awkward time. The second difficulty is related to quality. Just as the Japanese products in the 50s and 60s were thought cheap and low quality, some of the work done overseas today is a lower standard than expected. This is true of all software of course, but it’s harder to control the quality when you write a spec and throw it over the wall.

In short, offshoring is in its infancy. More of it is coming down the line, but, as the hidden costs are discovered, the benefits of local teams will come back into focus. I can do much of my consulting work from my house, yet I still find that most companies want me to come into their office. Why? To some extent it’s control (should be careful not to blog my way out of a job here), but it also is because communication, in each direction, is easier with someone who’s on site. There’s extra effort expended in any kind of virtual communication. And by meeting me, the customer and I build a relationship, which is perhaps more important for trust in business than anything else.

I also want to address Michael Yuan’s comments:

“3. Coding is a dying profession in the long run. If the jobs have not been outsourced to developing countries, the new generation of model-focused automatic code generation tools will eliminate the need for basic coders anyway. The jobs that have a future are system designing and architecting (the real engineering jobs). I think the ability to design end-to-end systems using whatever tools available is an important skill for the future.”

I’ve commented enough, I hope on, the idea that all coding work will or should be outsourced to developing countries. But I think that the idea that ‘the need for basic coders’ will be eliminated due to improvements in ‘automatic code generation tools’ is foolish for a number of reasons.

1. If you don’t understand basic coding on a system, when the automatic tool doesn’t do what you need it to, the means you’re screwed. Basic coding is not something you can pick up at school–you need to be out in the real world, working on crufty systems that are too expensive to change and older than dirt, to really appreciate what automatic tools can and cannot do, and what systems can and cannot do. In addition, easier code generation means more code out there, not less. And, easier code generation means more code out there. And, the easier the code is to generate, a la Visual Basic, the more likely the person doing the generation won’t get it right, or won’t document it, or won’t understand why there’s certain behavior exhibited. In this case, someone who has a deep, visceral understanding of the system and language will be called in.

2. Yuan talks about architecture and system design being the ‘real engineering jobs.’ I’ll agree that those are more stimulating and challenging than coding, and also have a higher value add. But again, how many folks are system architects right out of school? I certainly wouldn’t want to work on a project that had been designed by someone with exactly 0 years of real world experience. If you abolish entry level positions, you eat your seed corn, as I mention in my comments about offshoring above.

3. There is no silver bullet. The hard part isn’t writing the software, it’s determining what software needs to be written.

4. Someone’s got to write the tools. I remember an old science fiction story, by Isaac Asimov I think, about a world of taped learning. You took a test at 18 that determined what your skills were, and then you listened to a set of tapes that taught you all you needed to know. The protagonist was distraught because the test just didn’t seem to fit him–he was outcast and ridiculed for not knowing his profession. Of course it turned out that the folks that couldn’t be tape educated were exactly the folks who were able to write the tapes. Someone’s got to write the tools. (I don’t pretend, however, that anywhere near the same number of folks are needed to write tools as to use them.)

Michael and others are certainly correct that things are going to change in the software world. It’s not going to be possible to just know a language–you have to be involved in the business and maintain relationships. But change is nothing new for IT workers, right?

In short, it appears that Neal Stephenson was correct. I leave you with an excerpt from Snow Crash:

“When it gets down to it–talking trade balances here–once we’ve brain-drained all our technology into other countries, once things have evened out, they’re making cars in Bolivia and microwave ovens in Tadzhikistan and selling them here–once our edge in natural resources has been made irrelevant by giant Hong Kong ships and dirigibles that can ship North Dakota all the way to New Zealand for a nickel–once the Invisible Hand has take all those historical inequities and smeared them out into a broad global layer of what a Pakistani brickmaker would consider to be prosperity–y’know what? There’s only four things we [America] do better than anyone else

music
movies
microcode (software)
high-speed pizza delivery”

And none of the four above are guaranteed (except perhaps the high-speed pizza delivery).

Happy New Year.

Technology and Society | moore |

Java Tidbits

Quick, take this true/false quiz and test your Java(tm) knowledge:

1. Private members can only be accessed by the object which owns them.

2. The contents of a finally block are guaranteed to run.

Everybody knows that both of these are true, right?

Actually each of these is false. In the first case, this program shows that objects can twiddle with private members of other objects of the same class:

class PrivateMember {
   private int i; 
   public PrivateMember() { 
      i = 2;
   }
   public void printI() {
      System.out.println("i is: "+i);
   }
   public void messWithI(PrivateMember t) {
      t.i *= 2;
   }
   public static void main (String args[]) {
      PrivateMember sub = new PrivateMember();
      PrivateMember obj = new PrivateMember();
      obj.printI();
      sub.messWithI(obj);
      obj.printI();
   }
}

and this program shows that finally blocks don’t run if the JVM exits:

class TryCatch {
   public static void main (String args[]) {
      try {
         System.out.println("in try");
         throw new Exception();
      } catch (Exception e) {
         System.out.println("in catch");
         System.exit(0);
      } finally {
         System.out.println("in finally");
      }
   }
}
Java | moore |

Book Review: Sixth Column

If you’re looking for an introduction to Robert A. Heinlein’s vast corpus of fantastic science fiction, don’t read Sixth Column, read The Moon is a Harsh Mistress. If you’re a Heinlein junkie who’s read all his other stuff and you want a quick, fun read covering the typical Heinlein themes (the able man, war, gee whiz technology, “long live democracy”), then you’ll definitely want to pick up this book.

The basic premise is: the USA has been invaded by “PanAsians,” and the government effectively destroyed. Having subjugated India, the “PanAsians” know how to tie down the USA–lots of labor camps, citizen registration and public executions as punishment for any rebellion. But they also have learned not to interfere with their subjects’ religion(s). One small military base, a research laboratory, has escaped destruction; luckily the plucky soldiers have an able commander and lots of technology the invaders simply can’t match.

From there, it’s just a matter of time. The reader gets to watch how these men build a movement, screw with the “PanAsian” leadership, and eventually free the USA. Of course the technology is hokey and the dialog can be a bit offensive, but it’s realistic (yeah, I think slurs are allowable if they’re marching your family off to the labor camps). This book was written in 1949 and reflects some of the paranoia that Heinlein later gave voice to in Expanded Universe.

But, it’s a fun, quick read and if you like Heinlein, you’ll probably like it. It’s no classic, but not every book can be.

Books | moore |

J2ME development advice

Here is some advice for any of you thinking about doing any development for J2ME. I’ve written before on J2ME considerations, but the project I’m working on is a bit further on now, and I thought I’d share some hard earned knowledge.

1. Get Windows.

Make sure you have access to a windows box, running a modern version of windows. While Sun’s Wireless Tool Kit supports Linux and Solaris, and IBM’s WebSphere Device Developer supports Linux, no other emulators do (nor do they work under wine).

Whether you want to support Motorola, Nokia, or NEC, you are pretty much need windows to run the emulator. And an emulator is crucial, because it allows you to rapidly develop an application. And testing on as many emulators as possible means that your application will be as tight as possible. However, when you get something that is close to finished, you’ll need a real phone to test it on.

2. Get (a) real phone(s).

While emulators can tell you a lot, they certainly can’t assure you that an application will run on a real phone. One project I worked on had an emulator that ran the app just fine, but the app was locking up on the real phone. It turned out that the networking code needed to run in a separate thread. There are many things that differ between an emulator and a real phone. Installation of a midlet is different (and differs between phones as well). Instead of accessing a file on disk, you have to use OTA provisioning (sure you can emulate that with the WTK, but that’s just an emulation. I’ve run into issues with DNS that just don’t show up on the emulator). Also, as mentioned above, the networking capability differs, and the connection is much slower. The amount of memory that you can use while developing on the desktop is effectively unlimited (some of the emulators let you monitor the amount used, but I don’t know of any that lets you limit it), but phones have hard limits. Don’t you want to know what happens when you try to use 101KB of memory on a device that only has 100KB? The limitations you face on user interface are also more real on a phone, when you can’t use the keyboard to enter your username or the backspace key to fix errors. For all these reasons, you should get a phone as soon as you can.

3. Explore existing resources.

A couple of good books: Enterprise J2ME is a great survey book, with some very good ideas for building business applications (with large numbers of users) but not a whole lot of nuts and bolts. Wireless Java: Developing with J2ME, Second Edition is a good nuts and bolts book (it explains how to do your own Canvas manipulations, for example). Check out what else Amazon suggests for other ideas.

A couple of helpful urls: Fred Grott’s weblog, MicroJava, the EnterpriseJ2ME site, Sun’s site, of course, and the javaranch saloon is pretty helpful too.

The various carrier websites are useful, if only to find out what kind of phones you want to target: AT&T, Sprint, T-Mobile, Nextel. (Verizon in the USA is BREW only.)

4. Have fun.

Java Mobile Technology | moore |

Comments on “Mobility is more than J2ME (and the job martket for 2004)”, pt I

Michael Yuan had a pretty inflammatory post recently. Here are the first wave of my comments on this interesting topic.

“1. The move to mobility is inevitable in the enterprise. The IT revolution has to reach hundreds of millions of mobile workers in order to realize its promise. There is no other way. However, the real question is how and when this will happen. With the IT over-investment in the last decade, this might take several more years.”

Agreed. Allowing mobile users to access the corporate datacenter is an inevitability. When it does happen, however, it certainly won’t have the sexiness or big bang of the internet revolution. In fact, it’s much more an evolution than a revolution. Folks already have access to corporate databases right now; the mobile revolution simply combines the portability of paper with the real time nature of laptops. However, letting knowledge workers such as sales people and truckers have real time information on such a cheap, reliable device really will change the nature of the business. But we won’t see sock rabbits or dot.com millions, since such changes will favor existing businesses.

“2. When enterprises move to mobility, a key consideration is to preserve existing investment. Fancy flashy J2ME games will not do it. The task is often to develop specialized gateway servers and J2ME integration software to incorporate smart mobile frontends into the system. That requires the developers to have deep understanding of both J2EE and J2ME. I think that the “end-to-end” sector is where the real opportunities are in the next several years. That is also what “Enterprise J2ME” is all about. :)”

Now, don’t be so quick to judge. Gamers pushed the boundaries of the PC in terms of computing power, and I wouldn’t be surprised to see the same thing happen on the MIDP platform. That said, I’m not a gamer. However, I still have issues with the idea of folks paying to play a game on a cell phone. I play snake, but that’s a simple, free game, and I’m certainly not dedicated to it. Fred Grott claims that MMORPGs are going to drive J2ME game development–I just don’t see folks doing that when you can get a much, much richer experience from the XBox or Game Cube or PC sitting at home.

And I don’t see why Michael ties J2ME and J2EE so tightly. The whole point of web services is to decouple the server and the client. I dont’ see any reason why you couldn’t have J2ME talk to a .NET server, or a BREW client talk to a J2EE server. To me, the larger issue with the mobile revolution is the architecture of the J2ME applications, since I think that such small, non networked, memory constrained applications (with either extremely limited portability or extremely limited user interfaces, take your pick) are going to be a world apart from the standard java developer’s experience (which is HTML generation, not swing).

I’m going to leave his third point for another post, as the outsourcing issue is…worthy of a separate discussion.

Mobile Technology Technology | moore |