Skip to content

“Choose the right tool for the job!”

When you’re writing a program to perform some business function, there are usually many different options. Whether it’s the particular language, the database, the platform, or the hardware, you have to make some decisions. Like a carpenter, who chooses screws when he needs to attach two planks and a saw when he needs to shorten a dowel, programmers are supposed to choose the correct tool for the task. However, since programming is so new, changes so much, and is so abstract, it’s a bit more complex than that.

There are many things that affect the right tool, and some of the considerations aren’t directly technical:

Strategic change is one criteria. When I was working at a consultancy in 2000, there was a grand switch in language choice. perl was out, java was in. This decision was not made at a technical level, but rather at a strategic one. No matter that we’d have to retrain folks, and throw away a significant portion of our code base. No matter that for the sites we were doing, perl and java were a wash, except for the times that java was overkill. What did matter is that the future was seen to belong to java, and management didn’t want to be left behind.

Cost of the solution is another important factor. TCO is a buzzword today, but it’s true that you need to look at more than the initial cost of any piece of technology to get an idea of the true price. Linux has an initial cost of $0, but the TCO certainly isn’t. There’s the cost of maintaining it, the cost of paying for administrators, the upgrade cost, the security patch cost, the retraining cost, and the lock in cost. Windows is the same way–and though it’s hard to put a number on it, it’s clear that the future cost of a windows server is not going to be minimal, as you’ll eventually be forced to upgrade or provide support yourself.

The type of problem is another reason to preference one technology over the other. Slashdot is a database backed website. They needed speed (because of the vast number of hits they receive daily), but they didn’t need transactions. Hence, mysql was a perfect datastore, because it didn’t (at the time) support transactions, but was very fast.

The skill sets of folks available for implementation also should affect the choice. I recently worked at a company with a large number of perl applications that were integral to the company working for them. But they are slowly replacing all of them, because most of the folks working there don’t know perl. And it’s not just the skill set of the existing workers, but also the pool of available talent. I’ve heard great things about Lisp and how efficient Lisp programmers can be, but I’d never implement a business function in Lisp, because it’d be very hard to find someone else to maintain it.

The existing environment is a related influence. If everything in your organization is Windows, then a unix solution, no matter how elegant it may be to one particular problem, is going to be a poor choice. If all your previous applications were written in perl, your first java application is probably going to use perlish data structures and program flow, and is probably going to be a poor java program. I know my first server side java fell into this pit.

Time is also a factor, in a couple of different senses. How quickly are you trying to churn this code out? Do you have time to do some research into existing solutions and best practices, or to build a prototype and then throw it away? If not, then you should probably use a tool/solution that you’re familiar with, even if it’s not the best solution. Some tools add to productivity and some languages are made for quick prototyping (perl!). How long will the code be around? The answer to that is almost always ‘longer than you think,’ although in some of the projects I worked on, it was ‘only as long as the dot com boom lasts.’ You need to think about the supportability of the platform. I’m working with a Paradox client server application right now. As much as I dislike the MS monopoly, I wish it were Access, because there’s simply more information out there about Access.

There are many factors to consider when you choose a technology, and the best way to choose is not obviously clear, at least to me. Every single consideration outlined above could be crucial to a given project. Or it might be a no brainer. You can’t really know if you’ve chosen the correct technology until you’ve built the project out, and then, unless you have a forgiving boss or client, it’s probably to late to correct the worst of the mistakes. No wonder so many software projects fail.

Programming | moore |

Why I hate IDEs

I’m working on a project with Websphere Device Developer, and it constantly reminds me of why I hate integrated development environments (IDEs).

Why do I hate IDEs? Let me count the ways.

1. It’s a whole new interface that you have to learn. How do I save files? How do I save a project? How do I move around in an editor? All these questions need to be answered when I move to a new IDE; but they all lead to a more fundamental question: why should I have to relearn how to use my keyboard every time I get a new IDE.

2. One way to do things. Most IDEs have one favored way of doing anything. They may support other means, but only haphazardly. For instance, WSDD supports me editing files the filesystem, rather than through their editor, but freaks out if I use CVS (it has problems if I use most anything other than commit and update). But sometimes you aren’t even allowed the alternate method. I’m trying to get a project that was developed with one CVS repository to move to another CVS repository. WSDD lets you change repository information, but only if the project is still talking to the *same* host with the *same* cvs root. Thanks a lot guys.

3. IDEs are big pieces of code and as the size of a piece of code increases, the stability tends to decrease. In addition, they are being updated a lot more with new features (gotta give the companies some reason to buy, right). This means that you have to be aware of the environment (how often do I have to save, what work arounds do I have to use) and hence less focused on what you’re really trying to do, which is write code.

4. My biggest gripe with IDEs, however, is that they do stuff I don’t understand when I write or compile code. Now, I don’t think I should have to understand everything–I have no idea how GCC works in anything other than the most abstract sense. But and IDE is something that I interact with every day, and it’s my main interface to the code. When one does something I don’t understand to the product of my time, that scares me. I’m not just talking about code generation, although that is scary enough (just because something else generated the code, that doesn’t mean that it is right or that you won’t have to wade in and maintain it–and maintaining stuff that I write from the ground up is hard enough for me without bringing in machine generated code). I’m also talking about all the meta state that IDEs like to maintain. What files are included where, external libraries, etc. Now, of course, something has to maintain that state, but does it have to be a monolithic program with (probably) poor documentation on that process?

Some people will say that IDEs aren’t all that bad and can lead to huge productivity increases. This may be the case during the coding phase, but how much of that is lost when you have to learn a new IDE’s set of behaviors or spend time figuring out why your environment is broken by the IDE? Give me simple, reliable, old fashioned and well understood tools any day over the slick, expensive, tools that I don’t know and will have to learn each time I use a new one.

Programming | moore |

Will you be my friendster?

Friendster is an interesting phenomenon. The premise of site is that it’s easier to meet and become friends with folks if you are somehow connected to them. This is common sense and much validated by my experience–one of the things that made meeting folks in hostels when traveling was that you knew you had at least one thing in common with them: you were interested in travel. And this is true of other clubs and special interest groups–the Elks, adult sports teams, volunteer organizations, book discussion groups–all these are venues for adults to hang out with other people, knowing they have a common interest (which is whatever the purpose of the organization is).

Friendster takes this to a new level by making the social connections, by which we all have benefited, automated. Instead of having to introduce all my college friends to all my friends in Boulder, I can just invite both the Friendster, and let them check each other out. Of course, this is a pale imitation of true networking, but it’s a start. And, as many folks can attest, something that starts out as a simple on line friendship can become as deep and real as any other.

What’s interesting to me is that the level of effort to ‘get to know’ someone is very much reduced. You just look at their profile and you see what’s important to them. It’s almost as though there’s another level of friendship being created–you know more about these people than strangers or acquaintances, but less than real friends. I’ve had people email me, asking me to be their ‘friendster.’ This level of familiarity is disintermediated (I can operate entirely virtually) and permanent (unless I delete my profile, it’s going to be there as long as Friendster is around) and public (anyone connected to me can see my profile–family, friends, enemies). This means that the level of intimacy and sharing on Friendster is drastically less than you’d find at other ‘meeting places,’ including a house party.

Another interesting topic is: how the heck is Friendster going to survive. They’ve obviously put a lot of time and effort into their software. (For that matter, the members of Friendster have also put in a substantial time and data commitment.) How can the website make money (at least enough to make the site a wee bit faster)? I can see four ways:

1. Selling user information. Not very palatable, and I think this would drastically affect the quality of information that folks would be willing to give them. I’m not a big fan of giving corporations something valuable of mine to sell, and my connections definitely are valuable to me. In terms of selling information generated by users, by posting anything to Friendster, I grant them “an irrevocable, perpetual, non-exclusive, fully paid, worldwide license to use, copy, perform, display, and distribute such information and content and to prepare derivative works of, or incorporate into other works, such information and content, and to grant and authorize sublicenses of the foregoing.” In terms of selling information about users, their privacy policy doesn’t mention the possibility, other than specifying that if they change their use of personal information, they’ll email us.

2. Advertising–they already have some on the site, but we’ve all seen how profitable advertising funded websites are. Even if you’re getting a tremendous number of hits a day, the advertising has to be very focused to be successful.

3. Selling subscriptions. This is definitely coming down the pike. It will be interesting to see how many folks bail. Personally, the content on Friendster just isn’t compelling enough to pay for. If I wanted to stay in contact with old friends, especially in the age of free long distance on the weekends, I’d just call them.

4. Affiliation with product vendors. This would be easy to implement (after all, Friendster is already capturing book, movie and music info about users), wouldn’t impinge on current usage, and would offer a valuable service to users. Frankly, I’m surprised they haven’t done it already.

I like Friendster, and I like the idea of a new set of folks to ask questions of, interact with, and send email to. But I’m just not sure how long it’s going to survive. Enjoy it while it’s here.

Technology and Society | moore |

Software as commodity

So, I was perusing the Joel On Software archives last night, and came upon Strategy Letter V in which Joel expounds on the economics of software. In particular, he mentions that commodifying hardware is easier than commodifying software. This is because finding or building substitutes for software is hard.

Substitutes for any item need to have the same attributes and behavior. The new hard drive that I install in my computer might be slower or faster, larger or smaller, but it will definitely save files and be accessible to my operating system. There are two different types of attributes of any substitute. There are required attributes (can the hard drive save files?) and ancillary attributes (how much larger is the hard drive?). A potential substitute can have all the ancillary features in the world, but it isn’t a substitute until it has all the required features. The catch to building a substitute is knowing what are required and what are ancillary–spending too much time on ancillary can lead to the perfect being the enemy of the good, but spending too little means that you can’t compete on features (because, by definition, all your viable competitors will have all the required features). (For an interesting discussion of feature selection, check out this article.)

Software substitutes are difficult because people don’t like change (not in applications, not in URLs, not in business). And software is how the user interacts with the computer, so the user determines the primary attributes of any substitute. And those are different with every user, since every user uses their software in a different manner.

But, you can create substitutes for software, especially if

  1. The users are technically apt (because such users tend to resent learning new things less).
  2. You take care to mimic user interfaces as much as you can, to minimize the new things a user had to learn.
  3. It’s a well understood problem, which means the solutions are probably pretty well understood also (open standards can help with this as well)

Bug tracking software is an example of this. Now, I’m not talking about huge defect tracking systems like Rational’s ClearCase that can, if you believe the marketing, track a bug throughout the software life cycle, up into requirements and out into deployment. I’m talking about tools that allow small teams to write better code by making sure nothing slips between the cracks. I’ve worked with a number of these tools, including Joel’s own FogBUGZ, TestTrack, Mozilla’s Bugzilla and PHPBT. I have to say that I think the open source solutions (Bugzilla and PHPBT) are going to eat the commercial solutions’ lunch for small projects, because they are a cheaper substitute with all the required attributes (bug states, email changes, users, web access).

I used to like Bugzilla, but recently have become a fan of PHPBT because it’s even simpler to install. If you have local access to sendmail, a mysql database and a web server (all of which WestHost provides for $100/year or you can get on a $50 Redhat CD and install on a old Intel box). It tracks everything that you’d need to know. It ain’t elegant, but it works.

I think that in general, the web has helped to commodify software, just because it imposes a certain uniformity of user interface. Everyone expects to use forms, select boxes, and the back button. However, as eBay knows and Yahoo! Auctions found out, there are other factors that prevent substitution of web applications.

Technology | moore |

Book Review: Afghanistan

Updated 2/25/2007: Added amazon link.

Afghanistan: A Short History of Its People and Politics, by Martin Ewans, is a fantastic book. This fascinating account of this plucky country was chock full of facts that have immediate relevance. Covering from ancient times to 2002, this book provides a traditional history–no stories of the working classes or women. But it covers the byzantine regime changes of Afghanistan very well. It als does a fine job of explaining how the Afghanistan state was in constant tension between the local tribal powers and the more modern central authority of the king. The foreign situation was also an exercise in balance, with the Afghans depending on money, guns and expertise from British India to fend off the Russian Empire. However, the relationship with the Brits wasn’t entirely golden, as the three Anglo-Afghan wars suggest.

While the history was intensely interesting, the last chapters of the book, which cover the politics and battles of the last two decades which have left Afghanistan such a mess, were the most relevant for me. If you want to know how mcuh the CIA spent supporting the Taliban, it’s in there. If you want to know which external nations supported which of the warring factions, it’s in there. If you want to know why Afghanistan grows the majority of the world’s opium, it’s in there.

I won’t say this book was easy to get through. The writing is quite dense. The frequent re-appearance of characters was at times confusing, but I fear that is more a feature of Afghan history than a shortcoming of the book. For a concise political history of a nation that we’re becoming more and more involved with, check it out.

Link to the book on Amazon.

Books | moore |

Amazon Web Services

I remember way back when, in 2000, when EJBs were first hot. Everyone wanted to use EJBs in projects, mostly for the resume value. But there was also a fair bit of justified curiosity in this new technology that was being hyped so much. What did they do? Why were they cool? How did they help you?

I did some reading, and some research, and even implemented one or two toy EJBs. I remember talking to a more experienced colleague, saying “Well, all EJBs provide you is life-cycle assistance–just automatic pooling of objects, a set of services you can access, transaction support, and maybe SQL code generation.” Now, I’m young and inexperienced enough to never have had the joy of doing a CORBA application, but my colleague, who I believe had had the joy of doing one or three of those, must have been rolling her eyes when I said this. ‘Just’ life-cycle assistance, eh?

I just looked at Amazon’s web services, and I’m beginning to understand how she felt. Sure, all web services provides you is easy, (relatively) standardized access to the resources and data available in a web application. Sure, I could get the same information by screen-scraping (and many an application has done just that). But, just as EJB containers made life easier by taking care of grimy infrastructure, so do web services make life easier by exposing the information of a web application in a logical format, rather than one dependent on markup.

Using perl and XSLT (and borrowing heavily from the Developer Kit, I built an application using Amazon’s web services (the XML over HTTP API, not the full SOAP API). I was amazed at how easy it was to put together. This was partly due to the toy-like nature of the application, and how much it leveraged what Amazon already provided, but it was also due to the high level of abstraction I was able to use. Basically, Amazon exported their data model to me, and I was able to make small manipulations of that model. It took me the better part of three hours to put together an application which allows you to search on a keyword or ISBN and gives all the related books that Amazon has for that book. You know, the ‘Customers who bought this book also bought’ section.

I’ve always felt that that was the most useful bit of Amazon, and a key differentiator. This feature, as far as I can tell, leverages software to replace the knowledgeable bookstore employee. It does this by correlating book purchases. This software lends itself to some interesting uses. (I wanted to have a link to an app I found a while ago, where you entered two different artists/authors and it found the linkage between the two. But I can’t find it!)

I like this feature, but it also sucks. The aforementioned bookstore employee is much better than Amazon. Buying a book doesn’t mean that I’ll enjoy it–there are many books I’ve purchased that I wonder why I did so, even one hour after leaving the store–so that linkage isn’t surefire. In addition, purchase is a high barrier, and will probably cause me to not branch out as much as I should–rather than waste my money picking a random book, I’ll pick a book from an area I know. The book store employee, if good, can overcome both of these faults, because the process is more interactive, and the suggester has intelligence. But he doesn’t scale nearly as well as cheaply, nor does he have the breadth of Amazon’s database. (And he hates staying up all night responding to HTTP requests.)

Books Technology | moore |

Yahoo! mail problems

One of my clients had a problem earlier today. I helped them choose a new email setup, and they went with Yahoo Business Edition Mail. It’s worked like a charm for them until today. Oh, sure, they’ve had to adjust their business processes a bit, but it was a vast improvement over their previous situation, where only allowed one person in to view his or her mailbox at a time. And it’s quite low maintenance and fairly easy to learn, as it was entirely browser based.

But today, the web client for Yahoo! was busted. New layouts, new colors, old functionality gone, intermittent changes in the GUI, the whole bit. I got on the phone with Yahoo! support, and they assured me it was simply a webmail client problem. No mail was being lost. But, as Joel explains, for most folks, the interface is the program. You try explaining the difference between SMTP and POP and mail storage and local clients to an insurance agent who just wants to send her customers an application.

One of the worst bits is that, other than getting on the phone with Yahoo! support, there was absolutely nothing I could do to fix the problem. Alright, this isn’t entirely true–I could have worked on migrating them off the web client, and perhaps off Yahoo! entirely. And, had the outage continued, I probably would have begun that process. But fixing the web client was entirely out of my hands. That’s the joy and the pain of outsourcing, right? The problems aren’t yours to fix (yay!) but you can’t fix the problems yourself (boo!). Also, chances are the outsourcing provider is never going to be more enthusiastic than you about fixing your problems, and might be significantly less.

Technology | moore |

Put the pieces together: A Linux Small Business Server

I noticed an article on InfoWorld about Microsoft Small Business Server (SBS). This software package brings together important software packages for small business organizations in an easy to configure and install bundle. The primary features of SBS are, according to this article, email and calendaring, file and printer sharing, and file backup. There are additional features that you can plug in, including email clients, a fax server, remote access via the web, and possibly integration with a back end database.

All this software exists for Linux. For email, you have qmail and imap (we aren’t concerned about the client, because they’ll use Outlook, if they have Office). For calendaring, I haven’t found anything quite as slick as Outlook, but Courier promises web based calendaring, and Mozilla Calendar, when paired with a WebDAV enabled web server, like Apache with mod_dav, allows you to share calendars (it does require you to install the Mozilla browser on every client, though). For file and print sharing, there’s Samba and for backups, there’s the super stable Amanda. Remote access can be handled via VNC and fax server solutions can be built, although the author of the InfoWorld article prefers a fax over IP service which should work fine as long as you have MS Office. As for back end databases, you’d probably want PostgreSQL, probably managed via MS Access. Wrap it all up with Webmin for administration. (Full disclosure–I haven’t used all this software, just most of it.)

So, I set out on the web to see if anyone had gathered all these components together, tested them, and made it easy to install and configure. Basically, an SBS competitor that could compete on features, with the added bonus of Linux’s open nature and stability.

First, I checked out what Redhat and SuSe (1, 2) had to offer. While they had standard servers that were cheaper than the $1500 quoted for SBS, the Linux packages didn’t have all the features either. Then, I did a web search, which didn’t turn up much, except for a LUG project: the Windows Replacement Network. I’m not sure how active this project is, but at least it’s a start. I checked on SourceForge, but didn’t see anything that looked similar.

I really think that there’s an enormous opportunity for the open source community to piggy back off of MS. They’ve already done the market research, they’ve already determined the feature set that will sell to small businesses. And almost all the software is written for the Linux version of the SBS–all that really needs to happen is some configuration and documentation to make all these features work together. Cut it to a CD-ROM and start passing them out at LUG meetings. This would provide one more option in a consultant’s toolbox and give consumers one more choice.

Technology | moore |

Book Review: A Farce To Be Reckoned With

I just finished ‘A Farce to Be Reckoned With’ by Roger Zelazny and Robert Sheckley. I’ve read a fair bit of Zelazny–the Amber novels and Lord Of Light and some others. This book looked more light hearted, but I figured I’d give it a try.

I was sorely disappointed. There’s no plot. Or, rather, there is a plot, but it makes no sense. Plot turns are introduced (like the Greek gods getting free) and then dropped, willy nilly. There’s a character called Peter Westfall who gets Pandora’s Box at the beginning, but we never hear from him again. And at the end, we have a fight scene that is a total deus ex machina–the end of the book comes with no explanations.

Normally, you expect characters to have reasons for things they do. They can do weird things, but they should justify it to themselves, and have the actions be a natural outgrowth of their past. This is called characterization. Characters in this book have one sentence justifications for absurd actions. We have a nun who decides to deal with the devil, and an angel who is ordered to spy. There’s a set of religious pilgrims headed toward Venice during the Middle Ages. A demon joins them, proves himself to be a demon, and they don’t even run from him.

The dialog is wretched. Everyone converses in a stilted manner. The description is campy; the authors apparently decided to focus on the clothing of women–there are attractive wimples and red low cut blouses galore.

It feels like this book has been subjected to random editing. Or perhaps worse than random, as I feel that there may have been malicious intent at confusing the reader. Characters pop up, disappear for a while, then pop up again with no explanation (an example is the young lady named Priscilla [or Puss]).

But you know what? All of the above flaws could have been forgiven if there had been any scene, any scene at all, that was funny. I wanted to forgive the flaws–I wanted to laugh–I read the entire book, didn’t I? But I didn’t even crack a smile the entire book. There were times I put it down and thought to myself, ‘Why are you wasting your time?’ I will admit, I finished the book (I think for the same reasons that folks slow down to look at a wreck on the highway).

Don’t buy this book. If you want some funny fantasy, read ‘A Night in the Lonesome October’ (which is great!) or anything by Blaylock. Don’t buy this book.

Books | moore |

Sending Binary Data Via Struts

Struts is a MVC web framework. It allows you to specify the paths through a web application in an XML configuration file, and provides some nice tag libraries for JSP manipulation. The views for the states are usually JSPs, and the controller is a servlet provided for you.

This standard setup is fine, most of the time. But I’ve run into situations, as have others (1,2), where you need Struts to output a binary file. In my case, I’m sending a dynamic image to a non browser client, and I basically need to write a ByteArrayOutputStream to the response output stream.

Now, you’d think I’d be able to do this with a JSP. After all, a JSP is just a servlet turned inside out, right? Well, according to the specification, you’d be right. From page 42 of JSP 1.2 spec:

—————-
The JSP container should not invoke response.getWriter() until the time when the first portion of the content is to be sent to the client. This enables a number of uses of JSP, including using JSP as a language to glue actions that deliver binary content, or reliably forwarding to a servlet, or change dynamically the content type of the response before generating content. See Chapter JSP.3.
—————-

But, according to Tomcat 4.1.24 on the Linux platform, you’d be wrong. When calling ‘java.io.OutputStream rs = response.getOutputStream();’ in my JSP, I get this code generated:

—————

....snip...
      application = pageContext.getServletContext();
      config = pageContext.getServletConfig();
      session = pageContext.getSession();
      out = pageContext.getOut();
      _jspx_out = out;
java.io.OutputStream rs = response.getOutputStream();
....snip...

—————

The JSP is taking my stream before my code has a chance. Therefore, I get an “getOutputStream() has already been called for this response” error. The weird bit is that this doesn’t seem to happen on Tomcat 4.1.24 on Windows (same version of struts).

So, what do you do? You write a servlet instead. That way you have utter control over the output stream:

—————

import javax.servlet.http.*;
import javax.servlet.*;
import java.io.*;

public class BinaryStreamServlet extends HttpServlet {

   public void service(HttpServletRequest req, HttpServletResponse res)
throws ServletException, IOException {
      String contentType =
(String)req.getAttribute("contentType");
      if (contentType == null || "".equals(contentType)) {
         contentType = "image/png"; // default
      }
      res.reset();
      res.setContentType(contentType);
      OutputStream sos = res.getOutputStream();
      ByteArrayOutputStream baos = (ByteArrayOutputStream)req.getAttribute("baos");
      baos.writeTo(sos);
   }
}

—————

I set up my action classes to cache the ByteArrayOutputStream in the request, with the name “baos.” I added these lines to my web.xml:

—————

<servlet>
      <servlet-name>binaryFileServlet</servlet-name>
       <servlet-class>BinaryStreamServlet</servlet-class>
  </servlet>
....snip...
  <servlet-mapping>
    <servlet-name>binaryFileServlet</servlet-name>
    <url-pattern>/binaryFile</url-pattern>
  </servlet-mapping>

—————

and this to my struts-config.xml for any actions that needed to be able to send binary data:

—————

<forward name="success"         path="/binaryFile"/>

—————

Works like a charm. Leave the JSPs to the character data, and use this simple servlet for binary delivery.

Java | moore |