Skip to content

All posts by moore - 88. page

Lesson learned: copying bytes from an OutputStream to an InputStream in Weblogic 8

A few weeks back, I posted about calling one servlet from another, passing a multipart/form-data for submission. I ended up using an HttpURLConnection and POSTing to the second servlet. My target platform was Weblogic 8 on Solaris 8, but I was developing on Windows Server 2003; both had Weblogic running on a Java 1.4 platform.

I set both the doOutput and doInput properties of the connection to true, so I can first post the needed data and then read the response, which I can parse and pass on to the user. Here’s the setup:

    URL url = new URL("http://localhost/myws/myservlet");
    URLConnection con = url.openConnection();
    HttpURLConnection conn = (HttpURLConnection)con;
    conn.setDoOutput(true);
    conn.setDoInput(true);
    conn.setRequestProperty("Content-Type", request.getHeader("Content-Type"));
    conn.setRequestProperty("Cookie", request.getHeader("Cookie"));
    BufferedInputStream bis = new BufferedInputStream(request.getInputStream());
    BufferedOutputStream bos = new BufferedOutputStream(conn.getOutputStream());

I set the Cookie property so that the user does not have to reauthenticate to the second servlet.

However, when I was reading from the first servlet’s InputStream and writing to the second servlet’s OutputStream, I encountered some issues. The below code ran perfectly fine in the development environment, but in the production environment, the file being transferred was corrupted. The corruption happened in a number of ways, actually. At first, the file was about twice the size. Then, after fiddling a bit and moving the bis.close() and bos.close() closer to the write/read, I was able to get the file to be smaller than the posted file. However, it was a zip file and was still corrupt.)

    byte[] data = new byte[1024];
    int res = bis.read(data);
    while (res != -1) {
        bos.write(data);
        res = bis.read(data);
    }
    bos.flush();
    // other stuff, like getting response code
    bis.close();
    bos.close();

You can see that I’m double buffering the InputStream here, first with the BufferedInputStream, and then with my byte array, data[]. That’s apparently unnecessary; I don’t know where I got the idea. It actually leads to loss of data on the Solaris box.

Luckily, the Java Cookbook (and for that matter, the I/O section of the Java Tutorial) suggested code similar to this:

    int res;
    while ((res = bis.read()) != -1) {
        bos.write(res);
    }
    bis.close();
    bos.close();

This code works on both Windows and Solaris. I’m a bit mystified as to why the first byte transfer method worked on Windows but not Solaris, and I thought I’d post this so that any other folks struggling with the same problem don’t endure the debugging that I did.

Singing the praises of vmware

In the past few months, I’ve become a huge fan of vmware, the Workstation in particular. If you’re not familiar with this program, it provides a virtual machine in which you can host an operating system. If you’re developing on Windows, but targeting linux, you can run an emulated machine and deploy your software to it.

The biggest, benefit, however, occurs when starting a project. I remember at a company I worked at a few years ago, I was often one of the first people on a project. Since our technology stack often changed to meet the clients’ needs, I usually had to learn how to install and troubleshoot a new piece of server software (ATG Dynamo, BEA Weblogic, Expresso, etc). After spending a fair amount of time making sure I knew how to install and deploy the new platform, I then wrote up terse yet complete (hopefully) installation documents for the future members of the team as the project rolled into development. Of course, that was not the end of it; there were slight differences in environment and user capability which meant that I was a resource for the rest of the team regarding platform configuration.

These factors made me a strong proponent of server based development, where you buy a high powered box and everyone develops (via CVS, samba or some other network protocol) and deploys (via a virtual server for each developer) to that box. Of course, setting it up is a hassle, but once it’s done, adding new team members is not too much of a hassle. Compare this with setting up a windows box to do java development, and some of the complications that can ensue due to the differing environments.

But vmware changes the equation. Now, I, or someone like me, can create a development platform from scratch that includes everything from the operating system up. Combined with portable hard drives, which have become absurdly cheap (many gig for a few hundred bucks) you can distribute the platform to a new team member in less than hour. He or she can customize it, but if they ruin the image, it’s easy enough to give the developer another copy. No weird operating system problems and no complicated dev server setup. This software saves hours and hours of development time and lets developers focus on code and not configuration. In addition, you can actually do development on an OS with a panoply of tools (Windows) and at the same time test deployment to a serious server OS (a UNIX of some kind).

However, there are still some issues to be aware of. That lack of knowledge often means that regular programmers can’t help debug complicated deployment issues. Perhaps you believe that they shouldn’t need to, but siloing the knowledge in a few people can lead to issues. If they leave, knowledge is lost, and other team members have no place to start when troubleshooting problems. This is, of course, an issue with or without vmware, but with vmware regular developers really don’t even think about the install process; with an install document, they may not fully understand what they’re doing, but probably have a bit more knowledge regarding deployment issues.

One also needs to be more vigilant than ever about keeping everything in version control; if the vmware platforms diverge, you’re back to troubleshooting different machines. Ideally, everyone should mount a local directory as a network drive in vmware and use their own favorite development tools (Eclipse, netbeans, vi, even [shudder] emacs) on the host. Then each team member can deploy on the image and rest assured that, other than the code or configuration under development, they have the exact same deployment environment as everyone else.

In addition, vmware is a hog. This is to be expected, since it has an entire operating system to support, but I’ve found that for any real development, 1 gig of RAM is the absolute minimum. And if you have more than one image running at the same time, 1 gig is not enough. A fast processor is needed as well.

Still, for getting development off to a galloping start, vmware is a fantastic piece of software. Even with the downsides, it’s worth a look.

Running your company on webapps

Here’s an interesting post on running your company on webapps.

Of course, the issues of security (who’s responsible for it? who do you call when an employee leaves?), data ownership (how can you export your precious data if you want to move to a different provider?), legality (using gmail for business is a violation of their terms of service–didn’t check the other services), and access (if your internet access is disabled, your business is too) are skipped over entirely.

On the plus side, hey, it’s easy to get started, and the ongoing maintenance is minimal! But consider the downsides outlined above before you jump in.

It is interesting to me that that broadband is enough of a utility now, if you can get it, that a business can think of putting something as crucial as their calendar on a remote website.

Exchanging PostgreSQL for Oracle

I have a client who was building some commercial software on top of PostgreSQL. This plans to be a fairly high volume site, 1.8 million views/hour 500 hits a second. Most of the software seemed to work just fine, but they had some issues with Postgres. Specifically, the backup was failing and we couldn’t figure out why. Then, a few days ago, we saw this message:

ERROR: could not access status of transaction 1936028719
DETAIL: could not open file “/usr/local/postgres/data/pg_clog/0836”: No such file or directory

After a bit of searching, I saw two threads suggesting fixes, which ranged from deleting the offending row to recreating the entire database.

I suggested these to my client, and he thought about it for a couple of days and came up with a solution not suggested on these threads: move to Oracle. Oracle, whose licensing and pricing has been famously opaque, now has a pricing list available online, with prices for the Standard Edition One and Enterprise Edition versions of their database, as well as other software they sell. And my client decided that he could stomach paying for Oracle, given:

1. The prices aren’t too bad.
2. The amount of support and knowledgeable folks available for Oracle dwarfs the community of Postgres.
3. He just wants something to work. The value add of his company is in his service, not in the back end database (as long as it runs).

I can’t fault him for his decision. PostgreSQL is full featured, was probably responsible for Oracle becoming more transparent and reasonable in pricing, and I’ve used it in the past, but he’d had enough. It’s the same reason many folks have Macs or Windows when there is linux, which is a free tank that is “… invulnerable, and can drive across rocks and swamps at ninety miles an hour while getting a hundred miles to the gallon!”.

I’ll let you know how the migration goes.

Building a Full-Text Search Engine from Open Source Components

A friend and former colleague did a presentation a few months ago about “Building a Full-Text Search Engine from Open Source Components”. The slides are up. From the abstract:

In addition to the many useful open source applications that are available ready-to-run, there are quite a few open source APIs out there that are just crying out to be combined in new, useful, and interesting ways. By “just” writing a few lines of code to join them together it should be possible to build a new application that has a unique set of features.

Calling one servlet from another

So, I’m building a RESTful web service for a client, which is going to accept a large (60 mbish) file and a set of parameters that are attributes of the file, using the multipart/form-data enctype. The idea is to have this service be available for external programs to post to, but to also provide a nice web interface. I built another servlet which generates the usable user interface (the UI servlet), and am now having trouble pushing the data over to the RESTful servlet. After the RESTful service is called, the UI servlet needs to ensure that any errors are understandable to the user.

It looks like there are a couple of options:

1. Use RequestDispatcher to hand the request entirely over to the service. This is easy, but it means that the service now needs to return a human readable response, or you need to insert a filter to provide one.

1a. Have the RESTful service take a parameter which indicates whether its caller is another program or a human being, and use the RequestDispatcher from the UI servlet.

1b. Have no UI servlet at all, but just have the RESTful servlet be able to generate a nice user interface (or redirect to a pleasant looking JSP) via a given parameter.

2. Use the URL and HttpURLConnection objects to have the UI servlet post to the RESTful servlet just as you’d post to any other remote resource on the internet with java. This seems to work ok (I think), but requires (I also think) an absolute URL and also requires a bit of I/O to push the stream of bytes from the UI servlet to the RESTful servlet.

I can’t think of any other ways to solve this problem, and the only other solution that searching turned up is a no-no in modern servlet engines.

Web calendars: jwebcalendar and FlatCalendarXP

Just a quick note. I just spent a few days looking at web calendars for a client. In particular, they wanted to show a 12 months view, rather than the more typical month at a time view. It’s in a java environment.

Stay away from jwebcalendar. Hasn’t been developed for a few years, and even though the feature set looks great and the screen shots look gorgeous, I wasn’t able able to get it to run (in tomcat 4 or tomcat 5). After putting in some log statements, and learning that you needed to specify the xsl and xml locations on the url line (like so: localhost:8080/jwebcalendar/calendar?LAYOUT=form.url&XSL=webcalendar.form.url.xsl&XSLbase=./data/webcalendar/xsl/&XML=webcalender.form.url.vm.xml&XMLbase=./data/webcalendar/xml/&XMLfilter=.xml&HTMLbase=&TITLE=PPres) I ended up seeing this error message:

Error org.apache.xmlbeans.XmlException: error: Element type “input” must be foll owed by either attribute specifications, “>” or “/>”. org.apache.xmlbeans.XmlException: error: Element type “input” must be followed b y either attribute specifications, “>” or “/>”. at org.apache.xmlbeans.impl.store.Root$SaxLoader.load…

Since I’d burned enough time, I didn’t follow the path any further. Major bummer, as it seemed like it would be a good fit.

After that, we looked at other calendar systems, and FlatCalendarXP seemed to fit the bill. It’s payware (for commercial software) but it has an elegant API and has worked well so far. Recommended.

Breaking WEP: a Flash Presentation

About two years ago, I wrote about how to secure your wireless network by changing your router password, your SSID, and turning on WEP. Regarding WEP, I wrote:

This is a 128 bit encryption protocol. It’s not supposed to be very secure, but, as my friend said, it’s like locking your car–a thief can still get in, but it might make it hard enough to not be worth their while.

Now, some folks have created a flash movie showing just how easy it is to break WEP. Interesting to watch, and has a thumping soundtrack to boot.

Via Sex, Drugs, and Unix.

Blogging and Legal Issues

Any new technology needs to fit into existing societal infrastructures, whether it’s the printing press or the closed circuit TV system. Blogging is no different. I sometimes blog about what I get paid to work on, but always check with my employer first to make sure they’re comfortable with it. Since I’m a developer, it’s often general solutions to problems, and if need be, I omit any identifying information. Some folks take it farther..

Now, our good friends at the EFF have produced a legal guide for bloggers, which looks to be very useful, but is aimed only at those who live in the United States of America.