Skip to content

All posts by moore - 87. page

Set up your own geocode service

Update, 2/9/06: this post only outlines how to set up a geocode engine for the United States. I don’t know how to do it for any other countries.

Geocoder.us provides you with a REST based geocoding service, but their commercial services are not free. Luckily, the data they use is public domain, and there are some helpful perl modules which make setting up your own service a snap. This post steps you through setting up your own geocoding service (for the USA), based on public domain census data. You end up with a Google map of any address in the USA, but of course the lat/long you find could be used with any mapping service.

First, get the data.

$ wget -r -np -w 5 --random-wait ftp://www2.census.gov/geo/tiger/tiger2004se/

If you only want the data for one state, put the two digit state code at the end of the ftp:// url above (eg ftp://www2.census.gov/geo/tiger/tiger2004se/CO/ for Colorado’s data).

Second, install the needed perl modules. (I did this on cygwin and linux, and it was a snap both times. See this page for instructions on installing to a nonstandard location with the CPAN module and don’t forget to set your PERL5LIB variable.)

$ perl -MCPAN -e shell
cpan> install S/SD/SDERLE/Geo-Coder-US-1.00.tar.gz
cpan> install S/SM/SMPETERS/Archive-Zip-1.16.tar.gz

Third, import the tiger data (this code comes from the Geo::Coder::US perldoc, and took 4.5 hours to execute on a 2.6ghz pentium4 with 1 gig of memory). Note that if you install via the CPAN module as shown above, the import_tiger_zip.pl file is under ~/.cpan/:

$ find www2.census.gov/geo/tiger/tiger2004se/CO/ -name \*.zip 
  | xargs -n1 perl /path/to/import_tiger_zip.pl geocoder.db

Now you’re ready to find the lat/long of an address. Find one that you’d like to map, like say, the Colorado Dept of Revenue: 1375 Sherman St, Denver, CO.

$ perl -MGeo::Coder::US -e 'Geo::Coder::US->set_db( "geocoder.db" );
my($res) = Geo::Coder::US->geocode("1375 Sherman St, Denver, CO" ); 
print "$res->{lat}, $ res->{long}\n\n";'

39.691702, -104.985361

And then you can map it with Google maps.

Now, why wouldn’t you just use Yahoo!’s service (which provides geocoding and mapping APIs)? Perhaps you like Google’s maps better. Perhaps you don’t want to use a mapping service at all, you just want to find lat/longs without reaching out over the network.

Technology | moore |

Running Tomcat on port 80

The typical java web application is fronted by a web server (usually Apache) for a number of reasons. Apache handles static content well, and also is easier to configure to listen on privileged ports (under 1024). I’ve written before about different options for connecting Tomcat and Apache, but there are times when all you need is a servlet engine, and installing Apache is overkill. If you don’t want users to see a nonstandard port in their url (http://foo.com:8080/webapp/), then you have a couple of options.

You can run tomcat as root. This is probably not a good idea, since anyone who can write a jsp can now execute arbitrary commands as root. I don’t know how Tomcat’s security is, but in general, the fewer applications running with super user privileges, the better.

If you share my dislike of Tomcat running as root, here’s an excellent rundown of the options for running Tomcat on port 80. I went the route of jsvc. This seemed to work just fine, though every time we shut down tomcat, we would get an entry in the error log file: jsvc.exec error: Service exit with a return value of 143.

That didn’t start to disturb me until I realized that the destroy method of our servlets weren’t being called. This method cleaned up after the servlet and it was important that it get executed. A bit of googling turned up a discussion of this very problem. The version of jsvc that ships with Tomcat 5.0.27 doesn’t shut down Tomcat very nicely.

I downloaded and compiled subversion, because that’s the version control system that the daemon jakarta project (of which jsvc is a part) used. I then checked out the version of the source tagged daemon-1_0_1 (svn co http://svn.apache.org/repos/asf/jakarta/commons/proper/daemon/tags/daemon-1_0_1/) and rebuilt jsvc. This new version allows tomcat to call the destroy methods of servlets, and everything seems to be happy.

Java | moore |

Amazon’s Mechanical Turk

I did some work a long time ago with Amazon Web Services; I gave them an email address and they periodically send me newsletters about their web services. The most recent one contained a link to an article about a new service: Amazon Mechanical Turk. This service provides ‘Artificial Artificial Intelligence’ and lets developers place tasks in front of humans in a scalable, standardized manner. Amazon, with their infrastructure, makes sure that the task is completed and pays the human who completes the task. Right now, I only saw one set of tasks, sponsored by Amazon, so I’m not sure of the uptake. But this is certainly an fascinating idea–an interesting inverse of the normal computer/human relationship.

Technology and Society | moore |

unescaping a string with PL/SQL

I’ve written about PL/SQL before, but I’ve recently started working on a project that uses it heavily. Given the amount of code written for Oracle databases, I’m rather suprised that there’s not a PL/SQL Cookbook, where, like the Perl Cookbook and the Java Cookbook (more cookbooks from O’Reilly are listed here). There is an Oracle Cookbook, but based on a quick scan of Amazon, it’s is focused, as you’d expect, more on the database design than on PL/SQL programming. (Interestingly, there is a Oracle+PHP cookbook, and a PL/SQL sample code page but neither of those is quite what I’m looking for.)

The reason that I’d like a PL/SQL cookbook is that there are large sets of problems that routinely need to be solved in PL/SQL, but the language is so low level (though they just added some regex support in 10g; bravo!) that doing these routine tasks and making sure they’re correctly implemented can be difficult and tedious. This is especially true when it’s a programmer from a different language who’s used to higher levels of abstraction (like, for example, the good folks who author CPAN modules provide)–it’d be well worth my $70 to make sure that I never had to deal with a problem like, say, unescaping a string.

For that’s the problem I recently had to solve. Essentially, we have a string that looks like this: yellow,apple. This string represents two values, which need to be put in different places by splitting them up into ‘yellow’ and ‘apple’. All well and good until the possiblity of embedded commas arises, for it’s possible that the desired end values were ‘yellow,blue’ and ‘apple,banana’. The answer, of course, is to escape the commas on the way in (turning the second input into something like this: yellow:,blue,apple:,banana, and when processing to unescape those special characters (both the comma and the escape character, which in the example is the colon). That’s what these three functions do. They take a string like the above examples and parse it into a table, to be iterated over at your leisure.

/* ------------------- function splitit ------------------*/
FUNCTION splitit(p_str VARCHAR2, p_del VARCHAR2  := ',',p_idx PLS_INTEGER, p_esc VARCHAR2

:= ':')
RETURN INTEGER
IS
l_idx       PLS_INTEGER;
l_chars_before      VARCHAR2(32767);
l_escape_char       VARCHAR2(1) := p_esc;
l_chars_before_count        PLS_INTEGER := 0;
BEGIN
>
LOOP
l_idx := instr(p_str,p_del, p_idx);
IF l_idx > 0 then
WHILE substr(p_str, l_idx-l_chars_before_count-1, 1) = l_escape_char LOOP
l_chars_before_count := l_chars_before_count +1;
END LOOP;

IF mod(l_chars_before_count, 2) = 0 THEN
-- if chars_before_count is even, then we're at a segment boundary
RETURN l_idx;
ELSE
-- if odd, then we're at an escaped delimiter, want to move past
RETURN splitit(p_str, p_del, l_idx+1, p_esc);
END IF;
l_chars_before_count := 0;
ELSE
RETURN l_idx;
EXIT outer;
END IF;
END LOOP;
END splitit;
/* ------------------- function splitit ------------------*/

/* ------------------- function unescape ------------------*/

FUNCTION unescape(p_str VARCHAR2, p_del VARCHAR2 := ',', p_esc VARCHAR2 := ':')
RETURN VARCHAR2
IS
l_str VARCHAR2(32767);
BEGIN
l_str := replace(p_str, p_esc||p_del, p_del);
l_str := replace(l_str, p_esc||p_esc, p_esc);
RETURN l_str;
END unescape;
/* ------------------- function unescape ------------------*/

/* ------------------- function split ------------------*/

FUNCTION split(p_list VARCHAR2, p_del VARCHAR2 := ',')
RETURN split_tbl
IS
l_idx       PLS_INTEGER;
split_idx   PLS_INTEGER     := 0;
l_list      VARCHAR2(32767) := p_list;
l_chars_before      VARCHAR2(32767);
l_escape_char       VARCHAR2(1) := ':';
l_array split_tbl := split_tbl('','','','','','','','','','');
BEGIN
l_list := p_list;
LOOP
split_idx := split_idx + 1;
IF split_idx > 10 then
EXIT;
END IF;

l_idx := splitit(l_list, p_del, 1, l_escape_char);
IF l_idx > 0 then
l_array(split_idx) := unescape(substr(l_list,1,l_idx-1), p_del,

l_escape_char);
l_list := substr(l_list,l_idx+length(p_del));
ELSE
l_array(split_idx) := l_list;
EXIT;
END IF;
END LOOP;
RETURN l_array;
END split;
/* ------------------- function split ------------------*/

/* in the header file, split_tbl is defined */
TYPE split_tbl IS TABLE of varchar2(32767)

Not all of this code is mine–I built on a solution from a colleague. But I hope this saves one other person from the afternoon I just endured. And if you are a PL/SQL expert and care to critique this solution, please feel free.

Databases Oracle Programming | moore |

Article on open formats

Gervase Markham has written an interesting article about open document formats. I did a bit of lurking on the bugzilla development lists for a while and saw Gervase in action–quite a programmer and also interested in the end user’s experience. I think he raises some important issues–if html had been owned by a company, the internet (as the web is commonly known, even though it’s only a part of the internet) would not be where it is today. If Microsoft Word (or WordPerfect) had opened up their document specification (or worked with other interested parties on a common one), other companies could have competed on features and consumers would have benefited. More on OpenDocument, including a link to a marked up version of a letter from Microsoft regarding the standard.

Technology | moore |

XHTML Compatibility in the mobile world

Here is an interesting outline of some of the issues faced in making mobile user interfaces work well with today’s technologies. What’s old is new again–browsers on cell phones are dealing with the same standards compliance and diversity issues that desktop browsers were faced with 10 years ago. The difference is that there’s no one (yet) with large enough market share to rule by fiat (like Navigator and then Internet Explorer did).

Mobile Technology | moore |

Cross browser javascript/css development issues

I’m working on an application that needs to be supported on a wide variety of browsers, and unfortunately includes some interesting javascript and css. There are three problems we’ve encountered so far.

1. Finding Browser share

When you want to support most users, you have to try to figure out what they’re using. There are at least three or four different sites which give you their browser share, but I think you have to pay if you want really accurate, detailed information; here’s one source, here’s another, and here’s one last site. Update, 11/3: here are stats for the www.bbc.co.uk homepage.

2. Javascript specifications

Perhaps it’s just me, but I’ve had a devil of a time finding a list of javascript events supported by various browsers. I’ll give it to Microsoft, they have some documentation on supported events; I couldn’t find a similar list of events anywhere on the mozilla site. Here’s the Mozilla Javascript page but I don’t see anything resembling an API there. (All I want is a javascript javadoc!) Here is the best comparison of event support on modern browsers that I found. Update 10/31: here is a list of events that Gecko recognizes.

3. Getting ahold of old browsers and older operating systems, so you can test

Luckily, this is fairly easy to solve. VMWare (which I’ve written about previously) takes care of the various operating systems (well, that and a mac mini) that we need to test under. And a simple google search turned up a fantastic archive of old browsers: browsers.evolt.org, which has many different browsers going all the way back to NCSA Mosaic!.

Programming | moore |