{"id":391,"date":"2007-01-11T10:02:52","date_gmt":"2007-01-11T16:02:52","guid":{"rendered":"http:\/\/www.mooreds.com\/wordpress\/archives\/000391"},"modified":"2007-02-16T23:35:48","modified_gmt":"2007-02-17T05:35:48","slug":"large-varchar-columns-can-lead-to-huge-esri-exports-when-using-ogr2ogr","status":"publish","type":"post","link":"https:\/\/www.mooreds.com\/wordpress\/archives\/391","title":{"rendered":"Large varchar columns can lead to huge ESRI exports when using ogr2ogr"},"content":{"rendered":"<p>I was recently using <a href=\"http:\/\/ogr.maptools.org\/ogr2ogr.html\">ogr2ogr<\/a> to convert, on the fly, some data in a <a href=\"http:\/\/postgis.refractions.net\/\">PostGIS<\/a> to other standard formats (ESRI and MapInfo).  The ESRI export in particular had some problems&#8211;it took about 4 minutes for the export of an table with 11K rows and 37 columns, and it generated a 700M dbf file.  This file was then zipped (with the other config files), and in around 6 minutes was compressed to a 7M zip file, that was sent to the browser.  Now, you can imagine how thrilled a user would be to wait 10 minutes for an export.  Apache was timing out (the <a href=\"http:\/\/httpd.apache.org\/docs\/2.0\/mod\/core.html#timeout\">default timeout is 5 min<\/a>) and I was at a loss as to how to address the performance issue.<\/p>\n<p>I mentioned this to a colleague who has significantly more experience with GIS tools, and he pointed out that in the source table there were several <code>varchar(4000)<\/code> fields.  Now, in PostgreSQL, <a href=\"http:\/\/www.postgresql.org\/docs\/8.1\/static\/datatype-character.html\">[i]f the string to be stored is shorter than the declared length &#8230; values of type character varying [varchar] will simply store the shorter string<\/a>.  But the ESRI export does not do that&#8211;each <code>varchar(4000)<\/code> field was padded to a length of 4000, even though none of the fields approached that length.<\/p>\n<p>The solution?  A few simple <code>select max(length(colname)) from table<\/code> and <code>alter table<\/code> statements, and the <code>varchar(4000)<\/code>columns were decreased in size.  The dbf file decreased to a 50M file, uncompressed, and the entire zip file decreased to 5M.  As you can guess, the download time was slashed.<\/p>\n<p><em>Update 2\/16: The kind members of the <a href=\"http:\/\/lists.maptools.org\/mailman\/listinfo\/gdal-dev\">GDAL mailing list<\/a> pointed me to a document <a href=\"http:\/\/www.gdal.org\/ogr\/drv_shapefile.html\">listing all the limitations of the ESRI driver for ogr2ogr<\/a>.\u00a0 Check out the &#8220;Creation Issues Section&#8221;.<\/em><\/p>\n<p>[tags]PostGIS, ESRI, ogr2ogr[\/tags]<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I was recently using ogr2ogr to convert, on the fly, some data in a PostGIS to other standard formats (ESRI and MapInfo). The ESRI export in particular had some problems&#8211;it [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13,6],"tags":[],"class_list":["post-391","post","type-post","status-publish","format-standard","hentry","category-databases","category-programming"],"_links":{"self":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts\/391","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/comments?post=391"}],"version-history":[{"count":0,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts\/391\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/media?parent=391"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/categories?post=391"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/tags?post=391"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}