Skip to content

Useful Tools - 8. page

Review of Emergent One

A few weeks ago I sat down with some folks at Emergent One and got a demo of their product.  The reason I reached out to them is because I heard great things about their demo at GlueCon.  My company is considering building a mobile app (who isn’t, right!) and I thought that what EmergentOne offered was a great way to accelerate the server side development of that app.

We are slightly outside of their target market–our API would be solely for internal use of a small team, while it seems like they are aiming at companies who want to make an API available to a larger audience (either external or a larger internal development staff).

Regardless of target market, they have a slick product.  They have a self service web application which can generate APIs directly from database tables.  You allow the app select access to your database.  (I believe only mysql and postgresql are supported at the moment, but I know things are moving fast over there as well.)  You then work within the app to build an API based on the tables in your database.  You can have derived fields as well as fields that map to columns in your database directly.  You can filter your data (so if you only want to expose a subset of your data, you can create an endpoint that only displays that: “users over 40”, for example).  You can also add comments to fields.

After you define as many API endpoints as you want, you can manage access to them with application keys or usernames and passwords.  Automatic documentation with datatypes and whatever comments you have added is generated, and there is a developer portal where it is easy to play around with the APIs and see what you missed.

What they showed me is great, but this product is still in private beta.  That means there are some rough edges.  The biggest hole (a fix for which the demoer promised was coming very soon in their development plan) is that you can’t search against the API.  So, if you have an API exposing your pets table, you can create an endpoint to retrieve all pets, and you can retrieve one pet based on id.  You can create a ‘dogs-only’ endpoint, and get all dogs or one dog.  But you can’t query the dogs-only endpoint for dogs that weigh over 25 pounds and have short hair (or any other type of querying).  I only played with read-only APIs, so I’m not sure how the write-access APIs work.

There’s also always the issue of introducing another vendor into the system.  Since we are looking at this for mobile apps, performance is very important.  It seemed like the demoer was well aware of this issue.  He mentioned an SLA would be likely when they went public, and also talked about some of the steps they are taking to make sure their app uses indexes and other metadata about the tables being exposed to execute as quickly as possible.

I haven’t built an API with any of the other tools out there, so I can’t compare the ease of Emergent One with, say, tools like jboss resteasy that work with a java layer, or usergrid, which autogenerates an API but requires moving data into it.  But I can say that this was a very easy way to go database to API in less than an hour (with iptables troubleshooting mixed in!).  If they get searching right–making it easy to use and performant–this will be a fantastic product.

Cloud Computing Mobile Technology Useful Tools | moore |

Why I’m using a RSS reader (again)

Many moons ago, I moved from a personal handwritten RSS reader to Bloglines.  Then, a few years ago, I stopped using Bloglines (before they were bought).  I had too many feeds on Bloglines and was spending less time on the computer after work.  I also had some big personal happenings taking a lot of time, and was burnt out on learning new technologies.

So, for a couple of years, I read blogs occasionally, but didn’t subscribe to them in a reader.  The ones that really spoke to me were either visited regularly (by memory) or subscribed to via email (and often unsubscribed quickly).

But recently, I have been consuming a lot more content.  I think there are a number of reasons for this, but the biggest is that the household has an iPad. The iPad is crappy for creating content, but is fantastic for consuming it.  I also enjoy the ability to email articles with the tap of the finger.

Last week, I found myself visiting the same seven or eight sites over and over again, to see if there was anything new posted.  After the third go-around, I mentally kicked myself and said, ‘that’s what an RSS reader is for!’.

By happenstance, I saw an article on HackerNews about the NewsBlur founder around the same time, and decided to check NewsBlur out.  I actually appreciate the limited number of blogs available on the free version of NewsBlur–I hope that will help me avoid the blog overflow that occured last time.

Blog Useful Tools | moore |

Using Munin To Track Business Values

Munin is a great piece of software that we use at my company to track overall trends in disk usage, CPU and other system purposes.  Now, we don’t have a ton of servers, so I’m not sure how munin scales for many machines, but it has been invaluable in troubleshooting problems and giving us historic context.

One thing we’ve started to do is to incorporate business specific metrics into munin.  This is good because it ties the technical operations more tightly to the business, making us aware when there are issues.

Anything you can run a sql query or do a wget for, you can graph in munin.  (Here’s something I wrote about writing munin plugins a year ago.)

I don’t think that munin is acceptable as a general purpose dashboard.  I’d probably look at Google Analytics if I was web drivingdriven (updated Feb 25 2012), and at statsmix if I needed to integrate a bunch of disparate services.  But for bringing additional business awareness to a technical team, writing a few custom munin plugins that will graph key business metrics can be very useful.

Business Tips Useful Tools | moore |

Help A Reporter Out

This site, Help A Reporter Out/HARO, is a great resource for anyone with expertise in any field who wants to be better known. (It’s also a resource for journalists, but I don’t have any experience with that side of the site.)

To participate as a source, you sign up and then are sent three emails every work day. Every email consists of 35-50 reporter queries, grouped by area (‘Travel’, ‘Tech’, ‘Education’, etc). Included in each query is the deadline, name of the reporter (if provided), anonymized reporter email address, and media outlet. There’s also some advertising, but I tend to skip past that (although I did click once on an ad that led me to learn about Google Apps Scripting).

Once you get the email, you scan the queries and see if you can and want to respond to any. I recently responded to one, but before that I’d passed off a number of requests for information to other people. Such handoffs are a great way to help other people out, and it’s kinda fun–who doesn’t want to talk to a reporter? (Psst, if you’re looking for a job, sending over a reporter query related to a company’s business is a great way to build rapport with people there.)

As I said above, a few days ago I’d finally found a query I felt I could help with, and responded with an email answering the reporter’s questions. The reporter responded, and I ended up have a 10 minute phone call about the story. So even when you actually participate, it’s pretty low impact.

I will say the hardest part of participating in HARO for me is scanning the emails–scanning 150 queries a day wears me down. I’ve stopped scanning them all, but still check from time to time.

I just think this is the coolest example of something that the internet allows, but couldn’t happen (at scale) any other way. The costs, both in money and time, of sending out and responding to reporter’s queries would be just too high.

Business Jobs Technology and Society Useful Tools | moore |

Running a Google Apps Script Once a Month

I needed a way to email a Google spreadsheet to my boss once a month, for some reporting purposes.  I could have put an entry in my calendar reminding me to do it, but I thought it would be a great time to try out the Google Docs scripting that I had read about for a year or two, and seen an AppSumo video about.  (I got the AppSumo video for free, from an ad on HARO.)

It was laughably easy to get write the actual script (here’s a great set of tutorials).  The only rub was Google doesn’t allow you to run scripts in month intervals, only hourly, daily or weekly.  A small bit of scripting got around that.

Here’s the final script (edited to remove sensitive data):

function myFunction() {
  var dayOfMonth = Utilities.formatDate(new Date(), "GMT", "dd");
  if (dayOfMonth == 05){
    MailApp.sendEmail("email@example.com", "Spreadsheet Report Subject", 
'https://spreadsheets.google.com/a/mydomain.com/ccc?key='+SpreadsheetApp.getActiveSpreadsheet().getId());
  }
}

I set up a daily trigger for this script and installed it within the spreadsheet I needed to send.

I really really like Google Apps Script.  I think it has the power to be the VB of the web, in the way that VB made it easy to automate MS Office, reduce drudgery, and allow non developers to build business solutions.  It also ties together some really powerful tools–check out all the APIs you can access.

Once you let non developers develop, which is what Google Apps Script does, you do run into some maintenance issues (versioning, sharing the code, testing), but the same is true with Excel Macros, and solving those issues is for greater minds than mine.

Business Cloud Computing Dynamic Languages Google Apps Useful Tools | moore |

google_debug parameter: Google Ad Manager Help

I’m using Google Ad Manager because OpenX didn’t get back to me quickly enough when I requested a free hosted account.  Sorry, it’s not Google Ad Manager, it’s Double Click for Publishers.  Whoops.

It’s relatively complicated software, probably because serving ads is relatively complicated business.  I have hacked my way through it, but the single most helpful tip I found was to append <code>?google_debug</code> to the page you’re trying to get the ad displaying on.  This pops up a window with what is basically a stack trace of the javascript execution on your page.  You might not understand all of it, but you can definitely cut and paste the message you find into the Google Ad Manager help search, and find more useful answers.

Useful Tools | moore |

Quick Donation Website Setup

This is based off a question I got from a friend–how can non techies easily set up a website to take donations and publicize a cause?

It’s not hard.

Of course, Google Sites isn’t the most flexible publishing platform (see, for instance, this discussion on how to link to an uploaded PDF), but it will get the job done for most simple sites.

Long live the non expert publisher!

Technology and Society Useful Tools | moore |

On My Brief Experience with Wireframe Tools

I was looking for a tool to quickly and easily build wireframes.  Oh, it had to be free too.  Luckily, I found this useful list of wireframing tools.

The first one I picked was Simple Diagrams.  I like Adobe AIR apps, already have the runtime and it looked nice.  Even had some tutorials, even if they were third party generated.

But then, I’m actually using it, and I go to save the image and there’s no save!  You can export as PNG, but you can’t, using the free version, save a project, shut down your computer, turn it on again, and then open up your project to where you were.  Now, I’m no opponent of freemium business models, but if you can’t save a diagram and come back to it, the program is really crippled.  Call it a demo, but it’s not anywhere near a free functioning version.

I then tried Web Site Wireframe, but wanted something that would work offline.

Finally, I tried Pencil Prject.  It’s not the most polished (one of the tabs in the main screen is “My Stuffs”) but it worked–I could edit files offline, and save projects and re-open them later. It’s also kinda cool because it is built on top of the XUL and Firefox application frameworks (they have a plugin and a standalone app available).

Any wireframe tools you would recommend?

RIA Useful Tools | moore |

Parsing street addresses in a java application

I recently had to find a way to parse a street address into its component parts, and thought I’d share my adventure.

The idea is to take a string like “123 S Main Street” and break it apart into the street number (123), the street direction (S), the street name (Main) and the street type (Street).

At first, I thought that regular expressions would work, but the sheer variety of legal postal street addresses quickly dissuaded me, as did my boss’s misgivings.

Stackoverflow has a nice discussion of the problem, which gave me some additional pointers.  There’s a commercial solution, which is available as a COM component or a web service–I didn’t try this.  There is a free, but application/attribution required, web service provided by a university that did a great job (thanks, California tax payers).  This solution is also available in a for-pay variant.

Neither of these were desirable because we needed to parse a lot of addresses quickly, and calling out over the web can be slow.  Some more digging turned up this stack question and  JGeocoder, which has a fairly robust address parser.  It’s not perfect, but it was free and open source.  I am not sure if it is still in development (the author didn’t respond to my email) but it does what we need it to do.

As an added bonus, we’re using pentaho for the data processing, and you can call java classes directly from your data processing steps, so I didn’t even have to wrap the java call in a shell script or anything.

Java Useful Tools | moore |

Own your social media–install Storytlr

I guess I’m just not very trusting, because I like to have copies of my data.  I host my own blog, rather than use blogger or wordpress.com.  I host my own email (or at least one of my two main accounts).  I prefer to document interesting things on my blog, rather than a site like Quora or Stack Overflow (though I do have an account on the latter).  Heck, even though I use an open ID provider, my own domain is the master, and I just delegate to myopenid.com.

So, since I recently have been putting a bit more effort into my social media presence (you can find me on twitter here), I looked around to find a backup solution.  I did find one–Storytlr–via this article on backing up your twitter feed.  It apparently used to be a hosted service, but now is open source–code here, install instructions here.  (There’s at least one for pay service too, but then, you don’t really own your data, plus I’m cheap.)

It was pretty trivial to install.  I ran into this issue with Storytlr not recognizing that PDO was installed, but the fix (hacking the install script) worked, and I didn’t run into the Zend error also in that bug post.

I also ran into an issue where I chose an admin password of less than six characters on install.  Storytlr was happy to let me do that, but then wouldn’t let me enter the exact same password when I was logging in for the first time.  To fix this, I had to update the password column in the users table with a new MD5 string, created using this tool.

So, what does Storytlr actually give me?

  • Access to my data: I set up feeds to be polled regularly (requires access to cron) and can export them to CSV whenever I want.  And I keep them as long as I want to.
  • One single point of view of all my social content.
  • Really easy way to add more feeds if I join a new social network.  Here are the sites/networks Storytlr supports right now.

The issues I ran into are:

  • Technical issues, resolved as documented above.
  • No support for facebook.  (Well, there’s this experimental support, announced here, but nothing that is part of the project.)  This is big, given how bad Facebook is with respect to privacy.  I am not sure what my next steps are here.
  • Not wanting others to have access to my lifestream.  This was easily fixed with a Auth directive.

If you are depending on social media sites, have some technical chops, a server to host it on, and want to ensure a historical archive, you should look at Storytlr.

Blog Social Media Useful Tools Web Applications | moore |