“Spreadsheets are code”:
Useful Tools - 6. page
Lessons from curating a link blog
I maintain a link blog about Colorado food and local food in general. I use Tumblr, but I’m only incidentally interested in Tumblr traffic. Tumblr hooks up to Facebook and Twitter, and pushes links there. (I realize that I am missing interaction on Twitter and Facebook by using these networks as broadcast only, but I don’t have time to fully engage, so I thought a limited presence was better than nothing.)
Having maintained this link blog for over two years, I have learned a few things.
- It is easy to start a project like this, but hard to finish. There’s always more to do. I think I’ll stop when it stops being interesting.
- Deciding to do this is a great way to gain a broad understanding of a field while providing some value (via curating). As you find more and more sources of links, videos, articles and audio content, you’ll gain a sense of what is happening. Even if you don’t painstakingly read every article, you’ll still get a sense.
- Speaking of sources, Google alerts is your friend. I get emailed alerts on a variety of searches, and about 25% of the results are worth posting. Facebook and twitter are additional great sources of links.
- An RSS reader can help you if you are really diving in.
- Giving someone notice that you’ve referenced their article via an ‘@’ mention will get you their attention.
- Queuing up posts on Tumblr is a life saver. This lets you stack up posts and portion them out one per day. I typically have between 15 and 30 posts in my queue. This makes timely posts more difficult, but frees me up to forget about the link blog for weeks at a time.
- A link blog like this is a great use of your in between time, especially if you have a smartphone. In five minutes I can scan and post two or three links, where five minutes is barely enough time to think of a regular blog post. The Tumblr app is very good.
- A linkblog is a great resource for other content generation. I have a newsletter about local food as well, and a key section of that is interesting links. Those are almost entirely drawn from the Tumblr.
The linkblog approach is very similar to Twitter, but differs in a few crucial ways:
- the permanence of the link repository (Twitter only stores your last 3200 tweets)
- less interaction, unless you login to each social network
- the ability to post richer media
- you own the content (and can host it on your domain)
- ability to queue up content (without paying)
These attributes make a linkblog a fine complement to Twitter.
There are some problems with this model, however.
- Limited interaction with followers, either on Tumblr, Facebook or Twitter.
- I’ve found that engaging on Twitter and Facebook directly is far more effective if you want content to be viewed or links to be clicked.
- A linkblog like this is not truly building my tribe
So, if you have limited time, want to gain insight into a particular area of interest, and are OK with the drawbacks, create a linkblog.
Lob Postcard Review
A few months ago, I wrote a Zapier app to integrate with the Lob postcard API. I actually spent the 94 cents to get a postcard delivered to me (I paid 24 cents too much, as Lob has now dropped their price). The text of the postcard doesn’t really matter, but it was an idea I had to offer a SaaS that would verify someone lived where they said they lived, using postal mail. Here are the front and back of the postcard (address is blacked out).
Here is the PDF that Lob generated from both a PDF file I generated for the front (the QR code was created using this site) and a text message for the back.
A few observations about the postcard.
- The card is matte and feels solid.
- The QR code is smudged, but still works.
- The text message on the back appears a bit closer to the edge on the actual postcard than it does on the PDF image.
- The front of the postcard appears exactly as it was on the PDF.
- It took about 5 business days (sorry, working from memory) for delivery.
So, if I were going to use Lob for production, I would send a few more test mailings and make sure that the smudge was a one off and not a systemic issue. I would definitely generate PDFs for both the front and back sides–the control you have is worth the hassle. Luckily, there are many ways to generate a PDF nowadays (including, per Atwood’s Law, javascript). I also would not use it for time sensitive notifications. To be fair, any postal mail has this limitation. For such notifications, services like Twilio or email are better fits.
In the months since I discovered Lob, I’ve been looking for a standalone business case. However, business needs that are:
- high value enough to spend significant per notification money and
- slow enough to make sending mail a viable alternative to texting or emailing and
- split apart from a larger service (like dentist appointment scheduling)
seem pretty few and far between. You can see a short discussion I kicked off on hackernews. However, they’ve raised plenty of money, so they don’t appear to be going anywhere soon.
But the non-standalone business cases for direct postcard mail are numerous (just look in your mailbox).
King Soopers, Please Fix Your HomeShop Service
King Soopers is leaving money on the table.
I tweeted about this a while ago, but thought it was worth expanding into a full blog post.
Now, first, let me say that I haven’t used new shopping services like Instacart, but I agree with Gary Vee (link is a video, you want to go to 6:50)–home shopping gives me more time and that is an extremely precious commodity that I am willing to trade money for.
King Soopers, for readers outside of Colorado, is one of the two major grocery store chains–it’s owned by Kroger, a name that may be more familiar to you. (The other is Safeway, and they don’t do home delivery to my area.) The King Soopers online shopping experience, is abysmal.
Let me count the ways:
- It’s difficult to find the link from their home page (here it is if you want to try King Soopers HomeShop).
- There is no clear delivery map–you have to enter your address and then you find out if they do deliveries to your location.
- The UI of the shopping experience hasn’t changed since 2009 (the first time I used it).
- Searching is tough–synonyms are not handled (searching for ‘scallions’ returns no produce results, whereas searching for ‘green onions’ does, even though they are the same vegetable).
- Product description for produce is unclear. For instance, here is the description some onions: “King Soopers/Onions Yellow Organic Fresh Produce (Each)“, but the price is is by the pound. So, when I put in 1 into the quantity section, am I getting one onion or one pound of onions? I don’t know.
- Sorting is limited to ‘Brand’ or ‘Description’. That’s right folks, an ecommerce site where you can’t sort on price.
- Browse options are not clear–they have an ‘aisle’ metaphor, but organic produce is in a different category than regular produce.
- Product images are spotty.
- They don’t have a submit event fire when you hit ‘enter’ in the search box. To be fair, this issue only affects FireFox.
- They don’t have every product a store has. But wait, aren’t they fulfilling from the same warehouse? I don’t know.
So, why would you ever use King Soopers HomeShop? Because home delivery is so dang easy, and spending an extra $11 to not have to go to the store is worth it, especially for a big shopping trip. There are other reasons as well:
- No (or far fewer) impulse buys. Maybe this is the reason they aren’t investing in this product?
- You can save a list of staples and just re-order it.
- Did I mention they deliver food to your house? The drivers have been very nice, and even brought the bags into our house at times.
- The UI for picking a pickup time is good, and I haven’t encountered any traffic jams.
- I’ve been happy about the quality of non fungible items I’ve had picked out for me (such as lettuce).
- They are good about substitutions. If something you request is out of stock, there are three choices: ‘no substitutions, just omit’ or ‘please substitute’. If you choose the latter, you can either accept or reject the substitution at delivery.
But the best reason for using HomeShop is the convenience of having your food delivered to you for a reasonable price. As a busy person, this is huge.
I really really want King Soopers to repair the crappy parts of this service and make it amazing. Heck, I’ll even consult with you on it. Home delivery is a win in so many ways–please fix it, King Soopers.
Get Developers Going with Vagrant
I just ran across the Data Science at the Command Line, a new book, website and VM, which gives you many, many unix tools to manipulate large data sets.
I have played around with the tools, but what really stood out to me was how easy it was to get them installed. While the author does provide install instructions, the VM is ready to go in only five steps (two if you already have vagrant and virtual box installed).
I have used Vagrant in the past and it makes managing VMs very easy. If you are distributing a developer tool, make it easy to get up and running by letting the user download a virtual environment that is correctly configured. Then trying out your software becomes a 30 second test, not a 10 minute test.
Of course, the author of Data Science at the Command Line is not alone. I noticed Ionic is providing a VM for android development too. It looks like Vagrantcloud lets you share virtual instances with the world as well.
Frankly, I used to download and install cygwin first thing on every Windows computer I bought. But VMs are so easy now it’s easier to just install VirtualBox, Vagrant and a version of Linux (CentOS or Ubuntu).
Consolidate external dependency notifications using Zapier
As I wrote over at the Geek Estate Blog, if you build your business on vendors, you should monitor them. In the past, I’ve used a variety of services to monitor vendor services, from pingdom to wget/cron to nagios. These services are great about telling you when some external service is unavailable, but are not so hot at telling you when a service is going to be down (for planned maintenance) or back up.
For that, you need to be monitoring, and reading, vendor announcements, however the vendor has decided to provide them, whether that is as a blog/RSS feed, twitter feed, email newsletter, a status page or something else.
However, it can be tough to monitor and read announcements in two or more places. Here, Zapier or a similar service can help. Pick one place to be notified. For me, that’s typically an email inbox, because, frankly, other data sources can be ignored (except phone texts), but I’ll always check my email.
Then, use Zapier’s zaps to transform any announcements from the other sources to emails. For instance, there is an RSS trigger for new items in a feed and a Twitter trigger for tweets from a user. Status pages often provide RSS feeds (Google’s does). If the service provider doesn’t provide a structured method like an RSS feed to notify you of changes, but does provide a webpage of announcements, you could look at a service like changedetection.com and have the email sent to your inbox or parsed by Zapier and pushed to your notification location.
And for the output side, you can just use Zapier’s ‘send outbound email’ action. If you want to have all notifications pushed to your phone, an RSS reader or Twitter acount, you can use Zapier to send texts, create RSS items or tweets as well.
Building a System with IronMQ and Python
One of my most recent projects was writing a system to deliver real estate listing data to a content management system. The CMS was not in my control. Since the listing data source was bursty and I wasn’t sure how the CMS would handle the load, I decided to use a message queue, where the messages would have a JSON payload. Message queues are great at decoupling components of a system.
For the queue, I used IronMQ. The company already was using it, it has a free tier (up to 24 messages a second), the service has been stable and reliable, it has great language SDKs, and setting up a durable message queue is something I’d rather outsource. (I do wish Zapier supported it.) In other situations (when posting messages from mobile apps), we ran Varnish in front of IronMQ so that it could be replaced easily. In this case, we didn’t because there were fewer moving pieces (it was server to server communication and it would be easier to swap out IronMQ should that be required).
I wrote the bridge code from the listing database to the message queue in python. The shop was mostly Java and some python, and python seemed a better fit for a small ‘pull from here, push to there’ application. I used pytest for unit testing, jenkins to run the unit tests in a CI environment, and autopep8 for formatting. My colleague was a more experienced python programmer, so I was able to lean on him for questions. I didn’t find python hard to pick back up (I’d scripted in python a little years ago), and it was a fun language to code in. Reminded me of perl w/r/t packages and quick developer feedback. I did miss Java’s dependency management, though (my college recommended virtualenv as a possible solution).
The JSON payload would allow developers writing the message consumer to use almost any language they wanted–any language if they used the IronMQ REST API rather than an SDK.
I can’t share the code, but for this kind of problem, python was a great solution. And I’ll reach for IronMQ any time I need a message queue. This pair of technologies was quick to implement, easy to deploy, and high performance wasn’t really a requirement, since the frequency of the listing delivery was the real bottleneck.
WP-Inject Rocks
Inspired by Drew Meyers over at the Geek Estate Blog, I’ve made an effort to start putting images into my blog posts. I manually searched Flickr for commercially licenseable images, upload them to my server and put them into my post. After I did this a couple of times, I thought–surely there’s a plugin for this. And if not, I should write one!
A quick search turned up WP-Inject. I installed it and never looked back. It takes care of searching Flickr (and another site that I’ve never heard of called Pixabay). It takes care of the attribution link. It uploads the file to my server. It puts the image into my blog post. Well worth the install if you want to add any images to your posts at all.
My only wish is that it handled captions a bit better, but this could be a config option I’m overlooking. And that’s a small flaw for such an awesome plugin!
Building an automated postcard mailing system with Lob and Zapier
I was looking at automated paper mailing systems recently (and listed what I found), and was especially impressed with Lob, especially the ease of its API.
Among other printing services, Lob will let you mail a postcard with a custom PDF on both sides, or a custom PDF on one side and a text message on the other, anywhere in the USA for $0.94. (Sorry, not sure about international postcards) The company for which I work sends out tens of thousands of postcards every quarter. The vendor which we use charges them a similar fee (less, but in the same ballpark) but there’s a manual process to deliver the collateral and no API. So an on-demand, one by one post card sending system is very interesting to me.
Note that I haven’t received the Lob postcard which I sent myself, so I can’t speak to quality. Yet.
The Lob API is a bit weird, because the request is form encoded rather than a JSON payload. It also uses basic auth, but only the username, not the password. But the API seems to have all the pieces you’d need to generate all kinds of postcards–reminder postcards, direct mail postcards, photo postcards, etc.
After testing out the service via the web interface and cURL examples, I thought that it’d be fun to build a Zapier zap. In particular, being able to send a postcard for an entry in a Google spreadsheet seemed like a useful use case. Plus, Zapier is awesome, and I’d wanted to test out their integration environment for myself.
So, I built a Zapier integration for Lob, using the Zapier developer docs in combination with the Lob developer docs. It was actually easy. The most complicated step was translating the Zapier action data, which is a one or two dimensional array of typed data, into the Lob data format, which wanted a couple of text fields and two address arrays. Zapier has a scripting environment that let me modify data from APIs pre and post send, and even had an example about form encoded APIs. Zapier’s JavaScript scripting development environment was full featured, including syntax and error highlighting. It had no real debugging available, but I could use the venerable debug-by-log-statement method fairly easily.
Where could I take this next? Everywhere people use postcards in real life. The postcards depend on PDF files (see a sample), so if you are generating a custom postcard for each interaction things become more complex, but there are a few APIs (based on a 30 second google search, here and here) available for dynamic PDF generation. There are also limits on API call throughput, if I stuck to the Zapier integration–I could send at most 300 postcards a day, unless I managed multiple spreadsheets.
I see reminders of high value events (dentist, house maintenance, etc), contests and marketing as key opportunities for this type of service. And if I had a product where direct mail was a key component, using Lob directly would be worth serious consideration.
Regarding the Zap, I believe I cannot make this Zap available to anyone else. Since I’m not a representative of Lob, I couldn’t commit to maintaining this Zap, and Zapier doesn’t want to have any of their customers depending on an integration that could disappear or be unsupported at any time–a fair position.
If the Zapier or Lob folks want to take this integration and run with it, I’d be happy to share my code–just leave a comment. If anyone else is interested in being able to generate Lob postcards from a Google spreadsheet (or any other compatible API) via Zapier integration, let me know and I’ll see what I can do.
Zapier is awesome!
APIs are sprouting up everywhere. This is great for developers (and for end users) because it allows all kinds of automation. However, there are times when the investment of writing code to connect two APIs is too much.
Why might writing code be too much?
- The problem is still fluid and writing code will lock in a solution.
- There are more pressing business problems to solve.
- There are no engineering resources available, and/or no money to hire a dev.
If any of these reasons apply to a problem you are facing, consider Zapier (or a competitor like IFTTT). These services are much like Excel macros–they require less software engineering expertise but can leverage some of the power of programming to automate away work. I’m only going to write about Zapier, since that is the solution with which I am familiar.
Zapier runs the connections between each service (called ‘Zaps’) at regular intervals. Zaps are built using a web only interface that leverages the APIs in a manner that, while not completely intuitive, is thoroughly manageable by anyone who can sum up a column in Excel.
Here’s the class of problems for which Zapier is good:
- Connecting two services for which Zapier has connectors–this list is quite extensive.
- Syncing needs to happen no more than every 5 minutes (15 for the free account).
- No processing of the data during the transfer is needed (except possibly omitting some fields)–you are simply moving data from one place to another–no Yahoo! Pipes like transformations are possible in transit.
- One way sync is OK (though there are workarounds).
- You don’t need to bulk load initial data via Zapier–you either can disregard initial data or load it in some other fashion.
- You have a reasonable number of sources and sinks–each linked source and sink will take up one Zap.
- You have to have valid accounts with each source and sink.
That’s a fair number of limitations, but even so there are a large number of common problems that are solvable by Zapier. Some examples:
- Syncing a list of contacts from one source to another.
- Taking a google form submission and adding a user to a mailing list.
- Moving a row from one Google spreadsheet to another.
- Taking an email and adding it to a database.
- Adding a customer who you have just invoiced in QuickBooks to an email list.
These are all types of problems that can be done manually, but if frequency or scale increases, the process can run people ragged.
I want to call special attention to the email processing ability of Zapier. If you have a well-formatted email that you often receive that you want to further process, Zapier can parse it into interesting fields and send that data along to a sink like a Google Spreadsheet. Examples of well-formatted emails include order confirmations, contact us forms, and newsletter subscriptions.
I have found the Zapier support folks very responsive, whether that was troubleshooting an issue with Google docs, finding out how to pronounce the company name, or explaining why having 100 Zaps reading from one Google spreadsheet was a bad idea. Having responsive support staff reassures me, because once Zapier gets embedded into business processes, ripping it out is going to be very painful and a lot of work.
You can also write your own Zaps if you have custom APIs that you’d like to integrate with. I haven’t explored this much–it does seem like a developer centric task.
Zapier is not an all purpose tool nor a total replacement for developers, but it is definitely a great app to have in your toolbox. Take a look and see what little (or big) niggling problem that you haven’t had time to write code for (or, if you can’t write code, what you haven’t been able to get an engineer to write code for) it might solve.
Follow me on Twitter
Follow me on Blue Sky