Skip to content

Making my Twitter feed richer with Zapier and hnrss

twitter photo
Photo by marek.sotak

I read Hacker News, a site for startups and technologies, and occasionally post as well.  A few months back, I realized that the items that I post to HN, I want to tweet as well.  While I could have whipped something up with the HN RSS feed and the Twitter API (would probably be easier than Twitversation), I decided to try to use Zapier (which I’ve loved for a while).  It was dead simple to set up a Zap reading from my HN RSS feed and posting to my Twitter feed.  Probably about 10 minutes of time, and now I doubled my posts to Twitter.

Of course, this misses out on one of the huge benefits of Twitter–the conversational nature of the app.  When my auto posts happen, I don’t have a chance to follow up, or to cc: the authors, etc.

However, the perfect is the enemy of the good, and I figured it was better to engage in Twitter haphazardly and imperfectly than not at all.

Using basic authentication and Jetty realms to protect Apache Camel REST routes

So, just finished up looking at authentication mechanisms for a REST service that runs on Camel for a client.  I was using Camel 2.15.2 and the default Jetty component, which is Jetty 8.  I was ready to use Oauth2 until I read this post which makes the very valid point that unless you have difficulty exchanging a shared secret with your users, SSL + basic auth may be enough.  Since these APIs will be used by apps that are built for the same client, and there is no role based access, it seemed like SSL _ basic auth would be good enough.

This example is different from a lot of other Camel security posts, which are all about securing routes within camel.  This example secures the external facing REST routes defined using the REST DSL. I’m pretty sure the spring security example could be modified to protect REST DSL routes as well.

So, easy enough.  I looked at the Camel Jetty security configuration docs, and locking things down with basic auth seemed straightforward.  But, they were out of date.  Here’s an updated version.  (I’ve applied for the ability to edit the Camel website and will update the docs when I can.)

First off, secure the rest configuration in your camel context:

<camelContext ...>
   <restConfiguration component="jetty" ...>
      <endpointProperty key="handlers" value="securityHandler"></endpointProperty>
   </restConfiguration>
...
</camelContext>

Also, make sure you have the camel-jetty module imported your pom.xml (or build.gradle, etc).

Now, create the security context. The below is done in spring XML, but I’m sure it can be converted to Spring annotation configuration easily. This example uses a HashLoginService, which requires a properties file. I didn’t end up using it because this was POC code, but here’s how to configure the HashLoginService.

<!-- Jetty Security handling -->
<bean id="constraint" class="org.eclipse.jetty.util.security.Constraint"> <!-- this class's package changed -->
   <property name="name" value="BASIC"/>
   <property name="roles" value="XXXrolenameXXX"/>
   <property name="authenticate" value="true"/>
</bean>

<bean id="constraintMapping" class="org.eclipse.jetty.security.ConstraintMapping">
   <property name="constraint" ref="constraint"/>
   <property name="pathSpec" value="/*"/>
</bean>

<bean id="securityHandler" class="org.eclipse.jetty.security.ConstraintSecurityHandler">
   <property name="loginService">
      <bean class="org.eclipse.jetty.security.HashLoginService" />
   </property>
   <property name="authenticator">
      <bean class="org.eclipse.jetty.security.authentication.BasicAuthenticator"/>
   </property>
   <property name="constraintMappings">
      <list>
         <ref bean="constraintMapping"/>
      </list>
   </property>
</bean>

As mentioned above, I ended up writing a POC authentication class which had hardcoded values. This post was very helpful to me, as was looking through the jetty8 code on using grepcode.

public class HardcodedLoginService implements LoginService {
  
        // matches what is in the constraint object in the spring config
        private final String[] ACCESS_ROLE = new String[] { "rolename" };
        ...

	@Override
	public UserIdentity login(String username, Object creds) {
		
        UserIdentity user = null;
        
        
        // HERE IS THE HARDCODING
		boolean validUser = "ralph".equals(username) && "s3cr3t".equals(creds);
		if (validUser) {
			Credential credential = (creds instanceof Credential)?(Credential)creds:Credential.getCredential(creds.toString());

		    Principal userPrincipal = new MappedLoginService.KnownUser(username,credential);
		    Subject subject = new Subject();
		    subject.getPrincipals().add(userPrincipal);
		    subject.getPrivateCredentials().add(creds);
		    subject.setReadOnly();
		    user=identityService.newUserIdentity(subject,userPrincipal, ACCESS_ROLE);
		    users.put(user.getUserPrincipal().getName(), true);
		}

	    return (user != null) ? user : null;
	}

	...
}

How, when you retrieve your rest routes, you need to pass your username and password or the request will fail with a 401 error: curl -v --user ralph:s3cr3t http://localhost:8080/restroute

Note that the ACCESS_ROLE variable in the login class must match the roles property of the constraint object, or you’ll get this message:

Problem accessing /restroute. Reason:
!role

You can find a working example on github (with the hardcoded login provider).

Thanks to my client Katasi for letting me publish this work.

Twitversation: how much do you converse on Twitter?

twitter photo
Photo by eldh

You know what I said a few days ago?

I’d love to have stats on this to make myself more accountable, but I wasn’t able to find an easy way to show my Twitter usage (new tweets vs replys vs retweets)–does anyone know one?

Well, I didn’t find anything and thought it’d be fun to learn some of the Twitter API, a bit of Django, Bootstrap, and how to host something on Heroku.  So, I wrote an app, Twitversation, which gives you a rough approximation of how much you converse on Twitter, as opposed to broadcasting.  You can enter your Twitter username and it presents a breakdown graph and a numeric score (I’m 60 out of 100, whereas patio11 scores 78 and Gary V scores a hefty 83.

Twitversation only pulls the last 200 tweets, so it’s not canonical, but it should be enough to give you a flavor.  Sarah Allen has a post up about her score.

What’d I learn?  Among other things:

  • Heroku is super easy to get started on. And it’s free!  Perfect for your MVP.
  • Django has an unfortunate term for the C in the MVC (they call it a view).
  • You can create a pie graph using only CSS and HTML.
  • Side projects take longer than you think.
  • Picking a side project that doesn’t require any feeding is liberating.  Twitversation will keep running without any attention on my part, as opposed to my other side project.
  • Python’s dependency management is a bear for a newbie.  I didn’t have to do much with this project, because it had its own vagrant vm, but I saw some of the complexity out of the corner of my eye.  Makes me long for the JVM and classpaths, and I never thought I’d say that.
  • Catchy names are hard to come up with.

Hope you enjoy!

Don’t forget to deactivate inactive items in easyrec

delete photo
Photo by Ervins Strauhmanis

I wrote before about easyrec, a recommendation system with an easy to integrate javascript API, but just recently realized that I was still showing inactive items as ‘recommended’.  This is because I was marking items inactive in my database, but not in my easyrec system.

Luckily, there’s an API call to mark items inactive.  You could of course manually login and mark them as inactive, but using the API and a bit of SQL lets me run this check for all the items.  Right now I’m just doing this manually, but will probably put it in a cron job to make sure all inactive items are marked so in easyrec.

Here’s the SQL (escaped so it doesn’t wrap):

select concat('wget "http://hostname/api/1.0/json/setitemactive?apikey=apikey&tenantid=tenantid&\
active=false&itemtype=ITEM&itemid=',id,'"; sleep 5;')
from itemtable where enabled = 0;

I have an item table that looks like this:

CREATE TABLE `itemtable` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
...
`enabled` tinyint(1) NOT NULL DEFAULT '1',
...

Where enabled is set to 0 for disabled items and 1 for enabled items. The id column is numeric and also happens to be the easyrec item id number, in a happy coincidence.

The end output is a series of wget and sleep commands that I can run in the shell. I added in the sleep commands because I’m on the demo host of easyrec and didn’t want to overwhelm their server with updates.

 

Dropwizard vs Spring Boot

wizard photo
Photo by seanmcgrath

I just rolled off a project where I chose to use Spring Boot to create a number of microservices.  I have also written a number of Dropwizard services, and wanted to compare the two while they were fresh in my mind.

They have a number of similarities, of course.  Both Spring Boot and Dropwizard create standalone jarfiles that can be deployed without needing a container.  Both favor convention over configuration and want to help you do the right thing out of the box.  Both are java based.  Both have monitoring, health checks, logging and other production nicitiies buit-in.  Both are opinionated–making a lot of choices for the developer, rather than forcing the developer to choose.  Both make it easy to leverage existing libraries.  Both have a focus on performance.

However, there were a number of reasons I choose Spring Boot over Dropwizard for the recent project, and these highlight the differences.  The first is that dependency injection is built into Spring Boot in a way that it simply isn’t with Dropwizard.  Of course, there are third party solutions for bolting DI onto Dropwizard, but we also needed a java DI framework that would handle lifecycle events, which pretty much means Spring.  Finally, this project wasn’t all about REST services, and while Dropwizard has some support for other types of services, it really is designed as a performant HTTP/REST layer, and certainly almost all the questions about Dropwizard online are about REST and APIs.  Spring Boot, on the other hand, aims to provide support for a plethora of different types of services.  Plus, you can leverage the rest of the large Spring codebase.

There are some other differences as well.  Dropwizard uses shading to build fat jars while Spring Boot uses nested jars.  As far as support, Spring, as usual, wins on the documentation front, with loads of accurate docs.  But Dropwizard definitely has a larger community around it (compare the activity of the DW google group to the Spring Boot forums).

If you are writing a REST API in Java, Dropwizard is a great choice (here’s a review of other options I did a few months ago).  If you want to build microservices that integrate with other types of components (queues, nosql, etc), Spring Boot is what I’d recommend.

Update 12/8: Per this tweet, the spring forums aren’t used because of spam, but you can find plenty of support on StackOverflow with questions tagged ‘spring-boot’.

Lob Postcard Review

A few months ago, I wrote a Zapier app to integrate with the Lob postcard API. I actually spent the 94 cents to get a postcard delivered to me (I paid 24 cents too much, as Lob has now dropped their price). The text of the postcard doesn’t really matter, but it was an idea I had to offer a SaaS that would verify someone lived where they said they lived, using postal mail. Here are the front and back of the postcard (address is blacked out).

Lob Postcard Sample, address side
Lob sample postcard, address side
Lob Postcard Sample, front
Lob sample postcard, front (from PDF)

Here is the PDF that Lob generated from both a PDF file I generated for the front (the QR code was created using this site) and a text message for the back.

A few observations about the postcard.

  • The card is matte and feels solid.
  • The QR code is smudged, but still works.
  • The text message on the back appears a bit closer to the edge on the actual postcard than it does on the PDF image.
  • The front of the postcard appears exactly as it was on the PDF.
  • It took about 5 business days (sorry, working from memory) for delivery.

So, if I were going to use Lob for production, I would send a few more test mailings and make sure that the smudge was a one off and not a systemic issue. I would definitely generate PDFs for both the front and back sides–the control you have is worth the hassle. Luckily, there are many ways to generate a PDF nowadays (including, per Atwood’s Law, javascript). I also would not use it for time sensitive notifications. To be fair, any postal mail has this limitation. For such notifications, services like Twilio or email are better fits.

In the months since I discovered Lob, I’ve been looking for a standalone business case. However, business needs that are:

  • high value enough to spend significant per notification money and
  • slow enough to make sending mail a viable alternative to texting or emailing and
  • split apart from a larger service (like dentist appointment scheduling)

seem pretty few and far between. You can see a short discussion I kicked off on hackernews.  However, they’ve raised plenty of money, so they don’t appear to be going anywhere soon.

But the non-standalone business cases for direct postcard mail are numerous (just look in your mailbox).

Java REST API Framework Options

resting photo
Photo by shioshvili

I’ve been working with a couple of REST API solutions that exist in the Java tech stack.  I haven’t seen any great analysis of REST API solutions (though Matt Raible does mention some in this exhaustive slide deck about Java frameworks [pdf]), so wanted to share my on the ground experience.

First up is restSQL.  This framework makes it easy to get data from a database to a JSON or XML REST API and back.  If you have a servlet container available, you write two configuration files, one with a SQL query and one with db connection information, and you have a RESTful API.  For prototyping and database access, it is hard to beat.

Pros:

  • Quick to set up
  • Only SQL knowledge is required
  • No programming required
  • Allows simple mapping of db table to resource, but can include one to one and one to many mappings
  • Supports all four REST operations out of the box
  • Supports XML as well as JSON
  • Is an embeddable java library as well as a standalone framework
  • Project maintainer is engaged and the project is moving forward

Cons:

  • Requires a servlet engine, and you have to restart it for changes to your configuration to be picked up
  • Output format has limited customization
  • Only works with mysql and postgresql databases (though there is some experimental support for Oracle and MS SQL)
  • Doesn’t work with views
  • The security model, while fine grained, isn’t modern/OAuth (can be solved with an API gateway (like 3scale, Tyk or ApiAxle) or proxy

The next framework I have experience with is Dropwizard.  This is a powerful framework that creates uberjars that you can run on any port as a standalone service.  It’s not limited to providing a JSON representation of database tables–if you can create a Java object, Dropwizard can serve it up as a JSON resource.

Pros:

  • Community support
  • Extreme output formatting flexibility, but be prepared to write a custom deserializer if you want to handle anything other than reads of custom formatted objects
  • Supports any database that hibernate supports
  • Built in testing support
  • Brings together ‘best of breed’ tools like Jersey, Jackson and Hibernate, so you don’t have to do the integration yourself
  • Great documentation

Cons:

  • Have to roll your own deployment solution (tarball, chef, puppet)
  • No services startup script provided
  • Shading can slow down development
  • Not yet at 1.0 release

The last one I don’t have familiarity with, but a colleague used it in the past.  It is Sparkjava.  This is a lightweight framework that fits when you have an existing Java library with functionality you want to expose.  I’m not competent to write pros/cons for this framework, but wanted to mention it.

The gorilla in the room that I haven’t had experience with (in terms of writing RESTful webs services) is Spring.  I would definitely include this in any greenfield solutions review.

I Want to Pay You Money! (Except When I Don’t)

money photo
Photo by 401(K) 2013

I saw this post from Kin Lane talking about Zapier and how one of the many advantages it has over similar services is its pricing.  I completely agree.  While I like free as much as the next person, when I’m building on something, I want to pay for it, or at least have it monetized in some fashion (Kin has a nice list of ways for API providers to monetize).  Paying for a service means:

  • the company can survive
  • great employees can be paid
  • when I complain, the company has an incentive to listen
  • the value I get from the service is above what I’m paying (aka consumer surplus), if I’m a reasonable facsimile of homo economicus.

All of these are really nice attributes of a technology I’m going to build on (not just ‘date’ as Kin says).  This is an interesting dichotomy, because the fastest way to growth is to provide a free service–then there’s no friction to signing up.

I guess the answer, at least for software products where the marginal cost is very very low, is a freemium offering, like Zapier.  Get the user in, show them how your value proposition works, and then ask them for money when they are hooked.  Just don’t make the freemium level unusable!

Consolidate external dependency notifications using Zapier

binoculars photo
Photo by M1key.me

As I wrote over at the Geek Estate Blog, if you build your business on vendors, you should monitor them.  In the past, I’ve used a variety of services to monitor vendor services, from pingdom to wget/cron to nagios.  These services are great about telling you when some external service is unavailable, but are not so hot at telling you when a service is going to be down (for planned maintenance) or back up.

For that, you need to be monitoring, and reading, vendor announcements, however the vendor has decided to provide them, whether that is as a blog/RSS feed, twitter feed, email newsletter, a status page or something else.

However, it can be tough to monitor and read announcements in two or more places.  Here, Zapier or a similar service can help.  Pick one place to be notified.  For me, that’s typically an email inbox, because, frankly, other data sources can be ignored (except phone texts), but I’ll always check my email.

Then, use Zapier’s zaps to transform any announcements from the other sources to emails.  For instance, there is an RSS trigger for new items in a feed and a Twitter trigger for tweets from a user.  Status pages often provide RSS feeds (Google’s does).  If the service provider doesn’t provide a structured method like an RSS feed to notify you of changes, but does provide a webpage of announcements, you could look at a service like changedetection.com and have the email sent to your inbox or parsed by Zapier and pushed to your notification location.

And for the output side, you can just use Zapier’s ‘send outbound email’ action.  If you want to have all notifications pushed to your phone, an RSS reader or Twitter acount, you can use Zapier to send texts, create RSS items or tweets as well.