Amazon Alexa

I had a lot of fun working on a one day ‘hackfest’ project with Amazon Alexa. I learned a lot about voice UX and Alexa implementation details.It’s an interesting platform, especially if you have broad brand recognition and can deliver high level valuable information via short chunks of text.

From my blog post on the Culture Foundry site:

The multi step interaction is a bit clunky, but I think it’s a great way to avoid collisions between different skills. Basically, the user calls out an ‘invocation’ like ‘open color picker’. Interactions with Alexa after that are send directly to that particular skill until an end point is reached in the interaction tree. Each of these interactions is triggered by a different voice command, and is handled by something called an ‘intent’. Intents can have multiple triggering commands (‘what is my favorite color’ vs ‘what is my color’, for example). There’s also a lightweight, session level storage while the entire invocation is occurring, which means you can easily pass data between intents without reaching out to a more persistent data storage.

You can read the whole post over there.


Useful gem: stripe_event

If you are going to use stripe for payments, you need to set up your webhooks. If you are using rails, the easiest solution I’ve found is stripe_event. This gem mounts a configurable endpoint and takes care of all the authentication you need to receive the webhooks.You then set up configuration in an initializer to receive the various webhooks you want to receive. The type of hooks you want depends on your application, but all the available events are listed here and the stripe support folks are happy to point you toward interesting ones if you approach them with a problem.

You can (and should) test the stripe events by using fixtures and request tests. I found the most difficult part of that testing process to be getting sample data for the json payload. The documentation has some, but you may need to run a sample event through your test dashboard and capture the json via a generic webhook capture. I ended up using this type of puts debugging to help get the json for events:

events.all do |event|
  ## debugging
  puts "xxxdebugging all events"
  puts event.to_s
end

In my experience, we never received enough load to really stress out this gem (I’ve seen maybe 30 requests a minute), but if you plan to have a high webhook load, you may want to do some load testing.

Definitely a gem worth having if you are using stripe.


Using WordPress as a CRUD Database, API Included

Ethernet cordBased on this HN discussion, which I discussed a while back, I looked at how to set up WP as a CRUD database accessible via API.

It wasn’t hard. Steps:

  1. Install WordPress (I used ec2 and the Cloudformation sample template)
  2. Install the following plugins
  3. I also installed the following optional plugins
  4. I created a custom post type of ‘todo’ and added a couple of custom fields.
  5. I was able to get the todos by going to these URLs (apparently you can have the API live at wp-json, but that required some rejiggering of url permalinks).
    • http://host/wordpress/?rest_route=/wp/v2/todo/8
    • http://host/wordpress/?rest_route=/wp/v2/todos

Here’s an example of the output:

{
  "id": 8,
  "date": "2018-03-05T02:38:26",
  "date_gmt": "2018-03-05T02:38:26",
  "guid": {
    "rendered": "http://host/wordpress/?post_type=todo&p=8"
  },
  "modified": "2018-03-05T02:40:01",
  "modified_gmt": "2018-03-05T02:40:01",
  "slug": "auto-draft",
  "status": "publish",
  "type": "todo",
  "link": "http://host/wordpress/todo/auto-draft/",
  "title": {
    "rendered": "Buy Milk"
  },
  "template": "",
  "acf": {
    "": false,
    "due_date": "20180308",
    "description": "please buy milk.",
    "who_owns_it": {
      "ID": "1",
      "user_firstname": "",
      "user_lastname": "",
      "nickname": "mooreds",
      "user_nicename": "mooreds",
      "display_name": "mooreds",
      "user_email": "...",
      "user_url": "",
      "user_registered": "2018-03-05 02:21:36",
      "user_description": "",
      "user_avatar": "..."
    },
    "done": false
  },
  "_links": {
    "self": [
      {
        "href": "http://host/wordpress/wp-json/wp/v2/todo/8"
      }
    ],
    "collection": [
      {
        "href": "http://host/wordpress/wp-json/wp/v2/todo"
      }
    ],
    "about": [
      {
        "href": "http://host/wordpress/wp-json/wp/v2/types/todo"
      }
    ],
    "wp:attachment": [
      {
        "href": "http://host/wordpress/wp-json/wp/v2/media?parent=8"
      }
    ],
    "curies": [
      {
        "name": "wp",
        "href": "https://api.w.org/{rel}",
        "templated": true
      }
    ]
  }
}

The custom post fields are all under the ACF key, and you can see that there was an expansion of the who_owns_it field. If you are going to do this, make sure have the the normal title tag be part of the custom post, otherwise the WP UX for editing the custom posts won’t be much use.

Not perfectly restful, but a super simple way to set up an API that non technical folks can use to create, update or delete records and that you can consume in other systems.


Swagger looks pretty good from here

SwaggerA few years ago I was working on an API that my client was going to make available to some of their clients. I used Swagger, which I’d heard about from a Gluecon presentation.

I was unimpressed. I recall having difficulty getting the online tool to work and the documentation generated was poor. This could very well have been user error or my misunderstanding of the sweet spot of the tool, but for whatever reason it wasn’t a fit for the problem.

Fast forward a few years and I was talking to a company about a position. They were planning to use Swagger to generate their API SDKs. They had an API which was crucial to their business and were currently supporting a large number of SDKs on a bespoke basis. I reviewed the documentation of Swagger and downloaded the 2.3 version. I was very quickly able to generate a number of client and server stubs using the codegen project. They have a long list of supported languages, but I quickly generated ruby, csharp and perl client bindings and a simple rails5 and spring server. I didn’t push these through to production and I’m sure that if I had I’d have learned about the rough edges (it’s always nice to check out Stack Overflow for rough edges for any technology–here’s questions for swagger). Regardless, the ability to take a simple JSON configuration file and create API front ends and backends in minutes was quite impressive.

Note that the codegen tools are all in java, so if leveraging that technology gives you the willies, you might want to look around for another solution. Note also that Swagger has now been moved to the OpenAPI project (as of version 2.0 of Swagger). If you want to know more about that, here’s the blog post announcing that move. Here’s a look at features of the next version.

If you are developing an API first company (and there are good reasons to be that), I’d recommend taking a long hard look at Swagger. The speed of adding SDK support as well as the community around this tooling look to be huge advantages.


Pact Testing

PadlockI attended the Google Developer Group meetup last week and enjoyed many of the talks. It was a lightning session, so there were ten speakers. In particular I really enjoyed “Pact Contract Testing” by Claire Chen. The idea behind Pact Testing, which has been around since 2013 and has had four major specification releases, is to formalize the contract between an API consumer and producer and allow each side of the API conversation to be developed independently. You can record the interactions between each consumer and producer and re-play them during testing to verify that no regressions have occurred. It’s really designed for a situation where you control both the consumer and the producer and want to verify that there are no breaking changes when either of them evolve.

So, this seems like mocks and stubs on steroids with the additional benefit of being cross platform (many languages are supported) and exercising the entire producer or consumer independently. You can also run an external server to maintain all the pacts independently.

If you are running a microservices architecture, I’d strongly recommend taking a look at this. Next time I’m involved in an API consumer/producer project, I’ll definitely be using this, and will report back then.

See also “convince me that Pact Testing is a good idea” and “what is Pact not good for?”.


Software infrastructure configuration options

I ran across this great article when I was reading up on Terraform.

It does a good job of running through the options (puppet, cloudformation, etc) on how to set up your infrastructure via software. Here’s a great quote on why they chose Terraform:

On the other hand, with the kind of declarative approach used in Terraform, the code always represents the latest state of your infrastructure. At a glance, you can tell what’s currently deployed and how it’s configured, without having to worry about history or timing. This also makes it easy to create reusable code, as you don’t have to manually account for the current state of the world. Instead, you just focus on describing your desired state, and Terraform figures out how to get from one state to the other automatically.


Things I wish I knew about Stripe

Caterpillar

Striped, but not charging your credit card

So, at The Food Corridor, we’ve been using Stripe happily since we launched in June of 2016.  As a developer, I’d used Stripe before in a couple of different ways, but this has definitely been my most sustained use of the payment service.  (If you don’t know what Stripe is, it is an API that makes charging customers as easy as an API call.  More here.)

I wanted to outline some of the things I’ve learned from months of using Stripe.

  • Stripe supports pulling money directly from bank accounts, via ACH, but it really isn’t the same ACH as your bank lets you do.  This is because Stripe isn’t a bank.  The biggest thing to be aware of here is that Stripe ACH takes 7 days to arrive in your bank account.  Another issue is that you have to do verification.  They have two ways of doing verification–micro deposits and Plaid.  Plaid is instant, but only supports major banks, which was a non starter for us (updated 9/8: Plaid supports around 1000 banks now).  The code for micro deposits is straightforward, but be prepared for some customer support issues.  Stripe deposits two amounts and withdraws just one amount, which was confusing for some of our users.  It also takes a couple of days, so if your users are hot to spend money, Stripe ACH may not be a fit.  The win?  Definitely cheaper.  (And I didn’t find any other service that would support both credit card and ACH transactions that was developer friendly.)
  • Don’t forget to set up your webhooks out of the gate.  Stripe mentions this, but I glossed over it in the early days, and missed some events that were important.  (The most relevant is that ACH is asynchronous, so when an ACH transfer fails, it is reported via webhook.  If bank account verification doesn’t work, you’ll get a different kind of webhook.  Review the docs and set up webhooks for all the ACH events.)  If you don’t have time for a full featured webhook processing implementation, Zapier can just send the webhook data to your email. This can be a great stopgap solution.  Or you can use stripe_event.
  • Per support, if a webhook post fails (because your app is down, for example), they are retried once an hour for 72 hours.
  • Speaking of stopgap solutions, the Stripe Dashboard is fantastic for manual processes.  Just because you can automate everything via an API, doesn’t mean you should.  There can be some complicated edge cases with payment processing, especially around refunds, but they can easily be handled with a google doc of instructions and the Stripe Dashboard.  I have found only one use case that the API can handle that the dashboard cannot (a partial refund of an ACH transaction).
  • I have found Stripe support to be excellent, quick and knowledgeable.
  • Occasionally customer charges will be declined because of bank fraud triggers.  Expect to occasionally ask your customers to call their bank.  (I think this has happend about once every third month).
  • Disputes are a total pain, because the process is opaque and slow (expect a resolution in about two months and know you are not in possession of the payment during that time).
  • Make sure to capture the payment id anytime you charge a card or run ACH.  It will make future automation a lot easier.
  • Monthly plans are complicated, so if you can lean on Stripe for management, even if you are doing manual plan management (applying coupons, adding, or removing users from plans via the dashboard), do that.
  • The first payment you charge takes 7 days to move from stripe to your​ bank account.  This is for fraud protection.  Payments thereafter typically take 2 days (but it depends on your country and industry).

And here are some special tips if you are using Stripe Connect (their marketplace product).

  • Read the docs!
  • Remember that first payment timeline?  It applies to every one of the connected accounts.  Think about charging your own credit card as soon as you connect an account to help with customer cash flow.
  • Consider whether you want to use managed vs standalone accounts.  Managed accounts are a lot more work but allow you to have a seamless UX that you control.  Standalone accounts, which we use, are far quicker to setup.  I think this depends on the number of sellers you have in your marketplace.
  • You also want to think about whether to place the charges on the platform account or on the connected accounts.  A major factor there is who bears the Stripe fees, the platform or the sellers.  We charged on the platform account because we wanted all our data in one place.  If you are selling plans, you can’t charge on the platform and use Stripe plans.
  • If you are charging on the platform account, and are using standalone accounts (where the sellers have to set up a stripe account) your sellers won’t see charge descriptions unless you manually copy the description over.  The code looks like:

# this will let the sellers know what invoice the charge was for
transfer_id = charge.transfer
transfer = Stripe::Transfer.retrieve(transfer_id, expand: ['destination_
payment'])
payment_id = transfer.destination_payment
payment = Stripe::Charge.retrieve(payment_id, {stripe_account: destinati
on_account_id})
payment.description = description
payment.save

Happy charging!


Making my Twitter feed richer with Zapier and hnrss

twitter photo

Photo by marek.sotak

I read Hacker News, a site for startups and technologies, and occasionally post as well.  A few months back, I realized that the items that I post to HN, I want to tweet as well.  While I could have whipped something up with the HN RSS feed and the Twitter API (would probably be easier than Twitversation), I decided to try to use Zapier (which I’ve loved for a while).  It was dead simple to set up a Zap reading from my HN RSS feed and posting to my Twitter feed.  Probably about 10 minutes of time, and now I doubled my posts to Twitter.

Of course, this misses out on one of the huge benefits of Twitter–the conversational nature of the app.  When my auto posts happen, I don’t have a chance to follow up, or to cc: the authors, etc.

However, the perfect is the enemy of the good, and I figured it was better to engage in Twitter haphazardly and imperfectly than not at all.


Using basic authentication and Jetty realms to protect Apache Camel REST routes

So, just finished up looking at authentication mechanisms for a REST service that runs on Camel for a client.  I was using Camel 2.15.2 and the default Jetty component, which is Jetty 8.  I was ready to use Oauth2 until I read this post which makes the very valid point that unless you have difficulty exchanging a shared secret with your users, SSL + basic auth may be enough.  Since these APIs will be used by apps that are built for the same client, and there is no role based access, it seemed like SSL _ basic auth would be good enough.

This example is different from a lot of other Camel security posts, which are all about securing routes within camel.  This example secures the external facing REST routes defined using the REST DSL. I’m pretty sure the spring security example could be modified to protect REST DSL routes as well.

So, easy enough.  I looked at the Camel Jetty security configuration docs, and locking things down with basic auth seemed straightforward.  But, they were out of date.  Here’s an updated version.  (I’ve applied for the ability to edit the Camel website and will update the docs when I can.)

First off, secure the rest configuration in your camel context:

<camelContext ...>
   <restConfiguration component="jetty" ...>
      <endpointProperty key="handlers" value="securityHandler"></endpointProperty>
   </restConfiguration>
...
</camelContext>

Also, make sure you have the camel-jetty module imported your pom.xml (or build.gradle, etc).

Now, create the security context. The below is done in spring XML, but I’m sure it can be converted to Spring annotation configuration easily. This example uses a HashLoginService, which requires a properties file. I didn’t end up using it because this was POC code, but here’s how to configure the HashLoginService.

<!-- Jetty Security handling -->
<bean id="constraint" class="org.eclipse.jetty.util.security.Constraint"> <!-- this class's package changed -->
   <property name="name" value="BASIC"/>
   <property name="roles" value="XXXrolenameXXX"/>
   <property name="authenticate" value="true"/>
</bean>

<bean id="constraintMapping" class="org.eclipse.jetty.security.ConstraintMapping">
   <property name="constraint" ref="constraint"/>
   <property name="pathSpec" value="/*"/>
</bean>

<bean id="securityHandler" class="org.eclipse.jetty.security.ConstraintSecurityHandler">
   <property name="loginService">
      <bean class="org.eclipse.jetty.security.HashLoginService" />
   </property>
   <property name="authenticator">
      <bean class="org.eclipse.jetty.security.authentication.BasicAuthenticator"/>
   </property>
   <property name="constraintMappings">
      <list>
         <ref bean="constraintMapping"/>
      </list>
   </property>
</bean>

As mentioned above, I ended up writing a POC authentication class which had hardcoded values. This post was very helpful to me, as was looking through the jetty8 code on using grepcode.

public class HardcodedLoginService implements LoginService {
  
        // matches what is in the constraint object in the spring config
        private final String[] ACCESS_ROLE = new String[] { "rolename" };
        ...

	@Override
	public UserIdentity login(String username, Object creds) {
		
        UserIdentity user = null;
        
        
        // HERE IS THE HARDCODING
		boolean validUser = "ralph".equals(username) && "s3cr3t".equals(creds);
		if (validUser) {
			Credential credential = (creds instanceof Credential)?(Credential)creds:Credential.getCredential(creds.toString());

		    Principal userPrincipal = new MappedLoginService.KnownUser(username,credential);
		    Subject subject = new Subject();
		    subject.getPrincipals().add(userPrincipal);
		    subject.getPrivateCredentials().add(creds);
		    subject.setReadOnly();
		    user=identityService.newUserIdentity(subject,userPrincipal, ACCESS_ROLE);
		    users.put(user.getUserPrincipal().getName(), true);
		}

	    return (user != null) ? user : null;
	}

	...
}

How, when you retrieve your rest routes, you need to pass your username and password or the request will fail with a 401 error: curl -v --user ralph:s3cr3t http://localhost:8080/restroute

Note that the ACCESS_ROLE variable in the login class must match the roles property of the constraint object, or you’ll get this message:

Problem accessing /restroute. Reason:
!role

You can find a working example on github (with the hardcoded login provider).

Thanks to my client Katasi for letting me publish this work.


Twitversation: how much do you converse on Twitter?

twitter photo

Photo by eldh

You know what I said a few days ago?

I’d love to have stats on this to make myself more accountable, but I wasn’t able to find an easy way to show my Twitter usage (new tweets vs replys vs retweets)–does anyone know one?

Well, I didn’t find anything and thought it’d be fun to learn some of the Twitter API, a bit of Django, Bootstrap, and how to host something on Heroku.  So, I wrote an app, Twitversation, which gives you a rough approximation of how much you converse on Twitter, as opposed to broadcasting.  You can enter your Twitter username and it presents a breakdown graph and a numeric score (I’m 60 out of 100, whereas patio11 scores 78 and Gary V scores a hefty 83.

Twitversation only pulls the last 200 tweets, so it’s not canonical, but it should be enough to give you a flavor.  Sarah Allen has a post up about her score.

What’d I learn?  Among other things:

  • Heroku is super easy to get started on. And it’s free!  Perfect for your MVP.
  • Django has an unfortunate term for the C in the MVC (they call it a view).
  • You can create a pie graph using only CSS and HTML.
  • Side projects take longer than you think.
  • Picking a side project that doesn’t require any feeding is liberating.  Twitversation will keep running without any attention on my part, as opposed to my other side project.
  • Python’s dependency management is a bear for a newbie.  I didn’t have to do much with this project, because it had its own vagrant vm, but I saw some of the complexity out of the corner of my eye.  Makes me long for the JVM and classpaths, and I never thought I’d say that.
  • Catchy names are hard to come up with.

Hope you enjoy!



© Moore Consulting, 2003-2017 +