Skip to content

All posts by moore - 20. page

Serverless Framework

I had coffee with an acquaintance who is doing a lot of event driven data processing. Whereas ten years ago to tackle this problem you might use an ETL tool like Pentaho or Talend, now his process runs entirely on AWS Lambda functions. He is leveraging the Serverless framework to manage and deploy these applications. As I understand it there is a thin shim layer between the business logic and the lambda event handler, but the business logic is isolated and knows nothing about its environment. That makes the business logic very testable.

His description of the Serverless framework intrigued me. As he described it, the framework is driven by a simple yaml file and takes care of, among other tasks, the complicated infrastructure set up to tie Lambda functions to a variety of AWS events. I haven’t done it myself, but I’ve heard that setting up a lambda to API Gateway link is a real bear. Doing so allows a lambda function respond to a web requests without any AWS authentication, and is a key use case.

You can write and deploy lambda functions in any language that AWS Lambda supports (unfortunately, not java 9 at the moment). Here’s a java/maven/serverless tutorial. It also supports multiple cloud providers, though I haven’t done much beyond note that the documentation exists.

However, using Serverless does require writing code. If evaluating a a complicated ETL process which non developers needed to be able to understand and support, Serverless would not be a good fit. I’m not aware of any abstraction layers on top of it, though I guess you could run, for example, Pentaho Kettle jobs within lambda. There’s also an issue around cold start times–when your code hasn’t been invoked for a while, it can take longer to start up when a request or event occurs. Apparently there are partial solutions, but your lambdas still get cycled every few hours regardless.

I worked through some of the tutorials and was impressed at just how easy it was to get started. If I had a simple API or data processing pipeline to build, Serverless would definitely be on my short list of possible implementation options. It is very inexpensive, scales easily and encourages encapsulation.

Incidentally, my acquaintance’s company is hosting a lunch and learn on this technology at the end of the month. More details here.

AWS Cloud Computing Java Testing | moore | | Leave a comment

Smashing: A Quick Dashboard Solution

I’m putting together a business metrics dashboard for The Food Corridor (what is old is new again, I remember a project at XOR, my first job out of school, that was all about creating a dashboard). I could have just thrown together some rails views, but I looked around and saw Smashing, which is a fork of Dashing, a dashboard project that came out of shopify.

Smashing is a sinatra app and is fairly simple to set up. It looks gorgeous, a lot better than anything I could hack together. I could install it on a free heroku dyno. Even though it will take a bit of time to spin up, it is now running for free. Smashing has nice MVC separation–you have dashboards which assemble widgets, and then jobs which push data to widgets on a schedule. Sending data looks something like this: send_event('val', { current: current }) where val is referenced in the widget.

You can create more than one dashboard (I did only one). They aren’t customizable by non developers, but once the widgets are written, they can be created by someone with a modicum of experience editing HTML.

Some tips:

  • Smashing stores its state in a file. If you are running on heroku, the filesystem is ephemeral. You have two options. You can store the state in an external data store like redis (patch mentioned here, I didn’t try it). Or you can rely on the systems you are polling for metrics to maintain the state. That’s the path I took.
  • The number widget has the ability to display percentage changed since last updated: send_event('val', { current: current, last: last }). Make sure that val is an integer–I sent a string like “100000” and that was treated as a zero for purposes of calculation.
  • If you are accessing any external systems, make sure you inject any secrets via environment variables.  For local development, I used dotenv.
  • You’ll want some kind of authentication system.
  • The widgets that come with Smashing aren’t complicated, but neither are they documented, so prepare to spend some time understanding what they expect.
  • I grouped jobs, which gather the data, by data source.  You can send multiple events per job, and I thought that made it clearer.  Connections to APIs or databases only needed to happen once as well.
  • The business metrics which I was displaying really only change on a monthly basis.  So I wanted to run the data gathering immediately, then in a week or two weeks.  Because of the ephemeral state, I expect the second run will never happy, but wanted to be prepared for it.  I did so by creating a function and calling it once on job load and then in the scheduler.

Here’s pseudo code for the job that pulls data from stripe:


Stripe.api_key = ENV['STRIPE_SECRET_KEY']

def stripe
  # pull data from stripe...
  send_event('stripeval', { current: current })
end

stripe

SCHEDULER.interval '1w' do
  stripe
end

Smashing is no full on technical metrics solution (like Scout or New Relic), but can be useful for displaying limited data in a beautiful format with a minimum of developmetn effort. If you’re looking for a dead simple dashboard, Smashing will work for you.

The Food Corridor Useful Tools Web Applications | moore | | Leave a comment

Getting access to the very nice date functionality in non Rails Ruby

I am doing some small ruby scripts for a dashboard and need to do some date calculations, like the timestamp of the first of the previous month and the timestamp of the end of the previous month. Rails makes this so easy with (DateTime.now - 1.month).beginning_of_month. I looked around for a way to do it with straight up Ruby, but didn’t see a good solution.

Luckily, some of the nice parts of Rails have been broken out into the active support gem. You need to add it to your Gemfile or however else you are managing your gems, of course. Confusingly, the gem is activesupport and the require statement is require active_support/... (see the underscore?).

There’s an entire guide on how to pull in just the active support functionality you need. Unfortunately, I couldn’t make the targeted includes work (I was trying to pull in both numeric and date extensions precisely, but kept getting the error message undefined method `month' for 1:Integer (NoMethodError).

Finally, I just pulled in active_support/time and everything worked.

Dynamic Languages Programming Rails | moore | | Leave a comment

Blast from the past: 5 worlds

5 Worlds is a Joel Spolsky classic. This article needs to be updated (it’s from 2002, when shrinkwrap software was still A Thing) but it still has a lot of wisdom and illustrates just how large the scope of work available to software developers is (even more so now that software is eating the world).

Whenever you read one of those books about programming methodologies written by a full time software development guru/consultant, you can rest assured that they are talking about internal, corporate software development. Not shrinkwrapped software, not embedded software, and certainly not games. Why? Because corporations are the people who hire these gurus. They’re paying the bill.

Note that assuming a software developer is a webdev is like assuming a lawyer is a trial attorney. Just like there’s lots of ways to practice law, there are lots of ways to build software. And, to be honest, this is probably true of every profession. If you go to a party and ask someone “what do you do” and really really listen, chances are you’ll get a startling view of the world, because everyone does something interesting.

Programming Technology and Society | moore | | Leave a comment

Switching Scales

Developers need the ability to switch scales. I don’t know if this is crucial for any other profession (maybe a general contractor building a home?). For developers the ability to zoom in and focus on a specific problem for a few hours, and then zoom out and focus on the bigger picture, whether that is project management and reporting, or system architecture, is crucial.

The way I do it is to compartmentalize rigorously. This means that when I am focused on a bug, I’m focused on the bug. Sometimes I’ll timebox such investigation so I don’t get wrapped around the axle of the problem. If I encounter other issues, I file them off in the issue tracker (you do have an issue tracker, don’t you)?

If, instead, I’m focusing on the big picture, paradoxically it can be harder to focus. I think best when writing, so I’ll start out with a document outlining the issue. This has the added benefit of allowing me to easily collaborate (google docs FTW), refind the problem definition and solution, and revisit it over time. Other folks work better with other artifacts (app click throughs, diagrams) but the information density and flexibility of written communication work best for me. However, either way I need to set aside some time to focus on the question at hand.

Once you compartmentalize successfully, you can start to have the each scale inform decisions made at the other. Bugs that may feel urgent to fix can be deferred because that system is not accessed often, or will be rewritten soon. System diagrams may be reworked because you know that sub components belong together (or don’t). You may know where the “dark, scary” areas of the application are and make time to rework them. There may be patterns of code or services worth extraction.

One of the hardest parts of management, from personal experience and discussions with others, is letting go of the details. As an engineering manager, you should absolutely let go of critical path implementation items–you shouldn’t be working on the hardest problem or any key components. Depending on the size and needs of your team, your focus may be on recruiting, providing political cover or knocking down impediments to team goals. But I think you need to have a touchpoint at the lowest level of your application. Whether that is achieved through non blocking code reviews, fixing non critical bugs, or knocking out a customer support request, zooming in to the implementation level will inform your worldview. Just don’t let it be too large a focus; definitely don’t let it block your team.

If the idea of letting go of product implementation doesn’t appeal to you, you may want to dive in to management more. There are plenty of details and challenges to management (read High Output Management for a great intro). In fact, there are similarities between building a successful piece of software and building a team that builds a successful piece of software. People aren’t code though–imagine if when you ran a function you got a slightly different result based on how the function felt, how its home life was, what previous work the function had done, etc, and you’ll have some idea of the complexity of management.

If you’ve tried management and it doesn’t work for you, you can remain an individual contributor. Realize this will have an impact on your compensation (in most orgs) and your leverage (link) to effect change.

But it may be worth it to remain a builder.

Business Programming | moore | | Leave a comment

A Few SQL Database Links

So, here are two different relational database links that I am currently interested in.

The first is PostgreSQL Exercises. This is a fun way to sharpen your SQL. While I’m not a big user of PostgreSQL, I have enjoyed it the few times I’ve touched it. And the basics of thinking in SQL are useful across any database.

The second is Modern SQL. I was pointed to this site (in particular this page on pivots) after asking a question on a slack (so I can’t link to it). My question was about how to transform a set of event values (X happened Y ago, then Z happened at Y+1, and so on) up into counts (X happened N times, Z happened M times). But the entire site is worth reading if you are a SQL nerd, especially the use cases.

Bonus, check out this Twitter thread about using a single SQL database as a data source for multiple microservices. Pragmatism!

Databases Tips | moore | | Leave a comment

Useful rails gem: rack mini profiler

Sometimes you need to do some profiling of your application to see where performance issues lie. I haven’t done an extensive survey of the options, but a friend recommended rack mini profiler. As the name suggests, this can be used with any rack based framework (so rails, hanami, etc).

There are a number of reasons to use this profiler. Learning to use a profiler, any profiler and how to performance tune is a key part of being a software developer, as opposed to programmer.

It can run in production. You can limit who can access the profiler, and it usually sits in the upper left hand corner with a “page load time”, but you can jump into more detail as needed.

It runs all the time. This lets you have an ambient view of the application (at least the parts that you are touching regularly).

When you are investigating a performance issue, you can easily dive into the views, controllers and SQL queries that your application is making.

It is simple to install, just a simple gem in your Gemfile.

Well worth getting to know, if you are supporting a rails application.

Rails | moore | | Leave a comment

Interview with a early stage SaaS founder

I had the chance to talk with my good friend and former colleague Corey Snipes about his SaaS project. He recently launched Meeting Star, a lightweight SaaS tool to help coordinate tech meetups. This interview has been lightly edited for clarity.

——-

Why did you come up with this product?

It began as a desire to fill my own need, for a lightweight and inexpensive place to manage small local tech events. It seemed like a fairly straightforward set of features, which wouldn’t take long to build and release as a product. (Don’t they always?) I was aware of several other tech meetup organizers who were looking for an alternative to meetup.com (often due to price, sometimes due to dislike of the feature set or UI). I did quite a bit of research last fall and didn’t find any suitable alternatives so I built it.

What do you hope to achieve with this app?

I have a few parallel interests here. I want a tool that’s useful to me as a meetup organizer. I want to leverage my experience building, marketing, and operating other software products — both my own, and for customers I’ve had over the years. I want to add a business line in my portfolio that provides value and makes people happy, while also being financially sustainable. I also run a separate, conference-related application and I anticipate some complementary lift between the two.

How much research did you do before plunging in and writing it?

Quite a bit. You can always do more, of course. I poked around online and found several lists, articles, and discussion threads about alternative platforms. I followed conversations of other meetup organizers discussing the relative merits of various methods. I made a list and tried seven or eight of what seemed like the top contender products. I was looking for a place to run my meetups, though, not specifically looking at competition. But in the end, everything was either trying to be a full-featured community management piece, or was such a terribly crafted alternative that I felt there were exactly zero real usable options for what I wanted to do.

Who is the product aimed at?

This particular [app] was born of my own needs, and I tend toward tech and entrepreneurship meetups. It’s well-suited to tech and software meetups. Those are the people in my network. Those are the meetups I attend and organize, and those are the users whose needs I can most easily identify and meet. Since it’s a reasonably lightweight application, it’s actually well-suited to many different kinds of groups, but software/tech/biz meetups are my focus.

What would make you consider this product a success?

Right now, success is getting ten paying, happy customers and getting things dialed in to their needs. I subscribe to Patrick McKenzie’s wisdom that the first ten customers are critical for turning your idea into something people want to use. And also, for proving it’s not a fluke. If you can get to ten, you can get to a hundred. And if you can get to a hundred, you can get to a thousand. For me, long-term success is a useful, sustainable product that has revenue to turn into improvements, runs smoothly, and maybe also puts a little money in my kids’ college fund.

——-

You can find out more about Corey, including why he’s moving to Cleveland, at his website.

Interview Useful Tools Web Applications | moore | | Leave a comment

Basecs: Basics of Computer Science

If you backed into software as a career like me (thanks physics degree!), you may not have a firm grounding in basic computer science concepts. This can be hard to gain during your work week, as clients care far more about feature delivery than they do about which data structure you used.

However, having an intuitive sense of how computers work and what basic algorithms and data structures are can enrich your experience as a developer. Why? For the same reason understanding how to grow peas lets you appreciate a fine meal–understanding fundamentals lets you peer down through abstractions and appreciate what you are building on top of.

There are plenty of intro articles and books out there, but I recently stumbled across basecs and have been enjoying it.

The author starts from the beginning (what is binary), covers various data structures and algorithms, and ends with a discussion of just in time compilation. Her prose is comprehensible and she adds illustrations which complement the text.

If you are a developer without fundamentals, like me, you may enjoy this year long series on the basics of computer science.

Programming | moore | | Leave a comment