Skip to content

How to solve the “this application is slow” type of problem

How do you feel when your boss says “the application is slow, please speed it up”?

Personally, my heart sinks and then I get excited. Though they can be frustrating, this kind of thorny performance issue is fun if you look at it the right way.

Whenever I’ve tackled problems like this, the first thing to do is define the start and the finish line, as precisely as you can.

This requires understanding the application’s behavior and architecture. Where is the data stored? How is it presented? How is it modified? Are there types of operations that happen regularly? What exactly is “slow” about the application?

Try to avoid jumping to conclusions here. I understand the temptation to make a change as soon as you think of one that might help, but it’s better to approach this systematically.

Suppose you find out the issue is the database. Operations are too slow and the CPU is not pegged. You know the type and version of the database, how the application calls it, and more.

Here, a good finish line might be “we need to be able to handle updating the main table with 50k items in 1 second”. If you don’t have a precise finish line, this type of work can be endless and frustrating. After all, it is almost always possible to “make it faster”, but you will reach the point of diminishing returns.

If possible, set up a test scenario/system that you can run through repeatedly as you make changes. If not, figure out some other way to test that changes have a positive impact.

Next brainstorm possible solutions and think of two numbers for each: level of effort and hoped improvement. Doesn’t need to be too precise, a scale of 1-5 is fine. Here’s an example for a database bottleneck:

  • upgrade the size of the database. LOE low, impact medium
  • increase the disk speed: LOW low, impact high
  • running explain plans and adding suggested indices: LOE medium, impact high
  • offloading operations to read replicas: LOE high, impact high
  • …etc, etc

Then start doing low effort, high impact changes. Run through your scenario and tests after each one. See if you get closer to the finish line. Rinse and repeat.

This type of performance issue is a case where hiring an outside consultant/contractor can make sense. You don’t have to spend a lot, since the scope of work can be limited. They can work with you to define the start and finish lines as well as possible steps if you don’t have the time or knowledge to do so. Then have an internal team take the specific actions and test each change to see if it helps.

Consulting tips

If you are thinking about making the move from being an employee to consulting, I have some thoughts. First, there is a difference between contracting and consulting. Contractors are paid for what they do, consultants are paid for what (and who) they know. I have found it’s a lot easier to get contracting work than consulting work. (Of course, sometimes they blur together.)

If you are focusing on consulting, think about:

What is your source of work? Finding work is a big part of consulting. Options include:

  • your network (it’s fine to hit folks up and say “I’m doing X, do you know anyone who needs it?”)
  • past client
  • courses
  • a community presence
  • books
  • blog/writing
  • advertising (though this can be easy to waste money on if you aren’t targeted)

I’m sure there are others. Think about and nurture this pipeline. Always ask happy clients for referrals. This is something you typically don’t think about as an employee (it’s someone else’s job). It’s part of your job as a consultant.

How will you get paid? This is the other piece of consulting that people jumping from employment don’t consider enough.

What will your terms be? Have a standard contract. Have buffer in the bank because there will be lean months; prepare for “feast or famine”. You’ll also need to be prepared to chase down payment. Doesn’t happen often and you can prioritize companies that pay promptly, but with what feels like an economic rough period ahead, companies will be stretching payment terms, esp to outside consultants.

Don’t expect to be paid when you do the work, 30 days net is typical. Use a solution like freshbooks (what I used, years ago, there may be better ones) to automate your invoices and possibly invoice nagging.

Sometimes you will be paid to do things that seem dumb and/or below your pay grade. Raise your concerns, but if you are told to continue, smile and do it. One of the joys of consulting is you are not really “one of the team” which can give you healthy separation from org problems you aren’t trying to solve.

You are not really “one of the team” with any of your clients. Esp if you are a one person show, this can be lonely sometimes if you are used to socializing with your coworkers. Not that they won’t invite you to lunch, etc, but there’s always an awareness that you are a hired gun rather than someone who is on the team. (They might try to recruit you, though. Be prepared for that.)

Taxes/ownership structure/insurance become a bigger thing. Find a CPA, preferably through a referral, and understand and set up the proper ownership structure and tax payments. Typically your tax burden will be higher, but you can write off more stuff as a business expense. You may need some kind of business insurance depending on who your clients are (Oracle made me get e&o, I carried general liability for a while). The nolo books are good on this if you want to get smart on your own, but you’ll really want a CPA in your court.

Think about if you want to be a one-person show or build a team? I personally never got past the one person show stage (though tried subcontractors a few times) because I didn’t want to manage and I wanted to do the work. If you are building a team, you will need to focus even more on bringing in work rather than doing it.

Consulting can be fun because you have control of your work and you get to work in different environments without switching jobs. It’s also more stable to have a few purchasers for your labor than just one (an employer). Think about what kind of work you want to specialize in, because the temptation (esp if cash flow is low) will be to take anything you can do (or maybe things you think you can do).

Spend time on professional development. That could be during the work day or after hours. Sometimes clients that know you will pay for you to learn something, but that is not typical in my experience.

If you have the time, create a course or ebook in the domain you are planning to consult in. This can give you some income, but the real benefit is to say “I wrote the book on X” or “I did a course on X”.

Think about passiveish income options. Courses or ebooks (as above) can offer that. So can productized services, web hosting, or access to a tool that codifies your knowledge. Don’t focus on this when you start, but having something like this will help buffer your cash flow.

Prepare to raise your rates yearly and with every new client. No one is going to give you a raise, you have to ask for one. I usually said something like “my new rate for next year is going to be X/hr, please let me know if you have any questions” in an email around this time every year. Be prepared for some clients to not be able to afford you, and to part ways.

Finally, some book recommendations:

  • The Secrets of Consulting is required reading, covers about the people side of consulting which is really critical.
  • Value Based Fees: How to charge value based fees rather than time and materials/hourly/daily. I never had the guts to do this, but it was eye opening to read about.

How To Start Improving a Legacy App

An interesting question appeared on HN recently: “Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

You can read that post and the answers there. I’m going to address a related, but different question in this post.

If you run encounter an application that has tremendous business value and yet is not following any modern software management processes, what are concrete steps you can take to help improve the application?

That is, what if you have (or are hired to be responsible  for) an app like the poster of the HN thread:

  • making plenty of money for the company
  • no version control
  • old school structure
  • a ball of mud architecture
  • no code deleted, just commented out
  • multiple versions of libraries on the front and back end
  • etc

First off, you need to convince someone that it is going to be worthwhile to invest in this process. If you can’t do that, you are dead in the water. So look for the pain points that occur when best practices are lacking:

  • slow delivery of features
  • catastrophic bugs which lose money, hurt the brand or impact data
  • talent hard to hire
  • hard to improve the application

If you can pinpoint pain caused by the app, you can start to build a case to improve it.

If you can’t, well then, maybe you shouldn’t touch it. If it ain’t broke, don’t fix it!

With that said, here’s my list of what to implement, in rough priority order. Don’t worry about best of breed for the tools, just pick what the company uses. If the tool isn’t in use at the company, pick something you and the team are familiar with. If there is nothing in that set, pick the industry standard. I include a recommendation for the latter.

1. Get the app under version control. Git is best if you don’t have any existing solution. GitHub or GitLab are great places to store your git repositories.

2. Start up a bug tracker. You have to have a place to keep track of all issues. GitHub issues is adequate, but there are a ton of options. This would be an awesome place to get buy-in from the existing team about whichever one they prefer. The truth it is doesn’t matter which particular bug tracker you use, just that you use one.

3. A way to get one click or zero click deploys. A SaaS tool like CircleCI, GitHub actions is fine. If you require “on prem”, Jenkins is a fine place to start. But you want to be able to deploy changes quickly.

4. Set up a staging environment. With one, you can manually test changes and debug issues without affecting production. Building this will also give you confidence that you understand how the system is deployed. Then you can can include that in the build tool process.

5. Unit and system/end to end testing. End to end testing can give you confidence in changes. However, it is overwhelming to add testing to an existing large, crufty codebase. I’d focus on two things: unit testing some of the weird logic; this is a relatively quick win. Second, setting up at least one or two end to end tests through core flows (login, purchase path, etc). In my experience, setting up the first instance of each of these is the toughest process, then it gets progressively easier. There’s usually an ‘xUnit’ framework in any language for unit testing. Look for that. I’m not sure what best practice is in end to end testing, but selenium or cypress are good for browser based applications.

6. Capture documentation. This might be a higher priority, depending on what your relationship with the existing team is. Few teams will say no to someone helping out with doc. Document high level architecture, deployment processes, key APIs, interfaces, data stores, and more. Capture this in google docs or a wiki if you don’t have an existing solution.

7. Start using data migrations. Having some way to automatically roll database changes forward and back is a huge help for moving faster.

None of these are about changing the code (except maybe the last one), but they all wrap the code in a blanket of safety.

After implementing one or more of these, the team should be able to move faster and with more confidence. This will build trust and allow you to suggest bigger changes, such as bringing in a framework or building abstraction layers.

Throwback Thursday: What is the difference between a programmer and a developer?

BicycleOne of the nice things about blogging for so long is that you get to see the wheat separated from the chaff. If I’m still thinking the same thoughts five years or a decade later, that means that I’ve stumbled on one of my truths. (That or I’m just echoing my thoughts because I haven’t learned anything new. I believe it is the former, especially when it comes to development, something I spend a lot of time thinking about.)

Here’s my 2012 post about the difference between a programmer and a developer. A programmer can be a great coder, but stops when the code ends. A developer uses code to build a business solution. (Pssst, a programmer is much more likely to be replaced by an offshore “resource”.)

One of my favorite analogies for developers is the bike messenger. She may not be as fast as a car or as flexible as a pedestrian, but she can weave back and forth between the street and the sidewalk in a way that neither of those two other modes of transportation can. And in the end she’s all about the destination, just like great developers are.

Qualifying “leads” with two simple questions

Toddler girl
Every “lead” started out as one of these.

I use the word “lead” carefully, because every lead is actually a person with desires and hopes and dreams and fears. And it’s worth humanizing them.

But, a “lead” is also a prospect for business. When I ran my consulting company, I was always happy to take coffee because you never knew what could turn up. However, I enjoyed this medium post about how Seamus qualifies leads for his consulting business by asking two simple questions. I also like that he’s explicit about projects that aren’t a fit. It’s hard and scary to niche and yet so worthwhile.

From the post:

I can’t control how I’m introduced to people or how OTL Ventures has been described. So I have found it helpful to be upfront about what OTL Ventures does. This also gives the person who wants to meet with me an opportunity to self-select out of the meeting if they aren’t a good fit. I’ve been doing this by including my answer to the same two questions in my response. It only seems fair.

When you think about it, having this kind of prep conversation is good for both sides. It makes everyone think about what kind of value they bring and can get from a meeting.

Ditching Hourly Rates

Back when I was consulting full time, I typically charged by the hour.  For some clients I’d do fixed bid pricing, based on an hourly guess, but that was typically after we had an established relationship.  Otherwise the risk of losing your shirt is just too high.  I’ve done that once or twice–no fun to be working to finish up a project and just knowing your hourly rate is heading far too rapidly towards single digits.

I’m not consulting now, but if I were I’d be following Jonathan Stark’s advice on value pricing.  You can sign up for his free email course, which is valuable.  After that you’re put on his generic email list which is 25% pitches for his business coaching and 75% good tips about value based pricing.

I will be honest, I’d have a lot of fear about moving to value based pricing, the same way I have fear about niching and focusing on a particular market.  Both are scary concepts because when I am a consultant, the feast or famine nature of the business makes me want to say yes to everyone (within my available skillset and time).

But these will be my first two business experiments if I ever go back to my solo consulting practice.  I’ve just read too many success stories (like this one) to not give it a try.

My “getting paid for the work” story

Consulting is about getting the work, doing the work, and getting paid for the work.

This is my “getting paid for the work” story.

I was a contractor helping build out an ecommerce site for a startup.  I had been introduced to this client by a colleague, and felt like I had a good relationship with the technical lead, “Bob”.  We were making progress on getting the site built out and I’d worked a couple of months with them–they were my primary client.  I believe I was billing semi-monthly.

One fall day, I got a note from “Bob” that he’s leaving, and I should send all my future invoices to “Joe”, from accounting.  I seem to recall that the project was over budget and was being shut down.  I had one outstanding invoice for about $4,000.

“Joe” wasn’t very interested in making me whole.  He probably was interested in trying to keep the company afloat and keep cash in the company’s pockets.  I was, however, interested in collecting that money.

I didn’t have much leverage since the project was shut down and my primary contact had moved on.  What I did have was persistence.  And I was also able to get “Joe”‘s skype handle.

Every two weeks I would re-send the invoice, always with the same format:

Hi,

I just wanted to send you this invoice for work I’ve done previously.  It was due on XX/XXXX.

Please let me know if you have any questions or concerns.

Dan

And then I’d ping “Joe” on skype to see if he had received the invoice.  Needless to say, it didn’t take long before “Joe” wasn’t on skype very often.  I still continued to send the invoice to his email address.

Every year I give holiday gifts to my clients as a way of saying “thank you”.  I gave a box of chocolates to the ecommerce startup that year. Even though they were stiffing me for thousands of dollars, I still appreciated the money they’d paid me and the work they’d let me do.

Within two weeks, I was paid in full.

Useful Rails Gems: Pretender

I’m constantly amazed at how productive you can be with rails. It simply lets you work on typical webapp problems at a much higher level. At 8z, we had a web application and a customer support team. Occasionally the customer support person had to ‘impersonate’ a normal user to troubleshoot an issue. We built a piece of software that let them assume that role. (We called it ‘sudo‘, obviously.) It’s been a few years, but as I recall it was complicated and error prone, lived on a different domain and wasn’t fully functional.

I needed to add similar functionality to a rails web app, and was able to find a couple of gems that looked useful. I selected pretender, mostly on the basis of documentation and google search results placement. I followed the instructions, tweaked a few settings and was off to the races in about an hour.  (Note this isn’t a fair apples to apples comparison of the underlying technologies, due to the differences in available open source libraries between the mid 2000s and the late 2010s.)

Now, all this gem does is what it says it does. It lets a certain user or set of users in your application pretend to be another user. It doesn’t handle auditing or anything else you might want with an elevated privilege system.

But it does do what it does well.

Five rules for troubleshooting an unfamiliar system

trouble photo
Photo by Ken and Nyetta

A few weeks ago, I engaged with a client who had a real issue.  They sold a variety of goods via a website (if this was the 90s, they would have been called an ‘e-tailer’), and had been receiving intermittent double orders through their ecommerce system.  Some customers were charged two times for one order.  This led, as you can imagine, to very unhappy customers.  This had been happening for a while and, unfortunately, due to some external obstacles, internal staff were not available to investigate the issue–they had their hands full with an existing higher priority project.

I was called in to see if I could solve this issue.  I had absolutely no familiarity with the system.  But in less than ten hours of time, I was able to find the issue and resolve it.  How I approached the situation can be summed up in five rules:

Number one: define the problem.  Ask questions, and capture the answers.  What is the exact undesired behavior?  When is the undesired behavior happening?  What seems to trigger it?  When did it start?  Were there any changes that happened recently?  Does the client have reproduction steps?

I gathered as much information as I could, but keep it high level.  I asked for architecture and system diagrams.  For the history of the application.  For access to all systems that could possibly be relevant (this will save you time in the future).  For locations of log files, source repositories, configuration files.  For database credentials and credentials for third party systems like CC processors.  It is important at this time to resist the temptation to dive in–at this point the job is to get a high level understanding so I can be efficient in the next steps.

You will get speculation about what the solution is when you are asking about the problem.  Feel free to capture that, but don’t be influenced by it.

Number two–find the finish line.  After getting a clear definition of the problem, I looked in the orders database and find out if the double orders were showing up there.  They were, which was a clue as to which part of the system was malfunctioning, but more importantly let me see the effectiveness of any changes I was making.  It also lets the customer know the objective end goal, which can be important if this is a t&m project, and it let me know the end state to which I was headed–important for morale.  (BTW, don’t do fixed bids for this type of project–overruns will be unpleasant, and there will be overruns.)

I was able to write a SQL script to find double orders over a given time frame.  I ended up writing a script which emailed the results of this query to myself and the client nightly, as an easy way to track progress.  The results of this query were a quantifiable, objective measure of the problem.

Number three–start where you are familiar.  I could have dove in and looked at the codebase, but due to my problem definition, I knew that there had been no changes to the checkout portion of the code base for years.  I also was unfamiliar with the particular software that managed the ecommerce site and could have wasted a lot of time getting up to speed on the control flow.  Instead, once I had the SQL query, I could find users that had been double charged, and look at their sessions in the web server logs.  I’ve been looking at apache http logs for over a decade and was very familiar with this piece of the system.

Number four–follow your nose. I followed a few of the user sessions using grep and noticed some weirdness in the logs.  There were an awful lot of messages that indicated the server had been restarted, and all the double orders I looked at had completed 5-6 seconds after the minute changed.  (It’s hard to define weirdness explicitly, which is why it behooved me to start with a portion of the system that I was experienced with–it made the “weirdness” more obvious.)  From here, I ended up looking at why or how the server was being restarted regularly.  Ended up finding an errant cron job which was restarting the server often enough that the ecommerce system was getting confused and double booking orders–once before the restart and once after.  This was easily fixed by commenting out the cron job.

Number five–know when to stop.  This ecommerce system obviously had a logic flaw–after all, restarting the web server shouldn’t cause an order to be entered twice, whether you restart it every hour or once a year.  I could have dug through the code to find that out.  But instead, I commented out the cron job, let the system run for a week or so and waited for more double orders.  There were none, indicating that the site was low traffic enough that whatever flaw was present didn’t get exercised often, if at all.  I confirmed with the client that this situation met his expectations of completeness, and called it good.

Being thrown into a new system, especially when troubleshooting, is a difficult task.  I am thankful the client was relatively responsive to my questions, and that pressure, while present, wasn’t intense.  These five steps should help you, if you are put in any troubleshooting situation.

Sometimes you just need a technical project manager

stacked blocks photo
Photo by A. Drauglis

As a developer, my skills are applicable across a wide variety of domains.  However,  I re-engaged recently with a prospective client that I wrote about a while ago. They had used another booking solution over the past few months and still had the same pain.  After thinking and doing some research, I can to a conclusion that hiring a developer was not the right answer for them.

This company had an interesting set of constraints:

  • Wedded to a platform (Shopify) because of previous investment and its excellent shopping experience.
  • The platform doesn’t provide all the functionality needed.
  • Users (both internal and customers) are ill served by another system to login and manage.
  • No third party plugins seem to meet the needs, either alone or in combination (at least, no solutions that I could find).  It’s apparently a fairly unique problem set.
  • They were not interested in solving the problem in incremental steps.
  • They had budget limitations (don’t we all).

In this situation, the best solution is to look for someone who can undertake the following steps:

  1. Write up a clear description of the problem. This doesn’t have to be a detailed requirements doc, but should be a clear explication of the issues, needs, timelines (if any), and current systems.
  2. Post the description to the shopify forums and contact the platform vendor directly, to see if anyone has encountered any of the same issues and ask how they solved them. The point of this isn’t to solve the problem, it’s to see if anyone else has solved pieces of the problem. This will help the company identify partners and/or adjust scope of the project. (If Shopify customer support says ‘whoa, we’ve never heard of this’, it’s a different size problem than if they say ‘well, you might want to bolt these three pieces together’.)
  3. After the client has more knowledge, send the requirements to Shopify focused dev shops (and possibly Elance contractors who work with Shopify, but not just Shopify themes). Work with at least two to three of them to see what a solution would cost, either custom or building on their current code. At this time, avoid getting development quotes from anyone who doesn’t have experience with Shopify development (like me!), simply because the integration with the platform is so critical.
  4. Evaluate the results of the RFP process, including following up any avenues that the experts or forums turn up. Consider whether the budget allows for a comprehensive solution or whether it makes more sense to look at point solutions for the high pain areas.
  5. If the results point to a comprehensive solution being within budget, engage the solution provider.  If not, identify the high pain areas and go back to step 1 with the smaller scope.

So, what this client really needs is a technical project manager (TPM). While most freelancers have this skill to some degree, as it is hard to survive without it, making a full time living as a contract project manager is difficult. I know of only one person in 15 years who was making a living as a contract project manager (and she’s not doing it anymore). This particular project doesn’t seem like a full time effort, so the client should be able to get by with a moonlighter, at least until the results of step 4 are known.

Good technical project managers are hard to find. From a friend’s company’s job req (his company who is looking for a TPM, and the job desc captures the description of the skill set well), they have:

balance between hands-on technical knowledge, a ravenous appetite for order, and an understanding of humans and how they work.

Developers (or former developers) who want to project manage and have people skills aren’t quite the unicorn as someone who can both design and develop, but they are almost as rare.  Unless the company can find a contract project manager who has technical chops, they’ll want to either source this internally or find an external contractor with another skill set (developer or designer) who wants to PM this project.

This is a great chance to for an employee to expand their skill set. Based on my experience managing vendors during my time at 8z (a 4 month website relaunch and a longer term, less time intensive data provisioning engagement), I would advise that this employee have technical skills.  Challenging custom solution providers on technical grounds, or at least being able to follow along, ensures the company will get the best solution. This doesn’t work for “take it or leave it” SaaS apps, but this project appears custom enough that the TPM really needs to have both business and technical considerations in mind when managing the development shop. If they are looking for an outside contractor, engaging with a designer or developer who has PM experience would be an option.

As far as stretching budget, proposing a lower price to the development shop in exchange for shared ownership of the code might make sense. The partner could market this code to other clients and recoup some costs, while the client retains a perpetual license.

Sometimes, a generic developer isn’t the right answer for a software development problem. A technical project manager (or someone wearing that hat) can often stretch budget and leverage skill sets of focused development teams.