{"id":2452,"date":"2017-08-02T17:51:48","date_gmt":"2017-08-02T23:51:48","guid":{"rendered":"http:\/\/www.mooreds.com\/wordpress\/?p=2452"},"modified":"2022-08-27T09:37:47","modified_gmt":"2022-08-27T15:37:47","slug":"heroku-503-errors","status":"publish","type":"post","link":"https:\/\/www.mooreds.com\/wordpress\/archives\/2452","title":{"rendered":"Heroku 503 errors"},"content":{"rendered":"<p>One day recently I woke up to a text from our monitoring service (part of <a href=\"http:\/\/www.mooreds.com\/wordpress\/archives\/2312\">minimal heroku operations tasks<\/a>) saying our application was down.<\/p>\n<p>Darn it.<\/p>\n<p>Here&#8217;s the recreation of my steps in troubleshooting this issue:<\/p>\n<ul>\n<li>Login, see that the app is indeed down.<\/li>\n<li>Look at newrelic.<\/li>\n<li>Look at the logs (papertrail!).<\/li>\n<li>Look at the deployment history.<\/li>\n<li>Note when the issue started&#8211;curses, not when I did a deploy.<\/li>\n<li>Open a ticket with heroku (after doing some research). (Love their support.)<\/li>\n<li>Double check that the database is good and hasn&#8217;t hiccuped.<\/li>\n<li>Look at the logs more.<\/li>\n<li>Add more dynos, see if that helps.<\/li>\n<li>Google the error message.\n<ul>\n<li><time class=\"tooltip-on\" datetime=\"2017-07-12T08:47:37-06:00\" data-tooltip=\"Jul 12 08:47:37\">Jul 12 08:47:37<\/time>\u00a0<span class=\"system\" data-tooltip=\"tfc-production\">&lt;app-name&gt;<\/span> <span class=\"program\" data-tooltip=\"heroku\/router\">heroku\/router:<\/span> <span class=\"message\">at=info method=POST path=&#8221;&lt;url&gt;&#8221; host=app.thefoodcorridor.com request_id=8a17648f-2d84-46ea-abf2-5903be894a2c fwd=&#8221;216.191.191.58&#8243; dyno=web.3 connect=1ms service=4ms status=503 bytes=477 protocol=https<\/span>&#8220;Notice that the issue is being stated right in the log file (passenger request queue filling up). \u00a0Here are sample error messages?<\/li>\n<li><time class=\"tooltip-on\" datetime=\"2017-07-12T08:47:44-06:00\" data-tooltip=\"Jul 12 08:47:44\">Jul 12 08:47:44<\/time> <span class=\"system\" data-tooltip=\"tfc-production\">&lt;app-name&gt;<\/span> <span class=\"program\" data-tooltip=\"app\/web.3\">app\/web.3:<\/span> <span class=\"message\">[ 2017-07-12 14:47:44.3688 65\/7f652dffd700 age\/Cor\/Con\/CheckoutSession.cpp:261 ]: [Client 3-281] Returning HTTP 503 due to: Request queue full (configured max. size: 100)<\/span><\/li>\n<\/ul>\n<\/li>\n<li>Find some posts about the error message.\u00a0 <a href=\"https:\/\/github.com\/phusion\/passenger\/issues\/1424\">Here<\/a> and <a href=\"https:\/\/stackoverflow.com\/questions\/20402801\/what-is-optimal-value-for-phusion-passenger-passengermaxrequestqueuesize\">here<\/a>.<\/li>\n<li>Start researching how to increase request queue size.<\/li>\n<li>Talk a walk to clear my head.<\/li>\n<li>Think about what external services we call, as that seems to be what might cause the request queue to back up.<\/li>\n<li>Read another post that says <a href=\"https:\/\/stackoverflow.com\/questions\/20537185\/the-website-is-under-heavy-load-ror\">restarting passenger helped<\/a>.<\/li>\n<li>Restart all dynos.<\/li>\n<li>Problem disappears.<\/li>\n<li>Look at logs more closely.<\/li>\n<li>Last dyno to be restarted was the only problematic dyno.<\/li>\n<li>Add comment to ticket about this being the cause.<\/li>\n<li>Heroku confirms that the issue may have been the dyno: &#8220;sometimes individual dynos will hang and cause errors with 503 responses&#8221;<\/li>\n<li>Write note to customers about the issue explaining how access to app was affected.<\/li>\n<li>Lower number of dynos.<\/li>\n<li>Breath a sigh of relief.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>One day recently I woke up to a text from our monitoring service (part of minimal heroku operations tasks) saying our application was down. Darn it. Here&#8217;s the recreation of [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[77,37,20],"tags":[],"class_list":["post-2452","post","type-post","status-publish","format-standard","hentry","category-rails","category-tips","category-web-applications"],"_links":{"self":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts\/2452","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/comments?post=2452"}],"version-history":[{"count":2,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts\/2452\/revisions"}],"predecessor-version":[{"id":2457,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts\/2452\/revisions\/2457"}],"wp:attachment":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/media?parent=2452"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/categories?post=2452"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/tags?post=2452"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}