Author Topic: Why the forums were down Sep 17 - Sep 19, 2016 and other stuff  (Read 420 times)

Corey Cooper

  • Administrator
  • Hero Member
  • *****
  • Posts: 5402
    • View Profile
Why the forums were down Sep 17 - Sep 19, 2016 and other stuff
« on: September 19, 2016, 12:58:03 PM »
TL;DR: I had to move the site to a new hosting service in a hurry.  Took me a few days to work out the kinks.

Long, novella-length version: The Tournament Director hosting service of over 10 years had to move the TD site to a new server this year.  Yes, the site has been on the same server for 10 years.  This created a little bit of disruption, but was quickly worked out.  However, once the site had been moved, the hosting service's support staff began contacting me regarding the site's usage of resources.  The TD site, like most business sites, is hosted on a shared server, and thus each site has a limited amount of resources it can use before it begins to infringe on the resources of other sites on the same server.  The hosting service support staff was claiming the TD site was using way too much CPU resource.  After investigation, I couldn't find a plausible reason.  The statistics over the years simply didn't add up.  But nevertheless I worked with them in cutting down the resources it was using.  Over several weeks and many hours of my time, I had cut down the amount of irrelevant traffic the site was seeing (web crawlers, spiders, that sort of thing) to an almost all-time low for 10 years.  But the support staff continued to claim it was a problem.  At this point I pointed to the fact that the only factor that had changed was the server, and I believed the new server was underpowered or otherwise resource compromised, and it wasn't the fault of the TD site.  They refused to listen and continued to pester me regarding getting the resources down.  Ultimately, the support staff moved the TD site in the dead of night to a new, temporary server, in a "gesture of good will" to allow me more dedicated time to trim down the resource usage.  At this point, I had given up on their staff having a clue, and was convinced it was a server issue.  So I didn't complain about the action, but decided to see how everything behaved on the new server.  After 5 or 6 days of watching the resource usage, it became abundantly clear it had been a server issue the entire time, as resource usage was now at an all time low.  On Saturday, Sept 17, with every intention of contacting the hosting service to put this issue to rest and ask for an apology for wasting my time for nearly 2 months, I realized the hosting service doesn't have technical support on the weekends (another issue altogether), so I would have to wait until Monday to contact them.  I put down the phone and checked my email instead, only to find several thousand alert emails from the Tournament Director website backend, alerting me to the fact that the site was broken, and had been since around midnight.  Apparently the hosting support staff had moved the site back to the production server with no warning, and failed to complete the move (they moved the content, but not the databases).  The site was now down with no hope of recovery without the support staff, who was now unavailable until Monday morning.  The only option was to move the site to a new hosting service, which I completed late Saturday evening.  With any move like this, there are bound to be issues.  And there were several.  At this time I think I have resolved them all.  The forums were the last known issue, and it turns out a single file had failed to copy properly.  Once I tracked it down, I was able to get the rest of the settings updated for the new site.  The forums appear to be functional again.

Thanks to everyone who wrote to tell me of issues they were having.  Some of them I wouldn't have discovered on my own, at least not for a while.  If anyone notices anything odd or not working, please let me know so I can get it fixed up ASAP.  Thanks for your patience folks!

cousguy

  • Newbie
  • *
  • Posts: 29
    • View Profile
Re: Why the forums were down Sep 17 - Sep 19, 2016 and other stuff
« Reply #1 on: October 21, 2016, 08:46:23 AM »
Since you're asking, i'm finding that the attachments in the template forum don't work.  They give a "404 - Attachment Not Found" error.  Not sure if it's because of the move or just old posts.

Corey Cooper

  • Administrator
  • Hero Member
  • *****
  • Posts: 5402
    • View Profile
Re: Why the forums were down Sep 17 - Sep 19, 2016 and other stuff
« Reply #2 on: October 21, 2016, 02:51:27 PM »
Definitely asking.  Thanks for bringing this to my attention.  I'll look into it immediately.

Corey Cooper

  • Administrator
  • Hero Member
  • *****
  • Posts: 5402
    • View Profile
Re: Why the forums were down Sep 17 - Sep 19, 2016 and other stuff
« Reply #3 on: October 21, 2016, 03:14:11 PM »
Fixed.  Thanks again!