Our site is down AGAIN, because of a server problems :(

[sehh]

I'm posting this to vent my frustration.

During the past several weeks our site has gone down at least three times. We also had down time during the summer, but now it happens more frequently. There have been a wide range of problems (mostly database related):

1) The /tmp directory on the server was full and MySQL stopped working.

2) All attempts to connect to the server timed out.

3) The /tmp directory stopped being writtable, again MySQL stopped working.

4) MySQL stopped working with an error: "too many open connections".


I have to say that tech support has solved all these problems within minutes, we recieved replies right after we posted a support ticket.

Unfortunately, the whole situation has left us feeling insecure. As a result, me and my colleagues are constantly keep checking the site like mad men in case it falls down again so we can click for the "Create New Ticket" link as fast as possible. We also received several complaints from our users.

We are hosted on Mercury if that means anything.

[Zacrifice]

very unfortunate!
However, I too have experianced these same problems before with the server I am hosted on, however if you can just wait out the ruff waters Jag will track down the problem and correct it.

They always do.

[JPC-Greg]

Hardware fails, nothing we can do to change that aspect of reality. Les was trying to isolate and replace the culprit but its just a good time to fast add these clients to a new machine. Remember our long running and slow process of replacing all older equipment with newer machines. They still remain the most powerful shared servers Ive heard a host using and they are quite costly but worth every dime.

[sehh]

We've found a simple solution to these problems.

A small cron script runs every 30 minutes and checks if the database is accessible as well as the free space on /tmp. If the database can't be accessed or /tmp is full then it automaticaly sends several emails to us.

My next step is to make it open a Support Ticket on its own

[the_ancient]

We've found a simple solution to these problems.

A small cron script runs every 30 minutes and checks if the database is accessible as well as the free space on /tmp. If the database can't be accessed or /tmp is full then it automaticaly sends several emails to us.

My next step is to make it open a Support Ticket on its own

if the site it THAT mission critcal, I suggest a dedicated box

[tank]

if the site it THAT mission critcal, I suggest a dedicated box

Bingo

[Gwaihir]

if the site it THAT mission critcal, I suggest a dedicated box

O come on. "Throw a lot more money at it." Is that the standard answer now? That is no miralce cure and pretty disproportionate to the situation sketched.

In fact, it may not even help AT ALL. One of my sites moved to a (nearly) dedicated box early this year -> suddenly we were the only ones to notice troubles, where earlier on there were probably a 100 other clients who could put in a trouble ticket before me..

[the_ancient]

O come on. "Throw a lot more money at it." Is that the standard answer now? That is no miralce cure and pretty disproportionate to the situation sketched.

In fact, it may not even help AT ALL. One of my sites moved to a (nearly) dedicated box early this year -> suddenly we were the only ones to notice troubles, where earlier on there were probably a 100 other clients who could put in a trouble ticket before me..

all depends on your site.

If you move to a Dedicated box and problems persit, 1 of 2 things is happening

1> (most likly) YOUR site was the problem on the shared server causing havoc for the rest of the users

2> You have more traffic than a single box can handle and need a cluster or other specialized set up.

in a shared enviroment your at the mercy of all the other users on the servers each server here is limited to 250 accounts, which is low compaired to some hosts out there. It only take 1 of those accounts to install a bad runaway script to bring down the server.

Anyone will tell you Dedicated is better. But there are factors, Jag uses some very stout hardware in their Dedicated Boxes, Many of the "Budget hosts" Use very low quality and very small powered setups, Cheaper, and pack them with hundereds and hundereds of clients

In the time I have been with jag, they have proven to be the most reliable of all the budget hosts, and IMO better than most of the "not so budget" hosts where a GigaDealX5 account will cost you 5-10X more than what Jag Charges.

If a site is soooo mission critcal that 15 mins of down time disrupts your hole life than I suggest 2 Dedicated Boxes switched so if one fails the traffic it rerouted to the other until the primary can be restored.

Is that expensive, You betcha, but there is no Free lunch,

You want $10/mo hosting, Deal with the MINOR problems with $10/mo hosting

[Ron]

We've found a simple solution to these problems.

A small cron script runs every 30 minutes and checks if the database is accessible as well as the free space on /tmp. If the database can't be accessed or /tmp is full then it automaticaly sends several emails to us.

My next step is to make it open a Support Ticket on its own

There are monitoring services out there (some for free!) that will watch a page for you and send an email (or a page) when it's down.

Your solution by its very nature can't let you know when most services are not available including the machine itself, the network, the network interface, the php interpreter, the cron daemon the popserver or exim or whatever, etc., etc., etc.

Put up a page that accesses mysql, writes to /tmp does whatever your system does, and get a monitoring service to watch it for you from another machine.

My service tries for 30 seconds, and if it doesn't get a response, it tries for 30 seconds from a second backbone. If all that fails it then considers the machine to be down. It tests every 15 minutes. I point it to a page on my forum, which exercises just about everything my site uses.

[sehh]

I think you are missing the point.

Asking for proper uptime isn't asking alot, even for a budget host.

Especialy when the causes of downtime are rather silly (full /tmp???).

We get more than a million hits per month, downtime can cause problems to everyone, to small or big sites. The answer to downtime isn't mirrored boxes and a fortune in expences.

[the_ancient]

Asking for proper uptime isn't asking alot, even for a budget host.

you have had proper uptime, NETWORK uptime is guaranteed on all shared hosting accounts, you problems stem from a Database overload, which was probally caused by an abuse of services by an account on that Box. thus the evil that is shared hosting

Especialy when the causes of downtime are rather silly (full /tmp???).

See my Posts, all it takes is ONE account of the 250 or less accounts on the box to screw it up for everyone

We get more than a million hits per month, downtime can cause problems to everyone, to small or big sites. The answer to downtime isn't mirrored boxes and a fortune in expences.

actually it is, but what do I know. I have only been building web sites for the better part of the last decade......

[Ron]

I think you are missing the point.

Asking for proper uptime isn't asking alot, even for a budget host.

Especialy when the causes of downtime are rather silly (full /tmp???).

We get more than a million hits per month, downtime can cause problems to everyone, to small or big sites. The answer to downtime isn't mirrored boxes and a fortune in expences.

While that's not what *I* was saying at all, actually the solution to downtime IS mirrored boxes and a fortune in expenses.

If you want good uptime, you've got it here. If you want GREAT uptime, you USUALLY get it here. If you want 99.999999% uptime, you gots to pay through the nose to engineer it. If your site is generating enough cash per hour that an hour of downtime a week would lose you a ton of money, I suggest that you investigate a higher availability setup.

There has been some discussion here between Jag and me in the past on a very simple approach to a high-availability solution for shared customers, but even that solution is going to set you back an extra monthly fee, should Jag implement anything like it.

*I* was just discussing a way to better monitor your sites.

I've had periods of machine instability here that were very frustrating at the time they were occuring, but in hindsight it was all just a blip. If I was down 20 hours one month during transition from one server to another brand new server, and if I lost a day's revenue, it's still much less money than having an HA setup for a month.

Stuff happens.

[the_ancient]

If you want GREAT uptime, you USUALLY get it here. If you want 99.999999% uptime.

as proof of this
My Personal Site hosted here at jag is sitting at 100% been monitoring for only 15 days though
http://host-tracker.com/site-uptime-stats/76834/

[Zacrifice]

I don't run a mission cricital site at all, but I can tell you, its bloody annoying when the site goes down, specially when I start getting emails about it because people can't access the site (I'm not annoyed at the person, am annoyed at the server)

Now I only have the gigasomething 5X package on my site, it suits my budget, it has enough resources for my intentions and it lets me be another person with a website.

However, just because I have the lowend hosting package does NOT MEAN I have no expectations of hosting uptime. I expect the server to be stable, That things when gone wrong will be corrected as quickly as possible, with regular problems they be tracked down and corrected as fast as humanly possible.

This are things Jag gives in great abundance, however, just because its annoying that it happens to the server I'm on does NOT MEAN that I should suddenly start paying more for a dedicated server. IT would be a wast of money to do that!

The original poster may not need a dedicated server and he as every right to be bloody annoyed when the server can't remain stable long enough. It isn't his fualt, yes it could be another users scripts, however Jag should track these down quickly enough via monitoring to make these things a small problem at best, NOT a long enduring process that requires weeks!

The guy either doesn't need nor want a dedicated server, however he is intitled to a stable server!!!

so stop telling people who have shared hosting to move onto dedicated when things go wrong and they are annoyed, they have every right too be!

Oh, and Jag, I know you do a great job and your searching for the problem on the server.

[JPC-Greg]

You should be able to expect some awesome uptime from host even at this budget level and especially from us pouring our hearts and souls into this business. This whole discussion though is rather moot I beleive, its a hardware failure. We saw it immediately, no extra monitoring could have changed anything.

Aside from doubling or tripling our servers and expenses to mirror entire machines there isnt much we can do in cases like this. In our new expensive machines we are doing things like raid and of course we have always had dual cpu machines and so forth. We took steps last year adding in extra power switching devices for all our servers to elliminate that to. We have done all thats physically possible with a single machine to keep it up as long as possible in the event of something failing.

Most hosts, and especially all the budget hosts are others leasing single p4 servers with 512ram and some single hard drive...not much more than a desktop machine really. We have pushed the limit of what budget hosting can do I think. Id like to raise the bar and we will continue to look for ways to do that as best we can. Of course as others mention above in the end it comes down to money and how far we can push the budget hosting envelope.