Anatomy of an Outage

[Vin DSL]

Woo Hoo! I just happened to be sitting at the prompt in my shell account when MySQL had a seizure on Neutron. I don't know if this will help the techs track down the problem, but look at these server load[s]. WHOA!!! I guess maybe I need to get a life, but I find this interesting as hell.

The first entry is shortly after MySQL died. The last entry is when MySQL returned. HTML worked fine the whole time incidently. These readings are in 5-15 second increments roughly:

$ uptime
  4:01am  up 10 days,  7:57,  0 users,  load average: 2.14, 0.81, 0.39
$ uptime
  4:01am  up 10 days,  7:57,  0 users,  load average: 2.51, 0.93, 0.44
$ uptime
  4:01am  up 10 days,  7:57,  0 users,  load average: 3.45, 1.18, 0.52
$ uptime
  4:01am  up 10 days,  7:57,  0 users,  load average: 4.30, 1.39, 0.60
$ uptime
  4:02am  up 10 days,  7:57,  0 users,  load average: 5.15, 1.62, 0.67
$ uptime
  4:02am  up 10 days,  7:57,  0 users,  load average: 6.98, 2.05, 0.82
$ uptime
  4:02am  up 10 days,  7:58,  0 users,  load average: 9.52, 2.76, 1.06
$ uptime
  4:02am  up 10 days,  7:58,  0 users,  load average: 10.60, 3.10, 1.18
$ uptime
  4:02am  up 10 days,  7:58,  0 users,  load average: 11.67, 3.44, 1.30
$ uptime
  4:02am  up 10 days,  7:58,  0 users,  load average: 13.22, 3.90, 1.46
$ uptime
  4:02am  up 10 days,  7:58,  0 users,  load average: 15.02, 4.59, 1.71
$ uptime
  4:02am  up 10 days,  7:58,  0 users,  load average: 15.66, 4.90, 1.83
$ uptime
  4:02am  up 10 days,  7:58,  0 users,  load average: 15.93, 5.13, 1.92
$ uptime
  4:03am  up 10 days,  7:58,  0 users,  load average: 16.12, 5.52, 2.08
$ uptime
  4:03am  up 10 days,  7:58,  0 users,  load average: 16.43, 5.76, 2.18
$ uptime
  4:03am  up 10 days,  7:59,  0 users,  load average: 18.13, 6.48, 2.45
$ uptime
  4:03am  up 10 days,  7:59,  0 users,  load average: 17.00, 6.63, 2.54
$ uptime
  4:03am  up 10 days,  7:59,  0 users,  load average: 15.88, 6.57, 2.54
$ uptime
  4:03am  up 10 days,  7:59,  0 users,  load average: 13.66, 6.40, 2.53
$ uptime
  4:03am  up 10 days,  7:59,  0 users,  load average: 11.56, 6.19, 2.50
$ uptime
  4:03am  up 10 days,  7:59,  0 users,  load average: 10.63, 6.08, 2.49
$ uptime
  4:04am  up 10 days,  7:59,  0 users,  load average: 9.78, 5.98, 2.47
$ uptime
  4:04am  up 10 days,  7:59,  0 users,  load average: 9.00, 5.88, 2.46
$ uptime
  4:04am  up 10 days,  8:00,  0 users,  load average: 7.61, 5.69, 2.43
$ uptime
  4:04am  up 10 days,  8:00,  0 users,  load average: 6.52, 5.52, 2.41
$ uptime
  4:04am  up 10 days,  8:00,  0 users,  load average: 5.67, 5.37, 2.40
$ uptime
  4:04am  up 10 days,  8:00,  0 users,  load average: 5.18, 5.27, 2.40
$ uptime
  4:04am  up 10 days,  8:00,  0 users,  load average: 4.54, 5.13, 2.38
$ uptime
  4:05am  up 10 days,  8:00,  0 users,  load average: 4.29, 5.06, 2.39
$ uptime
  4:05am  up 10 days,  8:01,  0 users,  load average: 3.78, 4.93, 2.37
$ uptime
  4:05am  up 10 days,  8:01,  0 users,  load average: 3.43, 4.81, 2.36
$ uptime
  4:05am  up 10 days,  8:01,  0 users,  load average: 3.06, 4.69, 2.35
$ uptime
  4:05am  up 10 days,  8:01,  0 users,  load average: 2.74, 4.56, 2.33
$ uptime
  4:05am  up 10 days,  8:01,  0 users,  load average: 2.47, 4.45, 2.32
$ uptime
  4:06am  up 10 days,  8:02,  0 users,  load average: 2.38, 4.33, 2.31
$ uptime
  4:06am  up 10 days,  8:02,  0 users,  load average: 2.43, 4.30, 2.32
$ uptime
  4:06am  up 10 days,  8:02,  0 users,  load average: 2.54, 4.20, 2.32
$ uptime
  4:06am  up 10 days,  8:02,  0 users,  load average: 2.75, 4.20, 2.34
$ uptime
  4:07am  up 10 days,  8:02,  0 users,  load average: 2.96, 4.17, 2.36
$ uptime
  4:07am  up 10 days,  8:03,  0 users,  load average: 2.60, 4.03, 2.35
$ uptime
  4:07am  up 10 days,  8:03,  0 users,  load average: 2.35, 3.93, 2.33
$ uptime
  4:07am  up 10 days,  8:03,  0 users,  load average: 2.23, 3.85, 2.32
$ uptime
  4:07am  up 10 days,  8:03,  0 users,  load average: 2.10, 3.74, 2.31
$ uptime
  4:08am  up 10 days,  8:03,  0 users,  load average: 2.40, 3.75, 2.33
$ uptime
  4:08am  up 10 days,  8:04,  0 users,  load average: 2.37, 3.68, 2.33
$ uptime
  4:08am  up 10 days,  8:04,  0 users,  load average: 2.33, 3.59, 2.33
$ uptime
  4:08am  up 10 days,  8:04,  0 users,  load average: 1.88, 3.43, 2.29
$ uptime
  4:09am  up 10 days,  8:04,  0 users,  load average: 1.54, 3.27, 2.26
$ uptime
  4:09am  up 10 days,  8:05,  0 users,  load average: 1.20, 3.11, 2.22
$ uptime
  4:09am  up 10 days,  8:05,  0 users,  load average: 1.16, 3.04, 2.21
$ uptime
  4:09am  up 10 days,  8:05,  0 users,  load average: 1.06, 2.93, 2.18
$ uptime
  4:10am  up 10 days,  8:05,  0 users,  load average: 1.70, 2.96, 2.21
$ uptime
  4:10am  up 10 days,  8:06,  0 users,  load average: 1.68, 2.92, 2.20
$ uptime
  4:10am  up 10 days,  8:06,  0 users,  load average: 4.25, 3.36, 2.37
$ uptime
  4:11am  up 10 days,  8:06,  0 users,  load average: 3.67, 3.27, 2.35
$ uptime
  4:11am  up 10 days,  8:06,  0 users,  load average: 3.26, 3.19, 2.34
$ uptime
  4:11am  up 10 days,  8:07,  0 users,  load average: 3.25, 3.18, 2.36
$ uptime
  4:12am  up 10 days,  8:08,  0 users,  load average: 1.85, 2.83, 2.27
$ uptime
  4:12am  up 10 days,  8:08,  0 users,  load average: 1.92, 2.79, 2.27
$ uptime
  4:13am  up 10 days,  8:08,  0 users,  load average: 2.14, 2.77, 2.27
$ uptime
  4:13am  up 10 days,  8:09,  0 users,  load average: 1.65, 2.60, 2.23
$ uptime
  4:14am  up 10 days,  8:10,  0 users,  load average: 1.12, 2.28, 2.14

Hrm... interesting, no??? Here's one last one, now that things have settled down and are back to normal...

$ uptime
  4:27am  up 10 days,  8:23,  0 users,  load average: 0.10, 0.43, 1.09

[Vin DSL]

This makes me realize how stessed that CPU was...

$ uptime
  6:06am  up 10 days, 10:02,  0 users,  load average: 0.21, 0.21, 0.19

[mattsiegman]

maybe it got mad 'cause you ran uptime every 5 seconds

[Vin DSL]

Originally posted by mattsiegman
maybe it got mad 'cause you ran uptime every 5 seconds

Nah! 'Uptime' doesn't hurt anything...

[mattsiegman]

[that was a joke]

[Vin DSL]

Originally posted by mattsiegman
[that was a joke]

LoL, I know, but some ppl think that way. I left that comment for others to read.

Really, we're jailed now. How much harm can we do, unintentional or otherwise? The 'top' command would have been better, but I'm not a tech, so I guess I don't need it. Would have been nice though, none-the-less. 'Uptime' was the only tool in my bag...

[Vin DSL]

Hrm...

$ uptime
  8:39am  up 15 days, 12:35,  0 users,  load average: 25.08, 23.29, 11.08
$ uptime
  8:39am  up 15 days, 12:35,  0 users,  load average: 24.59, 23.22, 11.13
$ uptime
  8:39am  up 15 days, 12:35,  0 users,  load average: 23.98, 23.12, 11.16
$ uptime
  8:39am  up 15 days, 12:35,  0 users,  load average: 22.94, 22.91, 11.16
$ uptime
  8:39am  up 15 days, 12:35,  0 users,  load average: 22.07, 22.73, 11.16

[Defender]

Now that's some mess up loads!

What causes that much load w/o any users?

[Vin DSL]

Originally posted by Defender
Now that's some mess up loads!

What causes that much load w/o any users?

This was the explanation last time...

Originally posted by lijoe
...some user had left back a few dead mysql and http processes.Killed them and the "anatomy of an outage" came back to normalcy...