Positive Fusion

While the various modifications we have made over the last week have allowed the server to cope with the very random and very severe increases in activity we have experienced lately, tonight we did suffer a significant slow-down in the server’s ability to handle the traffic. There was a blip, which is still being recovered from, where traffic rose instantly to numerous powers of magnitude more than normal.

There was no service outage. Load on the server and speed of processing requests should return to normal within the next few minutes. A manual reload of Apache was done to free up available slots, during this five second window you may have experienced a connection refusal from the server.

We have continued to experience unusual usage spikes, another this evening, though not nearly as severe as previously. There does not seem to be any clear source of cause for this particular problem, but we are continuing to constantly monitor the server checking for any reason this might be happening.

There has been another momentary outage while Apache restarts this evening. It appears there are floods of traffic and the server isn’t spinning up enough process threads to deal with the increased traffic in time. We are working on a solution. There may be additional outages as we work to resolve this.

We’re again being hit with a very unusual server load spike, this time in the 200 range. This is highly abnormal. We’re still working on tracking down the cause, it’s very likely a script gone haywire. We are processing a remote reboot and expect to have services online within the next five minutes.

Update. The ultimate cause of the problem has been determined to be a massive mailbombing. Action has been taken to minimize chances of this occuring again.

We’re currently investigating the sudden increase in traffic to the webserver. This increase has caused server load to spike beyond the ability of the server to handle and still send out pages in a reasonable period of time. While pages are still loading, the delay is significant. We hope to have this issue resolved within the next few minutes.

A load average like this has never been seen before, peaking at approximately 187.72.

The server is currently experiencing a denial of service style attack. We have isolated the attacker and have blocked access.