Servers Asplode!

During November we have had some severe site responsiveness issues which were capped off, most recently, by the loss of one of the three main servers that power E2.  Although most of the hardware from that server has been recovered, its motherboard is fried, and so we are temporarily going to have to do without it.

I appreciate that having the site slow or outright unavailable can be very frustrating. I empathize. I know many of you folks feel frustrated having a site you care about unavailable and being powerless to fix it. The words of support I've gotten from folks thanking me for trying to get the site back on its feet have been quite heartening, and I thank you all for sending them. I'm sorry I've not been able to get the site functioning better sooner.

This situation isn't the best for anybody involved. Although I am the most active person with regards to working with the server configuration, I would rather be spending more time improving site features and giving the place a more modern feel. Hopefully, I will be able to put more time into that area soon.

Here's a timeline of recent events (times in U.S. Central, since that's our servers' location):

Tuesday Nov. 22, initial downtime

Around 10:15am, somebody powered down and unplugged two of our servers. Sadly, the room in which our servers are currently stored, an electronics work room, has been used as a classroom during this semester, something that was not expected when our servers were placed there. This has resulted in downtime multiple times when folks unaffiliated with the site, and unaware of the consequence of what they were doing, powered down our servers.

When I saw the servers down, I contacted the folks on site by email to inform them of the situation. They were able to get to the server room a bit after noon, where a presentation was going on, so the room was dark. They brough the primary DB server online, but were unable to power up the other box. The site was very slow, but back up.

At 4:10pm, Joe was able to get in and turn on the other machine. He discovered it wasn't only powered down, but unplugged from the wall. Hence why its power button did not work earlier. This got all of our machines back online.

Tuesday Nov. 22, further downtime

Around 6:45pm, our primary database server went offline. Two folks were good enough to travel back to campus to check on it. Let's call them Dilligence and Hiro. They arrived to find the server powered up, fan spinning, but without monitor output and unresponsive to input. It was the day before Thanksgiving Break starts and our primary database server was down. The site went offline.

Let me stress that at no time was the data backing the site in danger. The drives holding the data for E2 are still in working order, the replication database server was still perfectly functional, and we have recent offsite backups of all nodes. I hope you'll forgive me for being redundant, as I say this every time there are server issues, but we care about the data that makes up E2 and have taken steps to keep it safe.

Wednesday Nov. 23

Joe and Dilligence spent most of the morning attempting to get a scavenged machine together to get us a replacement database server. Unable to get the scavenged bits to work together, they retasked our replication database server to fill in for the dead primary server. This brought the site back up, limping, after around a full day of downtime. This is the configuration we've been running on since then.

Unfortunately, our replication database server had been our static file server, as well as our cron job server, so we needed a new place for these. We moved the cron job server to one of the overtaxed web virtual machines. For the static file server, alex was kind enough to let us use one of his off-site machines. Hammering out this configuration was part of why the site looked very weird for a while; stylesheets were not loading.

Saturday Nov. 26
After profiling the site, made a small technical change to reduce load.
Sunday Nov. 27
The site still overtaxed, I made a somewhat risky code change. CPU load dropped significantly. Things are still problematic, but a fair bit faster than they were before. However, since it was a change that affects most of the code on the site, bugs may remain undiscovered. As always, please report problems to E2 Bugs, even if you suspect it may be just you. I'm afraid I don't have the time to respond to all of the messages I receive, let alone monitor the chatterbox 24/7, so if you report problems in one of those venues, it is less likely to be attended to by one of the coders.

Going Forward

The site administration, including alex, clampe, and nate are actively discussing the site's needs and potential for relocation to other servers in the coming weeks to months.  I'm advising as far as technical requirements go, and, should we relocate, will do what I can to assist in the transition.

If we do move, relocation and downtime will be anounced on the front page beforehand with a lead time of days if not weeks.  Since we have the luxury of good support staff, relocation will almost certainly have the hardware and site running on the hypothetical new machines prior to us shutting down the current servers.

Keep calm and don't panic.  If you are concerned about the site, please contact the people who know what's going on. I am more than happy to answer questions regarding site functioning addressed privately to me either via /msg or email.  Those things I can not answer, I will refer you to somebody who can. Or, if you think others may share your concern and benefit from a public response, drop a note in a daylog, or a Letters to the Editors. Any lack of transparency isn't due to shadowy plotting; that's not our bag. We just sometimes lack time or don't know folks are curious.

Slowness Prior to Downtime

Prior to the downtime in November, we have been seeing a fair number of error pages and slow pageloads. I can get fairly technical about this, but suffice it to say, E2 could really use a couple of extra servers right now. If you are seeing slow pageloads or frequent timeouts, for the time being, you may try going to your nodelet settings and disabling some of them. For some of our more expensive users, an average pageload spends more than half its time generating nodelets.

Offers to Help

Several people have offered to send money or hardware to get the site functioning better. Although the offers are appreciated, we can't take money right now. For those unfamiliar with how E2 got into its current hosting relationship, read clampe's writeup on the state of the site back in 2007. In the past, hardware was a difficult proposition as we couldn't guarantee resources were available to plug the boxes in, even if somebody could be generous enough to ship us up-to-date boxes. That is less of an issue now, but what with potential relocation, it might be sketchy to try to plug a new machine in, especially if we pull up stakes a week later. Regardless, if you are interested in such a thing, get in contact with alex. I can't really say yes or no on these things, but I get these questions a lot, so I wanted to provide a (non-authoritative) answer.

There are plenty of opportunities to improve E2's code and reap performance improvements with judicious patches. I know quite a few of the active users at E2, especially the more vocally irritated, have the technical background that could assist in this area. I've got a list as long as my arm of small projects that can make a difference which I simply can't get to. If you're at all interested, please join edev and let us know. Prior knowledge of Perl not required. We'd be happy to have you aboard.

Server Configuration

In my last root log, several months back, I covered our servers. They didn't change much since then, except I moved the dev server off of dom00, over to dom10, and I expanded its RAM to 1GB from 512MB. The configuration at the start of the month was like this:

dom00
4 core Intel Xeon X3360 (@2.83GHz), 8GB RAM, 147 GB SSD, RAID1 of unimpressive stats
Primary MySQL server
xen host to:
  • www1 - 2GB RAM, 2 core virtual web server
  • www2 - 2GB RAM, 2 core virtual web server
dom10
4 core Intel Xeon X3360 (@2.83GHz), 8GB RAM, RAID1 of unimpressive stats
xen host to:
  • www3 - 2GB RAM, 2 core virtual web server
  • www4 - 2GB RAM, 2 core virtual web server
  • dev3 - 1GB RAM, 1 core dev box
dom52
4 core Intel Xeon X3360 (@2.83GHz), 8GB RAM, 147 GB SSD, RAID1 of unimpressive stats
MySQL replication server
backup server
cron job server
nfs server
stats server
dom51
Intel P4 (@3GHz), 1GB RAM, a regular-old HD
haproxy server (gateway to the outside world)
Mercurial server
Itty bitty stats, big job

As you can see, dom00 had several pretty important jobs. Since the web servers and MySQL are on the same machine, they compete for CPU time. Further, any RAM assigned to one of the web servers is RAM that MySQL can't use. Ideally, the primary MySQL server would have its own machine, but we didn't have an extra machine to do that.

To reduce load, I tasked dom52 with the job of being a static file server, serving images, stylesheets, static Javascript, and all of Guest User's Javascript. This was a natural move since dom52 was being underutilized, and it was already the cron job server, so the generated files were already being dropped there. It moved about a quarter of our HTTP requests off of our primary web servers, which helped.

Here's the configuration at present:

dom00 - the new name for the machine that *was* dom52
4 core Intel Xeon X3360 (@2.83GHz), 8GB RAM, 147 GB SSD, RAID1 of unimpressive stats
Primary MySQL server
xen host to:
  • www1 - 2GB RAM, 2 core virtual web server
  • www2 - 2GB RAM, 2 core virtual web server
dom10
4 core Intel Xeon X3360 (@2.83GHz), 8GB RAM, RAID1 of unimpressive stats
xen host to:
  • www3 - 3GB RAM, 2 core virtual web server, cron server
  • www4 - 3GB RAM, 2 core virtual web server
  • dev server shut down to reduce load
old dom00
4 core Intel Xeon X3360 (@2.83GHz), 8GB RAM, 147 GB SSD, RAID1 of unimpressive stats, disassembled
dom51
Intel P4 (@3GHz), 1GB RAM, a regular-old HD
haproxy server (gateway to the outside world)
Mercurial server
Itty bitty stats, big job
alex's external server - e2forum
Unknown stats
Static file server

Since dom10 had RAM that was going to waste, we expanded the memory for www3 and www4. Since we lost our cron server, we had to make one of our web machines dual purpose, or assign resources from somewhere else. For now, www3 gets the burden. www3 also serves as the backup static file server if the external static server goes down. The dev server was shut down to reduce load on that machine.

Something I didn't realized had been overlooked until I wrote this up is the NFS server. The job of the NFS server is to hold uploaded images and writeup backups so they are consistent across all servers. I'll have to fix that up. Current dom00 was the old nfs server, so I should be able to copy the files off of there. Most likely dom51 will become their new home. Will update this when it's been taken care of.

Enumerated Patches

isSpecialDate
- I made Halloween last a *little* bit longer by making special date use L.A. time, so the special Halloween doohickeys kicked on at midnight UTC, briefly turned off at midnight UTC, and finally went off for 2011 at midnight U.S. Pacific Time. For future holidays, we will likely be having festivities run from midnight UTC until midnight Pacific from the start, but that code hasn't been put in place yet.
Your Nodeshells II
- A bug report (I forget by whom) came in that this no longer worked since we changed the owners of nodeshells. Changed it to use createdby_user instead of author_user and it works once more
document linkview page
- DonJaime pointed out that linkview wasn't making softlinks. So I applied appropriate classes so that softlinking works as expected; removed no-longer necessary angle-bracket direct linking stuff
showchatter
- Fixed a bug where /fireball would sometimes display wierdly, showing 'singe' as if it were a 'sing' command
message
- Big code cleanup; regularize spaces/tabs; remove dangerous /settings feature; remove dead code; clean up regexes; stop commands from being silently swallowed; make sendPrivateMessage pass arguments the new way
static javascript
stylesheet serve page
linkStylesheet
To speed up pageloads, I wrote a cron job to generate the stylesheets and Guest User's Javascript in static files that could be served from a separate server than our main application web servers. DonJaime helped out quite a bit pointing out items I had overlooked, like initially not generating softlink gradients, and not taking care of the autofixing behavior that stylesheet serve page does. Further, DonJaime patched things up so that the stylesheet author would always get the dynamically generated stylesheet, so they don't have to wait for the cron job to fire before seeing stylesheet changes on their end.
voteit
weblog
Random Nodes

After turning on htmlcode compile caching, some lexically scoped-variables were retaining there values between calls. For example:

my $subRef = sub {
  my $foo;
  $foo .= "Foo!\n";
  return $foo;
};

On its second call, &$subRef(); would return "Foo!\nFoo!\n" So we had to patch some htmlcodes to explicitly initialize their variables to resolve some resulting bugs. I am honestly confused why this happens; normally lexically-scoped variables simply expire when their scope ends, and a new one is generated when the scope is entered again. I can only guess this is an artifact of running in a mod_perl environment. Worst-case scenario, patches like this will allow us to eventually run E2 with use warnings, which will make coding easier anyhow.

DonJaime submitted a bunch more patches to cover potential bugs like this, as well, but I'm sure he'll cover those in his log.

Mercurial-side Patches

Fixed wonky error output (coders only)
Because printf was being used to generate them, error messages that had percent signs in them would get garbled. Fixed.
More logging on Apache crashes
When Apache crashed mid-pageload, it wasn't logging much data, and I had hoped it was being caused by some particular type of query. So I made it log all query parameters on these occasions. Unfortunately, it didn't reveal much. It seems Apache is receiving an external signal which causes it to shut down all procceses at once.
Fix tag balancing code
The tag balancing code had been broken for quite a while, and raincomplex brought it to my attention. I believe I resolved this, but didn't get to test it on the live server. If you're interested in testing, try making a draft with broken table tags and see if they still get closed properly. Post a link in E2 Bugs if it appears still broken.
Calculate date without shelling out
The logging code called `date` to get the date, which showed up surprisingly high in the profile data. Presumably because shelling out requires accessing the disk to load bash, then to check for .bash_profile and .bashrc, and then to load /bin/date. Replaced with an appropriate use of DateTime. I believe Tick Tock has this same issue and still needs patching, though it runs rarely and is cached.
Statically generate stylesheets and Guest User Javascript
As described above, there are now cron jobs to generate static versions of these files in an attempt to reduce load on our primary web servers. Since there are no scripts to distribute these files to another server as yet,the cron server must also be the static file server. Since we are using an external server for static files, updates to Javascript and stylesheets will not propagate without manually copying the files. If we get a reliable cron server soon, this won't be an issue. If not, we'll have to find a way to distribute these files when they're updated.
Avoid uninitialized value warnings
Made some patches to ecore to get us slightly closer to being able to run with use warnings
Set nodetypes to be static
Static nodetypes was an option built into ecore long ago, before I came on staff, intended to allow lower load at the expense of the site being less dynamic under code and design changes. When profiling, a significant portion of every page load was spent doing dynamic lookups for nodetypes, so I set this to static. Realistically, when a nodetype is updated, ecore should find all types that derive from that type, update them, and dump the type cache for each of those types. That would be the best of both worlds. For now, any changes to nodetypes will require a restart of apache.
Cache compiled htmlcodes and opcodes

E2 uses the function evalCode to dynamically evaluate Perl stored in the database. This feature, viewed contentiously, is very neat from a development perspective as it allows significant code changes without server access, and without restarting the web server. It also comes with the downside that we were compiling code for regularly called functions each time they were called.

Near the beginning of the year, I had attempted to cache the results of these compilations to reduce CPU load. When I did so, DonJaime pointed out I had messed up, and the value of $NODE was being frozen to whatever it was when it first ran. I effectively disabled the code (while leaving most of its guts in place) and left the project by the wayside, worrying that another attempt might result in skulking bugs or requiring a lot of cleverness to ensure reference to package level variables like $USER and $VARS didn't get frozen. But good DJ had previously suggested a course of action to try again, and seeing huge CPU spikes, I felt it would be foolish not to try.

So now we cache the result of htmlcodes and opcodes being compiled. This has greatly reduced CPU load, and has triggered few bugs that we've observed. What's even nicer is that the cached code is tied to the node, and so we get the benefits of the existing node caching. If we patch an htmlcode or opcode, the cached code will be silently discarded.

superdocs, pages, htmlpages, and containers could all potentially benefit from the same treatment, but doing so is not as straightforward, so that will have to wait until further profiling shows it's worthwhile.

Did I Forget Anything?

Please toss me a message if you feel there is anything I should have covered here but didn't. Aside from bug reports, this is effectively the only communication I have with you guys where I'm acting as a staff member, so I try to be thorough.

The most interesting things happening to Everyting2's code this month were OldMiner's heroic efforts to keep it afloat. May he reap the rewards prescribed by all applicable religions. My consequent tweaks have been adequately addressed in his root log.

But even if the ship is going down, the deck chairs still need arranging. And I made some contributions, too:

New features

Easy new categories

It is now possible to create a new category on the fly from the 'add to category' form. So when you are inspired to put something into a whole new category, you no longer have to go to Create Category to do so. This required patches to categoryform to provide the option and to category to do the business.

Advanced publish options

You can publish a draft to a writeup. Fine. But editors can create documents as well, and it's useful to polish a document up before revealing it to the masses. So editors had a 'publish as document' button in their Master Control. Which wasn't a good place, since everything else you can do with a draft is under it. So setdraftstatus was patched to provide a 'Publish as Document' button for Editors. So that it would work, publishdrafttodocument had to be changed to take a 'node_id' parameter instead of 'draft_id'. Then the old button could be taken out of admin toolset.

The special user account 'everyone' is there for anonymous writeups, or writeups for which people do not want to reap the benefits of upvotes. To publish a writeup as 'everyone' you had to ask alex for the password, or ask an editor to change the author of one of your existing works. This was all rather cumbersome and not very anonymous. So now users of level 1 and above have an 'Advanced option(s)' button at the bottom of their drafts pages, which when clicked reveals an option to publish as a different user. Most users only have the option to publish anonymously as 'everyone', members of e2docs can also publish as Virgil, and editors can publish as various other special accounts they will probably never need. (The 'Advanced options(s) button also reveals their 'publish as document' button, hence the '(s)'.)

The option to publish as a different user required a fair amount of plumbing to work:

canpublishas
This new htmlcode has two related functions: when passed a name, it returns '1' if the user can publish under the account with this name. When called without an argument it returns a form control to choose to publish anonymously or under a different account.
publishdraft
This opcode turns a draft into a writeup before it is attached to an e2node. If a user tries to publish under a different user account, it checks canpublishas to check that they are allowed to do so, and checks that the other account is allowed to publish to the target e2node. If the user is creating a new e2node, it sets a parameter to change its creator to the other account. Finally, it sets the author_user of the writeup to the user_id of the alternate account.
publishwriteup
This attaches a new writeup to an e2node and does any necessary housekeeping, such as adjusting the user's XP. Which it will now not do for anonymous publication
parentdraft
This provides the options to choose a parent e2node when a draft is to be published. As long as the user has previously chosen to publish as a different user and as long as they are allowed to, it calls canpublishas to provide the form control to maintain or reverse that choice.
setdraftstatus
This creates the controls shown under drafts to set their status, attach them to e2nodes and publish them, and now also provides the advanced publishing options when asked nicely.

Bug fixes

If you have searched for something with a particular title and do not find it, you are offered an opportunity to create it yourself. You are invited to correct the spelling and capitalisation of the title while doing so. Unfortunately, your corrections were ignored. This was because the form containing the corrected title was sent to an address including the uncorrected title, which led to the 'node' parameter being passed to the everything2 engine containing the wrong one. So on OldMiner's prompting I fixed e2createnewnode to send the form to the address '/', which the code does not do anything with.

In the course of tidying up the admin tools for editors I had lost sight of which were where with what options active. With the result that two site admins were missing some of their vital buttons. Fixing admin toolset gave them their 'reparent writeup' and 'change author' links back.

As a result of a small change to settings script, if you are using Chrome or Safari you can now add and remove nodelets at Nodelet Settings, as well as moving them around.

General tidiness

If you did a full-text search from the search box, you were taken to Google Search Beta. Since this page had been working without bugs for some years, there was no reason to call it 'beta'. There was another page, E2 Full Text Search, which had consisted of a promise to start working soon for most of the last decade. So now the working code is there, while the 'Beta' page shows a link to it.

zensearchform now fills in the title of the page you just came from in the search box whether or not a softlink has been made.

One of OldMiner's site-rescuing changes was caching user's stylesheets in static files, updated from time to time. Stylesheet authors may appreciate having the latest up to date version of their stylesheets. linkStylesheet will now give them it.

Log in or register to write something here or to contact authors.