display | more...

A Brief Review of E2's Server Errors

If you have used Everything2 for any length of time you have probably discovered that fanciful bit of nonsense known as the server error. Cryptic, technical, and unwelcome messages requiring (often indifferent) higher powers to decipher and address. They're sort of like tax law in this regard.

Basically, with E2, there are two flavors of server errors: the 500 and the 503.

503

The 503 means Apache is down. This is the inevitable result of a cascading failure that looks something like this:

  • Our database design has historically been pretty minimal, and we have a lot of SQ queries that could (and should) be optimized but are not. To put it in perspective, most queries we run clock in at less than a tenth of a second (quite a few at less than a hundredth.) But the queries at Voting Oracle, for example, take about 7 seconds to run, which is a lifetime in terms of database queries. And since we use MyISAM tables instead of InnoDB (the InnoDB engine not being available in MySQL when E2 was first written back in the early 17th century), this invokes a table lock on the vote table. So anybody else trying to vote, or read Voting Oracle, or do anything that involves voting, has to wait their turn. This causes lag. However, once that Voting Oracle query clears, everything catches up pretty readily.
  • However. Sometimes a query is run that takes 30 seconds or more. Sometimes a minute. Sometimes more. And depending on the tables it is locking up, this causes MySQL to basically come to a standstill.
  • Meanwhile, people who have a frozen page naively hit refresh or try to load up two other pages while they're waiting. The child processes in Apache start loading up, and then ...
  • 503! Until Apache restarts itself, anyway.

503s have been caused by other things: memcached filled up a hard drive with its log once, our Apache .conf mysteriously vomited at certain URL calls for awhile, and we've been DOSed on at least one occasion by a (I think benevolent) webcrawler.

500

A 500 error, on the other hand, is different. The server is just fine, but the request you're attempting is no good. There are typically 5 kinds of 500 errors, which we will now explain.

  • No Default Pages Loaded. Our most popular 500 error, this is the error message you get when you choose a displaytype on a node that does not have a htmlpage associated with that particular displaytype and node. For example, if you go to a nodelet node (say Epicenter, node_id 262) and add "&displaytype=printable" to the URL, you'll get a 500 error. This happens mostly for webcrawlers.
    Solution Set the displaytype to "display" if an htmlpage cannot be found. Implemented February 19, 2008.
  • Nodegroup issues. The second most popular 500 error, I can diagnose it but without some debugging I'm not quite sure how to address it without setting off a thousand other issues. Basically, non-admin users have different rights around here, so when they try to view certain pages, they get a "NO SALE" sign. Only for some types of nodes, this comes as a 500 error due to how E2 handles nodegroups (an e2node is a nodegroup consisting of writeups, etc.) The canonical example is trying to view a jscript node (try empty javascript). Which to me makes no sense, since jscript is just a clone of fullpage, which does not give such errors. But clearly this is a traceable error - if and when we get our dev server running, this'll be addressed.
    Solution Get better try-and-catch built into the nodegroup process.
  • Malformed XML. Generally our XML parsers and renderers are pretty well. The most common infringer is the catbox, people's Unicode and obscure characters (and our attempts to modify what people use from time to time) causing occasionally malformed XML and thus 500 errors on the sites that read that XML.
    Solution: Work harder to conform to XML spec. But really, not that big a deal.
  • "Near Matches" searching. This pulls up errors every once in awhile, and I'm not sure how to duplicate it. But there is some magic entry somewhere that makes the SQL query it runs go bad, which causes the whole page to crash and burn. And it actually happens kind of frequently, so if you experience this error, please let me know.
  • Sex crawlers. I'm not sure what causes this one, exactly. But people using some form of adult content search engine are coming across our writeups regarding sex (mostly newsgroup-y stuff like bestiality and animation porn) and then from there submitting a form POST (?), which throws up a 500. Needless to say, Googling this is too embarrassing to be helpful.

So there you have it, the common 500 errors. Generally they occur do to lack of appropriate error catching and handling in our code. On the positive side, there used to be a lot more such errors and they have been addressed on an ad-hoc base and to say that we only have 3 or 4 common causes of 500 errors is not so bad.

Hope this was enlightening!

We're approaching the end of the second month of me playing with the code in a hopefully-helpful fashion, so I figured it was about time I wrote up another Root Log. For what it's worth, I make most of my changes as patches in any case, so that I leave a trail and can undo things more easily when I mess them up. You can therefore see what I've done in some detail by going to Patch Manager.

In any case, here's what I've been up to in list form:

  • Added a very short demonstration of how links work to the 'addwriteup' box (the one where you can add a writeup to a node or edit an existing writeup in full e2node view) for Level 1 users, and a great big ugly list of HTML tags for everyone. To do: Hide that list by default using JavaScript.
  • Added Everything2 Help link to the Epicenter for everyone.
  • Turned on 'blab' boxes - the boxes which allow people to send messages direct from writeup headers - for everyone, unless they have specifically customised their wu headers to exclude them in Writeup Settings. Many people had never even noticed that these things existed, what with their being hidden in a somewhat confusing section of a rather obscure settings page, and previous efforts to turn them on for new users seem not to have been entirely successful (this may have been partly or perhaps entirely because removing the Level 3 restriction on customising writeup headers broke the system for turning blab boxes on without faffing with that stuff).
  • Made call's ingenious formatButton script usable, and added it to Scratch Pads. This inserts paragraph and line-break tags according to the line breaks in the textarea.
  • Made linkview available for Scratch Pads and E2 Scratch Pad, with some help from rootbeer277 and kthejoker in what struck me as a highly educational public code discussion.
  • Made the old-style E2 Scratch Pad available from the new Scratch Pads as 'Default Scratch Pad', more-or-less negating the need for the former. Couldn't get it to work for viewing other users' default pads though, kthejoker eventually stepped in and fixed that after much head-breaking.
  • Added a note to explain that the old Scratch Pad is still available, and corrected the bit where it said 'You have no scratch pads' to say that you have no other scratch pads.
  • Started switching links to the old Scratch Pad to point to the new ones - affected nodes include the Epicenter, addwriteup and nodelockreason, as well as E2 Quick Start and dem bones' writeup under Vitals. If you spot any more, please let me know!
  • Fixed a bug pointed out by BlackPawn where hitting 'Delete!' and then cancelling it would cause all the other buttons to act like delete buttons from then on. DonJaime suggested the neat solution, 'return deleteMe=false'.
  • Implemented a patch from DonJaime to move the posting button away from the others. I think this is sensible, especially since it avoids the confusion of selecting 'recipe', hitting 'Update!' and then finding that the wutype box says 'person' again.
  • Lengthened the period of time messages can show for in showchatter, to eight minutes rather than five.
  • Changed home node displays to link to the new Scratch Pads.
  • Made the Everything2 Help link in Guest User's Epicenter a little bit more obvious by putting it into a centre-aligned paragraph.
  • Implemented a patch from rootbeer277 to make Scratch Pads numbered, for ease of reference, and then a couple more patches to make it so that private pads aren't numbered, so that you can refer to your second pad and someone else will always see the same one.
  • Implemented another rb patch to put list tags around the Findings: page to make it valid HTML, except he wanted to number the list and I wasn't convinced so I changed it from ol to ul.
  • Replaced the contents of the default zen stylesheet with the contents of Kernel Blue, since the latter is really very much prettier and probably more complete than the old default stylesheet, and at this stage I think we should be more concerned with encouraging adoption of the new theme than encouraging as many people as possible to customise it.
  • Applied patch from Sam512 to Everything User Poll: 'Fix FAQ link, make links make sense, add future polls link'. Very sensible I thought.
  • Added Social Bookmarking to Writeup Settings. Not documented yet, because it feels a little unfinished to me - it does seem to work well, but I want to add Cool Man Eddie notification. Anyway, if you want to try it, add r:social (or l:social) to the box with all the writeup header codes on the Writeup Settings page.
  • Applied several patches from DonJaime to enhance the stylability of E2 pages.
  • Made a series of changes to reduce the spidering of pages we don't really want in search engines, like 'chaos' views and Findings pages. Nodes affected: e2node display page, writeup display page, zen e2node displayfull page, jukka e2node displayfull page, jukka container and zen stdcontainer.
  • Added special Google Ads comments around the main content of nodes in the zen container, to give it a clue that most of its ads should be targeted towards that stuff: <!-- google_ad_section_start --> <!-- google_ad_section_end -->
  • Created an Ads nodelet, which it's been suggested might look better than the ad bar at the top of the page. This is chiefly to bring us closer to being able to implement the Zen theme for Guest User.
  • Added format buttons to writeups.
  • Made zenwriteups links go to anchor tags for the writeups they're about (so that you're taken to the bit of the node with the relevant writeup, rather than to the top).
  • Copied the New Writeups - Zen nodelet code into the New Writeups nodelet, and removed the former from the group of nodelets which can be added in Nodelet Settings, so that we just have one New Writeups nodelet to support.

Log in or register to write something here or to contact authors.