root log: February 2012

Bless me father, for I have sinned. It has been one month since my last confession. Here is what I have done and ask for the forgiveness of all:

Feb 2nd
• Changed the ways softlink works slightly until I can get the right technical underpinnings for arbitrary metadata to exclude non Google AdSense-able nodes. For instance a softlink to Assassin's Creed would cause the node to not have any ads on it when guest user sees it. I put a change in to simply hide the softlink. I normally would not have touched it, but it was eating up about 40% of our advertising revenue. Again, this will get undone once we can tag nodes on a case-by-case basis for not being advertisable.

Feb 3rd
• More infrastructure junk. Cleaning up various types

Feb 4th
• Cleaned up all of the infrastructure stuff, and every type now exports the minimal amount necessary to capture relevant data. I still need to pare down the usergroups, but I think that's a smaller mission later on
• I took a snapshot of the production database with ecoretool and it showed me some areas where I can improve the paring down of various nodes, so I will be doing that first,and grabbing a snapshot again.

Feb 8th
• After export is mostly nailed, I now need to work on merging the development snapshot I'm working off of, and production, and that means narrowing that gap some, changing some node_ids in development, and working through the differences. A lot of development nodes are getting deleted out of git because I want it to represent production. If it is lost work for anyone, it's pretty simple to undelete them.

Feb 12th
• You can see that my github checkins are a lot about churn of individual testing nodes and creating tools to handle more change. I've been aligning my data dump of the dev server with production, including changing node_ids, etc, etc. After this has been made a lot more robust, I have my first ecoretool.pl import planned against the fields taken from an export of the live database

Feb 13th
• Removed the 'mvtest' sqltable from the mvtest type, because it doesn't exist
• Created the 'mvtest' dbtable, so that we can test phasing it out, if need be. This is purely a transitional move
• Updated the writeup at Getting the ecore development environment to work at the request of Oolong to make it a bit easier to get up and running for git newbies

Feb 14th
• Changed displayWriteupInfo (See Patch Manager for the exact code done), so that we don't go to the database again to get the hits information for a particular node; we just get it out of WRITEUP. This has the small downside of not always having the most accurate information if you get balanced to the exact same thread which has the node in cache, but whenever we need to re-get the node after it falls out of cache, it's fine. It's a pretty slim chance, and since the data is okay in the DB anyways, whichever. This one small change has slashed our database load pretty seriously.
• Commented out assign_test_condition() for now. in Everything::HTML
• Added root back into gods, since I'm going to make that the default scripted "gods" user.

Feb 15th
• Commented out the hits code live on production so that we can get better performance out of the database until we figure out a better way to do it.

Feb 19th
• Been tackling a lot more invisible crap in the site, to really clean up some old practices that I don't think are particularly scalable. These are some pre-cursors to fixing the utf-8 to latin1 problems that I'm seeing on export.
• Got homenode pictures working again. I'm so sorry folks.

Feb 20th
• Working on a new way to handle backups. I've converted all of the tables in the database over to InnoDB tables, and I'm going to be working on a way to do InnoDB checkpoint backups instead of the old mysqldump method. This is so that we have some disaster recovery available. The problem here is though that we don't have enough hardware or disk for the time being to really do it on-site, so I am using off-site resources to make it happen. For those that are curious, we are using Percona xtrabackup to handle this.

Feb 23rd
• Percona xtrabackup turned out to be kind of a dud. It was taking literally hours and the backup swelled over 6x before I cancelled it to 200Gb. So forget that. We'll now have a backup strategy that involves a read lock with --single-transaction with mysqldump.
• Created a script to dump a "staging" snapshot of the database. This will help in integration testing and in finalizing the last of the data dump and migration from our "testing" snapshot that used to live on dev1.everything2

Feb 25th
• Staging dump went in after 9 hours of processing, so that's good. It'll allow me to kind of frost that away for later use when I need to test something against a more integrated setup.
• Today for chicken-and-egg import reasons, I need to look into removing the nodetype_id from nodes and replacing it with either the type or let where it exists in the repo be implied.
• Patched Login to have a unique id for scripting purposes

Feb 27th
• Edited static javascript to add the title of the node that we are on, plus use standard JSON encoding

This will be my final Root Log as the nominal Lead Developer of Everything2. I've got a lot of stuff improved and put in place in this role, and I'm very proud of what the development team has achieved here in the last couple of years or so - besides what I have done myself, DonJaime, who overwhelmingly shares my vision for how the site ought to work, has achieved several major overhauls of things while making countless valuable interface tweaks; OldMiner has put in a huge amount of work to keep the site running at all, and to improve performance within the framework of systems which, to be frank, were not always up to the task at hand. Call's occasional work on things like the site's mobile version has also counted for a lot.

JayBonci has now taken over the reins of the site, and since he is a talented and dedicated programmer who probably has a bit more time to invest in the site than I have available myself, I am officially handing over the overall management of the code to him as well. I will continue to be a developer here when I can, but I have a lot of other things to sink my efforts into here - pulling together the E2 Podcast, general editing duties, and of course writing.

So, to the actual logging of my rooting - I have done a great deal of coding here since my last proper root log, which was way back in October 2010, so apologies for talking about some new features which are actually quite old by now.

1. The Messages nodelet is a big one. The old Chatterbox handled private messages and public chat entirely in the same area, making it excessively easy to make private things public by mistake. The Messages nodelet tidies that up with a little reply box for any message when you hit the 'Reply' button, with options to delete or archive the message under 'Remove'. DonJaime was helpful with this, and it built on some of his earlier work to integrate jQuery better with the site.
2. I re-did Guest Front Page in November last year, splitting it off decisively from the front page shown to logged in users. If I did my job, it is far less cluttered and more enticing than it used to be, but I would like to see this revisited again soon, along with the general styling of the site that guests see.
3. Buffalo Generator is pure frippery. It generates random grammatically-correct strings of words which are both plural nouns and verbs.
4. Buffalo Haiku Generator does the same thing in haiku form, naturally.
5. Podcaster, the list of Everything2 Podcast episodes, is largely new.
6. You can now see when a given e2node was made, by hovering your mouse over where it says 'Created by'.

For countless minor tweaks, see Patch Manager.