I want to give an update as to the state of the E2 codebase as I have a fair bit of momentum and wanted to talk about where things are headed currently. This site is very much constrained by my free time, which as with many passion projects is at the mercy of jobs and children, and the rest of life. That said, there's a number of things in flux that impacts the code contribution community here, so I want to talk about those changes coming up shortly.
All of the code for the site is hosted in github. That's also where I'm taking bug requests and using as my work queue. Any changes made to elements that have left the database have to be done as pull requests (or feature requests and I'll do them), and we'll push to production that way. Deploys are relatively smooth currently. Reminder that we are on AWS so any terminology here refers to AWS-specific services.
Recent accomplishments
We're now on autoscaling/autohealing so if the application crashes then it instantly puts enough capacity back into the pool to get the site back up. It happens a few times a week due to a crasher I don't 100% currently understand but am tracking down. It takes about 45 minutes to come all the way back up due to the image rebuild time.
The site uses cloud-native logging. Meaning we don't capture anything from the internal apache instances anymore, but rather we just use the load balancer log delivery to S3. This is so, so much easier than occasionally wrangling logs and helps to stop issues where we'd be storing logs for eternity. Database backups are also done automatically by RDS so if something happens, I can defrost and restore the site without much fuss. Basically, the ops and admin work of logs is fully automated, so that's one thing off of my plate
Errors are now reported to CloudWatch, so that whenever there's a perl error or Server Error!/Hamster Twinkie Overdose, it'll get recorded with a full stacktrace to AWS. This means that I can poll out the top occurring errors and occasionally fix them. They are typically warnings or pretty obscure code paths right now; most of the mainline stuff has gotten paved over. Perl is relatively stable as a language so even as things get updated there's not much to chase down.
The code continues to move things out of the database to the next phase. There is a mostly-complete core of new Everything functions as Everything::Node, which is a Moose-based set of objects that allows me to clean up database functions and encapsulate new elements. There's also a set of controllers that enables heavy code reuse as Everything::Page. What that means is that instead of having superdocs, restricted_superdocs, and oppressor_superdocs, and superdocnolinks, we can simply have pages with different Everything::Security mixins (Everything::Security::NoGuest, Everything::Security::StaffOnly and Everything::Security::Permissive), and attributes that say whether the output of the template needs link parsing. This helps a lot and takes a lot of the complexity and branching out of the code path.
The Controller/View separation code also works really well. We are using Mason2 templates (also Moose-based) to make it so that the code takes structured data and simply executes a really-easy-to-understand view routine on them. The code for Silver Trinkets is a good example of how it looks. E2 Staff is another good demonstration of how clean those templates can get.
Development in the future
As of this writing there's around 200 more code elements to move out of the database, but once that's done, it's going to be a whole new developer cadence. Here's what that means:
No more sandboxes: The ability to execute code through the front-end is a terrible security issue that we're just eating, and at some point I'm going to throw away all of the sandboxes, mine included. If you'd like to do development, I'm going to be refreshing the guide for how to get E2 working locally in Docker, but basically it's checking out the code in git, installing Docker, and running the devbuild script. A cup of coffee later, you're in business.
No more patch usertype: Again, since there's no code in the database, I'm going to tear out the patch system, and make sure that everyone who has ever submitted a patch gets credit in the codebase. This involves deleting all of the patch nodes and those viewtypes.
Edev insights will change: There's some stuff that people can do in edev today like create javascript documents and such that are also security issues. I do want to encourage code contributions from the community, but it's hard to balance that with something like stealing login cookies so I'm going to be re-evaluating those elements
CSS/JS elements will be moving to the build system: Users who manage a zen stylesheet will have to submit pull requests to change them in the future. We're eventually going to move to a new unified frontend anyway, but in the interim, editing inside of E2 will also be shut off. Currently we do weird things like using yui-compressor in process to minify things, but we need to pull that over to the CI/CD pipeline.
Modernization Elements
Modernization looks like this:
- Move to a single CI/CD pipeline - We have a couple of CodeBuild pipelines that can take information from E2 and build it in a semi-automated fashion. I have the hooks working to see whenever a commit happens to GitHub so the shell is there, but I need to add more elements to it to make it a full pipeline. Once it is a single build that includes CloudFormation, assets, and code, then we will be able to have a staging site in another account that works the way we need it to.
- Move to containers - Currently E2 builds as a series of layers which makes an almost-production-workable container. I've moved all of the log processing and the "state" off of the servers so I don't need to log into anything. Any data operations are fully handled by lambda jobs which I can run out of the ops/ directory. This should drive a fair amount of cost savings as we can be more granular on the capacity we buy. I have a lot of cost modeling to figure out what right looks like there.
- Finish pulling code out of the database - As stated above, this work continues, but as more of the code gets modernized and encapsulated, I can start making more major feature elements happening without increasing technical debt. Right now this happens by pulling code into the Everything::Delegation modules where the Everything::HTML execution environment is emulated with local variables.
- Build a QA/staging environment: I break things every now and again without knowing, and that's because there isn't a true QA environment. The goal is to build one that automatically sees the code, tries to instantiate it, and runs a battery of integration tests against it. An upshot of this is that it'll let other developers assist in the process by giving a few people access to a staging branch for code changes that can be automatically deployed to verify changes as a part of a gitops workflow. More hands can help make the code better.
- Retire lambda: Right now, E2 lives inside of AWS Lambda, which is "serverless", and runs our chat update and other cron elements to build things like the sitemap, various reports, etc. AWS is pretty bad at keeping the internal images of Lambda stable, and we've had libraries disappear on us with no warning. We're going to keep the lambda cron drivers, but move the execution back into the container so we don't have to keep a separate baseline.
- Move code to templates - The templates force a separation between Controller and page renderer and start to provide a starting point to move to React. It also means that I have a spot to put on individual pages whether it is a React page or not. Everything that is a template has been refactored to not talk to the database as directly and uses Everything::Node constructs to help encapsulate and use different features.
- Move to React - The templates allow me to move to React on a page by page basis, which is a modern web framework. Instead of assembling heavyweight HTML pages, we can use the React rendering engine to build components that enable dynamic, state-driven web elements. We're heavily jQuery now, and it relies on a lot of quirks of how E2 returns from opcodes. There's an Everything::API code path now that has more modern interfaces, but React will allow us to clean-room those efforts and get to a modular framework that calls APIs and has good separation from the codebase. React was built to live inside of a CI/CD pipeline so that's why we need it to be available.
- The future! - Once we have the assets in the React pipeline we can refactor how the site looks pretty easily. This will enable us to truly tackle mobile in a way that Google will stop penalizing us for. This will drive traffic and revenue growth and help make sure we're set up for success in the future. We'll be less perl and more Node.js at that point but it'll make for a cleaner app with a modern toolchain that I don't have to build myself.
Thanks for reading the nerd update here, and as elements come together, I'll start to signal changes in how people's control knobs work here. Any questions, drop me a /msg