In my last update I ended up sounding the alarm and talking a bit about the moves that we needed to make. The good news is that the first phase (stability) is now complete, and we are fully out of the woods. As I type this, the final parts of the performance burn-in are happening, and we will start to begin the next phase, which is the elimination of EC2 Classic. For the non-technical among us, we utilize Amazon web services to not be reliant on any generosity. E2 is a business, and is self-funding, and keeping costs down by using modern infrastructure is a huge benefit for the model and the continued survivability of the site
The major accomplishments here are pretty big and am happy to talk about them. First, we have moved over to AWS Opsworks to help manage and deploy our code on managed Ubuntu images, this is big because it means we can wind down our custom log shipping code and use the CloudWatch primitives for custom logs. This also means moving over from Debian Wheezy to Ubuntu Bionic and fully over to all of the modern versions of software available there. It will also make future upgrades considerably more simple.
The first phase of E2 has been moved over to about 80% "configuration as code". There are a few unmanaged resources which were created by hand, but those will get cut over as a part of phase two. It also means I could completely delete E2, and rebuild it from scratch in about ten minutes of Amazon time from a script. The push is to meant to make this sustainable beyond me, so all of these build scripts are a part of the E2 repository on Github, so should something ever happen to me and I am unable to look after the site anymore, the next maintainer can pick it up. Part of what I am focusing in on is disaster recovery plans for the business and the site, so that longer term operations can keep this piece of the web alive forever.
Certificates are now auto-managed by Amazon, which will alleviate a problem we saw twice with cron jobs not running properly and people getting TLS errors. It also means there is less to do, which is great, but it might cause problems.
Next, phase two begins some new cutover plans:
- Move the database from EC2 classic to the newer virtual datacenter that E2 lives in
- Eliminate the need for static secrets; the E2 webheads have inherited roles which can be used to access the appropriate cloud resources.
- Clean up the provisioning script to allow for migrations. Right now it only builds or destroys.
- Run some test deploys over code to make sure the process doesn't cause issues
- Scan the error logs for post-deployment issues
- Purchase reserved instances for a one-year term
More as I have it. Thank you all for what you contribute to this community. I'm very happy to help to my part technically to keep it alive.
--Jay