We've had a (mostly) strong month on the server front except for the past few days - and I wanted to give an update on what has been going on behind the scenes. There has been a lot of improvements, including upgrading the way that our servers are built, and the way that we monitor what is causing the most lag and how we launch new features.

Server Building Continuing to iterate on the changes we made a few months ago, I've sped up the amount of time it takes for our servers to build after code is pushed. This goes to a service called DeployBot which compiles all of our code and bundles it, which then spins up servers containing that code. This means that when we push new code, an entirely new server instance is started with all of the most up to date packages.

Monitoring In order to get best information, I've put a lot of work into beefing up the monitoring available to us from all of our servers. This includes the amount of time it takes for you to load pages and what specific pieces are causing the most strain. We can dig down deeper and deeper to the exact lines of code that are taking the longest to load.

Caching Using the above information, we found that the caching solution we'd implemented for our avatar building service wasn't performing like we'd have liked, and so we were able to use that to rewrite the caching to include a file based component. It reduced the time on avatars by more than 50%, speeding up the whole site significantly.

We've been looking at other places to make these kind of changes and have had pretty good success so far!

Feature Flags Over the past week I've been setting up the plumbing for something called feature flags. This allows us to turn on features on the site over time, or turn them off easily without needing to push code. We can write a feature and push it to the site without it being visible, and slowly turn it on either by groups or target a percentage of users. Then, as it comes online we can monitor our charts shown above to see how it's performing before turning on to more users.

A basic example is the return of the forum pulse on the top of the forums, which I've turned on to just be visible to gold accounts and staff while I monitor that the queries aren't doing anything weird. I can then start adding a percentage of users who aren't GA/staff until we're showing it to 100% of the users.