Server Update
Filter
Show Official Posts OnlyReplies
Hey everyone! We had a pretty significant downtime last week, and I've been meaning to write up an update of how that went.
Move to Google Cloud Unfortunately, this was unsuccessful. I'd written a lot of internal documentation on the move, and had all of the steps prepared. Once we moved Subeta and the database however, it was clear that the stability isn't where I'd have liked it, and that forced me to revert early in the morning. It's possible that we'll try this again, but instead I've decided to focus on making our AWS setup more robust.
What does that mean? Upcoming steps include:
Forum API rewrite Not really a "rewrite", but looking at how our forum data is used. There are a lot of strong forum packages out there that could handle the backend while we focus on the things that are unique to Subeta (the front end, displaying avatars, events, etc). This would mean significantly less work for us to make changes to the forums, and better cache handling.
I'm investigating this now, and have a branch using this new API. It'll be well tested and likely put up on a temporary server for all of you to battle test before we push it live.
Proxy layer Our database is the major bottleneck right now. There are a lot of ways to improve this and one of those ways is to put a proxy in front of it. I'm currently testing this on 1% of users, making sure it doesn't introduce any new errors.
The benefit of this is:
- Adding caching directly at the MySQL level. For queries that are run frequently (things like getting user details in the hovers, checking if someone is a minimod, etc) but almost never change, we can cache before it even gets to our code.
- Allow us to see all of the queries running through the database, and run better analyzing on them. Here is an example of the 20 queries over the last 5 minutes that took the most time. You'll see a really simple query that checks if someone is a minimod is run the most, which allows us to cache that (it's part of the filter - should they have a purple icon) and speed up a lot of page loads.
- Allows us to add new "secondary" servers more easily in the future. I've attempted this a few times through code and it's been difficult (and broken the site 😭) but proxysql has significantly better targeting for doing this. In the past we'd just make our single database stronger and more powerful, but that's not the best way to approach scaling up. With this, we can add secondary servers that are only for reading data during events and plots which can really speed everything up. 80% of the database load comes from reading data (like all of your vault or wardrobe items).
So I'm actively working on this now, and will continue to increase the number of users who are put through the proxy instead of connecting directly.
Other Changes
Through the ProxySQL and better reporting we get an idea of the slowest parts of the site and continue to rewrite some of those. An example is this morning I saw that the code that gives you items around the site (a general use function we call give_item(itemid) hasn't been updated in 5 years and uses an old syntax. Now it's updated!
tldr We aren't moving to google cloud, but are making improvements to speed up the site. I've also started working on a ✨ new feature ✨ for site wide events ;)
💖 ✨ 🤗
Thanks for the update and all the hard work everyone is putting in!
I am not old. I am gracefully aged
Is this the reason behind item losses and all the errors after the server work?
Thank you for the update!
My Wishlist
Holiday and Event Codes
[egg=Outlandish]
[tp=Outlandish]
Nope, that comes from me trying to restore the lost smails :(
💖 ✨ 🤗
Thank you for all the hard work, best wishes for the future!
Thanks for the update and and letting us know about this.
Ohhh, sh.. :-(
Thanks for the update
I don't really understand what your doing, but have noticed something and I'm wondering if its related.
In the past when there were errors - when buying, or getting game/quest rewards etc. - we'd still get the items involved in spite of the errors. Now, however, I'm not getting the items when there's an error. Additionally I'm getting screwed over because its still taking my sP, or quest items and so forth and not giving me a reward.
For example, I was buying scratchcards today. It gave me an error and I didn't get my scratchcard. It took my 1,000 sP AND gave me the 5 minute block on buying another. Now, I really don't care about the sP - but that 5 minutes, well I'd kill to get that back! If this happens only once, its no big deal, but if it happens regularly through the day and I'm also delayed in buying because of 504s, then it does really start to add up.
In loving memory of Need posting achievements?Then join Posting Frenzy Achievement Items - searchable list List of Borders and cutouts New at Fresh and Flavorful Ping Group
lol perfect timing for the site to go down UGH.
Unrelated to these posts 😭
💖 ✨ 🤗
username, userid, usertype, accounttype, frozen from user where username = BoaConstrictor limit 1) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Database/Connection.php(624): IlluminateDatabaseConnection->runQueryCallback(&;select usernam...', Array, Object(Closure)) #1 /var/www/html/s/vendor/laravel/framework/src/Illuminate/Database/Connection.php(333): IlluminateDatabaseConnection->run('select usernam...&;, Array, Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2027): IlluminateDatabaseConnection->select(&;select `usernam...&;, Array, true) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2015): IlluminateDatabaseQueryBuilder->runSelect() /var/www/html/s/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2499): IlluminateDatabaseQueryBuilder->IlluminateDatabaseQuery{closure}() /var/www/html/s/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2016): IlluminateDatabaseQueryBuilder->onceWithColumns(Array, Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Database/Concerns/BuildsQueries.php(77): IlluminateDatabaseQueryBuilder->get(Array) /var/www/html/s/vendor/agogeinc/subeta-filter/Filter/SubetaFilter.php(520): IlluminateDatabaseQueryBuilder->first() /var/www/html/s/vendor/agogeinc/subeta-filter/Filter/SubetaFilter.php(493): SubetaFilterSubetaFilter->parseUser(Array, &;&;, false) /var/www/html/s/vendor/agogeinc/subeta-filter/vendor/mjohnson/decoda/src/Decoda/Decoda.php(1718): SubetaFilterSubetaFilter->parse(Array, &;&;) /var/www/html/s/vendor/agogeinc/subeta-filter/vendor/mjohnson/decoda/src/Decoda/Decoda.php(694): DecodaDecoda->_parse(Array) /var/www/html/s/vendor/agogeinc/subeta-filter/Filter.php(68): DecodaDecoda->parse() /var/www/html/s/app/Http/Controllers/ForumsController.php(82): SubetaFilterFilter->filterText(&;[userping=BoaCo...&;) /var/www/html/s/app/Http/Controllers/ForumsController.php(213): AppHttpControllersForumsController->linkPings(&;...&;, Object(SubetaFilterFilter)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Cache/Repository.php(340): AppHttpControllersForumsController->AppHttpControllers{closure}() /var/www/html/s/vendor/laravel/framework/src/Illuminate/Cache/CacheManager.php(323): IlluminateCacheRepository->remember(&;forums:text:625...&;, 25, Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(223): IlluminateCacheCacheManager->__call(&;remember&;, Array) /var/www/html/s/app/Http/Controllers/ForumsController.php(216): IlluminateSupportFacadesFacade::__callStatic(&;remember&;, Array) [internal function]: AppHttpControllersForumsController->getPosts(894276, Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): call_user_func_array(Array, Array) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(45): IlluminateRoutingController->callAction(&;getPosts&;, Array) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Route.php(212): IlluminateRoutingControllerDispatcher->dispatch(Object(IlluminateRoutingRoute), Object(AppHttpControllersForumsController), &;getPosts&;) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Route.php(169): IlluminateRoutingRoute->runController() /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Router.php(679): IlluminateRoutingRoute->run() /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(30): IlluminateRoutingRouter->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/app/Http/Middleware/SubetaLogin.php(30): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(151): AppHttpMiddlewareSubetaLogin->handle(Object(IlluminateHttpRequest), Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): IlluminatePipelinePipeline->IlluminatePipeline{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php(41): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(151): IlluminateRoutingMiddlewareSubstituteBindings->handle(Object(IlluminateHttpRequest), Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): IlluminatePipelinePipeline->IlluminatePipeline{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/VerifyCsrfToken.php(75): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(151): IlluminateFoundationHttpMiddlewareVerifyCsrfToken->handle(Object(IlluminateHttpRequest), Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): IlluminatePipelinePipeline->IlluminatePipeline{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php(49): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(151): IlluminateViewMiddlewareShareErrorsFromSession->handle(Object(IlluminateHttpRequest), Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): IlluminatePipelinePipeline->IlluminatePipeline{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(63): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(151): IlluminateSessionMiddlewareStartSession->handle(Object(IlluminateHttpRequest), Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): IlluminatePipelinePipeline->IlluminatePipeline{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php(37): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(151): IlluminateCookieMiddlewareAddQueuedCookiesToResponse->handle(Object(IlluminateHttpRequest), Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): IlluminatePipelinePipeline->IlluminatePipeline{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php(66): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(151): IlluminateCookieMiddlewareEncryptCookies->handle(Object(IlluminateHttpRequest), Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): IlluminatePipelinePipeline->IlluminatePipeline{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(104): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Router.php(681): IlluminatePipelinePipeline->then(Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Router.php(656): IlluminateRoutingRouter->runRouteWithinStack(Object(IlluminateRoutingRoute), Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Router.php(622): IlluminateRoutingRouter->runRoute(Object(IlluminateHttpRequest), Object(IlluminateRoutingRoute)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Router.php(611): IlluminateRoutingRouter->dispatchToRoute(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(176): IlluminateRoutingRouter->dispatch(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(30): IlluminateFoundationHttpKernel->IlluminateFoundationHttp{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/fideloper/proxy/src/TrustProxies.php(57): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(151): FideloperProxyTrustProxies->handle(Object(IlluminateHttpRequest), Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): IlluminatePipelinePipeline->IlluminatePipeline{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(31): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(151): IlluminateFoundationHttpMiddlewareTransformsRequest->handle(Object(IlluminateHttpRequest), Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): IlluminatePipelinePipeline->IlluminatePipeline{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(31): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(151): IlluminateFoundationHttpMiddlewareTransformsRequest->handle(Object(IlluminateHttpRequest), Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): IlluminatePipelinePipeline->IlluminatePipeline{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php(27): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(151): IlluminateFoundationHttpMiddlewareValidatePostSize->handle(Object(IlluminateHttpRequest), Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): IlluminatePipelinePipeline->IlluminatePipeline{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/CheckForMaintenanceMode.php(62): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(151): IlluminateFoundationHttpMiddlewareCheckForMaintenanceMode->handle(Object(IlluminateHttpRequest), Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): IlluminatePipelinePipeline->IlluminatePipeline{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/app/Http/Middleware/CorsMiddleware.php(21): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(151): AppHttpMiddlewareCorsMiddleware->handle(Object(IlluminateHttpRequest), Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): IlluminatePipelinePipeline->IlluminatePipeline{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(104): IlluminateRoutingPipeline->IlluminateRouting{closure}(Object(IlluminateHttpRequest)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(151): IlluminatePipelinePipeline->then(Object(Closure)) /var/www/html/s/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(116): IlluminateFoundationHttpKernel->sendRequestThroughRouter(Object(IlluminateHttpRequest)) /var/www/html/s/public/index.php(55): IlluminateFoundationHttpKernel->handle(Object(IlluminateHttpRequest)) {main} Nope, that comes from me trying to restore the lost smails 😢
useful threadsThe Giant List of Usability and Random Improvement 2.0 Comprehensive Guide to Battle Opponents (v2) [topic not found]
|
[topic not found] |
All the sql quote is gone now, just Keith's info left in his answer to me.
I lost an item from my shop. I have it in there still "amount 0".
<p> I don&;t really understand what your doing, but have noticed something and I&;m wondering if its related.</p>
<p>In the past when there were errors - when buying, or getting game/quest rewards etc. - we&;d still get the items involved in spite of the errors. Now, however, I&;m not getting the items when there&;s an error. Additionally I&;m getting screwed over because its still taking my sP, or quest items and so forth and not giving me a reward.</p>
<p>For example, I was buying scratchcards today. It gave me an error and I didn&;t get my scratchcard. It took my 1,000 sP AND gave me the 5 minute block on buying another. Now, I really don&;t care about the sP - but that 5 minutes, well I&;d kill to get that back! If this happens only once, its no big deal, but if it happens regularly through the day and I&;m also delayed in buying because of 504s, then it does really start to add up.
Yup - this comes from the 5 year old give_item code I mentioned. The technical explanation is that we have an auto increment on that table, which increases +1 for each item put in user inventories (that's where the id=23849723489723947 comes from when you click on an item in your inventory). Previously this code was creating a blank number that MySQL would interpret and add the number itself. The proper way to do it is to not send that at all.
You were likely added into the group that used the proxy, and because the proxy didn't accept that code (the blank number) it would error, not giving you the item. The function has been rewritten to adhere to 2018 standards now, so that error won't happen to you again even if you get put back in the pool for the proxy.
I understand the frustration and the losing sP, and advice making a ticket for it :( But it won't happen again!
💖 ✨ 🤗
Thanks for the reply .
I'm good since it won't happen regularly :D.
In loving memory of Need posting achievements?Then join Posting Frenzy Achievement Items - searchable list List of Borders and cutouts New at Fresh and Flavorful Ping Group
Great architecting, Keith. Thanks for the rundown on all the work you've been doing!
I'm surprised the Google cloud is so inadequate stability-wise, but that's been what I've found for the Amazon cloud in my own work, so I shouldn't be. :)
Keith, not sure you fixed the item going missing thing yet. Just trying to open a ticket about it now:
ticket text
I just received a Dungarde Gauntlet Armor from the donation corner.
When I tried to put it in my wardrobe, I got this error:
SQL Error
SQL: INSERT INTO item_instances ( locid, ownerid , itemid ) VALUES ('2', '675292', '210600')
subeta_subeta subeta_subeta
Then the item was gone. Not in my wardrobe, and not in my inventory either. Account search turned up nothing.
The ticket does not seem to be being created with the above text, though. Have tried twice. It finally let me create the ticket! yay
Well I just had the weirdest thing happen, not sure if it's related to errors or just a weird hiccup. I tried to comment on a thread related to fireside and when I hit post, it took me to this page:
https://subeta.net/forums.php/read///1/
A thread titled "Profile HTML Help"
My post didn't go through because that thread is locked... I went back to the fireside thread and my post wasn't there so I tried again. It went through that time. o_O
he/him / 31 / EST |  | My best friend is |
Same happened to me.
Moving item to wardrobe now missing
I was moving a Delish Can Mascara to my wardrobe. Now item is missing from my account
SQL Error
SQL: INSERT INTO `item_instances` ( `locid`, `ownerid` , `itemid` ) VALUES ('2', '336084', '181390')
subeta_subeta subeta_subeta
[Kiss=Trubble] [Flower=Trubble] [ToT=Trubble]
Before the site went down I had both the above issues. Tried to post here about a wardrobe item also disappearing when adding to wardrobe and the error I got and after posting it went to a locked thread and my post never made it to this thread. Super odd error.
“Justice will not be served until those who are unaffected are as outraged as those who are.” - Benjamin Franklin
"Woe to those who make unjust laws, to those who issue oppressive decrees, to deprive the poor of their rights and withhold justice from the oppressed of my people, making widows their prey and robbing the orphan." - Isaiah 10:1-2