This is an OOC Post detailing technical issues about the State of the Game.
So. We’re all frustrated by the constant Net2 errors. What exactly is going on?
To TL;DR: The Interface Service between the MOULa “Authorization Server” (or Auth Server for short) and the Game Server and Database, is crashing/exiting, causing the Auth Server to boot everyone with Net2s while the service in question restarts.
We’re looking into it.
If technical stuff isn’t your game, you can stop reading here…
TECHNICAL TIMES AHOY!
…Still here? Okay, cool. This is going to be a bit longer, and while in more detail, it’s still a summary. I’m not going to go into extreme technical detail of every test we’ve performed.
So, that shortened situational summary down, what exactly are we dealing with?
Myst Online URU Live (Again) runs on an ORACLE Database, and its Game Server is a black-boxed piece of code that Cyan has not revealed to us in development. Shards like The GOW’s GEHN or OU’s MINKATA use DIRTSAND or MOSS, which are fan created Game Servers based on observation of Cyan’s running server code. These shards also use different Databases than MOULa’s Oracle.
The error code Chogon has provided us with comes from the server side crash reports, and is “0xebf00baa,” and it’s specifically an Oracle Database error. We’ve done the research, but nothing seems to really come up as to what this explicit error actually means in other situations where it’s cropped up.
WHAT WE’VE CHECKED…
We’ve sorted through a bunch of potential causes during this.
“Well, the net2s started in full after the update, was there something in there?”
To Begin with: It is very likely that this problem was beginning to crop up before the update that seemingly triggered the net2s to start, as a Net2 of this type was observed on August 7th, 2021, well before the update was installed, and thus, the crashes may not have anything to do with the update at all. Correlation =/= Causation.
Beyond that, just in case? The answer is ‘No.’ We’ve checked. There wasn’t anything in the update that we of the OU team provided to Cyan that modified how the Client interacts with any of the game servers in terms of what we’re seeing. So, was there something that did change on the server? Yes, and for a while, we suspected that was the cause and waited for a time it could be reverted. That reversion change was done, but the Net2s continued. So unfortunately it was not the cause.
What was that change targeted at, you may ask? Preventing the Overpopulation of DRC Hoods. That’s a whole other subject in and of itself, and I’m not going to get into it much here. It’s POSSIBLE that these net2s are related to the overcrowded DRC hoods, and I (Calum) Personally do believe there’s a significant chance they’re a contributing factor, but are they the *Only* factor? No, at this point I don’t believe they’re the only factor involved.
THAT SAID: Chogon *has* introduced a new server side fix that restored the original hood sorting measures, and we should not be seeing the DRC hoods growing beyond their current size status via automatic sorting. In the coming days we will be attempting to break up these overpopulated DRC hoods into smaller chunks, in hopes that this alleviates the net2 issue. But as I said, this is potentially not the root cause, but just *a* contributing factor. If the Net2s continue after this stage, and if we have not deduced an alternate potential cause before then, then it’s back to hunting for the cause of all of this.
Our next thought about this was that it was possible that a collision of client side interactions were causing these net2s. It was observed that net2s were seeming to be triggering when explorers were opening their KIs (which is an action that requests a LOT of data from the Servers), Pellet Runs (A score board altering action AND several rapid fire links), playing Ayoheek, or running Door Runs.
Over the last week, we tried several experiments of various natures on the MOULa Staging Server trying to see what happens when these actions happened in a controlled environment.
And On Saturday the 18th of September 2021, we performed several stress tests involving various permutations of potential server log spamming events… (Above Photo from said event) But despite all of that…
…Not much happened, to be honest. We had a few slowdowns but not much else. Despite all of that effort, we didn’t come across anything that could have triggered a Net2. How frustrating.
Unfortunately, that doesn’t rule out the possibility that these actions are triggering the net2s, but rather, simply exemplifies that we were performing these tests in too clean of an environment. Staging doesn’t have the historical age, data weight, or database size of active avatars on it that MOULa does. It also doesn’t have the potential of corrupted files in the database clogging things up and adding potential outside vectors outside of our control.
We’re unsure at this stage if these potential factors are, again, a causation, or merely correlation. More testing, specifically on MOULa itself, may be required at this juncture.
At this point in time, it seems HIGHLY UNLIKELY that a client triggered interaction is the cause of the net2 errors, which doesn’t give us a lot of options to work with. Please bare with us as we try to find out what the problem is.
SOO…. WHAT NOW?
We’re devising up potential solutions and trying to come up with new tests to figure out what’s going on. If all of this sounds tedious and frustrating, believe me, it is. It’s also incredibly stressful!
Please Remember that we are all Volunteers working on this, this is something we do out of love for the game. We want the Net2s to be ended as much as you do!
If you wish to join the discussions, or help us in testing, please join the Open Uru Discord channel.