[2023-03-12] Infrastructure RecoveryPersistent Universe Electronic AccessPartial Outage
Since the launch of Star Citizen Alpha 3.18.0 our teams have been monitoring a number of issues players have been encountering while attempting to log into the game. These have primarily manifested in the form of 19003 and 19004 with a mix of other similar codes.
While monitoring those issues, the team has been able to tune and make adjustments to the entitlement processing flow to ensure that a large portion of player attempts were still successful and able to make it into the PU.
However, at a point in the early morning UTC, the environment entered into a state which would require a more disruptive recovery process.
As of 1300 UTC on 2023-03-12 the environment has been in the process of recovering from the slow / non-responsive state it was in but we wanted to advise that, while this recovery effort is in-progress, players will experience periods of extreme difficulty getting into the PU. We expect players to see a mix of errors from the 19k, 30k, and 60k ranges.
A more detailed breakdown of each specific issue we're tracking can be found on the the RSI Knowledge Base
0055 UTC - The team is working through the various causes of error code 30009 which is preventing some players from getting into the game.
While that work continues, Star Citizen Alpha 3.18.1 has been made available on the PTU Environment for all backers.
In addition to a number of crash fixes and improvements, this patch brings the new vehicle delivery changes which were first mentioned here,
For more information about Star Citizen Alpha 3.18.1 including this weekend's playtest schedule, please check out this thread on spectrum
1300 UTC - Environment entered into recovery.
1510 UTC - Initial Notice, recovery in-progress.
1815 UTC - Shards are slowly coming online. Recovery still ongoing.
2000 UTC - Reducing severity of outage, will continue to monitor recovery.
1130 UTC - Escalating to Major Outage.
The team is working to recover the environment as the Global Database entered a bad state.
1200 UTC - Temporarily disabling Character Repair to protect accounts while recovery is in-progress.
1245 UTC - Environment is accepting new players.
1304 UTC - Character Repair access restored.
1930 UTC - Team is working to mitigate increasing error rates. Character Repair temporarily disabled.
0350 UTC - Reducing severity to partial outage while we validate the outcome of infrastructure maintenance.
0530 UTC - While the Live Environment continues to see periodic waves of instability, overall, the new infrastructure is trending in the right direction.
A new build with a series of fixes, based off analysis of the data collected since SC Alpha 3.18.0 went live, is currently being validated by QA.
Additional updates on this subject will be posted here and on spectrum as they become available.
1620 UTC - After rolling out a series of Server-side & Services hotfixes, a client hotfix has been tested alongside the Evocati Test Flight. We are now beginning our rollout of that client hotfix build to everyone to address a few key issues. More information about these fixes can be found in the updated SC Alpha 3.18.0 Patch Notes on Spectrum
In addition to this the team continues to work on a larger set of changes & fixes for SC Alpha 3.18.1, parts of which are already being validated by QA.
More information about some of these changes can be found on Spectrum here.
2117 UTC - The team is working to address an above average lag in the Global DB. While the system is catching up, players can expect to see more errors than usual.
[2023-03-23 Updates] 1450 UTC - A series of hotfixes have been applied to the Entity Graph to reduce the chances of requests being lost in transit.