{
  "is": "issue",
  "title": "[2023-03-12] Infrastructure Recovery",
  "body": "\u003cp\u003eSince the launch of Star Citizen Alpha 3.18.0 our teams have been monitoring a number of issues players have been encountering while attempting to log into the game. These have primarily manifested in the form of 19003 and 19004 with a mix of other similar codes.\u003c/p\u003e\n\u003cp\u003eWhile monitoring those issues, the team has been able to tune and make adjustments to the entitlement processing flow to ensure that a large portion of player attempts were still successful and able to make it into the PU.\u003c/p\u003e\n\u003cp\u003eHowever, at a point in the early morning UTC, the environment entered into a state which would require a more disruptive recovery process.\u003c/p\u003e\n\u003cp\u003eAs of 1300 UTC on 2023-03-12 the environment has been in the process of recovering from the slow / non-responsive state it was in but we wanted to advise that, while this recovery effort is in-progress, players will experience periods of extreme difficulty getting into the PU. We expect players to see a mix of errors from the 19k, 30k, and 60k ranges.\u003c/p\u003e\n\u003cp\u003eA more detailed breakdown of each specific issue we\u0026rsquo;re tracking can be found on the the \u003ca href=\"https://support.robertsspaceindustries.com/hc/en-us/articles/360056254754-Star-Citizen-Alpha-3-18-Known-Issues\"\u003eRSI Knowledge Base\u003c/a\u003e\u003c/p\u003e\n\u003c!-- raw HTML omitted --\u003e\n\u003cp\u003e\u003cstrong\u003e[2023-04-12 Updates]\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e2240 UTC - Since the last major update, the team has been using the ptu environments to stress test a number of SC Alpha 3.18.1 builds. As mentioned on Spectrum, this has resulted in 50+ hotfixes and tuning adjustments aimed at bringing more stability to the live environment.\u003c/p\u003e\n\u003cp\u003eWhile there is still room for improvement, we believe the SC Alpha 3.18.1-live.8430497 to be substantially better than the current live experience.\u003c/p\u003e\n\u003cp\u003eAs we\u0026rsquo;re entering the maintenance window to bring 3.18.1 live, this incident has been set to resolved with any new issues for 3.18.1 tracked in a new incident.\u003c/p\u003e\n\u003c!-- raw HTML omitted --\u003e\n\u003c!-- raw HTML omitted --\u003e\n\u003cp\u003e[2023-03-12 Updates]*\u003c/p\u003e\n\u003cp\u003e1300 UTC - Environment entered into recovery.\u003c/p\u003e\n\u003cp\u003e1510 UTC - Initial Notice, recovery in-progress.\u003c/p\u003e\n\u003cp\u003e1815 UTC - Shards are slowly coming online. Recovery still ongoing.\u003c/p\u003e\n\u003cp\u003e2000 UTC - Reducing severity of outage, will continue to monitor recovery.\u003c/p\u003e\n\u003cp\u003e[2023-03-13 Updates]\u003c/p\u003e\n\u003cp\u003e1130 UTC - Escalating to Major Outage.\u003c/p\u003e\n\u003cp\u003eThe team is working to recover the environment as the Global Database entered a bad state.\u003c/p\u003e\n\u003cp\u003e1200 UTC - Temporarily disabling Character Repair to protect accounts while recovery is in-progress.\u003c/p\u003e\n\u003cp\u003e1245 UTC - Environment is accepting new players.\u003c/p\u003e\n\u003cp\u003e1304 UTC - Character Repair access restored.\u003c/p\u003e\n\u003cp\u003e1930 UTC - Team is working to mitigate increasing error rates. Character Repair temporarily disabled.\u003c/p\u003e\n\u003cp\u003e[2023-03-14 Updates]\u003c/p\u003e\n\u003cp\u003e0350 UTC - Reducing severity to partial outage while we validate the outcome of infrastructure maintenance.\u003c/p\u003e\n\u003cp\u003e[2023-03-16 Updates]\u003c/p\u003e\n\u003cp\u003e0530 UTC - While the Live Environment continues to see periodic waves of instability,  overall, the new infrastructure is trending in the right direction.\u003c/p\u003e\n\u003cp\u003eA new build with a series of fixes, based off analysis of the data collected since SC Alpha 3.18.0 went live, is currently being validated by QA.\u003c/p\u003e\n\u003cp\u003eAdditional updates on this subject will be posted here and on spectrum as they become available.\u003c/p\u003e\n\u003cp\u003e[2023-03-21 Updates]\u003c/p\u003e\n\u003cp\u003e1620 UTC - After rolling out a series of Server-side \u0026amp; Services hotfixes, a client hotfix has been tested alongside the Evocati Test Flight. We are now beginning our rollout of that client hotfix build to everyone to address a few key issues. More information about these fixes can be found in the updated \u003ca href=\"https://robertsspaceindustries.com/spectrum/community/SC/forum/190048/thread/star-citizen-alpha-3-18-0-live-8389318-patch-notes\"\u003eSC Alpha 3.18.0 Patch Notes on Spectrum\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eIn addition to this the team continues to work on a larger set of changes \u0026amp; fixes for SC Alpha 3.18.1, parts of which are already being validated by QA.\u003c/p\u003e\n\u003cp\u003eMore information about some of these changes can be found on \u003ca href=\"https://robertsspaceindustries.com/spectrum/community/SC/forum/1/thread/star-citizen-alpha-3-18-ship-claim-update\"\u003eSpectrum here.\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e2117 UTC - The team is working to address an above average lag in the Global DB. While the system is catching up, players can expect to see more errors than usual.\u003c/p\u003e\n\u003cp\u003e[2023-03-23 Updates]\u003c/p\u003e\n\u003cp\u003e1450 UTC - A series of hotfixes have been applied to the Entity Graph to reduce the chances of requests being lost in transit.\u003c/p\u003e\n\u003cp\u003e[2023-03-25 Updates]\u003c/p\u003e\n\u003cp\u003e0055 UTC - The team is working through the various causes of error code 30009 which is preventing some players from getting into the game.\u003c/p\u003e\n\u003cp\u003eWhile that work continues, Star Citizen Alpha 3.18.1 has been made available on the PTU Environment for all backers.\u003c/p\u003e\n\u003cp\u003eIn addition to a number of crash fixes and improvements, this patch brings the new vehicle delivery changes which were first mentioned \u003ca href=\"https://robertsspaceindustries.com/spectrum/community/SC/forum/1/thread/star-citizen-alpha-3-18-ship-claim-update\"\u003ehere,\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eFor more information about Star Citizen Alpha 3.18.1 including this weekend\u0026rsquo;s playtest schedule, please check out \u003ca href=\"https://robertsspaceindustries.com/spectrum/community/SC/forum/1/thread/star-citizen-alpha-3-18-1-update/5831234\"\u003ethis thread on spectrum\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[2023-03-27 Updates]\u003c/p\u003e\n\u003cp\u003e1700 UTC - Over the weekend, a series of hotfixes were deployed to the Live Environment to help reduce the number of players stuck with error code 30009 or otherwise unable to pass infinite loading.\u003c/p\u003e\n\u003cp\u003eThe majority of these issues were caused by the player\u0026rsquo;s character either:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eBeing flagged as unstowed when already stowed.\u003c/li\u003e\n\u003cli\u003eBeing flagged as created when the player, in fact, does not exist after a failed character repair.\u003c/li\u003e\n\u003cli\u003eBeing set as owned by the universe when aboard another player\u0026rsquo;s ship that was insurance claimed.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe work on the SC Alpha 3.18.1 continues and we hope to have this build published to Live this week.\u003c/p\u003e\n\u003cp\u003e[2023-04-11 Updates]\u003c/p\u003e\n\u003cp\u003e1530 UTC - The Live Environment is currently recovering from a temporary outage.\u003c/p\u003e\n\u003c!-- raw HTML omitted --\u003e\n",
  "createdAt": "2023-03-12 22:40:00 +0000 UTC",
  "lastMod": "2026-01-28 15:34:52 -0500 -0500",
  "permalink": "https://status.robertsspaceindustries.com/issues/2023-03-12_infrastructure-recovery/",
  "severity": "partial",
  "resolved": true,
  "informational": false,
  "resolvedAt": "2023-04-12 22:40:00",
  "affected": ["Persistent Universe", "Arena Commander"],
  "filename": "2023-03-12_infrastructure-recovery.md"
}