NGS instability due to a bad deploy

Resolved
Resolved

At 17:55 UTC (1.55pm US/Eastern, 10.55am US/Western) we started the deploy of a new version of the nats-server to NGS; this completed at 18:19 UTC.

Unfortunately, under load in NGS we found the server crashing, in a scenario not caught in tests or our Test environment. These issues were spotted at 18:25 UTC (within 6 minutes). Investigation proceeded and at 18:42 UTC the call was made to roll back to the previous version.

The rollback completed at 19:11 UTC, fully restoring service.

Began at:

Affected components
  • NGS
    • GCloud
      • JetStream GCloud
      • Connectivity GCloud
    • AWS
      • JetStream AWS
      • Connectivity AWS
    • Azure
      • JetStream Azure
      • Connectivity Azure