Recent Server Downtime

[Skully]

RECENT SERVER DOWNTIME

---

Prison Server: Online
Skyblock Server: Online
Survival Server: Online

Hey there!

You may be aware that UltraNetwork has been offline for the last 24 hours, we posted an announcement within our Discord server providing details as they become available regarding a situation with our server hosts.

 

Why was the server down?
We were made aware of an issue regarding an on-site fire with servers from our server hosts on the night of March 11th at roughly 1:00 GMT+0. The building where all of our servers were hosted was completely lost to the fire, along with the servers inside of it, this also means that we lost access to our consoles and at this time we don't know what the status is of the live server data.

Has any data been lost? Will I lose my progress?
All our servers are backed up daily around midnight GMT+1, the servers where backups are kept are off-site and seperate from the servers that were destroyed so everything is saved. We will be using the latest backup that we have available to restore on our new servers when they are available.

Why did it taking so long?
We had received replacement servers and everything was ready to go, however we were still missing our databases that contain essential information such as user account login details, punishments, game stats such as balance, prestige, survival lands, skyblock islands, etc. and without the databases we couldn't resume. Our databases are stored in a server building opposite the one that was lost to the fire and as a precaution, we were told the power to the building will remain offline until the 19th for safety. Once the building is available and power is switched on to the servers, we were be able to retrieve our databases and bring the server back online. Until then unfortunately we had no choice but to wait.

When will the server be back online?
Our current estimation for when the server will be restored is pinned at the top of this thread.

We sincerely apologize for the inconvenience, this is just an unfortunate event and a lot of this is out of our hands. This is the first time ever since UltraNetwork started nearly four years ago that we are experiencing such significant downtime as a result of a server error.

We will continue to do our best in order to get everything back online and running as soon as possible for you to enjoy, thank you for your patience.

_______________________________________________________________________________________________

Update posted on 26th March at 17:09 GMT+0. (Latest Update)

After two weeks of downtime, we're excited to announce that UltraNetwork is back online!

Both the Prison and Skyblock servers have been restored successfully with minimal data loss, the total duration of how far of a roll back occurred from restoring our latest backups cannot be accurately determined though it is within the range of 6-12 hours.

Survival Server
Whilst prison and skyblock are both back online, we're still waiting on a separate server within our host's data center to come back online in order for us to fetch the latest data for it, the survival server should be online within the next few days and we will continue to provide updates as they become available.

We'd like to thank all of our players for sticking with us during this extreme and unfortunate event - rest assured we have been working around the clock to bring our network back online for you to enjoy and will continue to do so for the foreseeable future. We apologize for the extensive delay in getting everything back online, the majority of this was out of our hands though moving forward, we are making adjustments to ensure that we never experience such a lengthy period of time offline ever again - this was our first, and last time being offline for over a week!

 

Update posted on 26th March at 12:30 GMT+0.

Spoiler

We are pleased to finally announce that we have obtained all of our database backups and are now in the process of restoring everything as quickly as possible! All servers are expected to be back online within the next few hours, a following announcement will be made later when everything is ready to go!

Update posted on 19th March at 21:46 GMT+0.

Spoiler

 

With the process of getting all the servers back online, we have received information from the data center that during the power cycling stages of all servers a lot of smoke was discovered coming from some servers, they have stopped the operation to start up servers for the day and will resume tomorrow.

Unfortunately they did not not get around to cycling our servers on today as we were hoping, we currently have no estimation as to when our data will be accessible in order for us to bring the servers back online. At this time, we can expect the servers to be back online within this upcoming week.

We apologize sincerely for this delay, we understand how frustrating this might be for some players and we are just as eager as you to have everything back online as soon as possible. Sadly we have no option but to wait as the backups we need to recover from our data center servers is the only copy available.

We will continue to provide updates within this channel and on the forums as more information becomes available, we thank you kindly for your patience!

 

 

Update posted on 19th March at 11:13 GMT+0.

Spoiler

The data center containing our server backups that we require for today's startup have begun a gradual restart of all servers and are preparing them to be restored for service, at this time we're unsure exactly at what point during today our data will become available for us to use before we launch the servers again.

A following announcement will be posted later today with an update as soon as we have more information, we appreciate your patience.

Update posted on 17th March at 05:14 GMT+0.

Spoiler

 

Hey players, just providing an update as to where we are with everything right now following the last update 6 days ago.

We have just gotten in touch with the host to re-confirm our plan to have everything up and ready for the 19th of March since we're still waiting on our database backups on the offline servers. In the meantime, a temporary server was mentioned in the last post however due to complications with the new server we've received, it's currently experiencing significantly high ping and connectivity issues which has delayed the release - because it has taken too much time already, it doesn't look like the server will be ready to release before all main servers come back online however we will continue working on it in the meantime so it can be released at a later date.

If you haven't done so already, get involved with the giveaways we're hosting in our Discord server! If you have any further questions or concerns you are more than welcome to send me a message on the forums or via Discord at any time. We hope to see you all back in-game over this weekend!

 

Update Posted on 11th March at 23:11 GMT+0.

Spoiler

 

We have heard back from our server host and all servers excluding the Skyblock server have been restored successfully. Whilst the servers are ready to go, we are now facing another issue.

Every one of our servers connects to a database which stores all core player information such as ranks, statistics, player data, login details, punishment history, etc. Unfortunately it turns out that all of our databases are in a server located in the building opposite to the one that caught fire and was destroyed, as a precaution the power to the building will not be turned on until the 19th of March and as a result, the server is inaccessible along with the databases on it.

So while all servers are ready to go, we just don't have our databases to connect them to at this time which is extremely unfortunate as we were hoping to have the server up and running by today. As a result of this, we have no choice but to delay opening the servers until the 19th of March when we regain access to our database. Rest assured however that we have been informed no data has been lost and everything should still be intact when being restored.

Because we are looking at significant downtime for the next 8 days, we don't want to leave our players without anything to play on, we understand a lot of our players love our network and the content we provide, so we will be working on creating a temporary server as soon as possible with a new gamemode while we wait. At this time, we're unsure exactly when we'll have this temporary server up though hopefully we have more information about this tomorrow. Along with this, we will be starting giveaways shortly as a thank you to our players for your patience.

 

Update posted on 11th March at 04:56 GMT+0.

Spoiler

The network proxy, lobby and prison servers have been transferred to different EU servers and have successfully been restored to the latest backup. The Skyblock server has been transferred though we are still working on restoring the latest available backup to minimize data loss. The survival server is still in the process of moving to an EU location. We hope to have everything done by today and the server should be online within the next 24 hours assuming everything goes to plan.

Update posted on 10th March at 22:20 GMT+0.

Spoiler

We are still hard at work collaborating with our host in order to transfer all files to our new servers and make adjustments as required to have everything up and running as soon as possible. Unfortunately due to the large number of services affected by this, there are multiple servers and services doing this at the time which is resulting in more delays as resources are being used constantly.

Update posted on 10th March at 10:23 GMT+0.

Spoiler

 

We have started the process of restoring all backups of each server to a server node in New Jersey, USA. Due to the significant size of all server files, this will take several hours to transfer and prepare, at this time fortunately we do believe the server will be back online later today.

Since we are moving to a different network node in the US, you can expect that we will experience similar downtime in a few days once our original European location becomes available again so we can transfer back.

 

Update posted on 10th March at 09:52 GMT+0.

Spoiler

We have received confirmation that the data center is a total loss as a result of the fire though fortunately we have backups of all server data with a minimal rollback, we are working with our host to find a temporarily solution in terms of hosting our servers in a different location until the situation is resolved.

 

[flash]

I hope it will be online back soon!

[axzdbd]

i know you all have tried your best and i also hope the server comes back soon! gl

[Skully]

[Update 11/03/21 04:56 GMT+0]

The network proxy, lobby and prison servers have been transferred to different EU servers and have successfully been restored to the latest backup. The Skyblock server has been transferred though we are still working on restoring the latest available backup to minimize data loss. The survival server is still in the process of moving to an EU location. We hope to have everything done by today and the server should be online within the next 24 hours assuming everything goes to plan.

[Shepler]

I hope the server was backed up after i made the new wackdonalds

[michaeltepper]

Yes Thank You Skully

[Lil222]

we believe in you Skully! 

[OscarReyes24]

Thank You Skully 

[Skully]

[Update 11/03/21 23:11 GMT+0] We have heard back from our server host and all servers excluding the Skyblock server have been restored successfully. Whilst the servers are ready to go, we are now facing another issue.

Every one of our servers connects to a database which stores all core player information such as ranks, statistics, player data, login details, punishment history, etc. Unfortunately it turns out that all of our databases are in a server located in the building opposite to the one that caught fire and was destroyed, as a precaution the power to the building will not be turned on until the 19th of March and as a result, the server is inaccessible along with the databases on it.

So while all servers are ready to go, we just don't have our databases to connect them to at this time which is extremely unfortunate as we were hoping to have the server up and running by today. As a result of this, we have no choice but to delay opening the servers until the 19th of March when we regain access to our database. Rest assured however that we have been informed no data has been lost and everything should still be intact when being restored.

Because we are looking at significant downtime for the next 8 days, we don't want to leave our players without anything to play on, we understand a lot of our players love our network and the content we provide, so we will be working on creating a temporary server as soon as possible with a new gamemode while we wait. At this time, we're unsure exactly when we'll have this temporary server up though hopefully we have more information about this tomorrow. Along with this, we will be starting giveaways shortly as a thank you to our players for your patience.

[TitaniumGem]

So how long until you make the temporary server ?

 

[Jae]

Pog, thank you for the hard work! Few days more, server will be back and kickin' alive!

[Skully]

[Update 17/03/21 05:14 GMT+0] Hey players, just providing an update as to where we are with everything right now following the last update 6 days ago.

We have just gotten in touch with the host to re-confirm our plan to have everything up and ready for the 19th of March since we're still waiting on our database backups on the offline servers. In the meantime, a temporary server was mentioned in the last post however due to complications with the new server we've received, it's currently experiencing significantly high ping and connectivity issues which has delayed the release - because it has taken too much time already, it doesn't look like the server will be ready to release before all main servers come back online however we will continue working on it in the meantime so it can be released at a later date.

If you haven't done so already, get involved with the giveaways we're hosting in our Discord server! If you have any further questions or concerns you are more than welcome to send me a message on the forums or via Discord at any time. We hope to see you all back in-game over this weekend!

[Patrosse_Ops]

wow i hope this backup would work and hope the server would still be good

 

[Skully]

[Update 19/03/21 11:13 GMT+0]
The data center containing our server backups that we require for today's startup have begun a gradual restart of all servers and are preparing them to be restored for service, at this time we're unsure exactly at what point during today our data will become available for us to use before we launch the servers again.

A following announcement will be posted later today with an update as soon as we have more information, we appreciate your patience.

[Skully]

[Update 19/03/21 21:46 GMT+0]
With the process of getting all the servers back online, we have received information from the data center that during the power cycling stages of all servers a lot of smoke was discovered coming from some servers, they have stopped the operation to start up servers for the day and will resume tomorrow.

Unfortunately they did not not get around to cycling our servers on today as we were hoping, we currently have no estimation as to when our data will be accessible in order for us to bring the servers back online. At this time, we can expect the servers to be back online within this upcoming week.

We apologize sincerely for this delay, we understand how frustrating this might be for some players and we are just as eager as you to have everything back online as soon as possible. Sadly we have no option but to wait as the backups we need to recover from our data center servers is the only copy available.

We will continue to provide updates within this channel and on the forums as more information becomes available, we thank you kindly for your patience!

[BananaSlicer]

No wonder I didn't see the server popping up, but hopefully the issue gets resolved.

[Llamers]

Is the server back up yet im not at a computer and cant check im on my phone

[Skully]

[Update 26/03/21 12:30 GMT+0] We are pleased to finally announce that we have obtained all of our database backups and are now in the process of restoring everything as quickly as possible! All servers are expected to be back online within the next few hours, a following announcement will be made later when everything is ready to go!

[Skully]

[Update 26/03/21 17:09 GMT+0] (Final Update)

After two weeks of downtime, we're excited to announce that UltraNetwork is back online!

Both the Prison and Skyblock servers have been restored successfully with minimal data loss, the total duration of how far of a roll back occurred from restoring our latest backups cannot be accurately determined though it is within the range of 6-12 hours.

Survival Server
Whilst prison and skyblock are both back online, we're still waiting on a separate server within our host's data center to come back online in order for us to fetch the latest data for it, the survival server should be online within the next few days and we will continue to provide updates as they become available.

We'd like to thank all of our players for sticking with us during this extreme and unfortunate event - rest assured we have been working around the clock to bring our network back online for you to enjoy and will continue to do so for the foreseeable future. We apologize for the extensive delay in getting everything back online, the majority of this was out of our hands though moving forward, we are making adjustments to ensure that we never experience such a lengthy period of time offline ever again - this was our first, and last time being offline for over a week!