Category: "Chicago DC"

GeekStorage Primary DNS Server IP Change

October 12, 2012 at 3:04 PM

At this time we have brought online a new DNS server to function as our primary nameserver. This will require an IP modification at your registrar if you are using your own custom nameservers. If you are using nameservers provided & maintained by GeekStorage, you do not need to worry about this update. If you are using private nameservers, which is most common among our reseller client base, you will need to update the IP address associated with your NS1 subdomain as soon as possible.

If you are using custom nameservers, such as ns1.yourdomain.com and ns2.yourdomain.com, you will need to check if you are using 65.254.36.170 or 65.254.36.172 for your DNS. If so, you will need to update your domain registrar with the new IP address for your private nameserver. Nameservers using 208.100.54.x, 67.202.92.x, or 208.117.43.x do not require any modification.

If you are currently using 65.254.36.170 for one of your private nameservers, update the associated nameserver to the new IP: 67.202.92.4

If you are currently using 65.254.36.172 for one of your private nameservers, update the associated nameserver to the new IP: 67.202.92.6

These changes must be made before October 25th, 2012!

If you are using ns1.geekstorage.com & ns2.geekstorage.com or our white-label nameservers, you do not need to change anything!

For assistance with updating your nameserver IPs, should you need to make the adjustment noted above, please visit our knowledgebase video tutorial database for private nameserver registration at the link below.

http://kb.geekstorage.com/category/55/0/10/Video-Tutorials/Private-Name-Servers/

Xen HV3 Scheduled Maintenance

September 28, 2012 at 3:17 PM

We will be performing scheduled maintenance on HV3 due to an operations problem with the system. We will begin this maintenance at 7PM CDT on September 28th, 2012. We expect the downtime to be less than 20 minutes in duration. We apologize for any inconvenience this may cause.

This maintenance has been completed. Downtime was around 10 minutes.

Titan DDoS (Resolved)

September 18, 2012 at 2:27 PM

Titan is currently being targeted by a distributed denial of service (DDoS) attack, which is causing intermittent service interruptions to users on this server.

We are currently working to resolve this issue as quickly as possible.

09/18/2012 @ 03:46 PM CST

At this time, service should start returning to normal. Thank you for your patience while we worked to mitigate this incident.

Emergency Reboot HV3 (Resolved)

August 31, 2012 at 10:41 PM

08/31/2012 @ 10:36 PM CST

We are issuing an emergency reboot on the HV3 hypervisor to address a network issue that arose late this evening. We are seeing a halt in operation on the internal management network of HV3, causing many functions to be offline and inaccessible. We hope this reboot will resolve the issues. If we continue to see problems, we will need to prepare for further downtime to address a possible hardware issue. We will update this post when we know more.

08/31/2012 @ 10:46 PM CST

At this time the server has rebooted and VPS are coming back online. Estimated downtime for each VM on the server was less than 10 minutes. So far we have been able to confirm that it doesn't appear to be a hardware issue related to the server's NIC. We are still investigating the root cause of the network issue that occurred, which caused certain management functions to stop working. We do know that a mistake on a NOC tech's part led to the hypervisor's primary connection to the switch to be disconnected momentarily, causing VM's to go offline, as he was attempting to help us troubleshoot the original network issue.

Goliath Offline (Resolved)

August 29, 2012 at 9:55 AM

08/29/2012 @ 09:50 AM CST

We are actively working to resolve an issue with Goliath that is causing the server to be offline.

This post will be updated with more information as soon as it becomes available.

08/29/2012 @ 10:15 AM CST

We are seeing a kernel panic on Goliath. We are resolving the kernel issue at this time and we expect the server to be back online within 20 minutes.

08/29/2012 @ 11:15 AM CST

We are still working to resolve the kernel panic issue on the Goliath server. Websites will still appear offline as we work as quickly as possible to restore service. Our apologies for the inconvenience and you have our thanks for your continued patience.

08/19/2012 @ 12:20 PM CST

The root cause of the issue has been identified and corrected at this time. At this time, service has been restored.

More Details

We apologize for the significant downtime this morning. We have just resolved the issue and the system is coming back online at this time. You can expect your website to be accessible immediately.

We identified the issue initially as a kernel panic. Unfortunately, due to some problems with our debugging tools, our recovery time was delayed significantly versus typical similar situations. Once we were able to begin debugging the issue, it was identified as a filesystem configuration error or malfunction. Ultimately it was determined to be a filesystem configuration error on the primary (root) filesystem and it has now been resolved.

We would like to extend our deepest apologies for the downtime today. This downtime could not have been avoided; however, it could have been reduced significantly had our debug & recovery tools been more thoroughly tested prior to this issue. We are making corrections to our internal processes & recovery methods to eliminate extended downtime issues such as what occurred today.

We will offer SLA credits to all customers on the Goliath server upon request, up to 1 month service credit, per our SLA.

I would like to personally thank you for your continued patronage to our business and for your patience and understanding throughout the service outage today.

Regards,

Matt Eli