There are no Open Network Issues Currently
webm1/SmarterMail (Resolved) High

Affecting System - email - windows hosting - Smartermail

  • 2021/05/07 15:26
  • Last Updated 2021/05/07 16:26

This server is being restarted due to unscheduled maintenance required.

During this time - smtp75.entirelydigital.com and https://webm1.entirelydigital.com are not accessible.

--

Update - the second reboot is in progress.  Unfortunately there is no way to accelerate the process or interrupt it at this stage.  We must wait for the update process to complete.

--

Update - the process is now complete and normal services have resumed.

Exchange Servers Update (Resolved) Critical

Affecting System - Hosted Microsoft Exchange

  • 2021/03/03 15:00 - 2021/03/03 16:23
  • Last Updated 2021/03/06 12:06

All our hosted Exchange Servers are being patched immediately due to a zero-day exploit which is being used by malicious groups.

This is an out of band emergency maintenance window.

The patch is referenced here:

https://blogs.microsoft.com/on-the-issues/2021/03/02/new-nation-state-cyberattacks/

 

VPS Infrastructure Updates (Resolved) Medium

Affecting System - OpenVZ

  • 2020/12/18 22:00
  • Last Updated 2020/12/19 22:52

OpenVZ Virtual Server Hosts - all servers

  • Friday December 18 22:00 EST - Saturday December 19 02:00 EST
  • Saturday December 19 22:00 EST - Sunday December 20 02:00 EST

All services will be unavailable at certain times during this maintenance window as the hosts reboot. 

Maintenance items include but are not exclusive to:

  • Software updates
  • Kernel updates
  • Operating system package updates

Friday December 18 22:33 maintenance window opened

Friday December 18 23:47 maintenance window closed


Saturday December 19 22:16 maintenance window opened

Saturday December 19 22:49 maintenance window closed

DNS - Windows hosting (Resolved) Medium

Affecting System - ns1.entirelydigital.com, net0.bluecho.net

  • 2020/12/13 20:57 - 2020/12/13 22:46
  • Last Updated 2020/12/13 20:58

Unscheduled maintenance of this host is in progress...

Viper/Neiko (Resolved) Medium

Affecting System - Hosting - Windows - Viperhost/Neiko

  • 2020/12/13 20:56 - 2020/12/13 22:46
  • Last Updated 2020/12/13 20:57

Unscheduled maintenance of this host is in progress....

VPS host maintenance (Resolved) Medium

Affecting System - VMWARE hosts - some servers

  • 2020/11/28 22:00 - 2020/11/28 23:59
  • Last Updated 2020/11/29 00:15

Please note that we have scheduled maintenance on the following host server:
  VMWARE hosts - some servers
During the following time window: Friday November 28 22:00 EST - Friday November 28 24:00 EST

All services on this server will be unavailable at certain times during this maintenance window.  We do expect that the process will take much less time, however we have allowed for a the full 2 hours should there be unexpected downtime.  Please note that we will perform this update after a full backup of your VPS has been taken. During the maintenance your VPS will be shut down.

Maintenance items include but are not exclusive to: 

  • Software updates
  • Kernel updates
  • Operating system package updates

You can check back here for status updates.


Saturday Nov 28 22:30

Backup of all virtual servers is in progress and maintenance will proceed as scheculed


Saturday Nov 28 23:05

VPS servers are not available during the maintenance process which is now in progress.


Sunday Nov 29 00:05

Maintenance Complete

Unable to connect to backup server (Resolved) High

Affecting System - Offsite Backup

  • 2020/10/29 08:39 - 2020/10/30 10:22
  • Last Updated 2020/10/30 10:22

We are aware of an issue with our backup server whereby your backup agent cannot connect to our server.

We are actively working with the software vendor to resolve this issue.  Rest assured that your backup data is in tact and that once resolved, your agent will connect and continue where it left off.

--

This issue is now resolved

Server Down (Resolved) High

Affecting Server - ED-SH-CP7

  • 2020/08/26 02:39
  • Last Updated 2020/08/26 10:16

We are working on getting this server online - it is operational however we have network connectivity issues at the moment.

Server being reset (Resolved) High

Affecting Server - greywood.webnethost.net

  • 2020/07/24 15:56 - 2020/08/26 05:40
  • Last Updated 2020/07/24 15:56

This server was non-responsive and is being restarted.

Server Down (Resolved) High

Affecting Server - whitemountains.webnethost.net

  • 2020/06/23 19:28
  • Last Updated 2020/06/23 20:23

UPDATE - Resolved.

We are currently investigating the issue with this server - updates posted here.

 

MySQL not running (Resolved) High

Affecting Server - ED-SH-CP7

  • 2020/05/11 05:19 - 2020/06/23 19:29
  • Last Updated 2020/05/11 19:31

This issue is now resolved - all databases are loaded and sites working properly.  Should you have any issues, please login and open a support ticket.


The MySQL engine on this server is currently not running - we are actively investigating the cause.


Update - 11:00am - EST - we are restoring the mysql data structure from backup will update this network issue with it's progress


Update - 7:00pm - EST - the data has all been restored but we are seeing that local connections from websites are not possible.  We are working to resolve this one last issue and hope to have a resolution shortly.

Important Update (Resolved) High

Affecting Server - ED-SH-CP7

  • 2019/12/17 21:00 - 2019/12/18 01:23
  • Last Updated 2019/12/17 22:03

Server maintenance is now in progress...

Notification of critical maintenance required for this server.

Estimated Start:

December 17 @ 09:00 pm EST 

Estimated End: 

December 17 @ 12:00 am EST 

Estimated Duration: 

Approximately 3 Hours

Affected systems: 

All services on ED-SH-CP7.entirelydigital.com (Email and Web)

Impact: 

All services on this server will be unavailable during this window

Root cause: 

Server storage upgrade required

Contact Person: 

Please open a support ticket by logging in

Traffic Forwarding Stopped (Resolved) High

Affecting System - Traffic Forwarding Server

  • 2019/08/30 15:00 - 2019/09/06 14:32
  • Last Updated 2019/08/30 15:03

We have now stopped forwarding traffic from our old Datacenter to our new Datacenter IPs.

Our official stop date is August 31, however in order to transition properly and force check to see if anyone has not completed DNS updates we've stopped forwarding traffic.

If you need traffic forwarded for a specific IP, please open a support ticket.

Provide: old IP, new IP and the reason for forwarding and why you cannot complete an immediate DNS change.

ALL possibility to forward traffic stops without exception August 31 at 11:59pm.

Microsoft Exchange Email Down (Resolved) Critical

Affecting System - Microsoft Exchange Email

  • 2019/08/22 17:34
  • Last Updated 2019/08/23 09:21

We are experiencing problems with our Domain Controllers that authenticate you to Microsoft Exchange.

We are aggressively investigating the issue and hope to have a resolution shortly.

--

Update - we continue to work on this issue and hopeful to have the matter resolved soon

--

All services restored and queued emails delivered - no incoming email was lost.

 

Physical cPanel Server Migration (Resolved) Medium

Affecting System - Listed cPanel servers

  • 2019/06/30 09:00 - 2019/07/10 07:53
  • Last Updated 2019/06/30 12:57

The following servers have been scheduled for move to our new datacenter:

  • blackroot
  • ed-sh-cp8
  • whitemountains
  • greywood

Please check back here for migration updates.

---

12pm EST - Update - Servers have been moved and are now reachable on their new IP addresses - forwarding of traffic has started - please be patient as we add all the new IPs and enable all the forwarding rules.

---

12:57pm EST - Update - All IP / DNS changes are complete - forwarding is in place

Toronto DC1 (Resolved) Medium

Affecting System - Internet transit services

  • 2019/06/06 23:59 - 2019/06/27 14:06
  • Last Updated 2019/05/28 11:18

This message is to advise you of the following upcoming network maintenance work that could impact the Toronto DC1 transit link.

As a result of this maintenance Internet transit services for the Data Centre could be potentially impacted and you could experience periods of degradation of services (packet loss, connectivity drops) while upstream path re-convergence occurs.

Planned Start Date: June 6, 2019 12:00 AM (Eastern Time)
Planned End Date: June 6, 2019 06:00 AM (Eastern Time)

Impact Duration: Outage up to 30Min(s)

Scope of Work: Network gear upgrades

We apologize for any inconvenience that this may cause.

PayPal Payment Issue (Resolved) High

Affecting System - PayPal

  • 2019/05/21 14:12 - 2019/05/21 15:16
  • Last Updated 2019/05/21 14:13

We are unable to process PayPal payments at this time.  We are working with PayPal to resolve this issue and hope to have a resolution in place within the next 2 days.

We appreciate your patience.

Server Reboot In Progress (Resolved) Critical

Affecting Server - whitemountains.webnethost.net

  • 2019/05/15 14:19 - 2019/05/21 14:13
  • Last Updated 2019/05/15 14:22

A hung backup process has caused the server to be unstable and a reboot is required ad this time

---

Server has restarted and we are watching the boot process to ensure there are no errors.

Server Reboot In Progress (Resolved) High

Affecting Server - blackroot.redundant-webservers.com

  • 2019/05/07 08:42
  • Last Updated 2019/05/07 08:50

After fixing a hung update process, we are now rebooting this server

---

Reboot complete and all updates were successful.

Server restart required (Resolved) High

Affecting Server - whitemountains.webnethost.net

  • 2018/11/03 08:06 - 2019/01/23 12:03
  • Last Updated 2018/11/03 08:32

A critical BIOS update requires a restart.  Now in progress.

--

BIOS is complete, we are now completing other firmware updates and will restart once again.

cPanel server restarts (Resolved) High

Affecting System - cPanel servers

  • 2018/10/14 09:11 - 2018/10/14 10:37
  • Last Updated 2018/10/14 10:27

Multiple cPanel servers are being restarted for pending kernel updates.

Please be patient while they restart....

SmarterMail - Major Upgrade (Resolved) Medium

Affecting Server - ED-DNP01

  • 2018/09/24 20:00
  • Last Updated 2018/09/24 21:00

Smartermail servers are getting a major version upgrade including new servers on Windows 2016.  SmarterMail version will be updated to version 16 (from 13.x)

Here is the link to SmarterMail's release notes for your perusal: https://www.smartertools.com/smartermail/release-notes/16

Want to learn more about the new features: https://www.smartertools.com/smartermail/whats-new#/mail-content-current

Updates to the process will be posted here.

During the migration period the old mail server will be stopped - last bit of data migrated to the new server and then then mail server services are to resume normal operation on the new server.

---

Transfer has been started - all mail services are temporarily stopped.

---

Migration complete.

Restart in progress (Resolved) High

Affecting Server - whitemountains.webnethost.net

  • 2018/07/20 17:54
  • Last Updated 2018/07/20 17:59

This server is being restarted...

--

Server has restarted.  Please allow some time for all services to load.

Updates in progress (Resolved) Medium

Affecting System - R1Soft Backup Servers

  • 2018/07/10 10:27
  • Last Updated 2018/07/10 11:09

All R1Soft Backup Servers are currently being updated and have been put in maintenance mode.
During this time no restore/backup operations are permitted.

--

Kernel updates + package updates + R1Soft > 6.6.2 build

--

Complete

Kernel update in progress (Resolved) High

Affecting Server - cpanel43.onlinemountain.com

  • 2018/07/01 09:29
  • Last Updated 2018/07/01 10:52

Kernel update in progress - please be patient while the server restarts.

--

Update complete

Cpanel Servers - Kernel Updates (Resolved) Critical
  • 2018/05/26 16:31 - 2018/05/26 17:23
  • Last Updated 2018/05/26 16:41

We are updating seversl CPANEL servers kernels - as this is a critical security update
You will lose connectivity during this time - please be patient while your server restarts.

Network Manager Service Down (Resolved) High

Affecting Server - whitemountains.webnethost.net

  • 2018/03/31 10:30 - 2018/03/31 10:47
  • Last Updated 2018/03/31 10:47

A recent update has made changes to the network manager and broken connectivity to the network.  We are investigating the issue.

---

This issue is now resolved.
A line was added to the configuration file at the end:
ONBOOT=no
That will break it!

LiteSpeed WebServer being installed (Resolved) High

Affecting Server - cpanel43.onlinemountain.com

  • 2018/03/19 14:07 - 2018/03/20 13:45
  • Last Updated 2018/03/20 09:45

LiteSpeed WebServer being installed to reduce loads on this server.
In addtion CloudLinux MySQL governer is being installed.

---

Strang yum behaviour requires a restart.

High Loads (Resolved) Critical

Affecting Server - cpanel43.onlinemountain.com

  • 2018/03/15 19:26
  • Last Updated 2018/03/15 20:00

This server is under high load and we are investigating the cause.

--

WHM backups were enabled on this server in error causing high disk io.

We are investigating the cause.

--

We have killed the running processes and loads look normal

Critical updates in progress (Resolved) Critical

Affecting Server - cpanel43.onlinemountain.com

  • 2018/03/01 16:55 - 2018/03/01 19:03
  • Last Updated 2018/03/01 19:03

A critical kernel update is in progress.  Once the update is complete the server will be restarted.

--

reboot in progress

---

reboot complete

Meltdown and Spectre BIOS and CPU (Resolved) High

Affecting System - Sever BIOS Updates - Various Firmware Updates

  • 2018/01/30 14:40 - 2018/05/26 16:40
  • Last Updated 2018/03/01 17:01

IMPORTANT NOTICE: We are actively investigating the recently discovered CPU vulnerabilities known as “Meltdown” (CVE-2017-5754) and “Spectre” (CVE-2017-5715 and CVE-2017-5753). We are closely examining any developments around possible exploits of these vulnerabilities, and taking rigorous steps to roll out patches as they become available. 

99% of our severs are made by Dell.  We are subject to firmwares updates provided by Dell EMC.

Note that because of the number of servers involved and the time it takes to install firmware and kernel udpates, these will be unscheduled installs - sometimes during normal operating hours.
Restarts usually only approximately 5 minutes so there is little impact.

Critical updates in progress (Resolved) Critical

Affecting Server - ED-SH-CP8

  • 2018/02/28 18:22 - 2018/02/28 18:45
  • Last Updated 2018/02/28 18:23

Once updates are completed, this server will be restarted.

Server updates in progress (Resolved) Critical

Affecting Server - ED-SH-CP7

  • 2018/02/28 17:34 - 2018/02/28 18:07
  • Last Updated 2018/02/28 17:35

This server requires critical updates and will be restarted shortly

EasyApache 3 > 4 Migration (Resolved) High

Affecting Server - cpanel43.onlinemountain.com

  • 2018/02/27 10:34 - 2018/02/27 13:53
  • Last Updated 2018/02/27 13:22

New versions of PHP and Apache in progress - other checks in progress as well.

---

Migration has failed and we are re-running the conversion process.

---

Conversion is now complete and we are updating the apache configuration

EasyApache 3 > 4 Migration (Resolved) High

Affecting Server - cpanel42.onlinemountain.com

  • 2018/02/26 17:57 - 2018/02/27 12:05
  • Last Updated 2018/02/27 11:10

We are updating EasyApache on this server from version 3 to 4 to support PHP 7.0 and 7.1

--

Conversion has failed and a reinstall is in progress.  We apologize for the inconvenience.

---

Conversion is now complete and we are checking the apache configuration.

---

There is a conflict between CloudLinux and EA4 which we have escalated to cpanel and cloudlinux support.
Support ticket with cpanel and cloudlinux is in progress

cpanel41 - Apache Rebuild for CloudLinux (Resolved) High

Affecting Server - cpanel41.onlinemountain.com

  • 2018/01/29 12:01 - 2018/02/26 17:57
  • Last Updated 2018/02/01 10:15

Server is currently stable and we are monitoring it.  We are going to schedule a migration of all the accounts on this server to new hardware.

Current status for Apache:

Current Time: Thursday, 01-Feb-2018 10:13:43 EST
Restart Time: Tuesday, 30-Jan-2018 22:28:24 EST
Parent Server Config. Generation: 14
Parent Server MPM Generation: 13
Server uptime: 1 day 11 hours 45 minutes 18 seconds
Server load: 9.45 8.73 8.63
Total accesses: 1312316 - Total Traffic: 43.0 GB
CPU Usage: u158.86 s952.23 cu33175.6 cs8048.17 - 32.9% CPU load
10.2 requests/sec - 350.1 kB/second - 34.3 kB/request
28 requests currently being processed, 122 idle workers



We are trying one more build of apache to see if the issue subsides.
---
Apache running but loads are very high.  We are still looking at the reason.
---
Getting apache to behave has become more difficult than anticipated, even with Level3 cpanel support involved.  We are now making a full backup of this server and may have to restore to different hardware.
We will continue to update this notice. 
---
Currently evidence is pointing to a DDOS attack on the sites on this server.  We may have to wait until the attack subsides but we are investigating.  There are too many IP's involved to block anything significant.
---
We are now making several updates to see if we can absorb some of this traffic while mitigating some of it.  Server will require a few restarts.
---
Server has been restarted after a successful kernel update.  We are now running a firmware update on some components to see if there is a possible issue with our RAID controller firmware.
---
We will be switching to EA4 vs EA3 and ensuring that CloudLinux is limiting any offending users.
---
Conversion from EA3 to EA4 CloudLinux is now in progress
--
In order to speed up this process we are now blocking port 80 and 443 so that the conversion can complete without being bogged down by apache requests.
---
Starting EA3 to EA4 php.ini conversion in the background (EXPECT HEAVY CPU and I/O LOADS)
---
Conversion complete - we are now building the CloudLinux profile in EA4 - this is the last step.
---
MPM worker needs to be changed to MPM prefork for apache on this server for faster apache processing
---
This server is still under high load and we are investigating the cause.
---
By blocking port 80 and 443, server loads are now normal and we can investigate properly
during this time IMAP/POP and SMTP functions as well as FTP are still avaialbe but websites will not load.  Blocking these ports is necessary so that we can actively look at logs without being bogged down by high loads.
---
We have restarted the server - we have also opened a ticket with cpanel for assistance as this seems to be related to the EasyApache process of going from version 3 to 4 and the MPM configuration.
---
In order for cpanel to investiate, we've opened the ports back up but loads have spiked again - please bear with us while we figure out what's going on with Apache.

Server restart in progress (Resolved) High

Affecting Server - cpanel42.onlinemountain.com

  • 2018/01/28 16:35 - 2018/01/29 12:01
  • Last Updated 2018/01/28 16:36

Server restart in progress
Kernel updated
R1Soft backup agent installed for ehanced backups

High Loads - ongoing (Resolved) High

Affecting Server - cpanel43.onlinemountain.com

  • 2018/01/11 15:21 - 2018/01/24 12:57
  • Last Updated 2018/01/11 15:38

Server under high load.  We have observed this behaviour on/off for serveral days.
This seems to be related to apache only.
Note that in order to troubleshoot this, we will be turning port 80 and 443 off and back on
Please check this Network Issue ticket for udpated information.

---

We have noticed an entry in iptables that may be loading an extremely large number of IPs into our firewall, thus causing a high load.
We have disabled this list and are watching the loads.

High Loads (Resolved) High

Affecting Server - cpanel43.onlinemountain.com

  • 2017/12/20 20:09
  • Last Updated 2018/01/11 15:21

Server under high load.
--
Some services restarted with no effect.  Investigation continues
--
High loads seem to be MySQL related and we are investigating further
--
combination of exploit scanning with log processing and high mysql load has led to many queued processes.
we wil allow the scan to complete then continue to monitor.
--
we have found a hacked account causing high disk io.  This account has been suspended and files removed.
loads are normal
--
we are rebooting this server
--
server is booting up, we are monitoring

 
 

Imunify Firewall Problem (Resolved) High

Affecting Server - greywood.webnethost.net

  • 2017/12/26 12:11 - 2018/01/11 15:21
  • Last Updated 2017/12/26 12:12

An error in the Imunify Firewall has forced us to remove it.
We are now attempting a reinstall - note that Apache may become non-responsive for a few minutes.

Server High Load (Resolved) High

Affecting Server - cpanel41.onlinemountain.com

  • 2017/12/18 10:31 - 2017/12/20 20:11
  • Last Updated 2017/12/18 13:50

We are investigating the cause of this server's very high load.
Currently, no sites are loading.

--

Server is under high load because of a long running backup operation.
We will allow the backup to complete at this time after which the server loads should go back to normal.

 

MySQL databases being moved (Resolved) Critical

Affecting Server - cpanel42.onlinemountain.com

  • 2017/12/15 10:17
  • Last Updated 2017/12/15 11:09

In order to protect the integrity of MySQL data on this server, we are moving the data on an emergency basis to a new data drive.
Please be patient while the moving process completes.

--

We are about 33% through the copy process...

--

Almost there...

--

Task complete

 

Reseller Limits Not being Enforced (Resolved) High

Affecting System - cPanel/WHM - all versions

  • 2017/11/15 11:41 - 2017/12/18 13:50
  • Last Updated 2017/12/06 22:51

CPANEL-16366 internal case has been opened for this issue.
Updates to this case will be added to cpanel's  changelogs when they're available.
You can check them here:

https://documentation.cpanel.net/display/CL/68+Change+Log

--

This isssue is still not resolved.

 

Server not responsive (Resolved) Critical

Affecting Server - whitemountains.webnethost.net

  • 2017/12/06 22:08 - 2017/12/06 22:48
  • Last Updated 2017/12/06 22:08

We are investigating an issue with this server not responding.

Exchange 2010 - Maintenance Window (Resolved) Medium

Affecting System - Exchange 2010

  • 2017/10/21 17:17 - 2017/11/15 11:47
  • Last Updated 2017/10/21 17:34

We are installing updates to the Exchange infrastructure - you may lose mailbox connectivity while servers restart.  This is expected and service will be restored normally once the server starts up.
These are critical security patches and are integral to providing a secure environment.

---

Restarts are now in progress, please be patient while servers install their updates on restart.

Windows Hosting - Management System Down (Resolved) High

Affecting System - manage.entirelydigital.com

  • 2017/10/21 17:16 - 2017/11/15 11:47
  • Last Updated 2017/10/21 17:17

manage.entirelydigital.com is currently unavailable as we have found a possible vulnerability.  Any required changes should be submitted via support ticket until the portal becomes live once again.

Reboot in progress (Resolved) High

Affecting Server - ED-SH-CP7

  • 2017/09/25 15:21 - 2017/09/25 15:30
  • Last Updated 2017/09/25 15:22

The server is being restarted to install a critical kernel update.

Disk Space Issue (Resolved) Critical

Affecting Server - cpanel42.onlinemountain.com

  • 2017/09/22 09:25 - 2017/09/25 09:25
  • Last Updated 2017/09/22 09:26

We have found a disk space issue in / on this server and have cleared out some files.  We are now restarting the server to ensure all services are running normally.
ETA 10 minutes

Backup Drive Replacement (Resolved) Medium

Affecting Server - cpanel43.onlinemountain.com

  • 2017/08/29 21:52 - 2017/09/22 09:26
  • Last Updated 2017/08/30 10:17

We are rebooting this server to replace a backup drive.

--

Drive is being formatted - io is high so loads may be higher than normal.  In addition start up processes cause higher than normal loads.  Please exercise patience during this maintenance window.

--

We have started an out of schedule full backup which will impact DISK IO - during this time you will see higher than normal loads.  Please exercise patience during this maintenance window.

Apache will not start (Resolved) Critical

Affecting Server - cpanel43.onlinemountain.com

  • 2017/08/01 11:29
  • Last Updated 2017/08/01 13:55

Apache is not starting on this server due to multiple configuration issues from a failed cpanel update.
We have escalated this to cpanel support and they are currently investigating.

The issue has been resolved and sites should now be loading correctly with loads on the server back to normal.

High Loads - DOS Attack (Resolved) Critical

Affecting Server - cpanel42.onlinemountain.com

  • 2017/07/05 16:15 - 2017/07/06 10:03
  • Last Updated 2017/07/05 16:16

This server is under high loads and we are working to resolve.

High Loads - DOS Attack (Resolved) Critical

Affecting Server - ED-SH-CP7

  • 2017/07/05 16:15 - 2017/07/06 10:03
  • Last Updated 2017/07/05 16:15

This server is under high loads and we are working to resolve.

Server Restart in progress (Resolved) High

Affecting Server - cpanel41.onlinemountain.com

  • 2017/07/01 10:23 - 2017/07/01 10:36
  • Last Updated 2017/07/01 10:24

This server requires a reboot - now in progress....

MySQL Server down (Resolved) High

Affecting Server - cpanel41.onlinemountain.com

  • 2017/06/24 15:40 - 2017/06/26 11:35
  • Last Updated 2017/06/25 21:22

MySQL server cannot strat due to the mysql data drive mounted as read only.  We are investigating the cause and looking for a resolution.

We were able to get the drive back online and MySQL is now operating normally with no errors.
--

Reopeing this issue as some users are experiencing authentication errors.
We are working with cpanel to resolve.

Apache build failed (Resolved) High

Affecting Server - cpanel42.onlinemountain.com

  • 2017/06/12 21:11 - 2017/06/12 22:09
  • Last Updated 2017/06/12 21:12

Rebuild of apache has failed and we have escalated the issue to cpanel support

High Loads (Resolved) High

Affecting Server - cpanel41.onlinemountain.com

  • 2017/05/17 10:30 - 2017/06/12 21:12
  • Last Updated 2017/05/17 10:31

We are seeing extremely high loads on this server and are investigating the cause.

High Loads (Resolved) High

Affecting Server - cpanel42.onlinemountain.com

  • 2017/05/15 14:50
  • Last Updated 2017/05/16 17:22

This server is experiencing high loads and we are currently investigating possible causes.

UPDATE
-we have found some IP addresses that are attacking some wordpress sites on the server and are mitigating the effects.

UPDATE
-this server is once again under high loads.

UPDATE
-we have found the issue - a huge ip.pag file from the DDOS attack previously reported was causing high disk IO in apache which was causing a cascading effect and failure.
-should be ok now.

root compromised - backup/restore in progress (Resolved) High
  • 2017/04/24 15:54 - 2017/05/15 14:50
  • Last Updated 2017/04/25 13:06

Our security scans have revealed that this server is root compromised.  As such, we have started provisioning a new server (with a newer OS) and will:
1. Backup your cpanel account
2. Take the current suitland.webnethost.net server offline.
--during this time, your site and email will be offline.
3. Once the new server is online, restore your cpanel account to the new server.  Once your account is restored, your website and email will be online once again.
4. There will be no IP change to your domain.

This process may take 1-3 hours with intermittent connectivity to your site and email.  This is an urgent matter and must be addressed immediately.

Please check back for updates.

---

Apr 24 - 17:36 EST (GMT -5:00)

The new server is set up and the backups are complete. I will be taking the compromised server down shortly and putting the new server in it's place. I will then run scripts on the new server that will install your services. After the post-install scripts are finished I will begin restoring your accounts to their proper IP address.

I will update you on the progress as best I can. Please be patient, however, as this will take some time.

---

Apr 25 - 10:00 EST (GMT -5:00)

Account restores are complete - cpanel account passwords have all been reset.  We will advise your new password shortly.  Some MIVA Merchant sites on this server are not functional - we are working with MIVA support to get the miva components re-installed.

---

Apr 25 - 13:05 EST (GMT -5:00)

Please note that all cpanel passwords were reset to protect the integrity of your account.  Please login to your client area at https://www.entirelydigital.com/manage and then click on your services icon, then drill in to the account that is on this server.  From there you will be able to reset your cpanel password from the left menu.

Auto SSL Failure (Resolved) High
  • 2017/02/07 16:24 - 2017/04/24 15:55
  • Last Updated 2017/02/07 18:25

The AutoSSL feature of this server has failed and certificates which expired today have not been renewed.  We are investigating the cause and hope to have a resolution shortly.

This issue has been resolved and the SSL certificates are queued at Comodo for installation.

Drive Maintenance (Resolved) High
  • 2017/01/16 22:00 - 2017/01/17 04:50
  • Last Updated 2017/01/16 20:48

potomac.webnetservers.com is displaying some issues with I/O. We require a maintenance window with the server offline for 1-4 hours to clone the primary drive.

Server unresponsive (Resolved) Critical
  • 2016/12/29 13:20 - 2016/12/29 13:43
  • Last Updated 2016/12/29 13:43

This server has become unresponsive and the issue is being actively investigated.

--

This isssue is now resolved.

SMTP Slowdown (Resolved) Critical

Affecting Server - cpanel43.onlinemountain.com

  • 2016/12/12 18:22 - 2016/12/13 11:03
  • Last Updated 2016/12/12 18:22

Because of a spamming account, this server's SMTP queue is being cleared of spam messages
There are several hundred thousand messages to clear.
Once clear we will divert outbound mail via an alternate IP.

SMTP Slowdown (Resolved) Critical

Affecting Server - cpanel42.onlinemountain.com

  • 2016/12/12 18:21 - 2016/12/13 11:03
  • Last Updated 2016/12/12 18:22

Because of a spamming account, this server's SMTP queue is being cleared of spam messages
There are several hundred thousand messages to clear.
Once clear we will divert outbound mail via an alternate IP.

Hardware Maintenance Window (Resolved) High
  • 2016/12/06 21:00 - 2016/12/07 14:13
  • Last Updated 2016/12/06 16:02

A hardware maintenance window is required for this server in the times shown
All services on this server will be unavailable during the maintenance window as the server will be powered off
Actual time will be approximately 1 hour.

Hardware Maintenance Window (Resolved) Medium
  • 2016/12/01 01:00
  • Last Updated 2016/12/06 16:00

A hardware maintenance window is required for this server in the times shown
Dec 1, 01:00-05:00 EST (GMT -5:00)
All services on this server will be unavailable during the maintenance window as the server will be powered off
Actual time will be approximately 1 hour.

IIS6 WH-01 Being Moved to New HOST (Resolved) Medium

Affecting Server - ED-DNP01

  • 2016/11/19 17:10 - 2016/11/19 17:39
  • Last Updated 2016/11/19 17:35

Our legacy IIS 6 server is being moved to a new host for improved stability.  During this move, websites on this server will be inaccessible.
Move time is estimated at 15minutes.

Migration complete

Server Loads Extremely High (Resolved) High
  • 2016/11/16 16:28 - 2016/11/16 17:56
  • Last Updated 2016/11/16 16:28

We are investigating extremely high loads on this server.

Kernel update and reboot (Resolved) Medium

Affecting Server - cpanel41.onlinemountain.com

  • 2016/09/03 11:16
  • Last Updated 2016/09/03 12:11

System kernel update in progress + reboot of server

--

FSCK in progress forced - startup delay.

---

Reboot complete.

Server Load High (Resolved) High

Affecting Server - cpanel43.onlinemountain.com

  • 2016/09/02 13:07 - 2016/09/03 11:13
  • Last Updated 2016/09/02 17:36

The high load on this server is being investigated.  While our investigation proceeds, port 80 has been blocked to allow our technicians the ability to work in a timely manner.  All other ports remain open (POP, IMAP, FTP, SMTP, Cpanel)

---

Apache / PHP rebuild in progress.

---

Build complete - we are now doing a kernel update.

---

System being rebooted now - new kernel installed and updates complete.

---

We've found the root cause of this issue and have resolved it.  Loads are now normal and the server is responsive.  We are monitoring it continuously.


Incoming email problem (Resolved) Critical

Affecting Server - ED-SH-CP7

  • 2016/08/12 16:28 - 2016/08/12 20:11
  • Last Updated 2016/08/12 20:11

Currently, all email is queued and not being delivered to individual user mailboxes.  This is an issue with a failed cpanel update.
We are investigating and hoping to have a resolution shortly.

ALL DELIVERY ISSUES HAVE NOW BEEN RESOLVED.

Hosted Exchange Update (Resolved) Medium

Affecting System - Hosted Microsoft Exchange

  • 2016/08/06 08:00
  • Last Updated 2016/08/06 12:04

We have scheduled some mailbox server updates - Microsoft Hosted Exchange 2010
These are general service and stability updates for Exchange and Windows Server as well as funcitionality updates for OWA (Outlook Web Access)

Updates started

1st reboot in progress....

Reboot complete.
Starting second set of updates

2nd reboot in progress....

Exchange updates in progres - your mailbox will be unavailalbe until completion...

Mailboxes now available

Further updates may make your mailbox unavailable for a few minutes....

Last set of updates require a reboot to complete.  Reboot in progress....

UPDATES COMPLETE

cp.neiko.net - login unavailable (Resolved) Medium

Affecting Server - VIPER-HELM-SH

  • 2016/07/21 11:39
  • Last Updated 2016/07/29 14:10

We are currently experiencing login issues with http://cp.neiko.net - this is a licensing problem with the control panel server which we are working to resolve.

cPanel servers (Resolved) Critical

Affecting System - cPanel Security Team - CVE-2016-3714 ImageMagick

  • 2016/05/04 12:18 - 2016/05/04 15:14
  • Last Updated 2016/05/04 12:20

cPanel Security Team - CVE-2016-3714 ImageMagick

Background Information

On Tuesday, May 3 2016, ImageMagick announced a vulnerability in all versions of the ImageMagick software. ImageMagick is a software
package commonly used by web services to process images.

Impact

One of the reported vulnerabilities can potentially be exploited for remote code execution (RCE).

Releases

ImageMagick has not released a fix, but plans to publish a new version of ImageMagic with the fixes soon. cPanel normally releases all builds at once in order to limit the ability to reverse engineer fixes. However, this vulnerability is already wildly known and we have seen reports of its use. In this instance, we plan to release builds as soon as they become available.

At this time the following builds are available:
11.56    11.56.0.13
EDGE 11.55.9999.193
CURRENT  11.56.0.13
RELEASE  11.56.0.13

We are currently updating all our cpanel servers to this build.  If you cannot access cpanel or WHM then a cpanel update is in progress.  Please be patient as the update completes.  

Update in progress (Resolved) Medium

Affecting Server - cpanel43.onlinemountain.com

  • 2016/04/30 11:43 - 2016/04/30 12:15
  • Last Updated 2016/04/30 11:44

cpanel43.onlinemountain.com is doing a cpanel udpate to 11.56 - as such cpanel and WHM are currently unavailable.

Normal operations should resume shortly.

Server Inaccessible - Unscheduled Maintenance (Resolved) Critical

Affecting Server - cpanel43.onlinemountain.com

  • 2016/03/06 18:01 - 2016/03/07 16:16
  • Last Updated 2016/03/07 16:16

We are investigating an issue with this server - updates to follow in this Network Issue notification

--UPDATE--

We see an issue with the installed kernel - we are reverting to the previous kernel and rebooting
Investigation will continue.

--UPDATE--

Server is operational - we will be rebooting the server to a newer kernel later tonight.

--UPDATE--

Kernel has been updated correctly and verified - we are rebooting the server so that dependencies with CloudLinux function correctly.

--REBOOT COMPLETE

--UPDATE--

We are once again noticing high loads on this server - invesigtating the issue - we will cycle through services and block specific ports to track down the culprit.

--UPDATE--

We are now running further updates to kernel and other drivers/cloudlinux in an attempt to narrow down the possible issue.

--UPDATE--

In order to expedite this process, we've had to block port 80 on this server as the update process was taking too long.  We have restarted the update process with port 80 blocked (low load) and hope to reboot the server shortly.

--UPDATE--

we have resolved the issue and loads are now normal.

Server Reboot (Resolved) High

Affecting Server - cpanel41.onlinemountain.com

  • 2016/03/06 20:15
  • Last Updated 2016/03/06 20:29

A kernel dependancy requires that we reboot the server for maximum stability.
Reboot has begun and server should be back online in 5-10 min.

--REBOOT COMPLETE

SHA1 Certs - onlinemountain.com (Resolved) Critical

Affecting System - SSL Certificates - SHA1 replacement

  • 2015/12/19 12:04 - 2016/03/06 18:02
  • Last Updated 2015/12/19 12:05

We are currently replacing all onlinemountain.com COMODO SSL certificates with SHA2 encrypted certificates.

Prior to replacement you will get a warning in Google Chrome.

Thank you for your patience

Core Infrastructure Upgrades (Resolved) Medium

Affecting System - Core Network

  • 2015/12/06 18:30 - 2015/12/06 20:59
  • Last Updated 2015/12/06 17:31

We will be updating some core switches.  During the upgrades there may be some downtime you experience - Approximately 5 minutes while switchports come online.
-This maintanance windows has been postponed.  A new item will be opened once the maintenance window has been determined.

Core Infrastructure Upgrades (Resolved) Medium

Affecting System - Toronto DC

  • 2015/11/22 21:00 - 2015/11/22 22:00
  • Last Updated 2015/11/22 20:16

We will be updating some core switches.  During the upgrades there may be some downtime you experience - Approximately 5 minutes while switchports come online.
-This maintanance windows has been postponed.  A new item will be opened once the maintenance window has been determined.

Apache 500 Errors - Cloudlinux (Resolved) Critical

Affecting Server - cpanel42.onlinemountain.com

  • 2015/11/01 14:38 - 2015/11/01 18:05
  • Last Updated 2015/11/01 17:39

Most websites throwing 500 errors due to a Cloudlinux CageFS issue.
We have escalted this to our Level3 support and hope to have a resolution shortly.

All 208.92.222.x IP network is down. (Resolved) High

Affecting System - All 208.92.222.x IP network

  • 2015/09/15 19:56 - 2015/09/15 20:13
  • Last Updated 2015/09/29 09:13

Due to a DDOS in progress all 208.92.222.x IP network is down.
We are working with our downstream provider to resolve the issue.

Reboot in progress (Resolved) High

Affecting Server - cpanel41.onlinemountain.com

  • 2015/05/29 16:02 - 2015/05/29 17:15
  • Last Updated 2015/05/29 16:28

This server was under very heavy load - a full restart is in progress.
- file system quotas check in progress (fsck) - this must complete before the server comes back online.

IMAP/POP logins failing (Resolved) Critical

Affecting Server - cpanel41.onlinemountain.com

  • 2015/03/25 16:18 - 2015/03/25 18:17
  • Last Updated 2015/03/25 16:19

We are aware of the issue regarding this server.
POP3 and IMAP connections may fail with an authenticator error.
We are actively investigating this issue and should have a resolution shortly.

Server down (Resolved) Critical

Affecting Server - cpanel42.onlinemountain.com

  • 2015/03/20 15:11
  • Last Updated 2015/03/20 15:23

We are awere of the issue with this server and are actively investigating.

--issue is now resolved.

MAINTENANCE - HARDWARE Upgrade (Resolved) Medium

Affecting Server - cpanel41.onlinemountain.com

  • 2015/02/25 21:21 - 2015/02/25 21:44
  • Last Updated 2015/02/25 21:22

We are currently doing a hardware upgrade on this server. During the upgrade no services are available.

PHP update in progress (Resolved) Medium

Affecting Server - cpanel41.onlinemountain.com

  • 2015/02/23 20:02 - 2015/02/25 21:42
  • Last Updated 2015/02/23 20:02

During updates websites will be unavailable.

Webmail/IMAP/POP Access down (Resolved) High

Affecting System - Mail server - WEBM1 - TOR DC 1

  • 2015/02/06 14:20 - 2015/02/10 10:23
  • Last Updated 2015/02/06 14:21

We are seeing intermittent webmail issues with this server (https://webm1.entirelydigital.com) and are working to resolve.
You may see continuing intermittent outages while we restart services and update core files.

POP/IMAP Webmail Authentication Failure (Resolved) High

Affecting Server - cpanel41.onlinemountain.com

  • 2015/02/03 08:06 - 2015/02/03 10:49
  • Last Updated 2015/02/03 08:43

This server is refusing to authenticate POP, IMAP and WEBMAIL connections (via IMAP) once more.  We've escalated this to cPanel.net for immediate investigation and permanent resolution.

---

This is now resolved but we are investigating the cause.

Apache/PHP recompile (Resolved) Medium

Affecting Server - cpanel41.onlinemountain.com

  • 2015/02/01 17:08
  • Last Updated 2015/02/02 10:48

We are updating apache on this server - websites will be down for approximately 10-15 min.

---

ETA changed - additional updates required for security reasons.

---

Now complete - websites available.

IMAP/POP/SMTP Authentication Failure (Resolved) High

Affecting Server - cpanel41.onlinemountain.com

  • 2015/01/28 10:31 - 2015/01/29 12:01
  • Last Updated 2015/01/28 10:32

This server is experiencing IMAP/POP/SMTP Authentication Failure for most accounts.
We are actively investigating this issue and hope to have a resolution shortly.

Server maintenance (Resolved) Medium

Affecting Server - cpanel42.onlinemountain.com

  • 2015/01/18 07:12 - 2015/01/18 10:34
  • Last Updated 2015/01/18 09:14

Server is currently undergoing maintenance - some hosting services will be affected during our upgrades

Server Maintenance (Resolved) Medium

Affecting Server - cpanel43.onlinemountain.com

  • 2015/01/18 07:13 - 2015/01/18 09:34
  • Last Updated 2015/01/18 09:14

Server is currently undergoing maintenance - some hosting services will be affected during our upgrades

Server under high load (Resolved) High

Affecting Server - cpanel41.onlinemountain.com

  • 2015/01/13 09:40 - 2015/01/14 09:18
  • Last Updated 2015/01/13 19:50

This server experienced a system failure during updates and we had to gracefully restart it.  Unfortunately a forced filesystem check is in progress and must complete.
Hopefully won't be much longer.

--

We are trying to deterimine what processes are causing high load - during our troubleshooting process, different services will become unavailable.

--

Currently httpd is turned off while we update cloudlinux cageFS - email and other services are up - this is required to ensure that cloudlinux LVE is working properly before we start httpd

--

Kernel updates are complete as well as updates to CloudLinux LVE - we're rebooting the server to load the new kernel
We'll be watching the server carefully over the next hours to ensure the loads stay in check.

--

Server being monitored and processes are being kept in check

--

In cooperation with cpanel.net support we are close to a resolution regarding the performance of this server.  We appreciate your patience.

--

We've identified this is a DDOS attack and are taking steps to attempt to mitigate the attack

--

The DDOS seems to be targeted to specific customers - we are working to mitigate the attack however DDOS can be difficult to stop.

--

Cloud Backup Server Upgrade (Resolved) Medium

Affecting Server - OBS

  • 2015/01/09 13:45 - 2015/01/09 14:00
  • Last Updated 2015/01/09 13:46

Our cloud backup service is being updated to the latest build - during the upgrade offsite backup and online portal access is unavailable

Apache/PHP recompile (Resolved) Medium

Affecting Server - cpanel43.onlinemountain.com

  • 2014/12/08 15:47 - 2014/12/19 12:31
  • Last Updated 2014/12/08 15:48

PHP is being rebuilt (native 5.4) to include MBSTRING as it was not included in the original build.

Hosted MS Exchange Certificate Warning (Resolved) Critical

Affecting System - Hosted MS Exchange

  • 2014/12/08 10:44
  • Last Updated 2014/12/08 11:32

Clients are receiving invalid SSL certificate warnings in webmail and outlook - we are working with Godaddy SSL to re-issue the SSL certificate in question.

UPDATE - Certificate has been rekeyed and installed - there are no furhter issues with webmail

PHP update in progress (Resolved) Medium

Affecting Server - cpanel42.onlinemountain.com

  • 2014/12/06 10:45 - 2014/12/06 11:39
  • Last Updated 2014/12/06 10:45

PHP and Apache being updated to the latest build

PHP update in progress (Resolved) Medium

Affecting Server - cpanel41.onlinemountain.com

  • 2014/12/06 10:45 - 2014/12/06 12:42
  • Last Updated 2014/12/06 10:45

PHP and Apache being updated to the latest build

PHP update in progress (Resolved) Medium

Affecting Server - cpanel43.onlinemountain.com

  • 2014/12/06 10:44 - 2014/12/06 11:39
  • Last Updated 2014/12/06 10:45

PHP and Apache being updated to the latest build

Cloud Backup Server Upgrade (Resolved) Medium

Affecting Server - OBS

  • 2014/09/08 16:08 - 2014/09/09 22:00
  • Last Updated 2014/09/08 20:17

Update of Cloud Backup Server in progress.
Disk Check in progress

---

Disk check is still in progress but almost complete - should be operational shortly.

PHP update in progress (Resolved) High

Affecting Server - cpanel41.onlinemountain.com

  • 2014/08/19 10:16 - 2014/08/20 12:32
  • Last Updated 2014/08/19 10:17

To address possible security vulnerabilities in php - we are updating PHP to 5.4.31 outside the normal maintenance window
This is only a MID-LEVEL security issue.

Failed cpanel update being re-run (Resolved) High
  • 2014/08/18 10:22 - 2014/08/19 10:16
  • Last Updated 2014/08/18 10:23

This server had a failed cpanel update causing some issues - we are re-running it now to stabilize the server.

Rebuild Apache/PHP (Resolved) Medium
  • 2014/07/30 18:25 - 2014/07/31 10:22
  • Last Updated 2014/07/30 18:25

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.

Rebuild Apache/PHP (Resolved) Medium

Affecting Server - ED-SH-CP6

  • 2014/07/26 10:56 - 2014/07/28 15:29
  • Last Updated 2014/07/26 12:58

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.

-- update --
We are now upgrading MySQL to 5.5 for compatiblity and upgrade paths to WHM 11.44

-- update --
We are now upgrading WHM to 11.44

Rebuild Apache/PHP (Resolved) Medium
  • 2014/07/23 17:01 - 2014/07/26 10:56
  • Last Updated 2014/07/23 17:15

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.

Rebuild Apache/PHP (Resolved) Medium
  • 2014/07/23 11:15 - 2014/07/23 12:21
  • Last Updated 2014/07/23 11:14

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.

Rebuild Apache/PHP (Resolved) Medium
  • 2014/07/16 22:02 - 2014/09/08 16:08
  • Last Updated 2014/07/16 19:02

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.

Rebuild Apache/PHP (Resolved) Medium
  • 2014/07/16 22:02 - 2014/07/16 19:02
  • Last Updated 2014/07/16 19:02

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.

Rebuild Apache/PHP (Resolved) Medium
  • 2014/07/16 19:01 - 2014/09/08 16:08
  • Last Updated 2014/07/16 19:01

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.

Rebuild Apache/PHP (Resolved) Medium
  • 2014/07/16 19:01 - 2014/09/08 16:08
  • Last Updated 2014/07/16 19:01

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.

Rebuild Apache/PHP (Resolved) Medium
  • 2014/07/16 19:00 - 2014/07/17 11:53
  • Last Updated 2014/07/16 19:00

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.

Rebuild Apache/PHP (Resolved) Medium
  • 2014/07/16 19:00 - 2014/07/17 12:46
  • Last Updated 2014/07/16 19:00

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.

Rebuild Apache/PHP (Resolved) High
  • 2014/07/16 16:53 - 2014/07/16 17:12
  • Last Updated 2014/07/16 16:53

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.

Rebuild Apache/PHP (Resolved) High
  • 2014/07/16 16:53 - 2014/07/16 17:28
  • Last Updated 2014/07/16 16:53

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.

Rebuild Apache/PHP (Resolved) High
  • 2014/07/16 15:14 - 2014/07/16 16:50
  • Last Updated 2014/07/16 15:14

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.

Rebuild Apache/PHP (Resolved) High
  • 2014/07/16 15:14 - 2014/07/16 16:50
  • Last Updated 2014/07/16 15:14

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.

Rebuild Apache/PHP (Resolved) High
  • 2014/07/16 14:59 - 2014/07/16 16:50
  • Last Updated 2014/07/16 15:14

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.

Rebuild Apache/PHP (Resolved) High
  • 2014/07/16 14:45 - 2014/07/16 14:55
  • Last Updated 2014/07/16 14:45

An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.

Server not responding (Resolved) High
  • 2014/07/02 10:10 - 2014/07/02 11:58
  • Last Updated 2014/07/02 10:14

This server is currently down - we are actively investigating the cause.

---

Server is booting howerver a mandatory forced FSCK (disk check) is now in progress.  Usually takes approx 20 min

Server reboot in progress (Resolved) High
  • 2014/06/22 14:46 - 2014/06/22 15:43
  • Last Updated 2014/06/22 14:47

We have had to restart this server and it is currently performing a quota operation mandatory in OpenVZ.  Once this operation completes, the server should come back online.
ETA 20 min

Server listed at bl.spamcop.net (Resolved) High
  • 2014/06/18 11:51 - 2014/06/22 10:17
  • Last Updated 2014/06/18 11:51

We have resolved the issue causing this listing and have requested delist from spamcop.net.

Queue flooded with spam (Resolved) Medium
  • 2014/06/14 17:44 - 2014/06/14 22:30
  • Last Updated 2014/06/14 17:45

A hacked mail account has caused the mail queue to be flooded with spam.  We have identified and stopped the spammer and clearing the queue - at this time the server is under high load cleaning he affected queue.

Server mysql under high load (Resolved) Medium
  • 2014/06/11 21:42 - 2014/06/14 17:44
  • Last Updated 2014/06/11 21:43

We are investigating a continuous load issue with this server - in this process we are shutting down mysql for approx 20min to see if the processes are to heavy for the server.

Server reboot in progress (Resolved) High
  • 2014/06/11 11:20 - 2014/06/11 21:42
  • Last Updated 2014/06/11 11:21

We have had to restart this server because of load issue investigations.  Because of the restart, a mandatory disk check is now in progress.  We realize the timing on the forced disk check is during normal business hours however it is better to allow the disk check to complete at this time.  ETA 30 min.

Server not responding (Resolved) High
  • 2014/06/06 02:00 - 2014/06/06 11:09
  • Last Updated 2014/06/10 11:09

We are investigating a server issue affecting multiple servers including this one.  We hope to have an update soon.

MySQL Upgrade / cPanel Upgrade (Resolved) Medium
  • 2014/06/01 09:48 - 2014/06/06 08:40
  • Last Updated 2014/06/01 09:50

In order to update cpanel/WHM to the latest version, MySQL 5.5 is required.  MySQL 5.0 is deprecated and no longer supported.  As such, in order to bring this server in line with all others in our infrastructure we are in the process of updating MySQL.  During this time, websites requiring mysql queries may not be available.

Apache/PHP recompile (Resolved) Medium
  • 2014/05/27 10:43 - 2014/05/27 11:27
  • Last Updated 2014/05/27 10:43

Due to some errors we are force to recompile Apache and PHP to the latest versions (2.2.x and 5.4.x respectively) - during this process your websites may become unavailable.

Queue flooded with spam (Resolved) High
  • 2014/05/05 16:22 - 2014/05/05 21:52
  • Last Updated 2014/05/05 16:23

This server is experiencing slowdowns due to the mail queue being full of spam.  We have identified the offending sender and disabled their account.  We are clearing the queue and will take steps to mitigate this behaviour in the future.

cPanel Update (Resolved) Medium
  • 2014/04/20 11:56 - 2014/05/04 20:46
  • Last Updated 2014/04/20 11:57

With a successful MySQL update completed, cPanel is being updated to the lastest RELEASE version 11.42.1.x

MySQL upgrade in progress (Resolved) Medium
  • 2014/04/20 11:19 - 2014/04/20 11:56
  • Last Updated 2014/04/20 11:20

MySQL is being upgraded to 5.5 - during the upgrade process, websites which use databases will be unavailable

PHP update in progress (Resolved) Medium
  • 2014/04/20 11:08 - 2014/04/20 11:41
  • Last Updated 2014/04/20 11:09

PHP/Apache is being recompiled to the latest versions

Server updates (Resolved) Medium
  • 2014/03/15 21:35 - 2014/03/15 21:53
  • Last Updated 2014/03/15 21:35

This server is being updated - as such some sites are not responding as apache and PHP are being updated.

WHM/Cpanel Uprade to 11.40.11 (Resolved) Medium
  • 2014/02/28 20:00
  • Last Updated 2014/02/28 21:10

We are currently upgrading MySQL from 5.0.x to 5.5.x
We are currently upgrading WHM/cPanel from 11.38.x (EOL) to 11.40.11 - current release
You may see some interruption in web services while the system updates it's packages.

Backup LAN GB (Resolved) Medium

Affecting System - Backup Swich - TOR1 DC1

  • 2014/02/03 09:50 - 2014/02/03 13:58
  • Last Updated 2014/02/03 09:51

One of our backup switches used for backend communcations is down.  This will affect communiction between websites and databases and backup operations.
We are investigating the issue and hope to have a resolution shortly.

PHP upgrade to 5.4 (Resolved) Medium
  • 2014/01/20 20:09 - 2014/01/20 21:00
  • Last Updated 2014/01/20 20:38

We have now started upgrading / recompiling apache with PHP 5.4 on this server.
Should be done shortly.
Build complete - PHP is now 5.4.24

MS Information store hung (Resolved) Critical

Affecting System - Hosted MS Exchange 2007

  • 2013/12/14 09:15
  • Last Updated 2013/12/16 20:00

Customers of our hosted MS Exchange have noticed intermittent connectivity this morning - We are working to resolve this and have initiated a server restart.

--UPDATE--
An integrity check and fix is required on one of the information stores.
This necessitates bringing the store offline during the check.
During this time some mailboxes may be unavailable depending on which store your mailbox resides.
We appreciate your patience during this check/fix operation.
--

--UPDATE--
Please note that is is a long procedure.  We estimate the time to completion at 14:30
Messages will be queued on other servers until the information store becomes available
--

--UPDATE--
Test revealed further errors which are now being fixed through another procedure.  We esitmate another 4 hours before completion.
Our apologies for the inconvenience
--

--UPDATE--
Integrity check is taking much longer than anticipated and we may have to restore from backup as we cannot continue to keep the server offline.
We will work to also restore the data from the restore point until this morning but keep customer up to date via this post.
--

--UPDATE / SUNDAY MORNING--
We have added some additional disk arrays and will be doing a restore from backup shortly.  Unfortunately our DB fix did not complete successfully and so we must restore from backup.  We will work to recover data manually from the corrupt DB after the server is back online.
Again, we thank you for your patience.
--

--UPDATE--
Restore is in progress and completion is estimated in approx 3 hours
--

IMAP/POP connections failing (Resolved) Critical
  • 2013/12/14 09:35 - 2013/12/14 10:19
  • Last Updated 2013/12/14 09:36

We are investigating an issue whereby all IMAP and POP accounts cannot login to retrieve mail

Server not responding - High Load (Resolved) Critical
  • 2013/12/13 12:38 - 2013/12/13 13:01
  • Last Updated 2013/12/13 12:53

This server is currently under high load and may not respond to web/email requests.
-Update
We have found an excessive amount of messages in the mail queue and are investigating.

Server not responding - High Load (Resolved) High
  • 2013/12/13 12:38 - 2013/12/14 09:16
  • Last Updated 2013/12/13 12:38

This server is currently under high load and may not respond to web/email requests.

Smartermail - Michigan (Resolved) High

Affecting Server - VIPER-HELM-SH

  • 2013/12/12 12:54 - 2013/12/13 12:38
  • Last Updated 2013/12/12 12:55

Currently our MI email server webm1-MI.entirelydigital.com is at 100%CPU - likely due to a DDOS attack - we are currently investigating.

Mandatory Disk Check (Resolved) High
  • 2013/12/11 18:23 - 2013/12/11 21:47
  • Last Updated 2013/12/11 21:47

We have had to restart this server and force a mandatory disk check to ensure integrity.  During this time, the server and it's services will be unavailable.
We anticipate downtime of approx 1 hour.

Apache offline temporarily (Resolved) Medium
  • 2013/12/09 14:49 - 2013/12/14 10:19
  • Last Updated 2013/12/09 14:50

We are recompling apache to address some web server issues.  It should be up shortly.

cpanel20.onlinemountain.com currently down (Resolved) Critical
  • 2013/12/02 10:19
  • Last Updated 2013/12/02 12:22

WHM/Cpanel host cpanel20.onlinemountain.com is currently not repsonding.  We have our NOC technician looking at the server and we are investigating the issue.
We hope to have it up and running shortly.

--UPDATE--
Server is undergoing a forced fsck (disk check) which we cannot stop.  This is a mandatory check after a hard reset.  We will investigate the cause of the issue to ensure that the resource drain does not re-occur.
--UPDATE--

--UPDATE--
Server is running - we are now performing system checks before bringing WHM/cpanel back online
--UPDATE--

--UPDATE--
Server has finished checks and WHM/cPanel is now initializing
--UPDATE--

cPanel Webmail Auto-Responder issue (Resolved) Low

Affecting System - cPanel Webmail

  • 2013/10/15 16:18
  • Last Updated 2013/11/22 14:35

There is an issue with webmail on all cPanel servers.  cPanel 11.40 does not allow you to set custom start and stop date/times for your autoresponder.  No calendar drop down appears when you click on the date.  This is a known issue and will be fixed with the next cPanel update.

System restart (Resolved) High
  • 2013/10/26 08:58 - 2013/10/26 13:38
  • Last Updated 2013/10/26 08:59

ed-sh-cp1.entirelydigital.com is currently undergoing a file system check.  Once complete the server will be back onine.

Exchange 2007 System Down (Resolved) Critical

Affecting System - Exchange 2007

  • 2013/10/01 07:30 - 2013/10/07 00:00
  • Last Updated 2013/10/15 16:18

We are experiencing an outage on our Exchange 2007 server - we are working as quickly as possible to resolve the issue.  Mailbox access, webmail access is affected.
UPDATE - All hardware upgraded and issue is now resolved

SmarterMail MI (Resolved) Medium

Affecting System - webmail MI webmail.neiko.net

  • 2013/09/24 21:55
  • Last Updated 2013/09/24 22:03

We are upgrading all our SmarterMail Servers to enterprise 11.6.x

Datacenter Maintenance (Resolved) Medium

Affecting System - UPS Battery

  • 2013/05/14 17:01 - 2013/11/27 10:14
  • Last Updated 2013/05/10 16:09

EntirelyDigital will be replacing a single UPS Battery at the Toronto data center location on Tuesday May 14, 2013 from 16:01hrs to 17:30hrs. There is no expected service impact during this maintenance

IP Blacklisted (Resolved) Critical
  • 2013/04/03 10:13 - 2013/04/11 10:14
  • Last Updated 2013/04/03 10:13


Due to an internal spammer/compromised account, 208.92.222.102 has been blacklisted.

We are investigating the issue and will request immediate delist once the account has been disabled.

High load - SMTP/POP/IMAP unresponsive (Resolved) High

Affecting System - cas1.entirelydigital.com/smtp75.entirelydigital.com

  • 2013/03/06 14:11 - 2013/03/08 17:15
  • Last Updated 2013/03/06 14:12


We are experiencing very high loads on this server and are investigating the cause.  SMTP/POP and IMAP connections will time out and server will appear unresponsive.

Drive Failure (Resolved) Critical
  • 2012/11/29 11:25 - 2013/03/04 22:27
  • Last Updated 2012/11/29 11:25

ED-SH-CP1.entirelydigital.com experienced a drive failure a short time ago.  We are investigating and hope to have the server up and running shortly.

Telephone Support Unavailable (Resolved) Critical

Affecting System - Telephone Technical Support

  • 2012/07/01 00:00 - 2012/07/01 18:22
  • Last Updated 2012/07/01 00:54

We are currently experiencing issues with our telephone service. As you may or may not know, our phone servers are located in our US office. Last night a severe storm knocked out power, Internet, and essential services to almost a million homes in Northern Virginia. Luckily our US staff was unharmed by the storm. As of right now, the Gov. Bob McDonnell has declared a state of emergency for Virginia.

Our vendors providing the DIDs and 'dial tone' for all of our phone lines have been forced to shut down all power, including UPS, so emergency crews can work to repair damaged power lines safely.

As inconvenient and unfortunate as this is, we are doing everything we can to assist/comply with emergency crews and services in hopes of having this resolved soon.

This telephony outage has no impact on helpdesk support services we provide.

We do not have a firm ETA however we will keep you updated as soon as we get additional information from the upstream providers and/or Virginia Dominion Power.

You can reach our support staff as usual by emailing helpdesk@entirelydigital.com

We thank you for your patience and understanding. Our thoughts are with those who lost loved ones in last nights storm.

Network Outage - USA Datacenter (Resolved) Critical

Affecting Other - Network Outage - USA Datacenter

  • 2012/06/29 15:15 - 2012/06/29 15:35
  • Last Updated 2012/06/29 15:56

We are experiencing a network outage in our USA datacenter - this is not a server issue but an issue with a downstream provider

We are currently invesitgating and this issue is priority 1 - Critical

-- update this issue is now resolved and we are actively monitoring the hardware in question

Michigan Data Center Outage (Resolved) Critical

Affecting System - Michigan Data Center - some servers

  • 2012/06/21 08:07 - 2012/06/21 10:05
  • Last Updated 2012/06/21 08:14

We are experiencing a major outage in our Michigan data center and are actively investigating.

Please do not email our helpdesk regarding your site or email being down as we are already aware of the issue and it is a network issue not a server issue or an issue with your particular domain.

US Datacenter Outage (Resolved) Critical

Affecting Other - US Datacenter

  • 2012/06/03 15:41 - 2012/06/03 00:00
  • Last Updated 2012/06/03 16:54

We are experiencing a system wide datacenter outage in our USA datacenter.

An investigation is currently in progress.  We are aware of the issue so please do not call our support line or email our helpdesk as your website/service/email will come back on line automatically once the issue is resolved.

Server listed at senderbase.org (Resolved) Medium
  • 2012/04/30 12:38 - 2012/06/03 16:54
  • Last Updated 2012/04/30 12:39

We are currently investigating why this server (208.92.222.102) is listed at senderbase.org. This may affect you sending email to certain domains that use Cisco firewalls as their antispam solution.

Kernel security, bug fixes, and enhancement u (Resolved) Medium

Affecting System - CPanel Servers

  • 2012/04/20 02:00 - 2012/07/01 00:55
  • Last Updated 2012/04/20 15:38

We will be scheduling an important kernel security, bug fixes and enhancement update for all our CentOS 5.x servers this weekend.

This would take place as far as possible during the off-peak hours. A reboot is required to complete the upgrade. The downtime should not exceed 30 minutes and it will be minimize as much as possible. We will track each server till it returns to service after the upgrade and reboot.

This upgrade is scheduled as follows:
Date: 21 April 2012 (Saturday) to 23 April 2012 (Monday)
Time: Between 2AM and 8AM EST

Details of Kernel security, bug fixes and enhancement update
 
* A flaw in the xfrm6_tunnel_rcv() function in the Linux kernel's IPv6 implementation could lead to a use-after-free or double free flaw in tunnel6_rcv(). A remote attacker could use this flaw to send
specially-crafted packets to a target system that is using IPv6 and also has the xfrm6_tunnel kernel module loaded, causing it to crash. (CVE-2012-1583, Important)

If you do not run applications that use xfrm6_tunnel, you can prevent the xfrm6_tunnel module from being loaded by creating (as the root user) a "/etc/modprobe.d/xfrm6_tunnel.conf" file, and adding the following line to it:

blacklist xfrm6_tunnel

This way, the xfrm6_tunnel module cannot be loaded accidentally. A reboot is not necessary for this change to take effect.

This update also fixes various bugs and adds an enhancement.

HSphere Server Migration (Resolved) Medium
  • 2012/04/10 19:57 - 2012/04/12 11:42
  • Last Updated 2012/04/10 19:58

We are in the process of migrating the Hsphere control panel server. During this move, access to account functions will not be available.  Web and email services are unaffected.

Michigan IDC - Unscheduled Maintenance (Resolved) High

Affecting System - Michigan IDC - All Services

  • 2012/01/13 01:14 - 2012/01/13 12:01
  • Last Updated 2012/01/13 01:15

We are currently performing some unscheduled maintenance on our Michigan SAN infrastructure.  We estimate service will resume at approx 4am EST

Email services unavailable (Resolved) High

Affecting System - some email servers

  • 2011/12/06 09:35 - 2011/12/06 10:21
  • Last Updated 2011/12/06 09:36

One of our email servers is currently not accepting requests for POP/IMAP - service will be restored shortly.

Cpanel Licence Error (Resolved) Medium
  • 2011/11/14 10:00 - 2011/11/14 11:34
  • Last Updated 2011/11/14 10:01

CP3.shieldhost.com is currently showing an error.  This affects webmail and control panel access.  We are working with the control panel vendor to resolve this.

Unable to access plesk / download mail (Resolved) Critical

Affecting Server - NET-209

  • 2011/10/14 16:01 - 2011/10/17 14:17
  • Last Updated 2011/10/14 16:20

Clients trying to access POP and IMAP mail will not be able to login and will recieve an error in their mail clients as well as webmail.

We are aware of the issue and working toward a resolution.

Worldwide BES issues (Resolved) High

Affecting System - Hosted Exchange 2007 / BES - Blackberry Enterprise Server

  • 2011/10/12 07:33 - 2011/10/17 14:17
  • Last Updated 2011/10/12 09:39

Blackberry's (RIM) outage has spread to Canada and affects all BES users on hosted MS exchange. This is not an issue with EntirelyDigital's networks but stems from RIM itself.

There is no ETA for resolution.

iPhone, Android, Windows Phone users are unaffected by this outage.

Hosted Exchange 2007 - Maintenance (Resolved) Medium

Affecting Other - Hosted Exchange 2007

  • 2011/03/27 22:00 - 2011/04/22 20:12
  • Last Updated 2011/03/26 19:29

Dear Microsoft Exchange User, (french message follows)
On Sunday March 27th, 2011 at approximately 22:00 (10pm) EST (GMT -5.00) we will be doing some software and hardware upgrades on our Microsoft Exchange infrastructure.
During this time your mailbox will be unavailable. The duration of this maintenance window is estimated to be approximately 1 hour.
With these upgrades we will be increasing our capacity which means larger mailbox quotas coming soon.
This upgrade is part of many infrastructure upgrades we will be doing in 2011.
Regards,
EntirelyDigital NOC Team

---

Cher utilisateur de Microsoft Exchange,

Le 27 Mars 2011, à environ 22h00 EST (GMT -5.00), nous allons faire quelques mises à jour logicielles et matérielles de notre infrastructure Microsoft Exchange.
Pendant ce temps, votre boîte aux lettres ne seront pas disponibles. La durée de cette fenêtre de maintenance est estimée à environ 1 heure.

Avec ces améliorations, nous augmenterons notre capacité ce qui signifie que les quotas de boîtes aux lettres plus à venir.

Cette mise à jour fait partie de nombreuses améliorations à l'infrastructure que nous allons faire en 2011.

Cordialement,

EntirelyDigital équipe CNP

Network Upgrades (Resolved) Low

Affecting Other - Michigan IDC

  • 2011/01/18 00:00 - 2011/01/26 20:42
  • Last Updated 2011/01/26 20:42

EntirelyDigital is scheduling a service window from Tuesday 1/18/2011 11:00PM
to Wednesday 1/19/2011 2:00AM to perform upgrades on key network equipment in our Michigan Data Centre. We
do not expect any network outages durring this service window. Customers may notice
a temporary changes in routing.

Level (3) connectivity will temporarily shutdown durring the service widow. All
customer traffic will route through other peers during this time.

This scheduled maintenance is part of EntirelyDigital's constant commitment
to providing customers with the very best in Internet service.

Exchange server unavailable for some users (Resolved) High

Affecting System - Exchange 2007

  • 2011/01/14 20:49 - 2011/01/14 22:00
  • Last Updated 2011/01/18 09:50

We are moving some mailboxes to a new storage array and as such they may be unavailable for approx 15 minutes.

Your connection will become available once the operation is complete.

Hosted Exchange Temporarily Unavailable (Resolved) Critical

Affecting Other - Postoffice.entirelydigital.com

  • 2011/01/13 07:52 - 2011/01/13 11:12
  • Last Updated 2011/01/13 07:54

We had a backup run longer than expected and have cut it short this morning. As a preventative measure we have restarted the server. All exchange services will be back online in approximately 5 minutes. All messages queued at the Edge servers will be delivered as soon as it comes online.

Sorry for any inconvenience.

NOC Team

Mail servers updates (Resolved) Low

Affecting Other - Updates

  • 2020/10/12 15:22 - 2010/12/21 22:57
  • Last Updated 2010/12/15 17:07

All mail servers in the EntirelyDigital network will be updated to their latest build starting 22:00 EST. Minor interruptions in mail flow and mailbox access expected during the upgrades.

Hosted Exchange Down (Resolved) Critical

Affecting System - Hosted Exchange Customers - All

  • 2010/12/04 22:22 - 2010/12/10 10:50
  • Last Updated 2010/12/05 12:32

We are experiencing hosted Exchange Connectivity issues.

Currently all messages are flowing correctly, however we will be doing a full database consistency check at approximately 10:00PM EST

Email server down sporatically (Resolved) Critical

Affecting System - Email - Toronto Data Centre

  • 2010/11/04 16:00 - 2010/11/04 16:23
  • Last Updated 2010/11/04 16:24

Email services were temporarily down between 16:00 EST and 16:30 EST today. We have resolved the issue and the server is operating normally.

Michigan Data Centre Network Issue (Resolved) Critical
  • 2010/09/16 11:19 - 2010/09/16 11:28
  • Last Updated 2010/09/16 11:20

We are experiencing connectivity issues with some of our core routers in our Michigan Data Centre.

We are investigating the proplem and hope to have a resolution shortly.

Server Down (Resolved) Critical

Affecting Server - VIPER-DNP01-SH

  • 2010/08/26 07:58 - 2010/08/26 11:34
  • Last Updated 2010/09/09 17:34

We are experiencing an outage with some of our customers websites hosted through the Michigan data centre.

Our NOC team is aware of the issue and is actively working on the problem.

Some webmail clients may not see all messages (Resolved) Medium
  • 2010/06/28 00:00 - 2010/08/01 00:25
  • Last Updated 2010/06/28 23:54

In some cases once you logon to your webmail client, you may not see your messages or you may only see a partial list.

If this is the case your webmail client needs to be reset and messages re-indexed by the application.  Should this be the case, please submit a helpdesk ticket and logout of your webmail.  We will re-initialize your client and advise once you can logon again.

Upgrade to Offsite Backup Server and Software (Resolved) Medium

Affecting Server - OBS

  • 2010/06/03 13:00 - 2010/06/28 23:52
  • Last Updated 2010/06/03 10:19

We will be upgrading our backup server to the latest build 5.5.7.  All installers will also be updated and your autoupdate utility should update your backup software to this version.  You may receive old failure notifications as we purge some of our logs.

Updates to hosted exchange infrastructure (Resolved) Critical

Affecting System - Hosted Exchange 2007

  • 2010/06/02 19:00 - 2010/06/03 10:19
  • Last Updated 2010/06/03 10:19

We will be performing updates to our hosted exchange infrastrucutre.  Mailbox access will be temporarily unavailable at various points during the update.

Mail will be queued at the edge transport level so no messages will be lost.

Updates to Viperhost DNP (Resolved) Medium

Affecting Server - VIPER-DNP01-SH

  • 2010/04/23 21:00 - 2010/04/25 21:00
  • Last Updated 2010/04/29 13:49

We will be updating the control panel software which connects/configures all Viperhost services over the weekend.  You may experience periods where your website / email may not be operational for brief periods (less than 5 minutes at a time) while we restart services.

Updates to Hosted Exchange Server (Resolved) Medium

Affecting System - Hosted Exchange

  • 2010/04/23 21:00 - 2010/04/29 13:49
  • Last Updated 2010/04/22 21:33

Between April 23rd 9:00pm EST and April 25th 9:00pm EST we will be performing maitenance on our Hosted Exchange infrastructure.  You may experience connectivity issues during the weekend only for a few minutes while we restart services.

Viperhost Michigan Data Centre (Resolved) Critical

Affecting System - Viperhost Michigan Data Centre

  • 2010/04/20 14:52 - 2010/04/20 15:00
  • Last Updated 2010/04/20 16:31

EntirelyDigital experienced a failure of a core router in Southfield. Some of
the fail-over mechanisms did not work, requiring manual innervation. This resulted
in an interruption to some customers in the Viperhost Michigan data center. Currently service
has been restored and was unavailable for approximately 9 minutes

Scheduled Network Upgrades (Resolved) Medium

Affecting System - Michigan Data Centre

  • 2010/04/09 00:00 - 2010/04/09 00:00
  • Last Updated 2010/04/13 20:46

Affecting:
Customers in Viperhost Michigan Data Center

Details:
EntirelyDigital Viperhost is scheduling a service window from 7:00PM-10:00PM Friday, April
9th, 2010, to perform upgrades on key network equipment.

Level (3) connectivity will temporarily shutdown for a period of not more then
approximately 15 minutes. All customer traffic will route through other peers during
this time.  As such no downtime is expected