cPanel server restarts (Resolved)
  • Priority - High
  • Affecting System - cPanel servers
  • Multiple cPanel servers are being restarted for pending kernel updates.

    Please be patient while they restart....

  • Date - 2018/10/14 09:11 - 2018/10/14 10:37
  • Last Updated - 2018/10/14 10:27
SmarterMail - Major Upgrade (Resolved)
  • Priority - Medium
  • Affecting Server - ED-DNP01
  • Smartermail servers are getting a major version upgrade including new servers on Windows 2016.  SmarterMail version will be updated to version 16 (from 13.x)

    Here is the link to SmarterMail's release notes for your perusal: https://www.smartertools.com/smartermail/release-notes/16

    Want to learn more about the new features: https://www.smartertools.com/smartermail/whats-new#/mail-content-current

    Updates to the process will be posted here.

    During the migration period the old mail server will be stopped - last bit of data migrated to the new server and then then mail server services are to resume normal operation on the new server.

    ---

    Transfer has been started - all mail services are temporarily stopped.

    ---

    Migration complete.

  • Date - 2018/09/24 20:00
  • Last Updated - 2018/09/24 21:00
Restart in progress (Resolved)
  • Priority - High
  • Affecting Server - whitemountains.webnethost.net
  • This server is being restarted...

    --

    Server has restarted.  Please allow some time for all services to load.

  • Date - 2018/07/20 17:54
  • Last Updated - 2018/07/20 17:59
Updates in progress (Resolved)
  • Priority - Medium
  • Affecting System - R1Soft Backup Servers
  • All R1Soft Backup Servers are currently being updated and have been put in maintenance mode.
    During this time no restore/backup operations are permitted.

    --

    Kernel updates + package updates + R1Soft > 6.6.2 build

    --

    Complete

  • Date - 2018/07/10 10:27
  • Last Updated - 2018/07/10 11:09
Kernel update in progress (Resolved)
  • Priority - High
  • Affecting Server - cpanel43.onlinemountain.com
  • Kernel update in progress - please be patient while the server restarts.

    --

    Update complete

  • Date - 2018/07/01 09:29
  • Last Updated - 2018/07/01 10:52
Cpanel Servers - Kernel Updates (Resolved)
  • Priority - Critical
  • We are updating seversl CPANEL servers kernels - as this is a critical security update
    You will lose connectivity during this time - please be patient while your server restarts.

  • Date - 2018/05/26 16:31 - 2018/05/26 17:23
  • Last Updated - 2018/05/26 16:41
Network Manager Service Down (Resolved)
  • Priority - High
  • Affecting Server - whitemountains.webnethost.net
  • A recent update has made changes to the network manager and broken connectivity to the network.  We are investigating the issue.

    ---

    This issue is now resolved.
    A line was added to the configuration file at the end:
    ONBOOT=no
    That will break it!

  • Date - 2018/03/31 10:30 - 2018/03/31 10:47
  • Last Updated - 2018/03/31 10:47
LiteSpeed WebServer being installed (Resolved)
  • Priority - High
  • Affecting Server - cpanel43.onlinemountain.com
  • LiteSpeed WebServer being installed to reduce loads on this server.
    In addtion CloudLinux MySQL governer is being installed.

    ---

    Strang yum behaviour requires a restart.

  • Date - 2018/03/19 14:07 - 2018/03/20 13:45
  • Last Updated - 2018/03/20 09:45
High Loads (Resolved)
  • Priority - Critical
  • Affecting Server - cpanel43.onlinemountain.com
  • This server is under high load and we are investigating the cause.

    --

    WHM backups were enabled on this server in error causing high disk io.

    We are investigating the cause.

    --

    We have killed the running processes and loads look normal

  • Date - 2018/03/15 19:26
  • Last Updated - 2018/03/15 20:00
Critical updates in progress (Resolved)
  • Priority - Critical
  • Affecting Server - cpanel43.onlinemountain.com
  • A critical kernel update is in progress.  Once the update is complete the server will be restarted.

    --

    reboot in progress

    ---

    reboot complete

  • Date - 2018/03/01 16:55 - 2018/03/01 19:03
  • Last Updated - 2018/03/01 19:03
Meltdown and Spectre BIOS and CPU (Resolved)
  • Priority - High
  • Affecting System - Sever BIOS Updates - Various Firmware Updates
  • IMPORTANT NOTICE: We are actively investigating the recently discovered CPU vulnerabilities known as “Meltdown” (CVE-2017-5754) and “Spectre” (CVE-2017-5715 and CVE-2017-5753). We are closely examining any developments around possible exploits of these vulnerabilities, and taking rigorous steps to roll out patches as they become available. 

    99% of our severs are made by Dell.  We are subject to firmwares updates provided by Dell EMC.

    Note that because of the number of servers involved and the time it takes to install firmware and kernel udpates, these will be unscheduled installs - sometimes during normal operating hours.
    Restarts usually only approximately 5 minutes so there is little impact.

  • Date - 2018/01/30 14:40 - 2018/05/26 16:40
  • Last Updated - 2018/03/01 17:01
Critical updates in progress (Resolved)
  • Priority - Critical
  • Affecting Server - ED-SH-CP8
  • Once updates are completed, this server will be restarted.

  • Date - 2018/02/28 18:22 - 2018/02/28 18:45
  • Last Updated - 2018/02/28 18:23
Server updates in progress (Resolved)
  • Priority - Critical
  • Affecting Server - ED-SH-CP7
  • This server requires critical updates and will be restarted shortly

  • Date - 2018/02/28 17:34 - 2018/02/28 18:07
  • Last Updated - 2018/02/28 17:35
EasyApache 3 > 4 Migration (Resolved)
  • Priority - High
  • Affecting Server - cpanel43.onlinemountain.com
  • New versions of PHP and Apache in progress - other checks in progress as well.

    ---

    Migration has failed and we are re-running the conversion process.

    ---

    Conversion is now complete and we are updating the apache configuration

  • Date - 2018/02/27 10:34 - 2018/02/27 13:53
  • Last Updated - 2018/02/27 13:22
EasyApache 3 > 4 Migration (Resolved)
  • Priority - High
  • Affecting Server - cpanel42.onlinemountain.com
  • We are updating EasyApache on this server from version 3 to 4 to support PHP 7.0 and 7.1

    --

    Conversion has failed and a reinstall is in progress.  We apologize for the inconvenience.

    ---

    Conversion is now complete and we are checking the apache configuration.

    ---

    There is a conflict between CloudLinux and EA4 which we have escalated to cpanel and cloudlinux support.
    Support ticket with cpanel and cloudlinux is in progress

  • Date - 2018/02/26 17:57 - 2018/02/27 12:05
  • Last Updated - 2018/02/27 11:10
cpanel41 - Apache Rebuild for CloudLinux (Resolved)
  • Priority - High
  • Affecting Server - cpanel41.onlinemountain.com
  • Server is currently stable and we are monitoring it.  We are going to schedule a migration of all the accounts on this server to new hardware.

    Current status for Apache:

    Current Time: Thursday, 01-Feb-2018 10:13:43 EST
    Restart Time: Tuesday, 30-Jan-2018 22:28:24 EST
    Parent Server Config. Generation: 14
    Parent Server MPM Generation: 13
    Server uptime: 1 day 11 hours 45 minutes 18 seconds
    Server load: 9.45 8.73 8.63
    Total accesses: 1312316 - Total Traffic: 43.0 GB
    CPU Usage: u158.86 s952.23 cu33175.6 cs8048.17 - 32.9% CPU load
    10.2 requests/sec - 350.1 kB/second - 34.3 kB/request
    28 requests currently being processed, 122 idle workers



    We are trying one more build of apache to see if the issue subsides.
    ---
    Apache running but loads are very high.  We are still looking at the reason.
    ---
    Getting apache to behave has become more difficult than anticipated, even with Level3 cpanel support involved.  We are now making a full backup of this server and may have to restore to different hardware.
    We will continue to update this notice. 
    ---
    Currently evidence is pointing to a DDOS attack on the sites on this server.  We may have to wait until the attack subsides but we are investigating.  There are too many IP's involved to block anything significant.
    ---
    We are now making several updates to see if we can absorb some of this traffic while mitigating some of it.  Server will require a few restarts.
    ---
    Server has been restarted after a successful kernel update.  We are now running a firmware update on some components to see if there is a possible issue with our RAID controller firmware.
    ---
    We will be switching to EA4 vs EA3 and ensuring that CloudLinux is limiting any offending users.
    ---
    Conversion from EA3 to EA4 CloudLinux is now in progress
    --
    In order to speed up this process we are now blocking port 80 and 443 so that the conversion can complete without being bogged down by apache requests.
    ---
    Starting EA3 to EA4 php.ini conversion in the background (EXPECT HEAVY CPU and I/O LOADS)
    ---
    Conversion complete - we are now building the CloudLinux profile in EA4 - this is the last step.
    ---
    MPM worker needs to be changed to MPM prefork for apache on this server for faster apache processing
    ---
    This server is still under high load and we are investigating the cause.
    ---
    By blocking port 80 and 443, server loads are now normal and we can investigate properly
    during this time IMAP/POP and SMTP functions as well as FTP are still avaialbe but websites will not load.  Blocking these ports is necessary so that we can actively look at logs without being bogged down by high loads.
    ---
    We have restarted the server - we have also opened a ticket with cpanel for assistance as this seems to be related to the EasyApache process of going from version 3 to 4 and the MPM configuration.
    ---
    In order for cpanel to investiate, we've opened the ports back up but loads have spiked again - please bear with us while we figure out what's going on with Apache.

  • Date - 2018/01/29 12:01 - 2018/02/26 17:57
  • Last Updated - 2018/02/01 10:15
Server restart in progress (Resolved)
  • Priority - High
  • Affecting Server - cpanel42.onlinemountain.com
  • Server restart in progress
    Kernel updated
    R1Soft backup agent installed for ehanced backups

  • Date - 2018/01/28 16:35 - 2018/01/29 12:01
  • Last Updated - 2018/01/28 16:36
High Loads - ongoing (Resolved)
  • Priority - High
  • Affecting Server - cpanel43.onlinemountain.com
  • Server under high load.  We have observed this behaviour on/off for serveral days.
    This seems to be related to apache only.
    Note that in order to troubleshoot this, we will be turning port 80 and 443 off and back on
    Please check this Network Issue ticket for udpated information.

    ---

    We have noticed an entry in iptables that may be loading an extremely large number of IPs into our firewall, thus causing a high load.
    We have disabled this list and are watching the loads.

  • Date - 2018/01/11 15:21 - 2018/01/24 12:57
  • Last Updated - 2018/01/11 15:38
High Loads (Resolved)
  • Priority - High
  • Affecting Server - cpanel43.onlinemountain.com
  • Server under high load.
    --
    Some services restarted with no effect.  Investigation continues
    --
    High loads seem to be MySQL related and we are investigating further
    --
    combination of exploit scanning with log processing and high mysql load has led to many queued processes.
    we wil allow the scan to complete then continue to monitor.
    --
    we have found a hacked account causing high disk io.  This account has been suspended and files removed.
    loads are normal
    --
    we are rebooting this server
    --
    server is booting up, we are monitoring

     
     

  • Date - 2017/12/20 20:09
  • Last Updated - 2018/01/11 15:21
Imunify Firewall Problem (Resolved)
  • Priority - High
  • Affecting Server - greywood.webnethost.net
  • An error in the Imunify Firewall has forced us to remove it.
    We are now attempting a reinstall - note that Apache may become non-responsive for a few minutes.

  • Date - 2017/12/26 12:11 - 2018/01/11 15:21
  • Last Updated - 2017/12/26 12:12
Server High Load (Resolved)
  • Priority - High
  • Affecting Server - cpanel41.onlinemountain.com
  • We are investigating the cause of this server's very high load.
    Currently, no sites are loading.

    --

    Server is under high load because of a long running backup operation.
    We will allow the backup to complete at this time after which the server loads should go back to normal.

     

  • Date - 2017/12/18 10:31 - 2017/12/20 20:11
  • Last Updated - 2017/12/18 13:50
MySQL databases being moved (Resolved)
  • Priority - Critical
  • Affecting Server - cpanel42.onlinemountain.com
  • In order to protect the integrity of MySQL data on this server, we are moving the data on an emergency basis to a new data drive.
    Please be patient while the moving process completes.

    --

    We are about 33% through the copy process...

    --

    Almost there...

    --

    Task complete

     

  • Date - 2017/12/15 10:17
  • Last Updated - 2017/12/15 11:09
Reseller Limits Not being Enforced (Resolved)
  • Priority - High
  • Affecting System - cPanel/WHM - all versions
  • CPANEL-16366 internal case has been opened for this issue.
    Updates to this case will be added to cpanel's  changelogs when they're available.
    You can check them here:

    https://documentation.cpanel.net/display/CL/68+Change+Log

    --

    This isssue is still not resolved.

     

  • Date - 2017/11/15 11:41 - 2017/12/18 13:50
  • Last Updated - 2017/12/06 22:51
Server not responsive (Resolved)
  • Priority - Critical
  • Affecting Server - whitemountains.webnethost.net
  • We are investigating an issue with this server not responding.

  • Date - 2017/12/06 22:08 - 2017/12/06 22:48
  • Last Updated - 2017/12/06 22:08
Exchange 2010 - Maintenance Window (Resolved)
  • Priority - Medium
  • Affecting System - Exchange 2010
  • We are installing updates to the Exchange infrastructure - you may lose mailbox connectivity while servers restart.  This is expected and service will be restored normally once the server starts up.
    These are critical security patches and are integral to providing a secure environment.

    ---

    Restarts are now in progress, please be patient while servers install their updates on restart.

  • Date - 2017/10/21 17:17 - 2017/11/15 11:47
  • Last Updated - 2017/10/21 17:34
Windows Hosting - Management System Down (Resolved)
  • Priority - High
  • Affecting System - manage.entirelydigital.com
  • manage.entirelydigital.com is currently unavailable as we have found a possible vulnerability.  Any required changes should be submitted via support ticket until the portal becomes live once again.

  • Date - 2017/10/21 17:16 - 2017/11/15 11:47
  • Last Updated - 2017/10/21 17:17
Reboot in progress (Resolved)
  • Priority - High
  • Affecting Server - ED-SH-CP7
  • The server is being restarted to install a critical kernel update.

  • Date - 2017/09/25 15:21 - 2017/09/25 15:30
  • Last Updated - 2017/09/25 15:22
Disk Space Issue (Resolved)
  • Priority - Critical
  • Affecting Server - cpanel42.onlinemountain.com
  • We have found a disk space issue in / on this server and have cleared out some files.  We are now restarting the server to ensure all services are running normally.
    ETA 10 minutes

  • Date - 2017/09/22 09:25 - 2017/09/25 09:25
  • Last Updated - 2017/09/22 09:26
Backup Drive Replacement (Resolved)
  • Priority - Medium
  • Affecting Server - cpanel43.onlinemountain.com
  • We are rebooting this server to replace a backup drive.

    --

    Drive is being formatted - io is high so loads may be higher than normal.  In addition start up processes cause higher than normal loads.  Please exercise patience during this maintenance window.

    --

    We have started an out of schedule full backup which will impact DISK IO - during this time you will see higher than normal loads.  Please exercise patience during this maintenance window.

  • Date - 2017/08/29 21:52 - 2017/09/22 09:26
  • Last Updated - 2017/08/30 10:17
Apache will not start (Resolved)
  • Priority - Critical
  • Affecting Server - cpanel43.onlinemountain.com
  • Apache is not starting on this server due to multiple configuration issues from a failed cpanel update.
    We have escalated this to cpanel support and they are currently investigating.

    The issue has been resolved and sites should now be loading correctly with loads on the server back to normal.

  • Date - 2017/08/01 11:29
  • Last Updated - 2017/08/01 13:55
High Loads - DOS Attack (Resolved)
  • Priority - Critical
  • Affecting Server - cpanel42.onlinemountain.com
  • This server is under high loads and we are working to resolve.

  • Date - 2017/07/05 16:15 - 2017/07/06 10:03
  • Last Updated - 2017/07/05 16:16
High Loads - DOS Attack (Resolved)
  • Priority - Critical
  • Affecting Server - ED-SH-CP7
  • This server is under high loads and we are working to resolve.

  • Date - 2017/07/05 16:15 - 2017/07/06 10:03
  • Last Updated - 2017/07/05 16:15
Server Restart in progress (Resolved)
  • Priority - High
  • Affecting Server - cpanel41.onlinemountain.com
  • This server requires a reboot - now in progress....

  • Date - 2017/07/01 10:23 - 2017/07/01 10:36
  • Last Updated - 2017/07/01 10:24
MySQL Server down (Resolved)
  • Priority - High
  • Affecting Server - cpanel41.onlinemountain.com
  • MySQL server cannot strat due to the mysql data drive mounted as read only.  We are investigating the cause and looking for a resolution.

    We were able to get the drive back online and MySQL is now operating normally with no errors.
    --

    Reopeing this issue as some users are experiencing authentication errors.
    We are working with cpanel to resolve.

  • Date - 2017/06/24 15:40 - 2017/06/26 11:35
  • Last Updated - 2017/06/25 21:22
Apache build failed (Resolved)
  • Priority - High
  • Affecting Server - cpanel42.onlinemountain.com
  • Rebuild of apache has failed and we have escalated the issue to cpanel support

  • Date - 2017/06/12 21:11 - 2017/06/12 22:09
  • Last Updated - 2017/06/12 21:12
High Loads (Resolved)
  • Priority - High
  • Affecting Server - cpanel41.onlinemountain.com
  • We are seeing extremely high loads on this server and are investigating the cause.

  • Date - 2017/05/17 10:30 - 2017/06/12 21:12
  • Last Updated - 2017/05/17 10:31
High Loads (Resolved)
  • Priority - High
  • Affecting Server - cpanel42.onlinemountain.com
  • This server is experiencing high loads and we are currently investigating possible causes.

    UPDATE
    -we have found some IP addresses that are attacking some wordpress sites on the server and are mitigating the effects.

    UPDATE
    -this server is once again under high loads.

    UPDATE
    -we have found the issue - a huge ip.pag file from the DDOS attack previously reported was causing high disk IO in apache which was causing a cascading effect and failure.
    -should be ok now.

  • Date - 2017/05/15 14:50
  • Last Updated - 2017/05/16 17:22
root compromised - backup/restore in progress (Resolved)
  • Priority - High
  • Our security scans have revealed that this server is root compromised.  As such, we have started provisioning a new server (with a newer OS) and will:
    1. Backup your cpanel account
    2. Take the current suitland.webnethost.net server offline.
    --during this time, your site and email will be offline.
    3. Once the new server is online, restore your cpanel account to the new server.  Once your account is restored, your website and email will be online once again.
    4. There will be no IP change to your domain.

    This process may take 1-3 hours with intermittent connectivity to your site and email.  This is an urgent matter and must be addressed immediately.

    Please check back for updates.

    ---

    Apr 24 - 17:36 EST (GMT -5:00)

    The new server is set up and the backups are complete. I will be taking the compromised server down shortly and putting the new server in it's place. I will then run scripts on the new server that will install your services. After the post-install scripts are finished I will begin restoring your accounts to their proper IP address.

    I will update you on the progress as best I can. Please be patient, however, as this will take some time.

    ---

    Apr 25 - 10:00 EST (GMT -5:00)

    Account restores are complete - cpanel account passwords have all been reset.  We will advise your new password shortly.  Some MIVA Merchant sites on this server are not functional - we are working with MIVA support to get the miva components re-installed.

    ---

    Apr 25 - 13:05 EST (GMT -5:00)

    Please note that all cpanel passwords were reset to protect the integrity of your account.  Please login to your client area at https://www.entirelydigital.com/manage and then click on your services icon, then drill in to the account that is on this server.  From there you will be able to reset your cpanel password from the left menu.

  • Date - 2017/04/24 15:54 - 2017/05/15 14:50
  • Last Updated - 2017/04/25 13:06
Auto SSL Failure (Resolved)
  • Priority - High
  • The AutoSSL feature of this server has failed and certificates which expired today have not been renewed.  We are investigating the cause and hope to have a resolution shortly.

    This issue has been resolved and the SSL certificates are queued at Comodo for installation.

  • Date - 2017/02/07 16:24 - 2017/04/24 15:55
  • Last Updated - 2017/02/07 18:25
Drive Maintenance (Resolved)
  • Priority - High
  • potomac.webnetservers.com is displaying some issues with I/O. We require a maintenance window with the server offline for 1-4 hours to clone the primary drive.

  • Date - 2017/01/16 22:00 - 2017/01/17 04:50
  • Last Updated - 2017/01/16 20:48
Server unresponsive (Resolved)
  • Priority - Critical
  • This server has become unresponsive and the issue is being actively investigated.

    --

    This isssue is now resolved.

  • Date - 2016/12/29 13:20 - 2016/12/29 13:43
  • Last Updated - 2016/12/29 13:43
SMTP Slowdown (Resolved)
  • Priority - Critical
  • Affecting Server - cpanel43.onlinemountain.com
  • Because of a spamming account, this server's SMTP queue is being cleared of spam messages
    There are several hundred thousand messages to clear.
    Once clear we will divert outbound mail via an alternate IP.

  • Date - 2016/12/12 18:22 - 2016/12/13 11:03
  • Last Updated - 2016/12/12 18:22
SMTP Slowdown (Resolved)
  • Priority - Critical
  • Affecting Server - cpanel42.onlinemountain.com
  • Because of a spamming account, this server's SMTP queue is being cleared of spam messages
    There are several hundred thousand messages to clear.
    Once clear we will divert outbound mail via an alternate IP.

  • Date - 2016/12/12 18:21 - 2016/12/13 11:03
  • Last Updated - 2016/12/12 18:22
Hardware Maintenance Window (Resolved)
  • Priority - High
  • A hardware maintenance window is required for this server in the times shown
    All services on this server will be unavailable during the maintenance window as the server will be powered off
    Actual time will be approximately 1 hour.

  • Date - 2016/12/06 21:00 - 2016/12/07 14:13
  • Last Updated - 2016/12/06 16:02
Hardware Maintenance Window (Resolved)
  • Priority - Medium
  • A hardware maintenance window is required for this server in the times shown
    Dec 1, 01:00-05:00 EST (GMT -5:00)
    All services on this server will be unavailable during the maintenance window as the server will be powered off
    Actual time will be approximately 1 hour.

  • Date - 2016/12/01 01:00
  • Last Updated - 2016/12/06 16:00
IIS6 WH-01 Being Moved to New HOST (Resolved)
  • Priority - Medium
  • Affecting Server - ED-DNP01
  • Our legacy IIS 6 server is being moved to a new host for improved stability.  During this move, websites on this server will be inaccessible.
    Move time is estimated at 15minutes.

    Migration complete

  • Date - 2016/11/19 17:10 - 2016/11/19 17:39
  • Last Updated - 2016/11/19 17:35
Server Loads Extremely High (Resolved)
  • Priority - High
  • We are investigating extremely high loads on this server.

  • Date - 2016/11/16 16:28 - 2016/11/16 17:56
  • Last Updated - 2016/11/16 16:28
Kernel update and reboot (Resolved)
  • Priority - Medium
  • Affecting Server - cpanel41.onlinemountain.com
  • System kernel update in progress + reboot of server

    --

    FSCK in progress forced - startup delay.

    ---

    Reboot complete.

  • Date - 2016/09/03 11:16
  • Last Updated - 2016/09/03 12:11
Server Load High (Resolved)
  • Priority - High
  • Affecting Server - cpanel43.onlinemountain.com
  • The high load on this server is being investigated.  While our investigation proceeds, port 80 has been blocked to allow our technicians the ability to work in a timely manner.  All other ports remain open (POP, IMAP, FTP, SMTP, Cpanel)

    ---

    Apache / PHP rebuild in progress.

    ---

    Build complete - we are now doing a kernel update.

    ---

    System being rebooted now - new kernel installed and updates complete.

    ---

    We've found the root cause of this issue and have resolved it.  Loads are now normal and the server is responsive.  We are monitoring it continuously.


  • Date - 2016/09/02 13:07 - 2016/09/03 11:13
  • Last Updated - 2016/09/02 17:36
Incoming email problem (Resolved)
  • Priority - Critical
  • Affecting Server - ED-SH-CP7
  • Currently, all email is queued and not being delivered to individual user mailboxes.  This is an issue with a failed cpanel update.
    We are investigating and hoping to have a resolution shortly.

    ALL DELIVERY ISSUES HAVE NOW BEEN RESOLVED.

  • Date - 2016/08/12 16:28 - 2016/08/12 20:11
  • Last Updated - 2016/08/12 20:11
Hosted Exchange Update (Resolved)
  • Priority - Medium
  • Affecting System - Hosted Microsoft Exchange
  • We have scheduled some mailbox server updates - Microsoft Hosted Exchange 2010
    These are general service and stability updates for Exchange and Windows Server as well as funcitionality updates for OWA (Outlook Web Access)

    Updates started

    1st reboot in progress....

    Reboot complete.
    Starting second set of updates

    2nd reboot in progress....

    Exchange updates in progres - your mailbox will be unavailalbe until completion...

    Mailboxes now available

    Further updates may make your mailbox unavailable for a few minutes....

    Last set of updates require a reboot to complete.  Reboot in progress....

    UPDATES COMPLETE

  • Date - 2016/08/06 08:00
  • Last Updated - 2016/08/06 12:04
cp.neiko.net - login unavailable (Resolved)
  • Priority - Medium
  • Affecting Server - VIPER-HELM-SH
  • We are currently experiencing login issues with http://cp.neiko.net - this is a licensing problem with the control panel server which we are working to resolve.

  • Date - 2016/07/21 11:39
  • Last Updated - 2016/07/29 14:10
cPanel servers (Resolved)
  • Priority - Critical
  • Affecting System - cPanel Security Team - CVE-2016-3714 ImageMagick
  • cPanel Security Team - CVE-2016-3714 ImageMagick

    Background Information

    On Tuesday, May 3 2016, ImageMagick announced a vulnerability in all versions of the ImageMagick software. ImageMagick is a software
    package commonly used by web services to process images.

    Impact

    One of the reported vulnerabilities can potentially be exploited for remote code execution (RCE).

    Releases

    ImageMagick has not released a fix, but plans to publish a new version of ImageMagic with the fixes soon. cPanel normally releases all builds at once in order to limit the ability to reverse engineer fixes. However, this vulnerability is already wildly known and we have seen reports of its use. In this instance, we plan to release builds as soon as they become available.

    At this time the following builds are available:
    11.56    11.56.0.13
    EDGE 11.55.9999.193
    CURRENT  11.56.0.13
    RELEASE  11.56.0.13

    We are currently updating all our cpanel servers to this build.  If you cannot access cpanel or WHM then a cpanel update is in progress.  Please be patient as the update completes.  

  • Date - 2016/05/04 12:18 - 2016/05/04 15:14
  • Last Updated - 2016/05/04 12:20
Update in progress (Resolved)
  • Priority - Medium
  • Affecting Server - cpanel43.onlinemountain.com
  • cpanel43.onlinemountain.com is doing a cpanel udpate to 11.56 - as such cpanel and WHM are currently unavailable.

    Normal operations should resume shortly.

  • Date - 2016/04/30 11:43 - 2016/04/30 12:15
  • Last Updated - 2016/04/30 11:44
Server Inaccessible - Unscheduled Maintenance (Resolved)
  • Priority - Critical
  • Affecting Server - cpanel43.onlinemountain.com
  • We are investigating an issue with this server - updates to follow in this Network Issue notification

    --UPDATE--

    We see an issue with the installed kernel - we are reverting to the previous kernel and rebooting
    Investigation will continue.

    --UPDATE--

    Server is operational - we will be rebooting the server to a newer kernel later tonight.

    --UPDATE--

    Kernel has been updated correctly and verified - we are rebooting the server so that dependencies with CloudLinux function correctly.

    --REBOOT COMPLETE

    --UPDATE--

    We are once again noticing high loads on this server - invesigtating the issue - we will cycle through services and block specific ports to track down the culprit.

    --UPDATE--

    We are now running further updates to kernel and other drivers/cloudlinux in an attempt to narrow down the possible issue.

    --UPDATE--

    In order to expedite this process, we've had to block port 80 on this server as the update process was taking too long.  We have restarted the update process with port 80 blocked (low load) and hope to reboot the server shortly.

    --UPDATE--

    we have resolved the issue and loads are now normal.

  • Date - 2016/03/06 18:01 - 2016/03/07 16:16
  • Last Updated - 2016/03/07 16:16
Server Reboot (Resolved)
  • Priority - High
  • Affecting Server - cpanel41.onlinemountain.com
  • A kernel dependancy requires that we reboot the server for maximum stability.
    Reboot has begun and server should be back online in 5-10 min.

    --REBOOT COMPLETE

  • Date - 2016/03/06 20:15
  • Last Updated - 2016/03/06 20:29
SHA1 Certs - onlinemountain.com (Resolved)
  • Priority - Critical
  • Affecting System - SSL Certificates - SHA1 replacement
  • We are currently replacing all onlinemountain.com COMODO SSL certificates with SHA2 encrypted certificates.

    Prior to replacement you will get a warning in Google Chrome.

    Thank you for your patience

  • Date - 2015/12/19 12:04 - 2016/03/06 18:02
  • Last Updated - 2015/12/19 12:05
Core Infrastructure Upgrades (Resolved)
  • Priority - Medium
  • Affecting System - Core Network
  • We will be updating some core switches.  During the upgrades there may be some downtime you experience - Approximately 5 minutes while switchports come online.
    -This maintanance windows has been postponed.  A new item will be opened once the maintenance window has been determined.

  • Date - 2015/12/06 18:30 - 2015/12/06 20:59
  • Last Updated - 2015/12/06 17:31
Core Infrastructure Upgrades (Resolved)
  • Priority - Medium
  • Affecting System - Toronto DC
  • We will be updating some core switches.  During the upgrades there may be some downtime you experience - Approximately 5 minutes while switchports come online.
    -This maintanance windows has been postponed.  A new item will be opened once the maintenance window has been determined.

  • Date - 2015/11/22 21:00 - 2015/11/22 22:00
  • Last Updated - 2015/11/22 20:16
Apache 500 Errors - Cloudlinux (Resolved)
  • Priority - Critical
  • Affecting Server - cpanel42.onlinemountain.com
  • Most websites throwing 500 errors due to a Cloudlinux CageFS issue.
    We have escalted this to our Level3 support and hope to have a resolution shortly.

  • Date - 2015/11/01 14:38 - 2015/11/01 18:05
  • Last Updated - 2015/11/01 17:39
All 208.92.222.x IP network is down. (Resolved)
  • Priority - High
  • Affecting System - All 208.92.222.x IP network
  • Due to a DDOS in progress all 208.92.222.x IP network is down.
    We are working with our downstream provider to resolve the issue.

  • Date - 2015/09/15 19:56 - 2015/09/15 20:13
  • Last Updated - 2015/09/29 09:13
Reboot in progress (Resolved)
  • Priority - High
  • Affecting Server - cpanel41.onlinemountain.com
  • This server was under very heavy load - a full restart is in progress.
    - file system quotas check in progress (fsck) - this must complete before the server comes back online.

  • Date - 2015/05/29 16:02 - 2015/05/29 17:15
  • Last Updated - 2015/05/29 16:28
IMAP/POP logins failing (Resolved)
  • Priority - Critical
  • Affecting Server - cpanel41.onlinemountain.com
  • We are aware of the issue regarding this server.
    POP3 and IMAP connections may fail with an authenticator error.
    We are actively investigating this issue and should have a resolution shortly.

  • Date - 2015/03/25 16:18 - 2015/03/25 18:17
  • Last Updated - 2015/03/25 16:19
Server down (Resolved)
  • Priority - Critical
  • Affecting Server - cpanel42.onlinemountain.com
  • We are awere of the issue with this server and are actively investigating.

    --issue is now resolved.

  • Date - 2015/03/20 15:11
  • Last Updated - 2015/03/20 15:23
MAINTENANCE - HARDWARE Upgrade (Resolved)
  • Priority - Medium
  • Affecting Server - cpanel41.onlinemountain.com
  • We are currently doing a hardware upgrade on this server. During the upgrade no services are available.

  • Date - 2015/02/25 21:21 - 2015/02/25 21:44
  • Last Updated - 2015/02/25 21:22
PHP update in progress (Resolved)
  • Priority - Medium
  • Affecting Server - cpanel41.onlinemountain.com
  • During updates websites will be unavailable.

  • Date - 2015/02/23 20:02 - 2015/02/25 21:42
  • Last Updated - 2015/02/23 20:02
Webmail/IMAP/POP Access down (Resolved)
  • Priority - High
  • Affecting System - Mail server - WEBM1 - TOR DC 1
  • We are seeing intermittent webmail issues with this server (https://webm1.entirelydigital.com) and are working to resolve.
    You may see continuing intermittent outages while we restart services and update core files.

  • Date - 2015/02/06 14:20 - 2015/02/10 10:23
  • Last Updated - 2015/02/06 14:21
POP/IMAP Webmail Authentication Failure (Resolved)
  • Priority - High
  • Affecting Server - cpanel41.onlinemountain.com
  • This server is refusing to authenticate POP, IMAP and WEBMAIL connections (via IMAP) once more.  We've escalated this to cPanel.net for immediate investigation and permanent resolution.

    ---

    This is now resolved but we are investigating the cause.

  • Date - 2015/02/03 08:06 - 2015/02/03 10:49
  • Last Updated - 2015/02/03 08:43
Apache/PHP recompile (Resolved)
  • Priority - Medium
  • Affecting Server - cpanel41.onlinemountain.com
  • We are updating apache on this server - websites will be down for approximately 10-15 min.

    ---

    ETA changed - additional updates required for security reasons.

    ---

    Now complete - websites available.

  • Date - 2015/02/01 17:08
  • Last Updated - 2015/02/02 10:48
IMAP/POP/SMTP Authentication Failure (Resolved)
  • Priority - High
  • Affecting Server - cpanel41.onlinemountain.com
  • This server is experiencing IMAP/POP/SMTP Authentication Failure for most accounts.
    We are actively investigating this issue and hope to have a resolution shortly.

  • Date - 2015/01/28 10:31 - 2015/01/29 12:01
  • Last Updated - 2015/01/28 10:32
Server maintenance (Resolved)
  • Priority - Medium
  • Affecting Server - cpanel42.onlinemountain.com
  • Server is currently undergoing maintenance - some hosting services will be affected during our upgrades

  • Date - 2015/01/18 07:12 - 2015/01/18 10:34
  • Last Updated - 2015/01/18 09:14
Server Maintenance (Resolved)
  • Priority - Medium
  • Affecting Server - cpanel43.onlinemountain.com
  • Server is currently undergoing maintenance - some hosting services will be affected during our upgrades

  • Date - 2015/01/18 07:13 - 2015/01/18 09:34
  • Last Updated - 2015/01/18 09:14
Server under high load (Resolved)
  • Priority - High
  • Affecting Server - cpanel41.onlinemountain.com
  • This server experienced a system failure during updates and we had to gracefully restart it.  Unfortunately a forced filesystem check is in progress and must complete.
    Hopefully won't be much longer.

    --

    We are trying to deterimine what processes are causing high load - during our troubleshooting process, different services will become unavailable.

    --

    Currently httpd is turned off while we update cloudlinux cageFS - email and other services are up - this is required to ensure that cloudlinux LVE is working properly before we start httpd

    --

    Kernel updates are complete as well as updates to CloudLinux LVE - we're rebooting the server to load the new kernel
    We'll be watching the server carefully over the next hours to ensure the loads stay in check.

    --

    Server being monitored and processes are being kept in check

    --

    In cooperation with cpanel.net support we are close to a resolution regarding the performance of this server.  We appreciate your patience.

    --

    We've identified this is a DDOS attack and are taking steps to attempt to mitigate the attack

    --

    The DDOS seems to be targeted to specific customers - we are working to mitigate the attack however DDOS can be difficult to stop.

    --

  • Date - 2015/01/13 09:40 - 2015/01/14 09:18
  • Last Updated - 2015/01/13 19:50
Cloud Backup Server Upgrade (Resolved)
  • Priority - Medium
  • Affecting Server - OBS
  • Our cloud backup service is being updated to the latest build - during the upgrade offsite backup and online portal access is unavailable

  • Date - 2015/01/09 13:45 - 2015/01/09 14:00
  • Last Updated - 2015/01/09 13:46
Apache/PHP recompile (Resolved)
  • Priority - Medium
  • Affecting Server - cpanel43.onlinemountain.com
  • PHP is being rebuilt (native 5.4) to include MBSTRING as it was not included in the original build.

  • Date - 2014/12/08 15:47 - 2014/12/19 12:31
  • Last Updated - 2014/12/08 15:48
Hosted MS Exchange Certificate Warning (Resolved)
  • Priority - Critical
  • Affecting System - Hosted MS Exchange
  • Clients are receiving invalid SSL certificate warnings in webmail and outlook - we are working with Godaddy SSL to re-issue the SSL certificate in question.

    UPDATE - Certificate has been rekeyed and installed - there are no furhter issues with webmail

  • Date - 2014/12/08 10:44
  • Last Updated - 2014/12/08 11:32
PHP update in progress (Resolved)
  • Priority - Medium
  • Affecting Server - cpanel42.onlinemountain.com
  • PHP and Apache being updated to the latest build

  • Date - 2014/12/06 10:45 - 2014/12/06 11:39
  • Last Updated - 2014/12/06 10:45
PHP update in progress (Resolved)
  • Priority - Medium
  • Affecting Server - cpanel41.onlinemountain.com
  • PHP and Apache being updated to the latest build

  • Date - 2014/12/06 10:45 - 2014/12/06 12:42
  • Last Updated - 2014/12/06 10:45
PHP update in progress (Resolved)
  • Priority - Medium
  • Affecting Server - cpanel43.onlinemountain.com
  • PHP and Apache being updated to the latest build

  • Date - 2014/12/06 10:44 - 2014/12/06 11:39
  • Last Updated - 2014/12/06 10:45
Cloud Backup Server Upgrade (Resolved)
  • Priority - Medium
  • Affecting Server - OBS
  • Update of Cloud Backup Server in progress.
    Disk Check in progress

    ---

    Disk check is still in progress but almost complete - should be operational shortly.

  • Date - 2014/09/08 16:08 - 2014/09/09 22:00
  • Last Updated - 2014/09/08 20:17
PHP update in progress (Resolved)
  • Priority - High
  • Affecting Server - cpanel41.onlinemountain.com
  • To address possible security vulnerabilities in php - we are updating PHP to 5.4.31 outside the normal maintenance window
    This is only a MID-LEVEL security issue.

  • Date - 2014/08/19 10:16 - 2014/08/20 12:32
  • Last Updated - 2014/08/19 10:17
Failed cpanel update being re-run (Resolved)
  • Priority - High
  • This server had a failed cpanel update causing some issues - we are re-running it now to stabilize the server.

  • Date - 2014/08/18 10:22 - 2014/08/19 10:16
  • Last Updated - 2014/08/18 10:23
Rebuild Apache/PHP (Resolved)
  • Priority - Medium
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
    PHP is being updated to the newest minor version - no major version upgrades are being installed.

  • Date - 2014/07/30 18:25 - 2014/07/31 10:22
  • Last Updated - 2014/07/30 18:25
Rebuild Apache/PHP (Resolved)
  • Priority - Medium
  • Affecting Server - ED-SH-CP6
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
    PHP is being updated to the newest minor version - no major version upgrades are being installed.

    -- update --
    We are now upgrading MySQL to 5.5 for compatiblity and upgrade paths to WHM 11.44

    -- update --
    We are now upgrading WHM to 11.44

  • Date - 2014/07/26 10:56 - 2014/07/28 15:29
  • Last Updated - 2014/07/26 12:58
Rebuild Apache/PHP (Resolved)
  • Priority - Medium
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
    PHP is being updated to the newest minor version - no major version upgrades are being installed.

  • Date - 2014/07/23 17:01 - 2014/07/26 10:56
  • Last Updated - 2014/07/23 17:15
Rebuild Apache/PHP (Resolved)
  • Priority - Medium
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
    PHP is being updated to the newest minor version - no major version upgrades are being installed.

  • Date - 2014/07/23 11:15 - 2014/07/23 12:21
  • Last Updated - 2014/07/23 11:14
Rebuild Apache/PHP (Resolved)
  • Priority - Medium
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
    PHP is being updated to the newest minor version - no major version upgrades are being installed.

  • Date - 2014/07/16 22:02 - 2014/09/08 16:08
  • Last Updated - 2014/07/16 19:02
Rebuild Apache/PHP (Resolved)
  • Priority - Medium
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
    PHP is being updated to the newest minor version - no major version upgrades are being installed.

  • Date - 2014/07/16 22:02 - 2014/07/16 19:02
  • Last Updated - 2014/07/16 19:02
Rebuild Apache/PHP (Resolved)
  • Priority - Medium
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
    PHP is being updated to the newest minor version - no major version upgrades are being installed.

  • Date - 2014/07/16 19:01 - 2014/09/08 16:08
  • Last Updated - 2014/07/16 19:01
Rebuild Apache/PHP (Resolved)
  • Priority - Medium
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
    PHP is being updated to the newest minor version - no major version upgrades are being installed.

  • Date - 2014/07/16 19:01 - 2014/09/08 16:08
  • Last Updated - 2014/07/16 19:01
Rebuild Apache/PHP (Resolved)
  • Priority - Medium
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
    PHP is being updated to the newest minor version - no major version upgrades are being installed.

  • Date - 2014/07/16 19:00 - 2014/07/17 11:53
  • Last Updated - 2014/07/16 19:00
Rebuild Apache/PHP (Resolved)
  • Priority - Medium
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
    PHP is being updated to the newest minor version - no major version upgrades are being installed.

  • Date - 2014/07/16 19:00 - 2014/07/17 12:46
  • Last Updated - 2014/07/16 19:00
Rebuild Apache/PHP (Resolved)
  • Priority - High
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.

  • Date - 2014/07/16 16:53 - 2014/07/16 17:12
  • Last Updated - 2014/07/16 16:53
Rebuild Apache/PHP (Resolved)
  • Priority - High
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.

  • Date - 2014/07/16 16:53 - 2014/07/16 17:28
  • Last Updated - 2014/07/16 16:53
Rebuild Apache/PHP (Resolved)
  • Priority - High
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.

  • Date - 2014/07/16 15:14 - 2014/07/16 16:50
  • Last Updated - 2014/07/16 15:14
Rebuild Apache/PHP (Resolved)
  • Priority - High
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.

  • Date - 2014/07/16 15:14 - 2014/07/16 16:50
  • Last Updated - 2014/07/16 15:14
Rebuild Apache/PHP (Resolved)
  • Priority - High
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.

  • Date - 2014/07/16 14:59 - 2014/07/16 16:50
  • Last Updated - 2014/07/16 15:14
Rebuild Apache/PHP (Resolved)
  • Priority - High
  • An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.

  • Date - 2014/07/16 14:45 - 2014/07/16 14:55
  • Last Updated - 2014/07/16 14:45
Server not responding (Resolved)
  • Priority - High
  • This server is currently down - we are actively investigating the cause.

    ---

    Server is booting howerver a mandatory forced FSCK (disk check) is now in progress.  Usually takes approx 20 min

  • Date - 2014/07/02 10:10 - 2014/07/02 11:58
  • Last Updated - 2014/07/02 10:14
Server reboot in progress (Resolved)
  • Priority - High
  • We have had to restart this server and it is currently performing a quota operation mandatory in OpenVZ.  Once this operation completes, the server should come back online.
    ETA 20 min

  • Date - 2014/06/22 14:46 - 2014/06/22 15:43
  • Last Updated - 2014/06/22 14:47
Server listed at bl.spamcop.net (Resolved)
  • Priority - High
  • We have resolved the issue causing this listing and have requested delist from spamcop.net.

  • Date - 2014/06/18 11:51 - 2014/06/22 10:17
  • Last Updated - 2014/06/18 11:51
Queue flooded with spam (Resolved)
  • Priority - Medium
  • A hacked mail account has caused the mail queue to be flooded with spam.  We have identified and stopped the spammer and clearing the queue - at this time the server is under high load cleaning he affected queue.

  • Date - 2014/06/14 17:44 - 2014/06/14 22:30
  • Last Updated - 2014/06/14 17:45
Server mysql under high load (Resolved)
  • Priority - Medium
  • We are investigating a continuous load issue with this server - in this process we are shutting down mysql for approx 20min to see if the processes are to heavy for the server.

  • Date - 2014/06/11 21:42 - 2014/06/14 17:44
  • Last Updated - 2014/06/11 21:43
Server reboot in progress (Resolved)
  • Priority - High
  • We have had to restart this server because of load issue investigations.  Because of the restart, a mandatory disk check is now in progress.  We realize the timing on the forced disk check is during normal business hours however it is better to allow the disk check to complete at this time.  ETA 30 min.

  • Date - 2014/06/11 11:20 - 2014/06/11 21:42
  • Last Updated - 2014/06/11 11:21
Server not responding (Resolved)
  • Priority - High
  • We are investigating a server issue affecting multiple servers including this one.  We hope to have an update soon.

  • Date - 2014/06/06 02:00 - 2014/06/06 11:09
  • Last Updated - 2014/06/10 11:09
MySQL Upgrade / cPanel Upgrade (Resolved)
  • Priority - Medium
  • In order to update cpanel/WHM to the latest version, MySQL 5.5 is required.  MySQL 5.0 is deprecated and no longer supported.  As such, in order to bring this server in line with all others in our infrastructure we are in the process of updating MySQL.  During this time, websites requiring mysql queries may not be available.

  • Date - 2014/06/01 09:48 - 2014/06/06 08:40
  • Last Updated - 2014/06/01 09:50
Apache/PHP recompile (Resolved)
  • Priority - Medium
  • Due to some errors we are force to recompile Apache and PHP to the latest versions (2.2.x and 5.4.x respectively) - during this process your websites may become unavailable.

  • Date - 2014/05/27 10:43 - 2014/05/27 11:27
  • Last Updated - 2014/05/27 10:43
Queue flooded with spam (Resolved)
  • Priority - High
  • This server is experiencing slowdowns due to the mail queue being full of spam.  We have identified the offending sender and disabled their account.  We are clearing the queue and will take steps to mitigate this behaviour in the future.

  • Date - 2014/05/05 16:22 - 2014/05/05 21:52
  • Last Updated - 2014/05/05 16:23
cPanel Update (Resolved)
  • Priority - Medium
  • With a successful MySQL update completed, cPanel is being updated to the lastest RELEASE version 11.42.1.x

  • Date - 2014/04/20 11:56 - 2014/05/04 20:46
  • Last Updated - 2014/04/20 11:57
MySQL upgrade in progress (Resolved)
  • Priority - Medium
  • MySQL is being upgraded to 5.5 - during the upgrade process, websites which use databases will be unavailable

  • Date - 2014/04/20 11:19 - 2014/04/20 11:56
  • Last Updated - 2014/04/20 11:20
PHP update in progress (Resolved)
  • Priority - Medium
  • PHP/Apache is being recompiled to the latest versions

  • Date - 2014/04/20 11:08 - 2014/04/20 11:41
  • Last Updated - 2014/04/20 11:09
Server updates (Resolved)
  • Priority - Medium
  • This server is being updated - as such some sites are not responding as apache and PHP are being updated.

  • Date - 2014/03/15 21:35 - 2014/03/15 21:53
  • Last Updated - 2014/03/15 21:35
WHM/Cpanel Uprade to 11.40.11 (Resolved)
  • Priority - Medium
  • We are currently upgrading MySQL from 5.0.x to 5.5.x
    We are currently upgrading WHM/cPanel from 11.38.x (EOL) to 11.40.11 - current release
    You may see some interruption in web services while the system updates it's packages.

  • Date - 2014/02/28 20:00
  • Last Updated - 2014/02/28 21:10
Backup LAN GB (Resolved)
  • Priority - Medium
  • Affecting System - Backup Swich - TOR1 DC1
  • One of our backup switches used for backend communcations is down.  This will affect communiction between websites and databases and backup operations.
    We are investigating the issue and hope to have a resolution shortly.

  • Date - 2014/02/03 09:50 - 2014/02/03 13:58
  • Last Updated - 2014/02/03 09:51
PHP upgrade to 5.4 (Resolved)
  • Priority - Medium
  • We have now started upgrading / recompiling apache with PHP 5.4 on this server.
    Should be done shortly.
    Build complete - PHP is now 5.4.24

  • Date - 2014/01/20 20:09 - 2014/01/20 21:00
  • Last Updated - 2014/01/20 20:38
MS Information store hung (Resolved)
  • Priority - Critical
  • Affecting System - Hosted MS Exchange 2007
  • Customers of our hosted MS Exchange have noticed intermittent connectivity this morning - We are working to resolve this and have initiated a server restart.

    --UPDATE--
    An integrity check and fix is required on one of the information stores.
    This necessitates bringing the store offline during the check.
    During this time some mailboxes may be unavailable depending on which store your mailbox resides.
    We appreciate your patience during this check/fix operation.
    --

    --UPDATE--
    Please note that is is a long procedure.  We estimate the time to completion at 14:30
    Messages will be queued on other servers until the information store becomes available
    --

    --UPDATE--
    Test revealed further errors which are now being fixed through another procedure.  We esitmate another 4 hours before completion.
    Our apologies for the inconvenience
    --

    --UPDATE--
    Integrity check is taking much longer than anticipated and we may have to restore from backup as we cannot continue to keep the server offline.
    We will work to also restore the data from the restore point until this morning but keep customer up to date via this post.
    --

    --UPDATE / SUNDAY MORNING--
    We have added some additional disk arrays and will be doing a restore from backup shortly.  Unfortunately our DB fix did not complete successfully and so we must restore from backup.  We will work to recover data manually from the corrupt DB after the server is back online.
    Again, we thank you for your patience.
    --

    --UPDATE--
    Restore is in progress and completion is estimated in approx 3 hours
    --

  • Date - 2013/12/14 09:15
  • Last Updated - 2013/12/16 20:00
IMAP/POP connections failing (Resolved)
  • Priority - Critical
  • We are investigating an issue whereby all IMAP and POP accounts cannot login to retrieve mail

  • Date - 2013/12/14 09:35 - 2013/12/14 10:19
  • Last Updated - 2013/12/14 09:36
Server not responding - High Load (Resolved)
  • Priority - Critical
  • This server is currently under high load and may not respond to web/email requests.
    -Update
    We have found an excessive amount of messages in the mail queue and are investigating.

  • Date - 2013/12/13 12:38 - 2013/12/13 13:01
  • Last Updated - 2013/12/13 12:53
Server not responding - High Load (Resolved)
  • Priority - High
  • This server is currently under high load and may not respond to web/email requests.

  • Date - 2013/12/13 12:38 - 2013/12/14 09:16
  • Last Updated - 2013/12/13 12:38
Smartermail - Michigan (Resolved)
  • Priority - High
  • Affecting Server - VIPER-HELM-SH
  • Currently our MI email server webm1-MI.entirelydigital.com is at 100%CPU - likely due to a DDOS attack - we are currently investigating.

  • Date - 2013/12/12 12:54 - 2013/12/13 12:38
  • Last Updated - 2013/12/12 12:55
Mandatory Disk Check (Resolved)
  • Priority - High
  • We have had to restart this server and force a mandatory disk check to ensure integrity.  During this time, the server and it's services will be unavailable.
    We anticipate downtime of approx 1 hour.

  • Date - 2013/12/11 18:23 - 2013/12/11 21:47
  • Last Updated - 2013/12/11 21:47
Apache offline temporarily (Resolved)
  • Priority - Medium
  • We are recompling apache to address some web server issues.  It should be up shortly.

  • Date - 2013/12/09 14:49 - 2013/12/14 10:19
  • Last Updated - 2013/12/09 14:50
cpanel20.onlinemountain.com currently down (Resolved)
  • Priority - Critical
  • WHM/Cpanel host cpanel20.onlinemountain.com is currently not repsonding.  We have our NOC technician looking at the server and we are investigating the issue.
    We hope to have it up and running shortly.

    --UPDATE--
    Server is undergoing a forced fsck (disk check) which we cannot stop.  This is a mandatory check after a hard reset.  We will investigate the cause of the issue to ensure that the resource drain does not re-occur.
    --UPDATE--

    --UPDATE--
    Server is running - we are now performing system checks before bringing WHM/cpanel back online
    --UPDATE--

    --UPDATE--
    Server has finished checks and WHM/cPanel is now initializing
    --UPDATE--

  • Date - 2013/12/02 10:19
  • Last Updated - 2013/12/02 12:22
cPanel Webmail Auto-Responder issue (Resolved)
  • Priority - Low
  • Affecting System - cPanel Webmail
  • There is an issue with webmail on all cPanel servers.  cPanel 11.40 does not allow you to set custom start and stop date/times for your autoresponder.  No calendar drop down appears when you click on the date.  This is a known issue and will be fixed with the next cPanel update.

  • Date - 2013/10/15 16:18
  • Last Updated - 2013/11/22 14:35
System restart (Resolved)
  • Priority - High
  • ed-sh-cp1.entirelydigital.com is currently undergoing a file system check.  Once complete the server will be back onine.

  • Date - 2013/10/26 08:58 - 2013/10/26 13:38
  • Last Updated - 2013/10/26 08:59
Exchange 2007 System Down (Resolved)
  • Priority - Critical
  • Affecting System - Exchange 2007
  • We are experiencing an outage on our Exchange 2007 server - we are working as quickly as possible to resolve the issue.  Mailbox access, webmail access is affected.
    UPDATE - All hardware upgraded and issue is now resolved

  • Date - 2013/10/01 07:30 - 2013/10/07 00:00
  • Last Updated - 2013/10/15 16:18
SmarterMail MI (Resolved)
  • Priority - Medium
  • Affecting System - webmail MI webmail.neiko.net
  • We are upgrading all our SmarterMail Servers to enterprise 11.6.x

  • Date - 2013/09/24 21:55
  • Last Updated - 2013/09/24 22:03
Datacenter Maintenance (Resolved)
  • Priority - Medium
  • Affecting System - UPS Battery
  • EntirelyDigital will be replacing a single UPS Battery at the Toronto data center location on Tuesday May 14, 2013 from 16:01hrs to 17:30hrs. There is no expected service impact during this maintenance

  • Date - 2013/05/14 17:01 - 2013/11/27 10:14
  • Last Updated - 2013/05/10 16:09
IP Blacklisted (Resolved)
  • Priority - Critical

  • Due to an internal spammer/compromised account, 208.92.222.102 has been blacklisted.

    We are investigating the issue and will request immediate delist once the account has been disabled.

  • Date - 2013/04/03 10:13 - 2013/04/11 10:14
  • Last Updated - 2013/04/03 10:13
High load - SMTP/POP/IMAP unresponsive (Resolved)
  • Priority - High
  • Affecting System - cas1.entirelydigital.com/smtp75.entirelydigital.com

  • We are experiencing very high loads on this server and are investigating the cause.  SMTP/POP and IMAP connections will time out and server will appear unresponsive.

  • Date - 2013/03/06 14:11 - 2013/03/08 17:15
  • Last Updated - 2013/03/06 14:12
Drive Failure (Resolved)
  • Priority - Critical
  • ED-SH-CP1.entirelydigital.com experienced a drive failure a short time ago.  We are investigating and hope to have the server up and running shortly.

  • Date - 2012/11/29 11:25 - 2013/03/04 22:27
  • Last Updated - 2012/11/29 11:25
Telephone Support Unavailable (Resolved)
  • Priority - Critical
  • Affecting System - Telephone Technical Support
  • We are currently experiencing issues with our telephone service. As you may or may not know, our phone servers are located in our US office. Last night a severe storm knocked out power, Internet, and essential services to almost a million homes in Northern Virginia. Luckily our US staff was unharmed by the storm. As of right now, the Gov. Bob McDonnell has declared a state of emergency for Virginia.

    Our vendors providing the DIDs and 'dial tone' for all of our phone lines have been forced to shut down all power, including UPS, so emergency crews can work to repair damaged power lines safely.

    As inconvenient and unfortunate as this is, we are doing everything we can to assist/comply with emergency crews and services in hopes of having this resolved soon.

    This telephony outage has no impact on helpdesk support services we provide.

    We do not have a firm ETA however we will keep you updated as soon as we get additional information from the upstream providers and/or Virginia Dominion Power.

    You can reach our support staff as usual by emailing helpdesk@entirelydigital.com

    We thank you for your patience and understanding. Our thoughts are with those who lost loved ones in last nights storm.

  • Date - 2012/07/01 00:00 - 2012/07/01 18:22
  • Last Updated - 2012/07/01 00:54
Network Outage - USA Datacenter (Resolved)
  • Priority - Critical
  • Affecting Other - Network Outage - USA Datacenter
  • We are experiencing a network outage in our USA datacenter - this is not a server issue but an issue with a downstream provider

    We are currently invesitgating and this issue is priority 1 - Critical

    -- update this issue is now resolved and we are actively monitoring the hardware in question

  • Date - 2012/06/29 15:15 - 2012/06/29 15:35
  • Last Updated - 2012/06/29 15:56
Michigan Data Center Outage (Resolved)
  • Priority - Critical
  • Affecting System - Michigan Data Center - some servers
  • We are experiencing a major outage in our Michigan data center and are actively investigating.

    Please do not email our helpdesk regarding your site or email being down as we are already aware of the issue and it is a network issue not a server issue or an issue with your particular domain.

  • Date - 2012/06/21 08:07 - 2012/06/21 10:05
  • Last Updated - 2012/06/21 08:14
US Datacenter Outage (Resolved)
  • Priority - Critical
  • Affecting Other - US Datacenter
  • We are experiencing a system wide datacenter outage in our USA datacenter.

    An investigation is currently in progress.  We are aware of the issue so please do not call our support line or email our helpdesk as your website/service/email will come back on line automatically once the issue is resolved.

  • Date - 2012/06/03 15:41 - 2012/06/03 00:00
  • Last Updated - 2012/06/03 16:54
Server listed at senderbase.org (Resolved)
  • Priority - Medium
  • We are currently investigating why this server (208.92.222.102) is listed at senderbase.org. This may affect you sending email to certain domains that use Cisco firewalls as their antispam solution.

  • Date - 2012/04/30 12:38 - 2012/06/03 16:54
  • Last Updated - 2012/04/30 12:39
Kernel security, bug fixes, and enhancement u (Resolved)
  • Priority - Medium
  • Affecting System - CPanel Servers
  • We will be scheduling an important kernel security, bug fixes and enhancement update for all our CentOS 5.x servers this weekend.

    This would take place as far as possible during the off-peak hours. A reboot is required to complete the upgrade. The downtime should not exceed 30 minutes and it will be minimize as much as possible. We will track each server till it returns to service after the upgrade and reboot.

    This upgrade is scheduled as follows:
    Date: 21 April 2012 (Saturday) to 23 April 2012 (Monday)
    Time: Between 2AM and 8AM EST

    Details of Kernel security, bug fixes and enhancement update
     
    * A flaw in the xfrm6_tunnel_rcv() function in the Linux kernel's IPv6 implementation could lead to a use-after-free or double free flaw in tunnel6_rcv(). A remote attacker could use this flaw to send
    specially-crafted packets to a target system that is using IPv6 and also has the xfrm6_tunnel kernel module loaded, causing it to crash. (CVE-2012-1583, Important)

    If you do not run applications that use xfrm6_tunnel, you can prevent the xfrm6_tunnel module from being loaded by creating (as the root user) a "/etc/modprobe.d/xfrm6_tunnel.conf" file, and adding the following line to it:

    blacklist xfrm6_tunnel

    This way, the xfrm6_tunnel module cannot be loaded accidentally. A reboot is not necessary for this change to take effect.

    This update also fixes various bugs and adds an enhancement.

  • Date - 2012/04/20 02:00 - 2012/07/01 00:55
  • Last Updated - 2012/04/20 15:38
HSphere Server Migration (Resolved)
  • Priority - Medium
  • We are in the process of migrating the Hsphere control panel server. During this move, access to account functions will not be available.  Web and email services are unaffected.

  • Date - 2012/04/10 19:57 - 2012/04/12 11:42
  • Last Updated - 2012/04/10 19:58
Michigan IDC - Unscheduled Maintenance (Resolved)
  • Priority - High
  • Affecting System - Michigan IDC - All Services
  • We are currently performing some unscheduled maintenance on our Michigan SAN infrastructure.  We estimate service will resume at approx 4am EST

  • Date - 2012/01/13 01:14 - 2012/01/13 12:01
  • Last Updated - 2012/01/13 01:15
Email services unavailable (Resolved)
  • Priority - High
  • Affecting System - some email servers
  • One of our email servers is currently not accepting requests for POP/IMAP - service will be restored shortly.

  • Date - 2011/12/06 09:35 - 2011/12/06 10:21
  • Last Updated - 2011/12/06 09:36
Cpanel Licence Error (Resolved)
  • Priority - Medium
  • CP3.shieldhost.com is currently showing an error.  This affects webmail and control panel access.  We are working with the control panel vendor to resolve this.

  • Date - 2011/11/14 10:00 - 2011/11/14 11:34
  • Last Updated - 2011/11/14 10:01
Unable to access plesk / download mail (Resolved)
  • Priority - Critical
  • Affecting Server - NET-209
  • Clients trying to access POP and IMAP mail will not be able to login and will recieve an error in their mail clients as well as webmail.

    We are aware of the issue and working toward a resolution.

  • Date - 2011/10/14 16:01 - 2011/10/17 14:17
  • Last Updated - 2011/10/14 16:20
Worldwide BES issues (Resolved)
  • Priority - High
  • Affecting System - Hosted Exchange 2007 / BES - Blackberry Enterprise Server
  • Blackberry's (RIM) outage has spread to Canada and affects all BES users on hosted MS exchange. This is not an issue with EntirelyDigital's networks but stems from RIM itself.

    There is no ETA for resolution.

    iPhone, Android, Windows Phone users are unaffected by this outage.

  • Date - 2011/10/12 07:33 - 2011/10/17 14:17
  • Last Updated - 2011/10/12 09:39
Hosted Exchange 2007 - Maintenance (Resolved)
  • Priority - Medium
  • Affecting Other - Hosted Exchange 2007
  • Dear Microsoft Exchange User, (french message follows)
    On Sunday March 27th, 2011 at approximately 22:00 (10pm) EST (GMT -5.00) we will be doing some software and hardware upgrades on our Microsoft Exchange infrastructure.
    During this time your mailbox will be unavailable. The duration of this maintenance window is estimated to be approximately 1 hour.
    With these upgrades we will be increasing our capacity which means larger mailbox quotas coming soon.
    This upgrade is part of many infrastructure upgrades we will be doing in 2011.
    Regards,
    EntirelyDigital NOC Team

    ---

    Cher utilisateur de Microsoft Exchange,

    Le 27 Mars 2011, à environ 22h00 EST (GMT -5.00), nous allons faire quelques mises à jour logicielles et matérielles de notre infrastructure Microsoft Exchange.
    Pendant ce temps, votre boîte aux lettres ne seront pas disponibles. La durée de cette fenêtre de maintenance est estimée à environ 1 heure.

    Avec ces améliorations, nous augmenterons notre capacité ce qui signifie que les quotas de boîtes aux lettres plus à venir.

    Cette mise à jour fait partie de nombreuses améliorations à l'infrastructure que nous allons faire en 2011.

    Cordialement,

    EntirelyDigital équipe CNP

  • Date - 2011/03/27 22:00 - 2011/04/22 20:12
  • Last Updated - 2011/03/26 19:29
Network Upgrades (Resolved)
  • Priority - Low
  • Affecting Other - Michigan IDC
  • EntirelyDigital is scheduling a service window from Tuesday 1/18/2011 11:00PM
    to Wednesday 1/19/2011 2:00AM to perform upgrades on key network equipment in our Michigan Data Centre. We
    do not expect any network outages durring this service window. Customers may notice
    a temporary changes in routing.

    Level (3) connectivity will temporarily shutdown durring the service widow. All
    customer traffic will route through other peers during this time.

    This scheduled maintenance is part of EntirelyDigital's constant commitment
    to providing customers with the very best in Internet service.

  • Date - 2011/01/18 00:00 - 2011/01/26 20:42
  • Last Updated - 2011/01/26 20:42
Exchange server unavailable for some users (Resolved)
  • Priority - High
  • Affecting System - Exchange 2007
  • We are moving some mailboxes to a new storage array and as such they may be unavailable for approx 15 minutes.

    Your connection will become available once the operation is complete.

  • Date - 2011/01/14 20:49 - 2011/01/14 22:00
  • Last Updated - 2011/01/18 09:50
Hosted Exchange Temporarily Unavailable (Resolved)
  • Priority - Critical
  • Affecting Other - Postoffice.entirelydigital.com
  • We had a backup run longer than expected and have cut it short this morning. As a preventative measure we have restarted the server. All exchange services will be back online in approximately 5 minutes. All messages queued at the Edge servers will be delivered as soon as it comes online.

    Sorry for any inconvenience.

    NOC Team

  • Date - 2011/01/13 07:52 - 2011/01/13 11:12
  • Last Updated - 2011/01/13 07:54
Mail servers updates (Resolved)
  • Priority - Low
  • Affecting Other - Updates
  • All mail servers in the EntirelyDigital network will be updated to their latest build starting 22:00 EST. Minor interruptions in mail flow and mailbox access expected during the upgrades.

  • Date - 2020/10/12 15:22 - 2010/12/21 22:57
  • Last Updated - 2010/12/15 17:07
Hosted Exchange Down (Resolved)
  • Priority - Critical
  • Affecting System - Hosted Exchange Customers - All
  • We are experiencing hosted Exchange Connectivity issues.

    Currently all messages are flowing correctly, however we will be doing a full database consistency check at approximately 10:00PM EST

  • Date - 2010/12/04 22:22 - 2010/12/10 10:50
  • Last Updated - 2010/12/05 12:32
Email server down sporatically (Resolved)
  • Priority - Critical
  • Affecting System - Email - Toronto Data Centre
  • Email services were temporarily down between 16:00 EST and 16:30 EST today. We have resolved the issue and the server is operating normally.

  • Date - 2010/11/04 16:00 - 2010/11/04 16:23
  • Last Updated - 2010/11/04 16:24
Michigan Data Centre Network Issue (Resolved)
  • Priority - Critical
  • We are experiencing connectivity issues with some of our core routers in our Michigan Data Centre.

    We are investigating the proplem and hope to have a resolution shortly.

  • Date - 2010/09/16 11:19 - 2010/09/16 11:28
  • Last Updated - 2010/09/16 11:20
Server Down (Resolved)
  • Priority - Critical
  • Affecting Server - VIPER-DNP01-SH
  • We are experiencing an outage with some of our customers websites hosted through the Michigan data centre.

    Our NOC team is aware of the issue and is actively working on the problem.

  • Date - 2010/08/26 07:58 - 2010/08/26 11:34
  • Last Updated - 2010/09/09 17:34
Some webmail clients may not see all messages (Resolved)
  • Priority - Medium
  • In some cases once you logon to your webmail client, you may not see your messages or you may only see a partial list.

    If this is the case your webmail client needs to be reset and messages re-indexed by the application.  Should this be the case, please submit a helpdesk ticket and logout of your webmail.  We will re-initialize your client and advise once you can logon again.

  • Date - 2010/06/28 00:00 - 2010/08/01 00:25
  • Last Updated - 2010/06/28 23:54
Upgrade to Offsite Backup Server and Software (Resolved)
  • Priority - Medium
  • Affecting Server - OBS
  • We will be upgrading our backup server to the latest build 5.5.7.  All installers will also be updated and your autoupdate utility should update your backup software to this version.  You may receive old failure notifications as we purge some of our logs.

  • Date - 2010/06/03 13:00 - 2010/06/28 23:52
  • Last Updated - 2010/06/03 10:19
Updates to hosted exchange infrastructure (Resolved)
  • Priority - Critical
  • Affecting System - Hosted Exchange 2007
  • We will be performing updates to our hosted exchange infrastrucutre.  Mailbox access will be temporarily unavailable at various points during the update.

    Mail will be queued at the edge transport level so no messages will be lost.

  • Date - 2010/06/02 19:00 - 2010/06/03 10:19
  • Last Updated - 2010/06/03 10:19
Updates to Viperhost DNP (Resolved)
  • Priority - Medium
  • Affecting Server - VIPER-DNP01-SH
  • We will be updating the control panel software which connects/configures all Viperhost services over the weekend.  You may experience periods where your website / email may not be operational for brief periods (less than 5 minutes at a time) while we restart services.

  • Date - 2010/04/23 21:00 - 2010/04/25 21:00
  • Last Updated - 2010/04/29 13:49
Updates to Hosted Exchange Server (Resolved)
  • Priority - Medium
  • Affecting System - Hosted Exchange
  • Between April 23rd 9:00pm EST and April 25th 9:00pm EST we will be performing maitenance on our Hosted Exchange infrastructure.  You may experience connectivity issues during the weekend only for a few minutes while we restart services.

  • Date - 2010/04/23 21:00 - 2010/04/29 13:49
  • Last Updated - 2010/04/22 21:33
Viperhost Michigan Data Centre (Resolved)
  • Priority - Critical
  • Affecting System - Viperhost Michigan Data Centre
  • EntirelyDigital experienced a failure of a core router in Southfield. Some of
    the fail-over mechanisms did not work, requiring manual innervation. This resulted
    in an interruption to some customers in the Viperhost Michigan data center. Currently service
    has been restored and was unavailable for approximately 9 minutes

  • Date - 2010/04/20 14:52 - 2010/04/20 15:00
  • Last Updated - 2010/04/20 16:31
Scheduled Network Upgrades (Resolved)
  • Priority - Medium
  • Affecting System - Michigan Data Centre
  • Affecting:
    Customers in Viperhost Michigan Data Center

    Details:
    EntirelyDigital Viperhost is scheduling a service window from 7:00PM-10:00PM Friday, April
    9th, 2010, to perform upgrades on key network equipment.

    Level (3) connectivity will temporarily shutdown for a period of not more then
    approximately 15 minutes. All customer traffic will route through other peers during
    this time.  As such no downtime is expected

  • Date - 2010/04/09 00:00 - 2010/04/09 00:00
  • Last Updated - 2010/04/13 20:46