Affecting Server - cpanel42.onlinemountain.com
We have found that the mail server (Send / SMTP) is not able to send mail and mail is being rejected with:
"Temporary local problem" error
We have begun our investigation
Update: we are still investigating the cause for this, but it seems to be an issue with Exim's ability to resolve host names.
No other server functions are affected
Affecting System - OpenVZ Host Node
Emergency maintenance is required on some OpenVZ Host VPS nodes.
This will require restarts. During this time, your VPS will be unavailable.
Check back here to see when the maintenance window is complete.
---
Some hosts have been updated and rebooted we will continue to run updates throughout the day
Affecting Server - cpanel41.onlinemountain.com
The MySQL server on this cpanel server is down and will not start. We have escalated the issue to cpanel support for further investigation
MySQL has been restored to full operation.
Affecting System - General Network
We have seen that some customers are experiencing slow connections to our services.
EntirelyDigital is not experiencing any issues at our Data Centres at this time.
We have noticed that the slowdowns have been from certain Rogers customers and some American ISPs that may be routing through Rogers.
We have an open ticket with our downstream providers and are currently investigating the issue further.
If you are experiencing an issue, please do a trace route to 192.34.55.1 and send us a screen shot or copy paste of the results by opening a support ticket. Please also let us know your ISP and where you are physically located in the support ticket.
At this time, we have no reports of slowdowns. We will continue to monitor the situation but it seems the issue has resolved "itself". And by "itself", we mean by the ISP backbone and routing providers that were experiencing intermittent issues earlier today.
Affecting System - Technical Support / Telephone
While the access to power has improved we are still experiencing some delays.
Estimates on full power restoration indicate that full power should be restored some time this weekend.
In the meantime we are doing our best to support you as quickly as we can.
Due to major and ongoing power outages in the maritimes our response times via our helpdesk (email/website) may see some delays. Telephone support may not be available at times. It is best to submit tickets online by logging in or sending an email to helpdesk.
We appreciate your patience while some of our support agents work to refuel their generators and work through this difficult situation.
Affecting Server - cpanel43.onlinemountain.com
This server is scheduled for restart at 22:00 EST - updates are required to resolve an ongoing issue with auto responders producing bounce back errors.
Affecting System - All cPanel/WHM servers
All ability to restore files/databases on cpanel servers is currently unavailable via the Acronis icon in your cpanel account.
We are working with Acronis at this time and hope to have a resolution shortly.
Backups continue to proceed as scheduled are uninterrupted - only the ability to perform restore functions from within cpanel are affected.
If you require a file restore please open support ticket.
Affecting System - Cable Internet
Currently - any connectivity products based on Rogers Telecommunications are down Canada Wide - this includes:
There is no ETA for resolution - this is a Rogers issue - opening a helpdesk ticket with us will not provide resolution for your connectivity issue.
Backup agent and kernel updates are being applied to this server - during the reboot process all services will be unavailable.
Affecting Server - cpanel42.onlinemountain.com
RESCHEDULED
This server is scheduled for updates today durning this maintenance window during which a server reboot is required. During reboot, no services are available.
Affecting Server - blackroot.redundant-webservers.com
Backup agent needs an update - In addition, critical kernel and PHP updates are being applied to this server which requires a restart.
Backup agent needs an update - In addition, critical kernel and PHP updates are being applied to this server which requires a restart.
Affecting System - email - windows hosting - Smartermail
This server is being restarted due to unscheduled maintenance required.
During this time - smtp75.entirelydigital.com and https://webm1.entirelydigital.com are not accessible.
--
Update - the second reboot is in progress. Unfortunately there is no way to accelerate the process or interrupt it at this stage. We must wait for the update process to complete.
--
Update - the process is now complete and normal services have resumed.
Affecting System - Hosted Microsoft Exchange
All our hosted Exchange Servers are being patched immediately due to a zero-day exploit which is being used by malicious groups.
This is an out of band emergency maintenance window.
The patch is referenced here:
https://blogs.microsoft.com/on-the-issues/2021/03/02/new-nation-state-cyberattacks/
Affecting System - OpenVZ
OpenVZ Virtual Server Hosts - all servers
All services will be unavailable at certain times during this maintenance window as the hosts reboot.
Maintenance items include but are not exclusive to:
Friday December 18 22:33 maintenance window opened
Friday December 18 23:47 maintenance window closed
Saturday December 19 22:16 maintenance window opened
Saturday December 19 22:49 maintenance window closed
Affecting System - ns1.entirelydigital.com, net0.bluecho.net
Unscheduled maintenance of this host is in progress...
Affecting System - Hosting - Windows - Viperhost/Neiko
Unscheduled maintenance of this host is in progress....
Affecting System - VMWARE hosts - some servers
Please note that we have scheduled maintenance on the following host server:
VMWARE hosts - some servers
During the following time window: Friday November 28 22:00 EST - Friday November 28 24:00 EST
All services on this server will be unavailable at certain times during this maintenance window. We do expect that the process will take much less time, however we have allowed for a the full 2 hours should there be unexpected downtime. Please note that we will perform this update after a full backup of your VPS has been taken. During the maintenance your VPS will be shut down.
Maintenance items include but are not exclusive to:
You can check back here for status updates.
Saturday Nov 28 22:30
Backup of all virtual servers is in progress and maintenance will proceed as scheculed
Saturday Nov 28 23:05
VPS servers are not available during the maintenance process which is now in progress.
Sunday Nov 29 00:05
Maintenance Complete
Affecting System - Offsite Backup
We are aware of an issue with our backup server whereby your backup agent cannot connect to our server.
We are actively working with the software vendor to resolve this issue. Rest assured that your backup data is in tact and that once resolved, your agent will connect and continue where it left off.
--
This issue is now resolved
We are working on getting this server online - it is operational however we have network connectivity issues at the moment.
Affecting Server - greywood.webnethost.net
This server was non-responsive and is being restarted.
Affecting Server - whitemountains.webnethost.net
UPDATE - Resolved.
We are currently investigating the issue with this server - updates posted here.
This issue is now resolved - all databases are loaded and sites working properly. Should you have any issues, please login and open a support ticket.
The MySQL engine on this server is currently not running - we are actively investigating the cause.
Update - 11:00am - EST - we are restoring the mysql data structure from backup will update this network issue with it's progress
Update - 7:00pm - EST - the data has all been restored but we are seeing that local connections from websites are not possible. We are working to resolve this one last issue and hope to have a resolution shortly.
Server maintenance is now in progress...
Notification of critical maintenance required for this server.
|
Estimated Start: |
December 17 @ 09:00 pm EST |
|
Estimated End: |
December 17 @ 12:00 am EST |
|
Estimated Duration: |
Approximately 3 Hours |
|
Affected systems: |
All services on ED-SH-CP7.entirelydigital.com (Email and Web) |
|
Impact: |
All services on this server will be unavailable during this window |
|
Root cause: |
Server storage upgrade required |
|
Contact Person: |
Please open a support ticket by logging in |
Affecting System - Traffic Forwarding Server
We have now stopped forwarding traffic from our old Datacenter to our new Datacenter IPs.
Our official stop date is August 31, however in order to transition properly and force check to see if anyone has not completed DNS updates we've stopped forwarding traffic.
If you need traffic forwarded for a specific IP, please open a support ticket.
Provide: old IP, new IP and the reason for forwarding and why you cannot complete an immediate DNS change.
ALL possibility to forward traffic stops without exception August 31 at 11:59pm.
Affecting System - Microsoft Exchange Email
We are experiencing problems with our Domain Controllers that authenticate you to Microsoft Exchange.
We are aggressively investigating the issue and hope to have a resolution shortly.
--
Update - we continue to work on this issue and hopeful to have the matter resolved soon
--
All services restored and queued emails delivered - no incoming email was lost.
Affecting System - Listed cPanel servers
The following servers have been scheduled for move to our new datacenter:
Please check back here for migration updates.
---
12pm EST - Update - Servers have been moved and are now reachable on their new IP addresses - forwarding of traffic has started - please be patient as we add all the new IPs and enable all the forwarding rules.
---
12:57pm EST - Update - All IP / DNS changes are complete - forwarding is in place
Affecting System - Internet transit services
This message is to advise you of the following upcoming network maintenance work that could impact the Toronto DC1 transit link.
As a result of this maintenance Internet transit services for the Data Centre could be potentially impacted and you could experience periods of degradation of services (packet loss, connectivity drops) while upstream path re-convergence occurs.
Planned Start Date: June 6, 2019 12:00 AM (Eastern Time)
Planned End Date: June 6, 2019 06:00 AM (Eastern Time)
Impact Duration: Outage up to 30Min(s)
Scope of Work: Network gear upgrades
We apologize for any inconvenience that this may cause.
Affecting System - PayPal
We are unable to process PayPal payments at this time. We are working with PayPal to resolve this issue and hope to have a resolution in place within the next 2 days.
We appreciate your patience.
Affecting Server - whitemountains.webnethost.net
A hung backup process has caused the server to be unstable and a reboot is required ad this time
---
Server has restarted and we are watching the boot process to ensure there are no errors.
Affecting Server - blackroot.redundant-webservers.com
After fixing a hung update process, we are now rebooting this server
---
Reboot complete and all updates were successful.
Affecting Server - whitemountains.webnethost.net
A critical BIOS update requires a restart. Now in progress.
--
BIOS is complete, we are now completing other firmware updates and will restart once again.
Affecting System - cPanel servers
Multiple cPanel servers are being restarted for pending kernel updates.
Please be patient while they restart....
Affecting Server - ED-DNP01
Smartermail servers are getting a major version upgrade including new servers on Windows 2016. SmarterMail version will be updated to version 16 (from 13.x)
Here is the link to SmarterMail's release notes for your perusal: https://www.smartertools.com/smartermail/release-notes/16
Want to learn more about the new features: https://www.smartertools.com/smartermail/whats-new#/mail-content-current
Updates to the process will be posted here.
During the migration period the old mail server will be stopped - last bit of data migrated to the new server and then then mail server services are to resume normal operation on the new server.
---
Transfer has been started - all mail services are temporarily stopped.
---
Migration complete.
Affecting Server - whitemountains.webnethost.net
This server is being restarted...
--
Server has restarted. Please allow some time for all services to load.
Affecting System - R1Soft Backup Servers
All R1Soft Backup Servers are currently being updated and have been put in maintenance mode.
During this time no restore/backup operations are permitted.
--
Kernel updates + package updates + R1Soft > 6.6.2 build
--
Complete
Affecting Server - cpanel43.onlinemountain.com
Kernel update in progress - please be patient while the server restarts.
--
Update complete
We are updating seversl CPANEL servers kernels - as this is a critical security update
You will lose connectivity during this time - please be patient while your server restarts.
Affecting Server - whitemountains.webnethost.net
A recent update has made changes to the network manager and broken connectivity to the network. We are investigating the issue.
---
This issue is now resolved.
A line was added to the configuration file at the end:
ONBOOT=no
That will break it!
Affecting Server - cpanel43.onlinemountain.com
LiteSpeed WebServer being installed to reduce loads on this server.
In addtion CloudLinux MySQL governer is being installed.
---
Strang yum behaviour requires a restart.
Affecting Server - cpanel43.onlinemountain.com
This server is under high load and we are investigating the cause.
--
WHM backups were enabled on this server in error causing high disk io.
We are investigating the cause.
--
We have killed the running processes and loads look normal
Affecting Server - cpanel43.onlinemountain.com
A critical kernel update is in progress. Once the update is complete the server will be restarted.
--
reboot in progress
---
reboot complete
Affecting System - Sever BIOS Updates - Various Firmware Updates
IMPORTANT NOTICE: We are actively investigating the recently discovered CPU vulnerabilities known as “Meltdown” (CVE-2017-5754) and “Spectre” (CVE-2017-5715 and CVE-2017-5753). We are closely examining any developments around possible exploits of these vulnerabilities, and taking rigorous steps to roll out patches as they become available.
Affecting Server - ED-SH-CP8
Once updates are completed, this server will be restarted.
This server requires critical updates and will be restarted shortly
Affecting Server - cpanel43.onlinemountain.com
New versions of PHP and Apache in progress - other checks in progress as well.
---
Migration has failed and we are re-running the conversion process.
---
Conversion is now complete and we are updating the apache configuration
Affecting Server - cpanel42.onlinemountain.com
We are updating EasyApache on this server from version 3 to 4 to support PHP 7.0 and 7.1
--
Conversion has failed and a reinstall is in progress. We apologize for the inconvenience.
---
Conversion is now complete and we are checking the apache configuration.
---
There is a conflict between CloudLinux and EA4 which we have escalated to cpanel and cloudlinux support.
Support ticket with cpanel and cloudlinux is in progress
Affecting Server - cpanel41.onlinemountain.com
Server is currently stable and we are monitoring it. We are going to schedule a migration of all the accounts on this server to new hardware.
Current status for Apache:
Current Time: Thursday, 01-Feb-2018 10:13:43 EST
Restart Time: Tuesday, 30-Jan-2018 22:28:24 EST
Parent Server Config. Generation: 14
Parent Server MPM Generation: 13
Server uptime: 1 day 11 hours 45 minutes 18 seconds
Server load: 9.45 8.73 8.63
Total accesses: 1312316 - Total Traffic: 43.0 GB
CPU Usage: u158.86 s952.23 cu33175.6 cs8048.17 - 32.9% CPU load
10.2 requests/sec - 350.1 kB/second - 34.3 kB/request
28 requests currently being processed, 122 idle workers
Affecting Server - cpanel42.onlinemountain.com
Server restart in progress
Kernel updated
R1Soft backup agent installed for ehanced backups
Affecting Server - cpanel43.onlinemountain.com
Server under high load. We have observed this behaviour on/off for serveral days.
This seems to be related to apache only.
Note that in order to troubleshoot this, we will be turning port 80 and 443 off and back on
Please check this Network Issue ticket for udpated information.
---
We have noticed an entry in iptables that may be loading an extremely large number of IPs into our firewall, thus causing a high load.
We have disabled this list and are watching the loads.
Affecting Server - cpanel43.onlinemountain.com
Server under high load.
--
Some services restarted with no effect. Investigation continues
--
High loads seem to be MySQL related and we are investigating further
--
combination of exploit scanning with log processing and high mysql load has led to many queued processes.
we wil allow the scan to complete then continue to monitor.
--
we have found a hacked account causing high disk io. This account has been suspended and files removed.
loads are normal
--
we are rebooting this server
--
server is booting up, we are monitoring
Affecting Server - greywood.webnethost.net
An error in the Imunify Firewall has forced us to remove it.
We are now attempting a reinstall - note that Apache may become non-responsive for a few minutes.
Affecting Server - cpanel41.onlinemountain.com
We are investigating the cause of this server's very high load.
Currently, no sites are loading.
--
Server is under high load because of a long running backup operation.
We will allow the backup to complete at this time after which the server loads should go back to normal.
Affecting Server - cpanel42.onlinemountain.com
In order to protect the integrity of MySQL data on this server, we are moving the data on an emergency basis to a new data drive.
Please be patient while the moving process completes.
--
We are about 33% through the copy process...
--
Almost there...
--
Task complete
Affecting System - cPanel/WHM - all versions
CPANEL-16366 internal case has been opened for this issue.
Updates to this case will be added to cpanel's changelogs when they're available.
You can check them here:
https://documentation.cpanel.net/display/CL/68+Change+Log
--
This isssue is still not resolved.
Affecting Server - whitemountains.webnethost.net
We are investigating an issue with this server not responding.
Affecting System - Exchange 2010
We are installing updates to the Exchange infrastructure - you may lose mailbox connectivity while servers restart. This is expected and service will be restored normally once the server starts up.
These are critical security patches and are integral to providing a secure environment.
---
Restarts are now in progress, please be patient while servers install their updates on restart.
Affecting System - manage.entirelydigital.com
manage.entirelydigital.com is currently unavailable as we have found a possible vulnerability. Any required changes should be submitted via support ticket until the portal becomes live once again.
The server is being restarted to install a critical kernel update.
Affecting Server - cpanel42.onlinemountain.com
We have found a disk space issue in / on this server and have cleared out some files. We are now restarting the server to ensure all services are running normally.
ETA 10 minutes
Affecting Server - cpanel43.onlinemountain.com
We are rebooting this server to replace a backup drive.
--
Drive is being formatted - io is high so loads may be higher than normal. In addition start up processes cause higher than normal loads. Please exercise patience during this maintenance window.
--
We have started an out of schedule full backup which will impact DISK IO - during this time you will see higher than normal loads. Please exercise patience during this maintenance window.
Affecting Server - cpanel43.onlinemountain.com
Apache is not starting on this server due to multiple configuration issues from a failed cpanel update.
We have escalated this to cpanel support and they are currently investigating.
The issue has been resolved and sites should now be loading correctly with loads on the server back to normal.
Affecting Server - cpanel42.onlinemountain.com
This server is under high loads and we are working to resolve.
This server is under high loads and we are working to resolve.
Affecting Server - cpanel41.onlinemountain.com
This server requires a reboot - now in progress....
Affecting Server - cpanel41.onlinemountain.com
MySQL server cannot strat due to the mysql data drive mounted as read only. We are investigating the cause and looking for a resolution.
We were able to get the drive back online and MySQL is now operating normally with no errors.
--
Reopeing this issue as some users are experiencing authentication errors.
We are working with cpanel to resolve.
Affecting Server - cpanel42.onlinemountain.com
Rebuild of apache has failed and we have escalated the issue to cpanel support
Affecting Server - cpanel41.onlinemountain.com
We are seeing extremely high loads on this server and are investigating the cause.
Affecting Server - cpanel42.onlinemountain.com
This server is experiencing high loads and we are currently investigating possible causes.
UPDATE
-we have found some IP addresses that are attacking some wordpress sites on the server and are mitigating the effects.
UPDATE
-this server is once again under high loads.
UPDATE
-we have found the issue - a huge ip.pag file from the DDOS attack previously reported was causing high disk IO in apache which was causing a cascading effect and failure.
-should be ok now.
Our security scans have revealed that this server is root compromised. As such, we have started provisioning a new server (with a newer OS) and will:
1. Backup your cpanel account
2. Take the current suitland.webnethost.net server offline.
--during this time, your site and email will be offline.
3. Once the new server is online, restore your cpanel account to the new server. Once your account is restored, your website and email will be online once again.
4. There will be no IP change to your domain.
This process may take 1-3 hours with intermittent connectivity to your site and email. This is an urgent matter and must be addressed immediately.
Please check back for updates.
---
Apr 24 - 17:36 EST (GMT -5:00)
The new server is set up and the backups are complete. I will be taking the compromised server down shortly and putting the new server in it's place. I will then run scripts on the new server that will install your services. After the post-install scripts are finished I will begin restoring your accounts to their proper IP address.
I will update you on the progress as best I can. Please be patient, however, as this will take some time.
---
Apr 25 - 10:00 EST (GMT -5:00)
Account restores are complete - cpanel account passwords have all been reset. We will advise your new password shortly. Some MIVA Merchant sites on this server are not functional - we are working with MIVA support to get the miva components re-installed.
---
Apr 25 - 13:05 EST (GMT -5:00)
Please note that all cpanel passwords were reset to protect the integrity of your account. Please login to your client area at https://www.entirelydigital.com/manage and then click on your services icon, then drill in to the account that is on this server. From there you will be able to reset your cpanel password from the left menu.
The AutoSSL feature of this server has failed and certificates which expired today have not been renewed. We are investigating the cause and hope to have a resolution shortly.
This issue has been resolved and the SSL certificates are queued at Comodo for installation.
potomac.webnetservers.com is displaying some issues with I/O. We require a maintenance window with the server offline for 1-4 hours to clone the primary drive.
This server has become unresponsive and the issue is being actively investigated.
--
This isssue is now resolved.
Affecting Server - cpanel43.onlinemountain.com
Because of a spamming account, this server's SMTP queue is being cleared of spam messages
There are several hundred thousand messages to clear.
Once clear we will divert outbound mail via an alternate IP.
Affecting Server - cpanel42.onlinemountain.com
Because of a spamming account, this server's SMTP queue is being cleared of spam messages
There are several hundred thousand messages to clear.
Once clear we will divert outbound mail via an alternate IP.
A hardware maintenance window is required for this server in the times shown
All services on this server will be unavailable during the maintenance window as the server will be powered off
Actual time will be approximately 1 hour.
A hardware maintenance window is required for this server in the times shown
Dec 1, 01:00-05:00 EST (GMT -5:00)
All services on this server will be unavailable during the maintenance window as the server will be powered off
Actual time will be approximately 1 hour.
Affecting Server - ED-DNP01
Our legacy IIS 6 server is being moved to a new host for improved stability. During this move, websites on this server will be inaccessible.
Move time is estimated at 15minutes.
Migration complete
We are investigating extremely high loads on this server.
Affecting Server - cpanel41.onlinemountain.com
System kernel update in progress + reboot of server
--
FSCK in progress forced - startup delay.
---
Reboot complete.
Affecting Server - cpanel43.onlinemountain.com
The high load on this server is being investigated. While our investigation proceeds, port 80 has been blocked to allow our technicians the ability to work in a timely manner. All other ports remain open (POP, IMAP, FTP, SMTP, Cpanel)
---
Apache / PHP rebuild in progress.
---
Build complete - we are now doing a kernel update.
---
System being rebooted now - new kernel installed and updates complete.
---
We've found the root cause of this issue and have resolved it. Loads are now normal and the server is responsive. We are monitoring it continuously.
Currently, all email is queued and not being delivered to individual user mailboxes. This is an issue with a failed cpanel update.
We are investigating and hoping to have a resolution shortly.
ALL DELIVERY ISSUES HAVE NOW BEEN RESOLVED.
Affecting System - Hosted Microsoft Exchange
We have scheduled some mailbox server updates - Microsoft Hosted Exchange 2010
These are general service and stability updates for Exchange and Windows Server as well as funcitionality updates for OWA (Outlook Web Access)
Updates started
1st reboot in progress....
Reboot complete.
Starting second set of updates
2nd reboot in progress....
Exchange updates in progres - your mailbox will be unavailalbe until completion...
Mailboxes now available
Further updates may make your mailbox unavailable for a few minutes....
Last set of updates require a reboot to complete. Reboot in progress....
UPDATES COMPLETE
Affecting Server - VIPER-HELM-SH
We are currently experiencing login issues with http://cp.neiko.net - this is a licensing problem with the control panel server which we are working to resolve.
Affecting System - cPanel Security Team - CVE-2016-3714 ImageMagick
cPanel Security Team - CVE-2016-3714 ImageMagick
Background Information
On Tuesday, May 3 2016, ImageMagick announced a vulnerability in all versions of the ImageMagick software. ImageMagick is a software
package commonly used by web services to process images.
Impact
One of the reported vulnerabilities can potentially be exploited for remote code execution (RCE).
Releases
ImageMagick has not released a fix, but plans to publish a new version of ImageMagic with the fixes soon. cPanel normally releases all builds at once in order to limit the ability to reverse engineer fixes. However, this vulnerability is already wildly known and we have seen reports of its use. In this instance, we plan to release builds as soon as they become available.
At this time the following builds are available:
11.56 11.56.0.13
EDGE 11.55.9999.193
CURRENT 11.56.0.13
RELEASE 11.56.0.13
We are currently updating all our cpanel servers to this build. If you cannot access cpanel or WHM then a cpanel update is in progress. Please be patient as the update completes.
Affecting Server - cpanel43.onlinemountain.com
cpanel43.onlinemountain.com is doing a cpanel udpate to 11.56 - as such cpanel and WHM are currently unavailable.
Normal operations should resume shortly.
Affecting Server - cpanel43.onlinemountain.com
We are investigating an issue with this server - updates to follow in this Network Issue notification
--UPDATE--
We see an issue with the installed kernel - we are reverting to the previous kernel and rebooting
Investigation will continue.
--UPDATE--
Server is operational - we will be rebooting the server to a newer kernel later tonight.
--UPDATE--
Kernel has been updated correctly and verified - we are rebooting the server so that dependencies with CloudLinux function correctly.
--REBOOT COMPLETE
--UPDATE--
We are once again noticing high loads on this server - invesigtating the issue - we will cycle through services and block specific ports to track down the culprit.
--UPDATE--
We are now running further updates to kernel and other drivers/cloudlinux in an attempt to narrow down the possible issue.
--UPDATE--
In order to expedite this process, we've had to block port 80 on this server as the update process was taking too long. We have restarted the update process with port 80 blocked (low load) and hope to reboot the server shortly.
--UPDATE--
we have resolved the issue and loads are now normal.
Affecting Server - cpanel41.onlinemountain.com
A kernel dependancy requires that we reboot the server for maximum stability.
Reboot has begun and server should be back online in 5-10 min.
--REBOOT COMPLETE
Affecting System - SSL Certificates - SHA1 replacement
We are currently replacing all onlinemountain.com COMODO SSL certificates with SHA2 encrypted certificates.
Prior to replacement you will get a warning in Google Chrome.
Thank you for your patience
Affecting System - Core Network
We will be updating some core switches. During the upgrades there may be some downtime you experience - Approximately 5 minutes while switchports come online.
-This maintanance windows has been postponed. A new item will be opened once the maintenance window has been determined.
Affecting System - Toronto DC
We will be updating some core switches. During the upgrades there may be some downtime you experience - Approximately 5 minutes while switchports come online.
-This maintanance windows has been postponed. A new item will be opened once the maintenance window has been determined.
Affecting Server - cpanel42.onlinemountain.com
Most websites throwing 500 errors due to a Cloudlinux CageFS issue.
We have escalted this to our Level3 support and hope to have a resolution shortly.
Affecting System - All 208.92.222.x IP network
Due to a DDOS in progress all 208.92.222.x IP network is down.
We are working with our downstream provider to resolve the issue.
Affecting Server - cpanel41.onlinemountain.com
This server was under very heavy load - a full restart is in progress.
- file system quotas check in progress (fsck) - this must complete before the server comes back online.
Affecting Server - cpanel41.onlinemountain.com
We are aware of the issue regarding this server.
POP3 and IMAP connections may fail with an authenticator error.
We are actively investigating this issue and should have a resolution shortly.
Affecting Server - cpanel42.onlinemountain.com
We are awere of the issue with this server and are actively investigating.
--issue is now resolved.
Affecting Server - cpanel41.onlinemountain.com
We are currently doing a hardware upgrade on this server. During the upgrade no services are available.
Affecting Server - cpanel41.onlinemountain.com
During updates websites will be unavailable.
Affecting System - Mail server - WEBM1 - TOR DC 1
We are seeing intermittent webmail issues with this server (https://webm1.entirelydigital.com) and are working to resolve.
You may see continuing intermittent outages while we restart services and update core files.
Affecting Server - cpanel41.onlinemountain.com
This server is refusing to authenticate POP, IMAP and WEBMAIL connections (via IMAP) once more. We've escalated this to cPanel.net for immediate investigation and permanent resolution.
---
This is now resolved but we are investigating the cause.
Affecting Server - cpanel41.onlinemountain.com
We are updating apache on this server - websites will be down for approximately 10-15 min.
---
ETA changed - additional updates required for security reasons.
---
Now complete - websites available.
Affecting Server - cpanel41.onlinemountain.com
This server is experiencing IMAP/POP/SMTP Authentication Failure for most accounts.
We are actively investigating this issue and hope to have a resolution shortly.
Affecting Server - cpanel42.onlinemountain.com
Server is currently undergoing maintenance - some hosting services will be affected during our upgrades
Affecting Server - cpanel43.onlinemountain.com
Server is currently undergoing maintenance - some hosting services will be affected during our upgrades
Affecting Server - cpanel41.onlinemountain.com
This server experienced a system failure during updates and we had to gracefully restart it. Unfortunately a forced filesystem check is in progress and must complete.
Hopefully won't be much longer.
--
We are trying to deterimine what processes are causing high load - during our troubleshooting process, different services will become unavailable.
--
Currently httpd is turned off while we update cloudlinux cageFS - email and other services are up - this is required to ensure that cloudlinux LVE is working properly before we start httpd
--
Kernel updates are complete as well as updates to CloudLinux LVE - we're rebooting the server to load the new kernel
We'll be watching the server carefully over the next hours to ensure the loads stay in check.
--
Server being monitored and processes are being kept in check
--
In cooperation with cpanel.net support we are close to a resolution regarding the performance of this server. We appreciate your patience.
--
We've identified this is a DDOS attack and are taking steps to attempt to mitigate the attack
--
The DDOS seems to be targeted to specific customers - we are working to mitigate the attack however DDOS can be difficult to stop.
--
Our cloud backup service is being updated to the latest build - during the upgrade offsite backup and online portal access is unavailable
Affecting Server - cpanel43.onlinemountain.com
PHP is being rebuilt (native 5.4) to include MBSTRING as it was not included in the original build.
Affecting System - Hosted MS Exchange
Clients are receiving invalid SSL certificate warnings in webmail and outlook - we are working with Godaddy SSL to re-issue the SSL certificate in question.
UPDATE - Certificate has been rekeyed and installed - there are no furhter issues with webmail
Affecting Server - cpanel42.onlinemountain.com
PHP and Apache being updated to the latest build
Affecting Server - cpanel41.onlinemountain.com
PHP and Apache being updated to the latest build
Affecting Server - cpanel43.onlinemountain.com
PHP and Apache being updated to the latest build
Update of Cloud Backup Server in progress.
Disk Check in progress
---
Disk check is still in progress but almost complete - should be operational shortly.
Affecting Server - cpanel41.onlinemountain.com
To address possible security vulnerabilities in php - we are updating PHP to 5.4.31 outside the normal maintenance window
This is only a MID-LEVEL security issue.
This server had a failed cpanel update causing some issues - we are re-running it now to stabilize the server.
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.
Affecting Server - ED-SH-CP6
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.
-- update --
We are now upgrading MySQL to 5.5 for compatiblity and upgrade paths to WHM 11.44
-- update --
We are now upgrading WHM to 11.44
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
PHP is being updated to the newest minor version - no major version upgrades are being installed.
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
An apache/PHP rebuild is required and in progress - for breif periods during the build you website may be unavailable.
This server is currently down - we are actively investigating the cause.
---
Server is booting howerver a mandatory forced FSCK (disk check) is now in progress. Usually takes approx 20 min
We have had to restart this server and it is currently performing a quota operation mandatory in OpenVZ. Once this operation completes, the server should come back online.
ETA 20 min
We have resolved the issue causing this listing and have requested delist from spamcop.net.
A hacked mail account has caused the mail queue to be flooded with spam. We have identified and stopped the spammer and clearing the queue - at this time the server is under high load cleaning he affected queue.
We are investigating a continuous load issue with this server - in this process we are shutting down mysql for approx 20min to see if the processes are to heavy for the server.
We have had to restart this server because of load issue investigations. Because of the restart, a mandatory disk check is now in progress. We realize the timing on the forced disk check is during normal business hours however it is better to allow the disk check to complete at this time. ETA 30 min.
We are investigating a server issue affecting multiple servers including this one. We hope to have an update soon.
In order to update cpanel/WHM to the latest version, MySQL 5.5 is required. MySQL 5.0 is deprecated and no longer supported. As such, in order to bring this server in line with all others in our infrastructure we are in the process of updating MySQL. During this time, websites requiring mysql queries may not be available.
Due to some errors we are force to recompile Apache and PHP to the latest versions (2.2.x and 5.4.x respectively) - during this process your websites may become unavailable.
This server is experiencing slowdowns due to the mail queue being full of spam. We have identified the offending sender and disabled their account. We are clearing the queue and will take steps to mitigate this behaviour in the future.
With a successful MySQL update completed, cPanel is being updated to the lastest RELEASE version 11.42.1.x
MySQL is being upgraded to 5.5 - during the upgrade process, websites which use databases will be unavailable
PHP/Apache is being recompiled to the latest versions
This server is being updated - as such some sites are not responding as apache and PHP are being updated.
We are currently upgrading MySQL from 5.0.x to 5.5.x
We are currently upgrading WHM/cPanel from 11.38.x (EOL) to 11.40.11 - current release
You may see some interruption in web services while the system updates it's packages.
Affecting System - Backup Swich - TOR1 DC1
One of our backup switches used for backend communcations is down. This will affect communiction between websites and databases and backup operations.
We are investigating the issue and hope to have a resolution shortly.
We have now started upgrading / recompiling apache with PHP 5.4 on this server.
Should be done shortly.
Build complete - PHP is now 5.4.24
Affecting System - Hosted MS Exchange 2007
Customers of our hosted MS Exchange have noticed intermittent connectivity this morning - We are working to resolve this and have initiated a server restart.
--UPDATE--
An integrity check and fix is required on one of the information stores.
This necessitates bringing the store offline during the check.
During this time some mailboxes may be unavailable depending on which store your mailbox resides.
We appreciate your patience during this check/fix operation.
--
--UPDATE--
Please note that is is a long procedure. We estimate the time to completion at 14:30
Messages will be queued on other servers until the information store becomes available
--
--UPDATE--
Test revealed further errors which are now being fixed through another procedure. We esitmate another 4 hours before completion.
Our apologies for the inconvenience
--
--UPDATE--
Integrity check is taking much longer than anticipated and we may have to restore from backup as we cannot continue to keep the server offline.
We will work to also restore the data from the restore point until this morning but keep customer up to date via this post.
--
--UPDATE / SUNDAY MORNING--
We have added some additional disk arrays and will be doing a restore from backup shortly. Unfortunately our DB fix did not complete successfully and so we must restore from backup. We will work to recover data manually from the corrupt DB after the server is back online.
Again, we thank you for your patience.
--
--UPDATE--
Restore is in progress and completion is estimated in approx 3 hours
--
We are investigating an issue whereby all IMAP and POP accounts cannot login to retrieve mail
This server is currently under high load and may not respond to web/email requests.
-Update
We have found an excessive amount of messages in the mail queue and are investigating.
This server is currently under high load and may not respond to web/email requests.
Affecting Server - VIPER-HELM-SH
Currently our MI email server webm1-MI.entirelydigital.com is at 100%CPU - likely due to a DDOS attack - we are currently investigating.
We have had to restart this server and force a mandatory disk check to ensure integrity. During this time, the server and it's services will be unavailable.
We anticipate downtime of approx 1 hour.
We are recompling apache to address some web server issues. It should be up shortly.
WHM/Cpanel host cpanel20.onlinemountain.com is currently not repsonding. We have our NOC technician looking at the server and we are investigating the issue.
We hope to have it up and running shortly.
--UPDATE--
Server is undergoing a forced fsck (disk check) which we cannot stop. This is a mandatory check after a hard reset. We will investigate the cause of the issue to ensure that the resource drain does not re-occur.
--UPDATE--
--UPDATE--
Server is running - we are now performing system checks before bringing WHM/cpanel back online
--UPDATE--
--UPDATE--
Server has finished checks and WHM/cPanel is now initializing
--UPDATE--
Affecting System - cPanel Webmail
There is an issue with webmail on all cPanel servers. cPanel 11.40 does not allow you to set custom start and stop date/times for your autoresponder. No calendar drop down appears when you click on the date. This is a known issue and will be fixed with the next cPanel update.
ed-sh-cp1.entirelydigital.com is currently undergoing a file system check. Once complete the server will be back onine.
Affecting System - Exchange 2007
We are experiencing an outage on our Exchange 2007 server - we are working as quickly as possible to resolve the issue. Mailbox access, webmail access is affected.
UPDATE - All hardware upgraded and issue is now resolved
Affecting System - webmail MI webmail.neiko.net
We are upgrading all our SmarterMail Servers to enterprise 11.6.x
Affecting System - UPS Battery
EntirelyDigital will be replacing a single UPS Battery at the Toronto data center location on Tuesday May 14, 2013 from 16:01hrs to 17:30hrs. There is no expected service impact during this maintenance
Due to an internal spammer/compromised account, 208.92.222.102 has been blacklisted.
We are investigating the issue and will request immediate delist once the account has been disabled.
Affecting System - cas1.entirelydigital.com/smtp75.entirelydigital.com
We are experiencing very high loads on this server and are investigating the cause. SMTP/POP and IMAP connections will time out and server will appear unresponsive.
ED-SH-CP1.entirelydigital.com experienced a drive failure a short time ago. We are investigating and hope to have the server up and running shortly.
Affecting System - Telephone Technical Support
We are currently experiencing issues with our telephone service. As you may or may not know, our phone servers are located in our US office. Last night a severe storm knocked out power, Internet, and essential services to almost a million homes in Northern Virginia. Luckily our US staff was unharmed by the storm. As of right now, the Gov. Bob McDonnell has declared a state of emergency for Virginia.
Our vendors providing the DIDs and 'dial tone' for all of our phone lines have been forced to shut down all power, including UPS, so emergency crews can work to repair damaged power lines safely.
As inconvenient and unfortunate as this is, we are doing everything we can to assist/comply with emergency crews and services in hopes of having this resolved soon.
This telephony outage has no impact on helpdesk support services we provide.
We do not have a firm ETA however we will keep you updated as soon as we get additional information from the upstream providers and/or Virginia Dominion Power.
You can reach our support staff as usual by emailing helpdesk@entirelydigital.com
We thank you for your patience and understanding. Our thoughts are with those who lost loved ones in last nights storm.
Affecting Other - Network Outage - USA Datacenter
We are experiencing a network outage in our USA datacenter - this is not a server issue but an issue with a downstream provider
We are currently invesitgating and this issue is priority 1 - Critical
-- update this issue is now resolved and we are actively monitoring the hardware in question
Affecting System - Michigan Data Center - some servers
We are experiencing a major outage in our Michigan data center and are actively investigating.
Please do not email our helpdesk regarding your site or email being down as we are already aware of the issue and it is a network issue not a server issue or an issue with your particular domain.
Affecting Other - US Datacenter
We are experiencing a system wide datacenter outage in our USA datacenter.
An investigation is currently in progress. We are aware of the issue so please do not call our support line or email our helpdesk as your website/service/email will come back on line automatically once the issue is resolved.
We are currently investigating why this server (208.92.222.102) is listed at senderbase.org. This may affect you sending email to certain domains that use Cisco firewalls as their antispam solution.
Affecting System - CPanel Servers
We will be scheduling an important kernel security, bug fixes and enhancement update for all our CentOS 5.x servers this weekend.
This would take place as far as possible during the off-peak hours. A reboot is required to complete the upgrade. The downtime should not exceed 30 minutes and it will be minimize as much as possible. We will track each server till it returns to service after the upgrade and reboot.
Details of Kernel security, bug fixes and enhancement update
* A flaw in the xfrm6_tunnel_rcv() function in the Linux kernel's IPv6 implementation could lead to a use-after-free or double free flaw in tunnel6_rcv(). A remote attacker could use this flaw to send
specially-crafted packets to a target system that is using IPv6 and also has the xfrm6_tunnel kernel module loaded, causing it to crash. (CVE-2012-1583, Important)
If you do not run applications that use xfrm6_tunnel, you can prevent the xfrm6_tunnel module from being loaded by creating (as the root user) a "/etc/modprobe.d/xfrm6_tunnel.conf" file, and adding the following line to it:
blacklist xfrm6_tunnel
This way, the xfrm6_tunnel module cannot be loaded accidentally. A reboot is not necessary for this change to take effect.
This update also fixes various bugs and adds an enhancement.
We are in the process of migrating the Hsphere control panel server. During this move, access to account functions will not be available. Web and email services are unaffected.
Affecting System - Michigan IDC - All Services
We are currently performing some unscheduled maintenance on our Michigan SAN infrastructure. We estimate service will resume at approx 4am EST
Affecting System - some email servers
One of our email servers is currently not accepting requests for POP/IMAP - service will be restored shortly.
CP3.shieldhost.com is currently showing an error. This affects webmail and control panel access. We are working with the control panel vendor to resolve this.
Affecting Server - NET-209
Clients trying to access POP and IMAP mail will not be able to login and will recieve an error in their mail clients as well as webmail.
We are aware of the issue and working toward a resolution.
Affecting System - Hosted Exchange 2007 / BES - Blackberry Enterprise Server
Blackberry's (RIM) outage has spread to Canada and affects all BES users on hosted MS exchange. This is not an issue with EntirelyDigital's networks but stems from RIM itself.
There is no ETA for resolution.
iPhone, Android, Windows Phone users are unaffected by this outage.
Affecting Other - Hosted Exchange 2007
Dear Microsoft Exchange User, (french message follows)
On Sunday March 27th, 2011 at approximately 22:00 (10pm) EST (GMT -5.00) we will be doing some software and hardware upgrades on our Microsoft Exchange infrastructure.
During this time your mailbox will be unavailable. The duration of this maintenance window is estimated to be approximately 1 hour.
With these upgrades we will be increasing our capacity which means larger mailbox quotas coming soon.
This upgrade is part of many infrastructure upgrades we will be doing in 2011.
Regards,
EntirelyDigital NOC Team
---
Cher utilisateur de Microsoft Exchange,
Le 27 Mars 2011, à environ 22h00 EST (GMT -5.00), nous allons faire quelques mises à jour logicielles et matérielles de notre infrastructure Microsoft Exchange.
Pendant ce temps, votre boîte aux lettres ne seront pas disponibles. La durée de cette fenêtre de maintenance est estimée à environ 1 heure.
Avec ces améliorations, nous augmenterons notre capacité ce qui signifie que les quotas de boîtes aux lettres plus à venir.
Cette mise à jour fait partie de nombreuses améliorations à l'infrastructure que nous allons faire en 2011.
Cordialement,
EntirelyDigital équipe CNP
Affecting Other - Michigan IDC
EntirelyDigital is scheduling a service window from Tuesday 1/18/2011 11:00PM
to Wednesday 1/19/2011 2:00AM to perform upgrades on key network equipment in our Michigan Data Centre. We
do not expect any network outages durring this service window. Customers may notice
a temporary changes in routing.
Level (3) connectivity will temporarily shutdown durring the service widow. All
customer traffic will route through other peers during this time.
This scheduled maintenance is part of EntirelyDigital's constant commitment
to providing customers with the very best in Internet service.
Affecting System - Exchange 2007
We are moving some mailboxes to a new storage array and as such they may be unavailable for approx 15 minutes.
Your connection will become available once the operation is complete.
Affecting Other - Postoffice.entirelydigital.com
We had a backup run longer than expected and have cut it short this morning. As a preventative measure we have restarted the server. All exchange services will be back online in approximately 5 minutes. All messages queued at the Edge servers will be delivered as soon as it comes online.
Sorry for any inconvenience.
NOC Team
Affecting Other - Updates
All mail servers in the EntirelyDigital network will be updated to their latest build starting 22:00 EST. Minor interruptions in mail flow and mailbox access expected during the upgrades.
Affecting System - Hosted Exchange Customers - All
We are experiencing hosted Exchange Connectivity issues.
Currently all messages are flowing correctly, however we will be doing a full database consistency check at approximately 10:00PM EST
Affecting System - Email - Toronto Data Centre
Email services were temporarily down between 16:00 EST and 16:30 EST today. We have resolved the issue and the server is operating normally.
We are experiencing connectivity issues with some of our core routers in our Michigan Data Centre.
We are investigating the proplem and hope to have a resolution shortly.
Affecting Server - VIPER-DNP01-SH
We are experiencing an outage with some of our customers websites hosted through the Michigan data centre.
Our NOC team is aware of the issue and is actively working on the problem.
In some cases once you logon to your webmail client, you may not see your messages or you may only see a partial list.
If this is the case your webmail client needs to be reset and messages re-indexed by the application. Should this be the case, please submit a helpdesk ticket and logout of your webmail. We will re-initialize your client and advise once you can logon again.
We will be upgrading our backup server to the latest build 5.5.7. All installers will also be updated and your autoupdate utility should update your backup software to this version. You may receive old failure notifications as we purge some of our logs.
Affecting System - Hosted Exchange 2007
We will be performing updates to our hosted exchange infrastrucutre. Mailbox access will be temporarily unavailable at various points during the update.
Mail will be queued at the edge transport level so no messages will be lost.
Affecting Server - VIPER-DNP01-SH
We will be updating the control panel software which connects/configures all Viperhost services over the weekend. You may experience periods where your website / email may not be operational for brief periods (less than 5 minutes at a time) while we restart services.
Affecting System - Hosted Exchange
Between April 23rd 9:00pm EST and April 25th 9:00pm EST we will be performing maitenance on our Hosted Exchange infrastructure. You may experience connectivity issues during the weekend only for a few minutes while we restart services.
Affecting System - Viperhost Michigan Data Centre
EntirelyDigital experienced a failure of a core router in Southfield. Some of
the fail-over mechanisms did not work, requiring manual innervation. This resulted
in an interruption to some customers in the Viperhost Michigan data center. Currently service
has been restored and was unavailable for approximately 9 minutes
Affecting System - Michigan Data Centre
Affecting:
Customers in Viperhost Michigan Data Center
Details:
EntirelyDigital Viperhost is scheduling a service window from 7:00PM-10:00PM Friday, April
9th, 2010, to perform upgrades on key network equipment.
Level (3) connectivity will temporarily shutdown for a period of not more then
approximately 15 minutes. All customer traffic will route through other peers during
this time. As such no downtime is expected