We were alerted that some of our services were down on Saturday 30th August 2025 just before lunch. Upon investigation, this looked to be limited to a few services supported by the Mucklestone network, however, it transpired that the wider area services like DNS did not fail over to other locations.
The initial cause was a double UPS failure in the rack in Mucklestone. The APC UPS’s perform a self check every day, and upon one doing a check, it somehow caused a circuit breaker to trip. The other UPS on the other circuit saw the trip and spike, and tried to go over to battery, but the battery failed. The dual power supplies of the servers are connected to each of these UPS’s. With both failing at the same time, the power to the server rack completely failed along with all services that these servers provide and support.
We have never seen any APC UPS go faulty before, and not at the same time that the other APC UPS decide it had a failed battery and go off-line too.
We asked a contractor to visit site, and reset the circuit breakers, and this brought the services back on-line, although some of the services may have still been off-line until just after 4pm, due to DNS delays.
We are also looking into why some of the DNS did not fail over to other locations in a timely manner, as this compounded the speed at resuming service.
We apologise for this loss of service.
We have ordered replacement equipment to swap out failed components, and hopefully these will be couriered to us by Tuesday.
