There is an intermittent issue with our Mucklestone network affecting MyServer users both for logging in and also web browsing inside their remote desktops due to intermittent packet loss. We are working closely with our ISP to resolve.
Updates:
31/05/2023
10:05 Reported the issue to our ISP.
11:35 Our ISP has confirmed an issue with the exchange and not our equipment.
11:40 Fault escalated to Openreach for urgent resolution.
14:20 Call from the ISP to say that Openreach has cleared the fault in the cabinet at the exchange. However, they will monitor for the next 24 hours.
01/06/2023
09:20 Users reporting problem returned.
09:25 Re-reported to Internet Service Provider who ran tests again, and same fault codes at exchange.
09:30 Reported ping losses of between 25% and 50%.
10:05 Problem escalated to Openreach as not fixed.
12:18 I think Openreach has rebooted something in the exchange because, although better, we are still seeing packet losses.
13:55 Speaking again with ISP as all Openreach appear to be doing is resetting the equipment at the exchange and closing the case, but, the problem returns a short period later. We need them to look further into the equipment and/or issue.
18:00 According to the ISP, this may impact into tomorrow (Friday 02/06/2023) while ongoing investigations continue…
02/06/2023
09:03 A formal complaint has been lodged with our ISP demanding a time frame to a resolution as this is now outside of our critical care service level agreement with them.
14:02 The ISP has reported that the whole area is affected and an incident management team at Openreach are actively investigating.
05/06/2023
11:50 Requested an update to this ongoing issue with our ISP.
14:30 The ISP is still seeing the whole area still affected and still investigating.
06/06/2023
09:30 There were two short outages whilst we believe (unconfirmed) that Openreach may have swapped some kit over. Service is better and we are continuing to monitor.
07/06/2023
11:15 Confirmation from our ISP that Openreach say they have resolved the issue in the exchange, and cleared the fault monitor. This will allow them to see if the fault on the links return. Our ISP is continuing to monitor for the next 48 hours.
14:28 Spoke too soon, the issue has returned. Re-reported to ISP.
09/06/2023
14:15 This is still on going, and being reported to our ISP on a daily basis. Although OK this morning, the issue still remains each afternoon just after lunch. Awaiting a further update today from Openreach.
14:30 Message via ISP from Openreach…
We have heard back from the escalations team to advise that they are still investigating this issue and have arranged for the NED (network end delivery team) to attend the street and look into this issue. We believe they have found the issue to be one of the CBT junction boxes that your Fibre goes through and they will try to resolve this as quickly as possible. Again they have asked us to check back in on the 12/13th to allow the team time to pick the job up, investigate and hopefully put in place the repair to stop this happening again but rest assured we have escalations in place for this area and the issue should be resolved ASAP.
10/06/2023
10:00 There was a short outage in order to replace the CBT junction box, see the comment from yesterday.
16:24 Since 10am this morning the links have been perfect. No packet losses.
21:11 Spoke too soon, packet losses again.
12/06/2023
10:00 No update yet.
13/06/2023
10:24 Technical message via ISP from Openreach…
We have been informed by Openreach that the engineers have now found that there is a high light level to the splitter but loss soon after so there are indeed 2 problems with the first now being resolved. To get this resolved for you as a priority, I have been able to find an engineer who is free this afternoon between 1 pm and 6 pm to look into this connection and get the issue resolved as soon as possible for both lines hopefully. Again we will keep a close eye on the case for you and hopefully, we are able to provide good news following the visit to confirm the issue is resolved.
11:18 Note: This may mean a short outage in order to swap links over.
14:02 Openreach are taking the link off line for a moment to test.
14:54 Openreach engineer registed a good strong -17dBm light reading from the end of the fibre. So the physical connection is working, so back to our ISP.
15:15 Openreach asked us to reset the ONT in order to apply updated firmware.
15:30 Continuing to monitor…
15/06/2023
12:25 We appear to have had no packet losses now for 24 hours, if we still have none by this time tomorrow, we will class this as resolved. We are continuing to monitor in the meantime.
14:45 Spoke too soon, we have packet losses again. Openreach engineer to visit 16/06/2023 requiring two short outages between 8am and 1pm tomorrow.
16/06/2023
12:00 Despite Openreach visiting this morning, the problem remains, disappointingly.
19/06/2023
10:00 It has been proven beyond a shadow of doubt that the problem appears with the exchange. So, where is the fault? Here is a diagram of the links from you to us.
You > Green box in your road > Your Exchange > Your ISP > {The Internet} …
… {The Internet} > Our ISP > Intermittent Fault > Our Exchange > Us
12:00 We have asked for a timeframe for a resultion of this fault.
19:38 Our service appears to be disconnected.
21:50 Service appears to be reconnected.
20/06/2023
08:57 Requested an update from Openreach (via ISP)
09:36 Our ISP replied with…
We have indeed just heard back from Openreach and our network operations team to advise that over the last 48 hours, they have been replacing equipment and cabling in this exchange and the parent exchange that serves the users in this area.
They have advised that they will be back out again today and tomorrow to try and swap out the optics that provide the service as well and then have the engineers look at each cable that brings the service into the exchange as well to try and resolve this faster whilst Openreach look into their side.
In terms of an ETA they are advising they will be back out today/tomorrow depending on availability as they are proactively looking into this as a wider area in the WN exchange but again we will do everything we can to get this resolved as quickly as possible and hopefully be back in touch asap to confirm they have resolved the fault.
15:40 There was a complete outage between 13:51 and 14:15, we assume in order to change something over upstream of us. However, since 15:15, the intermittent fault with poor network performance has returned.
18:35 In my opinion, whatever Openreach did today may have made the link worse. I will chase our ISP first thing in the morning, again.
21/06/2023
09:39 Update from Openreach (via ISP)…
Yesterday Openreach attended the exchange and cleaned/reseated the equipment on the OR side and found errors still after cleaning. this did improve the network but as you have seen as well, there are still a high number of discards and errors so we have arranged for our engineer to go back out and swap the unit entirely.
This should happen by roughly mid-day depending on traffic getting there and how long it will take to swap the unit out itself.
That being said there will be a small period of downtime today whilst this is moved over but should be resolved ASAP and should resolve the packet loss as well from what the Openreach engineers reported yesterday.
11:22 Outage started.
11:28 Equipment replaced and the link back operational.
11:30 Monitoring for the rest of the day.
16:00 So far, we have not yet had a problem since the equipment change at the exchange.
22/06/2023
10:15 Since the equipment change at the exchange yesterday, we have had no reported issues. We will continue to monitor until tomorrow and hopefully if there are still no more reported issues, we will consider this issue closed.
17:00 Still no reported issues. All looking good.
23/06/2023
09:00 Still no reported issues. All looking good.
