History Timeline

    20 June 2022

  • No activity on this day

    19 June 2022

  • No activity on this day

    18 June 2022

  • No activity on this day

    17 June 2022

  • No activity on this day

    16 June 2022

  • Description
    We are currently experiencing issues with calls, particularly calls failing to terminate.

    This is causing an increased load and an inability to make/receive calls.

    We are currently working to identify and resolve this issue.


    Resolved on: 16/06/2022 17:05


    ---

    Update on 17/06/2022 10:05
    RFO 16/06/2022

    We became aware of a number of issues affecting us and many other providers too. This had an impact on our service and therefore we took emergency action to mitigate the ongoing issues.

    We engaged our Disaster Recovery system and rerouted in excess of 8500 inbound calls each with a duration greater than 10 seconds.

    To alleviate the issues we disabled our Hosted VoIP Platform and BLF notifications temporarily, which enabled outbound and internal calls to be made.

    Once we were confident the issues were resolved we fully restored our services.

    An example of a couple of issues from suppliers:-
    ---
    DWS:- Dear partner, Hi To advise many services were affected today by an issue with a peering company lumin/lvl3. This is a peering partner who deals with hand-off traffic to the wider net for many suppliers. Between 13:15 and 13:45 there was a period of no traffic passing over their network. This was resolved by passing peering traffic to a different peer partner. This is not a DWS specific issue and likely affected multiple suppliers within the UK.

    June 16: Problems at BT (British Telecom) BT (British Telecom) is having issues since 09:00 AM GMT. Are you also affected? Leave a message in the comments section! We apologise for any inconvenience caused.
    ---

    ---

    Update on 16/06/2022 15:50
    SMS are now sending out as normal.

    Any sent to us during the outage would have been received and Queued. They are currently being processed now.

    ---

    Update on 16/06/2022 15:20
    Inbound calls have now been re-established, some services such as the portal and BLF's are still disabled as we continue to monitor the issue.

    We will continue to provide updates as these services are re-established.

    As per our previous updates, many services were affected today by an issue with a peering company lumin/lvl3.

    This is a peering partner who deals with hand-off traffic to the wider net for many suppliers.

    ---

    Update on 16/06/2022 14:25
    We are continuing to monitor this situation but we believe most services to be restored now.

    For Hosted VoIP - Outbound and Internal calls have been restored but inbound calls are still routing via the DR system (Disaster Recovery) to help reduce server load whilst connections restore back to the platform.

    We have had confirmation that a major "Tier 1" internet hub has failed, which is causing many internet services providers to have a knock on affect, our suppliers included.

    If you are experiencing any difficulties, please reach out to our service desk, via the usual channels.

    ---

    Update on 16/06/2022 13:45
    We have received an update regarding an ISP fault which appears to be the root cause of the problem affecting Hosted VoIP, Hosted Desktop and potentially some other services which are yet to be identified.

    The ISP provides internet access to the data centres, therefore with them having an issue, it's affecting multiple services for multiple providers and not just ourselves.

    ---
    One of our major internet providers is currently experiencing technical issues. Consequently, some customers are having difficulty connecting to our services, or cannot load web pages/access their email.

    We're currently working with customers experiencing this issue, as well as working with our comms partner for updates.

    We'll post an update shortly.
    ---

    ---

    Update on 16/06/2022 13:35
    We have identified a further issue with some Hosted Desktop connections.

    We believe there is a data center issue somewhere that is causing a further outage with more services.

    We're investigating to identify the root of these issues.

    ---

    Update on 16/06/2022 13:20
    We are continuing to slowly restore services and keeping the load on our servers low in order to ensure a stable recovery.

    You may notice some service restoring slowly, with the exception of the portal.

    ---

    Update on 16/06/2022 12:35
    We are currently investigating this incident.

    You may experience calls intermittently working as we try to bring our service back.

    ---

    Update on 16/06/2022 11:35
    We have disabled Hosted VoIP in our control panel temporarily to reduce the load on our servers to enable calls to be re-established. Please note we went into full disaster recovery mode 25 minutes ago.

    SMS will also be affected at this time, but you can still send via email and they will all Queue ready for when the system is back online.

    We hope to have this issue resolved shortly.