disarray on Nostr: Some rambling regarding the outage on November 5th: - NOC started noticing issues at ...
Some rambling regarding the outage on November 5th:
- NOC started noticing issues at 10:01 AM CST
- NOC immediately dispatched techs to start looking into this by taking OTDR shots at multiple locations.
- First fiber cut was 15 miles away in Oak Grove IL, 15 miles south and west of the data center. (Originally believed to be in Davenport, IA, but even further!)
- Their second redundant link was also cut about 20 minutes later in Clinton IA, 35 miles north east from the data center.
- These are both redundant fiber optic lines and both are baried underground.
- While it was storming that day and I don't know what exactly caused the Clinton, IA fiber cut due to that portion being owned by a 3rd party according to my ISP (I'm guessing they're leasing dark fiber in some areas) the Oak Grove, IL damage seems to be someone installing a sign post in the ground with a cement base without calling 811 to see if they can dig in in that area!
- While they sent in an emergency construction crew to both sites as fast as possible, repairs couldn't start until they got approval for an emergency dig order from 811 at 3:30 PM CST.
- My upstream is looking into adding a third fiber route to the data center.
Honestly I can't blame my upstream for this. They did everything right and this was an extremely unlikely situation to ever happen.
- NOC started noticing issues at 10:01 AM CST
- NOC immediately dispatched techs to start looking into this by taking OTDR shots at multiple locations.
- First fiber cut was 15 miles away in Oak Grove IL, 15 miles south and west of the data center. (Originally believed to be in Davenport, IA, but even further!)
- Their second redundant link was also cut about 20 minutes later in Clinton IA, 35 miles north east from the data center.
- These are both redundant fiber optic lines and both are baried underground.
- While it was storming that day and I don't know what exactly caused the Clinton, IA fiber cut due to that portion being owned by a 3rd party according to my ISP (I'm guessing they're leasing dark fiber in some areas) the Oak Grove, IL damage seems to be someone installing a sign post in the ground with a cement base without calling 811 to see if they can dig in in that area!
- While they sent in an emergency construction crew to both sites as fast as possible, repairs couldn't start until they got approval for an emergency dig order from 811 at 3:30 PM CST.
- My upstream is looking into adding a third fiber route to the data center.
Honestly I can't blame my upstream for this. They did everything right and this was an extremely unlikely situation to ever happen.