Subject: Pager Duty, v1.1.0
Posted on: 2017-10-17 23:06:00 UTC
(( 1.1.0: Expended on a few elements and tweaked vocabulary per some reviews. Further comments are welcome and appreciated. ))
2017-10-13 06:16:00 [ecosystems3.rack5.qq45ld09β-tl1.routing] tty1: NO CARRIER
2017-10-13 06:16:00 [cisco2.rack2.rr12zh34ε-tl1.routing] eth3: link down
[55385 similar messages elided]
2017-10-13 06:16:15 [ecosystems3.rack5.qq45ld09α-tl1.routing] no route to host vault-tec1.rack1.qa01aa04ξ-tl1.routing
2017-10-13 06:16:19 [cisco2.rack2.rr12zh34ε-tl1.routing] no route to host console04ab9e221e44b490ff.rc112358.dms
[103287 similar messages elided]
2017-10-13 06:16:59 [monitoring13.dosat] Region qa01 probably down - sev2 ticket #23499101 logged
Technician Tomash was startled awake by the console. He groped around for his glasses, put them on, and then took a look at the time.
"It's 6 in the morning, dammit." he grumbled. "What'm I getting woken up for?"
Tomash walked over to his console
2017-10-13 06:20:51 [ingress1.rack6.ra01aa07ν-tl1.routing] nlan1: link down (warp fluctuations)
2017-10-13 06:20:52 [tuberosum55.box2b.בc23ln43λ-tl1.routing] prot1: link down
[298232 similar messages elided]
2017-10-13 06:20:55 [console050102ky332761.rc007.dosat] no route to host monitoring.dosat
2017-10-13 06:21:01 [ingress1.rack6.ra01aa07ν-tl1.routing] no route to host console02.dosat
[408214 similar messages elided]
and started reading the beginning of the logs after glancing at the last few entries, which had come in around 6:19. "OK, zone outage, this'll be annoying. Let's see where that is..."
tomash@rc2718282 ~ 06:23 $ network-map qa01
Fatal error: Cannot look up room layout: no route to host
2017-10-13 06:23:23 [aperture4.control.dosat] External corruption on link fib3, preemptive halt
2017-10-13 06:23:24 [monitoring08.dosat] Kernel panic: Core nyph14tnyd,eucrkaogutns1\2443h4prs^X@
[10 similar messaged elided]
Tomash leaned down to make sure his console's network cable was still plugged in. This was a fortunate decision, as, about 30 seconds after that lookup command had failed, the console's screen and keyboard started glowing green.
Bits bits bits said the thing in the console.
Tomash wasn't quite sure what was going on, but whatever was in his console sounded like an infovore, and he'd gotten the talk on what to do about an actively spreading one a few weeks after he'd transferred in to DoSAT. Namely, kill the affected network and hope it leaves through the plothole it used to get in, and get the Weeds if it didn't. Therefore, he ran outside without bothering to put on shoes first and started heading for the main computing facilities. Headquarters, fortunately, was being cooperative, and he made it there rather quickly.
And so, about a minute later, a heavily breathing technician charged into a room full of routers, servers, and even more unusual equipment. Some of this equipment was glowing, and the fraction of eerily-green equipment was slowly growing. Tomash grabbed a nearby hammer, smashed some glass, pulled open a protective plastic cover, and slammed the big red button by the door.
2017-10-13 06:24:32 [panel.th02ln44μ.routing.dosat] Emergency shutdown logged
The blinking lights on the machines turned off.
The box of potatoes half a mile down the hall to the left stopped vibrating.
Everything was quiet.
But that might have just been the fact that piles of technical stuff are always at least somewhat noisy.
Fortunately, the glowing started receding out of the servers. Hungry hungry hungry it said.
Tomash looked at the diagram on a nearby wall and found out where qa01 should be. He started walking his way down the now emergency-lit hall, using a phone flashlight to check the door signs. Unfortunately, that was a rather theoretical diagram since HQ was back to its usual self. Now that the immediate threat was past, looking for a large bay full of equipment was a terrible strategy for finding it.
After a few increasingly frustrating minutes, Tomash made it to area qa01, where he started listening for eldritch muttering and glowing. He paced around the room, looking around every few steps, and didn't find anything. That was a bit concerning.
"OK, wait, the first thing that showed as down is in aa03 or so, hm let's see..." Tomash started walking over there, and then he looked around more carefully. "OK, so none of this stuff is glowy. Where the heck could that have been coming from?"
During all the staring, a faint blue shimmering from a cable run caught Tomash's eye. He went over to where the room's backup CAD was, and pointed it at the plothole in the cable run.
Natural portal to Laundryverse. Close? Y/n
Tomash hit the button to close the plothole, ending the shimmering. He then left and started heading for the main DoSAT labs.
Once he'd gotten there, he went through the Makes-Things Memorial Blast Doors (which were still named that even though Makes-Things had always been very alive) and called out. "Anyone know how to restart main routing? I got the scary green stuff out, but I can't find the startup docs."
"They're on the wiki!" said a nearby tech. A moment later, she'd remembered what was going on and added "Which is under that desk over there.", pointing towards said desk.
Tomash went over to the wiki server, turned it on, looked up the directions, and started the process of getting a good chunk of the network back up.
After the initial steps were dealt with, the procedure sped up significantly since it turned into something that could easily be split up among many people, and there were a lot of techs waiting around twiddling their thumbs (or other manipulatory appendages, if any) while waiting for the outage to clear up. Most of the work involved going to each part of the shut-down infrastructure, disengaging the emergency stop mechanisms, and turning certain supervisory computers on by hand, as they would restore the rest of the system automatically.
Even with all the help, getting everyone back online took a few hours, as there was a lot of infrastructure to restart. There were also the inevitable issues with bugs, such as the ones that got into the boxes of computational potatoes.
Once everything was mostly back up and running, Tomash went back to bed, yawning frequently along the way, since he really hadn't gotten enough sleep the night before.
He'd just about managed to settle into a nap when
2017-10-13 13:13:13 [aperture4.control.dosat] Emergency stop. In-progress canon integration detected.
2017-10-13 13:13:16 [monitoring01.dosat] Control AI unresponsive - sev3 ticket #23499666 logged