Featured Post

Dissecting the End of the World

I’m sure you’ve all heard by now that, on Saturday, January 13th, Hawaii’s Emergency Alert system sent out an SMS message to everybody that said:

Emergency Alert
BALLISTIC MISSILE THREAD INBOUND TO HAWAII. SEEK IMMEDIATE SHELTER. THIS IS NOT A DRILL.

A half hour later, a message was sent out that this message was a false alarm and there was no missile attack at all. (You can read Michelle Broder Van Dyke’s twitter thread that has pictures and everything here.)

Now, according to the governor, this was caused by an employee pushing the wrong button. Here’s the quotation from the article:

“It was a mistake made during a standard procedure at the change over of a shift, and an employee pushed the wrong button,” he said.

I am very relieved that the end of the world appears to be delayed for a couple more weeks but I have questions about this sort of thing and find it far too easy to question the official narrative.

Here’s what we know: a message saying “missile warning, this is not a drill” got sent out and it took 38 minutes to send out the message that said “whoops, yeah, this is a drill”. When asked about it, the authorities said that this was a mistake that happened when an employee pushed a button.

So let’s look at the possibilities:

1. The message was sent out mistakenly
2. The message was sent out deliberately but the sender knew that there wasn’t a missile
3. The message was sent out deliberately but the sender thought that there was a missile
4. The missile was sent, exploded, and we’re all living in a simulation or dead and in the afterlife or something

So let’s look at the first one:

As buttons go, having a single button that sends out a *VERY* specific message dealing with missile attacks makes me wonder exactly how many messages they have assigned to how many buttons. I mean, if a button got pushed that sent out an email that said “there’s a storm coming and it’s going to be a doozy, get to shelter”, that would make sense to me. Hawaii probably has doozies of a storm often enough that you’d want a message like that to be sendable with a single push of a button.

But a missile attack?

And, on top of that, it’s a button that can be pushed accidentally during a routine shift change? Maybe we’d want a plastic thing over that button that you have to flip up before the button is pressed. Perhaps two buttons (on two different sides of the room) that require being pressed at the same time. Maybe a key that has to be turned before the button is pushable.

I think that we’re lucky that the mistake does not yet appear to have a death toll because I could easily see how something awful could have happened by panicky people responding to a “THIS IS NOT A DRILL” message. Though I suppose that some of the panicky responses could result in a net positive number of lives due to the message… something for an intrepid journalist to investigate 9 months from now.

But if this was a mistake, there needs to be a full accounting, with maps, and descriptions of fixes, and somebody getting fired. This has created one heck of a callus for not only Hawaii but the entire country: the next time that there is an SMS message being sent out saying that there is an attack incoming and THIS IS NOT A DRILL, there will be a huge number of people who just won’t believe it.

So let’s look at the second one:

Off the top of my head, there are a couple of ways that a message could be sent out deliberately with the sender knowing that there wasn’t a missile. The first is that it was a “prank” by a stupid/malicious actor who hacked into the system. The second is that it was a “prank” by a stupid/malicious actor who had legitimate access to the system.

Given that the former exposes a *HUGE* vulnerability, the best response is the government covering up by saying something like “yeah, Joe Blow bumped the button with his elbow during shift change. Oh that Joe!” and figure out, right freaking now, what other systems have these vulnerabilities and patch those holes closed as quickly as possible and start pouring money into computer security.

The latter strikes me as being less likely for the authorities to be willing to cover up (but, I suppose, theoretically possible). If you have a prankster, it’s going to be turned into political hay by somebody. Better to deny the political hay and have the official story be Joe Blow “accidentally” bumped the “button”. Then you fire Joe if he’s firable and you transfer him if he’s not.

And the official story can remain that this was not done deliberately.

The third one is where we *REALLY* get into tin foil hat territory.

Somebody thought that there was a missile. The only proper response is to send out the alert. Maybe it’s because there *WAS* a missile! And our Star Wars program shot it down! But if The People knew that there was an attack, it’d demand a response! And so in order to prevent megadeaths (or even gigadeaths), we say that the SMS message was sent out in error. Whoops. Oh that Joe. Then we go to bed knowing that we saved the world until the next time.

(This strikes me as really, really, really, really unlikely. For one, South Korea and Japan would have known that there was a missile and this would have been all over Korean and Japanese twitter a few minutes before the SMS message went out. Korean and Japanese twitter was not all over this a few minutes before the SMS message went out. Therefore, there was no missile. Q.E.D.)

But maybe they thought there was a missile *MISTAKENLY*. Like they thought that a plane or a drone or a UFO or an undigested bit of beef, a blot of mustard, a crumb of cheese, a fragment of underdone potato. They responded in the only moral way they knew how: they pushed the button and warned their friends and loved ones. And, wouldn’t you know it, it was a bird.

And, at that point, well… there were a number of institutional failures there that need protections. Two-man verification from now on. No sending the SMS unless you’ve got an okay from the supervisor and the supervisor’s manager and the supervisor’s manager’s boss too. And the official story is that the button was pushed by accident.

As for the fourth one, that strikes me as not likely at all but, hey, it’s technically a possibility.

What *REALLY* happened? Well, we don’t know exactly quite yet and probably won’t for at least a few weeks until the official narrative coalesces fully.

As for “why did it take 38 minutes to send out the correction?”, we spoke to friends who had recently taken an extended vacation in Hawaii. They shrugged and said “Island time”. So maybe that’s the explanation there.

All that to say: I’m glad it’s not the end of the world. Yet, anyway.


Contributor
Home Page Twitter 

Jaybird is Birdmojo on Xbox Live and Jaybirdmojo on Playstation's network. He's been playing consoles since the Atari 2600 and it was Zork that taught him how to touch-type. If you've got a song for Wednesday, a commercial for Saturday, a recommendation for Tuesday, an essay for Monday, or, heck, just a handful a questions, fire off an email to AskJaybird-at-gmail.com

Please do be so kind as to share this post.
Share

59 thoughts on “Dissecting the End of the World

  1. It probably wasn’t a button like on some control console somewhere, it was probably on a drop down list in the software they use to manage shift changes and what not, and someone had the wrong screen up, or clicked in the wrong ComboBox, or selected the wrong item in the list, and it’s wasn’t really Joe’s fault, it was whoever designed the software interface.

      Quote  Link

    Report

    • Yeah, you’re right. Here’s a paragraph from the Fox article:

      Rather than triggering a test of the system, it went into actual event mode. He confirmed that to trigger the alert, there is a two-step process involving only one employee — who both triggers the alarm, then also confirms it.

      “There is a screen that says, ‘Are you sure you want to do this?'” Miyagi said. The employee confirmed the alert, inadvertently causing a panic in a state already on edge over saber-rattling missile threats from North Korea.

      So someone had the wrong screen up and clicked in the wrong ComboBox.

      And then did it again.

        Quote  Link

      Report

        • If I knew how much his salary was, it would help.

          If he was making $12/hr? I’d see his pushing the button a second time as pretty much inevitable and would immediately begin wondering why it took so long for this to happen.

          If he’s a salaried employee who had to do stuff like pass a background check to get his job?

          Yeah. I’m willing to be incredulous at that point.

            Quote  Link

          Report

          • Having done end user support for long enough (a decade or so) amongst people, young and old, who were ostensibly the best and brightest, I’m not incredulous at all. A badly designed GUI can readily trip up the smartest kinds of people, and badly designed GUIs are a chronic problem, especially in software marketed to government agencies, or built to spec for them.

            So, I want to see the UI, not the employee stats.

              Quote  Link

            Report

              • This isn’t even the first time. In 2005, an operator error caused an evacuation order for the entire state of Connecticut to go out. I have seen speculation that some of the state systems are running 20+ year old software.

                When I was a legislative budget analyst, it had become unfortunately common to see policy changes voted down because of the cost of putting those changes into the software systems (think tens of millions of dollars). Trump’s infrastructure dollars could possibly produce greater benefit if they were routed to the states for software replacements.

                  Quote  Link

                Report

        • And actually, I’m not kidding except for the QED. As greg notes below, there’s not only no reason (a priori!) to assume the system wasn’t hacked, we’ve been presented with lots of evidence (or reports of lots of evidence…) that foreign state actors are actively trying to hack into exactly these types of US state and federal systems. So the rational explanation given that we don’t know all the facts will be a probability assignment of each account based on what we do know.

            Quote  Link

          Report

          • The two things that make me most suspicious: the 38 minute response time and the somewhat wacky phrasing of the original message.

            As Kohole points out, “this is not a drill” doesn’t sound like official language (I think it sounds like something that a prankster would add). The 38 minute response time makes sense if nobody was at the console and then had to get to it… less sense if someone was sitting at the console and pressed the button himself twice.

              Quote  Link

            Report

            • The 38 minute reponse time makes sense in this scenario –

              Shift change, *saturday morning*. So the ‘regular’ people aren’t at work.

              Someone errs right as they are walking out the door, and nobody catches that they send a real world outgoing alert. (I have no idea how this system works, but there are similar systems that have ‘safe’ or ‘training’ modes where it’s not necessarily clear to certain users if you are in a training mode or not.

              There possibly are multiple locales that can send such an alert (e.g. a terminal at the Tsunami Warning Center, a terminal at Hawaiian Civil Defense HQ, a terminal at one of the military commands on the island).

              So when something went out, it takes a while for the watchcenters to know something went out. They don’t have their personal cellphones on them for security and paying attention reasons. The bosses at home do get an alert, (eventually), they call into their respective people, but it takes a while to determine who sent the alert, and to verify with ‘ground truth’ watchstanders (who only a limited number of people have access too) that indeed, the status boards are all green and nothing is actually hapenning in the real world.

              I could easily see this entire process taking 38 minutes from the time the alert went out, until when the governor has enough info to verify ‘all clear, false alarm’.

              Remember, it was a Saturday.

                Quote  Link

              Report

                • People on the previous shift doing a favor (or following procedures) to queue stuff up for a drill to be run on the next shift.

                  People on previous shift scheduled to run a drill and had stuff queued up but the drill was cancelled because of whatever reason.

                  People futzing around on the previous shift, loading message into the bin for training/demonstration (or for sh**s & giggles) and not clearing the buffer when they were done.

                  In either scenario that stuff could have been activated during the shift turnover when it wasn’t supposed due to not following correct turnover procedures or a flaw in the procedures that allowed this message to go live for real.

                  Further thought. Someone on previous shift showing an FNG on next shift ‘this is how you do this’ and it went too far.

                    Quote  Link

                  Report

                • Re the shift change hypothesis: why would the State run a test of the emergency systems during a shift change? Does that seem plausible?

                  Running one? No. Ending one? Yeah.

                  Say it’s the last test run of a long list (like, every part of the system) because it’s the least probable and the least used.

                  Then there are multiple delays. Joe didn’t know it was “live”. Jim didn’t know where the “kill this” switch even was (an even more unused part of the system), or maybe he even doesn’t have security access to this part of the system.

                  38 minutes is long enough for someone to walk out to his car, turn on the radio, realize what he’d done (panic on live news), and then run back to his desk.

                    Quote  Link

                  Report

      • I sort assumed that “pushed the wrong button” is managerial shorthand for “opened a couple of drop down menus, pushed the wrong buttons or three, and didn’t notice the difference between menu (a) and (III) argle bargle…:” since I doubt there are many systems left that are analog and are physical buttons.

          Quote  Link

        Report

        • “Pushed the wrong button” is something that gets me to say “hey, that could have happened to anybody”.

          “Opened a couple of drop down menus, pushed the wrong buttons or three, and didn’t notice the difference between menu (a) and (III) argle bargle” does not get me to say “hey, that could have happened to anybody” but “what the hell is going with training and procedures over there?”

            Quote  Link

          Report

          • Well, yeah, its a very good question as to who, and how, and under what circumstances a ‘”OMG WE ARE AT WAR !! WOOP WOOP WOOP” message is allowed to be sent;
            Like I mentioned below, I don’t get why any local entity is empowered to unilaterally decide to send this kind of message, even if they sincerely believed it to be real.

            I mean like,even since ancient times the military has an entire command and control system to control who is allowed to alert the troops and assemble them into fighting formation.

              Quote  Link

            Report

          • I’ve made wrong selections on drop-down menus before, but in my case, that meant the wrong statistical test got run, I cursed about it, and went back and did the right one. But if clicking the wrong thing on a drop-down menu would, for example, erase months of data I had spend weeks entering? I’d make DAMN sure of what I was doing before I clicked anything.

            That 38 minute gap is what I find so horrifying. I can’t imaging what would have been going through my mind, had I been in Hawaii. I hope no one died (heart attack, stroke….) as a result of the stress of those 38 minutes.

              Quote  Link

            Report

  2. I did ponder that there may be a spike in births in 9 months.

    There have been a bunch of report that Russian associated hackers have been working at getting access to more than just e-mail accounts.

    https://techcrunch.com/2018/01/12/russian-hackers-senate-pawn-storm-fancy-bear/

    Or course that is just the senate.

    Allthough in this bit, in the middle of the piece:

    “The U.S. is vulnerable in other areas, too. When Attorney General Jeff Sessions testified before the Senate Intelligence Committee in June, Senator John McCain turned his attention to an even more worrisome possibility: “Quietly, the Kremlin has been trying to map the United States telecommunications infrastructure,” he said, describing a series of steps hackers have taken to develop “a cyber weapon that can disrupt the United States power grids and telecommunications infrastructure.” When McCain asked Sessions if the administration had a plan to deal with such an eventuality, Sessions admitted that it did not.”

    https://www.vanityfair.com/news/2018/01/russian-hackers-may-be-preparing-another-major-us-attack

    So that is all scary especially since dealing with hacking by foreign actors has, perversely, become a deeply partisan issue. On the other hand i’d still bet this was a simple mistake since dumb errors are far more common than anything else in the world.

      Quote  Link

    Report

  3. I said already on twitter that the biggest problem is that the combox has a setting for ‘this is not a drill’ (and/or there are procedures that allow/require you to input that)

    Drill messages should be clearly marked as such, and real-world messsages should just have the message without any fluff, emphasis, or editorializing.

      Quote  Link

    Report

    • You’d think that there’d be a meeting or fourteen where they sat down and hammered out the wording of the battery of pre-written SMS emergency messages.

      “How do we want to phrase the hurricane one?”
      (Two whole hours devoted to whether all caps should be used, whether punctuation should be used, whether adjectives should be used…)
      “Okay. Break for lunch, then back here and we’ll figure out how to phrase the missile attack messages.”

        Quote  Link

      Report

      • I would think that the alert messages for the entire US Pacific rim would have enough commonalities that Hawaii didn’t need a bespoke home grown system.

        (and/or they would be the ones that innovated it and then sent it along to the other Pacific coast states)

          Quote  Link

        Report

        • I would like to know more about this myself.

          Wouldn’t it make sense that any messages about foreign attack should be tightly controlled by the Defense Dept in Washington?

          Its one thing to have localized conditions controlled locally, but it seems like the binary nature of “we are at war/ we are NOT at war” should be part of the command and control system.
          So like, instead of a warning screen like “Are you sure?” it should be “enter the DoD authorization code”.

          But then, I am not sure how it works to begin with.

            Quote  Link

          Report

          • Wouldn’t it make sense that any messages about foreign attack should be tightly controlled by the Defense Dept in Washington?

            I doubt it. An in bound missile would trump any lack of message from Washington. Hawaii is (historically) pretty isolated and could expect to know of some types of attacks (including a missile from NK) before Washington does. Think Pearl Harbor.

              Quote  Link

            Report

  4. Btw, this is also why ‘zombie apocalypse’ is now used so frequently in emergency response & mass casualities drills, because it can’t be mistaken for ‘the real thing’.

      Quote  Link

    Report

  5. One day, ninety-nine balloons
    were released into the air
    and one-by-one they crossed the wall
    from over here to over there
    and on the other side, they saw
    a blip upon a radar screen
    the operator said it was
    the strangest thing he’d ever seen

      Quote  Link

    Report

  6. Occam’s Razor says it was a stupid mistake during a shift change and nothing more. Speculating otherwise might be amusing but it is dangerous for political sanity and amusing ourselves to death.

      Quote  Link

    Report

    • That’s a weird response when the NatSec community and lots of Dems on the Hill think we’re potentially amusing ourselves to death by not addressing Russian efforts to penetrate US security and infrastructure systems.

        Quote  Link

      Report

      • I need a minute to catch up on the narrative. Aren’t we supposed to believe the Russians interfered with our elections, going so far as to hack into DNC email accounts, various Senators personal email accounts, and try to penetrate 26 states election systems? Is that just a conspiracy theory spun out of practiced ignorance?

        Maybe Trump is right about the Mueller investigation after all.

          Quote  Link

        Report

        • Last I heard, the Russians appear to have illegally placed roughly $200k(?) in advertising on FB… in an election where Billions of dollars were spent on advertising.

          Did it happen? Yes. Did it matter? Probably not.

          In theory that illegal advertising purchases the 20k(?) votes who swung the election. In practice we have lots of media professionals who are supposed to be good at influencing elections here in the US. If we can’t predict which 20k votes matter then I seriously doubt the Russians can.

          This is independent of Trump doing something heinous because Trump is always doing something heinous and none of his people were experienced at knowing the rules. So the rules were probably broken, and I expect Mueller can find something Vile about Trump.

          I also expect it didn’t really matter more than what we already know and it didn’t swing the election more than HRC’s email server and/or Trump’s twitter feed.

          This fascination with the Russians is an attempt to externize an internal problem.

            Quote  Link

          Report

    • “Please choose what you want to do…
      1. Text mom
      2. Flick lights in bathroom to confuse coworker
      3. Missile alert
      4. Test missile alert
      5. Launch missile
      6. Cat videos”

      “Don’t you think maybe we should have separate menus?”
      “It’s more efficient this way.”

        Quote  Link

      Report

  7. I read Van Dyke’s article and a few things stood out.

    She included a video of an alert being broadcast on the TV over a basketball game. The content of that alert was different than the SMS message. So, presumably, the “wrong button” triggered some sort of chain reaction of alerts. She also mentioned that these were 2 of the 3 ways they’d be alerted, with the third being sirens. She didn’t hear the sirens but lives far from them. She reached out to others who were closed and they said there were no sirens, but apparently others have claimed to have heard them.

    So this leaves me wondering…

    If we accept the official story, someone somewhere pushed the wrong button. What happened next? Did that button directly cause the SMS messaging? Did it cause the television alert? Or did that button being pushed cause a light to go off or a bell to ring elsewhere and somewhere who saw that light or heard that bell then pushed a button to send the SMS message and television alert? Or did someone at the TV station see the SMS message and decide to put out a television alert? Why were the alerts different? Why didn’t (or did?) the sirens go off? If all of these alerts aren’t automatically triggered by the wrong button (and I assume they are not if the sirens did indeed remain silent) why did no one double-check that the light or bell or whatever was indeed correct before pushing their buttons to send their alerts?

    I’m really curious to know more about the chain of events that begins with someone pushing a wrong button and ends with 2 alerts being sent but not a third.

      Quote  Link

    Report

      • I hadn’t even considered that the official story wasn’t more or less true until I read this post and I’m still inclined to believe it’s true or true enough.

        I think it is easy to think, “HOW CAN A MISTAKE LIKE THIS OCCUR?!” But if you assume there’s probably 3 shift changes a day every day for however many years and probably lots of button pushes every shift, the error rate starts to look very tiny and probably approaching what is reasonable. Layer on what sound like genuine design flaws (which seem less reasonable) and shit happens.

        So I don’t offer this to challenge the official story as much to say there is lots that is still unexplained to me (and maybe there are perfectly good answers that I just haven’t seen yet).

          Quote  Link

        Report

      • The employee who screwed up has been reassigned. That confirms that it was an employee screw up and confirms that we’re in a new era of government accountability. Previously a government employee could’ve accidentally triggered a global nuclear war and the most punishment they’d face was a paid leave of absence.

          Quote  Link

        Report

  8. When I was an undergrad at UC Santa Barbara–this would have been around 1982, a/k/a the height of the Cold War, a/k/a The Good Old Days, I was up late studying in the middle of the night with the radio on. The Emergency Broadcast System warning tone came on without the usual “This is a test” language before it. That was startling. Back in those days we all knew the nearest best target for nukes. Santa Barbara itself would be a waste of a good warhead, but it is down the coast from Vandenberg Air Force Base, which would have been prime. So I sat there for a couple of minutes contemplating the prevailing wind direction, then the warning tone stopped and the regular program resumed with no mention of it.

      Quote  Link

    Report

    • Very often software like this is customizable, with sets of menu options read from some list of customer-specified “actions”, and thus just listed together without a lot of thought. In other words, a well constructed application might have very clear “test mode” options, color coded, with another set of “for real” options, which then have extra fail-safes. But if these are just added modules hacked together onsite, then you will see UIs like this.

        Quote  Link

      Report

  9. There’s a fifth possibility, which is that they knew there was not a missile, but wanted to test the system AND wanted to test the public response. Joe Blow is just the fall guy.

      Quote  Link

    Report

Leave a Reply

Your email address will not be published. Required fields are marked *