
Wow. That was intense and one of those kick in the shin experiences that really was an eye opener on how dependent we have become to instantaneous communication and how it can all fail so quickly. Oh and how dependent we are on electricity as well.
What are you talking about?
A “black system event” refers to a situation where the entire power grid or a significant part of it has gone offline, resulting in a complete loss of electricity supply — a total blackout and that happened to the Iberian Peninsula (Spain, Portugal, and parts of southern France) yesterday. To recover a black start procedure has to be done which is tricky, time consuming and something you (hopefully) have plans for, but have never tested.
What caused it?
Good question and as of today (day after) still not totally clear. I have seen something about “induced atmospheric vibration” on 400KV lines some how caused 15 gigawatts to disappear from the load in 5 seconds (about 60% of usage at the time), but still waiting for a more explain it like I am five definitive answer.
What happened?
That one I can answer more definitively as I lived it:
- At 12:33 I lost power at my house.
- I went to raise the circuit breaker thinking the washer of something had tripped it to find all of the breakers ok.
- Looked out in the hallway and saw emergency lighting on, so at least building wide outage.
- Looked out on street and saw no buildings with lights and all traffic signals dead as well, so at least neighborhood outage.
- I tried to rejoin the Teams call I had been on from my mobile but found everyone had been disconnected, so at least a Madrid wide outage.
- That is when I got a bad feeling.
- I went to the roof to see if there were any mushroom clouds, bombers, or fires on the horizon, but saw none of them (phew).
- By 13:00 had confirmed via family chat in WhatsApp that it was affecting family members in Madrid, Galicia, and Catalonia, so was looking very much like all of Spain outage.
- By 13:15 my mobile phone service (calls and data) stopped working as well. Read that again: only after 45 minutes of power outage I had no phone or data. As the day progressed there was some luck getting SMS texts to work, but was extremely spotty and by nature SMS are not guaranteed delivery, so was hard to know when and if someone actually got the message as we no longer have an agreed protocol on top of the communication channel like in the old days with walky-talkies (copy, over, etcetera).
Power was finally restored at:
- Sister-in-law in Cataluña at 18:01 (5.5 hour outage)
- My house in Madrid at 10:19 (9.75 hour outage)
- Brother-in-law in Galicia 02:00 (13.5 hour outage)
Lessons learned:
- always have gas in the car (gas station pumps require power to pump gas)
- always have cash (card validation requires network communication that all went down[1])
- always have water at home (my water stopped working immediately but others had water for a while)
- always have battery powered radio (was the only way to get information on what the heck was happening)
- always have canned food at home (some super markets in the neighborhood had generators and were working with cash only sales, but they ran out of diesel eventually)
- always have flashlight with extra batteries (stairwell emergency lighting only lasted about an hour and then the sun went down and lighting was needed everywhere)
- always have downloaded offline google maps for your area (can you live without google maps to get anywhere?)
- always have planned meeting point with family (once totally cut off better to have pre-agreed meeting points)
[1] I have unconfirmed reports that:
- Orange (which I have) lasted less then an hour (this one I can confirm that lasted about 45 minutes)
- Movistar lasted a couple hours
- Vodafone lasted 6 or 7 hours
Links, References and things that helped with this
- Physics StackExchange: What is “Induced Atmospheric Vibration”?
- Wikipedia: List of major power outages
- BLACK SYSTEM SOUTH AUSTRALIA 28 SEPTEMBER 2016.pdf
- Wikipedia: Synchronization (alternating current)
Thanks for reading and feel free to give feedback or comments via email (andrew@jupiterstation.net).