Incident Un-Response

Its late at night, you’re on a walk, and you see something wrong. What do you do, who do you call? I walk several miles daily to try to keep my health up given that my career has me sitting at a desk all day. As a result of this, I somewhat regularly encounter situations that need to get called in, and over the years, I’ve become increasingly weary to try. Not because I don’t think its important to do so, but because of how universally bad the response is from the supposedly highly trained incident response groups is.

This blog post was prompted by a situation I encountered on my nightly walk. Because I have a career in large scale infrastructure, I tend to be pretty interested in the systems around those that I directly am responsible for. While I don’t personally handle high voltage distribution, it is interesting, so I’ve learned a fair deal about it. This also means that I have a pretty good idea of when something looks wrong. Those who work in infrastructure know exactly what I’m talking about here, an almost sixth-sense type response that you’re walking up on something out of place.

Tonight was one such night and while walking on a sidewalk that is on the back-lot of a school and a neighborhood street, I stopped dead just short of something hanging down in my path from overhead. There are no trees on this side of the street, so it could only be hanging from the overhead power lines. Quickly removing myself from the area to the other side of the street (as a colleague once put it “de-assing the area”), I pulled the flashlight off my belt and shone it on where I was just standing. Thankfully I had not almost walked head-first into a downed phase wire, but it was instead a sturdy kite line going up to a kite entangled in the primary supply at the top of the pole. Based on how it reflected my flashlight, I think its probably one of those cheap Mylar kites you can get at most retailers, emblazoned with popular characters and designs.

Not knowing if this was already called in, I made the choice to call the local utility responsible for the distribution infrastructure. This is already more than most consumers would know how to do given the unique and complicated way that the ERCOT grid operates, but knowing that I needed to get a hold of Oncor I found their after hours number and called it. What greeted me was a static laden interactive menu, probably running on some long abandoned Cisco equipment. After navigating the unintuitive menu, which really discourages you from filing a report at all, I decided I would report this under the category of “downed lines, damaged poles, dangerous equipment malfunction or life threatening situation.” This list is less than helpful, but is the most accurate to what I was looking at.

Lets take a step back and let me describe to you exactly what I was looking at. This is a quiet neighborhood street, and most of our utilities are serviced underground, including power distribution. In the center through-streets, however, exists overhead feeder equipment that brings in the main feeds for the neighborhood. It was in this overhead line equipment that the kite had become entangled and that its flight line was now dangling from all the way down to the sidewalk, where someone had left it looped over the school’s back metal fence.

Obviously this is not great, but isn’t immediately a life threatening issue. The kite was across at least one phase at the top of the primary span, but didn’t appear burnt away, so I had a pretty good idea the power on the span was de-energized. In my neighborhood, like most neighborhoods, there’s a set of lines that are energized in the normal modes of operation, and a handful of point to point lines that can be used to bridge out sections of the grid when the need arises. I’m fairly sure this span was one such tie-line that is normally not energized. If it was actually energized at the time, then that is the world’s luckiest kite.

The string, while not great to be hanging all the way down from a primary line to the sidewalk, isn’t itself conductive generally. Sure, at high enough voltages even air conducts, but dry kite string on its own isn’t that conductive. Notice I said “dry” kite string. From living here for several years, I know that the sprinklers in this field come on between midnight and 2AM, and they have some considerable over-spray, dousing the chain-link fence, the sidewalk, and well out into the street. While dry kite string doesn’t conduct that well, wet kite string is another matter. This situation, coupled with the proximity to a playground and the sidewalk itself convinced me it was worth calling this in after hours.

The first representative I spoke with wanted to know the service address associated with the address. There isn’t a service address for the pole and I navigate my own neighborhood by sight alone, so I had to jog back up the street to figure out what cross street I was on. I first tried to give the representative the ID number on the pole nearest to the span in question, but she informed me that the numbers don’t locate the pole. This is patently false, that is the entire point of the numbers on the pole, but I guess the dispatch center isn’t able to use the locating numbers that have the sole function of identifying physical infrastructure to locate the fault. Fine. I found the cross-street, described the situation, and said they could locate it by driving 4 poles to the East from the cross streets I gave. Because I answered “yes” to the script question “is the item hanging from the pole near where the public or children could interact with it” this triggered an escalation in response. The representative transferred me to another number, where I sadly didn’t get the representative’s name.

This second, absolutely useless representative re-took the entire report, then asked what department I was with. Confused, I explained I was a private citizen who had been transferred to this line from another part of Oncor’s after hours call center. This person then informed me they couldn’t take my report, because they could only take reports from police and fire personnel in their department. This is a spectacular process failure, and if you or someone you know works in Oncor management, please bring this to their attention. I don’t care if you’re the receptionist in some kind of sales office, if you get a hazardous report, you don’t try to pawn it off that the report is invalid because of who made it. They dumped me back into some hold queue, which then brought me back to another department, I suspect the one I’d just talked to. I re-made the entire report for a 3rd time to yet another representative who did an amazing job keeping a calm demeanor given my now growing frustration. They submitted it into some other queue that will probably take several days to actually get looked at, and will confirm my suspicions that Oncor really doesn’t have the machinery in place to take emergency reports seriously.

Why am I so frustrated over this?

This is not the first time I have encountered a useless “emergency” operations line. I’ve now had this happen with the bay area Valley Transportation Authority (VTA) who couldn’t localize an issue based on the signal number I gave them (the entire point of the signal number being to locate it precisely), Caltrain who couldn’t understand the severity of the overhead catenary wire having been knocked loose and fouling the main line, which I was desperately trying to get them to caution trains on before the next train barreled through a section with damage, and PG&E who tried to tell me that a loud crackling sound and purple glow on an insulator on a high tension transmission line was totally normal. For those of you wondering, that’s a phenomenon called corona-discharge, and is usually indicative of a breakdown of the insulator stack. By all means though its totally normal for the line to glow and crackle, and this is just another free service that PG&E provides (sarcasm, obviously).

The problems that I have with this are two fold. First, the general public is taught from a young age in the US that if you see a hazardous situation, you have a duty to report it. I remember presentations from the fire department, police, and the electrical and gas utilities in my town when I was in grade school. They underscored that this kind of infrastructure is not to be feared, but must be treated with respect, and you should always report something that looks wrong. After all, they can always tell you its fine, but they can’t know about situations that weren’t ever reported.

Secondly, this kind of poor response means that when something is legitimately hazardous, its hard to report it. As someone with an engineering background, I’m always torn on how much to use more proper terms versus playing dumb and giving a less useful description. In the example of the kite in the lines I reported it like this:

A kite is stuck on the wires at the top of the pole with the string reaching the sidewalk.

If I’d been talking to a more qualified individual, or at least someone who had given me some amount of confidence in their ability to take down a report, I would have reported it differently:

A kite that appears to be Mylar is entangled in the primary supply at the top of the span, fouling at least one phase; it is trailing a non-conductive line that is contacting the secondary span, the telecommunications span and the metal fence underneath the pole. You can locate this by either locating the transformer with ID <number> or by locating this cross street and proceeding 5 structures to the East along the span. The nearest adjacent address is <address>.

As you can see, the second description is much more accurate, clearly identifies the scope and scale of the problem, and uses more proper terms for things. Unfortunately in our litigious society words have power, and I’ve grown extremely wary over the years to use the real names for things after examples have cropped up of people being sued for practicing engineering without a license when reporting issues.

This is a rant, but I want there to be positive outcome of this, so here’s what I want you to walk away from this with:

• If you’re an operator in one of these call centers, learn more about what it is you’re taking a report for. You should be able to understand synonyms like “structure” and “pole”. You should be able to understand that the number on the pole absolutely identifies its exact position, and you should record it in your notes even if your system can’t take the number in directly. Most importantly, you should conduct yourself in a way that people talking to you believe you’re taking them seriously and you understand the information being reported to you. If you don’t understand, plainly state this and ask for clarification.

• If you’re a member of the general public, perhaps even someone unfortunate enough to lose your kite in the power lines, make sure you call in the situation. You’re almost never going to get in trouble for reporting the situation, and you can help keep everyone safe if you report the hazard as soon as it is created. Make sure you provide as much detail as you can, and do so in a way that someone not familiar with your area will understand. I like to use both streets/addresses and recognizable landmarks (near this restaurant, past the car-wash).

• Know how to report problems like this. First and foremost in an emergency where life is at stake call 911 if you’re in the US. If you can though, try to understand better who the report needs to go to. On the bottom of the bill for your water, electricity, and gas should be a small block of information about how to report hazardous situations. Don’t be afraid to navigate the phone tree past the fear mongering warnings and get a hold of a human after hours.

• If you’re an infrastructure operator, make sure that your systems can accept a wide variety of identifiers to supply location. The general public might not know what a CLLI code is, a signal head number, or a pole span identifier, but they recognize that this magic string of digits and numbers looks important, and its attached to the thing they want you to know about. By giving people options about how to identify the location they’re reporting, you speed up your total response time and cover cases where there is no cross street. The transmission line example from earlier was several miles down a bike path in a nature preserve, and the only realistic ID I could provide was the structure number. I wound up waiting at the entrance to the preserve at the cross street I gave the operator to tell the response crew the structure number.

• Finally, if you’re in any capacity where you’re taking in a report of an incident, take down everything that you’re told, even if you don’t understand it. I’ve had plenty of cases now where the operator refused to note something down that I knew was important, only for a repair crew to have to call me for that tidbit hours later. You may not understand why its important, but if someone feels the need to include it in their descriptions, you should write it down because they may know more than you about why this factoid is meaningful.

Overall, I am amazed at how bad the response is for incident management in the world of utilities and physical infrastructure. I work in a world of software and IT where we have rigorously tested incident response procedures that we regularly exercise to validate our ability to identify, report, and remediate a problem. The fact that we can do this so much better in a world where lives are not on the line makes me angry that in fields where safety is at stake the call-center response is so bad.

Hopefully this post has been an interesting read for you, but mostly I just want to put pen to paper on how bad this problem is so that hopefully the more technical readers of my blog can use this to catalyze a review in their own organizations. How well would your employer hold up in these situations? Would I have had a better time reporting this kind of fault to your department?

The views expressed in this rant are my own and do not reflect any positions of my employer, though their robust incident management procedures are part of why I wrote this. If you want to tell me I’m completely wrong, or on the off chance you’re from Oncor and want to follow up on this, you can reach me at maldridge@michaelwashere.net.