It's not pleasant to watch, but it happens.
As an employee of a large IT vendor, I've been at the receiving end of a reasonable number of vendor beatings.
Occasionally it's richly deserved. But, sometimes, it's masking a deeper set of issues that have very little to do any vendor whatsoever.
Such was the case a while back when I offered to do a meet-and-greet for a customer coming in to EMC's Executive Briefing Center.
The Role Of The EBC At EMC
EMC has invested heavily in customer briefings ever since I've been here, which is to say a very long time. There's the large EBC facility in Hopkinton, supplemented by regional centers in Cork (Ireland), Singapore, Santa Clara and a new capability in Durham, NC.
The Hopkinton facility is the busiest by far. On a given day, there's anywhere between five and twenty customer or partner groups coming through for a day or two. Sometimes they're looking for product updates, sometimes it's about a specific project or challenge, very often it's to get a better sense of vision and direction.
Part of our culture here is that we all spend considerable time with customers and partners; hopefully doing more listening than talking. Indeed, during hiring interviews, I make it pretty clear that EMC stands for Everyone Meets Customers. It starts with Joe Tucci and works its way throughout the organization from there.
Thanks to the EBC, any EMC employee nearby has no shortage of potential customers and partners to interact with -- if they choose, that is. The downside is sampling bias -- you need to balance EBC interactions with meeting people on their own turf.
I usually make a great effort to learn as much about specific situations before walking into the room with a customer or partner. Occasionally, circumstances force me to walk in largely blind, and figure out things as we go along. That can be, well, interesting ...
One of the things I'm often asked to do is greet people in the morning. That means we sit around the table, everyone introduces themselves, and I try to gain a better understanding of what's going on in their world.
The introductions went well enough: a few people on the storage team, their team leader, and the person he reported to, responsible for a portion of infrastructure and operations. So far, so good.
My opening gambit was pretty generic -- what's going on in your world, and what do you hope to accomplish today?
And then it started.
Here Comes The Flood
The storage team lead opened up with "We're spending way too much on storage". Hmmm, I'm thinking, that's not good.
"And we're having all sorts of performance problems". Huh, that's unusual. Wonder what's up with that?
"And there are way too many outages". If true, that's very serious stuff.
"And your people aren't able to help us". OK, this is getting very unusual indeed.
It just kept coming out. Complaint after complaint. Injustice after injustice. Multiple threats of switching vendors if the situation didn't improve.
It was Vendor Beating Time.
The Politics Of Getting Beat Up
When someone is pretty upset, it's usually best to hear them out -- no interruptions, no questions -- just let them go. If you're a vendor dealing with a very upset customer (justified or otherwise), that's especially true.
As the storage team lead started to get on a roll, you could feel the tension level rise in the room. That's natural.
The storage team lead's direct manager tried to interject and add some perspective to the discussion. I sort of sent him a look that said "hey, let this guy get it off his chest, it's OK". A few minutes in, the EMC sales rep started to get a bit defensive as well (after all, this is all looking pretty bad), but I sort of sent him the same message, back down, let it all out.
I started nodding and scribbling down notes to come back to later. After about ten minutes, you could feel the ferocity lessening a bit, the energy was starting to shift.
When it felt like the time, I asked if it was OK to come back and ask a few more questions about each of these issues. Of course it was -- what else could be said?
Help Me Understand
I started with his first complaint -- that he felt that he was spending far too much on storage. Can you tell me more about that? -- I asked.
The picture started to emerge. The entire IT function was under pressure to significantly cut costs (a separate topic probably worthy of discussion in itself, but not at this time). Storage expenditures were one of the most visible line items in the budget. Hence there was a lot of pressure on his team to spend less.
OK, I said. How much storage are we talking about? He tossed off a number of around 70 terabytes of primary storage, and about 95 terabytes of secondary (archival-ish) storage as raw capacity.
A good size, but not enormous I thought.
How fast was it growing? He wasn't sure, but he guessed raw capacity had doubled in the last twelve months. OK, I thought, that's something to ask more questions about later. And how many storage people do you have?
Seven primary storage administrators, exclusive of the backup team, server team, database team, etc. I didn't hide my astonishment at the large number very well. The storage team lead started to get defensive about my reaction, so I changed the subject.
OK, of the 70+95 terabytes of raw capacity, how much of this is visible to users? Stuff they can directly see -- like allocated and unallocated usable storage, visible snaps and the like.
He wasn't really sure.
I explained my interest that "conversion efficiency" was a useful quick-and-dirty metric. Here's X amount of raw capacity. Here's Y usable capacity directly visible and accessible to the people who use it. Do a good job, and you can often be north of 80% of your raw capacity being directly usable for work. Do a poor job, and it can drop below 50% or worse.
A long and pregnant pause. That angle wasn't working well.
So I came at it from another perspective -- what tools and processes do you have in place to manage storage usage, and report back to people how it's being used? Well, they had a bunch of spreadsheets that someone occasionally ran around and filled out. Hadn't done one in a while, too busy. Not sure if anyone really read them.
I decided to press my point. If I'm, say, the owner of the big SAP application, do I know what I'm using, what sort of service I'm getting, and how much I'm paying? No clear answer.
Enough on this topic. The picture was getting clearer, and it was time to move on.
Where Does It Hurt?
The next topic on the list was the complaint around performance -- where, exactly, were they experiencing performance problems? Database queries were getting slower, and the business users were starting to complain. Fine -- did they have any evidence as to where the problems might be -- storage media, storage array, storage path, server, database, query structure?
No clear answer.
OK, I asked the EMC team, had we offered to take a look and see what might be up?
The EMC presales engineer said that we had spent significant time looking at their setup, and came to the conclusion that the root cause was pretty obvious: the database environment had grown willy-nilly over the years -- it wasn't laid out well, the queries weren't particularly well written, and so on.
Sure, there were things we could do on the storage side (e.g. faster storage, better layouts, etc.), but it was a bigger issue that just storage performance. The EMC team had not only offered up a report, but had also shared our extensive library of best practices in an effort to help.
I then turned back to the storage team lead for his view. He said he had challenges working with the database and server teams to address the problem, and everybody was pointing fingers at the storage team.
The picture here was getting clearer as well. Time to move on.
Any Outage Is A Serious Outage
Look, at EMC, most of us are storage people at heart. If you can't get to your data at any time (and for any reason!), that's a really big important deal in our world. So I wanted to know more -- what the heck was going on here?
The storage team lead talked about "multiple SAN failures". Really? Turns out that they had a number of servers that periodically lost connectivity to the array, causing a huge turmoil.
Wow, I said, how could two paths fail at the same time? Something really bad must be happening here.
No, I was wrong. The problem was with servers that were single pathed -- to no-longer-supported operating systems, FC adapters and driver versions, to make the picture more complete. It got worse. The EMC presales engineer spoke up and said that every quarter they'd done an inventory of the customer's environment, flagged the problems as a serious concern, and no action had been taken.
For the last two years.
Another serious outage resulted when the customers' team were moving stuff around, and accidently trashed production data. EMC got the phone call after the damage was done. There were problems recovering from backup, and so forth.
I stopped that line of discussion, and decided I'd heard enough.
Time For Some Tough Love
I turned to face the storage lead's immediate boss, and decided to say what I thought needed to be said. Even if it wasn't going to make me any friends in the process.
"Look", I started "I get to see how hundreds and hundreds of IT organizations run their operations, and especially how they manage the storage function. So keep that in mind when I say what I'm about to say".
"And, please understand", I continued, "that no vendor is perfect, and there are certiainly many areas where EMC could be doing better, but I am absolutely convinced you'd have the same problems -- or most likely worse -- if you decided to go with another vendor".
"Dealing with rapid storage growth, demanding users and a tough budgeting environment isn't easy. It requires a partnership: we do our part, you do yours. Simply blaming your vendor for your troubles might make you feel good in the short term, but it won't solve anything".
I mentioned a friend that had now been unsuccessfully married four times. He has become quite adept in blaming each and every woman for the failures. It makes him feel better, but his life still sucks.
"You have a key choice to make here. Your storage environment is growing fast. What might have worked in years past clearly isn't going to work going forward. I think you can see that".
"Here's the choice: you either have to invest in building a modern storage management team that's organized, trained and equipped to deliver storage services, or you need to hand over the storage function to a managed service partner that knows how to do this. The problem isn't the technology, it's how it's being used".
The storage team lead didn't like where I was going one bit, and started to speak up in his defense. I ignored him for the time being. I was on a roll.
"Seven primary storage administrators, and no evidence of tools or processes? That's a problem. Poor relationships with your IT co-workers that prevent you from addressing user-visible issues? That's a problem. Multiple outages that were clearly preventable? That's a problem. No visible use of change control procedures? That's a problem. Not being able to act on the advice and help we're offering you? That's a problem."
"We can dress it all up with happy words and make it more palatable, but -- you asked me for my opinion -- and there it is. You -- as an IT manager -- have some decisions to make. We'll be glad to help you explore your options in more detail, and show you what we can do to help -- but I think the next big move is yours".
Long silence in the room. I think the poor EMC sales rep was going to burst.
The response was a bit awkward from the IT infrastructure manager -- after all, the storage team was sitting right there.
He sort of acknowledged the points I was making in a roundabout way, and sort of thanked me for my input. I suppose that's the best I could have hoped for, given the circumstances.
I'm Sure I Didn't Make Any Friends That Day
Sometimes I think I might be weird.
So many vendors take such extraordinary pains to please everyone all the time. Even if it's not in their customers' best interests. As I left the room, I felt sort of bad for the EMC sales rep; after all, I hadn't done the vendor-happy-talk and its-all-our-fault and we'll-try-to-do-better approach as might have been expected.
In this particular situation, I just couldn't do it. Something important had to be said. And I'm not saying that EMC was without fault -- either in this specific situation, or generically.
But -- if we fail to be transparent and honest with our assessment of the real problem, isn't that a concern as well? That's what real partners do for each other.
Sometimes I'm a technologist. Sometimes I'm a marketeer. Sometimes I'm a visionary. Sometimes I'm a career coach.
And, once in a while, I'm a therapist doing an intervention.