MidWest IT Survival

Whether you are working in a complete custom software development shop with little vendor interaction or a technology integration shop with vendor solutions integrated with other vendor solutions on top of yet other vendor solutions, you will have to manage vendor relationships to some degree as an IT manager in a MidWestern company.  This series looks at the complex arena of IT vendor management and offers some tips to make the arduous process a bit less arduous and possibly discover some additional benefits along the way.

Vendor Management Categories

• Role of the Sales Rep

Leverage your Vendor Sales Cheese to avoid the Architecture Astronauts!

In the previous article [], we explored how this role can benefit the IT engineer by cutting through the bologna that is the normal route to getting technical support from a vendor and putting one in direct contact with a peer senior technical resource that can solve tough problems quickly.  But what about the benefits to an IT manager?

IT Manager Dividends

Scenario 1 – You Don’t Have Bob the Engineer from the Previous Example

Well, let’s start with that same hypothetical problem scenario in the previous post [] with a production problem being reported in the service you and your team are responsible for, in this case the FlimFlam software.  You don’t have a Bob that has introduced himself to the Vendor Sales Cheese and established himself as the guy, that when he reports a problem with FlimFlam, there is indeed a problem with FlimFlam and Bob needs senior tech support ASAP.  It not, the Vendor Sales Cheese is going to spend his or her value time not selling but rather in fire suppression mode.

Once you notice your team is stuck in the troubleshooting process with the FlimFlam software and the trouble ticket that is open with the vendor is going no where, have your Vendor Sales Cheese contact info handy:

Sally the Manager: “Hey, Vendor Sales Cheese, it is Sally at ABC Company.  Well, no, everything isn’t quite fine.  We’ve got a problem.  FlimFlam is causing customer pain and is throwing an error 57 that no one, not even the tech folks behind your support web site are able to figure out.  You know I don’t waste your time with the trivial stuff, thus this isn’t trivial.  I am about to get on a conference call to explain to my peer management that we have our top engineers working on the problem but I don’t have a good answer when they ask what the vendor of FlimFlam is doing to aide us.  Yes, you can help, that is why I called.  Can you get a senior tech person to look at support ticket <blah> and then have that senior tech person call Joe on my team at <blah> to start resolving the issue?  I have your commitment someone is going to call Joe, right?  Good … now I have a much better story to tell everyone on the conference call.  I’ll be in touch.”

Similar to the previous example, you can leverage your relationship with the Vendor Sales Cheese to get priority service.  From a management perspective, you have just increased the technical capacity of your team without incurring any additional cost by leveraging the notion that the Vendor Sales Cheese would rather have a happy customer for which they can manage this problem more proactively since they were engaged early in the problem resolution process.  Any experienced Vendor Sales Cheese that been set on proverbial fire coming into a customer hot zone with threats of having the vendor and all provided products and services throw out because high level management had to get involved with a problem that, to them and rest of the organization, should never have occurred in the first place.  High level management sees the $$$ from their budget going to these monthly maintenance fees to their vendors as insurance that they will never have to directly deal with a problem caused by the vendor.  The Vendor Sales Cheese that can get engaged in a hot problem early, bring the right level of technical support and relationship support at the highest level and make the problem go away quick has actually earned positive face time with higher levels in the customer organization.  Their bet is the higher levels in the customer organization will see the support value in addition to product feature set from the vendor and look for that vendor to provide future solutions.

Thus, if the issue is heating up in your organization and you don’t have a Bob that has dragged the vendor into the troubleshooting mix, get on the phone to the Vendor Sales Cheese and give the Vendor Sales Cheese the opportunity to augment your team’s technical capacity as well as the opportunity to manage the customer relationship early rather then when asbestos underwear will be required later.

IT Manager Dividends

Scenario 2 – You Have Bob the Engineer

In this scenario, Bob on your team has already brought the vendor into the troubleshooting effort.  So, all you have to do is put your feet up on your desk and watch the magic happen as Bob and the senior vendor resource solve the issue before the day is over.  Well, you could, but what if it is 6PM and Bob and the vendor are stuck after hours of troubleshooting?  Once Bob has reached out to the Vendor Sales Cheese, wait a brief amount of time and then make a follow-up call yourself.  Let the Vendor Sales Cheese know that of all the problems you wrestle with every day, this one has your highest priority attention.  This further emphasizes to the Vendor Sales Cheese that this problem could become the problem that has the Vendor Sales Cheese sending $300+ an hour vendor solution architects to your company for free to appease the post resolution craziness that will surely develop if this problem isn’t given top priority and resolved in short order.

Sure, the Vendor Sales Cheese will quickly become high maintenance because they have had to work with customers in postmortems where a bunch of company architecture astronauts are swooping in after the system has been stabilized to philosophize on how a completely different solution that doesn’t even involve the vendor’s technology would be the best approach for senior management to implement to avoid this type of problem in the future.  The experienced Vendor Sales Cheese knows that once they are up against the customer’s architecture astronauts, the customer’s senior management has lost faith the vendor can save them from getting involved in distracting support issues.  Plus, senior management gets to throw out a vendor that failed and proceed to be courted by new vendors salivating at getting the opportunity to make the sale and have the “we replaced vendor X at company Y” story to enhance their sales pitches of the future.

Again, be prepared for the Vendor Sales Cheese to be high maintenance, but take comfort in the constant polling for status can easily be a positive when you need the vendor to jump higher and you already have the Vendor Sales Cheese with full issue context ready to say “how high?”

This concludes the perspectives on the role of the sale representative short of their role in the sales cycle and pricing.  I’ll dive into this aspect in the next article with more MidWestern IT perspectives on the topic of the “Sales Cycle and Pricing” in the spectrum of vendor management.

Whether you are working in a complete custom software development shop with little vendor interaction or a technology integration shop with vendor solutions integrated with other vendor solutions on top of yet other vendor solutions, you will have to manage vendor relationships to some degree as an IT manager in a MidWestern company.  This series looks at the complex arena of IT vendor management and offers some tips to make the arduous process a bit less arduous and possibly discover some additional benefits along the way.

Vendor Management Categories

• Role of the Sales Rep

The Vendor Sales Cheese can keep you off the phone and on your keyboard

In the previous article [], we explored this role more deeply and how, as an IT manager or IT engineer in a MidWestern company, you need to partner with this role to be successful in delivering you and your team’s services to the company.  We established the notion that as either an IT manager or IT engineer, instead of bolting for the nearest keyboard, there is benefit to spending five minutes introducing yourself to the Vendor Sales Cheese and giving him or her a clear understanding of your role within the company and how you are linked to the product or service the Vendor Sales Cheese is representing.  We left off suggesting that this brief exchange will pay off in tactical dividends.  So, enough with the preview, what are these so called dividends?

IT Engineer Dividends

In short, someone to suck into your troubleshooting effort to get you the technical expertise you need without having to sit on the phone on endless hold finding it yourself.  Someone you can say: “Hey, I did everything I was supposed to do to get help and I am stuck.  Get someone who can speak at my level that knows your product and can help me get this working ASAP.”  Take this typical scenario:

A technical problem makes its way through your business call center through the IT technical support helpdesk to your inbox.  Based on the brief explanation of the problem and the levels of “reboot your PC and try again” and “close your browser, clear your cache and try again” that have been tried with no success, this hypothetical situation suggests you are going to have to roll up your sleeves and figure this out because likely, no one else can in the company.  So, after much wailing and grinding of teeth, you are able to reproduce the problem in a test environment and have ruled out everything except the FlimFlam product.  From everything you can tell, you can now get the problem to occur at will, but all the FlimFlam system does is throw an “Error Code 57: Process died unexpectedly”.  The almighty Google is no help with error code 57.  The vendor’s tech support web site or “knowledge base” (which you have now dismissed as an oxymoron) completely mocks you with no reference whatsoever to error code 57.  So, no instant problem resolution gratification today, you have to log in to the vendor’s support web site with your company’s magic trouble ticket account to open a support issue.  You have been down this road plenty of times before, so you succinctly enter the exact end user steps to reproduce the problem and generate an error code 57.  You cut and paste in a copy of the system log that says “yep, I’m definitely throwing an error code 57 … and unexpectedly as well!”  You provide your platform and vendor product version, revision, and patch level down and dump the configuration out to the final detail.  You get back a trouble ticket number which you write down in the false hope your next email from the vendor’s support site will have the magic cure.

Off to lunch at the default route … err, food court at the mall

When you get back to your desk, you see an email from the vendor’s support site indicating your ticket has been closed.  You log back into the vendor’s site to see the last ticket status entry read:

“Upgrade to the latest version by applying patch 59837”

You switch over to the download tab and punch in “patch 59837”.  You quickly skim the release notes only to find absolutely no reference to error code 57 or anything that even resembles the problem you reported.  You’ve played this game many times before.  But you know, you have to play it or you get stuck at this level in the game, forever unable to advance.  So, you download patch 59837.  You install it in the test environment.  No install errors.  You re-test the system and low and behold, on the first attempt, you generate error code 57 with the same “unexpected-ness”.  So, you go back to the vendor support site.  You re-open your ticket; indicate you did exactly as told with no success.  Again, you’ve been down this road, so you re-state the platform and product versions showing the new patch applied.  You re-attach the log files and a dump of your configuration settings.  You re-attach the steps to create the error.  You raise the ticket to highest priority level you can.  You submit it back to the vendor.

Time goes by.

People in the company, including your boss, start asking: “Hey, when is that problem going to get fixed, people are complaining.” or “customers are getting irritated” or “we are starting to experience high call center call volume related to this problem.” Or whatever constitutes the inter-company fervor building to where you will soon be joining conference calls to explain what is going on and where things are at … rather than being allowed to actually fix the problem.

In anticipation of that first “please join the problem resolution conference line” alter, you re-check your ticket status online and see:

“Ticket Status = Pending”

… and nothing else.

Your world is about to get even more unpleasant as you see you’re frustrated and exhausted boss heading to your cube.

Wouldn’t it be great to have some human at the vendor to reach out to who is motivated to keep your company happily paying the monthly maintenance fees to help cut through the bureaucracy and get your technical peer at the vendor working on a fix for this problem?  Someone who can find that singular vendor engineer, that upon seeing your configuration can immediately go:

“Geez, they are running on OS 34 in 61-bit mode?  They need to add the ‘no cache during day light savings time=yes’ setting to their config file or else they will throw error 57 every time someone presses the ‘Q’ then ‘k’ keys.”

Here is where having the contact info for the Vendor Sales Cheese handy and having had that five minute conversation not too long ago with the Vendor Sales Cheese pays big tactical dividends.

Bob the Engineer: “Hey Vendor Sales Cheese, it is Bob at ABC Company.  Hey, I am getting the run around on support ticket <blah>.  I did everyone as instructed but we are still getting errors from FlimFlam.  A whole bunch of managers are going to get together and start talking about this problem with FlimFlam which means they are probably going to call you at some point if this problem doesn’t get resolved.  What do you need from me to get a senior tech guy to call me ASAP to avoid this pending mess?”  (Note the clever use of language to make your problem the Vendor Sales Cheese’s problem as well.)

The Vendor Sales Cheese doesn’t want to spend time putting out a fire at a customer’s site due to his product or service.  He or she wants to out selling their product or service to a new customer.  Plus, the Vendor Sales Cheese knows you from that fire minute conversation at which they “Check!” linked you to the top tech guy who knows his stuff and only needs help when something is going horribly, horribly wrong at ABC Compnay.  The Vendor Sales Cheese starts lighting fires within his company for get their FlimFlam tech expert on your phone ASAP.

That five minute awkward conversation with the Vendor Sales Cheese pays off big time when that FlimFlam senior tech guy or gal calls you with the magic config file setting that makes the whole problem go away.  And equally important, stops you from having to join the company trouble call and spend countless hours trying to explain to a conference call full of managers.

IT Manager Dividends

Ok, I see the value to the IT engineer, but what about the IT manager?   I’ll dive into this value proposition in the next article with more MidWestern IT perspectives on the topic of the “Role of the Sales Rep” in the spectrum of vendor management.

Whether you are working in a complete custom software development shop with little vendor interaction or a technology integration shop with vendor solutions integrated with other vendor solutions on top of yet other vendor solutions, you will have to manage vendor relationships to some degree as an IT manager in a MidWestern company.  This series looks at the complex arena of IT vendor management and offers some tips to make the arduous process a bit less arduous and possibly discover some additional benefits along the way.

Vendor Management Categories

• Role of the Sales Rep

You want to get to know the shinny suit

In the previous article, I concluded thoughts on vendor service integration challenges.  I made a cavalier reference to the “Vendor Sales Cheese” role.  This article will explore this role more deeply and how, as an IT manager in a MidWestern company, you need to partner with this role to be successful in delivering you and your team’s services to the company.

I don’t think there are more diametrically opposed roles in business than the IT engineer and the IT vendor sales representative.  One of the best descriptions of the complex persona that is the IT engineer is The Nerd Handbook.    IT engineers look at the world as an ever unfolding flow chart of logical constructs built on top of more logical constructs.  They are constantly learning and building.  They prioritize human interactions based on a peer level of technical appreciation and comprehension.  If someone isn’t at their level of knowledge on the subject at hand, the value of the exchange diminishes rapidly in their mind.  On the other hand, the Vendor Sales Representative or as I’ve affectionately relabeled as “Vendor Sales Cheese”, as viewed through the IT engineer lens, couldn’t be worth even a nod in the conversation spectrum.  If you align yourself more with the IT engineering mindset, I bet you are getting ready to HTTP 302 yourself off this article and on to something more technical.  I beg you to continue reading in the hopes I can influence you to consider a logical argument for the value of the Vendor Sales Cheese in your technical and/or management function.

So, as a typical IT engineer or engineering manager, your initial interactions with the Vendor Sales Cheese have you thinking: “This person is way too positive and friendly.  That sure is a slick and way too shinny suit.  I need to get outta this conversation and back to my keyboard ASAP.”  Yes, the Vendor Sales Cheese meets new people every fifteen minutes of every day.  Those people could be the tier one HelpDesk technician or the president of the company.  Hence, they error on the side of potentially meeting the president and bust out the shinny suit.  In meeting people, they need to quickly determine your role in the customer vendor relationship ASAP since there is going to be someone new to meet in another fifteen minutes.  Thoughts going through the Vendor Sales Cheese’s mind:

• How do you align in the organization against the product or service the Vendor Sales Cheese represents?
• Are you an end user that is going to be a source of complaints?
• Are you a decision influencer that won’t make the final purchase decision but could influence the decision maker and possibly tank the deal?
• Or are you’re the golden role, the decision maker that is the person between the Vendor Sales Cheese and closing the deal to get the big compensation bonus?

The Vendor Sales Cheese is trying to determine this as quickly as possible in the limited interaction time they are given.

Sure, you can return to the safety of your keyboard and the logical and controlled in order to avoid the seemingly unpleasant and awkward conversation.  But is another five minutes of conversation really going to kill you?  My recommendation is make this five minutes tactically productive by immediately describing your role within the organization and how it aligns with the product or service the Vendor Sales Cheese represents:

IT Engineer: “Hi, I’m Bob and I am the lead engineer in ABC Company that has the job of taking your FlimFlam software product and cramming it into our enterprise IT environment.  I can’t sign a PO and can’t buy anything.  But, when there is a tough problem with the FlimFlam software here, I get the call.  I’ve been working with it for X years.  So when I need tech support, I don’t need the 1-800 number level 1 tech.  I need access to the guy who, like me, knows how FlimFlam works inside and out.  How do I get that tech access so I don’t waste your company’s time?”  (Note the clever use of language to suck the “vsc” into the need to solve a problem for his company and help a customer at the same time.  How can the “vsc” resist this?)

IT Manager: “Hi, I’m Sally and I am the manager over the team that integrates your FlimFlam software with the rest of the technology here at ABC Company.  Let me start with the fact that I am not the guy that signs the PO, but I have the Director’s ear does.  We’ve had great success with FlimFlam but we know there is plenty of competition in this product space.  My biggest challenge with your company is X.  What is the best way to improve the X situation?”  (Again, sucking the “vsc” in by creating a scenario he/she can’t possibly walk away from since they are ever so relationship positive focused)

For the IT engineer, you have given the Vendor Sales Cheese exactly what they need to know:

• Bob = tech guy at ABC Company that can sing the praises of FlimFlam or make a lot of noise when we drop the ball failing to supporting his priority support needs.  Check.

For the IT manager, the Vendor Sales Cheese knows:

• Sally = manager at ABC Company that shouldn’t get the loge tickets to Sunday’s game at the stadium, but he needs some above average attention because she could tank the current/next deal by pitching to the VP/Director/Other that another company with a competing product could be integrated quicker/faster/cheaper is giving her more respect and support.  Check.

Ok, you are scratching your head … “ok, I see how the stage has been set for some tactical value from this Vendor Sales Cheese exchange, but what does this really do for me?  Doesn’t this just lead to more annoying conversation?”  I’ll dive into this value proposition in the next article with more MidWestern IT perspectives on the topic of the “Role of the Sales Rep” in the spectrum of vendor management.

Anyone that has had to participate in a meeting to determine why some IT system went down is echoing a collective groan as they read this title.  For both IT managers and engineers alike, it is the least desired activity following a system failure of any kind.  Business and/or product owners outside of IT are waiting, after the dust settles and the system is restored to working condition, to have primarily two questions answered

1. Why did the system go down in the first place?
2. What is IT going to do to make sure this doesn’t happen again?

In the first article, I outlined the business context of the root cause analysis exercise in general and the complexities in clearly and logically arriving at a true root cause for a system outage due to the interconnected players involved.  In the previous article, I outline a particular IT engineering resource approach entitled “Openly Be the Hero” to participating in the root cause analysis process.  This article introduces “Play it Safe”.

IT Engineering Participatory Approach C = Play it Safe

Play it Safe!

Having seen the potential pros and cons of approaches A and B, I assume you are wondering is there any way to play the root cause situation safely?  There is, but you are going to have to put your engineering brain and ego on hold a bit.

Step 1 = Resist the urge to be either “surprised and confused” or the Hero.

At the onset, avoid meetings, emails, hallway conversations and basically any situation that might put you in a position to start down the road of approach A or B.  Reply with vague “I’m not sure.  I think we are still looking into that.  I am waiting on <whatever>, let me get back to you” type answers.

Step 2 = Get with your management ASAP and give them a full run down of what is going on, the situation and the players involved.

As succinctly as humanly possible, state the problem “we may very well be part of the root cause for this outage”.  That should get management’s attention very quickly.  Follow-up with “here is what I know, stop me if I am going to fast or you already know all this already” and then quickly and briefly step through the problem clearly indicating where you believe/feel/think each “fact” ranks in authority.  In other words, don’t claim something is a fact unless you hold a log file printout in your hand that date and time stamps what you are saying.  “The commonly held view is the temporary storage volume filled up before anyone could purge files as the system needs, etc., etc., etc.”  “I am 50% confident based on this log entry that disk space was an issue.”  Be prepared to be stopped and asked all kinds of questions pertaining to how you know this, from whom, who else knows, etc.  What is happening is management is starting to build the story of what is taking place factually, the black and white versus gray-ness of those facts and how all the players are positioned to take the blame.

Step 3 = If management doesn’t define your role and thematically what to say and not to say, suggest your role and seek confirmation

Equally as important as step 2, confirming how management wants you to proceed is critical.  If you complete step 2 but then go off and “Being the Hero”, you will be susceptible to all of the cons associated with being the hero. Rather, if you are going forward and executing your role under the clear direction of management, as long as you indeed execute and seek clarity when a unclear situation presents itself, it will be exceedingly difficult to fall victim to the cons associated with “Being the Hero”.

Step 4 = Execute your role and keep management informed of major milestones

Go forth and help the post outage root cause investigation effort always being mindful of your role as indicated by your management.  As you are made aware of “major” milestones, make sure you go back and update management as soon as possible.  The timely updates directly assist in reshaping the story and may be accompanied by some tweak in direction to your role.  “Major” represents any event or new information that changes the shape of events.  “Bob in infrastructure just shared that the daily disk utilization report was indeed showing a reduction in free space for the last two weeks” = share ASAP.  “Bob just shared he forgot his lunch at home” = ignore.  Yes, these are rather obvious examples of what to share and not to share, but the goal here is to develop your own system for listening strategically to all the information that is being shared in order to parse out the noise and direct significant facts back to management.

In summary, the approaches and recommendations here may seem a bit extreme to many.  If you are lucky enough to belong to an organization that is culturally rational and fact based, you may not be forced into these scenarios.  Yet, all it takes is one situation to get out of control and the scenarios above become reality.  The rational, fact based logical analysis of an outage is replaced by the panicking, irrationality and emotion of engineers and managers faced with the notion of job loss due to failure to prevent an outage disaster that had major reputational and/or financial impacts.

Anyone that has had to participate in a meeting to determine why some IT system went down is echoing a collective groan as they read this title.  For both IT managers and engineers alike, it is the least desired activity following a system failure of any kind.  Business and/or product owners outside of IT are waiting, after the dust settles and the system is restored to working condition, to have primarily two questions answered:

1. Why did the system go down in the first place?
2. What is IT going to do to make sure this doesn’t happen again?

In the first article, I outlined the business context of the root cause analysis exercise in general and the complexities in clearly and logically arriving at a true root cause for a system outage due to the interconnected players involved.  In the previous article, I outline a particular IT engineering resource approach entitled “Surprised and Confused” to participating in the root cause analysis process.  This article introduces “Openly Be the Hero”:

IT Engineering Participatory Approach B = Openly Be the Hero

“I know what happened, the temporary storage volume …..”

I will save the day with facts no one can refute!

This approach, which is diametrically opposed to the surprised and confused approach, comes with some different risks.  By standing up and sharing every technical fact you can get your hands on to point out what really is going on can back fire in exactly the opposite way as the surprised and confused option.  People will tend to latch on to the one spouting off all the undeniable facts and suddenly the masses will associate the one with all of the answers as the one being in a position to have avoided the problem all together.  As far as your management goes, if they aren’t on board, you’ve placed them in a difficult spot to be supportive if the tide turns towards the root cause being the hero’s perceived lack of involvement.  Your peers, fearing their job might be in some jeopardy, will most likely slink down in their chairs to remain quiet and allow you to stand tall to take the proverbial daggers of blame.

Now if you are one that has put in the extra energy to understand how the system or systems were constructed, the “why” behind the seemingly architecturally backwards ways certain business processes are completed you may struggle with avoiding the hero trap.  You may be thinking: “The facts that I possess clearly indicate without compromise that what I known to be the root cause is the root cause.  Why can’t everyone just go with the facts and be done with it?”  Not everyone is comfortable accepting the facts even if they are the facts.  What if the facts suggest a particular individual or group of individuals have been linked to the last five system outages?  Maybe these five outages are legit and the individual or group is trying desperately to improve their system management activities.  The last thing they need is another problem piled on top of their previous problems to further put pressure on management to take some action.  In an effort to save their jobs and buy more time to get out from underneath their pile of problems they can redirect the masses to focus on the hero’s involvement and thus take the heat off themselves.

“Let me understand, the Hero knew that this problem was going to happen but didn’t do anything to stop it?  Why is the Hero hiding knowledge that would help the company?  This is yet another example of the Hero not sharing and not partnering.  How can the Hero just sit idly by and allow this to happen.  Something needs to be done about the Hero …”

And this “something that needs to be done” … and get ready, this is going to make any logical thinking IT engineer’s head spinning … could be as severe as disciplinary action cast upon the Hero.  Why is such an illogical outcome such as the individual that amassed such valuable knowledge to be able to assemble together all the puzzle pieces of the problem become the victim of some disciplinary action?  The answer falls more on the organizational hierarchy than on conventional logic.  If the individual that is uttering those statements about the Hero is significantly high on the organizational chart, then the layers below, who have been focusing on all sorts of other fires, are caught without a good story as to why this situation occurred and why the Hero is not the root of all evil.  Not being armed with a story that shields the Hero, the management layers in between are somewhat constrained and thus the blame lands on the Hero.  For more of the management side of the Hero’s plight, see the articles that cover this in the management section.

Sure, your peers might find you after meetings and give you kudos for standing up for the facts, but is being technically “right” worth the cost of being put through this ancillary pain?

The next article introduces the hybrid approach which I’ve entitled “Play it Safe”

Anyone that has had to participate in a meeting to determine why some IT system went down is echoing a collective groan as they read this title.  For both IT managers and engineers alike, it is the least desired activity following a system failure of any kind.  Business and/or product owners outside of IT are waiting, after the dust settles and the system is restored to working condition, to have primarily two questions answered:

1. Why did the system go down in the first place?
2. What is IT going to do to make sure this doesn’t happen again?

In the previous article, I outlined the business context of the root cause analysis exercise in general and the complexities in clearly and logically arriving at a true root cause for a system outage due to the interconnected players involved.  In this article, I outline a particular IT engineering resource approach to participating in the root cause analysis process.

IT Engineering Participatory Approach A = Surprised and confused

“There is a temporary storage volume?  Really?  What is it used for?  You mean it can fill up?  Wow … I’m surprised!  If someone had told me about the temporary storage volume I would have obviously blah, blah, blah”

What technology? I'm supposed to know about this "technology"?

This approach is synonymous with playing dumb about the whole situation as well as the technology itself that was aligned with the outage.  If you haven’t experienced someone using this technique, you might be thinking to yourself: oh come on, can this really work, doesn’t this just make one come across as inept?  The answer is: absolutely.  So why does this approach based in ineptitude actually work.  I will put forth two arguments based on human thought processes and the pressures of time.

I’ve witnessed time and time again where the immediate reaction to the surprised and confused IT engineer is to get caught up in following that line of reasoning.  People do not by and large enjoy being surprised at work with questions about their performance or work quality or responsiveness, etc.  I am not a sociologist, but in my experience, when experiencing someone else suggesting they were surprised and caught off guard by this work event, they immediately identify with the notion of dreading the feelings associated with being surprised themselves and implicitly support this individual’s claim of surprise and confusion.  Rarely have I witnessed the alternative reaction supersede the previous with: “wait a minute … isn’t it your job to be responsible for health of this IT service and thus why are you surprised at how the service you are responsible for functions?”  If someone indeed starts to go down this logic road, the confused aspect starts to take precedence of the surprise with “well, I obviously would have known and fixed the temporary storage problem if someone had told me about it … but I’m confused, who should have notified me ….”  The perception of the problem again, has been shifted from the individual off to some more nebulous area that suggests there was nothing the individual could have done because of this nebulous third party’s role as a barrier for the individual to do their job.  The confusion can keep growing: “wasn’t there a project that was supposed to fix this alerting problem?  Wasn’t <insert random but related engineering group name here> working on this?  Isn’t there a ticket open with the vendor on this issue?  Wasn’t <insert random IT resource name here> working on a fix for this?”  The larger the organization, the more effort it will take to follow up on each accusation in order to find validity.  And that validity gets more and more difficult as the word spreads that the proverbial witch hunt has begun:

<insert random IT resource name here>: “Working on a fix for that?  Yah, but that got assigned to what’s-his-name in the quality team.  You might want to follow up with whose-his-face in testing services because I think they now are responsible for the quality team.”

So this option sure sounds great since it appears the perception is always some external force that can’t be controlled is restricting one from doing their job.  With the converse being if the restriction didn’t exist, I would have done my job and there wouldn’t have been a problem in the first place.

Finally, for this option to be successful, you have to stay completely away from the problem resolution itself as much as possible.  If you are visibly involved in finding and fixing the problem or if you are poking around systems related to the problem and thus are appearing in audit and history logs, you can lose plausible deniability.  Someone could surface at the least opportune time and reveal your involvement in the resolution process, hence reducing your legitimate claim to surprise and confusion.

My suggestion, don’t pursue this option unless explicitly directed by management with some explanation of what role you are playing in the greater root cause exercise.  Why since it seems such an easy out?  In one word: reputation.  You will quickly be branded inept by your peers and management will see you as weak in the sense you are junior, not worthy of being trusted with an important assignment and finally, not promotion material; the latter being the most difficult to rescind if you have your heart on a different job within the organization.  Ok, so you might be thinking, right now, just staying where I am is just fine.  That might be true right now … but what if some new project comes along or the company purchases some cool new technology that you might want to participate in the near future?  The likelihood that you will get such an opportunity given your propensity to be surprised and confused about your job assignments is exceedingly low.  I will finally venture to say that when the company is facing difficult financial stress and the option of work force reduction seems eminent, guess who falls into the X percent category of people the organization can survive without: those that are perceived as surprised and confused by work.

In the next article I’ll outline the pros and cons associated with “Openly Be the Hero”.

Anyone that has had to participate in a meeting to determine why some IT system went down is echoing a collective groan as they read this title.  For both IT managers and engineers alike, it is the least desired activity following a system failure of any kind.  Business and/or product owners outside of IT are waiting, after the dust settles and the system is restored to working condition, to have primarily two questions answered:

1. Why did the system go down in the first place?
2. What is IT going to do to make sure this doesn’t happen again?

Everything is broken!

Since the business folks have had their otherwise perfectly aligned piles of work to do that doesn’t involve IT completely interrupted by IT, they start the root cause analysis process by contacting that person in IT that represents their “relationship” with IT.  That IT person is usually high enough in the management hierarchy that their need for answers to these two seemingly straight forward questions generates an urgency that has all the IT stakeholders rallying together in camps.  As each camp is forming, one underlying theme prevails: no one wants to be the engineer that broke the system and no manager wants to be responsible for that engineer and thus the breakage itself.

Engineering Perspective:

The last thing an IT engineer wants to hear from his boss is that after being up all night fixing the problem, that he or she has to come into the office to participate in a root cause or root cause analysis meeting to talk about what happened.

Bob the Engineer: “But we all know what really happened … Storage Support was supposed to monitoring the temp storage volume and when it starts to fill up, get someone on the DBA team to start dumping transaction history data … but since Storage Support let the volume fill up, of course the DB transactions are going to halt, which backs up everything in the system and then it crashes!”

Sure, that could very well be an accurate root cause to the problem.  Take a deeper dive into the group suggested to be dropping the proverbial ball on this one: Storage Support could be waiting on Storage Engineering to build/provide/implement a monitoring and alerting solution due to some past recognition that there doesn’t exist a reliable way to alert appropriate folks when a storage volume reaches a certain threshold.  And yet, Storage Engineering is actually actively working on a solution as part of a formal IT project already in flight based on some past disaster that kicked of said “ruggedization” project.  Thus, once “projectified”, the ownership of the delivery of an alerting and monitoring solution for the temporary storage volume is arguably not Storage Support, nor Storage Engineering but rather the more nebulous ruggedization project itself.  Thus, if the ruggedization project has been providing frequent and accurate updates to project sponsors and stakeholders as to the status and ultimate delivery dates of the project itself, one could argue that the project sponsors failed to assign appropriate priority to the ruggedization effort itself.  And since the project sponsors are most likely IT managers the situation becomes increasingly complex in that the logic (or illogic if you may) proposes that IT management, the same people that very well could be getting the phone calls from the business people, are actually the root cause of this hypothetical outage.

Whew … now if you are still reading and haven’t passed out yet, congratulations!  As an engineer caught in the middle of this interconnected web of essentially competing priorities of limited resources, you have a couple different ways to participate.  Each participatory approach has its positives and negatives, thus choose the approach you are most comfortable with to achieve the approaches associated outcome.

The next article will outline some participatory approaches and their associated pros and cons.

I think it is safe to say that anytime any company wants to get something new accomplished they “kick off” this thing called a “project”.  Now if you have spent any time in IT you have probably had a role on one of these so called projects.  You know when you are on a good project:

• Everyone knows what the end goal is.
• Everyone knows each others roles.
• Everyone has a sense that all participants are accountable for their tasks.
• Things are getting done.
• Everyone feels a sense of progress.
• Everyone feels a sense of team.

You also know when you are on a bad project:

• Everyone is seemingly bumping into each other.
• No one can really articulate the goal of the project
• No one knows what the end state is supposed to look like.
• Time is going by with meetings and minutes and status reports but no real tangible work is getting done.

This series of articles will cover different aspects on how to survive as an engineer that has a role to play on an IT project.

In the previous article, the concept of a “pre-task” was introducing a gap in the project plan and introducing gaps into the plan causes panic.  Thus, how can you redirect this panic to work for you?  That is were the ability to introduce a gap or a “pre-task” from the example in the previous article that redirects the panic on to another individual or entity while at the same time, implying the gap should have been already accounted for because everyone knows it is a required tasks works.  Breaking down the previous example:

How does a pre-task help me?

“…I can’t finish until the Architect signs off on my design document” introduces a gap in the plan that doesn’t involve or imply more work on your part.  Rather, it brings in a task that is a barrier for you to complete your task.  Thus, and here is the selfish benefit, this technique allows you to work on your task without someone hovering asking every five minutes … “is it done yet?  How about now, is it done yet?” as project managers tend to do.

“All designs must have Architect sign-off” is the follow-up sentence implying that it is the project manager’s job to know this standard operating procedure or process.  Thus, it isn’t your job and thus you can take no blame for not having this step in the plan from the beginning.

In summary, whether you find yourself with too many chiefs or too many indians, using the tools above should help you succeed in your role on a project.  Now if you have read this far and are thinking to yourself, “geez, this sure sounds like a lot of work above and beyond doing real engineering work.” my response is yes, it is.  And yes, none of it is seemingly appealing to the engineer that just wants to do engineering work.  But, in my experience, I have seen, time and time again, the good natured engineer get himself, his boss and his team in hot water because all the chiefs were pointing the finger of blame squarely on the good natured engineer that became the almighty reason for the project delay.  The finger is pointed regardless of all the prior missteps that get conveniently forgotten in these scenarios or have already been covered up.  To better position yourself not to get caught up in these nontechnical exercises with chiefs, it pays dividends to invest some time understanding what the formal and informal processes are in relation to your project role and get into the practice of always having a pre-task in your back pocket to use when chiefs are starting to dig into the project tasks.

Key points from this series of articles:

• Know your role on the project and ensure your boss is in agreement.
• Always have your boss in 110% support of your performing tasks/functions outside of your traditional role
• Have a working knowledge of the official project and engineer processes leveraged by your company
• Always have a pre-task ready to articulate when ever possible to create that external dependency on any tasks you are performing

I think it is safe to say that anytime any company wants to get something new accomplished they “kick off” this thing called a “project”.  Now if you have spent any time in IT you have probably had a role on one of these so called projects.  You know when you are on a good project:

• Everyone knows what the end goal is.
• Everyone knows each others roles.
• Everyone has a sense that all participants are accountable for their tasks.
• Things are getting done.
• Everyone feels a sense of progress.
• Everyone feels a sense of team.

You also know when you are on a bad project:

• Everyone is seemingly bumping into each other.
• No one can really articulate the goal of the project
• No one knows what the end state is supposed to look like.
• Time is going by with meetings and minutes and status reports but no real tangible work is getting done.

This series of articles will cover different aspects on how to survive as an engineer that has a role to play on an IT project.

In the previous article two concepts were introduced to help you survive.  Now it is time for the most powerful concept, the “pre-task”.  As much as possible at all times, create the perception that you are waiting on something from someone else in the organization.  Make sure in every situation possible that the project manager or development lead or whoever is managing the checklist of project tasks believes someone else needs to complete their task before you can complete your task.

Not having a “pre-task”:

Sally the Project Manager: “So Bob, according to my plan, you should be working on that widget … are you done yet?”

Bob: “Um, yah, I’m working on it but I told everyone it was going to take me five days and this is day two.”

Sally the Project Manager: “Not according to my plan … it says you build the widget and then we start testing.  You are holding up testing.  I’m going to have to escalate!”  And rushes out of your cube before you can formulate a response.

Having a “pre-task” at the ready:

Sally the Project Manager: “So Bob, according to my plan, you should be working on that widget … are you done yet?”

Bob: “Um, yah, I’m working on it but I can’t finish until the Architect signs off on my design document.  All designs must have Architect sign-off.”

Sally the Project Manager: “Um …” <has no clue what that means but now knows there is a delay building> “I better go find our Architect on this project and get him or her involved”.  And again, rushes out of your cube.

Now, in this example, there are multiple issues generating confusion.  One is task duration.  The likelihood that someone actually recorded your need for five days to actually do work is very low.  I am continually amazed that the less technical a person is the greater the likelihood they will assume a technical task takes an amazingly short amount of time.  Another is the propensity for project managers or people whose primary job it is to track other peoples work are quick to react less than rationally when their plan gets messed up.  They have their plan and they remain rational when, as time goes by, the plan remains unchanged and tasks are sequentially marked complete.  But when the plan itself appears to have a gap, panic and irrationality can ensue.

Ok, why does someone whose sole job is to track a checklist loses their marbles when there is a missing tasks or four additional tasks need to be inserted somewhere?  A quick glimpse into the project manager’s world reveals their greatest value to the organization is in their ability to predict the future with their plan.  Based on the future date their plan indicates, all kinds of business-ish things could be set to kick off.  Thus, something as simple as a plan that says when a system upgrade will be deployed could also be the starting point for a marketing team to formally kick off a set of media ads, plus a customer training/education packet could be already in the mail with dates for when each customer will be forced to use the new upgrade system, plus an accounting group could be poised to show a drop in license costs (i.e., save big money) due to the upgrade, plus, some executive is counting on the license money savings making his departments balance sheet come just under a magical finance number that triggers a quarterly bonus.  Hence, when the upgrade implementation date starts slipping from a failure of the future predictor (aka. project manager) being accurate, all kinds of heat from all kinds of folks downstream from any of the technical work associated with the project can come out of the woodwork and come down hard on the project manager for giving bad data.  Hence, fearing such heat, project managers react in a variety of ways and some of which are not particularly 100% rational.

In the next article, I’ll outline how to use this panic to work for you.

I think it is safe to say that anytime any company wants to get something new accomplished they “kick off” this thing called a “project”.  Now if you have spent any time in IT you have probably had a role on one of these so called projects.  You know when you are on a good project:

• Everyone knows what the end goal is.
• Everyone knows each others roles.
• Everyone has a sense that all participants are accountable for their tasks.
• Things are getting done.
• Everyone feels a sense of progress.
• Everyone feels a sense of team.

You also know when you are on a bad project:

• Everyone is seemingly bumping into each other.
• No one can really articulate the goal of the project
• No one knows what the end state is supposed to look like.
• Time is going by with meetings and minutes and status reports but no real tangible work is getting done.

This series of articles will cover different aspects on how to survive as an engineer that has a role to play on an IT project.  The previous article can be found here.

How to survive with too many chiefs or too many indians

First and foremost, make sure your role is clear on the project.  Make sure at all times you are executing the functions associated with your role.  If you are unsure if your role requires you to perform a certain function, ask around to confirm and if you aren’t getting concrete yes or no answers, confirm with your boss.  It might seem like you are annoying or bothering your boss but believe me, your boss would rather answer a 30 second role clarification question than sit in hours of meetings dancing around why someone on his team did or didn’t do a particular function on a project.

Second, if you have opportunity or are asked to perform a function outside the bounds of your role, consider all the angles before just up and completing the function.  It might seem cool to build the widget that connects the two systems together to allow the transactions to flow, but if that is not your explicit role, you might be putting your boss and your team in jeopardy.  How so?  Well, if you team is a support rather than a development group in the organization, you have just forced your boss to have more ownership in the system than he has been charged to have.  Your boss will be in political hot water when you are pulled back into the project make changes to the widget while other systems your boss is responsible for need coverage and you can’t be in both places at once.  The last thing you want to be doing is creating a headache for your boss when your boss has the most direct influence on your job duties and compensation.

Not all functions outside of your role are filled with danger.  Some are filled with opportunity for praise for helping the project move forward.  In the example above, there may not be a role that is supposed to build this widget.  Maybe the project is plugging all the technology together in order to test the final solution and someone didn’t realize these two systems needed a widget.  You could be the engineer that saves the proverbial day.  When everyone is patting each others backs when the project is successfully implemented and everyone is overly positive, someone could exclaim: “Wow, glad Bob pulled the widget out of his ear at the last moment, we thought we had a big delay on our hands!”  Yes, these moments are few and far between, but they do exist, they are fleeting, so enjoy them for the brief moment they do exist.  The key to pulling this off successfully, starting even before building a working widget, is to make 110% sure your boss is completely onboard with what you are proposing to do outside your role and the risks associated.  Be prepared for your boss to agree that you are completely capable of building the miracle widget, but because of company politics that if you tried to put them together in your mind would make you pass out, he asks you not to build the widget.  Don’t get discouraged if this happens.  There will be many projects and many opportunities to stretch your role and position yourself for exceeding expectations.  Learn from these rejections on how to better sell your boss on your ideas for the future.

In the next part of the series I’ll continue cover how to survive with too many chiefs or too many indians with one of the most powerful tools the “pre-task”.