Latest Blog Posts / Page 4

Protecting yourself from your OEM.

Quite a few calls we get are from people who are getting the shaft from their OEM, which is not to say that there are not good OEMs out there, just that these people aren’t working with them. Often times at least part of the fault lies with the client as well. So lets look at the OEM/Client relationship and what the major fault is.

1. The OEM wants to sell the client something.

2. Once the client buys it, the OEM is either done, or wants to sell the client something.

The second proposition is the problem. Once you have the equipment in your plant, the OEM can only make more money off of you by selling you something else. And they already know that you have money (bought their machine, didn’t you?) and what you have. So that makes you low hanging fruit.

3. The OEM engineering department and Bill of Materials change over time.

So the guy who integrated 10 years of their machines with Siemens has retired (or been run over) and the new guy likes AB, or ABB. His solution to fix both proposition 2 and proposition 3 is to figure that any problem with the old stuff means that the client needs a complete control system update. The upper management likes him because he is monetizing old clients and he enjoys putting in the new stuff. The problem hits when the old clients don’t want to pay 10k or 100k or 1m to upgrade their machine. But that’s not a problem, because if they don’t, that means that they don’t have any money and the OEM shouldn’t be wasting their time on them. Or they need a new machine. Either way, it’s a solution of sorts.

So as a client, how do you protect yourself?

1. Backups. Yes, we are back the “Known Good Backup” again. If you have the program and parameters, we can get you going again. If not, you are out of business if the OEM doesn’t have a copy and is willing to give it to you (or sell it, see #2)

2. Get someone who knows the equipment. Yes, it will cost money. It will, however, be cheap. I am working right now with a customer who’s maintenance guy pulled the battery on a DLC card. The program for the DLC evaporated while the maintenance guy was checking the battery with a meter. The OEM thinks that a system upgrade is in order and is not being helpful. Oh, and get some smarter maintenance guys too. That is another thing that will pay in the long run. I have had customers pay for a service call to get a known good backup and a survey of what they need for long term operations. While they have whined about the cost at the time, I have never had one come back to me 10 or 20 years later and say “Ya know, that money I spent was a complete waste”. Usually, it is “Boy, am I glad I got you guys in here” about the 4th time we saved their butt.

3. Don’t take the word of the guy trying to sell you an upgrade if the equipment is obsolete or not. Do you take the word of the used car salesman about “Driven by a little old lady…”? I have had people start off conversations with “I know this is obsolete, but” at which point I cut them off and tell them to forget everything the person that told them this is obsolete told them, because it is probably all as wrong as that statement is.

4. Spares. Look at what I recommend as spares. Cables are cheap and they are the first spares you should buy…but after the backup.


Sometimes, Service Calls are unavoidable

My main purpose in life is to avoid service calls. To a certain extent, Service Calls are indicative of failure. That can either be my failure to be able to properly aid the customer in the resolution of their problem, or the customers failure to have any sore of technical knowhow in the bank for when disaster strikes.

For your consideration:

Get a call from a OEM first (a bad sign) and then the customer on a packaging machine. The information from the OEM was a bit sparse, they did say “Battery Error”. With the code that they gave me on the drives, this sounds like a standard “The customer needs to replace the battery and then re-reference the machine” problem. So then the customer calls and things get worse really quick. They have 4 drives with the “Battery Error” on them, and the machine control is also showing that the battery probably failed, which means the main control program has evaporated. This is why I want everyone to have “Known Good Backups” available. Amazingly enough, the OEM is able to email them a copy of the machine control program, and they evidently left a copy of the programing software on the PC interface.

Which leads to the second problem. Nobody knows how to reload the program. Nobody has even opened the software before. As this is German software, the learning curve can be considered a learning cliff face, very nearly vertical. German software is much less forgiving as well, it will allow you to erase the control and the copy of the program without one of the Microsoft “Warning, this will format and erase all information, Proceed Y/N?” questions. German software figures that if you were man enough to open the program, you should be able to live with the results. If you didn’t know what you were doing, why did you open up the software in the first place?

While the entire plant is down is not really the time to try to teach someone how to use a highly complex program…over the phone…with the student having no real knowledge of what they are doing at all. This is one of the things that has changed over the years. When I was Engineering Manager at a large automotive Belt plant, all of the maintenance people (and Engineering staff) went through a 3 week training course on the controls we were using, with a few of us picking up an additional 3 weeks of advanced programming instruction. When we got a new person, they had to go through the same thing. Now it seems like companies would rather pay someone to come in and fix it rather than building their own people up to a level that they can handle larger problems. In this case, it means that the company got to pay weekend rates, plus at least 4 days of downtime. The weekend rates alone would have paid for one of their employees to go to class, and someone who had been to class would have understood what was going on long before it got to the point of bitbarfing the program. Hopefully, they would have also made a “Known Good Backup”.

So, the service call goes out. My failure of not being able to explain how to load the program up (actually 9 different programs) and the customer’s failure to have his people prepared for the problem. The cost though, is entirely on the customer.

You gonna have to spend some money to stay in the game.

My customers range from Fortune 100 Companies to Mom and Pop shops. I actually prefer helping the smaller companies, given a choice (I am not given a choice), they are more appreciative, keep their accounts current and are actually happy when I am able to keep them from sending everyone home for the day.  However, the one thing that is invariable from small to big companies is that the Second Law of Thermodynamics applies to everyone. In simple terms, the Second Law states that the instant you stop putting something together, it starts coming apart. That holds true from galaxies to servo drives.

The corollary to this is that the way you keep things from falling apart is by putting them back together again. That can be from tightening a nut (“They do not preach that their God will rouse them a little before the nuts work loose.” -Rudyard Kipling “The Sons of Martha”) to being sure the drive doesn’t have dust filling it and preventing cooling. It also means replacing things when they break. That costs money. We work with our clients to be sure that the money they spend is the most effective. Sometimes, however, we are unable to save them from themselves. Offered for your consideration:

The client that decided the factory battery with the wiring harness was too expensive, and went out and bought his own battery, disconnected the old factory battery from the unit, and very carefully cut the old wiring harness off and soldered it to the new battery. I congratulated him on his thriftiness and speed, as once he disconnected the battery, he had 90 seconds to do all that before his PROGRAM EVAPORATED. Since he didn’t have a Known Good Backup (or any backup at all), and the OEM was out of business, they wound up selling the machine for scrap.

The client who lost the display on his power supply and decided that he could connect the display (and boards) behind the drive to the power supply. He was quite surprised when he blew both units.

The client who decided to put an Ebay purchased unit in and it blew up the entire string of DKC drives. That drive cost him 60k in blown drives, not to count the downtime and lost orders. They did stay in business long enough to pay the bill to us, but not long enough to become a longer term customer…

We will work with you to make sure you use the most cost effective answers. We will help in any way we can. You have to listen though, we can’t save every customer from themselves.

Time for Dial-a-Prayer

So, had a late start to the morning the other day (or early, depending on how you look at it) and was setting at breakfast with Mrs Dr ThinkingEngineer, and I got a call on support. Since my headset was giving me fits, I took the call on speaker (had the phone forwarded to my cell) and had the following conversation.

Me: “Hyperdyne Systems, how can we help you?”

Person of Interest: “I have a problem with a power supply.”

Me: “Ok, what type of power supply?” Here I am looking for a type code or family so I know what I am dealing with.

Person of Interest: “I don’t know”

Mrs Dr ThinkingEnginner: “Raised Eyebrow” (I should note that after teaching ChemEng Students both Thermodynamics and Lab Classes, she is relatively immune to “the dog ate my homework”, but some of my calls are still surprising to her).

Me: Ok, what sort of problem are you having? What sort of drives are attached to the power supply?” An Error code will narrow down the family of Power Supplies he might have, as will knowing what drive is attached to it.

Person of Interest: “I don’t know”

At this point Mrs Dr ThinkingEngineer managed to get a napkin up to keep from cupspraying her morning coffee all over me. I am at a severe loss, he doesn’t know what kind of unit he has, and he doesn’t know what code is on the front of it. Despite the fact this call is going down in flames, I try one more time before yanking the ejection handle…

Me: “What does the display on the front of either the drive or Power supply read?”

Person of Interest: “They are both blank”

Me: “Ok, I don’t think I can help you…”

I can do many things, I have been handling this product line for over 30 years now, and I have been doing troubleshooting (both in the field and on the phone) all that time. But you gotta give me something I can use…

An expert is anyone from farther than 90 miles away

I have calls that irritate me. That’s bad, because I take between 50 and 90 calls a day, and unlike my father, tend to stew a bit. A bad call in the morning can leave me short until lunch, and two bad calls can mess up the whole day, leaving me short with people who actually do have problems and both deserve and need help.  Hey, I’m human.

One of the ones that irritates me is the guy who already knows everything (so why exactly, are you calling for tech support?). But the ones that Quick Fry me to a Crackly Crunch (as opposed to Baked to a Delicate Crunch) are “experts”. We give tech support for free (over the phone, if you want me to show up in person, that’s possible, but it will cost you), so when someone who is charging the customer calls me for tech support, I got to think his client probably doesn’t think he is paying hourly rate for on the job training. Let me tell you about a call I got last week.

Got a call from an “Integrator” looking to put a new PLC control on a machine, but keep the older servo drives. From my days in System Integration, I know an “Integrator” can be anyone from an actual engineer with a degree and license to a kid out of High School with a bandit copy of PLC programing software. Most of my last days in Sys Int were cleaning up the messes of the later types. Anyway, the guy starts off telling me how great the PLC is that he is putting on here but how the customer wants to keep his antique drives and motors. How it is so going to bring him down to have to put his new stuff on, and then hook it up to these old and dirty things…

Oh, and how can he do that?

The drives he is talking about aren’t that old, and they have discrete inputs for triggering Motion Blocks. However, he needs to be able to move block commands from his shiny new touchscreen to the drives, through the PLC. Could I tell him how to do this?

So, now I know.

1. He has never done this before.

2 He doesn’t know how to set up serial communication functionality on the PLC, making him more the second type of Integrator than the first.

3. He knows nothing about Motion Control.

4. He doesn’t (and I asked) have a manual.

Fortunately, I can send him to a different place (corporate Tech Support) where they get paid for this.  But I got to wonder if the company that hired him did any vetting of him or his company first, just hired the first person that was recommended to them (yeah, my Cousin Vinny can help) or just picked him up off the street.

And yes, I was short the rest of the day…

No really, where is your backup?

I have a whole number of posts and articles and campaign stickers about backups. It doesn’t take much to get me up on a soapbox in the middle of the town square, declaiming in iambic pentameter the virtues of a known good backup, and the evils of stale or non-existent ones. So, as I was coming back from saving the day again (why is there never a phone booth around anymore?) when I got a blast from the past. Or at least a phone call about it.

I used to do some system integration work in the old days. So I got a call about a machine that I had done the controls on. Was actually a system that I was quite proud of, it had a custom C coded front end, some specific communication DLL links and a large part storage file system. Was positively cutting edge FOR 1997. Haven’t seen or heard of it since…until now. The PC (Running WinNT) hard drive finally gave up the ghost, and there is no backup available on site. So they called me. Do I remember it? Sure. Did I make a backup? Absolutely, burned it onto a DVD and put one in the cabinet of the machine, a full Norton Ghost image. Only problem is that the DVD isn’t there anymore (color me shocked) and there is no other backup. Did I keep one? Yes, for about 15 years or so. So I start digging. I actually find the original programs and C compiled program, but trying to put that all back together for this company is going to cost a staggering amount. More than the machine is worth, I would bet.  I believe my initial recommendation is going to be for them to send the HD out for reconstruction. At a couple of K, it will be cheaper than paying me T&M to relearn what my 1997 self knew intuitively.

Oh, and make a backup.

Buying Beer on the 4th of July, 1984

So I had thought I might fly today, and went up to the airport. Low cloud deck meant that I got to spend an hour updating the maps in the avionics and not much else. On the way back home, I was instructed by “She who must be obeyed” to buy some brats and some beer to boil them in. So, I stopped at the local Springdale, AR Harps store. picked up the brats, some buns and some cheap beer to boil them in.

Welcome to 1984…

When I reached the checkout and got ready to pay, the girl asked for my ID. Being 54 and looking most of that age if not older, I muttered something about fishing for a tip and pulled out my wallet to display my license. I was then astounded when she asked me to pull the license out so she could scan it. Since I was a bit pressed for time I did so, but on the way home I kept thinking about the Orwellian implications.

1. What does Harp’s want with my license info? Name, address, phone number, DOB, that I am an organ donor and who knows what else. This is info that most places would be happy to pay for, yet I had to give it up just to pay THEM for a bottle of cheap beer.

2. On doing some research, I was unable to find a “Confidentiality of Driver’s License” Statute in Arkansas. In California, for instance, businesses cannot retain information like this that they (supposedly) scan for validation of DOB. Like, can the kids not do math anymore? Actually that may be a problem, but that is why they have the little calendars that say “Don’t even think of buying liquor unless you were born before xx”.

3. Is the fact that I bought beer tied to my AMEX card that I used to check out? My Insurance? What happens to the information? Why is it that Harp’s scans DLs when other places (including Liquor Stores) don’t?

Perhaps I am overthinking it. But the 4th of July is supposed to celebrate our freedom, and I got stuck showing my papers to a computer instead. Not going to be buying beer at Harp’s anymore, that’s for damn sure….

You had WHO try to repair this??

I have a couple of companies I love to hear on the phone. Not because I enjoy talking to them, but because they have company-wide agreements to let a large automation vendor repair all of their automation and servo parts. For the equipment we carry, this is a lot like taking a Mercedes to a Chevy dealer and then wondering why it doesn’t work afterwards (actually, I think most Chevrolet dealers would just turn you around, unlike this major automation vendor).

So why do I like these companies so much? Because they have 4 drives that have been “repaired” on the shelf. They have a drive fail, and none of the 4 “repaired” drives work. They then call us for emergency exchange units, that are typically flown out on charters (I have piloted a couple of these deliveries myself) for the highest prices we charge. We then will have the discussion about how the “Large automation vendor” can’t actually get the parts to repair these units, as the OEM doesn’t sell to ANYONE, and how it would be much better if they would go for factory repairs. They agree they will start sending out stuff to us for repair. However, then the company-wide repair agreement comes into play and they can’t send the stuff out.

So, in a month’s time, we do it again. and again. and again.

I may buy a new airplane….

Why exactly don’t you have a manual again?

I get a disturbing number of calls from customers who are trying to troubleshoot a very expensive piece of equipment that can be easily damaged by doing the wrong thing, without the manuals for the equipment. Coming from an engineering and aviation background, I find this astounding. For an aircraft mechanic, it is actually against the law (you can and will be fined, or worse if audited), to work on an aircraft without having a complete set of manuals on the premises. That is even true for painting an aircraft (mine was in hock at the paint shop for a couple of weeks because they didn’t have the manuals). I actually had to carry my personal set down to them so they could show them to the FAA before they would let them proceed on work.

So when I get a call from someone who is trying to troubleshoot, or even better, trying to get an entire plant going as people are being sent home without pay because of downtime, and they don’t have the manual that lists the error codes, I am amazed and upset. Manuals are easy to get. Our company websites have buttons to press to get to forms that are sent in to have manuals emailed to people. The corporate websites have downloadable manuals (admittedly, in a labyrinthine setup that makes you appreciate the Dewey Decimal System), and if you do a search on the web, you can typically find a PDF that shows up in the first few lines of Google or Bing.  So, manuals, the single most important thing you can have when troubleshooting a drive, are literally just a few keystrokes away. Anyone who doesn’t have one should reconsider their priorities.  Anyone supervising someone who doesn’t have one should perhaps consider it part of their job performance…or lack thereof.



Those durn meetings…

When someone asks me what I miss least about the corporate world, I can answer without a missed beat.


While I can safely say that Mrs ThinkingEngineer had many more at her last job than I did, I can say that they were the bane of my existence. The worst was the “Morning Staff Meeting”,  a meeting of department heads to see what the problems and issues of the day were. There were, however, more than a few problems with this concept.

1) At 9am, most of the day’s problems and issues hadn’t sprung up yet. So, the meeting became a “What happened last night that we missed meeting”.

2) The time of the meeting dropped it right into the period that you had figured out what had broken, and were formulating and implementing plans to fix them. This brought things to a screeching halt, while you listened to other Dept heads talking about personnel issues and such.

3) There are Dept heads that actually think their meeting performance (read talking time) actually is their job performance. Unless you have had a catastrophe, you should be able to summarize in under 5 minutes.

I did, however, find a solution. A thinking engineer solution. As Engineering Manager I had many of the machine and building controls accessible from my computer, including environmental controls. When the tired A/C in the conference room went out, I had a new more efficient and LARGER system put in, with very modern controls.

And every day as I walked into the morning meeting those modern controls did their duty, and turned the conference room temp down to 40 degrees.

Problem Solved.

« Newer Posts Older Posts »