As the school year starts, I think a bit of a commentary on school quality needs to be made. Schools are in a bit of a pickle these days, and not just because of rising tuition. I am familiar with a couple of schools, but let me rail on one. The school shall remain unnamed, to protect the guilty.

There are, in my opinion, only two factors in the quality of an Engineering School. The quality of the faculty, and the size of the classes. The quality of the faculty is difficult to judge from the outside, but certain assumptions can be made. Young faculty (under 30) isn’t going to have much actual experience teaching, so the students are going to get to learn while the professor does. That may not be the best environment. Watch out for a number of classes being taught by grad-students, or post docs. These can be told usually by the instructor being listed as “Dr Smith” as opposed to “Professor Smith”.  These folks are usually on their first or second class. In a reasonable environment, they will be given notes and previous tests to look at to work up their own material. The lazy ones will just take a problem from each of the previous tests, leaving people with access to previous tests (Greek Houses have extensive test files, although often not of engineering classes) with an advantage. Smart ones will take a problem from each of the previous test and make a subtle change to it, leaving people who have memorized previous tests running off a cliff like the coyote in the Road Runner cartoons.

Famous faculty is also problematic. A good friend of mine told me of his days at MIT, where he took a class with Prof. B****, of stereo fame. The only problem was he saw Prof B, one time on the first day of class, and after that, the class was taught by a post-doc. There are at least 3 professors that I know of at a Land-grant school who have paid out for teaching so they could do research. Herein lies a problem. If you took a job at a land grant school, you should know you aren’t going to be able to devote your life to research. You are gonna have to teach. If you wanted to be a researcher full time, you should have gone to Sandia or one of the other national labs. Don’t get me wrong, I have had great professors at land grant colleges. They can, however, be overwhelmed by…

CLASS SIZE

You want to know how good your school is? How big is Thermodynamics I course? If the answer is under 40, you are in good shape. If the answer is under 50, you are ok if it is taught by a real Professor with tenure (hopefully). If the answer is over 50, you have a problem. If the answer is over 70, you need to find a new school, no matter who is teaching it.

So I know a land grant school, with new stadiums and a great sports history. They just scraped past ABET accreditation, so tightly that the student body knew how tight it was. The website says the average class size is 24 in one place (but says “hard to pinpoint” in another). The Thermo I class size was over 80 the last semester, taught by a first timer. If you look at education in “bang for the buck” you are getting a wet firecracker here.  What sort of bang are you getting for your buck?

This is a question that I get all the time, related to the “What is Obsolete” question. There are 3 situations where you upgrade motion control (or control systems in general).

1. Absolute Obsolescence.  The controls are no longer available, no longer repairable and the machine is still required. This is a no-brainer. If the control goes down at this point you are dead in the water until the upgrade is done. Wouldn’t you really rather schedule that instead? However, be sure you know the control is actually Absolutely Obsolete, don’t take a used car salesman’s word.

2. Performance Requirement Change. The controls will not perform to a change in specification, or production. This is different from the controls can’t match their previous performance, which may mean a control issue (or mechanical one). Beware speed upgrades, as often times the physical machinery cannot handle the speed increase that production would like.

3. Economic Obsolescence. This one is tricky. I get a lot of “That’s too expensive, I will just upgrade to something newer” in my job. This statement is typically made by someone who has no idea either of what the current system is doing (I had one guy insist that the job that a 3 axis servo system was doing could be done by 3 pwm drives, his replacement called me to get the drives repaired and refitted into the machine), or who has given no thought to the requirements of the replacement. A fine example is TDM series drives. The drives and motors are still repairable, but are expensive to get done. A retrofit looks easy, as the TDM drives take a standard +/-10v servo signal for control, but it turns out that the units also are typically powered by 230 volt, and the motors cannot be run by anything else. So, you are ripping out the drives, motors and cabling, rewiring the cabinet for 480 and reworking the control system to accept the new drive’s ready and enable setups. I have done this in a couple of days with a good electrical crew, but I have been doing this for 30 years too.

The other thing to think of is that the best time to upgrade your servo system is as late in the life of the system as possible. This makes sure of two things. One, you get the money out of the old system, and two, you get as new of a system as possible. Servo systems families typically change every 7 to 10 years, depending on the manufacturer. Upgrading at the year before the entire family change takes place means that a year from now you will have the non-current system. It is a lot like buying computers. This week’s star is next weeks dog, so wait as long as possible and then plant your flag, knowing that soon this too will be a legacy system.

I have a seemingly endless parade of clients who come to me with the first words out of their mouth being “I know this is obsolete, but…” Usually I stop them right there as they have been told that something is obsolete by a competitor trying to get them to buy something. Lying to a client is a particularly short term strategy, I have clients that have been such for 25 years, and they didn’t get that way by me being wrong all the time.

So lets define obsolescence. We can split this into three types, Functional, Economic, and Absolute. Functional obsolescence is what my plane is. It was built in 1979, and has not been built since then. There are parts available for it (a couple of years ago someone built a new one out of new surplus stock). The engine, propeller, wheels, instruments are all built by 3rd party companies and should be available for the foreseeable future. Until the life limited parts time out (which will be past my flying days at the current rate) the plane will be maintainable at a cost comparable to a new airplane of the same class (in some cases cheaper).  Functional obsolescence means that while you can’t get one new, it makes sense to continue to put parts into the existing one. A 2-3 year old car is another example.

Economic obsolescence means that while parts or repairs are still available, it doesn’t make economic sense to continue to keep pumping money into the equipment. The average light truck in the US reaches this point at about the 16 year mark. At that point, enough things start going wrong that pumping money into it no longer makes sense. Why then, are their older trucks on the road? Well, sometimes there isn’t enough time to go buy a new one when you are needing to get something done right then. This is why a number of my clients keep running older equipment, the higher cost of the repairs is nothing compared to the cost of being shut down while an upgrade is taking place. Sometimes the capital isn’t available, but repair money is (1M to upgrade vs 10k to repair). Sometimes there are other issues. This doesn’t meant that the clients are wrong to  repair vs upgrade either. A decision that looks uneconomic in a vacuum (ie, the cost of repair of this part is higher than the cost of upgrading to something else) looks quite reasonable in a production environment (ie the cost of repair of this will be 12k, which is the same as upgrading, but we will loose 12M in production while the upgrade takes place).

Absolute obsolescence is just that. You can’t buy new, you can’t get parts or repairs. Stick a fork in it, it’s done. It’s dead, Jim.

So, as the person responsible for getting product out the door, what do you do? Equipment in Functional obsolescence often times normal in a plant. You can get parts, or repairs, but buying new is no longer an option. This is not something to panic about, or even be worried about, just something to keep in the back of your mind.  Economic obsolescence also is a normal state, most OEMs raise the prices on equipment as it gets older. It reflects the higher cost of building equipment at non-production rates. People complain about not being able to buy a ’65 Mustang still. Leaving out the improvements in cars since then (airbags anyone?), can you imagine the cost of a Mustang if Ford agreed to build you just one? You’re gonna need a bigger check. So older replacement parts will cost more. That doesn’t mean it’s time to replace the machine unless that starts getting onerous. However, what you do have to watch out for is Economic obsolescence turning into Absolute obsolescence. That means you crash into a wall and are now scrambling to upgrade or replace while loosing 1M an hour. Or that your successor is doing the scrambling while you clean out your desk…

Quite a few calls we get are from people who are getting the shaft from their OEM, which is not to say that there are not good OEMs out there, just that these people aren’t working with them. Often times at least part of the fault lies with the client as well. So lets look at the OEM/Client relationship and what the major fault is.

1. The OEM wants to sell the client something.

2. Once the client buys it, the OEM is either done, or wants to sell the client something.

The second proposition is the problem. Once you have the equipment in your plant, the OEM can only make more money off of you by selling you something else. And they already know that you have money (bought their machine, didn’t you?) and what you have. So that makes you low hanging fruit.

3. The OEM engineering department and Bill of Materials change over time.

So the guy who integrated 10 years of their machines with Siemens has retired (or been run over) and the new guy likes AB, or ABB. His solution to fix both proposition 2 and proposition 3 is to figure that any problem with the old stuff means that the client needs a complete control system update. The upper management likes him because he is monetizing old clients and he enjoys putting in the new stuff. The problem hits when the old clients don’t want to pay 10k or 100k or 1m to upgrade their machine. But that’s not a problem, because if they don’t, that means that they don’t have any money and the OEM shouldn’t be wasting their time on them. Or they need a new machine. Either way, it’s a solution of sorts.

So as a client, how do you protect yourself?

1. Backups. Yes, we are back the “Known Good Backup” again. If you have the program and parameters, we can get you going again. If not, you are out of business if the OEM doesn’t have a copy and is willing to give it to you (or sell it, see #2)

2. Get someone who knows the equipment. Yes, it will cost money. It will, however, be cheap. I am working right now with a customer who’s maintenance guy pulled the battery on a DLC card. The program for the DLC evaporated while the maintenance guy was checking the battery with a meter. The OEM thinks that a system upgrade is in order and is not being helpful. Oh, and get some smarter maintenance guys too. That is another thing that will pay in the long run. I have had customers pay for a service call to get a known good backup and a survey of what they need for long term operations. While they have whined about the cost at the time, I have never had one come back to me 10 or 20 years later and say “Ya know, that money I spent was a complete waste”. Usually, it is “Boy, am I glad I got you guys in here” about the 4th time we saved their butt.

3. Don’t take the word of the guy trying to sell you an upgrade if the equipment is obsolete or not. Do you take the word of the used car salesman about “Driven by a little old lady…”? I have had people start off conversations with “I know this is obsolete, but” at which point I cut them off and tell them to forget everything the person that told them this is obsolete told them, because it is probably all as wrong as that statement is.

4. Spares. Look at what I recommend as spares. Cables are cheap and they are the first spares you should buy…but after the backup.

My main purpose in life is to avoid service calls. To a certain extent, Service Calls are indicative of failure. That can either be my failure to be able to properly aid the customer in the resolution of their problem, or the customers failure to have any sore of technical knowhow in the bank for when disaster strikes.

For your consideration:

Get a call from a OEM first (a bad sign) and then the customer on a packaging machine. The information from the OEM was a bit sparse, they did say “Battery Error”. With the code that they gave me on the drives, this sounds like a standard “The customer needs to replace the battery and then re-reference the machine” problem. So then the customer calls and things get worse really quick. They have 4 drives with the “Battery Error” on them, and the machine control is also showing that the battery probably failed, which means the main control program has evaporated. This is why I want everyone to have “Known Good Backups” available. Amazingly enough, the OEM is able to email them a copy of the machine control program, and they evidently left a copy of the programing software on the PC interface.

Which leads to the second problem. Nobody knows how to reload the program. Nobody has even opened the software before. As this is German software, the learning curve can be considered a learning cliff face, very nearly vertical. German software is much less forgiving as well, it will allow you to erase the control and the copy of the program without one of the Microsoft “Warning, this will format and erase all information, Proceed Y/N?” questions. German software figures that if you were man enough to open the program, you should be able to live with the results. If you didn’t know what you were doing, why did you open up the software in the first place?

While the entire plant is down is not really the time to try to teach someone how to use a highly complex program…over the phone…with the student having no real knowledge of what they are doing at all. This is one of the things that has changed over the years. When I was Engineering Manager at a large automotive Belt plant, all of the maintenance people (and Engineering staff) went through a 3 week training course on the controls we were using, with a few of us picking up an additional 3 weeks of advanced programming instruction. When we got a new person, they had to go through the same thing. Now it seems like companies would rather pay someone to come in and fix it rather than building their own people up to a level that they can handle larger problems. In this case, it means that the company got to pay weekend rates, plus at least 4 days of downtime. The weekend rates alone would have paid for one of their employees to go to class, and someone who had been to class would have understood what was going on long before it got to the point of bitbarfing the program. Hopefully, they would have also made a “Known Good Backup”.

So, the service call goes out. My failure of not being able to explain how to load the program up (actually 9 different programs) and the customer’s failure to have his people prepared for the problem. The cost though, is entirely on the customer.

My customers range from Fortune 100 Companies to Mom and Pop shops. I actually prefer helping the smaller companies, given a choice (I am not given a choice), they are more appreciative, keep their accounts current and are actually happy when I am able to keep them from sending everyone home for the day.  However, the one thing that is invariable from small to big companies is that the Second Law of Thermodynamics applies to everyone. In simple terms, the Second Law states that the instant you stop putting something together, it starts coming apart. That holds true from galaxies to servo drives.

The corollary to this is that the way you keep things from falling apart is by putting them back together again. That can be from tightening a nut (“They do not preach that their God will rouse them a little before the nuts work loose.” -Rudyard Kipling “The Sons of Martha”) to being sure the drive doesn’t have dust filling it and preventing cooling. It also means replacing things when they break. That costs money. We work with our clients to be sure that the money they spend is the most effective. Sometimes, however, we are unable to save them from themselves. Offered for your consideration:

The client that decided the factory battery with the wiring harness was too expensive, and went out and bought his own battery, disconnected the old factory battery from the unit, and very carefully cut the old wiring harness off and soldered it to the new battery. I congratulated him on his thriftiness and speed, as once he disconnected the battery, he had 90 seconds to do all that before his PROGRAM EVAPORATED. Since he didn’t have a Known Good Backup (or any backup at all), and the OEM was out of business, they wound up selling the machine for scrap.

The client who lost the display on his power supply and decided that he could connect the display (and boards) behind the drive to the power supply. He was quite surprised when he blew both units.

The client who decided to put an Ebay purchased unit in and it blew up the entire string of DKC drives. That drive cost him 60k in blown drives, not to count the downtime and lost orders. They did stay in business long enough to pay the bill to us, but not long enough to become a longer term customer…

We will work with you to make sure you use the most cost effective answers. We will help in any way we can. You have to listen though, we can’t save every customer from themselves.

So, had a late start to the morning the other day (or early, depending on how you look at it) and was setting at breakfast with Mrs Dr ThinkingEngineer, and I got a call on support. Since my headset was giving me fits, I took the call on speaker (had the phone forwarded to my cell) and had the following conversation.

Me: “Hyperdyne Systems, how can we help you?”

Person of Interest: “I have a problem with a power supply.”

Me: “Ok, what type of power supply?” Here I am looking for a type code or family so I know what I am dealing with.

Person of Interest: “I don’t know”

Mrs Dr ThinkingEnginner: “Raised Eyebrow” (I should note that after teaching ChemEng Students both Thermodynamics and Lab Classes, she is relatively immune to “the dog ate my homework”, but some of my calls are still surprising to her).

Me: Ok, what sort of problem are you having? What sort of drives are attached to the power supply?” An Error code will narrow down the family of Power Supplies he might have, as will knowing what drive is attached to it.

Person of Interest: “I don’t know”

At this point Mrs Dr ThinkingEngineer managed to get a napkin up to keep from cupspraying her morning coffee all over me. I am at a severe loss, he doesn’t know what kind of unit he has, and he doesn’t know what code is on the front of it. Despite the fact this call is going down in flames, I try one more time before yanking the ejection handle…

Me: “What does the display on the front of either the drive or Power supply read?”

Person of Interest: “They are both blank”

Me: “Ok, I don’t think I can help you…”

I can do many things, I have been handling this product line for over 30 years now, and I have been doing troubleshooting (both in the field and on the phone) all that time. But you gotta give me something I can use…

I have calls that irritate me. That’s bad, because I take between 50 and 90 calls a day, and unlike my father, tend to stew a bit. A bad call in the morning can leave me short until lunch, and two bad calls can mess up the whole day, leaving me short with people who actually do have problems and both deserve and need help.  Hey, I’m human.

One of the ones that irritates me is the guy who already knows everything (so why exactly, are you calling for tech support?). But the ones that Quick Fry me to a Crackly Crunch (as opposed to Baked to a Delicate Crunch) are “experts”. We give tech support for free (over the phone, if you want me to show up in person, that’s possible, but it will cost you), so when someone who is charging the customer calls me for tech support, I got to think his client probably doesn’t think he is paying hourly rate for on the job training. Let me tell you about a call I got last week.

Got a call from an “Integrator” looking to put a new PLC control on a machine, but keep the older servo drives. From my days in System Integration, I know an “Integrator” can be anyone from an actual engineer with a degree and license to a kid out of High School with a bandit copy of PLC programing software. Most of my last days in Sys Int were cleaning up the messes of the later types. Anyway, the guy starts off telling me how great the PLC is that he is putting on here but how the customer wants to keep his antique drives and motors. How it is so going to bring him down to have to put his new stuff on, and then hook it up to these old and dirty things…

Oh, and how can he do that?

The drives he is talking about aren’t that old, and they have discrete inputs for triggering Motion Blocks. However, he needs to be able to move block commands from his shiny new touchscreen to the drives, through the PLC. Could I tell him how to do this?

So, now I know.

1. He has never done this before.

2 He doesn’t know how to set up serial communication functionality on the PLC, making him more the second type of Integrator than the first.

3. He knows nothing about Motion Control.

4. He doesn’t (and I asked) have a manual.

Fortunately, I can send him to a different place (corporate Tech Support) where they get paid for this.  But I got to wonder if the company that hired him did any vetting of him or his company first, just hired the first person that was recommended to them (yeah, my Cousin Vinny can help) or just picked him up off the street.

And yes, I was short the rest of the day…

I have a whole number of posts and articles and campaign stickers about backups. It doesn’t take much to get me up on a soapbox in the middle of the town square, declaiming in iambic pentameter the virtues of a known good backup, and the evils of stale or non-existent ones. So, as I was coming back from saving the day again (why is there never a phone booth around anymore?) when I got a blast from the past. Or at least a phone call about it.

I used to do some system integration work in the old days. So I got a call about a machine that I had done the controls on. Was actually a system that I was quite proud of, it had a custom C coded front end, some specific communication DLL links and a large part storage file system. Was positively cutting edge FOR 1997. Haven’t seen or heard of it since…until now. The PC (Running WinNT) hard drive finally gave up the ghost, and there is no backup available on site. So they called me. Do I remember it? Sure. Did I make a backup? Absolutely, burned it onto a DVD and put one in the cabinet of the machine, a full Norton Ghost image. Only problem is that the DVD isn’t there anymore (color me shocked) and there is no other backup. Did I keep one? Yes, for about 15 years or so. So I start digging. I actually find the original programs and C compiled program, but trying to put that all back together for this company is going to cost a staggering amount. More than the machine is worth, I would bet.  I believe my initial recommendation is going to be for them to send the HD out for reconstruction. At a couple of K, it will be cheaper than paying me T&M to relearn what my 1997 self knew intuitively.

Oh, and make a backup.