A shortcut to save days costs weeks instead
If you’ve ever been tempted to take a shortcut in engineering, take my advice that there’s an excellent chance it will come back and bite you.
Early in my career, I learned that lesson in a particularly mortifying way.
It happened on my first job out of college. I was working in telecom, designing equipment for last-mile applications. My company manufactured customer premise and pole mount equipment, and I was responsible for doing the pre-qualification testing on one of our line card systems that worked as a digital multiplexer.
Telcos at the time had a performance specification for digital subscriber line (DSL) systems. The standard specified that for a distance of “x” you had to maintain a data rate of “y” or better on the cable, which carried both power and data. The issue is that along the length of the cable, there are different impedances and parasitics like capacitance and inductance that can introduce errors into the communication line, which can tank or degrade the signal.
The test procedure called for verifying link speed and signal strength at line-length intervals from 1,000 to 20,000 feet, looking for any interference on the line that would impact the system’s ability to perform as expected. Basically, we were supposed to record the data rate and the bit errors that occurred at increments of 1,000 feet, looking for any drop-off or dead spots. We also tracked the margin, which is the difference between signal and background noise. It normally falls off as the line lengthens, but we needed to make sure it didn’t drop precipitously anywhere.
We had fallen behind schedule, and our boss was getting pretty worked up. The only way it seemed possible to meet the timetable was to speed up the testing, which was an overnight process for each data point. But then I started thinking: I had done the testing at 1,000 feet, 2,000 feet, 3,000 feet—everything had checked out. Logically I thought (first mistake) that if the link works at the limit, surely it would work at every length short of that limit. So, we took measurements at 10,000, 15,000 and 20,000 feet.
Everything looked fine. At each additional point we tested, we were below the threshold for bit errors and above the threshold for communications and data speed. I was feeling pretty good as we were not too far off the original schedule.
We then submitted the design to a test lab to perform the full product qualification. Their job was to record and document everything so that we could submit the design for certification.
Things went downhill from there, fast. The lab came back after a few days and informed us that the product was failing at 8,000 feet—the signal was dropping out. I thought that sounded like crazy talk. Even though I had skipped that interval, I knew the unit had passed at 10,000 feet and beyond. Unfortunately, what I did not know was that the impedance at 8,000 feet just happened to be low enough that it started squashing the signal and dropping the link.
Any hope of staying on schedule quickly faded. We had already lost time because the failure had taken several days to discover. Turns out we had to completely redesign the line communication transformer. Then we had to do our testing over again, making sure we did not skip any intervals, and resubmit the redesign to the lab.
In the end, a clever shortcut that I thought would shave days off the process wound up costing us weeks. I was feeling pretty stupid at that point. Fortunately, I had a mentor who told me that a situation like this is one of the best ways for an engineer to learn because it’s something you won’t ever forget.
Everything looked fine, until it wasn't