Latest from Network Reliability/Testing & Assurance/Cybersecurity/Safety

ISE Columnist Don McCarty, OSP Expert

Start the New Year Right

Jan. 1, 2018
Rethink Your Commitment to Quality Service, Regardless of Cable Type — Most ILECs are transitioning from a copper infrastructure to a fiber-to-the-premises infrastructure. However, at this time less than 25% […]

Rethink Your Commitment to Quality Service, Regardless of Cable Type —

Most ILECs are transitioning from a copper infrastructure to a fiber-to-the-premises infrastructure. However, at this time less than 25% of US residential bandwidth customers are served by fiber-to-the-premises. The rest of the residential bandwidth customers are fed by copper cable pairs, high-speed wireless bandwidth, or cable TV companies. Meanwhile, until a full transition from copper to fiber occurs, if traditional ILECs want to thrive and retain customer loyalty, they must maintain the copper infrastructure until fiber is in place.

ILECs have been in transition since the telephone was invented, whether from aerial cable to underground, different types of cable, cable technology advancements bringing every service imaginable, and now, the move to fiber. Field technicians have always had to find and fix any circuit that failed for any reason. Whether we are looking at copper cable or structured fiber cable, that cable is a potential case of trouble that is tied to 2 other potential cases of trouble, that being in the central office/remote or customer end use equipment.

Most circuit failure root causes are readily identified, repaired, and restored. The tough ones are turned over to the old, experienced go-to technician who restores service. The problem with this is that in today’s environment the go-to technician has either retired or moved on to the fiber infrastructure, leaving the new technician with the copper infrastructure with little experience, little mentorship, and, often, very little back up from the management team.

Whatever technology you are servicing now or in the future, one key to fiscal responsibility and the best customer service is being proactive. Look at your business as your doctor looks at you; being proactive now may seem like a waste of time and money until you look at the cost and challenge of fixing a significant illness down the road.

What the Techs Need
Let’s start with what the techs need to do the right job. The first requirement is that the management team provides the right tools, and that includes training as skills are critical tools:

1. The best multi-functional test set that money can buy. Average costs: $5,000 for basic; $15,000 fully loaded.

2. If the field technician works in buried plant, they will need a quality cable locator for locating path and depth and an earth frame for finding sheath faults and conductor to earth faults. Average costs: $3,000-$5,000.

3. Training from equipment manufacturer is generally free but you must pull the tech out of work to train, so you have to backfill at $200/per hour approximately. Assume a 4-hour session with the manufacturer: $800 per technician assuming no travel or other expenses.

4. Training from contractor: Average is $400 per day per student. Assume a 4-day course so $1,600. Again, you have to backfill, so that’s another $200 per hour x 4 days or $6,400. Cost per technician for in depth training by a contractor: $8,000.

Is that a big cost per technician? You bet. So, at the high end that’s about $20,000 per technician for equipment and $8,800 for training — nearly $30,000.

Throughout the rest of this column, we will talk about costs if the tech doesn’t have the right equipment and training.

Where to Begin
So, let’s get started. Local knowledge and a customer complaint indicates whether to start fault locating at the central office/remote or the customer premises. Management benefits from creating teams consisting of the field technicians, central office/remote technicians, dispatch technicians, and engineering. If there is any adversary positioning between groups, then you are beat before you start.

The next step is a critical one that is so easy but often not taken: as soon as you arrive to work on a customer problem, either visit the customer in person, or call or text to let them know you are "on it."

When there is trouble in the cable, there is a 95% chance that it is in a terminal or pedestal in the distribution plant. The root cause of the trouble is usually a bat, insects, a rodent, or a "telephone technician rodent" that was in there yesterday. Driven by management demands to get the job done fast (instead of making the best fix), most field technicians transfer the customer’s circuit to another cable pair (the famous cut-to-clear approach).

3 Scenarios of Reactive Fault Locating

Scenario #1
Let’s consider the problem is a single-pair case such as a wire pinched in a pedestal causing a tip ground that is interrupting one customer’s service and there is no other potential trouble in that pedestal.

Taking the reactive approach, the technician doesn’t have to figure out the root problem which is the pinched wire. Instead they just move the circuit to another cable pair. Service is usually quickly restored and probably an average of 2 hours invested — about $800 loaded man hour rate.

However, the root cause was never identified nor dealt with. Rather than figure out how, why and where, they just move to a new pair.

Customer is happy, and the technician’s manager is happy. But they’ve now lost a pair, and that’s what has killed the business long term: pair by pair the plant is ruined. If the technician had the right tools, training, and the corporate mandate of doing proactive maintenance, the cost would have been about the same BUT you would have kept a pair in service.

Scenario #2
Next, what if the root cause is multiple faults at the pedestal often caused by rodents? Instead of a single-pair fault, chaos begins. Customer number one’s circuit is down because their pair is crossed with another pair in that pedestal. Rather than fixing the damaged pair and any other pairs damaged by the rodent, the technician deals only with the one bad pair, and does so by moving the customer to a clean vacant pair.

Either repair takes about the same amount of time: fixing the original damage to multiple pairs or moving one customer to a new pair.

However, now you’ve not only lost a pair, you will soon lose more pairs damaged by rodents as customers call to complain. With each call, you send out technicians one by one to "fix" the pairs that are failing. Think of that true cost. If 4 pairs are out and 4 technicians resolve the problem reactively, you have travel time and fully burdened technician time, so instead of 2 hours to proactively fix all the pairs at a burdened cost of $400, you have maybe 8 hours at $1,600.

Scenario #3
Trouble in the cable itself. A technician goes out and knows it’s bad between 2 pedestals or terminals, and that tech calls for a new cable because he doesn’t have the knowledge, time, or management permission, to chase down the problem. So that tech calls for a new cable — a 25-pair cable at 1,000 feet, plus splicing, and installing the cable (and the possibility of damaging other cables in the process). I have no figures on this but I’m sure it’s over $10,000; there are also permits, engineering, cable locating, truck roll, and the cable costs, that go into this.

Back in the day, I was doing quite a bit of training for a large West Coast ILEC. After training, I would take the techs in the field to do "real work." Every time we did section analysis on bad sections already submitted for replacement, we were able to fix more than 50% of the cables every time by digging 1 or 2 holes.

Based upon my experience, the average time to do good section analysis on a bad cable and dig the holes and the repair is about 8 hours. So, at $200/hour, that’s $1,600 vs. the cost to repair that cable. In this example, we repaired about 50% of the cables — the other 50% may still have to be replaced; nevertheless, you are still saving a great deal of money.

If a customer’s circuit fails because of cable trouble like the ones described above, a single, skilled proactive field technician using the fault locating equipment listed can proactively find and fix most root causes of cable troubles, eliminating repeat customer reports and unnecessary truck rolls.

So, isn’t it worth investing $30,000 in equipment and training, and even more for follow up training and updated equipment? They can earn back that $30,000 with a few proactively and correctly diagnosed and repaired cases of trouble.

My assumed rates, cost of equipment, and length of time in repair, are my guesses based upon 45 years in the industry but they may not reflect actual costs. Nevertheless, I believe even these rough numbers give you a good idea of the value of proactive maintenance.

Spend money to make money! Develop in-house experts by giving them tools, training, and a mandate to do proactive maintenance each and every truck roll. In addition, pair them up with a new recruit, and you have mentoring that can lead to additional savings.

Signing off
Future columns will cover operation of the features of the above-named test sets, and I will also provide tips for fault locating techniques in your copper infrastructure that will allow you to achieve maximum quality bandwidth. Contact me at [email protected] or call 831.818.3930. Your input and experience is valued by me and the readers of this column.

About the Author

Don McCarty

Don McCarty is the OSP EXPERT columnist for ISE magazine, discussing the issues around provisioning, testing, and maintaining copper for all services from POTs to IPTV. Don is also president of and the lead trainer for McCarty Products, a technical training and products company training field technicians, cable maintenance, installation repair, and Central Office technicians and managers. For more information, email [email protected] or visit