Part 3, The Final Analysis —
Ringgold Telephone Company (RTC) of Ringgold, Ga., and Northwestern Indiana Telephone Company (NITCO) have been gracious enough to share their experiences, improvement efforts, and performance data, to contribute to general industry knowledge.
RTC has a territory just outside of the suburbs of Chattanooga, Tenn. NITCO has a territory just outside of the suburbs of Chicago. Both providers are "poaching" subscribers away from their larger neighbor providers through their CLEC because they are successful in areas that many larger telecom providers are not.
"Ruburbia" is a 10-month study that examines why that is, and how both providers are dealing with the expectations of suburbanites moving into rural North America.
With all of that said, the results from the Ruburbia study were enlightening and certainly worth sharing. If you have not yet read Ruburbia Parts 1 and 2, they can be found on ISE magazine’s website at ise.mediaquest.co.1,2
This article focuses specifically on NITCO’s constant goal to improve their operational processes and prevent customer churn.
The Way It’s Always Been Done
When Gary Gray of NITCO was asked what motivated him to begin the process of improving operational performance, he shared this story.
Grandmother’s ham was the highlight of every family celebration. As time passed and kids grew up, Grandmother was watched during preparation and the process was dutifully documented and shared with all the daughters and daughters-in-law. One of the key steps was to cut the end off of the ham. As the kids grew up and years passed, dozens of mouthwatering hams were served. Slight changes were made to the recipe as each daughter personalized this family tradition, but every ham had the end cut off.
Eventually, it was time for one the granddaughters to learn how to cook Grandmother’s ham. Being inquisitive, the young lady asked why the end had to be cut off the ham. Her mother didn’t know, so she asked other family members. They didn’t know, so the decision was made to go directly to the source.
"Grandmother, why is it so important for us to cut the end off of the ham?" she asked.
"Dear, it is not important for you to do it," Grandmother replied. "I had to cut it off because my pan was too small."
Very few processes are written from scratch in our industry. Buried cable specifications were written based on aerial cable specs. Fiber optic cable specifications were written based on copper cable specifications. DSL specs were the worst. During initial deployments when even experts could barely spell DSL, nobody knew how good the cable pairs had to be. It was simply assumed that they had be better than the standards in place for POTS. Hundreds of millions of dollars were wasted chasing conditions that had no impact on DSL. If you have ever been forced by company procedures to find and fix a 122 MOhm short, you have seen an example of this.
Gary’s point was that he wanted to focus on the operational aspects that actually improved subscriber service and offered efficiencies that allowed him to do more with less. Sure, all of the existing processes are there for a reason. The question is whether that reason still applies to current conditions. In other words, maybe simply buying a bigger pan will save you tons of time cutting the ends off hams.
A Bigger Pan?
Gary’s improvement process at NITCO was planned during early 2016 and implemented in the latter half of the year.
• It was difficult to find the root cause of many issues because the trouble report and service order performance data was vague, confusing and often inaccurate.
• It was found that very few people in the Operations group knew how DSL was supposed to work or what impacted it when exposed to field conditions.
• It was found that cross-departmental handoffs were often dropped.
• It was found that OSP maintenance had largely been ignored for too long.
NITCO chose to pursue the following priorities:
Invest in Testing Tools. There were plenty of the Dynatel 965’s in the field, but some had no DSL synchronization functionality at all. The ones that could perform a sync test did not handle DSL bonding or VDSL sync or VDSL noise testing. They were literally blind to common trouble conditions and could not effectively evaluate circuit performance. NITCO chose the Viavi HST-3000 with bonded DSL functionality from ADSL1 to VDSL2 for the field technicians. Additionally, it was determined that the DSLAM’s had test and performance data that was not being used, but could be very valuable, if it could be understood.
Invest in Performance Tracking Tools. NITCO uses home grown-tools for trouble and service order tracking. Off the shelf solutions were determined to be cost prohibitive because of the required customization to NITCO’s unique requirements. Therefore, the home-grown systems were adjusted and augmented. Yes, this was more difficult, but they got exactly what they wanted and adjustments were faster because the knowledge required to make changes was in house.
Another important step in the process was adding more specific cause codes and driving home the benefits of religiously using the appropriate codes for the conditions. "Catch-all" codes or simply using a code with which you are familiar was strongly discouraged.
Invest in Cross-Departmental Training. I was chosen to design and perform the cross-departmental training.
• Outside Technicians, Inside Technicians, Help Desk Technicians, and their managers, were all included in the same classes.
• DSL and IPTV were covered from history, to design, to field deployment, to impairment impact and symptoms.
• A systematic and repeatable troubleshooting process was taught to all.
• Advanced HST-3000 and results analysis was taught to all.
• DSLAM test data and statistical analysis was taught to all.
• Don McCarty was brought in to teach advanced copper testing and troubleshooting.
Bonding and Grounding Best Practices: NITCO chose to not use contractors for this work. Every technician was (and still is) expected to complete 15 rehabs a month. These rehabs include cleaning up the bonds and grounds, but at least as important was the step to assure that pedestals had enough gravel to keep out rodents. These efforts were focused on the worst cable runs first, and then moving on the next worst. This focus caused the work to payoff much more than simply rehabbing individual sites without a plan.
Results
NITCO’s results were broken down into 6-month intervals.
• 2016 Q1&Q2 = Before any of the improvements were implemented
• 2016 Q3&Q4 = During improvement implementation
• 2017 Q1&Q2 = After improvements were implemented
It should be noted that you will see a slight increase in the trouble report rate during the study period. This is an industry trend that is universal for DSL services. The reason is that higher bitrates are constantly being pushed out. The higher the bitrate, the better the circuit has to be. Also, DSL bonding is becoming more and more common. Since you have 2 DSL circuits for each service, twice as many things can go wrong per subscriber. NITCO’s report rate increase over this timespan ranges from normal to below normal from month to month. (See Figure 1.)
Figure 1. Trouble found trends at NITCO.
"RRSS Repaired Restored Services" is the catch-all bucket. All service providers have this type of cause code. They have just different names for it; "Defective or Deteriorated" is a common one. Unfortunately, these codes are meaningless and therefore useless for performance analysis. In the beginning this code was far and away the most prolific at 40% in the beginning. We really could not tell what the most common problems were. There was a drop to 32% during implementation. This code dropped to half of the original percentage after improvement deployment. NITCO would like to see this drop more, but this was a significant improvement.
CBN is yet another catch-all. Modems fall into this this category. Traditionally, when we don’t know how to fix a DSL problem, we replace the modem. This is why modem replacement is a common cause of repeated reports. The real problem was not fixed the first time. This code started at 22% before improvement deployment, jumped to 36% during improvement deployment, but dropped to 4% after improvement deployment. This is indicative of advanced troubleshooting skills assuring that only bad modems are replaced.
NTF (or No Trouble Found) is another big hitter for causing repeated reports. Especially, when using POTS strategies for DSL, the cable pair looks good, nothing else is considered and the result is NTF. NTF started at 18%, dropped to 6% during improvement deployment and virtually went away to .5% after improvement deployment.
"Customer Education" and "Customer Cancelled Report" were both also cut dramatically over the trial period.
Figure 2. Trouble report trends at NITCO.
As was stated in Welcome Back to Ruburbia, Part 2, 17% 5-Day Repeated Report Rates are not uncommon. When 1 in 6 customers has to call back within 5 days of a supposed fix, we have a lot of angry customers and the subsequent churn is one of the most devastating trends in our industry. NITCO has never been that bad by modern standards. Their 5-Day and 30-Day repeated report at 6% and 17% respectively in the beginning would be cause for celebration in many circles. (See Figure 2.)
The 30-Day 5 repeated reports dropped to 12% during improvement deployment and stayed there after deployment. 5-Day repeated reports dropped to 2% during improvement deployment and increased slightly to 3% after improvement deployment. With such small numbers, the increase is seen as more of a statistical anomaly than deterioration of performance.
Repeated reports are one of the areas where simply focusing on them causes some improvement. Tying that focus to usable and systematic trouble shooting provides consistent and ongoing improvement. Mandatory post-testing to standards has been attempted to reduce repeated reports since automated testing was introduced in the early 1980s does not cause improvements. Individual technicians focused on providing good service armed with the right tools and knowledge does work.
DSL Deployment
NITCO is disappointed that their IPTV deployment has not gone according to schedule. They were certainly impacted by implementing with almost 100% internal talent. Even under the best conditions creating the infrastructure to deploy IPTV is difficult. Still, they now have all of their network issues worked out and it was done before their customers experienced them. After watching way too many subscribers pay for service providers that implemented new technology before it was ready, I applaud their patience and commitment to excellence.
Delivering reliable service is elusive. The minute you think you have it all figured out, you encounter something that had not considered. What was working suddenly does not work, and adjustments must be made. We have to be vigilant and watch for changing conditions that make our practices obsolete or less effective. The goal of constant improvement is the path to staying ahead of the game. Progressive companies like Ringgold and NITCO are winning new customers in just this way. There is no part of their process that is not scalable to work with larger providers. We can all learn from them.
[toggle title=”Life Is What Happens While You’re Planning Something Else” load=”hide”]We specifically want to thank Marty Black of RTC and Gary Gray of NITCO for their ongoing contributions to this study and their friendship to ISE Magazine.
John Lennon once said, "Life is what happens to you while you are planning something else". The Ruburbia study and the goals established early in the process did go according to plan, but not much else did in 2017.
This article was planned to be a report on the Ruburbia session at ISE EXPO 2017, along with the final performance analysis. Hurricane Irma made that session at ISE EXPO impossible.
If you have attended ISE EXPO and seen the professionalism and fluidness of every phase (from the sessions to the keynote speakers to the exhibits), the work, attention to detail and expense are obvious. The speakers, the vendors, and ISE, saw tremendous investments bashed against the rocks at Mother Nature’s whim. So much for plans.
The Ruburbia session in Orlando was planned to be the springboard where my customers and potential customers could invest in training in 2018 with proof positive that the right kind of training morphs directly into operational efficiency. The reader can determine whether the proof is there based on the content of this article and the 2 articles before this.[/toggle]
Endnotes
1. Welcome to Ruburbia, Part 1, by Vernon May, ISE magazine, February 2017. https://isemag.com/2017/02/welcome-to-ruburbia/
2. Welcome Back to Ruburbia, Part 2, by Vernon May, ISE magazine, July 2017. https://isemag.com/2017/07/welcome-back-to-ruburbia/
