Part 2 of a 2-Part Series.
Northwestern Indiana Telephone Company (NITCO) and Ringgold Telephone Company (or RTC) of Ringgold, Georgia, have been gracious to share their experiences, improvement efforts, and performance data to contribute to general industry knowledge.
NITCO has a territory just outside of the suburbs of Chicago. RTC has a territory just outside of the suburbs of Chattanooga, Tenn. Both providers are “poaching” subscribers away from their larger neighbor providers through their CLEC because they are successful in areas that many larger telecom providers are not. Ruburbia is a 10-month study that examines why that is, and how both providers are dealing with the expectations of prior suburbanites moving into rural North America.
We specifically want to thank Gary Gray of NITCO and Marty Black of RTC for their ongoing contributions to this study and their friendship to ISE magazine. Mr. Gray and Mr. Black (their real names) are visionaries striving for constant improvement in providing fast and reliable Internet services. They are examples of the high level of competency to be found in the too often unsung independent service providers that have been leading our industry from the beginning.
At the deadline for this article, there are still 4 months of data to be processed. Still, some intriguing patterns are already becoming evident, and some meaningful stories have already been shared.
It did finally rain in the southeastern US, so RTC is now facing what would be considered normal noise conditions. The utility pole butt plates and ground rods can now do their job, and more of the power is returning to the sub-station on the designed paths, rather than on sheaths. (For more on this subject, please see the OSP Expert column “Returning to Earth” in the March 2017 issue of ISE magazine (https://www.isemag.com/2017/03/returning-to-earth).
NITCO has implemented the plant rehab project that TEC’s Director of Plant Operations, Forrest Collier, promoted in his February 2017 article in ISE magazine. (See “Grounding = Better Reach” at https://www.isemag.com/2017/02/grounding-better-reach.) At least during non-peak times, every technician is expected to perform 10 rehabs per week. This fossil best-practice from the good ol’ days still pays off.
Furthermore, my associate, Don McCarty, recently trained the NITCO technicians on copper fault locating. One highly unusual situation he noted during that training is that NITCO had only 6 bad buried cable sections in their territory when Don arrived. His explanation is that NITCO avoided buried splices for decades. Even when cables were cut, the splices were put into pedestals. (Learn more about this from Don when he joins us at the Ruburbia Workshop in Orlando at ISE EXPO 2017.)
To date, NITCO IPTV installations are now at the “friendly customer” trial stage. Some issues have been encountered that have slowed progress, but nothing out of the ordinary. Adding IPTV is a major undertaking, regardless of the environment.
There are many KPIs considered when grading operational performance. From provisioning, to installations, to subscriber technical support and internal technical support, everybody is rated by some set of numbers.
Given enough time, a good statistician could make the maiden voyage of the RMS Titanic look like a great success. When we consider that budgets and head counts are based on these KPIs, it is not hard to imagine how departments with less than stellar performance can look great on paper: We handled 3,200 calls today. Only about 600 calls actually helped anybody, but look at our volume.
Historically, you can see how this happened. Up until 1984, profits guaranteed by the government built the American telecom infrastructure. In exchange for this guarantee, customer service standards had to be met or penalties were severe (ask GTE). During that time, for example, only 2% of Texas subscribers were supposed to report trouble each month. I know that from personal experience. If more troubles were reported than that over a couple of months, we were called into the boss’s office.
The working theory at that time was whatever we touched was supposed to last 30 years. This was possible because we were allowed to (really expected to) gold-plate everything. As a result, a 10-year return on investment was not a problem because it was universally understood that telecom was a “pay me now or pay me later” industry.
Now, nothing is guaranteed. Few things can be gold-plated, and there are fewer people to do more work. Subsequently, the decision to “pay later” is commonplace.
Let’s also respect that a 5-television IPTV install is more complicated than the old business key telephone systems were. Our knowledge base must be broader now more than ever before. We must be better and faster at everything than ever before just to survive. The providers that are quick, smart, and adaptable are the ones growing. We believe there is no coincidence that these providers are also our most active readers and most attentive students.
Choosing the Right KPIs
Ruburbia is a study into what happens when every operations department has the same primary KPIs. Naturally, we had to choose those KPIs carefully. We chose to keep it simple with our key KPI: Do it right the first time.
My first I/R supervisor taught us If you don’t have time to do it right, you don’t have time to come back and do it over. Today, this remains true. Therefore, examining repeated reports (going back after trouble is supposedly fixed) has been our primary focus to date.
Even with a single focal point, some interpretation is required. For example, most of the larger providers are tracking when a subscriber calls back within 5 days of the original dispatch. Many of the independent provider’s track when a subscriber calls back within 30 days. We see value in both. If our subscriber calls back in 5 days, somebody almost certainly missed something on the first try. A 30-day window allows environmental factors, and just plain bad luck, to impact the results, even when the first effort was completely successful.
Some telecom providers with 100,000 subscribers show an average of 17% on 5-Day repeated reports. It is not at all surprising that these providers are leaking subscribers at a high rate.
The opposite is true for both NITCO and RTC. These companies have 30-Day repeated reports below 10%. We selected these companies partially because they are high level performers. But, we can’t ignore some unique factors that contribute to these low statistics as well:
When a NITCO or RTC subscriber calls in with a problem, the phone is answered by a person that they have probably seen around town. Maybe, the person helping them even shared an Algebra class with them in High School. That familiarity factor alone helps to create a comfort that facilitates the type of communications required to fix problems on the first try over the phone.
Everyone involved in delivering quality service knows each other. Handoffs cause repeated reports when there is no accountability between people.
Even in 2017, small-town conditions often apply. You might not know everyone, but still, somebody they know knows somebody that you know. Often, what would count as a repeated report bypasses the system. Employees are called at home or stopped in the grocery store to be made aware of issues.
Because this study is using relatively small samples based on the number of subscribers, some apparent trends can be caused by an anomaly (i.e., a large cable cut). Therefore, the data must be tracked over a long period of time to establish reliable trends.
Improving the KPI tracking is one of the goals of Ruburbia, so tracking is being adjusted as part of the process. (See Figure 1 for NITCO Internet statistics, and Figure 2 for RTC Internet statistics.)
Initial Analysis of the Data
Again, we are dealing with some relatively small numbers and we are committed to withhold any conclusions until all the data is in and processed. Still, a couple of things are obvious at first glance.
1. The Repair/Restored cause code at NITCO is a catchall code that is meaningless when trying to capture trends. These types of codes can be avoided only by completely removing them in favor of more specific codes.
2. RTC is replacing way too many modems. Industrywide, modem replacement is the fallback in lieu of true trouble analysis and understanding. We expect to see this percentage decrease by the end of 2017.
Anecdotal Studies Offer Insight
It is rare to spend more than a few hours around technicians at any provider without hearing some complaints about the Technical Support (or Help Desk) group. Though sometimes justified, these complaints are rarely fair to the good folks there who are doing their best. They are often asked to help fix problems on technology for which they have never been properly trained.
This is not the case at NITCO and RTC. Their leaders give their Tech Support group the same classroom training as their technicians. They understand that trained and diligent Tech Support people can be a godsend.
Trained and diligent Tech Support people can be a godsend. One great example at NITCO is Angela Wicks. Here is an example of a Trouble Report where her analysis skills saved the day.
Customer experiences slow speeds and frequent sync drops.
• All narrow band copper tests pass.
• Noise tests are marginal, but it is rural America.
• DSL Bits per Bin Analysis show no noise-based bit rate reduction.
Dispatch #1: The modem was replaced, but the modem power supply was not. This is a bad practice. Even if we must move large furniture to replace the power supply, we should never replace one without the other. As a result, no service improvement was experienced.
Dispatch #2: The modem and power supply were replaced. Also, Cat5 house wire was installed from the NID to the modem. The practice of installing a Cat5 “home run” has yielded consistent improvement at NITCO overall, but not this time. No service improvement was experienced.
After this dispatch, this trouble was classified as “Chronic” and assigned to Angela for analysis. Within a day, Angela discovered that attenuation Upstream and Downstream was changing. She knew that this indicated a series (or high resistance) open and informed the technician.
Dispatch #3: A series open was located near the drop. It was fixed and the circuit has performed as designed since then.
We might ask why the longitudinal balance test passed on the first dispatch. The answer is that there must be at least 1,000 feet of copper beyond a series open before this test will fail. Keep that tidbit in your pocket. It commonly causes misidentification of this root cause.
Ruburbia was designed to be a learning experience for everyone involved, and it is accomplishing that goal. Improvement overall is being reported by both employees and subscribers. Still, we are not yet finished.
The final numbers for both NITCO and RTC will be tallied at the end of July 2017. Final analysis and conclusions will be shared in the Ruburbia Workshop at ISE EXPO 2017 in Orlando, Florida. Please watch for updates on the moderator list as ISE EXPO gets closer.
Editor’s Note: “Welcome to Ruburbia: Rate/Reach Challenges at the Edge of Suburbia, Part 1 of a 2-Part Series” is found in our February 2017 issue. You can read the article at https://www.isemag.com/2017/02/welcome-to-ruburbia.