A Model for Improving Backup Power
Batteries are the foundation for backup power across the telecommunications industry. However, despite the growing investment in expanding bandwidth and coverage, backup power has not enjoyed the same attention. As the Internet of Things expands (Cisco predicts 50 billion connected devices by 2020), telecommunications companies must ensure not just connectivity to PCs and phones but to billions of devices from home appliances to industrial equipment and public safety assets. With this proliferation of new devices, many of which will deliver critical services to individuals and businesses, the impact of service outages grows exponentially.
An Industry-Wide Problem
Gila River Communications in Chandler, Arizona, sought to address the challenges of improving backup power in their service area in the harsh Arizona desert. For backup batteries, the region’s climate posed significant challenges that led to higher CapEx and OpEx expenditures, and more maintenance time committed to resolving battery issues.
First and foremost is the desert heat. Heat is known to accelerate battery death by as much as half per 15°F. The heat is mitigated only by another desert feature: monsoon season, in which heavy rains typically knock out grid power. Gila River’s sites are spread out, meaning service always requires significant drive time. Like most wireline companies, Gila River could test batteries only a few times a year when a technician could get to a site and perform a manual reading. In contrast, the IEEE recommends a monthly general inspection with more robust quarterly and yearly tests.
Such periodic readings can leave sites with inadequate backup assets for months if a battery failure occurs between intermittent check-ups. Gila River technicians were spending too much time reacting to issues with their batteries. An outage would occur and technicians might discover the batteries were already dead. Battery service demands were also an opportunity cost, diverting resources from other projects.
Gila River was seeking a solution that would eliminate concerns about the reliability of their backup batteries and limit the time and cost of maintenance trips to remote desert locations of their wireline cabinets. The staff knew how the harsh climate in the Southwest was harming their batteries and the impact on more frequent replacement cycles and wanted to find a solution that would extend battery life.
The dilemma Gila River faced was not unique to them nor is it unique to the Southwest. Telecommunications companies across the country grapple with the same pain points that all result from reactive battery management, mainly the sheer number of batteries to test. According to Millennium Research, North America is home to a wireline network of over 33,000 Central Offices (COs) and Remote COs, 39,000 Controlled Environmental Vaults (CEVs) or huts, 258,000 Digital Loop Carriers (DLCs), and 577,000 Feeder Distribution Interfaces (FDIs). In addition, CTIA estimates over 298,000 cell sites (not including "small cell" sites now being rapidly built). All of these locations have batteries, and all should be tested regularly to ensure readiness.
Because there is no visibility into battery health between time-consuming manual tests, dead batteries are often discovered in the most inconvenient way: when they fail during an outage. When technicians are reacting to failures there is less time for deep analysis that may uncover a more important power issue such as a faulty rectifier that caused battery failure. The problem is misdiagnosed as a bad battery.
Trying to solve with more technician time is not scalable because outside plant cabinets can be so remote. This would just result in higher costs. And the problem is not just in the Southwest. Other climates have their own challenges. Heat is also an issue in the Southeast, as are unpredictable hurricanes, which can wipe out power to broad swaths. Snow can inhibit service to remote sites in Northern states, making routine checks or even battery service impossible. Micro-climates abound that bring their own issues.
The Proactive Approach
An emerging solution to these ongoing challenges is a proactive battery management. Proactive battery management utilizes a method for finding battery issues before they are exposed in an outage or discovered manually. Such an approach can deliver significant cost savings and operational efficiencies.
A proactive battery management approach requires a few basic elements. First, there needs to be some way to collect data from the batteries remotely and frequently. In other words, a sensor that can constantly monitor batteries should replace infrequent technician visits. Second, companies need a software application that can highlight battery issues to technicians before they manifest as outages. This component is essential to enable the operational efficiencies that reduce costs. Some solutions now have the ability to go beyond monitoring alone and can actually deliver remote control over the batteries allowing charging to be modified in order to extend battery life or to ensure battery readiness.
The solutions on the market that fit these requirements — and that Gila River was considering — cover a spectrum of varying complexity. Some products simply collect data and compile it in a rudimentary way, requiring data analysis after the fact (using spreadsheets, for example) to divine insights. There are a number of companies offering software to visualize the data and determine necessary maintenance procedures as well. Servato, the company Gila River ultimately selected for battery management, is capable of particularly accurate measurements of each battery’s state of health and uses this data to adaptively charge batteries based on their needs. This system optimizes battery life based upon conditions. (See Figure 1.)
Figure 1. Battery Management software capable of simply visualizing complex data.
An Unexpected Result and Benefit
Shawn Manning, Network Operations Manager at Gila River, was well aware of these issues (accelerated battery replacement, inadequacy of manual testing, and an overall concern about reliability and insight) when he decided to move towards a proactive solution. Coupled with the ability to reduce CapEx and OpEx by lengthening replacement cycles and lowering maintenance needs, the solution provided a way to help the entire team at Gila River worry less about their backup power and the reliability of their network.
Upon installation Gila River found a variety of issues at remote sites big and small. Using ActiViewTM battery management application software, Gila River began registering small, brief power outages. (See Figure 2.)
Figure 2. Outages registered by battery management software.
As batteries were being charged, the load drawn would drop the rectifier voltage down far below normal for the rectifier. The new hardware and software was able to detect a voltage drop in the rectifier, which signaled a faulty rectifier. Without the unique diagnostic capabilities, this issue would probably have not been exposed. The faulty rectifiers would need to be replaced to protect the batteries.
Gila River was able to hone in on the bad rectifiers. When Manning’s team examined the rectifier logs they found indications of bad rectifier blades. As they began replacing the rectifiers at sites experiencing these brief outages the results were clear: the "mini" outages had been caused by faulty rectifiers. Had they not been replaced the outage problem would have continued, and the batteries would have gone bad sooner.
Discovering these issues proved to Manning what he had suspected: lack of insight into DC power assets had left Gila River exposed to reliability issues. These issues went beyond the accelerated battery degradation in the Arizona heat that they were already aware of.
Choosing Monitoring vs. Management
As is common with most carriers, Gila River’s manual maintenance left them in the dark most of the time about the status of backup DC power in their cabinets. Now technicians receive daily and real-time updates about any issues at their sites. The power of robust battery monitoring equipment allows team members to view all of Gila River’s sites in table or map format for a high-level picture of vulnerabilities. The telecommunications company can send technicians on power-related service less frequently, and devote more time and energy to other projects. The expectation is that the adaptive, intermittent charging will deliver life expectancy benefits as well.
The technology that Gila River has decided to embrace is just one option. For network operators, engineers, and managers interested in improving reliability and reducing costs, there are a few key considerations. The main choice is deciding on the importance of remotely monitoring batteries or remotely managing them. For monitoring, inexpensive sensors are available to gather limited information on the batteries. However, the accuracy of these measurements is only as good as the sensors and often may prove to be of limited value if their accuracy becomes questionable. Managing batteries remotely using more accurate data may seem like the more expensive option as better sensors and analytical capability can mean higher upfront costs. However, this upfront investment can generate more significant cost savings down the road by extending battery life and reliably streamlining maintenance regimens.
Gila River has changed their beliefs and behaviors around battery management, and has become a more efficient organization. The risks Gila River and all carriers that use manual testing regimens face can be greatly mitigated. By exploring battery monitoring and management options other companies can improve confidence in their batteries, gain clarity into equipment reliability, and reduce costs.
