Choose your language:
A leading broadband company sought TEKsystems’ support in establishing a more comprehensive product acceptance testing (PAT) framework and strategy to deliver an in-market testing capability, ultimately yielding a better end-customer experience.
The client is a leading broadband company providing video, high-speed internet and phone services to 6 million customers in nearly 30 states. TEKsystems has been a trusted IT services provider for this company since 2013.
JIRA, Multiplexers, Slingbox, TestRail, Witbe
As a leading broadband company, the client regularly releases diverse products and media across a wide range of product lines, spanning video, voice, mobile, portal and streaming services. A large portion of its products are available to customers via a content rendering system (CRS) that allows subscribers to access media via the internet or through in-home, set-top boxes.
But with so many products being released across different channels, it was difficult to anticipate where glitches in availability or performance might occur once products hit the market. The client lacked a formal in-market testing strategy to test product hardware and software in a controlled, customer-like environment that simulated the end-customer experience. Also, while the client did have a centralized lab in which some testing was conducted, such testing would not provide the actual variations of the in-market cable infrastructures that existed in each of their target in-market regions.
The absence of a comprehensive, formal testing framework incorporating an in-market testing capability meant product quality could be compromised if product code did not perform properly in the real world. With the high risk that even minimal disruption to the customer experience could result in a customer seeking alternative options for service—consequently impacting the client’s financial performance—it was critical that the client have an effective product acceptance testing (PAT) capability.
In order to improve the overall customer experience, the client required a product testing capability that included a formal in-market testing strategy that integrated into their enterprise software testing life cycle prior to production release. Testing conducted in this pre-production window would mirror the customer experience with the CRS and help identify defects or delivery shortcomings before products ever reached subscribers.
The client sought an IT services partner that could develop a PAT capability to achieve in-market testing and better prepare products for market. The ideal partner would have expertise in providing high-performing resources as well as deep experience implementing QA solutions that integrate seamlessly into client environments and management structures with minimal disruption.
TEKsystems had previously partnered with the client for resource-based solutions. Familiar with our Quality Management Services, the client requested we provide a solution enabling a formal PAT framework that included, among other benefits, the following key targets:
In addition to providing highly skilled resources, we were also asked to provide thought leadership focused on continuous improvement throughout our partnership.
Our solution would incorporate a wide range of service areas:
TEKsystems would leverage our Quality Management practice to provide a practice architect to serve as part of our delivery team and develop a formal PAT framework. This framework would include a comprehensive testing strategy, standardized processes (e.g., work estimation and reporting standards), reusable templates and other key components.
Our solution would be delivered by highly skilled resources, including a QA manager aligned with client stakeholders, and would integrate directly into the client’s project team. Further, we would provide team leads, and their associated team resources of various levels, across all in-market locations to ensure a lean but effective communication and coordination mechanism that could perform dynamically in the challenging client environment.
We would refine our testing strategy to incorporate a risk-based testing approach to accommodate the client’s aggressive timelines for product releases, focusing on efficiency with minimal risk to overall product quality.
Our initial efforts toward the scope of testing would largely focus on manual testing and also include application programming interface (API) testing of the various product platforms, including mobile and web. After accomplishing several successful testing events based on manual testing, we would establish an automation roadmap and develop this support capability leveraging the client’s chosen tools to complement our manual testing efforts.
These ongoing efforts would focus on regression functionality. Because of our established test design approach, this effort would not require any major refactoring of manual test cases to be adopted into our automation suite.
An in-market testing strategy would be developed to include defined facility requirements, equipment needs and target team structure. Resource profiles detailing the success criteria for the team would be developed and screening would be enhanced to ensure testing teams in each target market included the right balance of QA capabilities and technical skills.
Central to our solution would be establishing testing facilities in four of the client’s key markets. Our collaborative analysis with client stakeholders defined four initial target markets because they accounted for roughly 70 percent of the client’s subscriber base, and also included the majority of the client’s different set-top box models and variations in their independent infrastructure. This required working with the client to identify commercial real estate in the vicinity of the hub ends, setting up local test teams and equipment representative of those found in the local markets.
We would work collaboratively with client stakeholders, sharing results and metrics as well as insight to help them address decisions on product releases. From a PAT framework perspective, we would leverage this data to address tactical or strategic decisions in our testing efforts as well as take the opportunity to identify any process improvements toward greater efficiencies.
Throughout this service life cycle, our quality management practice architect would provide thought leadership as appropriate across process and strategy and serve as a trusted advisor for testing efforts, a focal point and differentiator for the client within their industry.
Over the course of a year, we supported in-market testing for approximately 25 different products across multiple product lines, including internet, video, voice, mobile, portable and streaming service. We established new test facilities in the client’s four core markets and provided more than 31 highly skilled resources to execute our solution.
We designed and executed test cases in TestRail and logged product defects in JIRA, while also coordinating the logistics of testing various products. Remote testing capabilities via Slingbox and multiplexer devices were also leveraged which increased test efficiency. We executed approximately 60,000 test cases in one month at the peak of testing. Additionally, new automation processes were implemented using Witbe.
We aggregated key reporting across multiple tools which allowed us to work collaboratively on key tactical and strategic decisions based on metrics and key performance indicators (KPIs) derived from the thousands of test cases executed each month. With our support, the in-market testing program helped improve the overall quality of the client’s products and end-customer experience—achieving a decrease in defect leakage to less than 5 percent. Through the integration and implementation of testing, defect and automation tools, we were also able to expand test coverage and reduce in-field service repairs costs.
The client was very proactive about improving product quality and the overall customer experience. During the planning process, TEKsystems’ team worked directly with client stakeholders to build an in-market QA strategy to align with their pre-established product rollout roadmaps. During the execution of our solution, the client’s testing and software teams worked side by side with our teams in the four testing facilities.
Through our Quality Management practice, we provided a practice architect with extensive experience developing QA strategies incorporating risk-based testing. Tapping into the architect’s expertise proved to be a key differentiator in our solution and enabled a successful engagement. She was able to design a custom QA strategy to fit the client’s unique needs and prepare the various test facilities with adequate testing parameters and processes. The quality and performance metrics we implemented also helped standardize product code review.
With more than 100 locations across the country, TEKsystems has relationships with skilled IT resources in every market. Leveraging our local market presence, we were able to ramp up teams as needed in the client’s core markets.