Choose your language:
TEKsystems partnered with a national retail organization to bring approximately 5,000 Windows and Linux servers to current compliance levels, updating roughly 75,000 security and application patches within a short timeframe.
The client, a leading national retailer with nearly 2,000 stores across North America, provides more than 40,000 different products to over 15 million retail and professional customers every week. TEKsystems has partnered with the client since 1999.
Recent data breaches at several large retailers have elevated IT security to a top priority for businesses and their IT departments. A company’s ability to maintain an uncompromised IT environment hinges on the aptness of their information security program and capacity to properly safeguard their assets, systems and customers.
Patch management is an essential element of any effective information security program. Patches keep critical software, applications and operating systems secure and running efficiently while delivering optimized performance and business benefits. However, patch management can be a cumbersome and resource-intensive process as software and application providers release new functionality and security patches on a perennial basis.
Being able to keep pace with patch releases and push updates to servers in an expedient fashion is imperative for minimizing risk to the business and reducing opportunities for malicious exploitation. It also helps ensure the reliability of a business’ IT environment and enables internal personnel to reap the benefits of the patches intended by software and application providers.
In the wake of recent, largely publicized data breaches at several major U.S. retailers, our client, a national retailer, recognized that patch management was a significant vulnerability within their organization. They felt their current patch management process was not sufficient and they lacked the internal capabilities to fully secure their data and information in their existing IT environment.
To help protect their business from future risks, the client needed to bring nearly 5,000 of their servers to current security and application patch levels. Business and IT leadership not only placed this initiative as their No.1 network infrastructure priority, but wanted it completed within an extremely condensed timeframe to prevent any existing vulnerabilities from being exploited.
Given the number of security and application patches needed throughout the client’s Windows and Linux servers—and the accelerated timeline outlined by leadership—the client required outside support to complete this project.
To expedite progress, the client sought a services provider specialized in patch management to deliver an IT solution that included working remotely to install patches under the direction of on-site project management at their headquarters. Given the tight timeframe for this project, the provider also needed their third-party partner to be able to ramp up teams quickly and begin work almost immediately, while working within their stringent screening and compliance requirements.
Given the sensitive nature of this project and associated information security risks, TEKsystems Global Services® immediately recommended a solution that would provide overarching logistical support for the entire initiative and deliver troubleshooting services as needed to improve overall service delivery.
We proposed a mixed on-site, off-shore support model that would leverage our Whitefield Solution Center in Bangalore, India, to maximize cost efficiencies and streamline project execution.
Over the course of 17 weeks, Tier 1 Windows and Linux server technicians in India would remotely apply more than 75,000 security and application patches to the client’s nearly 5,000 servers. Patch work would only be done during client-specified maintenance windows, and any issues that arose during each work session would be escalated to our Tier 3 support to analyze and troubleshoot.
We would also provide an on-site project manager and project coordinator to work at client headquarters, enabling collaboration between TEKsystems and the client’s network infrastructure team and project management office. TEKsystems would be responsible for prioritizing patches, orchestrating scheduling and ensuring approval for patch work before work occurred.
Our team would follow change management processes and procedures outlined by the client for conducting all patch work. Tier 1 technicians would follow a defined script for notification, patching, rebooting and verification. To keep track of progress, our team would meticulously record progress after each shift in a master server list that would be shared with our on-site team and the client’s teams.
Patching would only be conducted during narrow IT maintenance windows that had been predetermined by the client. Every client site would have its own set maintenance window which would vary depending on the type of facility, its hours of operation and the volume of back-office transactions that had to occur overnight, outside of business hours. The majority of locations had six-hour maintenance windows, though some had windows as small as two hours.
Based on our proposed mix of on-site and off-shore support, and our dedicated practice experience with large-scale patch management initiatives, the client selected TEKsystems for this engagement.
Over the course of 17 weeks, TEKsystems brought nearly 5,000 Windows and Linux servers up to date with the most current security and application patches. During the course of the project, we worked within narrow maintenance windows to ensure the client’s servers were available 99.9 percent of the time, reducing disruption to business and the client’s daily operating rhythm.
The patch management processes for the client’s Linux and Windows servers drastically varied. For the majority of our Linux server work, we were able to make updates via automated scripts. On a given night, we updated upwards of 50 servers, applying between 2,000 and 10,000 patches via automation.
Some of the client’s Windows servers were very outdated and required more customized support and manual patch application. Technicians logged in to a given server, clicked through security patch notifications to apply what security updates they could within that current version, rebooted the server, and then logged back in to perform the cycle again until the server was fully updated. The manual application was very cumbersome, sometimes taking one technician an entire eight-hour shift just to update one server. On average, our team applied approximately 300 patches to between 20 and 40 Windows servers a night, given the narrow maintenance windows.
Upon beginning this work, we quickly identified several gaps in processes where we helped mature the client’s procedures to improve overall efficiency and the quality of patch work.
At the start of the engagement, a formal approval process for conducting patch work was absent from the client’s original change management framework. If a server was identified as needing updates, our team was directed to access the server and begin work whenever its maintenance window permitted. This left application and server owners in the dark as to when actual work was being conducted within their entities, a breakdown in communication that our team saw as possibly detrimental to the business and IT environment.
We recommended integrating an approval process to require signoff from the application or server owner prior to our team proceeding with any work in their environments. This would ensure all owners were aware of any possible server or application downtime, and what work was being conducted during maintenance windows, ultimately limiting business disruptions.
The client lacked well-documented processes for change management, leaving vast differences between their development, test and quality (DTQ) environment and their production environment. This meant that, although some application and security patches successfully passed testing on the client’s DTQ servers, they failed when our team applied the same patches in their production environment.
To address this issue, we applied a large number of patches on an individual basis by application and server, rather than making sweeping updates per server. This detailed approach helped identify troublesome patches in real time so they could be addressed by our Tier 3 support team, and also minimize the number of rejected patches at the end of a maintenance window.
Overall, client stakeholders were very impressed by our change management process recommendations and ability to adapt to their complex IT environment. The client ultimately saw our patch management work as extremely beneficial in bringing their patches up to date and making their IT environment more secure.
Leveraging our Whitefield Solution Center, we were able to stand up a fully functioning technical team in an extremely condensed period of time. Working within the client’s compliance requirements for sourcing and screening candidates, our skilled off-site technicians and on-site project manager and coordinator were able to begin work almost immediately.
Our mixed on-site, off-shore team provided a flexible delivery model that allowed us to work during nonbusiness hours, given the time difference between India and the client’s headquarters. This allowed our team to make steady progress applying a high volume of patches to approximately 5,000 servers within a short period of time. Additionally, our team remained flexible when navigating the client’s challenging IT environment, leveraging process improvements to ensure completion of work.
At the onset of our partnership, the client planned to approach their patch management project via staff augmentation; however, we advised them that, given the sensitive nature of this project, a Global Services solution would be better suited for this type of project. Throughout the course of this engagement, we identified gaps in processes and proactively provided the client with feedback and ideas on ways to improve internal documentation and approval procedures.