Choose your language:

Hong Kong
New Zealand
United Kingdom
United States

Building a Big Data Platform for a Leading High-Tech Manufacturer


Download PDF
Big Data Platform for Manufacturer

A leading high-tech manufacturer partnered with TEKsystems to build a big data platform to improve manufacturing reliability and product quality.

Client Profile

Our client is the leading manufacturer of innovative optical communication products and next-generation 3D-sensing lasers worldwide. Their laser sensors and chips are used in leading consumer electronic products across the globe. TEKsystems Global Services® first worked with this client to set up their business intelligence structure in 2014.

Industry Landscape

In the competitive world of high-tech manufacturing, speed is everything. When a technology research and development (R&D) team comes to you with a request for a product, you need to be able to show that you can scale quickly to meet their design and demand. For example, if your customer is creating a new video game console and needs a certain sensor—you need to prove you can provide 10 million sensors to meet their order. Within a short period of time, you must make the product and support the supply chain process to carry it from design to build to scale (Design2Scale). Scalable manufacturing plants and processes are critical to landing large, lucrative contracts, and shortening the product development life cycle—while producing quality output—puts manufacturers in the most advantageous position against competition.

It is typical for high-tech manufacturers to work with contract manufacturers to increase scalability when opportunities arise. Contract manufacturers are independent entities with their own facilities and infrastructure, which limits the product manufacturer’s visibility into the supply chain process and may mean less investment on their part to optimize the supply chain. Working with multiple contract manufacturers (e.g., one for assembly, one for testing) within a single supply chain process can lead to greater visibility challenges.

To combat this challenge, high-tech manufacturers often look to centralize big data platforms to gain insight into their end-to-end manufacturing process in pursuit of consistently replicating the golden batch (i.e., ideal) product.


Our client was scaling up the supply chain of their key next-generation product to satisfy customer demand. They needed to validate their ability to satisfy the growing demand. The client wanted to look at its entire supply chain process from a data perspective to ensure they were delivering high-quality products in the most cost-effective and efficient manner possible. The supply chain process as well as the manufacturing vendor ecosystem was constantly evolving.

Having previously partnered with TEKsystems on a business intelligence project, the client reached out to us to build a scalable data platform that could report on multiple data elements throughout their end-to-end manufacturing process and use it to improve their entire business process. The platform would need to integrate data from multiple systems and make it centrally available in a timely manner to ultimately produce a better product.

Having previously partnered with TEKsystems on a business intelligence project, the client reached out to us to build a scalable data platform that could report on multiple data elements throughout their end-to-end manufacturing process and use it to improve their entire business process. The platform would need to integrate data from multiple systems and make it centrally available in a timely manner to ultimately produce a better product.

In developing our solution and approach, we needed to account for the following:

  • Timeline. The client wanted the data platform ready in time to provide on-demand manufacturing data and quality information during the product validation phase for their customer, which was within three months from us coming on board.
  • Production process. Understanding the end-to-end production process would be core to building the data platform. As this was a new product line with a new supply chain, the entire production process would be very fluid while it was being established and streamlined.
  • Multiple contract manufacturers. The client would be working with multiple contract manufacturers, and thus did not have total control over those manufacturers providing data in a timely, consistent and consumable manner. Additionally, some of the contract manufacturers lacked technical maturity, so finding the right technical stack would be critical.
  • Ensure minimal to no impact to production. The data platform would support some key operational activities local to the manufac¬turing facility. Our solution had to ensure day-to-day operational activities would not be impacted by any network outages. While we needed to capture and centralize the data, we also needed to make sure the plants were self-sufficient enough to continue operations.

Simultaneously, the client was driving a companywide initiative to increase focus on continual improvement in the quality and consistency in their product manufacturing to exceed customer expectation. This data platform would be critical in helping our client achieve this quality-first mindset.


Based on our previous success delivering projects across different technology stacks and our knowledge of their landscape and business processes—as well as successfully executing a test data proof of concept—TEKsystems was asked to design and implement a big data platform to capture, analyze and report data pertaining to manufacturing operations to provide the client with a unified picture of their end-to-end manufacturing process.

The scope of the engagement covered:

  • Creating a strategy and roadmap to address business challenges
  • Designing a scalable and integrated data platform to address their analytics-related needs
  • Building process and automation to collect and analyze data from multiple source systems across the contract manufacturer landscape
  • Setting up a data environment for data science and statistical analysis

TEKsystems recommended using Amazon Web Services (AWS) as the central data hub. Data would funnel from the individual contract manufacturing plants into AWS. Local infrastructure was set up at the contract manufacturing plants to manage data volume. On-premise edge compute and analytics nodes enable analytics and knowledge generation to occur at the source of the data (i.e., manufacturing plants). This set-up ensures the plants are capable of completing their day-to-day operational activities independently and production won’t shut down based on network connectivity. The nodes connect to a central location to provide information, but operate differently if the center system were to lose connectivity.

Our solution architect developed a decentralized solution with “all in the box” edge analytics and centralized cloud-based prescriptive analytics. The solution included:

  • Edge data acquisition and analytics leveraging spark infrastructure
  • Centralized cloud data platform with Hortonworks on AWS
    • S3 as persistent data store for all data sets and also the file system for the HDP on EC2
    • HDP cluster on EC2
    • Redshift for aggregated data used for reporting
  • Data lab built on R and MATLAB

We followed an Agile approach (Concept-Pilot- Execute) with a two-week sprint cycle for the technical implementation. This ensured a quick validation of the solution stack and architecture and helped identify and address hurdles, changes or course corrections early and quickly. We also set up a dedicated steering committee to help track issues related to coordinating and pushing integration activities with the contract manufacturers.


TEKsystems established a next-generation, scalable data framework based on open source big data technology (Hortonworks Data Platform). The platform provides data exploration and profiling capabilities, data science and modeling capabilities, and a data lake for capturing, archiving and storing petabyte-scale data. By consolidating data across databases and archiving it in the platform, we saved license and service costs related to database maintenance.

Thus far, we have built the platform; set up and optimized the process to capture, analyze and report streaming data from multiple plants; and automated data capture and transformation for manufacturing and quality information for four product lines. We also built the master data management for product and customer, built the statistical model and enhanced the planning process for warranty provisioning, and eliminated the manual process of reading XML files and interpreting test data information.

The client now has the ability to analyze and report data across multiple platforms, and build predictive statistical models leveraging historic data available in the data lake. Leveraging the cloud to capture and store all of that data as it happens gives them real-time visibility into the end-to-end manufacturing process, which will ultimately help inform improvements to their entire business process.

Given the dynamic nature of the requirements and evolving nature of the new supply chain process, the implementation strategy revolved around agility and thought leadership to respond to frequent changes and unforeseen hurdles.

Two strategies that were critical to our successful implementation were:

  • Leveraging our Big Data Center of Excellence (CoE) in Hyderabad to run parallel pilots with the on-site team to speed up learning, testing and development to turn concepts into executable solutions. Since we were leveraging new cutting-edge technologies, our CoE helped our on-site development team with experimentation and validation of the new technology without the burden of learning being charged to our client. Running the pilot internally allowed us to test multiple technology combinations.
  • Leveraging an Agile approach to implementation. Concept-Pilot-Execute approach enabled us to validate the solution as it was implemented. It allowed us make continuous progress based on available information while accommodating changes. Plus, our active engagement with the client sponsor helped us gather input and provide thought leadership throughout the engagement.

The client now has the ability to consume, store and analyze real-time data so they can provide real-time input into manufacturing and quality operations. They are also in a better position to replicate the successful outcome of the production parameters (i.e., the golden batch) captured at one production facility across other facilities rapidly.

Moving forward, we will be bringing the remaining 17 product lines data into the data lake and building a statistical model for quality and manufacturing process data. We also expect to reduce Design2Scale time by 2018 and increase their customer satisfaction rating by gaining visibility into product quality monitoring.

Technologies Supported

  • Hortonworks Data Platform (HDP) (on premise and AWS EC2)
  • Apache: Apache HAWQ; Apache Hive; Apache Kafka; Apache Nifi; Apache Oozie; Apache Spark; Apache Sqoop
  • Amazon Web Services (AWS): Amazon Elastic Compute Cloud (EC2); Amazon Kinesis; Amazon Redshift; Amazon Simple Storage Services (S3); AWS CodeDeploy; AWS CodeCommit; AWS CodePipeline

Related Case Studies

Contact us today
Send Us a Message
Choose one