- Initially, the client’s data warehouse was built on RDBMS based MPP system.
- The client was receiving data from various data sources like SAP ERP, JDA, IRI, Acousta and many others. The client was facing multiple challenges in integrating the data from these data sources & perform real-time analytics on the given data.
- The client wanted to switch to a scalable EDM system which Is easy to integrate with the other enterprise applications
- Minimizes the effort spend on data migration and processing
- Analyzed the various data sources and targets, and designed and implemented a suitable Hadoop architecture for the same.
- Designed near real-time ingestion system using Attunity and Kafka to perform real-time batch processing.
- Designed a data warehouse on Hive and HBase for all transactional data processing
- Used Talend ETL tool to design and develop Spark Streaming and batch jobs
- Designed OLAP cubes for reporting
Tools & Technologies
- Production rate can now be matched with order rate, thereby enabling faster and more informed decisions for sales, pricing, product, and manufacturing teams.
- Provides real-time operational visibility
- Able to take quicker and informed decisions
- Lower data integration cost
- Easily scalable architecture
- Ability to drill across cubes/data subject in pre-defined and ad-hoc ways
- Attunity replicates the integrated data into Kafta streams thereby providing a comprehensive view of it to the managers.