Challenges
- The client’s existing data warehouse was built on a RDBMS-based MPP system.
- Data was being collected from various data sources like SAP ERP, JDA, IRI, Acousta and many others. The client was facing multiple challenges in integrating and performing real-time analytics with the data from these data sources.
- The client wanted to switch to a scalable EDM system which is easy to integrate with other enterprise applications.
- Wanted to minimize the effort spent on data migration and processing.
Solutions
- Analyzed various data sources and targets, and designed and implemented a suitable Hadoop architecture for the same.
- Designed near real-time ingestion system using Attunity and Kafka to perform real-time batch processing.
- Designed a data warehouse on Hive and HBase for all transactional data processing.
- Used Talend ETL to design and develop Spark streaming and batch jobs.
- Designed OLAP cubes for reporting.
Tools & Technologies
Key benefits
- Production rate can now be matched with order rate, thereby enabling faster and more informed decisions for sales, pricing, product, and manufacturing teams.
- Real-time operational visibility.
- Quicker and better-informed decision making.
- Lower data integration cost.
- Easily scalable architecture.
- Ability to drill across cubes/data subject in pre-defined and ad-hoc ways.
- Attunity replicates the integrated data into Kafta streams thereby providing a comprehensive view to the managers.
