How GIM Cut ETL Time by 80% with a Modular Data Lakehouse on Azure Databricks

The Client

GIM (Global Industrial Manufacturer) is a leading producer of metal-cutting tools and tooling systems. With warehouses across North America, Europe, and Asia, GIM’s logistics operations span transportation planning, shipment tracking, inventory management, and supplier coordination.
To support these functions, GIM’s analytics team delivers dashboards and reports covering KPIs such as inventory velocity, carrier performance, cost-per-mile, and dock-to-stock times.

However, the company’s legacy Azure SQL Server–based data warehouse struggled to keep up with the growing volume and complexity of data. New sources – ranging from IoT telemetry to partner CSV files – created data silos, and refreshes for Power BI dashboards became increasingly delayed and unreliable.

The Challenge

Before the transformation, GIM faced a series of common but critical data challenges:

• Semi-Structured Data Overload: IoT devices and partner systems delivered data in diverse formats like JSON, XML, and CSV. Manual staging and parsing in SQL Server increased development overhead and reduced agility.
• Scaling Constraints: Scaling the existing SQL environment became prohibitively expensive. Adding compute drove licensing costs, while scaling out fragmented the data landscape.
• Rigid Development Environment: Data engineers had no sandbox for exploratory work, and even minor schema changes risked pipeline failures.
• Delayed Insights: ETL processes ran overnight or on weekends, and Power BI reports took hours to refresh – delaying decisions.
• Limited Data Lineage: Understanding where a single KPI came from often required manual investigation across dozens of scripts and tables.

Together, these issues slowed down innovation, strained operations, and reduced confidence in data-driven decision-making.

The Solution

To modernize its analytics infrastructure, GIM partnered with IPS to design and implement a modular, cloud-native Data Lakehouse on Azure Databricks. The architecture expanded on the bronze–silver–gold medallion model, introducing eight logical layers that ensured full traceability, governed transformations, and scalability.

Delta Lake provided ACID guarantees, while Power BI enabled intuitive, self-service reporting. Raw data was stored cost-effectively in Azure Data Lake Storage Gen2, and Databricks clusters provided flexible compute environments, supporting both interactive development and automated workflows.

Key architectural elements included:

• Eight Logical Layers: Based on an extended medallion model (bronze-silver-gold), each layer had a clear purpose – providing structure, data traceability, and easier governance. While the architecture referenced a “Data Vault” layer (L5), it was adapted to GIM’s needs and did not follow strict Data Vault modelling.
• Cost-Effective Storage: Raw files were stored in Azure Data Lake Storage Gen2. Compute was separated and spun up only when needed.
• Databricks Clusters for Development & Automation: Clusters supported both interactive development and scheduled jobs – boosting agility and reducing overhead.
• Power BI Semantic Models: A centralized dataset combined Import and DirectQuery tables, ensuring flexibility and scalability for business users.

The solution was delivered across five structured phases:

1. Assessment & Design:
   Conducted stakeholder workshops to define high-value KPIs (on-time delivery, dock-to-stock, cost per pallet), catalogued data gaps and latency requirements, and finalized an architecture blueprint including CI/CD patterns.
2. Platform Provisioning:
   Set up Azure Databricks workspaces, ADLS Gen2 storage, VNet peering, authentication and access controls (Azure AD, RBAC, ACLs), and established Git-based CI/CD pipelines with automated job deployment.
3. ETL Rebuild & Migration:
   Rewrote legacy SSIS packages as scalable PySpark and SQL jobs and implemented incremental load logic based on modified timestamps, in line with IPS architecture standards.
4. Semantic Modeling & Reporting:
   Built a composite Power BI model combining Import and DirectQuery tables, developed dashboards for key logistics metrics (inventory turnover, shipment exceptions, carrier scorecards), and conducted enablement workshops for analysts and operations managers.
5. Handover & Support:
   Delivered comprehensive documentation, implemented a governance framework for onboarding new sources and managing schema changes, and helped establish a Data & Analytics Center of Excellence to drive continuous improvement.

Results & Impact

The Data Lakehouse modernization delivered measurable performance improvements and laid the foundation for scalable, real-time analytics:

✅ ETL Processing Time Reduced by 80%
Nightly ETL windows shrank from ~10 hours to just 2 hours, thanks to distributed Spark pipelines.

✅ Power BI Report Refreshes 61% Faster
Dataset refresh times dropped from 90 to 35 minutes, enabling faster reporting and decision-making.

✅ Accelerated Time-to-Insight
New data sources can now be onboarded in a fraction of the time, improving agility across the logistics team.

✅ Improved Data Transparency and Lineage
Clear transformation layers and version control via Delta Lake increased auditability and trust in KPIs.

✅ Empowered Analysts and Operations
Business teams can now explore near-real-time dashboards, surface shipment exceptions instantly, and collaborate using shared semantic models.

Why it worked

GIM’s cloud-native Data Lakehouse modernization shows how manufacturers and large warehouses can break free from legacy constraints. IPS brought a deep understanding of both cloud architecture and logistics data requirements.

But the success of this transformation wasn’t just about technology – it was about strategy, structure, and close collaboration. By combining modular design, serverless compute, and modern DevOps practices, we helped GIM modernize their analytics without disrupting operations. Here’s what made the difference:

• Modular, Layered Design:
  IPS applied a clear separation of concerns across the eight-layer architecture. Each layer had a well-defined responsibility, allowing teams to isolate changes, reduce risk, and evolve specific parts of the pipeline without breaking downstream processes.
• Delta Lake on Databricks:
  By building on Delta Lake, IPS brought warehouse-grade capabilities – such as ACID transactions, schema enforcement, and time travel – to a scalable, cloud-native data lake. This gave GIM the reliability and data integrity needed to support critical logistics analytics.
• Serverless Compute & Job Clusters:
  Automated spin-up and tear-down of Databricks clusters allowed compute resources to scale with demand. This helped GIM optimize for cost during low-usage periods while ensuring full performance during peak data processing windows.
• CI/CD & Testing Frameworks:
  Git-backed notebooks and scripted deployments introduced consistency across environments. While GIM did not adopt automated testing tools like Great Expectations, IPS established a clear DevOps pipeline to enable repeatable deployments and version control.
• Business–Tech Collaboration:
  From day one, logistics SMEs were deeply involved in design decisions. This ensured technical development stayed closely aligned with operational realities and high-impact KPIs – maximizing business value at every stage.

With IPS, GIM didn’t just upgrade their data stack – they unlocked a foundation for continuous insight, smarter planning, and scalable growth.

Take the Next Step

Are your logistics insights held back by legacy pipelines and slow reporting?
Let’s build a scalable, cost-efficient analytics platform that moves at the speed of your business.

Schedule a consultation