Eventing Migration

The Integration & Business Support / Tech Delivery team at Bunnings had previously developed a custom event broker service, Event-API, to integrate upstream systems like Item Master, Inventory, and Locations with both internal APIs and external subscribers. This service utilized Azure Blob Storage and Event Grid to trigger events upon blob creation, which were then processed by Event-API. Depending on the configuration, blobs were either directly copied to a target account or delivered through other methods.

However, the design of this service presented some limitations. The storage account couldn’t be fully secured, and Event-API lacked advanced event broker features such as durability, scalability, dead-lettering, and guaranteed delivery, amongst other systemic issues.

Recognising these limitations and with Bunnings initiating a large-scale rollout of Solace’s PubSub+ platform, an opportunity arose to address the existing design deficiencies by migrating to Solace’s event broker. The main challenge was to transition existing publishers and subscribers to Solace without disrupting ongoing operations.

To address this, we conducted a thorough audit of all publishers and subscribers to determine which needed to be onboarded onto the Solace PubSub platform.

A dual-publication approach was adopted for each publisher (referred to as Feeds). This involved updating them to publish to both the existing Event-API service and Solace simultaneously. To facilitate this, a standard, shared library of reusable components was created. The initial goal was to achieve dual-publishing for all publishers and migrate all existing subscribers to Solace. Once all dependencies on the Event-API service were eliminated, a future project would revisit each publisher to remove the legacy publishing path and exclusively use Solace.

We successfully upgraded all existing publishers to support dual-publishing, and the majority of subscribers, including both Kubernetes services and Azure Function applications, were migrated to subscribe to Solace events.

A versatile capability library was developed, featuring a custom pipeline pattern implementation with standard components for performing ETL (extract, transform, and load) operations. This included functionalities for change detection, schema and business logic validation, domain mapping, and transformation. The library also provided support for seeding and allowed subscribers to automatically reconcile and request remediation for unprocessed events via targeted reseed events sent to the publisher.

Within this library, standard use-cases are provided using these components, which significantly streamlined the process of enabling dual-publishing for each publisher. These use-cases serve as templates with pre-built components for typical publisher and/or subscriber services.

This component library offers ongoing benefits and has the potential to be leveraged for numerous future integration scenarios, promising a significant return on investment.

While the decommissioning of Event-API is still in progress, the key achievement is the successful migration of all publishers and subscribers to Solace PubSub. This has made a substantial amount of information accessible via eventing for both current and future subscribers.