Workflow Engine: accelerating process automation | Mage Series
Low-code development has rapidly transformed the software industry, making life easier for developers, empowering power users to unlock their creativity, and enabling businesses to embrace digital transformation without the lengthy timelines of traditional development. Any software company with a strong foothold in low-code ensures its place among the giants.
Microsoft’s Power Platform, particularly Power Apps and Power Automate, have led the charge in this revolution. Power Apps enables developers and business users to build applications with ease, while Power Automate allows business logic to be defined through an intuitive, visual UI.
In this article, I want to share a personal story that explores my early attempts at addressing repetitive business logic, how it led to the development of an internal low-code solution, and the lessons I learnt along the way.
The chore of repetitive business logic
Many businesses define their processes as a series of stages in a flow. Each stage consists of steps that must be completed to move forward. This is universal and easy to understand. However, in the early days of Dynamics CRM, developers were left to handle much of the business logic manually, which led to repetitive tasks.
In 2013, Microsoft introduced Business Process Flow (BPF) in Dynamics CRM. BPFs made visualising business processes easier, allowing users to follow defined stages and steps. While this was a welcome improvement, BPFs only provided a visual guide with no connection to actual business logic. Developers still had to manually implement tasks like routing, notifications, and task assignments.
Over time, patterns emerged—common tasks and processes were repeated across different projects, but there were no complementary solutions to automate them. For example, linking the form state with the BPF stage or setting key performance indicators (KPIs) per stage were frequently requested features. It was clear that a solution was needed to alleviate these repetitive tasks.
A light-bulb moment
In late 2015, while working at Link Development, my colleagues (Michael Botros and Mira Ghaly) and I noticed this recurring pattern and began brainstorming a way to automate business processes and their associated logic. As a junior developer at the time, I was excited to contribute to what seemed like the perfect solution to our repetitive tasks.
We started with a simple concept: create a configuration record for each stage of a process, define the task owner for that stage, and apply this configuration automatically when the user entered the corresponding stage. This idea led to the birth of what we later called the Workflow Engine (WFE).
The birth of the Workflow Engine (WFE)
By early 2016, the WFE was introduced, and it quickly gained traction across multiple projects. Initially, the WFE was simple: a plugin triggered when the process or stage of a record (e.g., a case or incident) was updated. The plugin retrieved configuration details based on the stage and automatically created a task and assigned it to a preconfigured owner—whether that was the record’s owner, their manager, or even the customer in some cases.
This initial release addressed one of our biggest pain points: automating repetitive task assignments. But I wasn’t satisfied with just two features.
Feature expansion: notifications, routing, and stage-jumping
In the months following its release, I became obsessed with expanding the WFE’s capabilities.
By March 2016, the WFE had grown to include dynamic routing using CRM’s Text Parser (@CrmTextParser) and FetchXML queries (@AdvancedFind), allowing us to retrieve task owners dynamically. This gave us more flexibility to handle complex scenarios where static configuration wasn’t enough.
Next came the notifications module. I developed a rich editor with dynamic capabilities, allowing notifications to be sent via email, SMS, or push, all while dynamically retrieving recipients based on the business logic.
One of the most requested features was the ability to ‘jump’ stages in the BPF based on conditions. By integrating this functionality with FetchXML queries, we introduced a highly popular feature that made stage-jumping seamless and conditional.
The hiccup: performance challenges and overload
By mid-2016, the WFE had become deeply integrated into various projects, but this success came at a cost. The engine was growing rapidly, and with it came performance challenges. Complaints about slower system performance started to surface, and the complexity of the WFE’s logic made troubleshooting difficult.
As the sole developer of the WFE, I struggled to manage the constant influx of new features and the increasing number of issues. In retrospect, I should have taken a more controlled approach—perhaps limiting the WFE to a few pilot projects before rolling it out more broadly. This would have allowed for more thorough testing and refinement.
Further enhancements: SLA and form control
Despite the performance challenges, I continued expanding the WFE. By June 2016, I added several major features, including enhanced task flows, a custom Service Level Agreement (SLA) module, and form control.
The task flow feature allowed for more complex branching and looping within the business process, similar to how BPF operates but focused on tasks. The custom SLA module, designed to address limitations of the out-of-the-box (OoB) SLA, provided greater flexibility and performance improvements.
By popular demand, I also introduced least-loaded and round-robin task routing to balance workloads more efficiently. The ability to include attachments in notifications, such as generated reports, was also added as a standard feature.
Finally, since JavaScript is primarily used for show/hide, lock/unlock, and setting required fields, it was inevitable to introduce configurations that manage tabs, sections, and fields at each stage.
However, despite all these enhancements, the WFE had become bloated. It was now so deeply intertwined with the business logic that even small issues could cause significant disruptions. Maintenance had become a challenge, and I realised the need for a strategic pause.
Roadblocks and refocusing
At this point, I had to stop and reconsider the future of the WFE. The engine’s complexity had outgrown its original purpose, and the technical debt was starting to weigh heavily. I spent the next few months refactoring the code, focusing on performance improvements and stability. Thanks to a dedicated quality control team, we were able to stabilise the solution and significantly improve its reliability.
The impact of the WFE
Despite the rocky early days, the WFE achieved remarkable success. Projects that utilised the WFE saw a 30% reduction in implementation time, improved quality, and higher customer satisfaction. The engine automated many of the repetitive tasks that previously required custom logic, freeing developers to focus on unique business challenges.
However, there were also drawbacks. Developers felt disconnected from the core workings of Dynamics, and some complained that their role had been reduced to managing configurations rather than writing code. New developers, especially, lacked the opportunity to hone their coding skills due to the automation provided by the WFE.
Additionally, the WFE’s configuration-heavy nature made maintenance cumbersome, especially without a clear, visual interface. Misuse of the engine also led to complex, hard-to-trace issues that took significant time to resolve.
Conclusion
The Workflow Engine was an early attempt at accelerating business process development in much the same way that modern low-code platforms like Microsoft’s Power Platform do today. It helped automate common tasks, reduced repetitive work for developers, and sped up project delivery times.
Looking back, developing the WFE was a challenging but rewarding experience. It solved many of the issues we faced at the time and provided valuable lessons in balancing automation with flexibility. Today, with platforms like Power Platform leading the way, I can’t help but feel nostalgic when I encounter challenges that the WFE once resolved.
