Automating (legacy) business processes using cutting edge technologies - pt 2 (General Architecture)
In the last post I looked on the motivation that lead us to further explore what we lacked when automating processes at Gruner + Jahr. Now we want to take a look at the architecture we developed in order to achieve our goal: Automating business processes that span the whole spectrum from incoming E-Mails to SAP, SalesForce or ServiceNow.
In the center of our new business process automation architecture stands the Camunda Cloud as SaaS offering. We use it to deploy bpmn diagrams and have the workflow engine keep track of the process instances. It provides "job queues" for job types and we create workers for them. Camunda's Chief Technologist Bernd Rücker blogged about the considerations to take when using Camunda Cloud to orchestrate your (micro-)services and writing your workers. It's a really good read and we would have loved to have this insight when developing our architecture. Long story short: We developed our architecture a couple of months before Bernd shared his thoughts. We still use some of the concepts that he wrote about and are really happy with it.
You can watch me presenting the architecture and a use-case we implemented using it on Camunda Community Summit 2022 here.
Where do we come from? Where do we go?
When we developed the architecture we had some requirements. It should:
- enhance our existing product offering
- use as much as possible from our exisiting well maintained infrastructure
- make it easy to integrate NoCode / LowCode / RPA "mircoservices"
Especially the last requirement was important to us, as it would also heavily extend what our product could offer (first requirement). We had our customers ask us about automating things, that were (probably) possible to do with NoCode / LowCode / RPA. We had no experience with these tools but decided we wanted to use these new projects to change that. The market seems to go in that direction, so we might as well get comfortable with it. Coming from a "ProCode" point of view it was important to us as a team of software developers, that the new setup would make it easy to unplug the new solutions, if we decided that we did not feel comfortable with the new microservices, and be able to migrate them to ProCode.
Little heads up: having brought a bunch of services to production in a LowCode environment, we now can judge pretty well, what is and what isn't a good fit for the tooling we use. From the first project we now migrated some services to Java code, which was not effortless. One thing helped us a lot though: We had set up the process using bpmn and Camunda 8 as workflow engine. This had early on "forced" us to apply the separation of concerns principle. This implicitly takes care of defining clear interfaces at the worker/service boundaries. We only had to take care that the new worker implementation would implement the incoming and outgoing interface and were all set.
Below you can see the architecture we set up.
Going from left to right we see:
- Camunda 8 as bpmn workflow engine to orchestrate what is on the right of it
- EASI Platform as integration solution based on apache camel. It is a spring boot setup running on kubernetes with the possiblility to auto-scale horizontially. There are many EASI modules that most of the time integrate with one other service and speak its "dialect" while using AWS SQS queues to communicate among the EASI modules. For the use-cases that are related to M365 services and the process is orchestrated the two modules "EASI Camunda" and "EASI M365" play a central role. To communicate with the Camunda gRPC APIs we developed a custom camel component. If you are interested in using it contact me, we think about open sourcing it.
- Below the EASI Platform a couple of the service EASI integrates with are shown. There are many more.
- The AWS Serverless API Gateway setup is used for these use cases at the moment:
- An oauth authenticated api facade that enables third party services to drop off data for async processing by EASI modules
- An oauth authenticated api facade in front of cloud flows being used from EASI-M365 to invoke flows in M365 PowerPlatform
- An unauthenticated proxy for PowerAutomate cloud flows, that provide callback landingpages to give humans a possiblity to interact with the bpmn processes
- We use PowerAutomate CloudFlows in the PowerPlatform to interact with M365 Services in a LowCode fashion. The flows send results back to the EASI Platform via ApiGateway or are invoked in a synchronous way.
- RPA Flows are also available and from a view outside of the PowerPlatform behave just like CloudFlows
Why we are kind of in love with this architecture?
- This setup allows us to trigger bpmn process instances from an incoming e-mail in an Outlook mailbox or from an employee holding the identity card to a card read of our physical access control system in a very similar fashion with all the flexibility that apache camel offers.
- The fact that for a huge part we use serverless resources in AWS and Microsofts PowerPlatform is a huge benefit and cost efficient.
Next time we will look in more detail on why we chose camunda cloud and how we use it.
Newest job offers
Senior Azure DevOps Engineer
Intern in Applications Global Support (m/f)
Broadcast Network Operation Center Engineer (m/f)
Data Engineer - CRM (f/m/x)
Microsoft .NET Developer
Data Engineer - Personalization & Recommendation (f/m/x) (RTL Deutschland ISA 145/23)
Create Your Own Career
On our career website "Create Your Own Career" you can discover the wide range of entry and career opportunities at Bertelsmann and be inspired by our employee stories!Find more interesting jobs