Introducing Sitecore CDP/Personalize Serializer/CLI Tool

Well, I have wrapped up from Sitecore Symposium last week. Not fully recovered, but I wanted to blog about the topic for my session at Sitecore Symposium 2022 in Chicago. Now the topic that I did, was on Developing for the Engagement Cloud, but one of the topics I covered that I want to talk about/recap here today, is the PoC (Proof of Concept) that I had built for this presentation for CDP/Personalize. This PoC was meant to show a new and I will admit better way of building things in your tenant, using the tool that every developer is familiar with, which is Source Control.

Where Did This Idea Come From

So a couple of months ago, I was told about some APIs that could be used for migration. Now after exploring these APIs I realized they only covered a few specific use cases and that there was a greater need. At the time, I knew that customers were delivered a UAT and a Production tenant, but everything was manual to move items between those environments. However, the APIs that had been delivered was only for Decision Models, and not for what I considered developer tasks (I think this is a larger discussion for another blog post in the future). So I started to dig in and research if there were any more APIs that could be used to solve my challenges. This is when I stumbled upon a set of APIs that are being used by the frontend application of the tenant, but these same APIs are not currently documented anywhere. These APIs are very powerful, and open up the door for a bunch of different use cases:

• Using Source Control
• Migration
• One-Off Accelerator Deployments

What are these APIs?

As I just mentioned they are not currently in the official Sitecore documentation. I’ve worked to try to determine why that is, and it’s simply because the APIs aren’t officially supported by Sitecore currently. But does that mean you should avoid them? I don’t think that’s the case, and in fact, they open up and allow for automation scenarios that are too great to ignore. So how do you find out about these APIs? Well, it’s quite simple, if you log in to a CDP/Personalize tenant, open up Chrome Tools and view the network tab, you’ll notice as you interact with the interface, there are a lot of interactions taking place asynchronously, including with all of the APIs that I’m describing today. Now I’ve spent some time documenting what I’ve found here and created a very rough draft Swagger document of these APIs: https://dylanyoung-dev.github.io/sitecore-cdp-personalize-api-spec/. Just keep in mind that these are unofficial and something that I’ve personally spent time creating. These undocumented APIs are listed under Tenant Management, this isn’t an official name for these, but these are the ones that are taking place in the browser.

How to Start Using these APIs?

Okay, so now you know about the hidden APIs, but what do you do with them and how do you start using them? Well, the first major hurdle you’ll deal with when using these APIs is authentication. It’s different from the Batch, Stream, and Interactive APIs that you would typically use to ingest data to CDP/Personalize. These APIs use OAuth to handle authentication, and there are two core ways to authenticate and get back the access token that you’ll use as a bearer token in the rest of the API requests. There is the password grant type, which is where you would use your username and password for the tenant, to authenticate to use these APIs. The problem with this approach is that if you have Multi-Factor Authentication (MFA) enabled, which is likely the case in a production environment, then you’ll need to pass in your MFA code as well. This presents issues when you are trying to have a system run an automated process, such is the case during CI/CD DevOps deployments. The second method is the preferred method of using client credentials grant type, which will use the Client Key and API Token from the API access panel in your tenant:

Proof of Concept

So let’s talk a little more about the Proof of Concept (PoC) that I worked on that utilizes the APIs above. I had this crazy idea in my head, maybe 6 months ago, that source control could be used instead to create and control the flow of various assets to a specific tenant. Something like the diagram below was what I was envisioning. At the same time that I was dreaming up this new approach, the product team was talking about how it would be nice to test these further, but in my mind, how do you just test a tenant’s assets directly? I didn’t think the use case of randomly testing what was there, really made sense, and I thought it would make much more sense in a DevOps flow, where before you deploy to a tenant, you run unit tests and confirm they pass before you deploy, and then Automation tests could be done post-deploy to confirm if the changes were deployed successfully.

Now as of this writing, I have only some simple examples of Unit Testing, and no examples of how to handle automation testing, but rest assured, that is the next part of what I’ll be working on next. I now have two repositories that I’ll describe and link to below, but the serializer application needs to be broken apart into the SDK parts for calling the APIs, and then I should be able to share the Automated Testing scenarios utilizing that SDK. So more on that to come.

But before I get into the projects, just wanted to describe the illustration above. Simply put, a developer will no longer develop their changes directly in the tenant and instead will set up a repository with the serialized assets that they would build locally, and then there would be a process to deploy those changes to their tenant. That’s where we would get into the repositories and their use cases. Today all the examples use GitHub and Mocha for Unit testing, but my intention is for this PoC to be tech agnostic, so if you wanted to deploy changes, you could use Azure DevOps or GitLab, or any other Source Control Management system. I plan to create a lot more content coming up on different scenarios that could be done for this PoC. But with my PoC it is meant to be a community project, and thus all the repositories below are open source and I suspect will get way more traction as the community builds them out and other scenarios that these APIs can handle.

Serializer Repository

This is the brains of the operation and where all the magic happens. It takes everything that is in physical form (in a source control repository) and moves that (via a deploy command) using a Node CLI and moves that to a CDP/Personalize tenant. Currently, the PoC that was originally created only includes examples for Decision, Audience, and Web Experience templates, but the plan is that it really should support anything that you would create in your tenant directly. I always like to build things like this, to be unopinionated as possible, so those teams that have specific use cases should be able to handle their specific use cases using the same project. So for example, if your organization is structured in a way where everything is done by developers and directed by a marketing team then your developers can configure as much as they desire in Source Control and have those assets deployed to their tenants. Another use case, which I would assume is likely more common is that only certain assets are considered developer assets, such as templates, and then only those assets are in source control, while everything else falls outside of that and can be freely created in Source Control.

GitHub Repository: https://github.com/dylanyoung-dev/sitecore-cdp-serializer

Available on NPM: https://www.npmjs.com/package/sitecore-cdp-serializer

Examples Repository

This is the repository that shows how all this comes together to achieve the diagram above. Now it’s opinionated because it uses a specific set of technology, such as Mocha and Super test for Unit/Automated tests and it also uses VS Code and GitHub Workflow for CI/CD deployments. But it’s really easy to either create a template from this project and start creating it based on your own scenarios, or it’s also configured as a template repository, so when you land on the link below, you can click on Use template and you can start building based on this repository for your scenarios today.

GitHub Repository: https://github.com/dylanyoung-dev/cdp-personalize-examples

Extra Credit

So all of this is great for customers but this opens up a new world for the community or partners that want to build out starter kits or accelerators for a specific industry. For example, partner A builds experiences all the time for specific verticals like energy or health care, and could now use this strategy to either build a bunch of items for a specific industry so that those customers can get off the line faster. Now I don’t have any specific examples of this today, but I challenge the community to contribute to these projects and start to create out these use cases.