Enterprise CI/CD for MuleSoft CloudHub 2.0 using GitHub Actions & Anypoint CLI
I've worked on many MuleSoft projects over the years, and looking back, I was always annoyed by the mundane tasks of setting up an API on Anypoint Platform and maintaining coherent configuration across multiple APIs. Not to mention, the Platform UI isn't much fun to use either. Now, Anypoint CLI has been around for quite some time now and has matured a lot. I wanted to combine the capabilities of Anypoint CLI with DevOps CI/CD and share it with the world, not lock it behind a paywall.
This blog covers how I did this in GitHub Actions while maintaining enterprise-level quality and security. I also chose CloudHub 2.0 for my deployments to keep things relatively new. I'll show the entire setup so you can recreate and make it your own.
First, let me summarize the features of my implementation, like an AI :D
- CloudHub 2.0; GitHub Actions; Anypoint CLI
- MUnits (I've commented this code because MUnit plugin for CI/CD is only available for licensed MuleSoft users, not for Trial account holders)
- Uses JQ & XmlLint to read JSON and XML files (LLMs helped a lot here, because I never used these tools before)
- Create/Update API Manager Instance (plus Exchange RAML Sync, Auto-Discovery, Policies & SLA Tiers)
- Artifacts (In Anypoint Exchange & GitHub), Tags, Releases, Auto version bumps for next development cycle, Sync PRs
Sounds interesting? Let's get into it.
Repository URL
This project is built for 2 environments: Sandbox and Production. But once you understand it, you can easily scale it for multiple lower and higher environments as needed.
Don't forget to go through the README.md file to learn more about the setup required on GitHub and MuleSoft Anypoint Platform. That includes secrets, variables, tokens, connected apps and their required permissions/scopes.
I've purposefully chosen to keep the names of environments identical across GitHub and Anypoint Platform, as it allows me to keep most of the workflow code environment agnostic, but if you wish to have different names, you can still do that. For example:
To change Sandbox GitHub environment name, find and replace the following line in sandbox.yml –
environment: ${{ 'SANDBOX' }}
To change Sandbox Anypoint Platform environment name, find and replace the following two lines in sandbox.yml –
echo "ENVIRONMENT=SANDBOX" >> $GITHUB_ENV
echo "ENVIRONMENT=SANDBOX" >> $GITHUB_OUTPUT
Orchestration
Let's go through the individual workflow files to understand them better.
► Sandbox.yml
The sandbox workflow deploys the application to the SANDBOX environment only. It uses sandbox-specific environment credentials and keeps production isolated.
It contains 2 jobs:
- sandbox-prep
- sandbox-deployment
1. sandbox-prep
This job sets up an API on Anypoint Platform. Before starting, it selects the SANDBOX environment in GitHub and prepares a few variables for output (to be used by the next job). Afterwards, it performs the following tasks:
- Checks out the code in an Ubuntu VM and installs Corretto-17 JDK.
- Runs MUnit using Maven (I've commented out this code because I am using Anypoint Platform's Trial Account and didn't have Enterprise Nexus Repository access configured in settings.xml. If you have Enterprise Nexus Repository access, you can uncomment this code block to run MUnit tests).
- Installs JQ (for reading JSON files), XmlLint (for reading XML files) and Anypoint CLI in the Ubuntu VM.
- Selects Anypoint Platform Environment and Configuration JSON File (anypoint-config.json file in the repository). This contains valuable information used by Anypoint CLI to set up our API in Anypoint Platform as explained in following steps)
- Displays the selected environment and configuration file set in the previous step.
- Tests Anypoint CLI login using the Connected App credentials.
- Uses XMLLint to read project metadata from the code's POM (pom.xml) file. This includes Organization/Business Group ID, project Artifact ID and project Version. These values are to be used in later steps.
- Create/Update API Manager Instance
- Uses JQ to read the configuration JSON file and extract exchangeAssetId. If its value is empty or the key itself is missing, then API Manager-related steps will be skipped.
- Then it looks for the RAML asset in POM using the previously extracted exchangeAssetId. If it fails to find it, then again API Manager-related steps will be skipped. It reads and stores the RAML asset version for use in later steps.
- Then it builds a Label for API Manager instance to be created or updated using the environment, project Artifact ID, and RAML major version. This is crucial and acts like an anchor and signature that a particular instance is being managed by this pipeline. Example value - SANDBOX-demo-api-v2
- It looks for existing API Manager instances with the label crafted above.
- If not found, it creates a new API Manager instance and retrieves the Auto-discovery ID.
- If found, it will sync the RAML asset version to the one found in POM as necessary.
- Apply/Update Policies
- It reads the policies key in the configuration JSON file and prints the count. If nothing is found, this step is skipped.
- Next it reads the currently applied policies from Anypoint Platform API Manager and prints that count as well.
- Then it will start processing policies from the configuration JSON file one by one.
- It checks if the policy is already applied; if not, it applies the policy with version and configuration as present in the configuration JSON file.
- If a policy is already applied, then it compares the version. A version change requires removal and re-application of the policy.
- So, in case there is a version mismatch, it removes currently applied policy and applies the one it found in the configuration JSON file.
- If there is no version mismatch, it will only update the configuration using the one found in the configuration JSON file.
- You might be wondering how to find and set policy-specific configuration. First apply the desired policy to a Test API Manager instance. Then use the following command in your local terminal –
anypoint-cli-v4 api-mgr:policy:list \
"Use Auto-Discovery ID Here" \
--environment "Use Anypoint Platform Env Here" \
--client_id "Use CONNECTED_APP_ID Here" \
--client_secret "Use CONNECTED_APP_SECRET Here" \
--output json > appliedpolicies.json
The output JSON will have the policy Configuration key with a stringified JSON value. That you can convert to a proper JSON and store in your configuration JSON file.
FYI: This command is also used by this Apply/Update Policies step in the beginning to read and count the currently applied policies.
Sample Output of the above command –
[
{
"ID": 8620536,
"Template ID": "433807",
"Asset ID": "client-id-enforcement",
"Asset Version": "1.3.3",
"Label": null,
"Status": "Enabled",
"Configuration": "credentialsOriginHasHttpBasicAuthenticationHeader: customExpression\nclientIdExpression: #[attributes.headers['client_id']]\nclientSecretExpression: #[attributes.headers['client_secret']]",
"Updated": "6 days ago"
},
{
"ID": 8666644,
"Template ID": "433850",
"Asset ID": "spike-control",
"Asset Version": "1.2.2",
"Label": null,
"Status": "Enabled",
"Configuration": "maximumRequests: 1\ntimePeriodInMilliseconds: 1000\ndelayTimeInMillis: 1000\ndelayAttempts: 1\nqueuingLimit: 5\nexposeHeaders: false",
"Updated": "a minute ago"
}
] - Create SLA Tier
- Reads sla key in the configuration JSON file.
- If there is already an SLA on Anypoint Platform with the same name, the pipeline skips this step; otherwise, it creates a new SLA as defined in the configuration file. Unlike Policies section, here only the name is compared for simplicity.
- This concludes the sandbox-prep job. If any configuration is missing in the configuration JSON file, the pipeline will not error out, which is crucial for APIs where the event source is a Scheduler or Queue Subscriber.
2. sandbox-deployment
This job handles the actual deployment of our API on Anypoint Platform. Before starting, it selects the SANDBOX environment in GitHub and prepares a few variables to use from the previous job. Afterwards, it performs the following tasks:
- Checks out the code in an Ubuntu VM and installs Corretto-17 JDK.
- Installs JQ (for reading JSON files) and Anypoint CLI in the Ubuntu VM.
- Builds JAR Artifact using Maven and publishes it to Anypoint Exchange.
Why, you might ask? Well, this is just how CloudHub 2.0 works. You build your JAR artifact and publish it to Exchange. The Anypoint Exchange itself acts as a Release Artifact Repository for CloudHub 2.0 deployments. It is mandatory for a build artifact to exist in Exchange, because it is linked with its corresponding deployment in Runtime Manager.
Note that this artifact is not the same as RAML Asset that we publish for REST APIs. It is a different individual entity.
For Sandbox deployments, we don't need to bump the project version in POM file as long as it contains the -SNAPSHOT suffix. Anypoint Exchange sets this asset's lifecycle as Development and the same version can be overwritten infinitely.
However, for Production deployments, we will have to bump the project version every time in POM file and remove the -SNAPSHOT suffix. This time Anypoint Exchange will set the asset's lifecycle as Stable and won't allow any overwrite operation on it (unless you delete the specific asset version from Exchange while checking the box that allows the same version to be re-used – I will not recommend it though).
- Deploy the previously published JAR Artifact to CloudHub 2.0. It will also dynamically attach api.id property to the deployment if API Manager Instance was created/updated in previous job. The application name is dynamically prepared using anypoint platform environment and project Artifact ID from POM in lower-case. Example value - sandbox-demo-api
- If the application already exists, it will be updated; otherwise, a new application is created in Runtime Manager.
The best thing about this Anypoint CLI-based deployment that I personally love is that, since the Artifact was already built and published to Anypoint Exchange previously, this step barely takes 7 seconds!
So, in summary, the Maven build command that downloads numerous dependencies and stuff runs at max 2 times, one for MUnit and one for JAR build, and that's it. This significantly reduces deployment time.
If you review the script, you will also notice the way additional properties are injected in the new deployment and update existing deployment commands. You can always add more properties, like mule.env is shown.
► Release.yml
The release workflow deploys the application to the PRODUCTION environment only. It uses production-specific credentials and is separated from sandbox deployment to ensure controlled release handling.
It contains 3 jobs:
- release-prep
- release-and-deploy
- post-release-routine
You'll notice in the script that the first two jobs set GitHub environment as PRODUCTION. I created an Environment Deployment Protection Rule which mandates Manual Approval everytime this environment is used by a job (check the screenshot above for how the approval looks). We don't want anyone to deploy our code, do we? This rule looks like this:
1. release-prep
Just like sandbox-prep explained above, this job sets up our API on Anypoint Platform. Before starting, it selects the PRODUCTION environment in GitHub and prepares a few variables for output (to be used by the next job).
I wanted to keep this job as similar to the sandbox-prep job as possible. That's why after Anypoint CLI installation, I only added a Convert Snapshot to Release step, which basically just removes the -SNAPSHOT suffix from project version in POM). By doing so, I am able to keep rest of the script identical to the sandbox-prep job.
This had to be done here because of the Read Project Metadata step, which reads project version from POM and sets a variable that gets passed down to the deployment job.
2. release-and-deploy
This job handles the actual deployment of our API to Anypoint Platform. Before starting, it selects the PRODUCTION environment in GitHub and prepares a few variables to use from the previous job. Afterwards, it performs the following tasks:
- Checks out the code in an Ubuntu VM and installs Corretto-17 JDK.
- Installs JQ (for reading JSON files), XmlLint (for reading XML files) and Anypoint CLI in the Ubuntu VM.
- Converts Snapshot to Release again (because this job freshly checks out the code), so it removes the -SNAPSHOT suffix from project version in POM.
- Builds JAR Artifact using Maven and publishes it to Anypoint Exchange. This time it is a Stable lifecycle asset, and as explained previously, it can't be overwritten.
- Then the pipeline creates a Git Tag, followed by a Git Release on GitHub along with release notes based on commit messages. Check here.
- Finally, the previously published JAR Artifact is deployed to CloudHub 2.0. Just like in Sandbox workflow, it dynamically creates the application name and attaches the api.id property to the deployment if needed.
- Not to mention, if the application already exists, it will be updated; otherwise, a new application is created in Runtime Manager.
In the following screenshot, the app is in stopped state because my Trial Account expired.
3. post-release-routine
This job looked fairly simple, but it took me some time to figure out how to get around the GitHub PR rule – no self-approval. I had to migrate my repo to a Free Organization account and set up two users with at least the Maintainer role.
Another reason for this was because of my main branch protection ruleset. Apart from the obvious – Restrict deletions and Block force pushes – I had also enabled the Require a pull request before merging with atleast 1 required approval.
Before starting, it prepares a Project Version variable to use from the previous job. Afterwards, it performs the following tasks:
- Checks out the code in an Ubuntu VM and installs Corretto-17 JDK.
- Increment the current project version. Example – 2.0.19-SNAPSHOT becomes 2.0.20-SNAPSHOT.
- Next, the pipeline creates a new branch (name dynamically created), commits the project version change and pushes it to the remote. Then it creates a PR from this new branch to the main branch and stores the PR URL for next step.
This step uses a PAT (Fine-grained GitHub Personal Access Token) that belongs to the 1st member of the organization.
Note that, since this repository belongs to an organization account, I had to approve the PATs before use.
This PAT has following accesses – - Read access to metadata (Default)
- Read and Write access to actions, code, and pull requests
- Then, the next step approves the previously raised PR using a PAT that belongs to the 2nd member of this same organization.
This PAT has following accesses – - Read access to metadata (Default)
- Read and Write access to pull requests
- Then this PR is finally merged using the same PAT from the creation step and the source branch is deleted. This is how the next version gets committed to the repository's protected branch.
Following is a screenshot of how this PR flow looks –
- Finally, a Sync PR is created that proposes merge from main to develop branch for next development iteration. This one doesn't require elevated access PATs, just using a normal GitHub Actions default token.
- Developers can merge this one before starting next development at their will.
That’s it.
Hope this helps accelerate your CloudHub 2.0 migration.
Debugging
After every Anypoint CLI command you will find a line that can be uncommented to print the output of the CLI command for debugging. Please check the example below and be mindful of the indentations :)
Cool Facts
- I've pushed Design Center RAML Asset and last good run GH Actions Logs to the Repo for archiving purposes. It should help when GH clears the run history after some time.
- This hobby project took me 230 workflow runs to perfect and satisfy my OCD :)
Comments
Post a Comment