Create a CI/CD pipeline for APIs using Jenkins (2023)

In the modern digital era, many organizations adopt digital transformation technologies to keep pace with changes. APIs are one of the basic building blocks used in a digitally oriented organization. When the number of APIs managed by the organization grows, they need to have an automated process to handle the rapid API development process. Having a proper continuous integration and continuous deployment (CI/CD) process would give your organization an added advantage.

In this section, you will learn how to create an automated process using the WSO2 API Controller (apictl) with the WSO2 API Manager (WSO2 API-M). You will see how a solution is built using a version control system (Github), artifact repository (JFrog Artifactory) and CI/CD tool (Jenkins).

configuring the environment

Step 1 - Configure Jenkins

Before starting, you need to set up the Jenkins server on a remote machine that has public access. To use webhooks in the Github and Artifactory repository to trigger Jenkins jobs, you need a publicly accessible Jenkins server URL. If you are trying this on a local machine you may need to use a tool likesocketxpto create a tunnel.

In this section the instructions are tested on an ubuntu server setup on

Log into the server and install next.

  1. Install jq using the command given below.
    sudo apt-get install jq
  2. install theapictl tool. Extract to a location hereinafter referred to asCTL_HOME.

  3. Install java.

  4. Configure the Jenkins server by following the given instructionshere. You can choose to install the suggested default plugins. This will install commonly used plugins.

  5. Register a webhook in the JForg Artifact repository to notify of changes. To capture these events and trigger a Jenkins job, install a generic webhook trigger plugin.

  6. Set apictl home as an environment variable. I go to theManage Jenkinssection and selectconfigure system.

    Underglobal propertiesselectenvironmental variablesand set initial APICTL as below and clickTo save.

Now the Jenkins server is configured.

Step 2 - Create Github repositories

In this section, two repositories are created to illustrate this scenario.

  1. Source Repository -

  2. Deployment repository -

    (Video) How to Easily Build a Git Based CI/CD Pipeline for your APIs with Jenkins

The source repository contains the API-related files and metadata. The deployment repository contains settings related to each environment where these APIs are being deployed.

Step 2.1 - Configure the Source Repository

This section shows how to trigger a Jenkins job when a change is made to the repository. let's useWebhooks do Githubto trigger a Jenkins job.

  1. I go to thesourcerepository and under thesettings, selectWebhooksand add a webhook to the jenkins server.

  1. Use the url of your jenkins server (e.g.http://:8080/github-webhook/) as the URL andcontent typeas app/json.

  2. UnderWhat events would you like to trigger this webhook?section, selectLet me select individual eventsand selectpull requestepush.

The source repository is now configured.

Now let's initialize the source repository. This will be done on the developer's local machine.


If you haven't configured the API Controller, configure it using theapictl tool

  1. Clone the source repository. Navigate to the repository folder.

    $ git clone$ cd poc-cicd-source-repo
  2. Run the following command.

    $ apictl vcs init
    This will create avcs.yamlfile in the repository. Commit it to the source repository.

Now the source repository is configured.

Step 2.2 - Configure Deployment Repository

The deployment repository will be used to collect configurations for each deployment. To separate the configuration, we'll use git branches. Create a branch called 'dev' to hold configuration related to the dev environment.

Step 3 - Configure JFrog Artifactory

Before you start

Create a free cloud repository using After registration is complete, log in to

(Video) CI/CD Pipeline Using Jenkins | Continuous Integration & Continuous Deployment | DevOps | Simplilearn

  1. Create a user to upload artifacts. Log in using administrator credentials and go toIdentity and accessUsersand create a user. All roles are selected in this scenario.

  2. Make the public repository accessible. Go toSecurity → Settingsand selectAllow anonymous access

  3. Create a repository to upload the artifacts. Go toRepositoriesand selectadd repositories. selectGenerictype and provide a name (for example,myrepo).

    After saving you should be able to access

    Try loading a sample resource. An example is given below.

    $ touch resource.txt$ curl -u repouser:-X PUT -T resource.txt

    Go to the repository URL to view the resource.

  4. Configure a webhook to trigger the Jenkins job.

    Create a webhook to trigger the Jenkins job when a new feature is uploaded to the repository. Go toIn generalWebhooksand create a new one.

    URL -http://:8080/generic-webhook-trigger/invoke?token=123**

    addset of eventsasArtifact has been deployed.

    Select the repository in the new window that appears.

Step 4 - Configure API-M instances

Download WSO2 API Manager 4.2.0hereand start the instance. Set this in a separate instance like thedeveloperinstance.

Step 5 - Configure Jenkins Jobs

Step 5.1 - Configuration of global variables

not as shownConfiguring Jenkinssection, go toManage Jenkinssection and selectconfigure system. Underglobal propertiesselectenvironmental variablesand set the following variables with the details related to your environment.

(Video) CI/CD Pipeline for WSO2 API Manager with Jenkins

Variable Description
APIM_DEV_HOST IP or hostname of the dev instance (eg )
APIM_PROD_HOST IP or hostname of the prod instance (eg )
ARTIFACTORY_HOST Artificial hostname (ex:
ARTIFACTORY_PWD Artifact loading user password
ARTIFACTORY_REPO Repository name. (eg myrepo)
ARTIFACTORY_USER Username of the artifact upload user

Step 5.2 - Artifact Build and Upload Job Setup

  1. create a jenkinsPipelineproject.

  2. To useto bescript for the pipeline.


    The default Jenkins server workspace URL is/var/lib/jenkins/workspace/. If you installed the Jenkins server in a different location, change that path in the script accordingly.

  3. Undercreate triggersset the GitHub hook trigger toGITScm poll.

  4. Save the configuration. Now the Jenkins job is configured to listen for any changes in the source repository and upload any new updates to the artifact repository.

To set it up, run a build job. Note that this will fail.

Step 5.3 - Artifact Deployment Job Setup

The artifact deployment job will listen for all new updates in the Artifactory repository and deploy the artifact with the settings in the deployment git repository to thedeveloperAPI-M environment.

  1. Create a Jenkins pipeline usingto beroad map .

  2. Save the script and run a build to configure. Note that this will fail.

  3. To test that the webhook between the Artifactory and the Jenkins job is working, you can run the same upload test we did in the Configure JFrog Artifactory section

    $ touch resource.txt$ curl -u repouser:-X PUT -T resource.txt

If the configuration is working, this will trigger a new build.

(Video) DevOps : CI/CD Pipeline For Deploying APIs - Part II | Jenkins | MuleSoft | GIT | GitHub

test the scenario

To test this setup, use the OpenAPI definition-based method to create the APIs.

  1. Create API using an OAS definition. This will create a project namedPet store API. An example command is given below.

    $apictl init PetstoreAPI --oas
  2. Generate the deployment resource using the command given below.

    $apictl gen deployment-dir --source /Path/To/PetstoreAPI --destination/poc-cicd-deployment-repo

    You will see the resources created with the following structure.

    ├── DeploymentArtifacts_SwaggerPetstore-1.0.0│ ├── api_meta.yaml│ ├── certificates│ └── params.yaml
  3. open theparams.yamlfile and update the contents with the code given below.

    environments: - name: development settings: endpoints: production: url: '' deploymentEnvironments: - displayOnDevportal: true deploymentEnvironment: Default deploymentVhost: localhost
    The environment name is defined asdeveloper. This configuration will be used for thedeveloperenvironment.
  4. Commit it to thedeveloperbranch in the deployment repository.

  5. Copy oPet store APIfolder to the git location of the source repository.
  6. Create a file calledmeta.yamlwithinPet store APIfolder and add a version similar to the following.

    version: 1.0.0
    This version is used to deploy the package created using thePet store APIAPI in the Artifactory repository.
  7. The following is the content of the PetstoreAPI. Commit all files to the git repository

    ├── PetstoreAPI│ ├── Definitions│ │ └── swagger.yaml│ ├──│ ├── api.yaml│ ├── api_meta.yaml│ ├── deployment_environments. yaml│ └── meta .yaml

    When you commit to the source repository, theCICD_ARTIFACT_UPLOADwill fire first and load the created component into the Artifactory. This is displayed in the console log of the Jenkins job.

    Once the package is uploaded to the Artifactory repository, it will trigger the Jenkins deployment job. The package being deployed to API-M is displayed in the logs.

    Log in to the API Manager instance and you will see that the API has been created.

Let's update the API and see the changes. The API is inCREATEDstate. Let's publish this API by following the steps below.

  1. open thePetstoreAPI/api.yamlfile and change thelifeCycleStatusforPUBLISHED
  2. OpenPetstoreAPI/meta.yamlfile and change the version to some new value (say 1.0.1).

  3. Commit both files.

    (Video) DevOps : CI/CD Pipeline For Deploying APIs - Part I | Jenkins | MuleSoft | GIT | GitHub

After committing the changes to the source repository you will see the jenkins jobs being triggered. Log in to the Publisher Portal and verify the changes.

If you go to the artifact repository and check thePet store APIyou will see two versions in the repository.



How do I create a Jenkins pipeline for API? ›

Create Pipeline using API
  1. Create pipeline manually in Jenkins UI and test it.
  2. Export your pipeline using export import plugin or Jenkins jar to xml file or extract it from ${JENKINS_URL}/job/JOB_NAME/config. xml.
  3. Now use this xml file as a reference and create pipeline through API.
  4. Use below curl request to create pipeline.
Sep 25, 2021

How to deploy API through Jenkins? ›

Jenkins Pipeline stages to deploy your API
  1. Fetch the OpenAPI specification file. ...
  2. Import the OpenAPI Specification file. ...
  3. Create an application plan and an application. ...
  4. Run the end-to-end tests. ...
  5. Promote the configuration to the production gateway. ...
  6. Read more.
Jul 30, 2019

How do I create a CI CD pipeline in Jenkins with GitHub? ›

On the Jenkins dashboard, click on “New Item”. After that, enter the name of your project, choose “Freestyle Project” from the list, and hit the enter key. The next step is to configure the project on the Jenkins dashboard. In the General tab, choose your GitHub Project, and enter the GitHub Repository URL.

Does Jenkins have a REST API? ›

The jenkins-rest library is an object oriented Java project that provides access to the Jenkins REST API programmatically to some remote API Jenkins provides. It is built using the jclouds toolkit and can easily be extended to support more REST endpoints.

How do I trigger a Jenkins job with REST API? ›

Setting Jenkins Job to respond REST API

The REST API feature can be enabled per Job basis. To enable REST API trigger for a Job, Navigate to Your JobName ->Configure -> Build triggers TAB and Check on 'Trigger build remotely'.

What are the steps to create CI CD pipeline? ›

The 7 essential stages of a CI/CD pipeline
  1. The trigger.
  2. Code checkout.
  3. Compile the code.
  4. Run unit tests.
  5. Package the code.
  6. Run acceptance tests.
  7. Delivery or Deployment.
Sep 4, 2022

What is the limit of Jenkins API request? ›

The limit is 5000 for me and I see that every second about 100 requests are going and I'm not sure which job/service is using up the bot's requests. Ideally my other jenkins jobs should take up maximum 1 request per minute.

How to use API key in Jenkins? ›

Authentication with API Token
  1. Go to your Jenkins instance and login with the user you want to use in the app;
  2. Open the user profile page;
  3. Click on Configure to open the user configuration page: If you already have an API Token: click on Show API Token;

How do I create a Jenkins pipeline for deployment? ›

To get started quickly with Pipeline:
  1. Install the Docker Pipeline plugin through the Manage Jenkins > Plugins page.
  2. After installing the plugin, restart Jenkins so that the plugin is ready to use.
  3. Copy one of the examples below into your repository and name it Jenkinsfile.
  4. Click the New Item menu within Jenkins.

What is the difference between Jenkins and GitHub pipeline? ›

Jenkins and GitHub Actions share some similarities in workflow configuration: Jenkins creates workflows using Declarative Pipelines, which are similar to GitHub Actions workflow files. Jenkins uses stages to run a collection of steps, while GitHub Actions uses jobs to group one or more steps or individual commands.

How to create CI CD pipeline in Git? ›

Table of Contents
  1. Step 1: Create or choose a repository, and pick a project.
  2. Step 2: Open GitHub Actions in your repository to start building your CI/CD workflow.
  3. Step 3: Make changes to your code to trigger your CI/CD pipeline.
Feb 2, 2022

How do I deploy code in Jenkins pipeline? ›

Jenkins - Automated Deployment
  1. Step 1 − Go to Manage Jenkins → Manage Plugins. ...
  2. Step 2 − Go to your Build project and click the Configure option. ...
  3. Step 3 − In the Deploy war/ear to a container section, enter the required details of the server on which the files need to be deployed and click on the Save button.

How to automate REST APIs? ›

We'll cover how to:
  1. Send API commands to the server and validate responses.
  2. Use values from responses as parameters in test steps.
  3. Combine REST API and recorded UI steps within the same automated test to achieve end-to-end testing.
  4. Analyze reports.
Mar 17, 2020

Is every API a REST API? ›

The majority of HTTP APIs are on the verge of becoming completely RESTful. But not all HTTP APIs are REST APIs. To be termed a REST API, the API must meet the following architectural requirements: Client-Server: A server oversees the application's data and state in REST applications.

How to access Jenkins API with token? ›

Connect to Jenkins using API token authentication
  1. Navigate to Your Username > Configure > API Token.
  2. Click Add new Token.
  3. Click Generate.
  4. Copy the API token that is generated.
Feb 2, 2023

How to generate API key for REST API? ›

Generate a User Token REST API Key
  1. In the web app, navigate to User Icon My Profile User Settings.
  2. In the section API Access, click Create API User Token.
  3. Enter a Description to help you identify the key later.
  4. Click Create Key.

How do I create Jenkins credentials in REST API? ›

Create Credentials

According to the documentation, a POST request must be sent to $JENKINS_URL/<path to context>/credentials/store/<store id>/domain/<domain name>/createCredentials with the credentials XML configuration as body.

How do I remotely trigger a Jenkins job? ›

Create a remote Jenkins build trigger in three steps
  1. Create a Jenkins build job and enable the Trigger builds remotely checkbox.
  2. Provide an authentication token; This can be any text string of your choice.
  3. Invoke the Jenkins build URL to remotely trigger the build job.
Dec 22, 2020

What are the different types of pipelines in Jenkins? ›

A Jenkinsfile can be written using two types of syntax - Declarative and Scripted. Declarative and Scripted Pipelines are constructed fundamentally differently. Declarative Pipeline is a more recent feature of Jenkins Pipeline which: provides richer syntactical features over Scripted Pipeline syntax, and.

What are the four parts of the CI CD pipeline? ›

Stages of the CI/CD Pipeline
  • Source. Development teams use version control systems, such as Git. ...
  • Build. The build stage combines the source code with dependencies. ...
  • Test. ...
  • Deploy.
Mar 24, 2022

What is a real time example of CI CD pipeline? ›

CI/CD pipelines real-life example

Netflix and Facebook are the prime examples that have successfully implemented CI/CD pipelines to deliver software updates and features quickly and with high quality. Netflix uses a custom-built platform called Spinnaker to manage its CI/CD pipelines.

What is CI CD in Jenkins? ›

Overview. A continuous integration and continuous deployment (CI/CD) pipeline is a series of steps that must be performed in order to deliver a new version of software. CI/CD pipelines are a practice focused on improving software delivery throughout the software development life cycle via automation.

What are the stages of CI CD in Jenkins? ›

There are four stages of a CI/CD pipeline 1) Source Stage, 2) Build Stage, 3) Test Stage, 4) Deploy Stage. Important CI/CD tools are Jenkins, Bambo, and Circle CI. CI/CD pipeline can improve reliability.

What are the five stages of a development pipeline? ›

A continuous delivery pipeline consists of five main phases—build/develop, commit, test, stage, and deploy.

How many pipelines can be created in Jenkins? ›

1 Answer. There are two types of pipelines in Jenkins: Declarative. Scripted.

How many ways we can create pipelines in Jenkins? ›

Additionally, Jenkins provides two ways to create pipelines: Scripted and Declarative. Finally, the Jenkins pipeline gives the flexibility to define various stages and steps programmatically.

What is the difference between Jenkins project and pipeline? ›

The major difference between any Jenkins job and a Jenkins Pipeline Job is that the Pipeline Scripted job runs on the Jenkins master. This uses a lightweight executor which uses only some resources to translate in the master to atomic commands that execute or send to the agents.

How many requests can a REST API handle? ›

By default, it is set to 100 requests per 100 seconds per user and can be adjusted to a maximum value of 1,000. But the number of requests to the API is restricted to a maximum of 10 requests per second per user.

What is the API limit for Jenkins GitHub? ›

One of our service account hit Git Api rate limit of 5000 per hour. It is used in jenkins and Aws code pipeline.

What is the size limit of API? ›

A web server such as Apache allows a payload size of 2 GB. Google's Photo APIs allow photos with a maximum file size of 200 MB.

What is the best way to pass API key? ›

You can pass the API key via Basic Auth as either the username or password. Most implementations pair the API key with a blank value for the unused field (username or password). You will need to base64-encode the `username:password` content, but most request libraries do this for you.

Where do I put my API key? ›

Don't store your API key directly in your code.

Instead, store your API key and secret directly in your environment variables. Environment variables are dynamic objects whose values are set outside of the application. This will let you access them easily (by using the os.

What is an API key and how do I get one? ›

An application programming interface (API) key is a code used to identify and authenticate an application or user. API keys are available through platforms, such as a white-labeled internal marketplace. They also act as a unique identifier and provide a secret token for authentication purposes.

Why Gitlab is better than Jenkins? ›

Both Jenkins and Gitlab are designed to serve different requirements. While Jenkins boasts of a large plugin shelf, Gitlab is a comprehensive DevOps tool. While multiple plugins do your job efficiently, integration and management of these plugins might become a challenge when the project scales up.

How to set up Jenkins for continuous integration? ›

js application to show how to define the CI/CD process for a project.
  1. Prerequisites.
  2. Add the Jenkins User to the Docker Group.
  3. Create a Personal Access Token in GitHub.
  4. Add the GitHub Personal Access Token to Jenkins.
  5. Set Up Jenkins Access to GitHub.
  6. Set Up the Demonstration Application in your GitHub Account.
Jan 5, 2022

What is the first step in the pipeline workflow? ›

Always Put Build Workflow First — The Build Workflow should always be the first stage in the Pipeline. This enables the rest of the Pipeline to use the artifact it builds and collects.

What are the 3 types of pipelines in Jenkins? ›

Different Types of Jenkins CI/CD Pipelines. Scripted Pipeline. Declarative Pipeline. The Concept of Stages in Jenkins Pipeline.

What are 3 important stages in pipeline in Jenkins? ›

Stage: This block contains a series of steps in a pipeline. i.e., build, test, and deploy processes all come together in a stage. Generally, a stage block visualizes the Jenkins pipeline process.

Which is better Jenkins or GitHub? ›

When using GitHub Actions, you are essentially bound to Github's source code management platform. You may keep your code on any repository, including Gitlab, BitBucket, and others, by using Jenkins. Compared to GitHub Actions, Jenkins is more experienced and established.

How to implement CI CD pipeline by Jenkins and Docker? ›

Docker Jenkins CI/CD Pipeline
  1. Jenkins-Docker Continuous Integration & Continuous Deployment Pipeline. ...
  2. chmod 400 /var/run/docker.sock. ...
  3. unix://var/run/docker.sock. ...
  4. Pipeline syntax-> select withcredentials from drop down -> secret text → select dockerHubaccount -> generate script. ...
  5. chmod 400 /var/run/docker.sock.

How to create pipeline in Jenkins with GitLab? ›

In Jenkins, go to Manage Jenkins → Configure System and scroll down to the Gitlab section. Enter in a name for the connection, the URL to your GitLab instance, and then next to Credentials, click Add → Jenkins. This is where you will add your GitLab token from Step 2. Next to Kind, select GitLab API Token.

How to create CI CD pipeline with YAML? ›

CI/CD Pipeline using YAML file
  1. Case Study.
  2. dotnet new sln -o Test_CI_CDPipeline.
  3. cd Test_CI_CDPipeline.
  4. dotnet new mvc -n Test_CI_CDPipeline.
  5. dotnet sln HelloWorldApp.sln add HelloWorldApp.Web\HelloWorldApp.Web.csproj.
  6. dotnet restore.
  7. dotnet build –no-restore –configuration release.
Oct 4, 2022

Does Jenkins have an API? ›

Jenkins provides machine-consumable remote access API to its functionalities. Currently it comes in three flavors: XML. JSON with JSONP support.

Is Jenkins a CI tool or CD tool? ›

Jenkins is an open source continuous integration/continuous delivery and deployment (CI/CD) automation software DevOps tool written in the Java programming language. It is used to implement CI/CD workflows, called pipelines.

What can you automate with Jenkins? ›

Jenkins can be used to automate building an application, running tests, and deploying the application. It can be easily installed through native system packages, Docker, or even run standalone by any machine with a Java Runtime Environment (JRE) installed.

How do I create a Jenkins API token? ›

Creating a Jenkins API token

Sign in to Jenkins. Select your user name at the upper right, and then select Configure. Select Add new Token. Enter a name for the token and select Generate.

How do I get test results from Jenkins API? ›

You get to the Test Result page by clicking a build link on the Status or History page of your Jenkins project, or by clicking Test Result in the menu on the left. Click the image to enlarge it. At the top of the page, Jenkins displays summary information about the executed tests and their total execution time.

How do I generate API authentication token? ›

  1. Create an API token.
  2. Enable CORS.
  3. Find your Okta domain.
  4. Find your application credentials.
  5. Share app key credentials for IdPs.
  6. Set up SAML Tracer.
  7. Upgrade SAML apps to SHA256.
  8. Sign Okta certs with your own CA.

How to generate authorization token for API? ›

Generating an API token
  1. In Admin Center, click Apps and integrations in the sidebar, then select APIs > Zendesk APIs.
  2. Click the Settings tab, and make sure Token Access is enabled.
  3. Click the Add API token button to the right of Active API Tokens. ...
  4. Enter an API token description.

How to create GitLab API token for Jenkins? ›

In the configuration of your Jenkins job, in the GitLab configuration section, click 'Advanced' Click the 'Generate' button under the 'Secret Token' field. Copy the resulting token, and save the job configuration.

How do I authenticate Jenkins API? ›

If your Jenkins server requires authentication (and it SHOULD), you'll see a message saying "Authentication Required". The Jenkins API uses HTTP BASIC authentication and requires a username as well as an API token to connect. Then click the box named "Show API Token", and copy the token to your clipboard.

How to test an API request? ›

API testing flow is quite simple with three main steps:
  1. Send the request with necessary input data.
  2. Get the response having output data.
  3. Verify that the response returned as expected in the requirement.

How do I access Jenkins API? ›

Click your name on the top right corner on every page, then click "Configure" to see your API token. (The URL $root/me/configure is a good shortcut.) You can also change your API token from here. Note that Jenkins does not do any authorization negotiation.

How to create pipeline in Jenkins step by step with example? ›

Click the New Item menu within Jenkins. Provide a name for your new item (e.g. My-Pipeline) and select Multibranch Pipeline. Click the Add Source button, choose the type of repository you want to use and fill in the details. Click the Save button and watch your first Pipeline run.


1. Part1-SDET Essentials | Jenkins Build & Delivery Pipeline | CI & CD | How to create Jenkins Pipeline
(SDET- QA Automation Techie)
2. DevOps : CI/CD Pipeline For Deploying APIs - Part IV | Jenkins | Jenkinsfile
(Mule Technology Academy - Zero To Hero)
3. Postman 10.x Tutorial (Latest) - Configure & Run collection on Jenkins CI/CD pipeline using CLI
4. Building a Jenkins CI/CD Pipeline with WSO2 API Manager, WSO2 Virtual Learning
5. DevOps : CI/CD Pipeline For Deploying APIs - Part V | Jenkins | MuleSoft | Jenkinsfile
(Mule Technology Academy - Zero To Hero)
6. Jenkins Python Pipeline Tutorial
(Vincent Stevenson)


Top Articles
Latest Posts
Article information

Author: Annamae Dooley

Last Updated: 23/08/2023

Views: 6190

Rating: 4.4 / 5 (45 voted)

Reviews: 84% of readers found this page helpful

Author information

Name: Annamae Dooley

Birthday: 2001-07-26

Address: 9687 Tambra Meadow, Bradleyhaven, TN 53219

Phone: +9316045904039

Job: Future Coordinator

Hobby: Archery, Couponing, Poi, Kite flying, Knitting, Rappelling, Baseball

Introduction: My name is Annamae Dooley, I am a witty, quaint, lovely, clever, rich, sparkling, powerful person who loves writing and wants to share my knowledge and understanding with you.