In the modern digital era, many organizations adopt digital transformation technologies to keep pace with changes. APIs are one of the basic building blocks used in a digitally oriented organization. When the number of APIs managed by the organization grows, they need to have an automated process to handle the rapid API development process. Having a proper continuous integration and continuous deployment (CI/CD) process would give your organization an added advantage.
In this section, you will learn how to create an automated process using the WSO2 API Controller (apictl) with the WSO2 API Manager (WSO2 API-M). You will see how a solution is built using a version control system (Github), artifact repository (JFrog Artifactory) and CI/CD tool (Jenkins).
configuring the environment¶
Step 1 - Configure Jenkins¶
Before starting, you need to set up the Jenkins server on a remote machine that has public access. To use webhooks in the Github and Artifactory repository to trigger Jenkins jobs, you need a publicly accessible Jenkins server URL. If you are trying this on a local machine you may need to use a tool likesocketxpto create a tunnel.
In this section the instructions are tested on an ubuntu server setup onhttps://cloud.google.com/compute.
Log into the server and install next.
- Install jq using the command given below.
sudo apt-get install jq install theapictl tool. Extract to a location hereinafter referred to as
CTL_HOME.Install java.
Configure the Jenkins server by following the given instructionshere. You can choose to install the suggested default plugins. This will install commonly used plugins.
Register a webhook in the JForg Artifact repository to notify of changes. To capture these events and trigger a Jenkins job, install a generic webhook trigger plugin.
Set apictl home as an environment variable. I go to theManage Jenkinssection and selectconfigure system.
Underglobal propertiesselectenvironmental variablesand set initial APICTL as below and clickTo save.
Now the Jenkins server is configured.
Step 2 - Create Github repositories¶
In this section, two repositories are created to illustrate this scenario.
Source Repository -https://github.com/chamilaadhi/poc-cicd-source-repo
Deployment repository -https://github.com/chamilaadhi/poc-cicd-deployment-repo
(Video) How to Easily Build a Git Based CI/CD Pipeline for your APIs with Jenkins
The source repository contains the API-related files and metadata. The deployment repository contains settings related to each environment where these APIs are being deployed.
Step 2.1 - Configure the Source Repository¶
This section shows how to trigger a Jenkins job when a change is made to the repository. let's useWebhooks do Githubto trigger a Jenkins job.
- I go to thesourcerepository and under thesettings, selectWebhooksand add a webhook to the jenkins server.
Use the url of your jenkins server (e.g.
http://) as the URL andcontent typeas app/json.:8080/github-webhook/ UnderWhat events would you like to trigger this webhook?section, selectLet me select individual eventsand selectpull requestepush.
The source repository is now configured.
Now let's initialize the source repository. This will be done on the developer's local machine.
Observation
If you haven't configured the API Controller, configure it using theapictl tool
Clone the source repository. Navigate to the repository folder.
$ git clone https://github.com/chamilaadhi/poc-cicd-source-repo.git$ cd poc-cicd-source-repoRun the following command.
This will create a$ apictl vcs initvcs.yamlfile in the repository. Commit it to the source repository.
Now the source repository is configured.
Step 2.2 - Configure Deployment Repository¶
The deployment repository will be used to collect configurations for each deployment. To separate the configuration, we'll use git branches. Create a branch called 'dev' to hold configuration related to the dev environment.
Step 3 - Configure JFrog Artifactory¶
Before you start
Create a free cloud repository usinghttps://jfrog.com/start-free/#saas. After registration is complete, log in tohttps://
Create a user to upload artifacts. Log in using administrator credentials and go toIdentity and access→Usersand create a user. All roles are selected in this scenario.
Make the public repository accessible. Go toSecurity → Settingsand selectAllow anonymous access
Create a repository to upload the artifacts. Go toRepositoriesand selectadd repositories. selectGenerictype and provide a name (for example,
myrepo).After saving you should be able to access
https://..jfrog.io/artifactory/ / Try loading a sample resource. An example is given below.
$ touch resource.txt$ curl -u repouser:-X PUT https://xxxx.jfrog.io/artifactory/myrepo/ -T resource.txt Go to the repository URL to view the resource.
Configure a webhook to trigger the Jenkins job.
Create a webhook to trigger the Jenkins job when a new feature is uploaded to the repository. Go toIn general→Webhooksand create a new one.
URL -
http://:8080/generic-webhook-trigger/invoke?token=123** addset of eventsasArtifact has been deployed.
Select the repository in the new window that appears.
Step 4 - Configure API-M instances¶
Download WSO2 API Manager 4.2.0hereand start the instance. Set this in a separate instance like thedeveloperinstance.
Step 5 - Configure Jenkins Jobs¶
Step 5.1 - Configuration of global variables¶
not as shownConfiguring Jenkinssection, go toManage Jenkinssection and selectconfigure system. Underglobal propertiesselectenvironmental variablesand set the following variables with the details related to your environment.
| Variable | Description |
|---|---|
| APIM_DEV_HOST | IP or hostname of the dev instance (eg 10.100.5.96 ) |
| APIM_PROD_HOST | IP or hostname of the prod instance (eg 10.100.5.97 ) |
| ARTIFACTORY_HOST | Artificial hostname (ex: testrepo.jfrog.io) |
| ARTIFACTORY_PWD | Artifact loading user password |
| ARTIFACTORY_REPO | Repository name. (eg myrepo) |
| ARTIFACTORY_USER | Username of the artifact upload user |
Step 5.2 - Artifact Build and Upload Job Setup¶
create a jenkinsPipelineproject.
To useto bescript for the pipeline.
Observation
The default Jenkins server workspace URL is
/var/lib/jenkins/workspace/. If you installed the Jenkins server in a different location, change that path in the script accordingly.Undercreate triggersset the GitHub hook trigger toGITScm poll.
Save the configuration. Now the Jenkins job is configured to listen for any changes in the source repository and upload any new updates to the artifact repository.
To set it up, run a build job. Note that this will fail.
Step 5.3 - Artifact Deployment Job Setup¶
The artifact deployment job will listen for all new updates in the Artifactory repository and deploy the artifact with the settings in the deployment git repository to thedeveloperAPI-M environment.
Create a Jenkins pipeline usingto beroad map .
Save the script and run a build to configure. Note that this will fail.
To test that the webhook between the Artifactory and the Jenkins job is working, you can run the same upload test we did in the Configure JFrog Artifactory section
$ touch resource.txt$ curl -u repouser:-X PUT https://xxxxx.jfrog.io/artifactory/myrepo/ -T resource.txt
If the configuration is working, this will trigger a new build.
test the scenario¶
To test this setup, use the OpenAPI definition-based method to create the APIs.
Create API using an OAS definition. This will create a project namedPet store API. An example command is given below.
$apictl init PetstoreAPI --oas https://raw.githubusercontent.com/OAI/OpenAPI-Specification/main/examples/v2.0/yaml/petstore.yamlGenerate the deployment resource using the command given below.
$apictl gen deployment-dir --source /Path/To/PetstoreAPI --destination/poc-cicd-deployment-repo You will see the resources created with the following structure.
├── DeploymentArtifacts_SwaggerPetstore-1.0.0│ ├── api_meta.yaml│ ├── certificates│ └── params.yamlopen the
params.yamlfile and update the contents with the code given below.
The environment name is defined asenvironments: - name: development settings: endpoints: production: url: 'https://petstore.swagger.io/v2/' deploymentEnvironments: - displayOnDevportal: true deploymentEnvironment: Default deploymentVhost: localhostdeveloper. This configuration will be used for thedeveloperenvironment.Commit it to thedeveloperbranch in the deployment repository.
- Copy oPet store APIfolder to the git location of the source repository.
Create a file calledmeta.yamlwithinPet store APIfolder and add a version similar to the following.
This version is used to deploy the package created using thePet store APIAPI in the Artifactory repository.version: 1.0.0The following is the content of the PetstoreAPI. Commit all files to the git repository
├── PetstoreAPI│ ├── Definitions│ │ └── swagger.yaml│ ├── README.md│ ├── api.yaml│ ├── api_meta.yaml│ ├── deployment_environments. yaml│ └── meta .yamlWhen you commit to the source repository, the
CICD_ARTIFACT_UPLOADwill fire first and load the created component into the Artifactory. This is displayed in the console log of the Jenkins job.Once the package is uploaded to the Artifactory repository, it will trigger the Jenkins deployment job. The package being deployed to API-M is displayed in the logs.
Log in to the API Manager instance and you will see that the API has been created.
Let's update the API and see the changes. The API is inCREATEDstate. Let's publish this API by following the steps below.
- open thePetstoreAPI/api.yamlfile and change thelifeCycleStatusforPUBLISHED
OpenPetstoreAPI/meta.yamlfile and change the version to some new value (say 1.0.1).
Commit both files.
(Video) DevOps : CI/CD Pipeline For Deploying APIs - Part I | Jenkins | MuleSoft | GIT | GitHub
After committing the changes to the source repository you will see the jenkins jobs being triggered. Log in to the Publisher Portal and verify the changes.
If you go to the artifact repository and check thePet store APIyou will see two versions in the repository.
Principal
FAQs
How do I create a Jenkins pipeline for API? ›
- Create pipeline manually in Jenkins UI and test it.
- Export your pipeline using export import plugin or Jenkins jar to xml file or extract it from ${JENKINS_URL}/job/JOB_NAME/config. xml.
- Now use this xml file as a reference and create pipeline through API.
- Use below curl request to create pipeline.
- Fetch the OpenAPI specification file. ...
- Import the OpenAPI Specification file. ...
- Create an application plan and an application. ...
- Run the end-to-end tests. ...
- Promote the configuration to the production gateway. ...
- Read more.
On the Jenkins dashboard, click on “New Item”. After that, enter the name of your project, choose “Freestyle Project” from the list, and hit the enter key. The next step is to configure the project on the Jenkins dashboard. In the General tab, choose your GitHub Project, and enter the GitHub Repository URL.
Does Jenkins have a REST API? ›The jenkins-rest library is an object oriented Java project that provides access to the Jenkins REST API programmatically to some remote API Jenkins provides. It is built using the jclouds toolkit and can easily be extended to support more REST endpoints.
How do I trigger a Jenkins job with REST API? ›Setting Jenkins Job to respond REST API
The REST API feature can be enabled per Job basis. To enable REST API trigger for a Job, Navigate to Your JobName ->Configure -> Build triggers TAB and Check on 'Trigger build remotely'.
- The trigger.
- Code checkout.
- Compile the code.
- Run unit tests.
- Package the code.
- Run acceptance tests.
- Delivery or Deployment.
The limit is 5000 for me and I see that every second about 100 requests are going and I'm not sure which job/service is using up the bot's requests. Ideally my other jenkins jobs should take up maximum 1 request per minute.
How to use API key in Jenkins? ›- Go to your Jenkins instance and login with the user you want to use in the app;
- Open the user profile page;
- Click on Configure to open the user configuration page: If you already have an API Token: click on Show API Token;
- Install the Docker Pipeline plugin through the Manage Jenkins > Plugins page.
- After installing the plugin, restart Jenkins so that the plugin is ready to use.
- Copy one of the examples below into your repository and name it Jenkinsfile.
- Click the New Item menu within Jenkins.
Jenkins and GitHub Actions share some similarities in workflow configuration: Jenkins creates workflows using Declarative Pipelines, which are similar to GitHub Actions workflow files. Jenkins uses stages to run a collection of steps, while GitHub Actions uses jobs to group one or more steps or individual commands.
How to create CI CD pipeline in Git? ›
- Step 1: Create or choose a repository, and pick a project.
- Step 2: Open GitHub Actions in your repository to start building your CI/CD workflow.
- Step 3: Make changes to your code to trigger your CI/CD pipeline.
- Step 1 − Go to Manage Jenkins → Manage Plugins. ...
- Step 2 − Go to your Build project and click the Configure option. ...
- Step 3 − In the Deploy war/ear to a container section, enter the required details of the server on which the files need to be deployed and click on the Save button.
- Send API commands to the server and validate responses.
- Use values from responses as parameters in test steps.
- Combine REST API and recorded UI steps within the same automated test to achieve end-to-end testing.
- Analyze reports.
The majority of HTTP APIs are on the verge of becoming completely RESTful. But not all HTTP APIs are REST APIs. To be termed a REST API, the API must meet the following architectural requirements: Client-Server: A server oversees the application's data and state in REST applications.
How to access Jenkins API with token? ›- Navigate to Your Username > Configure > API Token.
- Click Add new Token.
- Click Generate.
- Copy the API token that is generated.
- In the web app, navigate to User Icon My Profile User Settings.
- In the section API Access, click Create API User Token.
- Enter a Description to help you identify the key later.
- Click Create Key.
Create Credentials
According to the documentation, a POST request must be sent to $JENKINS_URL/<path to context>/credentials/store/<store id>/domain/<domain name>/createCredentials with the credentials XML configuration as body.
- Create a Jenkins build job and enable the Trigger builds remotely checkbox.
- Provide an authentication token; This can be any text string of your choice.
- Invoke the Jenkins build URL to remotely trigger the build job.
A Jenkinsfile can be written using two types of syntax - Declarative and Scripted. Declarative and Scripted Pipelines are constructed fundamentally differently. Declarative Pipeline is a more recent feature of Jenkins Pipeline which: provides richer syntactical features over Scripted Pipeline syntax, and.
What are the four parts of the CI CD pipeline? ›- Source. Development teams use version control systems, such as Git. ...
- Build. The build stage combines the source code with dependencies. ...
- Test. ...
- Deploy.
What is a real time example of CI CD pipeline? ›
CI/CD pipelines real-life example
Netflix and Facebook are the prime examples that have successfully implemented CI/CD pipelines to deliver software updates and features quickly and with high quality. Netflix uses a custom-built platform called Spinnaker to manage its CI/CD pipelines.
Overview. A continuous integration and continuous deployment (CI/CD) pipeline is a series of steps that must be performed in order to deliver a new version of software. CI/CD pipelines are a practice focused on improving software delivery throughout the software development life cycle via automation.
What are the stages of CI CD in Jenkins? ›There are four stages of a CI/CD pipeline 1) Source Stage, 2) Build Stage, 3) Test Stage, 4) Deploy Stage. Important CI/CD tools are Jenkins, Bambo, and Circle CI. CI/CD pipeline can improve reliability.
What are the five stages of a development pipeline? ›A continuous delivery pipeline consists of five main phases—build/develop, commit, test, stage, and deploy.
How many pipelines can be created in Jenkins? ›1 Answer. There are two types of pipelines in Jenkins: Declarative. Scripted.
How many ways we can create pipelines in Jenkins? ›Additionally, Jenkins provides two ways to create pipelines: Scripted and Declarative. Finally, the Jenkins pipeline gives the flexibility to define various stages and steps programmatically.
What is the difference between Jenkins project and pipeline? ›The major difference between any Jenkins job and a Jenkins Pipeline Job is that the Pipeline Scripted job runs on the Jenkins master. This uses a lightweight executor which uses only some resources to translate in the master to atomic commands that execute or send to the agents.
How many requests can a REST API handle? ›By default, it is set to 100 requests per 100 seconds per user and can be adjusted to a maximum value of 1,000. But the number of requests to the API is restricted to a maximum of 10 requests per second per user.
What is the API limit for Jenkins GitHub? ›One of our service account hit Git Api rate limit of 5000 per hour. It is used in jenkins and Aws code pipeline.
What is the size limit of API? ›A web server such as Apache allows a payload size of 2 GB. Google's Photo APIs allow photos with a maximum file size of 200 MB.
What is the best way to pass API key? ›
You can pass the API key via Basic Auth as either the username or password. Most implementations pair the API key with a blank value for the unused field (username or password). You will need to base64-encode the `username:password` content, but most request libraries do this for you.
Where do I put my API key? ›Don't store your API key directly in your code.
Instead, store your API key and secret directly in your environment variables. Environment variables are dynamic objects whose values are set outside of the application. This will let you access them easily (by using the os.
An application programming interface (API) key is a code used to identify and authenticate an application or user. API keys are available through platforms, such as a white-labeled internal marketplace. They also act as a unique identifier and provide a secret token for authentication purposes.
Why Gitlab is better than Jenkins? ›Both Jenkins and Gitlab are designed to serve different requirements. While Jenkins boasts of a large plugin shelf, Gitlab is a comprehensive DevOps tool. While multiple plugins do your job efficiently, integration and management of these plugins might become a challenge when the project scales up.
How to set up Jenkins for continuous integration? ›- Prerequisites.
- Add the Jenkins User to the Docker Group.
- Create a Personal Access Token in GitHub.
- Add the GitHub Personal Access Token to Jenkins.
- Set Up Jenkins Access to GitHub.
- Set Up the Demonstration Application in your GitHub Account.
Always Put Build Workflow First — The Build Workflow should always be the first stage in the Pipeline. This enables the rest of the Pipeline to use the artifact it builds and collects.
What are the 3 types of pipelines in Jenkins? ›Different Types of Jenkins CI/CD Pipelines. Scripted Pipeline. Declarative Pipeline. The Concept of Stages in Jenkins Pipeline.
What are 3 important stages in pipeline in Jenkins? ›Stage: This block contains a series of steps in a pipeline. i.e., build, test, and deploy processes all come together in a stage. Generally, a stage block visualizes the Jenkins pipeline process.
Which is better Jenkins or GitHub? ›When using GitHub Actions, you are essentially bound to Github's source code management platform. You may keep your code on any repository, including Gitlab, BitBucket, and others, by using Jenkins. Compared to GitHub Actions, Jenkins is more experienced and established.
How to implement CI CD pipeline by Jenkins and Docker? ›- Jenkins-Docker Continuous Integration & Continuous Deployment Pipeline. ...
- chmod 400 /var/run/docker.sock. ...
- unix://var/run/docker.sock. ...
- Pipeline syntax-> select withcredentials from drop down -> secret text → select dockerHubaccount -> generate script. ...
- chmod 400 /var/run/docker.sock.
How to create pipeline in Jenkins with GitLab? ›
In Jenkins, go to Manage Jenkins → Configure System and scroll down to the Gitlab section. Enter in a name for the connection, the URL to your GitLab instance, and then next to Credentials, click Add → Jenkins. This is where you will add your GitLab token from Step 2. Next to Kind, select GitLab API Token.
How to create CI CD pipeline with YAML? ›- Case Study.
- dotnet new sln -o Test_CI_CDPipeline.
- cd Test_CI_CDPipeline.
- dotnet new mvc -n Test_CI_CDPipeline.
- dotnet sln HelloWorldApp.sln add HelloWorldApp.Web\HelloWorldApp.Web.csproj.
- dotnet restore.
- dotnet build –no-restore –configuration release.
Jenkins provides machine-consumable remote access API to its functionalities. Currently it comes in three flavors: XML. JSON with JSONP support.
Is Jenkins a CI tool or CD tool? ›Jenkins is an open source continuous integration/continuous delivery and deployment (CI/CD) automation software DevOps tool written in the Java programming language. It is used to implement CI/CD workflows, called pipelines.
What can you automate with Jenkins? ›Jenkins can be used to automate building an application, running tests, and deploying the application. It can be easily installed through native system packages, Docker, or even run standalone by any machine with a Java Runtime Environment (JRE) installed.
How do I create a Jenkins API token? ›Creating a Jenkins API token
Sign in to Jenkins. Select your user name at the upper right, and then select Configure. Select Add new Token. Enter a name for the token and select Generate.
You get to the Test Result page by clicking a build link on the Status or History page of your Jenkins project, or by clicking Test Result in the menu on the left. Click the image to enlarge it. At the top of the page, Jenkins displays summary information about the executed tests and their total execution time.
How do I generate API authentication token? ›- Create an API token.
- Enable CORS.
- Find your Okta domain.
- Find your application credentials.
- Share app key credentials for IdPs.
- Set up SAML Tracer.
- Upgrade SAML apps to SHA256.
- Sign Okta certs with your own CA.
- In Admin Center, click Apps and integrations in the sidebar, then select APIs > Zendesk APIs.
- Click the Settings tab, and make sure Token Access is enabled.
- Click the Add API token button to the right of Active API Tokens. ...
- Enter an API token description.
In the configuration of your Jenkins job, in the GitLab configuration section, click 'Advanced' Click the 'Generate' button under the 'Secret Token' field. Copy the resulting token, and save the job configuration.
How do I authenticate Jenkins API? ›
If your Jenkins server requires authentication (and it SHOULD), you'll see a message saying "Authentication Required". The Jenkins API uses HTTP BASIC authentication and requires a username as well as an API token to connect. Then click the box named "Show API Token", and copy the token to your clipboard.
How to test an API request? ›- Send the request with necessary input data.
- Get the response having output data.
- Verify that the response returned as expected in the requirement.
Click your name on the top right corner on every page, then click "Configure" to see your API token. (The URL $root/me/configure is a good shortcut.) You can also change your API token from here. Note that Jenkins does not do any authorization negotiation.
How to create pipeline in Jenkins step by step with example? ›Click the New Item menu within Jenkins. Provide a name for your new item (e.g. My-Pipeline) and select Multibranch Pipeline. Click the Add Source button, choose the type of repository you want to use and fill in the details. Click the Save button and watch your first Pipeline run.