In this blog, we will implement the Jenkins CI/CD Pipeline for Python applications. We will be building a pipeline as code, aka Declarative pipeline.
What is the declarative pipeline? The pipeline is the new feature of Jenkins, where we can write the Jenkins job as a sequence of steps. It is written in groovy. Since it is a script we can keep it along with our code and can be easily replicated in other projects. We can also invoke Jenkins plugins in pipeline scripts.
Directives: Jenkins pipeline provides a list of directives that will be used in defining various operations.
Agent: Specifies in which environment the job will be run.
Stages: A collection of stages or a working bundle.
Stage: A set of steps to perform a certain operation.
Steps: One or more steps will be executed under each stage. Typically, it is a command.
Post: it will run on job completion. we can trigger different options based on build status if successful, unstable, or failure.
Step 1: Choose a server and set working directories. By default, Jenkins chooses the home directory of the user to run jobs. We can change this if required.
agent {
node {
label "my-local-suer'
customWorkspace "/projects/project"
}
}
Agent: specifies which environment we need to run the job. It can be server/host, docker, or Kubernetes.
To run in any available server or host, we can specify agent as "any."
To run a specific server, we need to specify it in "label."
To add extra parameters like a custom working directory, we need to use node. "customWorkspace" is where our code will be checked out, and steps will be performed.
Step 2: Checking out code: To checkout the code, we would need to use the GIT plugin. We need to specify the following things. - branch: what branches we need to checkout - credentialsId: if we use the HTTP method to checkout, we need to specify the credentials to checkout. If it SSH, it's not required. - url: the repo URL.
stage("Checkout Code") {
steps {
script {
git branch: "master",
credentialsId: 'my-credentials',
url: 'https://user@github.org/myproject/sample-repo.git'
}
}
}
The code will be checked out in the current working directory.
Step 3: Installing the python packages: To install the python packages, we can use the shell command.
stage('Installing packages') {
steps {
script {
sh 'pip -r requirements.txt'
}
}
}
Step 4: Running Static Code Analysis: We choose Pylint to check the static code analysis and warnings next Generation plugin to analyze the Pylint report. The plugin has features to mark the build as unstable or failing based on the scores.
stage('Static Code Checking') {
steps {
script {
sh 'find . -name \\*.py | xargs pylint --load-plugins=pylint_django -f parseable | tee pylint.log'
recordIssues(
tool: pyLint(pattern: 'pylint.log'),
failTotalHigh: 10,
)
}
}
}
We need to use shell command to run the Pylint for all python files in the project folder.
Record Issues method of warnings next-generation plugin, and we have also specified the types of log and parameters. The build fails if the High category issues count is more than or equal to 10. there are multiple options available. Please visit https://www.jenkins.io/doc/pipeline/steps/warnings-ng/ for more information on configurations.
Step 5: Running unit test cases: We can use any test tools like a pytest or nose to run the unit test. To publish the report on Jenkins we choose "CoberturaPublisher" plugin. There is no straight way to use the plugin in the pipeline. We need to invoke its class.
stage('Running Unit tests') {
steps {
script {
sh 'python3.7 manage.py test --keepdb --with-xunit --xunit-file=pyunit.xml --cover-xml --cover-xml-file=cov.xml tests/*.py || true'
step([$class: 'CoberturaPublisher',
coberturaReportFile: "cov.xml",
])
junit "pyunit.xml"
}
}
We need to specify the report file, and it must be "XML" file.
We also have options to fail the build if tests fail or when the coverage is less. for detailed information on available options, please visit. https://www.jenkins.io/doc/pipeline/steps/cobertura/
To publish the test results we use Junit plugin
The pipeline script -
pipeline {
agent {
node {
label 'my_local_server'
customWorkspace '/projects/'
}
}
stages {
stage('Checkout project') {
steps {
script {
git branch: "master",
credentialsId: 'my-credentials',
url: 'https://user@github.org/myproject/sample-repo.git'
}
}
}
stage('Installing packages') {
steps {
script {
sh 'pip install -r requirements.txt'
}
}
}
stage('Static Code Checking') {
steps {
script {
sh 'find . -name \\*.py | xargs pylint -f parseable | tee pylint.log'
recordIssues(
tool: pyLint(pattern: 'pylint.log'),
unstableTotalHigh: 100,
)
}
}
}
stage('Running Unit tests') {
steps {
script {
sh 'pytest --with-xunit --xunit-file=pyunit.xml --cover-xml --cover-xml-file=cov.xml tests/*.py || true'
step([$class: 'CoberturaPublisher',
coberturaReportFile: "cov.xml",
onlyStable: false,
failNoReports: true,
failUnhealthy: false,
failUnstable: false,
autoUpdateHealth: true,
autoUpdateStability: true,
zoomCoverageChart: true,
maxNumberOfBuilds: 10,
lineCoverageTargets: '80, 80, 80',
conditionalCoverageTargets: '80, 80, 80',
classCoverageTargets: '80, 80, 80',
fileCoverageTargets: '80, 80, 80',
])
junit "pyunit.xml"
}
}
}
}
}
Automating the pipeline using Python -
This code uses the Jenkins library to connect to a Jenkins server, create a new pipeline job, build the job, and check the status of the pipeline. The Jenkins library needs to be installed via pip before use.
import jenkins
# Connect to Jenkins
server = jenkins.Jenkins('http://jenkins_url:port', username='your_username', password='your_password')
# Create a new pipeline job
server.create_job('pipeline_job_name', jenkins.EMPTY_CONFIG_XML)
# Build the pipeline job
server.build_job('pipeline_job_name')
# Wait for the pipeline to complete
while server.get_build_info('pipeline_job_name')['result'] == None:
sleep(5)
# Check the pipeline status
result = server.get_build_info('pipeline_job_name')['result']
if result == 'SUCCESS':
print("Pipeline completed successfully!")
else:
print("Pipeline failed.")
This is a basic example and can be modified or added with steps, triggers, and other functionality as per the requirement. You should also note that this code is for python 2, and for python 3, the library is JenkinsAPI.
You might also need to configure your Jenkins server to allow remote API access, and also make sure that you have the correct permissions to create jobs and perform other actions on the Jenkins server.
Hope it helps, thanks for reading this.
See you soon in the next blog ...