# Jenkins Pipeline Complete Guide
Jenkins pipelines are the backbone of most enterprise CI/CD systems. Whether you're building microservices, deploying to Kubernetes, or managing monorepo builds, understanding Jenkins pipelines deeply separates a functional setup from a production-grade one.
This guide walks through everything you need — from basic syntax to shared libraries, parallel execution, and production-ready examples. Every concept includes working code you can drop into your projects today.
---
1. Declarative vs Scripted Pipeline
Jenkins supports two pipeline syntaxes. Here's when to use each.
Declarative Pipeline
Declarative is the modern, opinionated syntax. It enforces structure, which makes pipelines easier to read and maintain. Use this for 90% of your pipelines.
pipeline {
agent any
stages {
stage('Build') {
steps {
sh 'npm install'
sh 'npm run build'
}
}
stage('Test') {
steps {
sh 'npm test'
}
}
stage('Deploy') {
steps {
sh './deploy.sh'
}
}
}
post {
success {
echo 'Pipeline succeeded!'
}
failure {
echo 'Pipeline failed!'
}
}
}
Scripted Pipeline
Scripted pipelines use raw Groovy. They're more flexible but harder to maintain. Use them when you need complex control flow that declarative can't handle.
node {
try {
stage('Checkout') {
checkout scm
}
stage('Build') {
sh 'mvn clean package -DskipTests'
}
stage('Test') {
sh 'mvn test'
junit '*/target/surefire-reports/.xml'
}
stage('Deploy') {
if (env.BRANCH_NAME == 'main') {
sh './deploy-prod.sh'
} else {
sh './deploy-staging.sh'
}
}
} catch (Exception e) {
currentBuild.result = 'FAILURE'
throw e
} finally {
cleanWs()
}
}
When to Choose Which
| Feature | Declarative | Scripted |
|---|---|---|
| Readability | High | Medium |
| Flexibility | Structured | Unlimited |
| Error handling | Built-in <code class="inline-code">post</code> | Manual try/catch |
| Restart from stage | Supported | Not supported |
| Blue Ocean compatibility | Full | Partial |
| Learning curve | Lower | Higher |
Rule of thumb: Start with declarative. Switch to scripted only when you hit a wall — complex loops, dynamic stage generation, or heavy Groovy logic.
---
2. Jenkinsfile Syntax Deep Dive
Stages, Steps, and Post
Every declarative pipeline has this skeleton:
pipeline {
agent any
environment {
APP_NAME = 'my-service'
DEPLOY_ENV = 'staging'
}
parameters {
string(name: 'BRANCH', defaultValue: 'main', description: 'Branch to build')
choice(name: 'ENV', choices: ['dev', 'staging', 'prod'], description: 'Target environment')
booleanParam(name: 'RUN_TESTS', defaultValue: true, description: 'Run test suite')
}
stages {
stage('Checkout') {
steps {
checkout scm
}
}
stage('Build') {
steps {
sh "echo Building ${env.APP_NAME}..."
sh 'make build'
}
}
stage('Test') {
when {
expression { params.RUN_TESTS == true }
}
steps {
sh 'make test'
}
post {
always {
junit '*/test-results/.xml'
}
}
}
stage('Deploy') {
when {
branch 'main'
}
steps {
sh "deploy --env ${params.ENV}"
}
}
}
post {
always {
cleanWs()
}
success {
slackSend channel: '#deploys', message: "✅ ${env.APP_NAME} deployed"
}
failure {
slackSend channel: '#deploys', message: "❌ ${env.APP_NAME} failed"
}
}
}
Key Directives Explained
agent— Where the pipeline runs. Useany, a label, or Docker.environment— Set env vars available to all stages. Supports credentials binding.parameters— User inputs at build time. Types:string,choice,booleanParam,password.when— Conditional stage execution. Supportsbranch,expression,environment,changeset.post— Actions after stages complete. Conditions:always,success,failure,unstable,changed.
---
3. Shared Libraries
Shared libraries let you reuse pipeline code across multiple Jenkinsfiles. This is how mature teams avoid copy-pasting 200-line Jenkinsfiles across 50 repos.
Directory Structure
jenkins-shared-library/
├── vars/
│ ├── buildApp.groovy # Global pipeline steps
│ ├── deployToK8s.groovy # Reusable deploy function
│ └── notifySlack.groovy # Notification helper
├── src/
│ └── org/
│ └── devopskit/
│ └── Pipeline.groovy # OOP-style classes
├── resources/
│ └── templates/
│ └── deployment.yaml # Static resources
└── Jenkinsfile # Optional: CI for the library itself
Writing a Shared Library Step
vars/buildApp.groovy:
def call(Map config = [:]) {
def appName = config.appName ?: 'unknown'
def buildCmd = config.buildCmd ?: 'make build'
def testCmd = config.testCmd ?: 'make test'
pipeline {
agent any
stages {
stage('Build') {
steps {
sh buildCmd
}
}
stage('Test') {
steps {
sh testCmd
}
}
stage('Package') {
steps {
sh "docker build -t ${appName}:${env.BUILD_NUMBER} ."
}
}
}
}
}
Using the Shared Library
In your Jenkinsfile:
@Library('my-shared-library@main') _
buildApp(
appName: 'user-service',
buildCmd: 'npm run build',
testCmd: 'npm test'
)
Configuring in Jenkins
- Name: my-shared-library
- Default version: main
- Source: Git repo URL
- Credentials: your Git token
Versioning Strategy
// Pin to a tag for stability
@Library('my-shared-library@v2.1.0') _
// Use a branch for development
@Library('my-shared-library@feature/new-deploy') _
// Use latest (risky in production)
@Library('my-shared-library') _
---
4. Multi-Branch Pipelines
Multi-branch pipelines automatically discover branches in your repo and create pipeline jobs for each one. This is the standard for teams using feature branches.
Setup
JenkinsfileJenkinsfile with Branch-Specific Logic
pipeline {
agent any
stages {
stage('Build') {
steps {
sh 'npm ci'
sh 'npm run build'
}
}
stage('Test') {
steps {
sh 'npm test'
}
}
stage('Deploy to Staging') {
when {
branch 'develop'
}
steps {
sh './deploy.sh staging'
}
}
stage('Deploy to Production') {
when {
branch 'main'
}
steps {
input message: 'Deploy to production?', ok: 'Deploy'
sh './deploy.sh production'
}
}
}
}
Branch Discovery Configuration
In the job config, set scan triggers:
- Scan interval: Every 1-5 minutes or use webhooks
- Branch filtering: Include
main,develop,feature/,release/ - Orphaned branch strategy: Delete builds for deleted branches after X days
---
5. Parallel Stages for Faster Builds
Running stages in parallel dramatically cuts build time. Use this for independent tasks like running tests across multiple platforms or services.
pipeline {
agent none
stages {
stage('Build') {
agent { label 'builder' }
steps {
sh 'npm ci'
sh 'npm run build'
stash includes: 'dist/**', name: 'build-artifacts'
}
}
stage('Test') {
parallel {
stage('Unit Tests') {
agent { label 'test-node' }
steps {
unstash 'build-artifacts'
sh 'npm run test:unit'
}
post {
always {
junit 'reports/unit/*.xml'
}
}
}
stage('Integration Tests') {
agent { label 'test-node' }
steps {
unstash 'build-artifacts'
sh 'npm run test:integration'
}
post {
always {
junit 'reports/integration/*.xml'
}
}
}
stage('E2E Tests') {
agent { label 'browser-node' }
steps {
unstash 'build-artifacts'
sh 'npm run test:e2e'
}
post {
always {
junit 'reports/e2e/*.xml'
}
}
}
stage('Security Scan') {
agent { label 'security' }
steps {
sh 'trivy fs --exit-code 1 --severity HIGH,CRITICAL .'
}
}
}
}
stage('Deploy') {
agent { label 'deployer' }
when {
branch 'main'
}
steps {
unstash 'build-artifacts'
sh './deploy.sh'
}
}
}
}
Key Points About Parallel Stages
- Use
agent noneat the top when parallel stages use different agents stash/unstashshares files between nodes- If one parallel stage fails, others continue by default (use
failFast trueto abort all on first failure)
stage('Test') {
failFast true
parallel {
// stages here will all abort if one fails
}
}
---
6. Credentials Management in Pipelines
Never hardcode secrets. Jenkins credentials store integrates directly with pipelines.
Binding Credentials to Environment Variables
pipeline {
agent any
environment {
DOCKER_CREDS = credentials('docker-hub-credentials')
SLACK_TOKEN = credentials('slack-webhook-token')
SSH_KEY = credentials('deploy-ssh-key')
}
stages {
stage('Docker Login') {
steps {
// DOCKER_CREDS_USR and DOCKER_CREDS_PSW are auto-created
sh 'echo $DOCKER_CREDS_PSW | docker login -u $DOCKER_CREDS_USR --password-stdin'
}
}
stage('Deploy via SSH') {
steps {
sshagent(['deploy-ssh-key']) {
sh 'ssh user@server "cd /app && git pull && docker compose up -d"'
}
}
}
}
}
Using withCredentials Block
For scoped credential access within a single stage:
stage('Push to Registry') {
steps {
withCredentials([
usernamePassword(
credentialsId: 'ecr-credentials',
usernameVariable: 'AWS_ACCESS_KEY_ID',
passwordVariable: 'AWS_SECRET_ACCESS_KEY'
)
]) {
sh '''
aws ecr get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com
docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/my-app:${BUILD_NUMBER}
'''
}
}
}
Credential Types
| Type | Use Case | Access Pattern |
|---|---|---|
| Username/Password | Docker registries, APIs | <code class="inline-code">_USR</code> and <code class="inline-code">_PSW</code> suffixes |
| Secret text | Tokens, API keys | Direct variable |
| Secret file | Kubeconfig, cert files | File path variable |
| SSH key | Remote server access | <code class="inline-code">sshagent</code> wrapper |
| Certificate | TLS/mTLS | PKCS#12 file path |
---
7. Docker Agent in Jenkins Pipelines
Running builds inside Docker containers ensures consistent environments and eliminates "works on my machine" issues.
Basic Docker Agent
pipeline {
agent {
docker {
image 'node:20-alpine'
args '-v /tmp:/tmp'
}
}
stages {
stage('Install') {
steps {
sh 'node --version'
sh 'npm ci'
}
}
stage('Build') {
steps {
sh 'npm run build'
}
}
stage('Test') {
steps {
sh 'npm test'
}
}
}
}
Per-Stage Docker Agents
Different stages can use different images:
pipeline {
agent none
stages {
stage('Build Frontend') {
agent {
docker { image 'node:20-alpine' }
}
steps {
sh 'npm ci && npm run build'
stash includes: 'dist/**', name: 'frontend'
}
}
stage('Build Backend') {
agent {
docker { image 'maven:3.9-eclipse-temurin-21' }
}
steps {
sh 'mvn clean package -DskipTests'
stash includes: 'target/*.jar', name: 'backend'
}
}
stage('Build Docker Image') {
agent { label 'docker-host' }
steps {
unstash 'frontend'
unstash 'backend'
sh 'docker build -t my-app:${BUILD_NUMBER} .'
}
}
}
}
Using a Custom Dockerfile
pipeline {
agent {
dockerfile {
filename 'Dockerfile.ci'
dir 'ci'
args '-v $HOME/.m2:/root/.m2'
additionalBuildArgs '--build-arg HTTP_PROXY=http://proxy:3128'
}
}
stages {
stage('Build') {
steps {
sh 'make build'
}
}
}
}
Docker-in-Docker (DinD)
When your pipeline needs to build Docker images inside a Docker agent:
agent {
docker {
image 'docker:24-dind'
args '--privileged -v /var/run/docker.sock:/var/run/docker.sock'
}
}
Security note: Mounting the Docker socket gives the container full host access. In production, consider using Kaniko or Buildah for rootless image builds.
---
8. Production-Ready Jenkinsfile Examples
Example 1: Node.js Application — Build, Test, and Deploy
pipeline {
agent none
environment {
REGISTRY = '123456789.dkr.ecr.us-east-1.amazonaws.com'
APP_NAME = 'user-service'
SLACK_CHANNEL = '#team-deploys'
}
options {
timeout(time: 30, unit: 'MINUTES')
disableConcurrentBuilds()
buildDiscarder(logRotator(numToKeepStr: '20'))
}
stages {
stage('Checkout') {
agent any
steps {
checkout scm
script {
env.GIT_COMMIT_SHORT = sh(
script: 'git rev-parse --short HEAD',
returnStdout: true
).trim()
env.IMAGE_TAG = "${env.BRANCH_NAME}-${env.GIT_COMMIT_SHORT}-${env.BUILD_NUMBER}"
}
}
}
stage('Install & Lint') {
agent { docker { image 'node:20-alpine' } }
steps {
sh 'npm ci'
sh 'npm run lint'
}
}
stage('Test') {
agent { docker { image 'node:20-alpine' } }
steps {
sh 'npm ci'
sh 'npm run test:ci -- --coverage'
}
post {
always {
junit 'reports/junit.xml'
publishHTML(target: [
reportName: 'Coverage Report',
reportDir: 'coverage/lcov-report',
reportFiles: 'index.html'
])
}
}
}
stage('Build & Push Image') {
agent { label 'docker-host' }
when {
anyOf {
branch 'main'
branch 'develop'
}
}
steps {
withCredentials([
string(credentialsId: 'aws-account-id', variable: 'AWS_ACCOUNT')
]) {
sh '''
aws ecr get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin ${REGISTRY}
docker build -t ${REGISTRY}/${APP_NAME}:${IMAGE_TAG} .
docker push ${REGISTRY}/${APP_NAME}:${IMAGE_TAG}
'''
}
}
}
stage('Deploy to Staging') {
agent { label 'deployer' }
when { branch 'develop' }
steps {
withCredentials([file(credentialsId: 'kubeconfig-staging', variable: 'KUBECONFIG')]) {
sh """
kubectl set image deployment/${APP_NAME} \
${APP_NAME}=${REGISTRY}/${APP_NAME}:${IMAGE_TAG} \
--namespace=staging
kubectl rollout status deployment/${APP_NAME} \
--namespace=staging --timeout=120s
"""
}
}
}
stage('Deploy to Production') {
agent { label 'deployer' }
when { branch 'main' }
steps {
input message: 'Deploy to production?', ok: 'Yes, deploy it'
withCredentials([file(credentialsId: 'kubeconfig-prod', variable: 'KUBECONFIG')]) {
sh """
kubectl set image deployment/${APP_NAME} \
${APP_NAME}=${REGISTRY}/${APP_NAME}:${IMAGE_TAG} \
--namespace=production
kubectl rollout status deployment/${APP_NAME} \
--namespace=production --timeout=180s
"""
}
}
}
}
post {
success {
slackSend channel: env.SLACK_CHANNEL, color: 'good',
message: "✅ ${APP_NAME} #${BUILD_NUMBER} succeeded | ${BRANCH_NAME}"
}
failure {
slackSend channel: env.SLACK_CHANNEL, color: 'danger',
message: "❌ ${APP_NAME} #${BUILD_NUMBER} failed | <${BUILD_URL}|View Logs>"
}
always {
cleanWs()
}
}
}
Example 2: Java Maven Application — Build, Test, and Deploy
pipeline {
agent none
environment {
MAVEN_OPTS = '-Xmx1024m'
REGISTRY = 'registry.company.com'
APP_NAME = 'payment-service'
}
options {
timeout(time: 45, unit: 'MINUTES')
buildDiscarder(logRotator(numToKeepStr: '15'))
timestamps()
}
stages {
stage('Compile') {
agent {
docker {
image 'maven:3.9-eclipse-temurin-21'
args '-v $HOME/.m2:/root/.m2'
}
}
steps {
sh 'mvn clean compile -B'
}
}
stage('Unit Tests') {
agent {
docker {
image 'maven:3.9-eclipse-temurin-21'
args '-v $HOME/.m2:/root/.m2'
}
}
steps {
sh 'mvn test -B'
}
post {
always {
junit '*/target/surefire-reports/.xml'
jacoco(
execPattern: '**/target/jacoco.exec',
classPattern: '**/target/classes',
sourcePattern: '**/src/main/java'
)
}
}
}
stage('Integration Tests') {
agent {
docker {
image 'maven:3.9-eclipse-temurin-21'
args '-v $HOME/.m2:/root/.m2 --network=host'
}
}
steps {
sh 'mvn verify -P integration-tests -B'
}
post {
always {
junit '*/target/failsafe-reports/.xml'
}
}
}
stage('SonarQube Analysis') {
agent {
docker {
image 'maven:3.9-eclipse-temurin-21'
args '-v $HOME/.m2:/root/.m2'
}
}
steps {
withSonarQubeEnv('sonarqube-server') {
sh 'mvn sonar:sonar -B'
}
}
}
stage('Quality Gate') {
steps {
timeout(time: 5, unit: 'MINUTES') {
waitForQualityGate abortPipeline: true
}
}
}
stage('Package & Push') {
agent { label 'docker-host' }
when {
anyOf {
branch 'main'
branch 'release/*'
}
}
steps {
sh 'mvn package -DskipTests -B'
sh """
docker build -t ${REGISTRY}/${APP_NAME}:${BUILD_NUMBER} .
docker push ${REGISTRY}/${APP_NAME}:${BUILD_NUMBER}
"""
}
}
stage('Deploy') {
agent { label 'deployer' }
when { branch 'main' }
steps {
withCredentials([file(credentialsId: 'kubeconfig-prod', variable: 'KUBECONFIG')]) {
sh """
helm upgrade --install ${APP_NAME} ./helm-chart \
--set image.tag=${BUILD_NUMBER} \
--set image.repository=${REGISTRY}/${APP_NAME} \
--namespace=production \
--wait --timeout=300s
"""
}
}
}
}
post {
failure {
emailext(
subject: "FAILED: ${APP_NAME} #${BUILD_NUMBER}",
body: "Build failed. Check: ${BUILD_URL}",
to: 'team@company.com'
)
}
always {
cleanWs()
}
}
}
---
9. Pipeline Best Practices (10 Items)
These come from running Jenkins at scale across hundreds of pipelines. Each one solves a real pain point.
1. Keep Jenkinsfiles Thin
Move logic into shared libraries. Your Jenkinsfile should be a configuration file, not a script.
// Good: thin Jenkinsfile
@Library('company-pipeline@v3.0.0') _
standardPipeline(
appName: 'my-service',
language: 'nodejs',
deployTo: ['staging', 'production']
)
2. Pin Your Tool Versions
Never use latest tags in Docker agents. Version drift causes mysterious build failures.
// Bad
agent { docker { image 'node:latest' } }
// Good
agent { docker { image 'node:20.11-alpine3.19' } }
3. Set Timeouts on Everything
Prevent zombie builds that hang forever and block your executor pool.
options {
timeout(time: 30, unit: 'MINUTES')
}
// Or per-stage
stage('Deploy') {
options {
timeout(time: 5, unit: 'MINUTES')
}
steps { / ... / }
}
4. Use <code class="inline-code">options</code> Block for Build Hygiene
options {
buildDiscarder(logRotator(numToKeepStr: '20', artifactNumToKeepStr: '5'))
disableConcurrentBuilds()
timestamps()
ansiColor('xterm')
}
5. Fail Fast in Parallel Stages
Don't waste compute on a doomed build:
stage('Tests') {
failFast true
parallel { / ... / }
}
6. Cache Dependencies
Mount dependency caches to speed up builds:
agent {
docker {
image 'node:20-alpine'
args '-v /var/cache/npm:/root/.npm'
}
}
7. Use <code class="inline-code">when</code> Conditions to Skip Unnecessary Work
stage('Deploy Docs') {
when {
changeset 'docs/**'
}
steps {
sh 'mkdocs build && mkdocs gh-deploy'
}
}
8. Always Clean Workspace in Post
Prevent leftover files from corrupting the next build:
post {
always {
cleanWs()
}
}
9. Use Replay for Debugging
Instead of committing 50 times to test a pipeline change, use Replay in the Jenkins build page. It lets you edit the pipeline on the fly without pushing code.
10. Externalize Configuration
Don't hardcode environment-specific values. Use Jenkins parameters, credential stores, or config files:
stage('Setup') {
steps {
script {
def config = readJSON file: 'ci/config.json'
env.DEPLOY_TARGET = config.environments[params.ENV].url
}
}
}
---
10. Troubleshooting Common Pipeline Failures
Scripts not permitted to use method
Problem: Jenkins sandbox blocks unsafe Groovy methods.
Fix: Approve the method in Manage Jenkins, In-process Script Approval, or better yet, wrap the logic in a shared library which runs outside the sandbox.
No such DSL method or No such property
Problem: Missing plugin or wrong pipeline syntax.
Fix:
- Verify the plugin is installed (e.g.,
pipeline-stage-step,workflow-aggregator) - Check if you are mixing declarative and scripted syntax (you cannot use
node {}insidepipeline {})
Docker Permission Denied
Problem: docker: Got permission denied while trying to connect to the Docker daemon socket
Fix:
# Add Jenkins user to docker group
sudo usermod -aG docker jenkins
sudo systemctl restart jenkins
Agent Goes Offline During Build
Problem: Build fails midway because the agent disconnects.
Fix:
- Increase agent JVM heap:
-Xmx512m - Set keep-alive intervals in agent config
- Use
retryfor flaky steps:
stage('Flaky Deploy') {
steps {
retry(3) {
sh './deploy.sh'
}
}
}
Stash/Unstash Fails with Large Files
Problem: Stash has a default 100MB limit.
Fix:
- Increase limit in Jenkins system config
- Use
archiveArtifactspluscopyArtifactsfor large files - Better approach: use an external artifact store (S3, Nexus) for large binaries
Credentials Not Found
Problem: Could not find credentials entry with ID
Fix:
- Verify credential ID in Manage Jenkins, Credentials
- Check the credential scope. System-scoped credentials are not available to pipeline jobs
- Ensure the credential domain matches
Pipeline Takes Too Long
Diagnosis checklist:
npm install instead of npm ci? The latter is faster in CI-v /cache/.m2:/root/.m2)checkout scm pulling the entire git history? Use shallow clone:checkout([
$class: 'GitSCM',
branches: [[name: '*/main']],
extensions: [[$class: 'CloneOption', depth: 1, shallow: true]],
userRemoteConfigs: [[url: 'https://github.com/org/repo.git']]
])
Build Works Locally But Fails in Jenkins
Common causes:
- Different Node/Java/Python version. Pin versions in Docker agent
- Missing environment variables. Check
environmentblock - File permission issues. Jenkins user is not your local user
- Different OS. Your Mac vs Linux Jenkins agent
Debug strategy:
stage('Debug') {
steps {
sh 'whoami'
sh 'pwd'
sh 'env | sort'
sh 'node --version || true'
sh 'java -version || true'
sh 'ls -la'
}
}
---
Wrapping Up
Jenkins pipelines are powerful but demand discipline. The key takeaways:
The Jenkinsfiles in this guide are production-tested patterns. Adapt them to your stack, commit them to your repo, and iterate. A well-crafted pipeline is an investment that pays off on every single commit.
---
Found this useful? Check out our other CI/CD guides on GitHub Actions, GitLab CI, and ArgoCD.
---
Frequently Asked Questions
What is the difference between declarative and scripted Jenkins pipelines?
Declarative pipelines use a structured pipeline { } block with predefined sections (stages, steps, post) and are easier to read and validate. Scripted pipelines use raw Groovy with a node { } block offering maximum flexibility but less structure. Start with declarative pipelines and only use scripted when you need advanced Groovy logic that declarative doesn't support.
How do I pass parameters to a Jenkins pipeline?
Define parameters in the parameters { } block with types like string, choice, booleanParam. Access them in stages with params.PARAM_NAME. Trigger parameterized builds via the UI "Build with Parameters" button, API calls, or upstream jobs using the build step with parameters: list.
Why is my Jenkins pipeline failing with "script not permitted"?
This means the pipeline uses a Groovy method or class not in Jenkins' script security sandbox whitelist. An admin must approve the signature in Manage Jenkins > In-process Script Approval. Alternatively, move the logic into a shared library marked as trusted, which bypasses sandbox restrictions.
How do I run parallel stages in Jenkins?
Use the parallel block inside a stage to run multiple branches concurrently. Each parallel branch can have its own steps, agent, and even downstream stages. Example: run unit tests, integration tests, and linting simultaneously. Set failFast true to abort all parallel branches if one fails.
How do I integrate Jenkins with Kubernetes?
Use the Kubernetes plugin to dynamically provision Jenkins agents as pods. Define pod templates in your pipeline with kubernetes { } agent specifying container images for each tool needed. This provides clean, isolated build environments that scale automatically and don't consume resources when idle.