# Gerrit and Jenkins Verified Lab

This kit deploys a fresh Gerrit 3.14.2 review server and a fresh Jenkins 2.555.1 controller, then connects them with Gerrit Trigger 3.1983. It does not cover Jenkins migration, high availability, reverse proxy setup, or production secret management.

All addresses and directory names are examples. Replace them before use. Never commit `env`, password files, Jenkins credentials, SSH private keys, or Gerrit's `secure.config`.

## Contents

- `VERSION.env`: pinned application versions and multi-architecture image digests.
- `env.example`: non-secret runtime settings.
- `gerrit/compose.yaml`: Gerrit service with bind-mounted persistence.
- `gerrit/configure-site.sh`: advertised SSH address and image-specific startup fixes.
- `gerrit/configure-ldap.sh`: LDAP authentication without putting the bind password on the command line.
- `gerrit/configure-smtp.sh`: SMTP settings with the password written to `secure.config`.
- `gerrit/backup.sh`: cold backup of authoritative and non-rebuildable data.
- `gerrit/project.config`: project-level Verified label and submit requirement.
- `gerrit/groups.example`: UUID mapping template for `refs/meta/config`.
- `jenkins/Dockerfile`: pinned Jenkins image with pinned plugins.
- `jenkins/compose.yaml`: single-controller Jenkins installation.
- `jenkins/quality-gate.pipeline.groovy`: fixed demo checks for Gerrit patch sets.
- `jenkins/review-commands.xml`: Gerrit 3.14-compatible review command templates.
- `scripts/verify.sh`: Compose and HTTP smoke checks.
- `SHA256SUMS`: integrity list for this kit.

## 1. Requirements

Use a Linux host with these tools:

- Docker Engine 24 or newer and Docker Compose v2.
- Git, OpenSSH, curl, and sha256sum.
- At least 4 CPU cores, 8 GiB memory, and 30 GiB free disk for both services.
- Network access from Jenkins to Gerrit HTTP and SSH ports.
- An LDAP directory and SMTP server only if those integrations are required.

The examples bind both web services to `127.0.0.1`. Set `GERRIT_BIND_ADDRESS` or `JENKINS_BIND_ADDRESS` to an internal interface only when direct network access is intentional.

## 2. Prepare Settings

```bash
cp env.example env
chmod 600 env
${EDITOR:-vi} env
set -a
# shellcheck source=/dev/null
source env
set +a
```

Validate the Compose files before starting anything:

```bash
docker compose --env-file env -f gerrit/compose.yaml config --quiet
docker compose --env-file env -f jenkins/compose.yaml config --quiet
sha256sum -c SHA256SUMS
```

## 3. Start and Configure Gerrit

```bash
docker compose --env-file env -f gerrit/compose.yaml up -d
docker compose --env-file env -f gerrit/compose.yaml ps
docker compose --env-file env -f gerrit/compose.yaml logs --tail=100 gerrit
./gerrit/configure-site.sh env
```

Wait until the container is healthy, then check the API:

```bash
curl -fsS "${GERRIT_WEB_URL%/}/config/server/version"
```

The first authenticated user becomes the initial Gerrit administrator. Complete this login before inviting other users.

### LDAP

The example uses a dedicated read-only LDAP bind account. Its password is supplied through stdin and stored only in Gerrit's mode-0600 `secure.config`:

```bash
install -m 600 /dev/null /secure/path/ldap-password
${EDITOR:-vi} /secure/path/ldap-password
./gerrit/configure-ldap.sh env /secure/path/ldap-password
```

Before applying the configuration, validate the exact equality query against the directory:

```bash
ldapsearch -x -H "$LDAP_SERVER" \
  -D "$LDAP_BIND_DN" -W \
  -b "$LDAP_ACCOUNT_BASE" \
  "(${LDAP_ACCOUNT_ATTRIBUTE}=your-user)" dn cn mail
```

A successful bind alone is not enough. The search must return exactly the intended user. Gerrit creates an account on the user's first successful LDAP login.

`auth.type=LDAP` authenticates the web UI and HTTP Git. Gerrit's SSH service still uses an SSH key; it does not accept an LDAP password.

### SMTP

```bash
install -m 600 /dev/null /secure/path/smtp-password
${EDITOR:-vi} /secure/path/smtp-password
./gerrit/configure-smtp.sh env /secure/path/smtp-password
```

Post a test review comment and confirm both the Gerrit sendemail log and the recipient mailbox. A TCP connection to the SMTP server does not prove mail delivery.

## 4. Create a Review Project

Create `gerrit-demo` in the Gerrit web UI or with an administrator SSH command:

```bash
ssh -p "$GERRIT_SSH_PORT" admin@"$GERRIT_HOST" \
  gerrit create-project --empty-commit gerrit-demo
```

Clone it and install the Gerrit `commit-msg` hook:

```bash
git clone "ssh://your-user@${GERRIT_HOST}:${GERRIT_SSH_PORT}/gerrit-demo"
cd gerrit-demo

git config core.hooksPath .git/hooks
curl -fsSLo .git/hooks/commit-msg "${GERRIT_WEB_URL%/}/tools/hooks/commit-msg"
chmod u+x .git/hooks/commit-msg
```

The local `core.hooksPath` override matters when a global Git configuration points hooks to another directory.

Create a review change:

```bash
printf '# Gerrit demo\n' > README.md
git add README.md
git commit -m "docs: add project readme"
git push origin HEAD:refs/for/main
```

Do not push review work directly to `refs/heads/main`. The `refs/for/main` push creates a Gerrit Change and emits the events consumed by Jenkins.

## 5. Add the Verified Submit Requirement

Fetch the project configuration branch into a separate worktree:

```bash
mkdir gerrit-demo-meta-config
cd gerrit-demo-meta-config
git init
git remote add origin \
  "ssh://your-user@${GERRIT_HOST}:${GERRIT_SSH_PORT}/gerrit-demo"
git fetch origin refs/meta/config
git checkout -b meta-config FETCH_HEAD

git config core.hooksPath .git/hooks
curl -fsSLo .git/hooks/commit-msg "${GERRIT_WEB_URL%/}/tools/hooks/commit-msg"
chmod u+x .git/hooks/commit-msg
```

Copy `gerrit/project.config` into the worktree. Run the following command and replace both placeholders in `groups.example` with the UUIDs from your own Gerrit instance:

```bash
ssh -p "$GERRIT_SSH_PORT" your-user@"$GERRIT_HOST" \
  gerrit ls-groups --verbose
```

Save the completed file as `groups`, then submit the configuration through review:

```bash
git add project.config groups
git commit -m "feat: require Verified approval"
git push origin HEAD:refs/for/refs/meta/config
```

Review and submit this configuration Change as a project owner or administrator. The supplied rule excludes `refs/meta/config` from the Verified requirement so a broken CI system cannot lock its own repair Change.

Before Jenkins is connected, give an ordinary Change `Code-Review +2`. Submit must remain blocked because Verified is unsatisfied.

## 6. Install Jenkins

Create a writable Jenkins home owned by the image's UID 1000, then build and start the pinned image:

```bash
mkdir -p jenkins/data/jenkins_home
sudo chown -R 1000:1000 jenkins/data/jenkins_home
chmod 750 jenkins/data/jenkins_home

docker compose --env-file env -f jenkins/compose.yaml build --pull
docker compose --env-file env -f jenkins/compose.yaml up -d
docker compose --env-file env -f jenkins/compose.yaml ps
```

Read the one-time password and finish the setup wizard:

```bash
sudo cat jenkins/data/jenkins_home/secrets/initialAdminPassword
```

Set the Jenkins URL to `JENKINS_URL` under **Manage Jenkins > System**. Configure the required authentication and authorization realm before exposing the web port beyond localhost.

## 7. Create the Gerrit CI Identity

Create the key as Jenkins UID 1000:

```bash
docker compose --env-file env -f jenkins/compose.yaml run --rm --no-deps \
  --entrypoint sh jenkins -eu -c '
    install -d -m 700 /var/jenkins_home/.ssh
    test -e /var/jenkins_home/.ssh/gerrit-ci-cd || \
      ssh-keygen -q -t ed25519 -N "" \
        -f /var/jenkins_home/.ssh/gerrit-ci-cd
    chmod 600 /var/jenkins_home/.ssh/gerrit-ci-cd
  '
```

Use an LDAP-backed `ci-cd` account that has logged in once, or create a dedicated internal Gerrit account. Add the public key and group membership:

```bash
cat jenkins/data/jenkins_home/.ssh/gerrit-ci-cd.pub | \
  ssh -p "$GERRIT_SSH_PORT" admin@"$GERRIT_HOST" \
    gerrit set-account --add-ssh-key - ci-cd

ssh -p "$GERRIT_SSH_PORT" admin@"$GERRIT_HOST" \
  "gerrit set-members --add ci-cd 'Service Users'"
```

Verify the identity from the Jenkins host:

```bash
ssh -i jenkins/data/jenkins_home/.ssh/gerrit-ci-cd \
  -p "$GERRIT_SSH_PORT" ci-cd@"$GERRIT_HOST" gerrit version

timeout 5 ssh -i jenkins/data/jenkins_home/.ssh/gerrit-ci-cd \
  -p "$GERRIT_SSH_PORT" ci-cd@"$GERRIT_HOST" gerrit stream-events
```

The second command should remain connected until `timeout` exits. `Permission denied` or an immediate capability error means the account is not ready.

If `stream-events` reports a capability error, grant **Stream Events** to `Service Users` under **All-Projects > Access > Global Capabilities**, then test again.

Also add the same private key to Jenkins Credentials as **SSH Username with private key**, username `ci-cd`, credential ID `gerrit-ci-cd`. Verify the Gerrit host key before adding it to Jenkins `known_hosts`.

## 8. Configure Gerrit Trigger

Open **Manage Jenkins > Gerrit Trigger** and add one server:

| Field | Value |
|---|---|
| Name | `gerrit-stu` |
| Gerrit host | `GERRIT_HOST` |
| Gerrit SSH port | `GERRIT_SSH_PORT` |
| Gerrit user | `ci-cd` |
| SSH key file | `/var/jenkins_home/.ssh/gerrit-ci-cd` |
| Frontend URL | `GERRIT_WEB_URL` |
| Notification level | `NONE` |

Test the SSH connection before saving.

Under the advanced Gerrit reporting settings, replace all six default review commands with the values from `jenkins/review-commands.xml`. Gerrit 3.14.2 rejects the plugin's default `--verified` flag. The compatible form is:

```text
--label Verified=<VERIFIED> --label Code-Review=<CODE_REVIEW>
```

Set the server vote values to:

| Build result | Verified |
|---|---:|
| Started | 0 |
| Successful | +1 |
| Failed | -1 |
| Unstable | -1 |
| Not built | 0 |
| Aborted | 0 |

Create a Pipeline job named `gerrit-demo-verified` and paste `jenkins/quality-gate.pipeline.groovy` as its trusted Pipeline script. Gerrit Trigger supplies `GERRIT_HOST` and `GERRIT_PORT` from the selected server at runtime; keep the `gerrit-ci-cd` credential ID aligned with Jenkins Credentials.

Configure **Gerrit event** for this job:

- Server: `gerrit-stu`.
- Project type: Plain, pattern: `gerrit-demo`.
- Branch type: Plain, pattern: `main`.
- Event: Patchset Created.
- Event: Comment Added Contains, pattern: `(?i)^recheck\s*$`.
- Successful vote: Verified `+1`.
- Failed and unstable vote: Verified `-1`.
- Code-Review vote: `0` for all results.

The supplied Pipeline runs only fixed checks controlled by the Jenkins administrator. Move real builds to an isolated ephemeral agent before executing code or a Jenkinsfile supplied by an untrusted review Change.

## 9. Verify the Closed Loop

Upload a Change and observe this sequence:

1. Gerrit emits `patchset-created`.
2. Gerrit Trigger starts `gerrit-demo-verified`.
3. Jenkins fetches `GERRIT_REFSPEC` and verifies `GERRIT_PATCHSET_REVISION`.
4. The fixed checks run.
5. `ci-cd` records Verified `+1` or `-1` on that Patch Set.
6. Gerrit evaluates Code-Review and Verified independently.

Run the local smoke check:

```bash
./scripts/verify.sh env
```

A successful build must produce a `ci-cd` Verified `+1` with tag `autogenerated:jenkins-gerrit-trigger`. Keep auto-submit disabled for the first rollout; a human can inspect the result and submit the Change.

### New Patch Sets and Recheck

Update the same Change by preserving its Change-Id:

```bash
git add .
git commit --amend
git push origin HEAD:refs/for/main
```

A code-changing Patch Set does not inherit the old Verified vote because `copyCondition = changekind:NO_CODE_CHANGE`. The new `patchset-created` event runs Jenkins again and records a vote on the new Patch Set.

If no code changed and the failure was transient, comment exactly `recheck` on the current Change. The Jenkins job runs again and replaces its own prior vote.

## 10. Operations

Check service state and recent logs:

```bash
docker compose --env-file env -f gerrit/compose.yaml ps
docker compose --env-file env -f gerrit/compose.yaml logs --tail=200 gerrit
docker compose --env-file env -f jenkins/compose.yaml ps
docker compose --env-file env -f jenkins/compose.yaml logs --tail=200 jenkins
```

Create a cold Gerrit backup during a maintenance window:

```bash
./gerrit/backup.sh env /secure/backups/gerrit
```

The backup includes Git/NoteDb data, configuration and secrets, AccountPatchReviewDb, plugin data, and installed plugins. Lucene indexes and caches are omitted because they are rebuildable. Protect the archive like a credential because `gerrit_etc` contains `secure.config`.

Before an upgrade, test restore and reindex in a separate environment. Do not use the existence of an archive as proof that recovery works.

## References

- Gerrit 3.14.2 documentation: <https://www.gerritcodereview.com/3.14.html>
- Gerrit submit requirements: <https://gerrit-documentation.storage.googleapis.com/Documentation/3.14.2/config-submit-requirements.html>
- Gerrit Change-Ids: <https://gerrit-documentation.storage.googleapis.com/Documentation/3.14.2/user-changeid.html>
- Gerrit Trigger plugin: <https://plugins.jenkins.io/gerrit-trigger/>
- Official Jenkins Docker image: <https://github.com/jenkinsci/docker>
- Gerrit Docker image: <https://hub.docker.com/r/gerritcodereview/gerrit>