Blogging with Hugo and GitLab (5): Let's Encrypt Renewal Automation

web development

Manually renewing certificates every three months is not going to be sustainable. Fortunately, there is the experience shared for automating the certificate renewal under GitLab. The basic idea is to apply the same operation of the manual certificate generation described in the previous article, but via the automation strategy with the GitLab pipeline schedule. Here I recompiled the scripts from the original article to not include any domain specific content, which allows them to be directly applied to any repositories (with domain specific variables defined in the pipeline setting).

Scripts for Let’s Encrypt

Download the following three Shell scripts (or the zip package containing them altogether). Place the scripts under the root directory (the same level as .gitlab-ci.yml) of your GitLab repository.

Below are the details of the scripts.

  • letsencrypt_generate.sh
#!/bin/sh

end_epoch=$(date -d "$(echo | openssl s_client -connect $ROOT_DOMAIN:443 -servername $ROOT_DOMAIN 2>/dev/null | openssl x509 -enddate -noout | cut -d'=' -f2)" "+%s")
current_epoch=$(date "+%s")
days_left=$((($end_epoch - $current_epoch) / 60 / 60 / 24))

echo "=============================="
echo "Renewal threshold: $RENEWAL_THRESHOLD_IN_DAYS_LEFT days"
echo "Root domain: $ROOT_DOMAIN"
echo "=============================="

if [ $days_left -le $RENEWAL_THRESHOLD_IN_DAYS_LEFT ]; then
    echo "=============================="
    echo "Certificate is $days_left days old, renewing now."
    echo "=============================="
    certbot certonly \
    --agree-tos \
    --debug \
    --manual \
    --manual-auth-hook letsencrypt_authenticator.sh \
    --manual-cleanup-hook letsencrypt_cleanup.sh \
    --manual-public-ip-logging-ok \
    --preferred-challenges http \
    -d $ROOT_DOMAIN \
    -d www.$ROOT_DOMAIN \
    -m $GITLAB_USER_EMAIL
    echo "=============================="
    echo "Certbot finished. Updating GitLab Pages domains."
    echo "=============================="
    curl --request PUT --header "PRIVATE-TOKEN: $CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN" --form "certificate=@/etc/letsencrypt/live/$ROOT_DOMAIN/fullchain.pem" --form "key=@/etc/letsencrypt/live/$ROOT_DOMAIN/privkey.pem" https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pages/domains/$ROOT_DOMAIN
    curl --request PUT --header "PRIVATE-TOKEN: $CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN" --form "certificate=@/etc/letsencrypt/live/$ROOT_DOMAIN/fullchain.pem" --form "key=@/etc/letsencrypt/live/$ROOT_DOMAIN/privkey.pem" https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pages/domains/www.$ROOT_DOMAIN
else
    echo "=============================="
    echo "Certificate still valid for $days_left days, no renewal required."
    echo "=============================="
fi
  • letsencrypt_authenticator.sh
#!/bin/sh

mkdir -p $CI_PROJECT_DIR/static/.well-known/acme-challenge
echo $CERTBOT_VALIDATION > $CI_PROJECT_DIR/static/.well-known/acme-challenge/$CERTBOT_TOKEN
git add $CI_PROJECT_DIR/static/.well-known/acme-challenge/$CERTBOT_TOKEN
git commit -m "GitLab runner - Added certbot challenge file for certificate renewal"
git push https://$GITLAB_USER_LOGIN:$CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN@gitlab.com/$CI_PROJECT_PATH.git HEAD:master

interval_sec=15
max_tries=80 # 20 minutes
n_tries=0
while [ $n_tries -le $max_tries ]
do
  status_code=$(curl -L --write-out "%{http_code}\n" --silent --output /dev/null https://$ROOT_DOMAIN/.well-known/acme-challenge/$CERTBOT_TOKEN)
  if [[ $status_code -eq 200 ]]; then
    exit 0
  fi

  n_tries=$((n_tries+1))
  sleep $interval_sec
done

exit 1
  • letsencrypt_cleanup.sh
#!/bin/sh

git rm $CI_PROJECT_DIR/static/.well-known/acme-challenge/$CERTBOT_TOKEN
git commit -m "GitLab runner - Removed certbot challenge file"
git push https://$GITLAB_USER_LOGIN:$CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN@gitlab.com/$CI_PROJECT_PATH.git HEAD:master

GitLab pipeline configuration

With the above three scripts, add a new job to .gitlab-ci.yml for the GitLab pipeline schedule. Similarly, there is no domain specific content. Simply copy the exact content should just work.

letsencrypt-renewal:
  image: loadbalancing/alpine-certbot:3.10
  only:
    - schedules
  variables:
    CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN: $CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN
    RENEWAL_THRESHOLD_IN_DAYS_LEFT: $RENEWAL_THRESHOLD_IN_DAYS_LEFT
    ROOT_DOMAIN: $ROOT_DOMAIN
  script:
    - export PATH=$PATH:$CI_PROJECT_DIR
    - git config --global user.name $GITLAB_USER_LOGIN
    - git config --global user.email $GITLAB_USER_EMAIL
    - chmod +x ./letsencrypt*.sh
    - ./letsencrypt_generate.sh

Note that the original article specifies a docker image with Alpine Linux 3.7, where the certificate renewal won’t succeed now. Specifically, Let’s Encrypt relies on the ACME protocol for certificate issuance and management. They are now replacing the old version ACMEv1 with ACMEv2. On the other hand, Certbot has ACME v2 support since Version 0.22.0, while its version in Alpine Linux 3.7 is 0.19.0. Therefore, I created another docker image to include a newer version of Alpine Linux (3.10, which has Certbot 0.35.1).

GitLab personal access token

As mentioned earlier, the above scripts just apply the same operation of the manual certificate generation. Recall that we need to add the certificate to the GitLab repository. A personal access token allows the pipeline schedule executing the script to update the GitLab repository, which automates the certificate generation.

To create a personal access token, click “Settings” of the avatar in the upper-right corner.

Then create a new personal access token.
“User Settings” > “Access Tokens” > “Personal Access Tokens”

  • Name: RENEWAL_THRESHOLD_IN_DAYS_LEFT
  • Expires at: (leave it empty)
  • Scopes: select “api”

Click “Create personal access token”
The page will show the generated token which will be used next. Copy it.

GitLab pipeline schedule

With all the above steps ready, let’s create a new pipeline schedule to run the job of certificate renewal.

“CI / CD” > “Schedules” > “New schedule”

  • “Description”: Let’s Encrypt Auto Renewal
  • “Interval Pattern”: select “Every week (Sundays at 4:00am)”
  • “Cron Timezone”: (your preferred timezone)
  • “Target Branch”: master
  • “Variables”:
    1. CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN: (paste the token generated in the previous step)
    2. RENEWAL_THRESHOLD_IN_DAYS_LEFT: 20
    3. ROOT_DOMAIN: loadbalancing.xyz

Click “Save pipeline schedule”

The three variables defined are used by the scripts for the specified domain. A certificate from Let’s Encrypt is valid for 90 days and the notification starts being sent when the certificate is going to expire in 20 days. Therefore, I set the renewal threshold at 20 days to avoid email noises (i.e., the certificate will be automatically renewed when there are less than 21 days left before the original certificate expires).

Cost

While the execution of scripts incurs certain computational cost, they are all within GitLab and thus is free to users. Another aspect of cost is that every certificate renewal will generate four commit records–two for adding files for the ACME challenge and another two for removing them after the ACME challenge completes (to keep the repository clean). With the schedule setting above, that is four additional commit records every 20 days, which looks not to bad to me. You can certainly minimize such noises by extending the renewal interval, but I suggest to leave some margin in case of renewal failures (e.g., for the ACMEv1 deprecation issue, I got 10 days for the root cause identification and fix before my website is blocked from the https access).

Contents