Compare commits

..

2 Commits

Author SHA1 Message Date
moilanik 50a1cb9a5c extra deps for chart build
CI Feature / Load example-gitea-env.conf to pipeline env (push) Successful in 30s
unit-tests Bats test report
CI Feature / Bats tests (push) Successful in 1m35s
acc-tests Cucumber test report
CI Feature / Cucumber tests (push) Successful in 1m55s
CI Feature / Report Summary (push) Successful in 6s
2026-06-23 12:41:50 +03:00
moilanik e1ebbe2fce gitops skill 2026-06-23 12:38:34 +03:00
14 changed files with 132 additions and 1100 deletions
+1 -1
View File
@@ -23,7 +23,7 @@ on:
jobs: jobs:
build-push: build-push:
runs-on: docker runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
+2 -2
View File
@@ -32,7 +32,7 @@ concurrency:
jobs: jobs:
build-push: build-push:
runs-on: docker runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
- uses: actions/checkout@v4 - uses: actions/checkout@v4
@@ -91,7 +91,7 @@ jobs:
fi fi
tag-commit: tag-commit:
runs-on: docker runs-on: ubuntu-latest
needs: [build-push] needs: [build-push]
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
+1 -14
View File
@@ -17,9 +17,6 @@ on:
GITOPS_REPO: GITOPS_REPO:
required: true required: true
type: string type: string
GITOPS_EXTRA_CMD:
required: false
type: string
secrets: secrets:
GITOPS_DISPATCH_TOKEN: GITOPS_DISPATCH_TOKEN:
required: true required: true
@@ -37,7 +34,6 @@ env:
GITOPS_SOURCE_COMMIT: ${{ github.sha }} GITOPS_SOURCE_COMMIT: ${{ github.sha }}
GITEA_API_URL: ${{ fromJson(inputs.env_json).GITEA_API_URL }} GITEA_API_URL: ${{ fromJson(inputs.env_json).GITEA_API_URL }}
GITOPS_TAG_PREFIX: ${{ fromJson(inputs.env_json).GIT_TAG_PREFIX || '' }} GITOPS_TAG_PREFIX: ${{ fromJson(inputs.env_json).GIT_TAG_PREFIX || '' }}
GITOPS_EXTRA_CMD: ${{ inputs.GITOPS_EXTRA_CMD || '' }}
GITOPS_WORKFLOW: gitops-service.yaml GITOPS_WORKFLOW: gitops-service.yaml
GITOPS_DISPATCH_TIMEOUT: 30 GITOPS_DISPATCH_TIMEOUT: 30
@@ -53,28 +49,19 @@ jobs:
ID=$(date +%s | md5sum | head -c 8) ID=$(date +%s | md5sum | head -c 8)
echo "dispatch_id=$ID" >> "$GITHUB_OUTPUT" echo "dispatch_id=$ID" >> "$GITHUB_OUTPUT"
- name: Resolve commit author
id: author
run: |
echo "name=$(git log -1 --format='%an')" >> "$GITHUB_OUTPUT"
echo "email=$(git log -1 --format='%ae')" >> "$GITHUB_OUTPUT"
- name: Dispatch to GitOps repo - name: Dispatch to GitOps repo
env: env:
GITEA_TOKEN: ${{ secrets.GITOPS_DISPATCH_TOKEN }} GITEA_TOKEN: ${{ secrets.GITOPS_DISPATCH_TOKEN }}
run: | run: |
INPUTS=$(jq -nc \ INPUTS=$(jq -nc \
--arg dispatch_id "${{ steps.gen.outputs.dispatch_id }}" \ --arg dispatch_id "${{ steps.gen.outputs.dispatch_id }}" \
--arg author_name "${{ steps.author.outputs.name }}" \
--arg author_email "${{ steps.author.outputs.email }}" \
--arg file "$GITOPS_FILE" \ --arg file "$GITOPS_FILE" \
--arg yq_tpl "$GITOPS_YQ_TPL" \ --arg yq_tpl "$GITOPS_YQ_TPL" \
--arg version "$GITOPS_VERSION" \ --arg version "$GITOPS_VERSION" \
--arg source_repo "$GITOPS_SOURCE_REPO" \ --arg source_repo "$GITOPS_SOURCE_REPO" \
--arg source_commit "$GITOPS_SOURCE_COMMIT" \ --arg source_commit "$GITOPS_SOURCE_COMMIT" \
--arg git_tag_prefix "${GITOPS_TAG_PREFIX:-}" \ --arg git_tag_prefix "${GITOPS_TAG_PREFIX:-}" \
--arg extra_cmd "${GITOPS_EXTRA_CMD:-}" \ '{dispatch_id: $dispatch_id, file: $file, yq_tpl: $yq_tpl, version: $version, source_repo: $source_repo, source_commit: $source_commit, git_tag_prefix: $git_tag_prefix}')
'{dispatch_id: $dispatch_id, author_name: $author_name, author_email: $author_email, file: $file, yq_tpl: $yq_tpl, version: $version, source_repo: $source_repo, source_commit: $source_commit, git_tag_prefix: $git_tag_prefix, extra_cmd: $extra_cmd}')
curl -s -X POST \ curl -s -X POST \
"${GITEA_API_URL}/api/v1/repos/${GITOPS_REPO}/actions/workflows/${GITOPS_WORKFLOW}/dispatches" \ "${GITEA_API_URL}/api/v1/repos/${GITOPS_REPO}/actions/workflows/${GITOPS_WORKFLOW}/dispatches" \
-H "Authorization: token $GITEA_TOKEN" \ -H "Authorization: token $GITEA_TOKEN" \
+12 -59
View File
@@ -151,102 +151,58 @@ ja sitä kautta Gitea ei tarvitse päivityksessä mitään temppuja.
Päivityksen jälkeen muista tappaa pod (käynnistyy automaattisesti uudelleen), että lataa varmasti kaikki uudesta. Sillä ConfigMap tms eivät lataudu Päivityksen jälkeen muista tappaa pod (käynnistyy automaattisesti uudelleen), että lataa varmasti kaikki uudesta. Sillä ConfigMap tms eivät lataudu
mikäli pod jatkaa ajamista. mikäli pod jatkaa ajamista.
Klusterissa on kaksi StatefulSetiä, joilla on eri labelit:
- **`act-runner`** — yleisrunneri (label `ubuntu-latest`). DinD-sidecar on olemassa mutta idle — `require_docker: false` estää runneria käyttämästä sitä.
- **`act-runner-docker`** — Docker-buildien runneri (label `docker`). DinD on aktiivinen, `require_docker: true`.
Vain `docker-build-push.yml` ja `ci-container-build-push.yml` käyttävät `docker`-labelia.
Kaikki muut workflowt (testit, lintit, helm-publish, gitops-dispatch) ajetaan `ubuntu-latest`-runnerilla.
Steppien suoritus ei mene Docker Daemonin läpi, joten konekielisiä kontteja luodaan suoraan K8s-runtimella.
```bash ```bash
helm repo add gitea https://dl.gitea.com/charts helm repo add gitea https://dl.gitea.com/charts
helm repo update helm repo update
# 1. Yleisrunneri (DinD idle)
helm upgrade --install act-runner gitea/actions \ helm upgrade --install act-runner gitea/actions \
--set enabled=true \ --set enabled=true \
--set giteaRootURL="$GITEA_URL" \ --set giteaRootURL="$GITEA_URL" \
--set existingSecret=act-runner-token \ --set existingSecret=act-runner-token \
--set existingSecretKey=token \ --set existingSecretKey=token \
--set statefulset.replicas=2 \ --set statefulset.replicas=3 \
--set statefulset.runner.tag=1.0.8 \ --set statefulset.runner.tag=1.0.8 \
--set statefulset.dind.tag=29.5.2-dind \ --set statefulset.dind.tag=29.5.2-dind \
--set statefulset.dind.resources.requests.memory=250Mi \
--set statefulset.dind.resources.limits.memory=750Mi \
--set-string 'statefulset.runner.config=log: --set-string 'statefulset.runner.config=log:
level: info level: info
cache: cache:
enabled: true enabled: false
container:
require_docker: false' \
--namespace "$GITEA_ACTIONS_NAMESPACE" \
--create-namespace
# 2. Docker-runner (DinD)
helm upgrade --install act-runner-docker gitea/actions \
--set enabled=true \
--set giteaRootURL="$GITEA_URL" \
--set existingSecret=act-runner-token \
--set existingSecretKey=token \
--set statefulset.replicas=1 \
--set statefulset.runner.tag=1.0.8 \
--set statefulset.dind.tag=29.5.2-dind \
--set statefulset.dind.resources.requests.memory=250Mi \
--set statefulset.dind.resources.limits.memory=750Mi \
--set-string 'statefulset.runner.config=log:
level: info
cache:
enabled: true
container: container:
require_docker: true require_docker: true
docker_timeout: 300s docker_timeout: 300s' \
runner: --namespace "$GITEA_ACTIONS_NAMESPACE" \
labels: --create-namespace
- "docker:docker://catthehacker/ubuntu:act-latest"' \
--namespace "$GITEA_ACTIONS_NAMESPACE"
``` ```
`path escapes from parent` -bugi korjattiin Docker 29.5.2:ssa. Tämän teko aikana default on 29.5.1 — juuri tämän alle jäävä versio. path escapes from parent -bugi korjattiin Docker 29.5.2:ssa. Tämän teko aikana default on 29.5.1 — juuri tämän alle jäävä versio.
Oletus-lokitaso on `debug` — suositeltu `info`. Näkee jobien aloitukset ja valmistumiset ilman konttikerrosten purkua (Downloading/Extracting-spämmiä). `debug` on tarpeen vain vianselvityksessä. Oletus-lokitaso on `debug` — suositeltu `info`. Näkee jobien aloitukset ja valmistumiset ilman konttikerrosten purkua (Downloading/Extracting-spämmiä). `debug` on tarpeen vain vianselvityksessä.
`cache.enabled: true` nappaa image-cachen käyttöön — ilman sitä jokainen ajo lataa konttikuvat uudestaan.
#### Docker (DinD) #### Docker (DinD)
> **Huomio:** Gitea 1.26.x ei tue vielä label-pohjaista runner-valintaa.
> `runs-on: ubuntu-latest`-jobi saattaa päätyä `docker`-labeliselle runnerille.
> Bugi on tunnettu Gitean FAQ:ssa — korjaus tulossa myöhemmässä versiossa.
> Katso: [docs.gitea.com/usage/actions/faq](https://docs.gitea.com/usage/actions/faq)
Helm chart deployaa DinD:n init-sidecarina (`docker:dind` samassa podissa). Helm chart deployaa DinD:n init-sidecarina (`docker:dind` samassa podissa).
Molemmissa StatefulSetissä on DinD-sidecar, mutta: `require_docker: true` kytkee jobit siihen — erillistä DinD-asennusta ei tarvita.
- **`act-runner`**: `require_docker: false` → runner ei käytä DinD:tä lainkaan, steppien suoritus menee suoraan K8s-runtimella
- **`act-runner-docker`**: `require_docker: true` → runner luo steppikontit DinD:n kautta (tarvitaan `docker build` -komentoja varten)
**DinD-tag pinottu:** `29.5.2-dind` molemmissa (ei chart-oletusta `29.5.1-dind`). Docker 29.5.1 aiheuttaa act-runnerissa **DinD-tag pinottu:** `29.5.2-dind` (ei chart-oletusta). Docker 29.5.1 aiheuttaa act-runnerissa
`path escapes from parent` -virheen job-kontin käynnistyksessä sekä `mkdirat var/run: file exists` -virheen tiedostojen kopioinnissa. `path escapes from parent` -virheen job-kontin käynnistyksessä.
Maven/npm-ajot käyttävät vain workflow'n `container:`-imagea; DinD tarvitaan vasta Docker-buildissä. Maven/npm-ajot käyttävät vain workflow'n `container:`-imagea; DinD tarvitaan vasta Docker-buildissä.
### 5. Varmista ### 3. Varmista
```bash ```bash
kubectl get pods -n gitea-actions kubectl get pods -n gitea-actions
# → act-runner-runner-0 Running # → act-runner-runner-0 Running
# → act-runner-docker-runner-0 Running
kubectl exec -n gitea-actions act-runner-docker-runner-0 -c dind -- docker version kubectl exec -n gitea-actions act-runner-runner-0 -c dind -- docker version
# → Server Version: 29.5.2 (tai uudempi) # → Server Version: 29.5.2 (tai uudempi)
``` ```
Gitean puolella runnerit ilmestyvät Active-tilaan pienellä viiveellä: Gitean puolella runner ilmestyy Active-tilaan pienellä viiveellä:
``` ```
Site Admin → Actions → Runners (tai Org → Settings → Actions → Runners) Site Admin → Actions → Runners (tai Org → Settings → Actions → Runners)
# → act-runner-runner-0 Active ubuntu-latest # → act-runner-runner-0 Active ubuntu-latest
# → act-runner-docker-runner-0 Active docker
``` ```
Tämän jälkeen `.gitea/workflows/ci.yml` triggeröityy automaattisesti pushista. Tämän jälkeen `.gitea/workflows/ci.yml` triggeröityy automaattisesti pushista.
@@ -323,7 +279,4 @@ Tarkka asennus: [skills/gitops-update/SKILL.md](skills/gitops-update/SKILL.md)
| `existingSecret` | Kubernetes secretin nimi, jossa token | | `existingSecret` | Kubernetes secretin nimi, jossa token |
| `existingSecretKey` | Avain secretin sisällä | | `existingSecretKey` | Avain secretin sisällä |
| `statefulset.dind.tag` | DinD-image tag (`29.5.2-dind` minimi) | | `statefulset.dind.tag` | DinD-image tag (`29.5.2-dind` minimi) |
| `statefulset.dind.resources.requests.memory` | DinD muistirequest (suositus `250Mi`) |
| `statefulset.dind.resources.limits.memory` | DinD muistilimitti (suositus `750Mi`) |
| `statefulset.runner.labels` | Mukautetut labelit | | `statefulset.runner.labels` | Mukautetut labelit |
-39
View File
@@ -149,42 +149,3 @@ curl -X PATCH https://ci-reports.helm-dev.keskikuja.site/owner/repo/commit/sha8/
- `git-pages-publish-token` = plaintext (luetaan Giteaan viedessä) - `git-pages-publish-token` = plaintext (luetaan Giteaan viedessä)
Tarkemmat secret-ohjeet: [docs/secrets.md](docs/secrets.md). Tarkemmat secret-ohjeet: [docs/secrets.md](docs/secrets.md).
---
## Testaus
Retention-logiikalle on unit-testit, jotka testaa funktiot ja Phase 3 -säännöt
erikseen ilman ulkoisia riippuvuuksia.
```bash
cd git-pages
bats tests/retention.bats
```
Testit käyttävät `<root>/files/retention-lib.sh` -jaettua kirjastoa, jota myös
`retention-cleanup.sh` sourceaa. Uutta testiä kirjoittaessa:
1. Luo config `write_config`-helperilla
2. Täytä `KEEP`-array testidatalla (muoto: `dir|owner|repo|branch|days`)
3. Kutsu `apply_retention "$CONFIG"`
4. Tarkista `TO_DELETE`-array ja `$output`
**Vaatimukset:** `bats`, `jq`, `date` (GNU date tai BSD date ISO 8601 -tuella).
---
## Retention
Ylläpitoscripti, joka poistaa vanhat raportit git-pagesista retentionsääntöjen mukaan.
Ajetaan sidecar tai cronjobtilassa Kubernetesissa.
### Air gap -yhteensopimattomuus
Retentionkontti asentaa tarvitsemansa työkalut (`curl`, `jq`) ajon aikana
packagemanagerilla (`apt-get` / `apk`). Tämä **ei toimi air gap -ympäristössä**,
jossa konttirekisteriin tai pakettivarastoihin ei ole verkkoyhteyttä.
**TODO:** Rakenna custom Dockerimage, jossa deps on valmiina:
`FROM alpine:latest && apk add --no-cache curl jq`.
Pushaa omaan rekisteriin ja päivitä `values.yaml`:n `retention.image`.
+104 -115
View File
@@ -13,11 +13,75 @@ curl_with_host() {
[ -f "$CONFIG" ] || { echo "ERROR: config missing: $CONFIG" >&2; exit 1; } [ -f "$CONFIG" ] || { echo "ERROR: config missing: $CONFIG" >&2; exit 1; }
declare -A REPO_BRANCHES_CACHE declare -A BRANCH_CACHE
declare -A REPO_STATUS branch_exists() {
local owner="$1" repo="$2" branch="$3" key="${owner}/${repo}/${branch}"
local status attempt
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" [ -z "$GITEA_API_URL" ] && return 0
source "$SCRIPT_DIR/retention-lib.sh" [ -z "$GITEA_TOKEN" ] && return 0
if [ "${BRANCH_CACHE[$key]:-}" = "1" ]; then
return 0
fi
# Retry up to 2 times on API errors (hardcoded)
for attempt in 1 2 3; do
status=$(curl -sS -o /dev/null -w "%{http_code}" \
-H "Authorization: token ${GITEA_TOKEN}" \
"${GITEA_API_URL}/api/v1/repos/${owner}/${repo}/branches/${branch}" 2>/dev/null || echo "000")
if [ "$status" = "200" ]; then
BRANCH_CACHE[$key]=1
return 0
fi
if [ "$status" = "404" ]; then
return 1
fi
# API error - retry if not last attempt
if [ "$attempt" -lt 3 ]; then
sleep 10
continue
fi
done
# All retries failed - keep report (fail-safe)
echo " WARN: Gitea API error for ${owner}/${repo}/${branch} (status ${status}) after 3 attempts - KEEPING report"
BRANCH_CACHE[$key]=1
return 0
}
default_max_age=$(jq -r '.branches.default.maxAgeDays // 90' "$CONFIG")
default_keep_min=$(jq -r '.branches.default.keepMin // 5' "$CONFIG")
rule_max_age() {
local branch="$1" v
v=$(jq -r --arg b "$branch" '.branches[$b].maxAgeDays // empty' "$CONFIG")
[ -n "$v" ] && echo "$v" || echo "$default_max_age"
}
rule_keep_min() {
local branch="$1" v
v=$(jq -r --arg b "$branch" '.branches[$b].keepMin // empty' "$CONFIG")
[ -n "$v" ] && echo "$v" || echo "$default_keep_min"
}
age_days() {
local published="$1" epoch_pub now
epoch_pub=$(date -u -d "$published" +%s 2>/dev/null || echo 0)
[ "$epoch_pub" -eq 0 ] && echo 99999 && return
now=$(date -u +%s)
echo $(( (now - epoch_pub) / 86400 ))
}
parse_path() {
local rel="$1"
OWNER="${rel%%/*}"
rest="${rel#*/}"
REPO="${rest%%/*}"
}
echo "Fetching manifest from ${PAGES_URL}/.git-pages/manifest.json" echo "Fetching manifest from ${PAGES_URL}/.git-pages/manifest.json"
MANIFEST=$(curl_with_host "${PAGES_URL}/.git-pages/manifest.json") MANIFEST=$(curl_with_host "${PAGES_URL}/.git-pages/manifest.json")
@@ -33,7 +97,6 @@ fi
echo "" echo ""
echo "=== Phase 1: collect reports ===" echo "=== Phase 1: collect reports ==="
declare -A SEEN_REPORTS declare -A SEEN_REPORTS
declare -A SEEN_ECHO_COMMITS
declare -a REPORTS declare -a REPORTS
while IFS= read -r meta_path; do while IFS= read -r meta_path; do
report_dir=$(dirname "$meta_path") report_dir=$(dirname "$meta_path")
@@ -54,105 +117,58 @@ while IFS= read -r meta_path; do
days=$(age_days "$published") days=$(age_days "$published")
REPORTS+=("${report_dir}|${OWNER}|${REPO}|${branch}|${days}") REPORTS+=("${report_dir}|${OWNER}|${REPO}|${branch}|${days}")
echo " ${OWNER}/${REPO} branch=${branch} age=${days}d"
commit_dir=$(dirname "$report_dir")
if [ -z "${SEEN_ECHO_COMMITS[$commit_dir]:-}" ]; then
SEEN_ECHO_COMMITS[$commit_dir]=1
echo " ${commit_dir} branch=${branch} age=${days}d"
fi
done <<< "$META_PATHS" done <<< "$META_PATHS"
[ "${#REPORTS[@]}" -eq 0 ] && { echo "No actionable reports"; exit 0; } [ "${#REPORTS[@]}" -eq 0 ] && { echo "No actionable reports"; exit 0; }
echo "" echo ""
echo "=== Phase 2: check branches/repos in Gitea ===" echo "=== Phase 2: check branches in Gitea ==="
if [ -z "$GITEA_API_URL" ] || [ -z "$GITEA_TOKEN" ]; then
echo "ERROR: GITEA_API_URL and GITEA_TOKEN must be set" >&2
exit 1
fi
declare -a TO_DELETE declare -a TO_DELETE
declare -a KEEP declare -a KEEP
declare -A SEEN_ECHO_BRANCHES
declare -A SEEN_ECHO_REPO_DELETED
declare -A UNIQUE_BRANCHES
declare -A REASON_MAP
declare -A COMMIT_BRANCH_MAP
# Build commit→branch mapping
for entry in "${REPORTS[@]}"; do
IFS='|' read -r dir _ _ branch _ <<< "$entry"
commit_dir=$(dirname "$dir")
[ -n "${COMMIT_BRANCH_MAP[$commit_dir]:-}" ] || COMMIT_BRANCH_MAP["$commit_dir"]=$branch
done
for entry in "${REPORTS[@]}"; do
IFS='|' read -r _ owner repo branch _ <<< "$entry"
UNIQUE_BRANCHES["${owner}/${repo}/${branch}"]=1
done
TOTAL_BRANCHES=${#UNIQUE_BRANCHES[@]}
BRANCHES_EXISTING=0
BRANCH_DELETED_COUNT=0
REPO_DELETED_COUNT=0
MAXAGE_DELETED=0
KEEPMIN_DELETED=0
for entry in "${REPORTS[@]}"; do for entry in "${REPORTS[@]}"; do
IFS='|' read -r dir owner repo branch days <<< "$entry" IFS='|' read -r dir owner repo branch days <<< "$entry"
branch_key="${owner}/${repo}/${branch}" if [ -n "$GITEA_API_URL" ] && [ -n "$GITEA_TOKEN" ]; then
if branch_exists "$owner" "$repo" "$branch"; then if branch_exists "$owner" "$repo" "$branch"; then
if [ -z "${SEEN_ECHO_BRANCHES[$branch_key]:-}" ]; then echo " BRANCH EXISTS: ${owner}/${repo}/${branch}"
SEEN_ECHO_BRANCHES[$branch_key]=1
BRANCHES_EXISTING=$((BRANCHES_EXISTING + 1))
echo " BRANCH EXISTS: ${branch_key}"
fi
KEEP+=("${dir}|${owner}|${repo}|${branch}|${days}") KEEP+=("${dir}|${owner}|${repo}|${branch}|${days}")
else else
if [ -z "${SEEN_ECHO_BRANCHES[$branch_key]:-}" ]; then echo " BRANCH DELETED: ${owner}/${repo}/${branch} -> DELETE"
SEEN_ECHO_BRANCHES[$branch_key]=1
repo_key="${owner}/${repo}"
if [ "${REPO_STATUS[$repo_key]:-}" = "deleted" ]; then
REPO_DELETED_COUNT=$((REPO_DELETED_COUNT + 1))
if [ -z "${SEEN_ECHO_REPO_DELETED[$repo_key]:-}" ]; then
SEEN_ECHO_REPO_DELETED[$repo_key]=1
echo " REPO DELETED: ${repo_key} -> DELETE ALL"
fi
reason="repo deleted"
else
BRANCH_DELETED_COUNT=$((BRANCH_DELETED_COUNT + 1))
echo " BRANCH DELETED: ${branch_key} -> DELETE"
reason="branch deleted"
fi
fi
REASON_MAP["$dir"]="$reason"
TO_DELETE+=("$dir") TO_DELETE+=("$dir")
fi fi
else
KEEP+=("${dir}|${owner}|${repo}|${branch}|${days}")
fi
done done
echo "" echo ""
echo "=== Phase 3: apply retention rules to remaining reports ===" echo "=== Phase 3: apply retention rules to remaining reports ==="
PHASE2_DELETED=${#TO_DELETE[@]} declare -A BRANCH_COUNTS
apply_retention "$CONFIG" if [ "${#KEEP[@]}" -gt 0 ]; then
PHASE3_DELETED=$(( ${#TO_DELETE[@]} - PHASE2_DELETED )) IFS=$'\n'
for entry in $(printf '%s\n' "${KEEP[@]}" | sort -t'|' -k4,4 -k5,5rn); do
IFS='|' read -r dir owner repo branch days <<< "$entry"
max_age=$(rule_max_age "$branch")
keep_min=$(rule_keep_min "$branch")
fmt_num() { if [ "$days" -gt "$max_age" ]; then
local n="$1" out="" echo " DELETE: ${dir} (age ${days}d > maxAge ${max_age}d, branch ${branch})"
[ -z "$n" ] && { echo "?"; return; } TO_DELETE+=("$dir")
n="${n##0}" # strip leading zeros continue
while [ "${#n}" -gt 3 ]; do fi
out=" ${n: -3}$out"
n="${n:0:${#n}-3}" key="${branch}"
count="${BRANCH_COUNTS[$key]:-0}"
count=$((count + 1))
BRANCH_COUNTS["$key"]=$count
if [ "$count" -gt "$keep_min" ]; then
echo " DELETE: ${dir} (kept ${keep_min}/${count}, exceeds keepMin, branch ${branch})"
TO_DELETE+=("$dir")
fi
done done
echo "${n}${out}" unset IFS
} fi
echo ""
echo "=== Summary ==="
echo " Branches:"
echo " existing: $(fmt_num $BRANCHES_EXISTING)"
echo " deleted: $(fmt_num $BRANCH_DELETED_COUNT)"
echo " repo gone: $(fmt_num $REPO_DELETED_COUNT)"
echo " Commits:"
echo " deleted by maxAge: $(fmt_num $MAXAGE_DELETED)"
echo " deleted by keepMin:$(fmt_num $KEEPMIN_DELETED)"
if [ "${#TO_DELETE[@]}" -eq 0 ]; then if [ "${#TO_DELETE[@]}" -eq 0 ]; then
echo "Nothing to delete" echo "Nothing to delete"
@@ -177,37 +193,14 @@ echo "Downloading archive.tar..."
HTTP_CODE=$(curl_with_host -o "$ARCHIVE_FILE" -w "%{http_code}" -sS "${PAGES_URL}/.git-pages/archive.tar") HTTP_CODE=$(curl_with_host -o "$ARCHIVE_FILE" -w "%{http_code}" -sS "${PAGES_URL}/.git-pages/archive.tar")
if [ "$HTTP_CODE" = "200" ] && tar -tf "$ARCHIVE_FILE" >/dev/null 2>&1; then if [ "$HTTP_CODE" = "200" ] && tar -tf "$ARCHIVE_FILE" >/dev/null 2>&1; then
OLD_KB=$(du -sk "$ARCHIVE_FILE" 2>/dev/null | awk '{print $1}') echo "Extracting archive..."
echo "Extracting archive (${OLD_KB}kB)..."
tar -xf "$ARCHIVE_FILE" -C "$SITE_DIR" tar -xf "$ARCHIVE_FILE" -C "$SITE_DIR"
declare -A GROUP_SEEN for dir in "${TO_DELETE[@]}"; do
declare -A GROUP_LINES if [ -d "$SITE_DIR/$dir" ]; then
for del in "${TO_DELETE[@]}"; do echo " Removing: $dir"
if [ ! -d "$SITE_DIR/$del" ]; then rm -rf "$SITE_DIR/$dir"
continue
fi fi
commit_dir=$(dirname "$del")
branch="${COMMIT_BRANCH_MAP[$commit_dir]:-?}"
reason="${REASON_MAP[$del]:-?}"
repo_path="${del%%/reports/*}"
commit_hash="${commit_dir##*/}"
key="${repo_path}/${branch} | Reason: ${reason}"
seen_key="${key}|${commit_hash}"
if [ -z "${GROUP_SEEN[$seen_key]:-}" ]; then
GROUP_SEEN[$seen_key]=1
GROUP_LINES["$key"]="${GROUP_LINES[$key]:-} $commit_hash"
fi
rm -rf "$SITE_DIR/$del"
done
for key in "${!GROUP_LINES[@]}"; do
echo " Removing: ${key}"
for hash in ${GROUP_LINES[$key]}; do
echo " commit: ${hash}"
done
done done
else else
echo "archive.tar failed (HTTP ${HTTP_CODE}) - falling back to manifest-based rebuild" echo "archive.tar failed (HTTP ${HTTP_CODE}) - falling back to manifest-based rebuild"
@@ -255,7 +248,6 @@ if [ -z "$(ls -A "$SITE_DIR" 2>/dev/null)" ]; then
fi fi
tar -cf "$NEW_TAR" -C "$SITE_DIR" . tar -cf "$NEW_TAR" -C "$SITE_DIR" .
NEW_KB=$(du -sk "$NEW_TAR" 2>/dev/null | awk '{print $1}')
echo "PUT: replacing site contents..." echo "PUT: replacing site contents..."
HTTP_CODE=$(curl_with_host -X PUT "${PAGES_URL}/" \ HTTP_CODE=$(curl_with_host -X PUT "${PAGES_URL}/" \
@@ -267,9 +259,6 @@ HTTP_CODE=$(curl_with_host -X PUT "${PAGES_URL}/" \
echo "HTTP ${HTTP_CODE}" echo "HTTP ${HTTP_CODE}"
if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "201" ] || [ "$HTTP_CODE" = "204" ]; then if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "201" ] || [ "$HTTP_CODE" = "204" ]; then
echo "Site rebuild completed." echo "Site rebuild completed."
if [ -n "${OLD_KB:-}" ]; then
echo " archive size: $(fmt_num $OLD_KB)kB → $(fmt_num $NEW_KB)kB"
fi
else else
echo "ERROR: PUT HTTP ${HTTP_CODE}" >&2 echo "ERROR: PUT HTTP ${HTTP_CODE}" >&2
exit 1 exit 1
-184
View File
@@ -1,184 +0,0 @@
#!/usr/bin/env bash
# Shared functions for retention-cleanup.sh
# Can be sourced by tests for unit testing
age_days() {
local published="$1" epoch_pub now
epoch_pub=$(date -d "$published" +%s 2>/dev/null || date -j -f "%Y-%m-%dT%H:%M:%SZ" "$published" +%s 2>/dev/null || echo 0)
[ "$epoch_pub" -eq 0 ] && echo 99999 && return
now=$(date -u +%s)
echo $(( (now - epoch_pub) / 86400 ))
}
parse_path() {
local rel="$1"
OWNER="${rel%%/*}"
rest="${rel#*/}"
REPO="${rest%%/*}"
}
read_rule() {
local config="$1" branch="$2" key="$3" default="$4"
v=$(jq -r --arg b "$branch" --arg k "$key" '.branches[$b][$k] // empty' "$config")
[ -n "$v" ] && echo "$v" || echo "$default"
}
# ---------------------------------------------------------------------------
# Gitea branch/repo checking via git ls-remote
# Uses global: GITEA_API_URL, GITEA_TOKEN
# Sets global: REPO_BRANCHES_CACHE, REPO_STATUS
# ---------------------------------------------------------------------------
# Fetch all branches for a repo (one git ls-remote call per repo).
# Sets REPO_STATUS[owner/repo].
# Echos branch list on success.
# Returns: 0=ok, 1=deleted, 2=cert_error, 3=error (fail-safe keep)
repo_branches() {
local owner="$1" repo="$2" key="${owner}/${repo}"
local attempt output
[ -z "$GITEA_API_URL" ] && return 0
[ -z "$GITEA_TOKEN" ] && return 0
# Check cached status first (avoids re-running git on every report)
case "${REPO_STATUS[$key]:-}" in
deleted) return 1 ;;
cert_error) return 2 ;;
error) echo "${REPO_BRANCHES_CACHE[$key]:-}"; return 3 ;;
esac
# Cache hit (success with branch list)
[ -n "${REPO_BRANCHES_CACHE[$key]:-}" ] && { echo "${REPO_BRANCHES_CACHE[$key]}"; return 0; }
local git_host
git_host=$(echo "$GITEA_API_URL" | sed -E 's|^https?://||' | sed 's|/.*$||')
local git_url="https://token:${GITEA_TOKEN}@${git_host}/${owner}/${repo}.git"
for attempt in 1 2 3; do
output=$(git ls-remote --heads "$git_url" 2>&1) && {
local branches
branches=$(echo "$output" | sed -n 's|.*refs/heads/||p')
REPO_BRANCHES_CACHE[$key]="$branches"
REPO_STATUS[$key]="ok"
echo "$branches"
return 0
}
# Repo deleted → no retry
if echo "$output" | grep -qiE "fatal:.*(not found|repository.*not|could not read)"; then
REPO_BRANCHES_CACHE[$key]="__REPO_DELETED__"
REPO_STATUS[$key]="deleted"
echo " REPO DELETED: ${owner}/${repo}" >&2
return 1
fi
[ "$attempt" -lt 3 ] && sleep 10
done
# Certificate verification failure → configuration error, stop
if echo "$output" | grep -qi "server certificate verification failed"; then
REPO_STATUS[$key]="cert_error"
echo "[ERROR] git-pages.retention: certificate verification failed for ${owner}/${repo}" >&2
echo "[ERROR] git-pages.retention: check CA certificates or set GIT_SSL_NO_VERIFY=1" >&2
echo "[ERROR] git-pages.retention: git output:" >&2
echo "$output" >&2
return 2
fi
# Other network errors → fail-safe keep, continue
REPO_BRANCHES_CACHE[$key]="__REPO_ERROR__"
REPO_STATUS[$key]="error"
echo "[WARN] git-pages.retention: cannot reach Gitea for ${owner}/${repo} — keeping all reports" >&2
echo "[WARN] git-pages.retention: git output:" >&2
echo "$output" >&2
return 3
}
# Check if a specific branch exists in a repo.
# Returns 0 (exists), 1 (not found/deleted).
# Returns 2 (cert error), 3 (network error).
branch_exists() {
local owner="$1" repo="$2" branch="$3"
local branches rc
[ -z "$GITEA_API_URL" ] && return 0
[ -z "$GITEA_TOKEN" ] && return 0
branches=$(repo_branches "$owner" "$repo")
rc=$?
# Return codes from repo_branches propagate through $() subshell:
# 0=ok, 1=deleted, 2=cert_error, 3=error
case $rc in
2) echo "[FATAL] git-pages.retention: cannot reach Gitea (${owner}/${repo}) — check configuration" >&2
exit 1 ;;
3) return 0 ;; # network error → fail-safe keep
1) return 1 ;; # repo/branch gone
esac
echo "$branches" | grep -qxF "$branch"
}
# Phase 3: apply retention rules to KEEP array, populate TO_DELETE
# Reads from global KEEP array
# Populates global TO_DELETE array
# Usage: apply_retention <config_path>
apply_retention() {
local config="$1"
local default_max_age default_keep_min
local max_age keep_min key count seen_key commit_dir
local entry dir owner repo branch days
default_max_age=$(jq -r '.branches.default.maxAgeDays // 90' "$config")
default_keep_min=$(jq -r '.branches.default.keepMin // 5' "$config")
declare -A BRANCH_COUNTS
declare -A SEEN_COMMITS
declare -A DELETED_COMMITS
if [ "${#KEEP[@]}" -eq 0 ]; then
return
fi
IFS=$'\n'
for entry in $(printf '%s\n' "${KEEP[@]}" | sort -t'|' -k4,4 -k5,5n); do
IFS='|' read -r dir owner repo branch days <<< "$entry"
max_age=$(read_rule "$config" "$branch" "maxAgeDays" "$default_max_age")
keep_min=$(read_rule "$config" "$branch" "keepMin" "$default_keep_min")
# Age check — per-report-type deletion
if [ "$days" -gt "$max_age" ]; then
echo " DELETE: ${dir} (age ${days}d > maxAge ${max_age}d, branch ${branch})"
TO_DELETE+=("$dir")
REASON_MAP["$dir"]="maxAgeDays exceed"
MAXAGE_DELETED=$((MAXAGE_DELETED + 1))
continue
fi
# keepMin — per-commit counting
commit_dir=$(dirname "$dir")
key="$branch"
seen_key="${key}|${commit_dir}"
if [ -z "${SEEN_COMMITS[$seen_key]:-}" ]; then
SEEN_COMMITS["$seen_key"]=1
count="${BRANCH_COUNTS[$key]:-0}"
count=$((count + 1))
BRANCH_COUNTS["$key"]=$count
else
count="${BRANCH_COUNTS[$key]:-0}"
fi
if [ "$count" -gt "$keep_min" ]; then
if [ -z "${DELETED_COMMITS[$commit_dir]:-}" ]; then
DELETED_COMMITS[$commit_dir]=1
echo " DELETE: ${commit_dir} (kept ${keep_min}/${count} commits, exceeds keepMin, branch ${branch})"
TO_DELETE+=("$commit_dir")
REASON_MAP["$commit_dir"]="keepMin exceed"
KEEPMIN_DELETED=$((KEEPMIN_DELETED + 1))
fi
fi
done
unset IFS
}
+1 -8
View File
@@ -70,15 +70,8 @@ spec:
set -euo pipefail set -euo pipefail
echo "Retention sidecar: installing deps..." echo "Retention sidecar: installing deps..."
apt-get update -qq apt-get update -qq
apt-get install -y --no-install-recommends curl jq git ca-certificates >/dev/null apt-get install -y --no-install-recommends curl jq python3 >/dev/null
echo "Retention sidecar: ready" echo "Retention sidecar: ready"
# Sleep until 01:00 so retention runs at night
now_epoch=$(date +%s)
target_epoch=$(date -d "today 01:00:00" +%s)
[ "$target_epoch" -le "$now_epoch" ] && target_epoch=$((target_epoch + 86400))
sleep_sec=$((target_epoch - now_epoch))
echo "Retention sidecar: next run in $((sleep_sec / 3600))h (at 01:00)"
sleep $sleep_sec
while true; do while true; do
/scripts/retention-cleanup.sh /scripts/retention-cleanup.sh
echo "Retention sidecar: next run in 24h" echo "Retention sidecar: next run in 24h"
@@ -8,8 +8,6 @@ metadata:
data: data:
retention.json: | retention.json: |
{{- .Values.retention.rules | toJson | nindent 4 }} {{- .Values.retention.rules | toJson | nindent 4 }}
retention-lib.sh: |
{{- .Files.Get "files/retention-lib.sh" | nindent 4 }}
retention-cleanup.sh: | retention-cleanup.sh: |
{{- .Files.Get "files/retention-cleanup.sh" | nindent 4 }} {{- .Files.Get "files/retention-cleanup.sh" | nindent 4 }}
retention-run.sh: | retention-run.sh: |
+1 -1
View File
@@ -33,7 +33,7 @@ spec:
- | - |
set -euo pipefail set -euo pipefail
apt-get update -qq apt-get update -qq
apt-get install -y --no-install-recommends curl jq git >/dev/null apt-get install -y --no-install-recommends curl jq >/dev/null
chmod +x /scripts/retention-run.sh /scripts/retention-cleanup.sh chmod +x /scripts/retention-run.sh /scripts/retention-cleanup.sh
/scripts/retention-run.sh /scripts/retention-run.sh
env: env:
-623
View File
@@ -1,623 +0,0 @@
#!/usr/bin/env bats
setup() {
source "$(dirname "$BATS_TEST_DIRNAME")/files/retention-lib.sh"
declare -gA REPO_BRANCHES_CACHE
declare -gA REPO_STATUS
declare -gA REASON_MAP
MAXAGE_DELETED=0
KEEPMIN_DELETED=0
CONFIG=$(mktemp)
}
teardown() {
rm -f "$CONFIG"
}
write_config() {
cat > "$CONFIG"
}
# ---------------------------------------------------------------------------
# read_rule
# ---------------------------------------------------------------------------
@test "read_rule returns default when branch has no override" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":5}}}
EOF
result=$(read_rule "$CONFIG" "nonexistent" "maxAgeDays" 90)
[ "$result" = "90" ]
}
@test "read_rule returns branch-specific value" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":5},"main":{"maxAgeDays":365,"keepMin":20}}}
EOF
result=$(read_rule "$CONFIG" "main" "keepMin" 5)
[ "$result" = "20" ]
}
@test "read_rule returns default for undefined key even if branch exists" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":5},"main":{"maxAgeDays":365}}}
EOF
result=$(read_rule "$CONFIG" "main" "keepMin" 5)
[ "$result" = "5" ]
}
# ---------------------------------------------------------------------------
# parse_path
# ---------------------------------------------------------------------------
@test "parse_path extracts owner and repo" {
parse_path "my-owner/my-repo/reports/abc123/go-test-unit"
[ "$OWNER" = "my-owner" ]
[ "$REPO" = "my-repo" ]
}
@test "parse_path handles owner with hyphens" {
parse_path "niko/agent-platform/reports/abc1234/go-test-bdd"
[ "$OWNER" = "niko" ]
[ "$REPO" = "agent-platform" ]
}
# ---------------------------------------------------------------------------
# apply_retention — keepMin per commit
# ---------------------------------------------------------------------------
@test "keepMin: 6 commits × 4 types, keepMin=10 → all kept (6 commits < 10)" {
# With old per-file counting, 24 files > 10 keepMin would delete 14.
# With per-commit counting, 6 commits < 10 keepMin keeps everything.
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":365,"keepMin":10}}}
EOF
KEEP=()
local -a commits=(c1 c2 c3 c4 c5 c6)
local -a ages=(100 80 60 40 20 5)
local -a types=(go-test-bdd go-test-unit helm-lint helm-kubeconform)
for i in "${!commits[@]}"; do
for t in "${types[@]}"; do
KEEP+=("niko/agent-platform/reports/${commits[$i]}/$t|niko|agent-platform|main|${ages[$i]}")
done
done
TO_DELETE=()
apply_retention "$CONFIG"
[ "${#TO_DELETE[@]}" -eq 0 ]
}
@test "keepMin: 8 commits × 1 type, keepMin=5 → deletes 3 oldest" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":365,"keepMin":5}}}
EOF
KEEP=()
local -a ages=(80 70 60 50 40 30 20 10)
for i in "${!ages[@]}"; do
KEEP+=("niko/r/reports/c$((i+1))/test|niko|r|main|${ages[$i]}")
done
TO_DELETE=()
apply_retention "$CONFIG"
# 5 newest (c8-c4) kept, 3 oldest (c3,c2,c1) deleted
[ "${#TO_DELETE[@]}" -eq 3 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/c3" ]]
[[ "${TO_DELETE[1]}" == "niko/r/reports/c2" ]]
[[ "${TO_DELETE[2]}" == "niko/r/reports/c1" ]]
}
@test "keepMin: 12 commits × 1 type, keepMin=5 → keeps 5 newest, deletes 7 oldest" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":5}}}
EOF
KEEP=()
for i in $(seq 1 12); do
KEEP+=("niko/r/reports/c${i}/test|niko|r|feature/foo|$(( 13 - i ))")
done
TO_DELETE=()
apply_retention "$CONFIG"
# 7 oldest commits deleted (12 - 5 = 7)
[ "${#TO_DELETE[@]}" -eq 7 ]
# Oldest 7 should be c1..c7 (highest days = oldest = processed last after sort)
# Sort is ascending by days, so processed as c12(1d), c11(2d), ..., c1(12d)
# keepMin=5: c12-c8 kept, c7-c1 deleted
[[ "${TO_DELETE[0]}" == "niko/r/reports/c7" ]]
[[ "${TO_DELETE[6]}" == "niko/r/reports/c1" ]]
}
@test "keepMin: 2 commits × 3 types, keepMin=5 → all kept (2 < 5)" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":5}}}
EOF
KEEP=()
KEEP+=("niko/r/reports/c1/test-a|niko|r|main|30")
KEEP+=("niko/r/reports/c1/test-b|niko|r|main|30")
KEEP+=("niko/r/reports/c1/test-c|niko|r|main|30")
KEEP+=("niko/r/reports/c2/test-a|niko|r|main|10")
KEEP+=("niko/r/reports/c2/test-b|niko|r|main|10")
KEEP+=("niko/r/reports/c2/test-c|niko|r|main|10")
TO_DELETE=()
apply_retention "$CONFIG"
[ "${#TO_DELETE[@]}" -eq 0 ]
}
@test "keepMin: 6 commits × 2 types, keepMin=3 → keeps 3 newest, deletes 3 oldest" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":365,"keepMin":3}}}
EOF
KEEP=()
local -a commits=(c1 c2 c3 c4 c5 c6)
local -a ages=(60 50 40 30 20 10)
local -a types=(jest pytest)
for i in "${!commits[@]}"; do
for t in "${types[@]}"; do
KEEP+=("niko/r/reports/${commits[$i]}/$t|niko|r|feature/x|${ages[$i]}")
done
done
TO_DELETE=()
apply_retention "$CONFIG"
# Sort by days ascending: c6(10d), c5(20d), c4(30d), c3(40d), c2(50d), c1(60d)
# keepMin=3: c6,c5,c4 kept; c3,c2,c1 deleted
[ "${#TO_DELETE[@]}" -eq 3 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/c3" ]]
[[ "${TO_DELETE[1]}" == "niko/r/reports/c2" ]]
[[ "${TO_DELETE[2]}" == "niko/r/reports/c1" ]]
}
# ---------------------------------------------------------------------------
# apply_retention — maxAge
# ---------------------------------------------------------------------------
@test "maxAge: report exceeding maxAge is deleted" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":5}}}
EOF
KEEP=(
"niko/r/reports/c1/test|niko|r|main|100"
"niko/r/reports/c2/test|niko|r|main|50"
)
TO_DELETE=()
apply_retention "$CONFIG"
# c1 (100d) > 90, deleted; c2 (50d) < 90, kept
[ "${#TO_DELETE[@]}" -eq 1 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/c1/test" ]]
}
@test "maxAge deletes report-level dir, not commit-level" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":30,"keepMin":5}}}
EOF
KEEP=(
"niko/r/reports/c1/test-a|niko|r|main|100"
"niko/r/reports/c1/test-b|niko|r|main|20"
"niko/r/reports/c2/test-a|niko|r|main|10"
)
TO_DELETE=()
apply_retention "$CONFIG"
# Only test-a for c1 is old; test-b for c1 is young, c2 is young
[ "${#TO_DELETE[@]}" -eq 1 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/c1/test-a" ]]
}
# ---------------------------------------------------------------------------
# apply_retention — maxAge + keepMin interaction
# ---------------------------------------------------------------------------
@test "maxAge takes precedence over keepMin — aged report deleted, not counted in keepMin" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":30,"keepMin":2}}}
EOF
# 3 commits, 1 type each. c1 is old (100d), c2 and c3 are young.
# With keepMin=2: c1 should be deleted by maxAge, c2 and c3 kept.
# Without the continue after maxAge check, c1 would consume a keepMin slot.
KEEP=(
"niko/r/reports/c1/test|niko|r|main|100"
"niko/r/reports/c2/test|niko|r|main|10"
"niko/r/reports/c3/test|niko|r|main|5"
)
TO_DELETE=()
apply_retention "$CONFIG"
# c1 deleted by maxAge, c2 and c3 within keepMin=2
[ "${#TO_DELETE[@]}" -eq 1 ]
}
# ---------------------------------------------------------------------------
# apply_retention — sorting (newest first)
# ---------------------------------------------------------------------------
@test "sort order: newest commits processed first within same branch" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":365,"keepMin":2}}}
EOF
KEEP=(
"niko/r/reports/c1/test|niko|r|main|100"
"niko/r/reports/c2/test|niko|r|main|50"
"niko/r/reports/c3/test|niko|r|main|10"
)
TO_DELETE=()
apply_retention "$CONFIG"
# Sort by days ascending: c3(10d) 1st, c2(50d) 2nd, c1(100d) 3rd
# keepMin=2: c3 and c2 kept, c1 deleted
[ "${#TO_DELETE[@]}" -eq 1 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/c1" ]]
}
@test "sort order: branches sorted alphabetically" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":365,"keepMin":1}}}
EOF
KEEP=(
"niko/r/reports/c1/test|niko|r|z-branch|50"
"niko/r/reports/c2/test|niko|r|a-branch|60"
"niko/r/reports/c3/test|niko|r|m-branch|10"
)
TO_DELETE=()
apply_retention "$CONFIG"
# Alphabetical: a-branch, m-branch, z-branch
# Each has 1 commit, keepMin=1 → nothing deleted
[ "${#TO_DELETE[@]}" -eq 0 ]
}
@test "multi-branch: each branch has own keepMin counter" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":2}}}
EOF
KEEP=(
"niko/r/reports/c1/test|niko|r|branch-a|30"
"niko/r/reports/c2/test|niko|r|branch-a|20"
"niko/r/reports/c3/test|niko|r|branch-a|10"
"niko/r/reports/c4/test|niko|r|branch-b|60"
"niko/r/reports/c5/test|niko|r|branch-b|50"
"niko/r/reports/c6/test|niko|r|branch-b|40"
"niko/r/reports/c7/test|niko|r|branch-b|30"
)
TO_DELETE=()
apply_retention "$CONFIG"
# branch-a: 3 reports → keep 2 newest (c2,c3), delete 1 oldest (c1)
# branch-b: 4 reports → keep 2 newest (c6,c7), delete 2 oldest (c4,c5)
# Actually: Sort is by branch, then by days ascending
# branch-a processed first: c3(10d) 1st, c2(20d) 2nd (keep), c1(30d) 3rd (delete)
# branch-b processed next: c7(30d) 1st, c6(40d) 2nd (keep), c5(50d) 3rd (delete), c4(60d) 4th (delete)
[ "${#TO_DELETE[@]}" -eq 3 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/c1" ]]
[[ "${TO_DELETE[1]}" == "niko/r/reports/c5" ]]
[[ "${TO_DELETE[2]}" == "niko/r/reports/c4" ]]
}
# ---------------------------------------------------------------------------
# apply_retention — empty / edge cases
# ---------------------------------------------------------------------------
@test "empty KEEP array → nothing deleted" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":5}}}
EOF
KEEP=()
TO_DELETE=()
apply_retention "$CONFIG"
[ "${#TO_DELETE[@]}" -eq 0 ]
}
@test "TO_DELETE preserves Phase 2 entries after apply_retention (no new deletions)" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":365,"keepMin":10}}}
EOF
KEEP=(
"niko/r/reports/c1/test|niko|r|main|10"
"niko/r/reports/c2/test|niko|r|main|5"
)
TO_DELETE=(
"niko/r/reports/abc/branch-gone"
"niko/r/reports/def/repo-gone"
)
apply_retention "$CONFIG"
[ "${#TO_DELETE[@]}" -eq 2 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/abc/branch-gone" ]]
[[ "${TO_DELETE[1]}" == "niko/r/reports/def/repo-gone" ]]
}
@test "TO_DELETE preserves Phase 2 entries AND adds retention deletions" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":365,"keepMin":3}}}
EOF
KEEP=(
"niko/r/reports/c1/test|niko|r|main|40"
"niko/r/reports/c2/test|niko|r|main|30"
"niko/r/reports/c3/test|niko|r|main|20"
"niko/r/reports/c4/test|niko|r|main|10"
)
TO_DELETE=(
"niko/r/reports/abc/branch-gone"
)
apply_retention "$CONFIG"
# 1 pre-existing + 1 commit deleted (c1, oldest of 4, keepMin=3)
[ "${#TO_DELETE[@]}" -eq 2 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/abc/branch-gone" ]]
}
# ---------------------------------------------------------------------------
# branch_exists — mocking REPO_STATUS / REPO_BRANCHES_CACHE
# ---------------------------------------------------------------------------
@test "branch_exists: branch in list → return 0" {
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test-token"
REPO_BRANCHES_CACHE["owner/repo"]=$'main\nfeature/x'
REPO_STATUS["owner/repo"]="ok"
run branch_exists "owner" "repo" "main"
[ "$status" -eq 0 ]
}
@test "branch_exists: branch not in list → return 1" {
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test-token"
REPO_BRANCHES_CACHE["owner/repo"]=$'main\nfeature/x'
REPO_STATUS["owner/repo"]="ok"
run branch_exists "owner" "repo" "nonexistent"
[ "$status" -eq 1 ]
}
@test "branch_exists: cert error → exit 1 with [FATAL]" {
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test"
REPO_STATUS["owner/repo"]="cert_error"
run branch_exists "owner" "repo" "any-branch"
[ "$status" -eq 1 ]
[[ "$output" == *"[FATAL]"* ]]
}
@test "branch_exists: repo deleted → return 1" {
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test-token"
REPO_BRANCHES_CACHE["owner/repo"]="__REPO_DELETED__"
REPO_STATUS["owner/repo"]="deleted"
run branch_exists "owner" "repo" "any-branch"
[ "$status" -eq 1 ]
}
@test "branch_exists: network error → return 0 (fail-safe keep)" {
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test-token"
REPO_BRANCHES_CACHE["owner/repo"]="__REPO_ERROR__"
REPO_STATUS["owner/repo"]="error"
run branch_exists "owner" "repo" "any-branch"
[ "$status" -eq 0 ]
}
@test "branch_exists: empty GITEA_API_URL → return 0 (skip)" {
GITEA_API_URL=""
GITEA_TOKEN="test-token"
run branch_exists "owner" "repo" "any-branch"
[ "$status" -eq 0 ]
}
@test "branch_exists: empty GITEA_TOKEN → return 0 (skip)" {
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN=""
run branch_exists "owner" "repo" "any-branch"
[ "$status" -eq 0 ]
}
# ---------------------------------------------------------------------------
# repo_branches — git ls-remote error detection patterns
# ---------------------------------------------------------------------------
@test "error detection: 'command not found' does NOT trigger repo deleted" {
# This must NOT match — "bash: git: command not found" is NOT a repo deletion
local msg="bash: git: command not found"
run grep -qiE "fatal:.*(not found|repository.*not|could not read)" <<< "$msg"
[ "$status" -eq 1 ]
}
@test "error detection: 'fatal: repo not found' triggers repo deleted" {
# This MUST match — genuine git error for deleted/missing repo
local msg="fatal: repository 'https://gitea.app/owner/repo.git' not found"
run grep -qiE "fatal:.*(not found|repository.*not|could not read)" <<< "$msg"
[ "$status" -eq 0 ]
}
@test "error detection: 'could not read from remote' triggers repo deleted" {
local msg="fatal: could not read from remote repository"
run grep -qiE "fatal:.*(not found|repository.*not|could not read)" <<< "$msg"
[ "$status" -eq 0 ]
}
# ---------------------------------------------------------------------------
# git ls-remote integration (real git, temp repo)
# ---------------------------------------------------------------------------
@test "git ls-remote parsing: lists branches correctly" {
local tmpdir=$(mktemp -d)
git -C "$tmpdir" init -b main source >/dev/null 2>&1
git -C "$tmpdir/source" config user.email "test@test"
git -C "$tmpdir/source" config user.name "test"
git -C "$tmpdir/source" commit --allow-empty -m "init" >/dev/null 2>&1
git -C "$tmpdir/source" branch feature/x >/dev/null 2>&1
git clone --bare "$tmpdir/source" "$tmpdir/repo.git" >/dev/null 2>&1
local url="file://$tmpdir/repo.git"
local output
output=$(git ls-remote --heads "$url" 2>&1)
local branches
branches=$(echo "$output" | sed -n 's|.*refs/heads/||p')
echo "$branches" | grep -qxF "main"
[ "$?" -eq 0 ]
echo "$branches" | grep -qxF "feature/x"
[ "$?" -eq 0 ]
! echo "$branches" | grep -qxF "nonexistent"
rm -rf "$tmpdir"
}
# ---------------------------------------------------------------------------
# repo_branches — retry + error output (using git mock)
# ---------------------------------------------------------------------------
@test "repo_branches: success returns branches immediately" {
local mockdir=$(mktemp -d)
cat > "$mockdir/git" << 'SCRIPT'
#!/usr/bin/env bash
echo "abc123 refs/heads/main"
echo "def456 refs/heads/feature/x"
SCRIPT
chmod +x "$mockdir/git"
local save_PATH="$PATH"
export PATH="$mockdir:$PATH"
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test"
REPO_BRANCHES_CACHE=()
REPO_STATUS=()
run repo_branches "owner" "repo"
[ "$status" -eq 0 ]
[[ "$output" == *"main"* ]]
[[ "$output" == *"feature/x"* ]]
export PATH="$save_PATH"
rm -rf "$mockdir"
}
@test "repo_branches: retries 3 times on transient error" {
local mockdir=$(mktemp -d)
cat > "$mockdir/git" << 'SCRIPT'
#!/usr/bin/env bash
echo "call" >> "$MOCKDIR/count"
echo "fatal: unable to access 'https://...'" >&2
exit 1
SCRIPT
chmod +x "$mockdir/git"
# Inject mockdir path into mock script via env var
sed -i '' "s|\$MOCKDIR|$mockdir|g" "$mockdir/git"
local save_PATH="$PATH"
export PATH="$mockdir:$PATH"
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test"
REPO_BRANCHES_CACHE=()
REPO_STATUS=()
local start=$SECONDS
run repo_branches "owner" "repo"
[ "$status" -eq 3 ]
[[ "$output" == *"[WARN] git-pages.retention"* ]]
[[ "$output" == *"keeping all reports"* ]]
[[ "$output" == *"git output:"* ]]
[[ "$output" == *"unable to access"* ]]
[ $(cat "$mockdir/count" | wc -l) -eq 3 ]
[ $(( SECONDS - start )) -ge 18 ]
export PATH="$save_PATH"
rm -rf "$mockdir"
}
@test "repo_branches: certificate error → [ERROR] + return 1" {
local mockdir=$(mktemp -d)
cat > "$mockdir/git" << 'SCRIPT'
#!/usr/bin/env bash
echo "call" >> "$MOCKDIR/count"
echo "fatal: unable to access 'https://gitea.app/owner/repo.git/': server certificate verification failed. CAfile: none CRLfile: none" >&2
exit 1
SCRIPT
chmod +x "$mockdir/git"
sed -i '' "s|\$MOCKDIR|$mockdir|g" "$mockdir/git"
local save_PATH="$PATH"
export PATH="$mockdir:$PATH"
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test"
REPO_BRANCHES_CACHE=()
REPO_STATUS=()
run repo_branches "owner" "repo"
[ "$status" -eq 2 ]
[[ "$output" == *"[ERROR]"* ]]
[[ "$output" == *"certificate verification"* ]]
[[ "$output" == *"git output:"* ]]
[[ "$output" == *"unable to access"* ]]
export PATH="$save_PATH"
rm -rf "$mockdir"
}
@test "repo_branches: repo not found returns immediately (no retry)" {
local mockdir=$(mktemp -d)
cat > "$mockdir/git" << 'SCRIPT'
#!/usr/bin/env bash
echo "call" >> "$MOCKDIR/count"
echo "fatal: repository 'https://gitea.app/owner/repo.git' not found" >&2
exit 1
SCRIPT
chmod +x "$mockdir/git"
sed -i '' "s|\$MOCKDIR|$mockdir|g" "$mockdir/git"
local save_PATH="$PATH"
export PATH="$mockdir:$PATH"
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test"
REPO_BRANCHES_CACHE=()
REPO_STATUS=()
run repo_branches "owner" "repo"
[ "$status" -eq 1 ]
[[ "$output" == *"REPO DELETED"* ]]
[[ "$output" != *"[WARN]"* ]]
[ $(cat "$mockdir/count" | wc -l) -eq 1 ]
export PATH="$save_PATH"
rm -rf "$mockdir"
}
+1 -4
View File
@@ -22,10 +22,7 @@ INPUTS=$(jq -nc \
--arg source_repo "$GITOPS_SOURCE_REPO" \ --arg source_repo "$GITOPS_SOURCE_REPO" \
--arg source_commit "$GITOPS_SOURCE_COMMIT" \ --arg source_commit "$GITOPS_SOURCE_COMMIT" \
--arg git_tag_prefix "${GITOPS_TAG_PREFIX:-}" \ --arg git_tag_prefix "${GITOPS_TAG_PREFIX:-}" \
--arg extra_cmd "${GITOPS_EXTRA_CMD:-}" \ '{file: $file, yq_tpl: $yq_tpl, version: $version, source_repo: $source_repo, source_commit: $source_commit, git_tag_prefix: $git_tag_prefix}')
--arg author_name "${GIT_USER_NAME:-}" \
--arg author_email "${GIT_USER_EMAIL:-}" \
'{file: $file, yq_tpl: $yq_tpl, version: $version, source_repo: $source_repo, source_commit: $source_commit, git_tag_prefix: $git_tag_prefix, extra_cmd: $extra_cmd, author_name: $author_name, author_email: $author_email}')
DIR="$(cd "$(dirname "$0")" && pwd)" DIR="$(cd "$(dirname "$0")" && pwd)"
echo "gitops-dispatch: dispatching to $GITOPS_REPO/$GITOPS_WORKFLOW..." echo "gitops-dispatch: dispatching to $GITOPS_REPO/$GITOPS_WORKFLOW..."
+2 -7
View File
@@ -84,12 +84,7 @@ _gitops_update() {
cd "${CLONE_DIR}" || _gitops_fail "Failed to enter clone directory" cd "${CLONE_DIR}" || _gitops_fail "Failed to enter clone directory"
yq eval -i "${YQ_EXPR}" "${INPUT_FILE}" || _gitops_fail "Failed to update ${INPUT_FILE}" yq eval -i "${YQ_EXPR}" "${INPUT_FILE}" || _gitops_fail "Failed to update ${INPUT_FILE}"
if [ -n "${GITOPS_EXTRA_CMD:-}" ]; then
eval "${GITOPS_EXTRA_CMD}" || _gitops_fail "Extra command failed: ${GITOPS_EXTRA_CMD}"
git add -A || _gitops_fail "Failed to stage all changes"
else
git add "${INPUT_FILE}" || _gitops_fail "Failed to stage ${INPUT_FILE}" git add "${INPUT_FILE}" || _gitops_fail "Failed to stage ${INPUT_FILE}"
fi
if git diff --cached --quiet; then if git diff --cached --quiet; then
echo "No changes — ${INPUT_FILE} already at ${VERSION}" echo "No changes — ${INPUT_FILE} already at ${VERSION}"
@@ -98,8 +93,8 @@ _gitops_update() {
exit 0 exit 0
fi fi
git -c user.name="${GIT_USER_NAME:-gitea-ci-bot}" \ git -c user.name="gitea-ci-bot" \
-c user.email="${GIT_USER_EMAIL:-ci@keskikuja.site}" \ -c user.email="ci@keskikuja.site" \
commit -m "[skip ci] gitops: update version to ${VERSION}" || _gitops_fail "Failed to commit" commit -m "[skip ci] gitops: update version to ${VERSION}" || _gitops_fail "Failed to commit"
GITOPS_SHA="$(git rev-parse HEAD)" GITOPS_SHA="$(git rev-parse HEAD)"
git push || _gitops_fail "Failed to push" git push || _gitops_fail "Failed to push"
+2 -36
View File
@@ -56,15 +56,6 @@ on:
git_tag_prefix: git_tag_prefix:
required: false required: false
type: string type: string
extra_cmd:
required: false
type: string
author_name:
required: false
type: string
author_email:
required: false
type: string
env: env:
INPUT_FILE: ${{ inputs.file }} INPUT_FILE: ${{ inputs.file }}
@@ -75,9 +66,6 @@ env:
GITOPS_REPO: ${{ github.repository }} GITOPS_REPO: ${{ github.repository }}
GITEA_API_URL: ${{ gitea.server_url }} GITEA_API_URL: ${{ gitea.server_url }}
GIT_TAG_PREFIX: ${{ inputs.git_tag_prefix || '' }} GIT_TAG_PREFIX: ${{ inputs.git_tag_prefix || '' }}
GITOPS_EXTRA_CMD: ${{ inputs.extra_cmd || '' }}
GIT_USER_NAME: ${{ inputs.author_name || '' }}
GIT_USER_EMAIL: ${{ inputs.author_email || '' }}
jobs: jobs:
update: update:
@@ -142,37 +130,15 @@ gitops-update:
This single job handles: dispatch → poll → find commit SHA → set commit-status on your commit → produce `GITOPS_SUMMARY` output. This single job handles: dispatch → poll → find commit SHA → set commit-status on your commit → produce `GITOPS_SUMMARY` output.
To run extra commands (e.g. `helm dependency update`) after the version bump and before the commit:
```yaml
gitops-update:
needs: [load-config, check-version, helm-build-push]
if: success()
uses: niko/gitea-ci-library/.gitea/workflows/gitops-dispatch.yml@v1
secrets: inherit
with:
env_json: ${{ needs.load-config.outputs.env_json }}
version: ${{ needs.check-version.outputs.version }}
GITOPS_FILE: Chart.yaml
GITOPS_YQ_TPL: '(.dependencies[] | select(.name == "agent-platform-helm") | .version) = "{{VERSION}}"'
GITOPS_REPO: niko/agent-platform-gitops
GITOPS_EXTRA_CMD: helm dependency update
```
When `GITOPS_EXTRA_CMD` is set, the script runs it after `yq` and stages all changes (`git add -A`) instead of only the input file — so any files generated by the extra command (e.g. `Chart.lock`, `charts/`) are included in the commit.
By default the GitOps commit is made as `gitea-ci-bot`. To use the original commit author instead, the dispatch workflow resolves it automatically from the consumer repo — no extra config needed. Just ensure the GitOps repo's `gitops-service.yaml` template has the `author_name` and `author_email` inputs and env mappings.
### 2.3 Parameters ### 2.3 Parameters
| Input | Required | Description | | Input | Required | Description |
|---|---|---|---| |---|---|---|
| `env_json` | Yes | Config JSON with `GITEA_API_URL`, optional `GIT_TAG_PREFIX` (for multi-component repos) | | `env_json` | Yes | Config JSON with `GITEA_API_URL`, optional `GIT_TAG_PREFIX` (for multi-component repos) |
| `version` | Yes | Version to write (e.g. `0.2.3`) | | `version` | Yes | Version to write (e.g. `0.2.3`) |
| `GITOPS_FILE` | Yes | Path in GitOps repo (e.g. `dev/Chart.yaml`) | | `GITOPS_FILE` | Yes | Path in GitOps repo (e.g. `dev/Chart.yaml`) |
| `GITOPS_YQ_TPL` | Yes | yq expression, `{{VERSION}}` is replaced at runtime | | `GITOPS_YQ_TPL` | Yes | yq expression, `{{VERSION}}` is replaced at runtime |
| `GITOPS_REPO` | Yes | GitOps repo slug (e.g. `niko/agent-platform-gitops`) | | `GITOPS_REPO` | Yes | GitOps repo slug (e.g. `niko/agent-platform-gitops`) |
| `GITOPS_EXTRA_CMD` | No | Shell command to run after yq update, before git commit (e.g. `helm dependency update`) |
### 2.4 Output ### 2.4 Output
@@ -211,7 +177,7 @@ report-summary:
## 4. What happens at runtime ## 4. What happens at runtime
1. Consumer's `gitops-dispatch.yml` generates a unique `dispatch_id` and POSTs it to the GitOps repo 1. Consumer's `gitops-dispatch.yml` generates a unique `dispatch_id` and POSTs it to the GitOps repo
2. GitOps workflow clones its own repo, applies `yq`, runs `GITOPS_EXTRA_CMD` if set, then commits + pushes 2. GitOps workflow clones its own repo, applies `yq`, commits + pushes
3. Consumer polls the GitOps repo's runs until the workflow completes 3. Consumer polls the GitOps repo's runs until the workflow completes
4. Consumer lists recent commits and finds the matching one by commit message `"gitops: update version to X.Y.Z"` 4. Consumer lists recent commits and finds the matching one by commit message `"gitops: update version to X.Y.Z"`
5. Consumer sets commit-status `gitops/{repo}[/{prefix}]` on its own commit with a link to the exact GitOps commit 5. Consumer sets commit-status `gitops/{repo}[/{prefix}]` on its own commit with a link to the exact GitOps commit