Compare commits

...

5 Commits

Author SHA1 Message Date
niko dd9cdf70b6 Featuer/gitops extra command (#47)
CI Main / Config load (push) Successful in 1m48s
CI Main / Latest versio (push) Successful in 1m38s
unit-tests Bats test report
CI Main / Bats tests (push) Successful in 2m15s
acc-tests Cucumber test report
CI Main / Cucumber tests (push) Successful in 2m46s
ci-docker-build-push Docker push 0.2.35
CI Main / Build & Push Docker (push) Successful in 40s
gitops/gitea-ci-library GitOps: 0.2.35
CI Main / GitOps (push) Successful in 1m34s
CI Main / Report Summary (push) Successful in 3s
CI Main / Move provider version tag (push) Successful in 11s
Co-authored-by: moilanik <niko.moilanen@tietoevry.com>
Reviewed-on: #47
2026-06-27 05:20:09 +03:00
niko 848ba723e4 Fix/git pages rentetin create tests (#46)
CI Main / Config load (push) Successful in 3m12s
CI Git-Pages Main / Config load (push) Successful in 3m4s
CI Main / Latest versio (push) Successful in 30s
CI Git-Pages Main / Latest version (push) Successful in 27s
ci-helm-build-push Helm push 0.1.9
unit-tests Bats test report
CI Main / Bats tests (push) Successful in 1m56s
CI Git-Pages Main / Build & Push Helm chart (push) Successful in 1m19s
acc-tests Cucumber test report
CI Main / Cucumber tests (push) Successful in 3m47s
ci-docker-build-push Docker push 0.2.34
CI Main / Build & Push Docker (push) Successful in 1m14s
gitops/gitea-ci-library/git-pages GitOps: git-pages 0.1.9
CI Git-Pages Main / GitOps (push) Successful in 3m37s
gitops/gitea-ci-library GitOps: 0.2.34
CI Main / GitOps (push) Successful in 40s
CI Main / Report Summary (push) Successful in 4s
CI Main / Move provider version tag (push) Successful in 12s
CI Git-Pages Main / Report Summary (push) Successful in 4s
Co-authored-by: moilanik <niko.moilanen@tietoevry.com>
Reviewed-on: #46
2026-06-26 08:07:02 +03:00
niko 1978a995a8 Update README.md (#45)
CI Main / Config load (push) Successful in 23s
CI Main / Latest versio (push) Successful in 21s
CI Main / Bats tests (push) Successful in 1m34s
CI Main / Cucumber tests (push) Successful in 1m33s
ci-docker-build-push Docker push 0.2.33
CI Main / Build & Push Docker (push) Successful in 50s
gitops/gitea-ci-library GitOps: 0.2.33
CI Main / GitOps (push) Successful in 47s
CI Main / Report Summary (push) Successful in 7s
CI Main / Move provider version tag (push) Successful in 14s
CI Feature / Load example-gitea-env.conf to pipeline env (push) Successful in 1m5s
unit-tests Bats test report
CI Feature / Bats tests (push) Successful in 2m9s
acc-tests Cucumber test report
CI Feature / Cucumber tests (push) Successful in 2m53s
CI Feature / Report Summary (push) Successful in 7s
Reviewed-on: #45
2026-06-25 09:19:01 +03:00
niko d6343438a3 Feture/gitops7 (#44)
unit-tests Bats test report
acc-tests Cucumber test report
CI Main / Cucumber tests (push) Successful in 1m44s
ci-docker-build-push Docker push 0.2.32
gitops/gitea-ci-library GitOps: 0.2.32
CI Main / Config load (push) Successful in 26s
CI Main / Latest versio (push) Successful in 21s
CI Main / Build & Push Docker (push) Has been skipped
CI Main / GitOps (push) Has been skipped
CI Main / Move provider version tag (push) Has been skipped
CI Main / Report Summary (push) Successful in 5s
CI Main / Bats tests (push) Has been skipped
Co-authored-by: moilanik <niko.moilanen@tietoevry.com>
Reviewed-on: #44
2026-06-23 12:45:42 +03:00
niko ed2703b7d7 commit status kutsujalle gitops repoon (#43)
CI Main / Config load (push) Successful in 19s
CI Main / Latest versio (push) Successful in 19s
unit-tests Bats test report
CI Main / Bats tests (push) Successful in 1m33s
acc-tests Cucumber test report
CI Main / Cucumber tests (push) Successful in 1m45s
ci-docker-build-push Docker push 0.2.31
CI Main / Build & Push Docker (push) Successful in 44s
gitops/gitea-ci-library GitOps: 0.2.31
CI Main / GitOps (push) Successful in 35s
CI Main / Report Summary (push) Successful in 7s
CI Main / Move provider version tag (push) Successful in 13s
Co-authored-by: moilanik <niko.moilanen@tietoevry.com>
Reviewed-on: #43
2026-06-22 14:24:25 +03:00
17 changed files with 1273 additions and 460 deletions
+1 -1
View File
@@ -23,7 +23,7 @@ on:
jobs:
build-push:
runs-on: ubuntu-latest
runs-on: docker
steps:
- uses: actions/checkout@v4
+2 -2
View File
@@ -32,7 +32,7 @@ concurrency:
jobs:
build-push:
runs-on: ubuntu-latest
runs-on: docker
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4
@@ -91,7 +91,7 @@ jobs:
fi
tag-commit:
runs-on: ubuntu-latest
runs-on: docker
needs: [build-push]
steps:
- uses: actions/checkout@v4
+44 -25
View File
@@ -17,12 +17,15 @@ on:
GITOPS_REPO:
required: true
type: string
GITOPS_EXTRA_CMD:
required: false
type: string
secrets:
GITOPS_DISPATCH_TOKEN:
required: true
outputs:
summary:
description: 'Pipe-format: component|version|status|commit_sha|repo'
description: "Pipe-format: component|version|status|commit_sha|repo"
value: ${{ jobs.dispatch.outputs.summary }}
env:
@@ -34,6 +37,7 @@ env:
GITOPS_SOURCE_COMMIT: ${{ github.sha }}
GITEA_API_URL: ${{ fromJson(inputs.env_json).GITEA_API_URL }}
GITOPS_TAG_PREFIX: ${{ fromJson(inputs.env_json).GIT_TAG_PREFIX || '' }}
GITOPS_EXTRA_CMD: ${{ inputs.GITOPS_EXTRA_CMD || '' }}
GITOPS_WORKFLOW: gitops-service.yaml
GITOPS_DISPATCH_TIMEOUT: 30
@@ -49,25 +53,39 @@ jobs:
ID=$(date +%s | md5sum | head -c 8)
echo "dispatch_id=$ID" >> "$GITHUB_OUTPUT"
- name: Resolve commit author
id: author
run: |
echo "name=$(git log -1 --format='%an')" >> "$GITHUB_OUTPUT"
echo "email=$(git log -1 --format='%ae')" >> "$GITHUB_OUTPUT"
- name: Dispatch to GitOps repo
env:
GITEA_TOKEN: ${{ secrets.GITOPS_DISPATCH_TOKEN }}
run: |
INPUTS=$(jq -nc \
--arg dispatch_id "${{ steps.gen.outputs.dispatch_id }}" \
--arg author_name "${{ steps.author.outputs.name }}" \
--arg author_email "${{ steps.author.outputs.email }}" \
--arg file "$GITOPS_FILE" \
--arg yq_tpl "$GITOPS_YQ_TPL" \
--arg version "$GITOPS_VERSION" \
--arg source_repo "$GITOPS_SOURCE_REPO" \
--arg source_commit "$GITOPS_SOURCE_COMMIT" \
--arg git_tag_prefix "${GITOPS_TAG_PREFIX:-}" \
'{dispatch_id: $dispatch_id, file: $file, yq_tpl: $yq_tpl, version: $version, source_repo: $source_repo, source_commit: $source_commit, git_tag_prefix: $git_tag_prefix}')
--arg extra_cmd "${GITOPS_EXTRA_CMD:-}" \
'{dispatch_id: $dispatch_id, author_name: $author_name, author_email: $author_email, file: $file, yq_tpl: $yq_tpl, version: $version, source_repo: $source_repo, source_commit: $source_commit, git_tag_prefix: $git_tag_prefix, extra_cmd: $extra_cmd}')
curl -s -X POST \
"${GITEA_API_URL}/api/v1/repos/${GITOPS_REPO}/actions/workflows/${GITOPS_WORKFLOW}/dispatches" \
-H "Authorization: token $GITEA_TOKEN" \
-H "Content-Type: application/json" \
-d "$(jq -nc --arg ref "main" --argjson inputs "$INPUTS" '{ref: "main", inputs: $inputs}')"
- uses: actions/checkout@v4
with:
repository: niko/gitea-ci-library
path: .ci
- name: Poll for completion
id: poll
env:
@@ -131,32 +149,12 @@ jobs:
--arg msg "$SEARCH_MSG" \
'[.[] | select(.commit.message | contains($msg))] | .[0].sha // empty')
# 3. If not found → no change → use HEAD
# 3. If not found → fail
if [ -z "$GITOPS_COMMIT" ]; then
GITOPS_COMMIT=$(echo "$COMMITS" | jq -r '.[0].sha // empty')
echo "ERROR: no matching GitOps commit found for version ${GITOPS_VERSION}" >&2
exit 1
fi
# 4. Set commit status on caller's own commit
PREFIX="${GITOPS_TAG_PREFIX%/}"
if [ -n "$PREFIX" ]; then
CONTEXT="gitops/$(basename "${GITOPS_SOURCE_REPO}")/${PREFIX}"
DESCRIPTION="GitOps: ${PREFIX} ${GITOPS_VERSION}"
else
CONTEXT="gitops/$(basename "${GITOPS_SOURCE_REPO}")"
DESCRIPTION="GitOps: ${GITOPS_VERSION}"
fi
STATUS_URL="${GITEA_API_URL}/api/v1/repos/${GITOPS_SOURCE_REPO}/statuses/${GITOPS_SOURCE_COMMIT}"
STATUS_BODY=$(jq -nc \
--arg state "success" \
--arg description "$DESCRIPTION" \
--arg context "$CONTEXT" \
--arg target_url "${GITEA_API_URL}/${GITOPS_REPO}/commit/${GITOPS_COMMIT}" \
'{state: $state, target_url: $target_url, description: $description, context: $context}')
curl -s -X POST "$STATUS_URL" \
-H "Authorization: token $GITEA_TOKEN" \
-H "Content-Type: application/json" \
-d "$STATUS_BODY" || true
echo "GITOPS_COMMIT=$GITOPS_COMMIT" >> "$GITHUB_OUTPUT"
echo "$GITOPS_COMMIT" > /tmp/gitops-commit
exit 0
@@ -181,3 +179,24 @@ jobs:
fi
COMPONENT="${GITOPS_TAG_PREFIX:-${GITOPS_FILE}}"
echo "GITOPS_SUMMARY=${COMPONENT}|${GITOPS_VERSION}|${STATUS}|${GITOPS_SHA}|${GITOPS_REPO}" >> "$GITHUB_OUTPUT"
- name: Set commit status
if: success()
env:
GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
GITOPS_SHA=$(cat /tmp/gitops-commit)
PREFIX="${GITOPS_TAG_PREFIX%/}"
if [ -n "$PREFIX" ]; then
CONTEXT="gitops/$(basename "${GITOPS_SOURCE_REPO}")/${PREFIX}"
DESCRIPTION="GitOps: ${PREFIX} ${GITOPS_VERSION}"
else
CONTEXT="gitops/$(basename "${GITOPS_SOURCE_REPO}")"
DESCRIPTION="GitOps: ${GITOPS_VERSION}"
fi
ROOT_REPO="${GITOPS_SOURCE_REPO}" ROOT_COMMIT="${GITOPS_SOURCE_COMMIT}" \
bash .ci/scripts/report-status.sh success \
"$DESCRIPTION" "$CONTEXT" "" \
"${GITEA_API_URL}/${GITOPS_REPO}/commit/${GITOPS_SHA}"
+13
View File
@@ -8,6 +8,12 @@ on:
version:
required: true
type: string
chart_path:
required: true
type: string
extra_dependency_paths:
required: false
type: string
secrets:
GITEA_TOKEN:
required: true
@@ -47,6 +53,13 @@ jobs:
repository: niko/gitea-ci-library
path: .ci
- name: Resolve extra subchart dependencies
if: inputs.extra_dependency_paths != ''
run: |
for path in $(echo "${{ inputs.extra_dependency_paths }}" | tr ',' '\n'); do
helm dependency update "${path}"
done
- name: Package Helm chart
run: |
CHART_DIR=$(dirname "${CHART_FILE}")
+61 -14
View File
@@ -141,7 +141,7 @@ kubectl create secret generic act-runner-token \
### 4. Helm install / upgrade
Menee samalla komennolla.
Menee samalla komennolla.
> Muista asettaa variables ennen ajoa.
@@ -151,58 +151,102 @@ ja sitä kautta Gitea ei tarvitse päivityksessä mitään temppuja.
Päivityksen jälkeen muista tappaa pod (käynnistyy automaattisesti uudelleen), että lataa varmasti kaikki uudesta. Sillä ConfigMap tms eivät lataudu
mikäli pod jatkaa ajamista.
Klusterissa on kaksi StatefulSetiä, joilla on eri labelit:
- **`act-runner`** — yleisrunneri (label `ubuntu-latest`). DinD-sidecar on olemassa mutta idle — `require_docker: false` estää runneria käyttämästä sitä.
- **`act-runner-docker`** — Docker-buildien runneri (label `docker`). DinD on aktiivinen, `require_docker: true`.
Vain `docker-build-push.yml` ja `ci-container-build-push.yml` käyttävät `docker`-labelia.
Kaikki muut workflowt (testit, lintit, helm-publish, gitops-dispatch) ajetaan `ubuntu-latest`-runnerilla.
Steppien suoritus ei mene Docker Daemonin läpi, joten konekielisiä kontteja luodaan suoraan K8s-runtimella.
```bash
helm repo add gitea https://dl.gitea.com/charts
helm repo update
# 1. Yleisrunneri (DinD idle)
helm upgrade --install act-runner gitea/actions \
--set enabled=true \
--set giteaRootURL="$GITEA_URL" \
--set existingSecret=act-runner-token \
--set existingSecretKey=token \
--set statefulset.replicas=3 \
--set statefulset.replicas=2 \
--set statefulset.runner.tag=1.0.8 \
--set statefulset.dind.tag=29.5.2-dind \
--set statefulset.dind.resources.requests.memory=250Mi \
--set statefulset.dind.resources.limits.memory=750Mi \
--set-string 'statefulset.runner.config=log:
level: info
cache:
enabled: false
enabled: true
container:
require_docker: true
docker_timeout: 300s' \
require_docker: false' \
--namespace "$GITEA_ACTIONS_NAMESPACE" \
--create-namespace
```
path escapes from parent -bugi korjattiin Docker 29.5.2:ssa. Tämän teko aikana default on 29.5.1 — juuri tämän alle jäävä versio.
# 2. Docker-runner (DinD)
helm upgrade --install act-runner-docker gitea/actions \
--set enabled=true \
--set giteaRootURL="$GITEA_URL" \
--set existingSecret=act-runner-token \
--set existingSecretKey=token \
--set statefulset.replicas=1 \
--set statefulset.runner.tag=1.0.8 \
--set statefulset.dind.tag=29.5.2-dind \
--set statefulset.dind.resources.requests.memory=250Mi \
--set statefulset.dind.resources.limits.memory=750Mi \
--set-string 'statefulset.runner.config=log:
level: info
cache:
enabled: true
container:
require_docker: true
docker_timeout: 300s
runner:
labels:
- "docker:docker://catthehacker/ubuntu:act-latest"' \
--namespace "$GITEA_ACTIONS_NAMESPACE"
```
`path escapes from parent` -bugi korjattiin Docker 29.5.2:ssa. Tämän teko aikana default on 29.5.1 — juuri tämän alle jäävä versio.
Oletus-lokitaso on `debug` — suositeltu `info`. Näkee jobien aloitukset ja valmistumiset ilman konttikerrosten purkua (Downloading/Extracting-spämmiä). `debug` on tarpeen vain vianselvityksessä.
`cache.enabled: true` nappaa image-cachen käyttöön — ilman sitä jokainen ajo lataa konttikuvat uudestaan.
#### Docker (DinD)
Helm chart deployaa DinD:n init-sidecarina (`docker:dind` samassa podissa).
`require_docker: true` kytkee jobit siihen — erillistä DinD-asennusta ei tarvita.
> **Huomio:** Gitea 1.26.x ei tue vielä label-pohjaista runner-valintaa.
> `runs-on: ubuntu-latest`-jobi saattaa päätyä `docker`-labeliselle runnerille.
> Bugi on tunnettu Gitean FAQ:ssa — korjaus tulossa myöhemmässä versiossa.
> Katso: [docs.gitea.com/usage/actions/faq](https://docs.gitea.com/usage/actions/faq)
**DinD-tag pinottu:** `29.5.2-dind` (ei chart-oletusta). Docker 29.5.1 aiheuttaa act-runnerissa
`path escapes from parent` -virheen job-kontin käynnistyksessä.
Helm chart deployaa DinD:n init-sidecarina (`docker:dind` samassa podissa).
Molemmissa StatefulSetissä on DinD-sidecar, mutta:
- **`act-runner`**: `require_docker: false` → runner ei käytä DinD:tä lainkaan, steppien suoritus menee suoraan K8s-runtimella
- **`act-runner-docker`**: `require_docker: true` → runner luo steppikontit DinD:n kautta (tarvitaan `docker build` -komentoja varten)
**DinD-tag pinottu:** `29.5.2-dind` molemmissa (ei chart-oletusta `29.5.1-dind`). Docker 29.5.1 aiheuttaa act-runnerissa
`path escapes from parent` -virheen job-kontin käynnistyksessä sekä `mkdirat var/run: file exists` -virheen tiedostojen kopioinnissa.
Maven/npm-ajot käyttävät vain workflow'n `container:`-imagea; DinD tarvitaan vasta Docker-buildissä.
### 3. Varmista
### 5. Varmista
```bash
kubectl get pods -n gitea-actions
# → act-runner-runner-0 Running
# → act-runner-docker-runner-0 Running
kubectl exec -n gitea-actions act-runner-runner-0 -c dind -- docker version
kubectl exec -n gitea-actions act-runner-docker-runner-0 -c dind -- docker version
# → Server Version: 29.5.2 (tai uudempi)
```
Gitean puolella runner ilmestyy Active-tilaan pienellä viiveellä:
Gitean puolella runnerit ilmestyvät Active-tilaan pienellä viiveellä:
```
Site Admin → Actions → Runners (tai Org → Settings → Actions → Runners)
# → act-runner-runner-0 Active ubuntu-latest
# → act-runner-docker-runner-0 Active docker
```
Tämän jälkeen `.gitea/workflows/ci.yml` triggeröityy automaattisesti pushista.
@@ -279,4 +323,7 @@ Tarkka asennus: [skills/gitops-update/SKILL.md](skills/gitops-update/SKILL.md)
| `existingSecret` | Kubernetes secretin nimi, jossa token |
| `existingSecretKey` | Avain secretin sisällä |
| `statefulset.dind.tag` | DinD-image tag (`29.5.2-dind` minimi) |
| `statefulset.dind.resources.requests.memory` | DinD muistirequest (suositus `250Mi`) |
| `statefulset.dind.resources.limits.memory` | DinD muistilimitti (suositus `750Mi`) |
| `statefulset.runner.labels` | Mukautetut labelit |
+6 -4
View File
@@ -103,7 +103,8 @@ joten `actions/checkout` toimii ilman node-asennuksia.
|-----------|------------|--------|
| `env_json` | Kyllä | Konffi `gitea-env.conf`:stä |
| `version` | Kyllä | Version string (check-version output) |
| `chart_path` | Ei | Polku Chart.yaml-hakemistoon, oletus `.` |
| `chart_path` | Kyllä | Polku Chart.yaml-hakemistoon |
| `extra_dependency_paths` | Ei | Pilkulla erotellut polut subcharttien dependeinceille, joille ajetaan `helm dependency update` ennen päächartin buildia |
**`env_json`-avaimet:**
@@ -124,9 +125,10 @@ build-push (helm package → helm push OCI) → tag-commit (git-tagin luonti)
**Steppien kuvaus `build-push`-jobissa:**
1. **Node.js-asennus**`apk add --no-cache nodejs` (vaaditaan `actions/checkout`-actionia varten)
2. **Checkout** — sovellusrepo ja gitea-ci-library `.ci/`-polkuun
3. **Package**`helm package` versiolla `$VERSION`
4. **Push OCI**`helm push` registryyn autentikoinnilla
5. **Report status** — commit-status + UI-linkki
3. **Resolve extra subchart dependencies**`helm dependency update` jokaiselle `extra_dependency_paths`-polulle (vain jos input on annettu)
4. **Package**`helm dependency update` + `helm package` versiolla `$VERSION`
5. **Push OCI**`helm push` registryyn autentikoinnilla
6. **Report status** — commit-status + UI-linkki
**Kompromissi:** Kontti `alpine/helm` ei sisällä node.js:ää, mutta
`actions/checkout@v4` on JavaScript-action ja vaatii sen. Siksi nodejs
+39
View File
@@ -149,3 +149,42 @@ curl -X PATCH https://ci-reports.helm-dev.keskikuja.site/owner/repo/commit/sha8/
- `git-pages-publish-token` = plaintext (luetaan Giteaan viedessä)
Tarkemmat secret-ohjeet: [docs/secrets.md](docs/secrets.md).
---
## Testaus
Retention-logiikalle on unit-testit, jotka testaa funktiot ja Phase 3 -säännöt
erikseen ilman ulkoisia riippuvuuksia.
```bash
cd git-pages
bats tests/retention.bats
```
Testit käyttävät `<root>/files/retention-lib.sh` -jaettua kirjastoa, jota myös
`retention-cleanup.sh` sourceaa. Uutta testiä kirjoittaessa:
1. Luo config `write_config`-helperilla
2. Täytä `KEEP`-array testidatalla (muoto: `dir|owner|repo|branch|days`)
3. Kutsu `apply_retention "$CONFIG"`
4. Tarkista `TO_DELETE`-array ja `$output`
**Vaatimukset:** `bats`, `jq`, `date` (GNU date tai BSD date ISO 8601 -tuella).
---
## Retention
Ylläpitoscripti, joka poistaa vanhat raportit git-pagesista retentionsääntöjen mukaan.
Ajetaan sidecar tai cronjobtilassa Kubernetesissa.
### Air gap -yhteensopimattomuus
Retentionkontti asentaa tarvitsemansa työkalut (`curl`, `jq`) ajon aikana
packagemanagerilla (`apt-get` / `apk`). Tämä **ei toimi air gap -ympäristössä**,
jossa konttirekisteriin tai pakettivarastoihin ei ole verkkoyhteyttä.
**TODO:** Rakenna custom Dockerimage, jossa deps on valmiina:
`FROM alpine:latest && apk add --no-cache curl jq`.
Pushaa omaan rekisteriin ja päivitä `values.yaml`:n `retention.image`.
+117 -106
View File
@@ -13,75 +13,11 @@ curl_with_host() {
[ -f "$CONFIG" ] || { echo "ERROR: config missing: $CONFIG" >&2; exit 1; }
declare -A BRANCH_CACHE
branch_exists() {
local owner="$1" repo="$2" branch="$3" key="${owner}/${repo}/${branch}"
local status attempt
[ -z "$GITEA_API_URL" ] && return 0
[ -z "$GITEA_TOKEN" ] && return 0
if [ "${BRANCH_CACHE[$key]:-}" = "1" ]; then
return 0
fi
# Retry up to 2 times on API errors (hardcoded)
for attempt in 1 2 3; do
status=$(curl -sS -o /dev/null -w "%{http_code}" \
-H "Authorization: token ${GITEA_TOKEN}" \
"${GITEA_API_URL}/api/v1/repos/${owner}/${repo}/branches/${branch}" 2>/dev/null || echo "000")
if [ "$status" = "200" ]; then
BRANCH_CACHE[$key]=1
return 0
fi
if [ "$status" = "404" ]; then
return 1
fi
# API error - retry if not last attempt
if [ "$attempt" -lt 3 ]; then
sleep 10
continue
fi
done
# All retries failed - keep report (fail-safe)
echo " WARN: Gitea API error for ${owner}/${repo}/${branch} (status ${status}) after 3 attempts - KEEPING report"
BRANCH_CACHE[$key]=1
return 0
}
declare -A REPO_BRANCHES_CACHE
declare -A REPO_STATUS
default_max_age=$(jq -r '.branches.default.maxAgeDays // 90' "$CONFIG")
default_keep_min=$(jq -r '.branches.default.keepMin // 5' "$CONFIG")
rule_max_age() {
local branch="$1" v
v=$(jq -r --arg b "$branch" '.branches[$b].maxAgeDays // empty' "$CONFIG")
[ -n "$v" ] && echo "$v" || echo "$default_max_age"
}
rule_keep_min() {
local branch="$1" v
v=$(jq -r --arg b "$branch" '.branches[$b].keepMin // empty' "$CONFIG")
[ -n "$v" ] && echo "$v" || echo "$default_keep_min"
}
age_days() {
local published="$1" epoch_pub now
epoch_pub=$(date -u -d "$published" +%s 2>/dev/null || echo 0)
[ "$epoch_pub" -eq 0 ] && echo 99999 && return
now=$(date -u +%s)
echo $(( (now - epoch_pub) / 86400 ))
}
parse_path() {
local rel="$1"
OWNER="${rel%%/*}"
rest="${rel#*/}"
REPO="${rest%%/*}"
}
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/retention-lib.sh"
echo "Fetching manifest from ${PAGES_URL}/.git-pages/manifest.json"
MANIFEST=$(curl_with_host "${PAGES_URL}/.git-pages/manifest.json")
@@ -97,6 +33,7 @@ fi
echo ""
echo "=== Phase 1: collect reports ==="
declare -A SEEN_REPORTS
declare -A SEEN_ECHO_COMMITS
declare -a REPORTS
while IFS= read -r meta_path; do
report_dir=$(dirname "$meta_path")
@@ -117,58 +54,105 @@ while IFS= read -r meta_path; do
days=$(age_days "$published")
REPORTS+=("${report_dir}|${OWNER}|${REPO}|${branch}|${days}")
echo " ${OWNER}/${REPO} branch=${branch} age=${days}d"
commit_dir=$(dirname "$report_dir")
if [ -z "${SEEN_ECHO_COMMITS[$commit_dir]:-}" ]; then
SEEN_ECHO_COMMITS[$commit_dir]=1
echo " ${commit_dir} branch=${branch} age=${days}d"
fi
done <<< "$META_PATHS"
[ "${#REPORTS[@]}" -eq 0 ] && { echo "No actionable reports"; exit 0; }
echo ""
echo "=== Phase 2: check branches in Gitea ==="
echo "=== Phase 2: check branches/repos in Gitea ==="
if [ -z "$GITEA_API_URL" ] || [ -z "$GITEA_TOKEN" ]; then
echo "ERROR: GITEA_API_URL and GITEA_TOKEN must be set" >&2
exit 1
fi
declare -a TO_DELETE
declare -a KEEP
declare -A SEEN_ECHO_BRANCHES
declare -A SEEN_ECHO_REPO_DELETED
declare -A UNIQUE_BRANCHES
declare -A REASON_MAP
declare -A COMMIT_BRANCH_MAP
# Build commit→branch mapping
for entry in "${REPORTS[@]}"; do
IFS='|' read -r dir _ _ branch _ <<< "$entry"
commit_dir=$(dirname "$dir")
[ -n "${COMMIT_BRANCH_MAP[$commit_dir]:-}" ] || COMMIT_BRANCH_MAP["$commit_dir"]=$branch
done
for entry in "${REPORTS[@]}"; do
IFS='|' read -r _ owner repo branch _ <<< "$entry"
UNIQUE_BRANCHES["${owner}/${repo}/${branch}"]=1
done
TOTAL_BRANCHES=${#UNIQUE_BRANCHES[@]}
BRANCHES_EXISTING=0
BRANCH_DELETED_COUNT=0
REPO_DELETED_COUNT=0
MAXAGE_DELETED=0
KEEPMIN_DELETED=0
for entry in "${REPORTS[@]}"; do
IFS='|' read -r dir owner repo branch days <<< "$entry"
if [ -n "$GITEA_API_URL" ] && [ -n "$GITEA_TOKEN" ]; then
if branch_exists "$owner" "$repo" "$branch"; then
echo " BRANCH EXISTS: ${owner}/${repo}/${branch}"
KEEP+=("${dir}|${owner}|${repo}|${branch}|${days}")
else
echo " BRANCH DELETED: ${owner}/${repo}/${branch} -> DELETE"
TO_DELETE+=("$dir")
branch_key="${owner}/${repo}/${branch}"
if branch_exists "$owner" "$repo" "$branch"; then
if [ -z "${SEEN_ECHO_BRANCHES[$branch_key]:-}" ]; then
SEEN_ECHO_BRANCHES[$branch_key]=1
BRANCHES_EXISTING=$((BRANCHES_EXISTING + 1))
echo " BRANCH EXISTS: ${branch_key}"
fi
else
KEEP+=("${dir}|${owner}|${repo}|${branch}|${days}")
else
if [ -z "${SEEN_ECHO_BRANCHES[$branch_key]:-}" ]; then
SEEN_ECHO_BRANCHES[$branch_key]=1
repo_key="${owner}/${repo}"
if [ "${REPO_STATUS[$repo_key]:-}" = "deleted" ]; then
REPO_DELETED_COUNT=$((REPO_DELETED_COUNT + 1))
if [ -z "${SEEN_ECHO_REPO_DELETED[$repo_key]:-}" ]; then
SEEN_ECHO_REPO_DELETED[$repo_key]=1
echo " REPO DELETED: ${repo_key} -> DELETE ALL"
fi
reason="repo deleted"
else
BRANCH_DELETED_COUNT=$((BRANCH_DELETED_COUNT + 1))
echo " BRANCH DELETED: ${branch_key} -> DELETE"
reason="branch deleted"
fi
fi
REASON_MAP["$dir"]="$reason"
TO_DELETE+=("$dir")
fi
done
echo ""
echo "=== Phase 3: apply retention rules to remaining reports ==="
declare -A BRANCH_COUNTS
if [ "${#KEEP[@]}" -gt 0 ]; then
IFS=$'\n'
for entry in $(printf '%s\n' "${KEEP[@]}" | sort -t'|' -k4,4 -k5,5rn); do
IFS='|' read -r dir owner repo branch days <<< "$entry"
max_age=$(rule_max_age "$branch")
keep_min=$(rule_keep_min "$branch")
PHASE2_DELETED=${#TO_DELETE[@]}
apply_retention "$CONFIG"
PHASE3_DELETED=$(( ${#TO_DELETE[@]} - PHASE2_DELETED ))
if [ "$days" -gt "$max_age" ]; then
echo " DELETE: ${dir} (age ${days}d > maxAge ${max_age}d, branch ${branch})"
TO_DELETE+=("$dir")
continue
fi
key="${branch}"
count="${BRANCH_COUNTS[$key]:-0}"
count=$((count + 1))
BRANCH_COUNTS["$key"]=$count
if [ "$count" -gt "$keep_min" ]; then
echo " DELETE: ${dir} (kept ${keep_min}/${count}, exceeds keepMin, branch ${branch})"
TO_DELETE+=("$dir")
fi
fmt_num() {
local n="$1" out=""
[ -z "$n" ] && { echo "?"; return; }
n="${n##0}" # strip leading zeros
while [ "${#n}" -gt 3 ]; do
out=" ${n: -3}$out"
n="${n:0:${#n}-3}"
done
unset IFS
fi
echo "${n}${out}"
}
echo ""
echo "=== Summary ==="
echo " Branches:"
echo " existing: $(fmt_num $BRANCHES_EXISTING)"
echo " deleted: $(fmt_num $BRANCH_DELETED_COUNT)"
echo " repo gone: $(fmt_num $REPO_DELETED_COUNT)"
echo " Commits:"
echo " deleted by maxAge: $(fmt_num $MAXAGE_DELETED)"
echo " deleted by keepMin:$(fmt_num $KEEPMIN_DELETED)"
if [ "${#TO_DELETE[@]}" -eq 0 ]; then
echo "Nothing to delete"
@@ -193,14 +177,37 @@ echo "Downloading archive.tar..."
HTTP_CODE=$(curl_with_host -o "$ARCHIVE_FILE" -w "%{http_code}" -sS "${PAGES_URL}/.git-pages/archive.tar")
if [ "$HTTP_CODE" = "200" ] && tar -tf "$ARCHIVE_FILE" >/dev/null 2>&1; then
echo "Extracting archive..."
OLD_KB=$(du -sk "$ARCHIVE_FILE" 2>/dev/null | awk '{print $1}')
echo "Extracting archive (${OLD_KB}kB)..."
tar -xf "$ARCHIVE_FILE" -C "$SITE_DIR"
for dir in "${TO_DELETE[@]}"; do
if [ -d "$SITE_DIR/$dir" ]; then
echo " Removing: $dir"
rm -rf "$SITE_DIR/$dir"
declare -A GROUP_SEEN
declare -A GROUP_LINES
for del in "${TO_DELETE[@]}"; do
if [ ! -d "$SITE_DIR/$del" ]; then
continue
fi
commit_dir=$(dirname "$del")
branch="${COMMIT_BRANCH_MAP[$commit_dir]:-?}"
reason="${REASON_MAP[$del]:-?}"
repo_path="${del%%/reports/*}"
commit_hash="${commit_dir##*/}"
key="${repo_path}/${branch} | Reason: ${reason}"
seen_key="${key}|${commit_hash}"
if [ -z "${GROUP_SEEN[$seen_key]:-}" ]; then
GROUP_SEEN[$seen_key]=1
GROUP_LINES["$key"]="${GROUP_LINES[$key]:-} $commit_hash"
fi
rm -rf "$SITE_DIR/$del"
done
for key in "${!GROUP_LINES[@]}"; do
echo " Removing: ${key}"
for hash in ${GROUP_LINES[$key]}; do
echo " commit: ${hash}"
done
done
else
echo "archive.tar failed (HTTP ${HTTP_CODE}) - falling back to manifest-based rebuild"
@@ -248,6 +255,7 @@ if [ -z "$(ls -A "$SITE_DIR" 2>/dev/null)" ]; then
fi
tar -cf "$NEW_TAR" -C "$SITE_DIR" .
NEW_KB=$(du -sk "$NEW_TAR" 2>/dev/null | awk '{print $1}')
echo "PUT: replacing site contents..."
HTTP_CODE=$(curl_with_host -X PUT "${PAGES_URL}/" \
@@ -259,6 +267,9 @@ HTTP_CODE=$(curl_with_host -X PUT "${PAGES_URL}/" \
echo "HTTP ${HTTP_CODE}"
if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "201" ] || [ "$HTTP_CODE" = "204" ]; then
echo "Site rebuild completed."
if [ -n "${OLD_KB:-}" ]; then
echo " archive size: $(fmt_num $OLD_KB)kB → $(fmt_num $NEW_KB)kB"
fi
else
echo "ERROR: PUT HTTP ${HTTP_CODE}" >&2
exit 1
+184
View File
@@ -0,0 +1,184 @@
#!/usr/bin/env bash
# Shared functions for retention-cleanup.sh
# Can be sourced by tests for unit testing
age_days() {
local published="$1" epoch_pub now
epoch_pub=$(date -d "$published" +%s 2>/dev/null || date -j -f "%Y-%m-%dT%H:%M:%SZ" "$published" +%s 2>/dev/null || echo 0)
[ "$epoch_pub" -eq 0 ] && echo 99999 && return
now=$(date -u +%s)
echo $(( (now - epoch_pub) / 86400 ))
}
parse_path() {
local rel="$1"
OWNER="${rel%%/*}"
rest="${rel#*/}"
REPO="${rest%%/*}"
}
read_rule() {
local config="$1" branch="$2" key="$3" default="$4"
v=$(jq -r --arg b "$branch" --arg k "$key" '.branches[$b][$k] // empty' "$config")
[ -n "$v" ] && echo "$v" || echo "$default"
}
# ---------------------------------------------------------------------------
# Gitea branch/repo checking via git ls-remote
# Uses global: GITEA_API_URL, GITEA_TOKEN
# Sets global: REPO_BRANCHES_CACHE, REPO_STATUS
# ---------------------------------------------------------------------------
# Fetch all branches for a repo (one git ls-remote call per repo).
# Sets REPO_STATUS[owner/repo].
# Echos branch list on success.
# Returns: 0=ok, 1=deleted, 2=cert_error, 3=error (fail-safe keep)
repo_branches() {
local owner="$1" repo="$2" key="${owner}/${repo}"
local attempt output
[ -z "$GITEA_API_URL" ] && return 0
[ -z "$GITEA_TOKEN" ] && return 0
# Check cached status first (avoids re-running git on every report)
case "${REPO_STATUS[$key]:-}" in
deleted) return 1 ;;
cert_error) return 2 ;;
error) echo "${REPO_BRANCHES_CACHE[$key]:-}"; return 3 ;;
esac
# Cache hit (success with branch list)
[ -n "${REPO_BRANCHES_CACHE[$key]:-}" ] && { echo "${REPO_BRANCHES_CACHE[$key]}"; return 0; }
local git_host
git_host=$(echo "$GITEA_API_URL" | sed -E 's|^https?://||' | sed 's|/.*$||')
local git_url="https://token:${GITEA_TOKEN}@${git_host}/${owner}/${repo}.git"
for attempt in 1 2 3; do
output=$(git ls-remote --heads "$git_url" 2>&1) && {
local branches
branches=$(echo "$output" | sed -n 's|.*refs/heads/||p')
REPO_BRANCHES_CACHE[$key]="$branches"
REPO_STATUS[$key]="ok"
echo "$branches"
return 0
}
# Repo deleted → no retry
if echo "$output" | grep -qiE "fatal:.*(not found|repository.*not|could not read)"; then
REPO_BRANCHES_CACHE[$key]="__REPO_DELETED__"
REPO_STATUS[$key]="deleted"
echo " REPO DELETED: ${owner}/${repo}" >&2
return 1
fi
[ "$attempt" -lt 3 ] && sleep 10
done
# Certificate verification failure → configuration error, stop
if echo "$output" | grep -qi "server certificate verification failed"; then
REPO_STATUS[$key]="cert_error"
echo "[ERROR] git-pages.retention: certificate verification failed for ${owner}/${repo}" >&2
echo "[ERROR] git-pages.retention: check CA certificates or set GIT_SSL_NO_VERIFY=1" >&2
echo "[ERROR] git-pages.retention: git output:" >&2
echo "$output" >&2
return 2
fi
# Other network errors → fail-safe keep, continue
REPO_BRANCHES_CACHE[$key]="__REPO_ERROR__"
REPO_STATUS[$key]="error"
echo "[WARN] git-pages.retention: cannot reach Gitea for ${owner}/${repo} — keeping all reports" >&2
echo "[WARN] git-pages.retention: git output:" >&2
echo "$output" >&2
return 3
}
# Check if a specific branch exists in a repo.
# Returns 0 (exists), 1 (not found/deleted).
# Returns 2 (cert error), 3 (network error).
branch_exists() {
local owner="$1" repo="$2" branch="$3"
local branches rc
[ -z "$GITEA_API_URL" ] && return 0
[ -z "$GITEA_TOKEN" ] && return 0
branches=$(repo_branches "$owner" "$repo")
rc=$?
# Return codes from repo_branches propagate through $() subshell:
# 0=ok, 1=deleted, 2=cert_error, 3=error
case $rc in
2) echo "[FATAL] git-pages.retention: cannot reach Gitea (${owner}/${repo}) — check configuration" >&2
exit 1 ;;
3) return 0 ;; # network error → fail-safe keep
1) return 1 ;; # repo/branch gone
esac
echo "$branches" | grep -qxF "$branch"
}
# Phase 3: apply retention rules to KEEP array, populate TO_DELETE
# Reads from global KEEP array
# Populates global TO_DELETE array
# Usage: apply_retention <config_path>
apply_retention() {
local config="$1"
local default_max_age default_keep_min
local max_age keep_min key count seen_key commit_dir
local entry dir owner repo branch days
default_max_age=$(jq -r '.branches.default.maxAgeDays // 90' "$config")
default_keep_min=$(jq -r '.branches.default.keepMin // 5' "$config")
declare -A BRANCH_COUNTS
declare -A SEEN_COMMITS
declare -A DELETED_COMMITS
if [ "${#KEEP[@]}" -eq 0 ]; then
return
fi
IFS=$'\n'
for entry in $(printf '%s\n' "${KEEP[@]}" | sort -t'|' -k4,4 -k5,5n); do
IFS='|' read -r dir owner repo branch days <<< "$entry"
max_age=$(read_rule "$config" "$branch" "maxAgeDays" "$default_max_age")
keep_min=$(read_rule "$config" "$branch" "keepMin" "$default_keep_min")
# Age check — per-report-type deletion
if [ "$days" -gt "$max_age" ]; then
echo " DELETE: ${dir} (age ${days}d > maxAge ${max_age}d, branch ${branch})"
TO_DELETE+=("$dir")
REASON_MAP["$dir"]="maxAgeDays exceed"
MAXAGE_DELETED=$((MAXAGE_DELETED + 1))
continue
fi
# keepMin — per-commit counting
commit_dir=$(dirname "$dir")
key="$branch"
seen_key="${key}|${commit_dir}"
if [ -z "${SEEN_COMMITS[$seen_key]:-}" ]; then
SEEN_COMMITS["$seen_key"]=1
count="${BRANCH_COUNTS[$key]:-0}"
count=$((count + 1))
BRANCH_COUNTS["$key"]=$count
else
count="${BRANCH_COUNTS[$key]:-0}"
fi
if [ "$count" -gt "$keep_min" ]; then
if [ -z "${DELETED_COMMITS[$commit_dir]:-}" ]; then
DELETED_COMMITS[$commit_dir]=1
echo " DELETE: ${commit_dir} (kept ${keep_min}/${count} commits, exceeds keepMin, branch ${branch})"
TO_DELETE+=("$commit_dir")
REASON_MAP["$commit_dir"]="keepMin exceed"
KEEPMIN_DELETED=$((KEEPMIN_DELETED + 1))
fi
fi
done
unset IFS
}
+8 -1
View File
@@ -70,8 +70,15 @@ spec:
set -euo pipefail
echo "Retention sidecar: installing deps..."
apt-get update -qq
apt-get install -y --no-install-recommends curl jq python3 >/dev/null
apt-get install -y --no-install-recommends curl jq git ca-certificates >/dev/null
echo "Retention sidecar: ready"
# Sleep until 01:00 so retention runs at night
now_epoch=$(date +%s)
target_epoch=$(date -d "today 01:00:00" +%s)
[ "$target_epoch" -le "$now_epoch" ] && target_epoch=$((target_epoch + 86400))
sleep_sec=$((target_epoch - now_epoch))
echo "Retention sidecar: next run in $((sleep_sec / 3600))h (at 01:00)"
sleep $sleep_sec
while true; do
/scripts/retention-cleanup.sh
echo "Retention sidecar: next run in 24h"
@@ -8,6 +8,8 @@ metadata:
data:
retention.json: |
{{- .Values.retention.rules | toJson | nindent 4 }}
retention-lib.sh: |
{{- .Files.Get "files/retention-lib.sh" | nindent 4 }}
retention-cleanup.sh: |
{{- .Files.Get "files/retention-cleanup.sh" | nindent 4 }}
retention-run.sh: |
+1 -1
View File
@@ -33,7 +33,7 @@ spec:
- |
set -euo pipefail
apt-get update -qq
apt-get install -y --no-install-recommends curl jq >/dev/null
apt-get install -y --no-install-recommends curl jq git >/dev/null
chmod +x /scripts/retention-run.sh /scripts/retention-cleanup.sh
/scripts/retention-run.sh
env:
+623
View File
@@ -0,0 +1,623 @@
#!/usr/bin/env bats
setup() {
source "$(dirname "$BATS_TEST_DIRNAME")/files/retention-lib.sh"
declare -gA REPO_BRANCHES_CACHE
declare -gA REPO_STATUS
declare -gA REASON_MAP
MAXAGE_DELETED=0
KEEPMIN_DELETED=0
CONFIG=$(mktemp)
}
teardown() {
rm -f "$CONFIG"
}
write_config() {
cat > "$CONFIG"
}
# ---------------------------------------------------------------------------
# read_rule
# ---------------------------------------------------------------------------
@test "read_rule returns default when branch has no override" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":5}}}
EOF
result=$(read_rule "$CONFIG" "nonexistent" "maxAgeDays" 90)
[ "$result" = "90" ]
}
@test "read_rule returns branch-specific value" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":5},"main":{"maxAgeDays":365,"keepMin":20}}}
EOF
result=$(read_rule "$CONFIG" "main" "keepMin" 5)
[ "$result" = "20" ]
}
@test "read_rule returns default for undefined key even if branch exists" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":5},"main":{"maxAgeDays":365}}}
EOF
result=$(read_rule "$CONFIG" "main" "keepMin" 5)
[ "$result" = "5" ]
}
# ---------------------------------------------------------------------------
# parse_path
# ---------------------------------------------------------------------------
@test "parse_path extracts owner and repo" {
parse_path "my-owner/my-repo/reports/abc123/go-test-unit"
[ "$OWNER" = "my-owner" ]
[ "$REPO" = "my-repo" ]
}
@test "parse_path handles owner with hyphens" {
parse_path "niko/agent-platform/reports/abc1234/go-test-bdd"
[ "$OWNER" = "niko" ]
[ "$REPO" = "agent-platform" ]
}
# ---------------------------------------------------------------------------
# apply_retention — keepMin per commit
# ---------------------------------------------------------------------------
@test "keepMin: 6 commits × 4 types, keepMin=10 → all kept (6 commits < 10)" {
# With old per-file counting, 24 files > 10 keepMin would delete 14.
# With per-commit counting, 6 commits < 10 keepMin keeps everything.
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":365,"keepMin":10}}}
EOF
KEEP=()
local -a commits=(c1 c2 c3 c4 c5 c6)
local -a ages=(100 80 60 40 20 5)
local -a types=(go-test-bdd go-test-unit helm-lint helm-kubeconform)
for i in "${!commits[@]}"; do
for t in "${types[@]}"; do
KEEP+=("niko/agent-platform/reports/${commits[$i]}/$t|niko|agent-platform|main|${ages[$i]}")
done
done
TO_DELETE=()
apply_retention "$CONFIG"
[ "${#TO_DELETE[@]}" -eq 0 ]
}
@test "keepMin: 8 commits × 1 type, keepMin=5 → deletes 3 oldest" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":365,"keepMin":5}}}
EOF
KEEP=()
local -a ages=(80 70 60 50 40 30 20 10)
for i in "${!ages[@]}"; do
KEEP+=("niko/r/reports/c$((i+1))/test|niko|r|main|${ages[$i]}")
done
TO_DELETE=()
apply_retention "$CONFIG"
# 5 newest (c8-c4) kept, 3 oldest (c3,c2,c1) deleted
[ "${#TO_DELETE[@]}" -eq 3 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/c3" ]]
[[ "${TO_DELETE[1]}" == "niko/r/reports/c2" ]]
[[ "${TO_DELETE[2]}" == "niko/r/reports/c1" ]]
}
@test "keepMin: 12 commits × 1 type, keepMin=5 → keeps 5 newest, deletes 7 oldest" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":5}}}
EOF
KEEP=()
for i in $(seq 1 12); do
KEEP+=("niko/r/reports/c${i}/test|niko|r|feature/foo|$(( 13 - i ))")
done
TO_DELETE=()
apply_retention "$CONFIG"
# 7 oldest commits deleted (12 - 5 = 7)
[ "${#TO_DELETE[@]}" -eq 7 ]
# Oldest 7 should be c1..c7 (highest days = oldest = processed last after sort)
# Sort is ascending by days, so processed as c12(1d), c11(2d), ..., c1(12d)
# keepMin=5: c12-c8 kept, c7-c1 deleted
[[ "${TO_DELETE[0]}" == "niko/r/reports/c7" ]]
[[ "${TO_DELETE[6]}" == "niko/r/reports/c1" ]]
}
@test "keepMin: 2 commits × 3 types, keepMin=5 → all kept (2 < 5)" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":5}}}
EOF
KEEP=()
KEEP+=("niko/r/reports/c1/test-a|niko|r|main|30")
KEEP+=("niko/r/reports/c1/test-b|niko|r|main|30")
KEEP+=("niko/r/reports/c1/test-c|niko|r|main|30")
KEEP+=("niko/r/reports/c2/test-a|niko|r|main|10")
KEEP+=("niko/r/reports/c2/test-b|niko|r|main|10")
KEEP+=("niko/r/reports/c2/test-c|niko|r|main|10")
TO_DELETE=()
apply_retention "$CONFIG"
[ "${#TO_DELETE[@]}" -eq 0 ]
}
@test "keepMin: 6 commits × 2 types, keepMin=3 → keeps 3 newest, deletes 3 oldest" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":365,"keepMin":3}}}
EOF
KEEP=()
local -a commits=(c1 c2 c3 c4 c5 c6)
local -a ages=(60 50 40 30 20 10)
local -a types=(jest pytest)
for i in "${!commits[@]}"; do
for t in "${types[@]}"; do
KEEP+=("niko/r/reports/${commits[$i]}/$t|niko|r|feature/x|${ages[$i]}")
done
done
TO_DELETE=()
apply_retention "$CONFIG"
# Sort by days ascending: c6(10d), c5(20d), c4(30d), c3(40d), c2(50d), c1(60d)
# keepMin=3: c6,c5,c4 kept; c3,c2,c1 deleted
[ "${#TO_DELETE[@]}" -eq 3 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/c3" ]]
[[ "${TO_DELETE[1]}" == "niko/r/reports/c2" ]]
[[ "${TO_DELETE[2]}" == "niko/r/reports/c1" ]]
}
# ---------------------------------------------------------------------------
# apply_retention — maxAge
# ---------------------------------------------------------------------------
@test "maxAge: report exceeding maxAge is deleted" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":5}}}
EOF
KEEP=(
"niko/r/reports/c1/test|niko|r|main|100"
"niko/r/reports/c2/test|niko|r|main|50"
)
TO_DELETE=()
apply_retention "$CONFIG"
# c1 (100d) > 90, deleted; c2 (50d) < 90, kept
[ "${#TO_DELETE[@]}" -eq 1 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/c1/test" ]]
}
@test "maxAge deletes report-level dir, not commit-level" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":30,"keepMin":5}}}
EOF
KEEP=(
"niko/r/reports/c1/test-a|niko|r|main|100"
"niko/r/reports/c1/test-b|niko|r|main|20"
"niko/r/reports/c2/test-a|niko|r|main|10"
)
TO_DELETE=()
apply_retention "$CONFIG"
# Only test-a for c1 is old; test-b for c1 is young, c2 is young
[ "${#TO_DELETE[@]}" -eq 1 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/c1/test-a" ]]
}
# ---------------------------------------------------------------------------
# apply_retention — maxAge + keepMin interaction
# ---------------------------------------------------------------------------
@test "maxAge takes precedence over keepMin — aged report deleted, not counted in keepMin" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":30,"keepMin":2}}}
EOF
# 3 commits, 1 type each. c1 is old (100d), c2 and c3 are young.
# With keepMin=2: c1 should be deleted by maxAge, c2 and c3 kept.
# Without the continue after maxAge check, c1 would consume a keepMin slot.
KEEP=(
"niko/r/reports/c1/test|niko|r|main|100"
"niko/r/reports/c2/test|niko|r|main|10"
"niko/r/reports/c3/test|niko|r|main|5"
)
TO_DELETE=()
apply_retention "$CONFIG"
# c1 deleted by maxAge, c2 and c3 within keepMin=2
[ "${#TO_DELETE[@]}" -eq 1 ]
}
# ---------------------------------------------------------------------------
# apply_retention — sorting (newest first)
# ---------------------------------------------------------------------------
@test "sort order: newest commits processed first within same branch" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":365,"keepMin":2}}}
EOF
KEEP=(
"niko/r/reports/c1/test|niko|r|main|100"
"niko/r/reports/c2/test|niko|r|main|50"
"niko/r/reports/c3/test|niko|r|main|10"
)
TO_DELETE=()
apply_retention "$CONFIG"
# Sort by days ascending: c3(10d) 1st, c2(50d) 2nd, c1(100d) 3rd
# keepMin=2: c3 and c2 kept, c1 deleted
[ "${#TO_DELETE[@]}" -eq 1 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/c1" ]]
}
@test "sort order: branches sorted alphabetically" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":365,"keepMin":1}}}
EOF
KEEP=(
"niko/r/reports/c1/test|niko|r|z-branch|50"
"niko/r/reports/c2/test|niko|r|a-branch|60"
"niko/r/reports/c3/test|niko|r|m-branch|10"
)
TO_DELETE=()
apply_retention "$CONFIG"
# Alphabetical: a-branch, m-branch, z-branch
# Each has 1 commit, keepMin=1 → nothing deleted
[ "${#TO_DELETE[@]}" -eq 0 ]
}
@test "multi-branch: each branch has own keepMin counter" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":2}}}
EOF
KEEP=(
"niko/r/reports/c1/test|niko|r|branch-a|30"
"niko/r/reports/c2/test|niko|r|branch-a|20"
"niko/r/reports/c3/test|niko|r|branch-a|10"
"niko/r/reports/c4/test|niko|r|branch-b|60"
"niko/r/reports/c5/test|niko|r|branch-b|50"
"niko/r/reports/c6/test|niko|r|branch-b|40"
"niko/r/reports/c7/test|niko|r|branch-b|30"
)
TO_DELETE=()
apply_retention "$CONFIG"
# branch-a: 3 reports → keep 2 newest (c2,c3), delete 1 oldest (c1)
# branch-b: 4 reports → keep 2 newest (c6,c7), delete 2 oldest (c4,c5)
# Actually: Sort is by branch, then by days ascending
# branch-a processed first: c3(10d) 1st, c2(20d) 2nd (keep), c1(30d) 3rd (delete)
# branch-b processed next: c7(30d) 1st, c6(40d) 2nd (keep), c5(50d) 3rd (delete), c4(60d) 4th (delete)
[ "${#TO_DELETE[@]}" -eq 3 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/c1" ]]
[[ "${TO_DELETE[1]}" == "niko/r/reports/c5" ]]
[[ "${TO_DELETE[2]}" == "niko/r/reports/c4" ]]
}
# ---------------------------------------------------------------------------
# apply_retention — empty / edge cases
# ---------------------------------------------------------------------------
@test "empty KEEP array → nothing deleted" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":90,"keepMin":5}}}
EOF
KEEP=()
TO_DELETE=()
apply_retention "$CONFIG"
[ "${#TO_DELETE[@]}" -eq 0 ]
}
@test "TO_DELETE preserves Phase 2 entries after apply_retention (no new deletions)" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":365,"keepMin":10}}}
EOF
KEEP=(
"niko/r/reports/c1/test|niko|r|main|10"
"niko/r/reports/c2/test|niko|r|main|5"
)
TO_DELETE=(
"niko/r/reports/abc/branch-gone"
"niko/r/reports/def/repo-gone"
)
apply_retention "$CONFIG"
[ "${#TO_DELETE[@]}" -eq 2 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/abc/branch-gone" ]]
[[ "${TO_DELETE[1]}" == "niko/r/reports/def/repo-gone" ]]
}
@test "TO_DELETE preserves Phase 2 entries AND adds retention deletions" {
write_config <<'EOF'
{"branches":{"default":{"maxAgeDays":365,"keepMin":3}}}
EOF
KEEP=(
"niko/r/reports/c1/test|niko|r|main|40"
"niko/r/reports/c2/test|niko|r|main|30"
"niko/r/reports/c3/test|niko|r|main|20"
"niko/r/reports/c4/test|niko|r|main|10"
)
TO_DELETE=(
"niko/r/reports/abc/branch-gone"
)
apply_retention "$CONFIG"
# 1 pre-existing + 1 commit deleted (c1, oldest of 4, keepMin=3)
[ "${#TO_DELETE[@]}" -eq 2 ]
[[ "${TO_DELETE[0]}" == "niko/r/reports/abc/branch-gone" ]]
}
# ---------------------------------------------------------------------------
# branch_exists — mocking REPO_STATUS / REPO_BRANCHES_CACHE
# ---------------------------------------------------------------------------
@test "branch_exists: branch in list → return 0" {
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test-token"
REPO_BRANCHES_CACHE["owner/repo"]=$'main\nfeature/x'
REPO_STATUS["owner/repo"]="ok"
run branch_exists "owner" "repo" "main"
[ "$status" -eq 0 ]
}
@test "branch_exists: branch not in list → return 1" {
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test-token"
REPO_BRANCHES_CACHE["owner/repo"]=$'main\nfeature/x'
REPO_STATUS["owner/repo"]="ok"
run branch_exists "owner" "repo" "nonexistent"
[ "$status" -eq 1 ]
}
@test "branch_exists: cert error → exit 1 with [FATAL]" {
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test"
REPO_STATUS["owner/repo"]="cert_error"
run branch_exists "owner" "repo" "any-branch"
[ "$status" -eq 1 ]
[[ "$output" == *"[FATAL]"* ]]
}
@test "branch_exists: repo deleted → return 1" {
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test-token"
REPO_BRANCHES_CACHE["owner/repo"]="__REPO_DELETED__"
REPO_STATUS["owner/repo"]="deleted"
run branch_exists "owner" "repo" "any-branch"
[ "$status" -eq 1 ]
}
@test "branch_exists: network error → return 0 (fail-safe keep)" {
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test-token"
REPO_BRANCHES_CACHE["owner/repo"]="__REPO_ERROR__"
REPO_STATUS["owner/repo"]="error"
run branch_exists "owner" "repo" "any-branch"
[ "$status" -eq 0 ]
}
@test "branch_exists: empty GITEA_API_URL → return 0 (skip)" {
GITEA_API_URL=""
GITEA_TOKEN="test-token"
run branch_exists "owner" "repo" "any-branch"
[ "$status" -eq 0 ]
}
@test "branch_exists: empty GITEA_TOKEN → return 0 (skip)" {
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN=""
run branch_exists "owner" "repo" "any-branch"
[ "$status" -eq 0 ]
}
# ---------------------------------------------------------------------------
# repo_branches — git ls-remote error detection patterns
# ---------------------------------------------------------------------------
@test "error detection: 'command not found' does NOT trigger repo deleted" {
# This must NOT match — "bash: git: command not found" is NOT a repo deletion
local msg="bash: git: command not found"
run grep -qiE "fatal:.*(not found|repository.*not|could not read)" <<< "$msg"
[ "$status" -eq 1 ]
}
@test "error detection: 'fatal: repo not found' triggers repo deleted" {
# This MUST match — genuine git error for deleted/missing repo
local msg="fatal: repository 'https://gitea.app/owner/repo.git' not found"
run grep -qiE "fatal:.*(not found|repository.*not|could not read)" <<< "$msg"
[ "$status" -eq 0 ]
}
@test "error detection: 'could not read from remote' triggers repo deleted" {
local msg="fatal: could not read from remote repository"
run grep -qiE "fatal:.*(not found|repository.*not|could not read)" <<< "$msg"
[ "$status" -eq 0 ]
}
# ---------------------------------------------------------------------------
# git ls-remote integration (real git, temp repo)
# ---------------------------------------------------------------------------
@test "git ls-remote parsing: lists branches correctly" {
local tmpdir=$(mktemp -d)
git -C "$tmpdir" init -b main source >/dev/null 2>&1
git -C "$tmpdir/source" config user.email "test@test"
git -C "$tmpdir/source" config user.name "test"
git -C "$tmpdir/source" commit --allow-empty -m "init" >/dev/null 2>&1
git -C "$tmpdir/source" branch feature/x >/dev/null 2>&1
git clone --bare "$tmpdir/source" "$tmpdir/repo.git" >/dev/null 2>&1
local url="file://$tmpdir/repo.git"
local output
output=$(git ls-remote --heads "$url" 2>&1)
local branches
branches=$(echo "$output" | sed -n 's|.*refs/heads/||p')
echo "$branches" | grep -qxF "main"
[ "$?" -eq 0 ]
echo "$branches" | grep -qxF "feature/x"
[ "$?" -eq 0 ]
! echo "$branches" | grep -qxF "nonexistent"
rm -rf "$tmpdir"
}
# ---------------------------------------------------------------------------
# repo_branches — retry + error output (using git mock)
# ---------------------------------------------------------------------------
@test "repo_branches: success returns branches immediately" {
local mockdir=$(mktemp -d)
cat > "$mockdir/git" << 'SCRIPT'
#!/usr/bin/env bash
echo "abc123 refs/heads/main"
echo "def456 refs/heads/feature/x"
SCRIPT
chmod +x "$mockdir/git"
local save_PATH="$PATH"
export PATH="$mockdir:$PATH"
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test"
REPO_BRANCHES_CACHE=()
REPO_STATUS=()
run repo_branches "owner" "repo"
[ "$status" -eq 0 ]
[[ "$output" == *"main"* ]]
[[ "$output" == *"feature/x"* ]]
export PATH="$save_PATH"
rm -rf "$mockdir"
}
@test "repo_branches: retries 3 times on transient error" {
local mockdir=$(mktemp -d)
cat > "$mockdir/git" << 'SCRIPT'
#!/usr/bin/env bash
echo "call" >> "$MOCKDIR/count"
echo "fatal: unable to access 'https://...'" >&2
exit 1
SCRIPT
chmod +x "$mockdir/git"
# Inject mockdir path into mock script via env var
sed -i '' "s|\$MOCKDIR|$mockdir|g" "$mockdir/git"
local save_PATH="$PATH"
export PATH="$mockdir:$PATH"
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test"
REPO_BRANCHES_CACHE=()
REPO_STATUS=()
local start=$SECONDS
run repo_branches "owner" "repo"
[ "$status" -eq 3 ]
[[ "$output" == *"[WARN] git-pages.retention"* ]]
[[ "$output" == *"keeping all reports"* ]]
[[ "$output" == *"git output:"* ]]
[[ "$output" == *"unable to access"* ]]
[ $(cat "$mockdir/count" | wc -l) -eq 3 ]
[ $(( SECONDS - start )) -ge 18 ]
export PATH="$save_PATH"
rm -rf "$mockdir"
}
@test "repo_branches: certificate error → [ERROR] + return 1" {
local mockdir=$(mktemp -d)
cat > "$mockdir/git" << 'SCRIPT'
#!/usr/bin/env bash
echo "call" >> "$MOCKDIR/count"
echo "fatal: unable to access 'https://gitea.app/owner/repo.git/': server certificate verification failed. CAfile: none CRLfile: none" >&2
exit 1
SCRIPT
chmod +x "$mockdir/git"
sed -i '' "s|\$MOCKDIR|$mockdir|g" "$mockdir/git"
local save_PATH="$PATH"
export PATH="$mockdir:$PATH"
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test"
REPO_BRANCHES_CACHE=()
REPO_STATUS=()
run repo_branches "owner" "repo"
[ "$status" -eq 2 ]
[[ "$output" == *"[ERROR]"* ]]
[[ "$output" == *"certificate verification"* ]]
[[ "$output" == *"git output:"* ]]
[[ "$output" == *"unable to access"* ]]
export PATH="$save_PATH"
rm -rf "$mockdir"
}
@test "repo_branches: repo not found returns immediately (no retry)" {
local mockdir=$(mktemp -d)
cat > "$mockdir/git" << 'SCRIPT'
#!/usr/bin/env bash
echo "call" >> "$MOCKDIR/count"
echo "fatal: repository 'https://gitea.app/owner/repo.git' not found" >&2
exit 1
SCRIPT
chmod +x "$mockdir/git"
sed -i '' "s|\$MOCKDIR|$mockdir|g" "$mockdir/git"
local save_PATH="$PATH"
export PATH="$mockdir:$PATH"
GITEA_API_URL="https://gitea.example.com"
GITEA_TOKEN="test"
REPO_BRANCHES_CACHE=()
REPO_STATUS=()
run repo_branches "owner" "repo"
[ "$status" -eq 1 ]
[[ "$output" == *"REPO DELETED"* ]]
[[ "$output" != *"[WARN]"* ]]
[ $(cat "$mockdir/count" | wc -l) -eq 1 ]
export PATH="$save_PATH"
rm -rf "$mockdir"
}
+4 -1
View File
@@ -22,7 +22,10 @@ INPUTS=$(jq -nc \
--arg source_repo "$GITOPS_SOURCE_REPO" \
--arg source_commit "$GITOPS_SOURCE_COMMIT" \
--arg git_tag_prefix "${GITOPS_TAG_PREFIX:-}" \
'{file: $file, yq_tpl: $yq_tpl, version: $version, source_repo: $source_repo, source_commit: $source_commit, git_tag_prefix: $git_tag_prefix}')
--arg extra_cmd "${GITOPS_EXTRA_CMD:-}" \
--arg author_name "${GIT_USER_NAME:-}" \
--arg author_email "${GIT_USER_EMAIL:-}" \
'{file: $file, yq_tpl: $yq_tpl, version: $version, source_repo: $source_repo, source_commit: $source_commit, git_tag_prefix: $git_tag_prefix, extra_cmd: $extra_cmd, author_name: $author_name, author_email: $author_email}')
DIR="$(cd "$(dirname "$0")" && pwd)"
echo "gitops-dispatch: dispatching to $GITOPS_REPO/$GITOPS_WORKFLOW..."
+8 -3
View File
@@ -84,7 +84,12 @@ _gitops_update() {
cd "${CLONE_DIR}" || _gitops_fail "Failed to enter clone directory"
yq eval -i "${YQ_EXPR}" "${INPUT_FILE}" || _gitops_fail "Failed to update ${INPUT_FILE}"
git add "${INPUT_FILE}" || _gitops_fail "Failed to stage ${INPUT_FILE}"
if [ -n "${GITOPS_EXTRA_CMD:-}" ]; then
eval "${GITOPS_EXTRA_CMD}" || _gitops_fail "Extra command failed: ${GITOPS_EXTRA_CMD}"
git add -A || _gitops_fail "Failed to stage all changes"
else
git add "${INPUT_FILE}" || _gitops_fail "Failed to stage ${INPUT_FILE}"
fi
if git diff --cached --quiet; then
echo "No changes — ${INPUT_FILE} already at ${VERSION}"
@@ -93,8 +98,8 @@ _gitops_update() {
exit 0
fi
git -c user.name="gitea-ci-bot" \
-c user.email="ci@keskikuja.site" \
git -c user.name="${GIT_USER_NAME:-gitea-ci-bot}" \
-c user.email="${GIT_USER_EMAIL:-ci@keskikuja.site}" \
commit -m "[skip ci] gitops: update version to ${VERSION}" || _gitops_fail "Failed to commit"
GITOPS_SHA="$(git rev-parse HEAD)"
git push || _gitops_fail "Failed to push"
+21 -4
View File
@@ -368,9 +368,18 @@ Pakkaa ja pushee Helm-chartin OCI-registryyn. Käyttää `alpine/helm`-konttia.
```yaml
HELM_REGISTRY: gitea.app.keskikuja.site/niko
VERSION_FILE: platform-helm/Chart.yaml # chart-hakemisto + versionlähde
VERSION_FILE: platform-helm/Chart.yaml # versionlähde, chart_path määrää chart-hakemiston
```
**Inputit:**
| Parametri | Pakollinen | Kuvaus |
|-----------|------------|--------|
| `env_json` | Kyllä | Konffi `gitea-env.conf`:stä |
| `version` | Kyllä | Version string (check-version output) |
| `chart_path` | Kyllä | Polku Chart.yaml-hakemistoon (esim. `platform-helm`) |
| `extra_dependency_paths` | Ei | Pilkulla erotellut polut subcharttien dependeinceille |
**Käyttö reitittimessä:**
```yaml
@@ -382,11 +391,19 @@ helm-build-push:
with:
env_json: ${{ needs.load-config.outputs.env_json }}
version: ${{ needs.check-version.outputs.version }}
chart_path: platform-helm
# extra_dependency_paths: subchart-a,subchart-b # tarvittaessa
```
Chart-hakemisto johdetaan `VERSION_FILE`-polusta: `dirname "${VERSION_FILE}"`.
Jos `VERSION_FILE` on `Chart.yaml`, konteksti on juuri. Jos `platform-helm/Chart.yaml`,
konteksti on `platform-helm/`.
`chart_path` on eksplisiittinen polku chart-hakemistoon (esim. `platform-helm`).
`VERSION_FILE` määrää version lähteen (`Chart.yaml:n` `version`-kenttä) —
nämä voivat olla eri polkuja, mutta tyypillisesti molemmat osoittavat samaan
chart-hakemistoon.
**`extra_dependency_paths`:** Jos chartilla on alikarttoja (subchartteja) jotka
vaativat `helm dependency update` -ajon ennen päächartin buildia, anna niiden
polut pilkulla eroteltuna. Provider ajaa `helm dependency update` jokaiselle
polulle ennen päächartin buildia.
**Yksittäisten Helm-UI-linkkien raportointi:** `HELM_UI_URL` on
tarkoitettu yleiselle registry UI:lle — provider muodostaa linkin
+139 -298
View File
@@ -1,10 +1,9 @@
---
name: gitops-update
description: |
Setting up GitOps version updates: GitOps-repo workflow template, code
repo dispatch, secret requirements, and two-repo commit-status pattern.
Activates when the user needs to wire up artifact builds to GitOps
configuration updates.
Getting GitOps configuration updates working for a consumer project —
GitOps repo setup, consumer pipeline wiring, secrets, and commit-status
output.
activation-gate: |
User mentions GitOps update, gitops-update, dispatch to another repo,
two-repo version bump, cross-repo deployment, or wiring build output to
@@ -13,47 +12,26 @@ category: ci
impact: high
---
# GitOps Update — Provider-palvelu
# GitOps Update — consumer setup
`scripts/gitops-update.sh` ja `scripts/dispatch-workflow.sh` muodostavat
GitOps-päivityspalvelun. Artifact buildataan code repossa, minkä jälkeen
code repo dispatchaa GitOps-repoon, joka päivittää konfiguraatiotiedoston
ja pushaa muutoksen.
## What you need
## Arkkitehtuuri
- **GitOps repo** — holds the configuration files (e.g. `Chart.yaml`, `values.yaml`)
- **Consumer repo** — builds artifacts and triggers the update
- **Bottitoken** — Gitea token with write access to the GitOps repo only
Kaksi erillistä repoa, eristetyt oikeudet:
Two repos, isolated access. The consumer never writes to GitOps directly;
it dispatches a workflow that clones, updates, commits, and pushes.
```
Code repo GitOps repo
(build & push artifact) (konfiguraatiot)
---
build & push onnistuu (v0.2.3)
│ dispatch ci-main.yml
│ {file, yq_tpl, version, source_repo, source_commit}
└────────────────────────────────────→┐
dispatch-workflow.sh pollaa ←─────────┘
code repo asettaa │ git clone, yq update,
oman commit-statusnsa │ git commit + push
dispatchin exit-koodilla │ status GitOps-repoon
```
## 1. GitOps-repo setup
**Token-periaate:** Vain GitOps-repoon kirjoitetaan. Code repo asettaa
oman commit-statusnsa dispatch-kutsun exit-koodin perusteella omalla
auto-tokenillaan. GitOps-repon auto-token ei tarvitse oikeuksia code
repoon.
## GitOps-repon workflow (ci-main.yml)
GitOps-repoon luodaan `.gitea/workflows/ci-main.yml`:
Create `.gitea/workflows/gitops-service.yaml`:
```yaml
name: GitOps Update
run-name: "GitOps Service (${{ inputs.dispatch_id || 'manual' }})"
run-name: "GitOps (${{ inputs.dispatch_id || 'manual' }})"
on:
workflow_dispatch:
inputs:
@@ -78,6 +56,15 @@ on:
git_tag_prefix:
required: false
type: string
extra_cmd:
required: false
type: string
author_name:
required: false
type: string
author_email:
required: false
type: string
env:
INPUT_FILE: ${{ inputs.file }}
@@ -86,27 +73,26 @@ env:
SOURCE_REPO: ${{ inputs.source_repo }}
SOURCE_COMMIT: ${{ inputs.source_commit }}
GITOPS_REPO: ${{ github.repository }}
GITOPS_BRANCH: ${{ github.ref_name }}
GITEA_API_URL: ${{ gitea.server_url }}
GIT_TAG_PREFIX: ${{ inputs.git_tag_prefix || '' }}
GITOPS_EXTRA_CMD: ${{ inputs.extra_cmd || '' }}
GIT_USER_NAME: ${{ inputs.author_name || '' }}
GIT_USER_EMAIL: ${{ inputs.author_email || '' }}
jobs:
update:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4
with:
repository: niko/gitea-ci-library
path: .ci
- name: Install yq
run: |
wget -qO /usr/local/bin/yq \
https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
chmod +x /usr/local/bin/yq
- name: Run GitOps update
env:
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
@@ -114,297 +100,152 @@ jobs:
bash .ci/scripts/gitops-update.sh
```
**Huomiot:**
- `GITEA_TOKEN` on Gitean auto-token — scopeutuu GitOps-repoon, riittää
cloneen, committiin, pushiin ja commit-statusiin GitOps-repossa
- `run-name` ja `dispatch_id` mahdollistavat dispatchaavan skriptin tunnistaa
tämän workflow-runin yksiselitteisesti `display_title`-kentästä, vaikka
samassa repossa olisi samanaikaisia ajoja
- yq ladataan lennossa (kompromissi, ks. "Tuleva CI-kontti")
> **⚠️ yq ladataan lennossa.** Tämä on väliaikainen kompromissi. Myöhemmin
> julkaistaan Docker Hubiin custom CI-kontti, jossa nodejs + git + yq
> valmiina. Sama patterni kuin `ci-bats` ja `ci-cucumber`.
> Ks. `skills/ci-container-build/SKILL.md`.
### Tulossa: custom CI-kontti
Key points:
- `run-name` must include `dispatch_id` — the consumer's poll step uses it to find the run
- `secrets.GITEA_TOKEN` is the **auto-token** — write access to the GitOps repo only, no consumer access needed
- Commit message becomes `"[skip ci] gitops: update version to X.Y.Z"` — used by consumer to find the commit SHA
Nykyinen job lataa yq:n lennossa. Myöhemmin rakennetaan oma kontti
(`ci-gitops`), jossa on nodejs + git + yq valmiina. Sama patterni kuin
`ci-bats` ja `ci-cucumber`. Ks. `skills/ci-container-build/SKILL.md`.
---
## Code-repon dispatch-step
## 2. Consumer-repo setup
Code repo dispatchaa GitOps-repon workflown artifact buildin onnistuttua:
### 2.1 Token
Create a Gitea token with write access to the GitOps repo:
1. Gitea → `Settings``Applications``Generate Token`
2. Select the GitOps repo, grant write access
3. Save as an Actions secret in the consumer repo: **`GITOPS_DISPATCH_TOKEN`**
### 2.2 Pipeline call
Add a job after your build step that calls the dispatch workflow:
```yaml
gitops-update:
needs: [helm-build-push]
needs: [build-push]
if: success()
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4
with:
repository: niko/gitea-ci-library
path: .ci
- name: Dispatch GitOps update
id: dispatch
env:
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
run: |
INPUTS=$(jq -nc \
--arg file "dev/Chart.yaml" \
--arg yq_tpl '(.dependencies[] | select(.name == "agent-platform-helm") | .version) = "{{VERSION}}"' \
--arg version "${{ needs.check-version.outputs.version }}" \
--arg source_repo "${{ github.repository }}" \
--arg source_commit "${{ github.sha }}" \
'{file: $file, yq_tpl: $yq_tpl, version: $version, source_repo: $source_repo, source_commit: $source_commit}')
OUTPUT=$(bash .ci/scripts/dispatch-workflow.sh \
"niko/agent-platform-gitops" \
"ci-main.yml" \
"main" \
"$INPUTS" \
"${{ fromJson(needs.load-config.outputs.env_json).GITEA_API_URL }}" \
"${{ secrets.GITEA_TOKEN }}" \
"30")
echo "$OUTPUT"
GITOPS_COMMIT=$(echo "$OUTPUT" | grep '^GITOPS_COMMIT=' | cut -d= -f2)
echo "gitops_commit=$GITOPS_COMMIT" >> "$GITHUB_OUTPUT"
uses: niko/gitea-ci-library/.gitea/workflows/gitops-dispatch.yml@v1
secrets: inherit
with:
env_json: ${{ needs.load-config.outputs.env_json }}
version: ${{ needs.version.outputs.version }}
GITOPS_FILE: dev/Chart.yaml
GITOPS_YQ_TPL: '.version = "{{VERSION}}"'
GITOPS_REPO: niko/your-gitops-repo
```
### Multi-artifact pipeline (kontti + helm)
This single job handles: dispatch → poll → find commit SHA → set commit-status on your commit → produce `GITOPS_SUMMARY` output.
Yksi main-haaran build tuottaa usein sekä Docker-imagen että Helm-chartin.
Kumpikin artefakti dispatchaa oman GitOps-päivityksensä rinnakkain:
To run extra commands (e.g. `helm dependency update`) after the version bump and before the commit:
```yaml
gitops-helm:
needs: [helm-build-push]
gitops-update:
needs: [load-config, check-version, helm-build-push]
if: success()
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4
with:
repository: niko/gitea-ci-library
path: .ci
- name: Update helm version
id: helm
run: |
INPUTS=$(jq -nc \
--arg file "dev/Chart.yaml" \
--arg yq_tpl '(.dependencies[] | select(.name == "git-pages") | .version) = "{{VERSION}}"' \
--arg version "${{ needs.check-version.outputs.version }}" \
--arg source_repo "${{ github.repository }}" \
--arg source_commit "${{ github.sha }}" \
--arg git_tag_prefix "helm" \
'{dispatch_id: "", file: $file, yq_tpl: $yq_tpl, version: $version, source_repo: $source_repo, source_commit: $source_commit, git_tag_prefix: $git_tag_prefix}')
OUTPUT=$(bash .ci/scripts/dispatch-workflow.sh \
"niko/gitea-ci-gitops-tests" "gitops-service.yaml" "main" \
"$INPUTS" "${{ fromJson(needs.load-config.outputs.env_json).GITEA_API_URL }}" \
"${{ secrets.GITOPS_DISPATCH_TOKEN }}" "30")
echo "$OUTPUT"
echo "helm_commit=$(echo "$OUTPUT" | grep '^GITOPS_COMMIT=' | cut -d= -f2)" >> "$GITHUB_OUTPUT"
gitops-docker:
needs: [docker-build-push]
if: success()
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4
with:
repository: niko/gitea-ci-library
path: .ci
- name: Update docker tag
id: docker
run: |
INPUTS=$(jq -nc \
--arg file "dev/values.yaml" \
--arg yq_tpl '.service.tag = "{{VERSION}}"' \
--arg version "${{ needs.check-version.outputs.version }}" \
--arg source_repo "${{ github.repository }}" \
--arg source_commit "${{ github.sha }}" \
--arg git_tag_prefix "docker" \
'{dispatch_id: "", file: $file, yq_tpl: $yq_tpl, version: $version, source_repo: $source_repo, source_commit: $source_commit, git_tag_prefix: $git_tag_prefix}')
OUTPUT=$(bash .ci/scripts/dispatch-workflow.sh \
"niko/gitea-ci-gitops-tests" "gitops-service.yaml" "main" \
"$INPUTS" "${{ fromJson(needs.load-config.outputs.env_json).GITEA_API_URL }}" \
"${{ secrets.GITOPS_DISPATCH_TOKEN }}" "30")
echo "$OUTPUT"
echo "docker_commit=$(echo "$OUTPUT" | grep '^GITOPS_COMMIT=' | cut -d= -f2)" >> "$GITHUB_OUTPUT"
uses: niko/gitea-ci-library/.gitea/workflows/gitops-dispatch.yml@v1
secrets: inherit
with:
env_json: ${{ needs.load-config.outputs.env_json }}
version: ${{ needs.check-version.outputs.version }}
GITOPS_FILE: Chart.yaml
GITOPS_YQ_TPL: '(.dependencies[] | select(.name == "agent-platform-helm") | .version) = "{{VERSION}}"'
GITOPS_REPO: niko/agent-platform-gitops
GITOPS_EXTRA_CMD: helm dependency update
```
Kaksi dispatchia, kaksi eri tiedostoa, kaksi eri `GIT_TAG_PREFIX`-arvoa.
Kummallakin on oma commit-status-linja ja oma summary-rivi.
`dispatch-workflow.sh` hoitaa rinnakkaisuuden `display_title`-matchauksella.
When `GITOPS_EXTRA_CMD` is set, the script runs it after `yq` and stages all changes (`git add -A`) instead of only the input file — so any files generated by the extra command (e.g. `Chart.lock`, `charts/`) are included in the commit.
**GITEA_TOKEN dispatch-vaiheessa:** Tarvitaan manuaalinen token,
jolla on **write-oikeus GitOps-repoon** (esim. org-tason token).
Code-repon auto-token ei oikeuta dispatchaamaan toiseen repoon.
Token luodaan Giteassa: `Settings → Applications → Generate Token`
ja asetetaan code-repoon Actions Secretiksi.
By default the GitOps commit is made as `gitea-ci-bot`. To use the original commit author instead, the dispatch workflow resolves it automatically from the consumer repo — no extra config needed. Just ensure the GitOps repo's `gitops-service.yaml` template has the `author_name` and `author_email` inputs and env mappings.
### Commit-status dispatchin perusteella
### 2.3 Parameters
`dispatch-workflow.sh` tulostaa `GITOPS_COMMIT=<sha>` stdoutiin onnistuneen
GitOps-päivityksen jälkeen. Code repo parsii sen ja asettaa commit-statusin
linkillä GitOps-committiin:
| Input | Required | Description |
|---|---|---|---|
| `env_json` | Yes | Config JSON with `GITEA_API_URL`, optional `GIT_TAG_PREFIX` (for multi-component repos) |
| `version` | Yes | Version to write (e.g. `0.2.3`) |
| `GITOPS_FILE` | Yes | Path in GitOps repo (e.g. `dev/Chart.yaml`) |
| `GITOPS_YQ_TPL` | Yes | yq expression, `{{VERSION}}` is replaced at runtime |
| `GITOPS_REPO` | Yes | GitOps repo slug (e.g. `niko/agent-platform-gitops`) |
| `GITOPS_EXTRA_CMD` | No | Shell command to run after yq update, before git commit (e.g. `helm dependency update`) |
### 2.4 Output
The workflow produces a `summary` output in pipe format:
```
component|version|status|commit_sha|repo
agent-platform-helm|0.2.3|success|abc789def|niko/agent-platform-gitops
```
Pass it to `report-summary.yml` for the pipeline summary:
```yaml
- name: Set commit-status with GitOps link
if: always()
env:
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
GITEA_API_URL: ${{ fromJson(needs.load-config.outputs.env_json).GITEA_API_URL }}
GITOPS_COMMIT: ${{ steps.dispatch.outputs.gitops_commit }}
VERSION: ${{ needs.check-version.outputs.version }}
run: |
GITOPS_URL="${GITEA_API_URL}/niko/agent-platform-gitops/commit/${GITOPS_COMMIT}"
CTX="gitops/$(basename ${{ github.repository }})"
DESC="Deploy to dev ${VERSION}"
if [ -n "$GITOPS_COMMIT" ]; then
bash .ci/scripts/report-status.sh success "$DESC" "$CTX" "" "$GITOPS_URL"
else
bash .ci/scripts/report-status.sh success "$DESC" "$CTX"
fi
report-summary:
needs: [load-config, gitops-update]
if: always()
uses: niko/gitea-ci-library/.gitea/workflows/report-summary.yml@main
with:
env_json: ${{ needs.load-config.outputs.env_json }}
suites: bats cucumber
gitops: ${{ needs.gitops-update.outputs.summary }}
```
`dispatch-workflow.sh` palauttaa:
- exit 0 = GitOps-päivitys onnistui (+ `GITOPS_COMMIT=<sha>`)
- exit 1 = GitOps-päivitys failasi
- exit 124 = aikakatkaisu (360 min oletus)
---
### Loppuraportti (report-summary)
## 3. Token summary
Code-repon viimeinen job (`report-summary`) lisää GitOps-päivityksestä
rivin GITHUB_STEP_SUMMARYyn:
| Token | Where | Scope | Purpose |
|---|---|---|---|
| `GITOPS_DISPATCH_TOKEN` (manual) | Consumer secrets | write GitOps repo | Dispatches the GitOps workflow |
| `GITHUB_TOKEN` (auto) | Consumer workflow | write consumer repo | Sets commit-status on consumer's commit |
| `GITEA_TOKEN` (auto) | GitOps workflow | write GitOps repo | Clone, push, commit-status in GitOps repo |
```yaml
- name: GitOps summary
if: always()
env:
GITEA_API_URL: ${{ fromJson(needs.load-config.outputs.env_json).GITEA_API_URL }}
GITOPS_COMMIT: ${{ steps.dispatch.outputs.gitops_commit }}
VERSION: ${{ needs.check-version.outputs.version }}
run: |
if [ -n "$GITOPS_COMMIT" ]; then
LINK="${GITEA_API_URL}/niko/agent-platform-gitops/commit/${GITOPS_COMMIT}"
else
LINK="#"
fi
cat >> "$GITHUB_STEP_SUMMARY" << 'GITOPS'
---
## GitOps updates
## 4. What happens at runtime
| Component | Version | Status | Commit |
|-----------|---------|--------|--------|
| agent-platform-helm | __VERSION__ | __STATUS__ | [link](__LINK__) |
GITOPS
sed -i "s|__VERSION__|${VERSION}|; s|__STATUS__|${{ job.status }}|; s|__LINK__|${LINK}|" \
"$GITHUB_STEP_SUMMARY"
1. Consumer's `gitops-dispatch.yml` generates a unique `dispatch_id` and POSTs it to the GitOps repo
2. GitOps workflow clones its own repo, applies `yq`, runs `GITOPS_EXTRA_CMD` if set, then commits + pushes
3. Consumer polls the GitOps repo's runs until the workflow completes
4. Consumer lists recent commits and finds the matching one by commit message `"gitops: update version to X.Y.Z"`
5. Consumer sets commit-status `gitops/{repo}[/{prefix}]` on its own commit with a link to the exact GitOps commit
6. If no matching commit is found (no change or error), the job fails
7. On failure, `GITOPS_SUMMARY` still flows through `report-summary` with `status=failure`
---
## 5. GIT_TAG_PREFIX (optional)
If the same consumer repo dispatches updates for multiple components (e.g. Docker image + Helm chart), set `GIT_TAG_PREFIX` in your `gitea-env.conf`:
```
GIT_TAG_PREFIX=docker/
```
## Secretit ja tokenit
Each component gets its own commit-status context:
| Secret | Missä | Scope | Kuvaus |
|--------|-------|-------|--------|
| `GITEA_TOKEN` (auto) | Code repo | Vain code repo | Asettaa commit-statusin dispatchin jälkeen |
| `GITEA_TOKEN` (auto) | GitOps repo | Vain GitOps repo | Klooni, push, commit-status GitOps-repossa |
| `GITOPS_DISPATCH_TOKEN` (manuaalinen) | Code repo | Write GitOps-repoon | Dispatchaa GitOps-repon workflow |
| Prefix | Context |
|---|---|
| (empty) | `gitops/agent-platform` |
| `docker/` | `gitops/agent-platform/docker` |
| `helm/` | `gitops/agent-platform/helm` |
**Tokenin luonti:**
1. Gitea → `Settings``Applications``Generate Token`
2. Valitse repo-oikeudet: valitse GitOps-repo, anna write-oikeudet
3. Token asetetaan code-repoon: `{repo} → Settings → Actions Secrets`
4. Salaisuuden nimi: esim. `GITOPS_DISPATCH_TOKEN`
This prevents status overwrites between parallel dispatch jobs.
## Provider-skriptit
---
### `scripts/gitops-update.sh`
## 6. What you do NOT need to know
Ajaan GitOps-repon workflow'ssa. Päivittää konfiguraatiotiedoston yq:llä,
committaa ja pushaa. Asettaa commit-statuksen vain GitOps-repoon.
- How `gitops-update.sh` works internally
- How the polling finds the run
- How the commit SHA is extracted
- Race condition handling
- CI container plans
**Input-ympäristömuuttujat:**
| Muuttuja | Pakollinen | Kuvaus |
|---|---|---|
| `INPUT_FILE` | Kyllä | Tiedosto GitOps-repossa (esim. `dev/Chart.yaml`) |
| `YQ_TPL` | Kyllä | yq-lauseke `{{VERSION}}`-placeholderilla |
| `VERSION` | Kyllä | Uusi versio (esim. `0.2.3`) |
| `SOURCE_REPO` | Kyllä | Code-repo slug (esim. `org/app`) |
| `SOURCE_COMMIT` | Kyllä | Code-repon commit SHA |
| `GITOPS_REPO` | Kyllä | GitOps-repo slug |
| `GITEA_API_URL` | Kyllä | Gitean API-URL |
| `GITEA_TOKEN` | Kyllä | Gitea API-token (write GitOps-repoon) |
| `GITOPS_BRANCH` | Ei | Branch (oletus `main`) |
| `GIT_TAG_PREFIX` | Ei | Komponentin tag-prefix status-nimeämiseen (esim. `agent-platform-helm`) |
| `GITOPS_CLONE_URL` | Ei | Yliajaa clone-URL (esim. eri protokolla) |
| `GITOPS_TARGET_DIR` | Ei | Yliajaa clone-kohdehakemisto |
**Commit-status muoto:**
GitOps-repoon asetetaan commit-status:
| Kenttä | Formaatti | Esimerkki |
|--------|-----------|-----------|
| Context | `{repo}/{GIT_TAG_PREFIX} {RUN_ID}` tai `{repo} {RUN_ID}` | `gitea-ci-library/agent-platform-helm 473` |
| Description | `Install to {env} {version}` | `Install to dev 0.2.0` |
| Target URL | Linkki code-repon committiin | `/niko/gitea-ci-library/commit/abc123` |
Jos tiedosto on jo halutussa versiossa (ei muutoksia), status saa descriptionin `Install to {env} {version} — no change`. Commit-pushia ei tehdä, GitOps-repo pysyy muuttumattomana.
- `{env}` parsitaan `INPUT_FILE`:stä (`dev/Chart.yaml``dev`)
- `{repo}` parsitaan `SOURCE_REPO`:sta (`niko/gitea-ci-library``gitea-ci-library`)
- `{GIT_TAG_PREFIX}` tulee env-varista (sama kuin `gitea-env.conf`:ssa)
### `scripts/dispatch-workflow.sh`
Dispatchaa workflow_dispatchin kohderepoon ja pollaa valmistumista.
Generoi automaattisesti `dispatch_id`-tunnisteen, lisää sen dispatch-
inputteihin ja tunnistaa workflow-runin kohdereposta `display_title`-
kentän perusteella. Toimii luotettavasti vaikka samassa repossa olisi
useita samanaikaisia dispatch-attribuutioita.
**Argumentit:**
| # | Pakollinen | Kuvaus |
|---|------------|--------|
| 1 | Kyllä | Kohderepo (esim. `niko/agent-platform-gitops`) |
| 2 | Kyllä | Workflow-tiedosto (esim. `ci-main.yml`) |
| 3 | Kyllä | Branch/ref |
| 4 | Kyllä | Inputs JSON |
| 5 | Kyllä | Gitea API URL |
| 6 | Kyllä | Gitea token |
| 7 | Ei | Aikakatkaisu minuutteina (oletus 360) |
Kutsujan ei tarvitse välittää `dispatch_id`:tä — skripti generoi sen
itse ja lisää inputteihin ennen dispatchia.
## [skip ci]
Commit-viestissä on `[skip ci]`, joka estää GitActions-runneria
triggeröimästä uutta CI-ajoa GitOps-repoon pushista. Näin vältetään
ääretön trigger-loop.
## Race condition
`dispatch-workflow.sh` tunnistaa jokaisen dispatchatun runin uniikilla
`dispatch_id`-tunnisteella `display_title`-kentästä. Vaikka useampi
artifakti dispatchaisi samaan aikaan ja useita workflow-runeja olisi
käynnissä rinnakkain, jokainen skripti löytää oikean runinsa.
## Sääntöjä
1. **Token ei kirjoita code repoon.** GitOps-repon workflow ei tarvitse
oikeuksia code repoon. Kaikki status-kutsut kohdistuvat vain
GitOps-repoon. Code repo asettaa oman statusnsa itse.
2. **Ei provider-workflowta.** GitOps-päivitys ei ole reusable workflow.
GitOps-repo ajaa `scripts/gitops-update.sh`:n suoraan.
3. **Vain `workflow_dispatch`.** GitOps-repon workflow:ta ei triggeröidä
pushista — se laukeaa vain dispatch-kutsusta.
4. **Dispatch ei palauta tarkkaa SHA:**ta. Code repo ei tiedä GitOps-
commitin SHA:ta ennen dispatch-valmistumista. Status asetetaan
dispatchin exit-koodin perusteella, ei GitOps-commitin tiedoilla.
5. **`dispatch_id` on pakollinen kohde-workflow'ssa** — ilman sitä
`dispatch-workflow.sh` ei löydä oikeaa runia moniajo-tilanteessa.
6. **`[skip ci]` commit-viestissä.** Pakollinen trigger-loopin estoon.
All of that is handled by `gitops-dispatch.yml`. You just call it.