- ci-engine.yml: 2 dummy test stepiä + agnostinen publish-stage (skannaa .meta-tiedostot, PATCH raportit, postaa status + linkki) - publish-git-pages.sh: palauta BASE URL (ilman index.html) - .meta-formaatti: lisää context, description, state kentät
This commit is contained in:
@@ -3,10 +3,45 @@ on:
|
||||
workflow_call:
|
||||
|
||||
jobs:
|
||||
hello:
|
||||
poc-report:
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
GITEA_API_URL: https://gitea.app.keskikuja.site
|
||||
PAGES_HOST: pages.helm-dev.keskikuja.site
|
||||
GIT_PAGES_PUBLISH_URL: https://pages.helm-dev.keskikuja.site
|
||||
GIT_PAGES_PUBLISH_TOKEN: ${{ secrets.GIT_PAGES_PUBLISH_TOKEN }}
|
||||
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Provider engine reachable
|
||||
run: echo "ci-engine (provider) running via uses binding"
|
||||
- name: Generate POC report
|
||||
run: |
|
||||
SHA8="${GITHUB_SHA:0:8}"
|
||||
mkdir -p "reports/${SHA8}"
|
||||
cat > "reports/${SHA8}/index.html" <<EOF
|
||||
<!DOCTYPE html>
|
||||
<html lang="fi">
|
||||
<head><meta charset="utf-8"><title>POC report ${SHA8}</title></head>
|
||||
<body>
|
||||
<h1>git-pages POC</h1>
|
||||
<p>Commit: ${GITHUB_SHA}</p>
|
||||
<p>Branch: ${GITHUB_REF_NAME}</p>
|
||||
<p>Run: ${GITHUB_RUN_ID}</p>
|
||||
</body>
|
||||
</html>
|
||||
EOF
|
||||
|
||||
- name: Publish report to git-pages
|
||||
id: publish
|
||||
run: |
|
||||
REPORT_URL=$(bash scripts/publish-git-pages.sh "reports/${GITHUB_SHA:0:8}")
|
||||
echo "report_url=${REPORT_URL}" >> "${GITHUB_OUTPUT}"
|
||||
echo "Published: ${REPORT_URL}"
|
||||
|
||||
- name: Set commit status with report link
|
||||
run: |
|
||||
bash scripts/report-status.sh \
|
||||
success \
|
||||
"POC report published" \
|
||||
"${{ steps.publish.outputs.report_url }}" \
|
||||
"ci-report"
|
||||
|
||||
@@ -7,3 +7,4 @@ on:
|
||||
jobs:
|
||||
call-engine:
|
||||
uses: niko/gitea-ci-library/.gitea/workflows/ci-engine.yml@plan/0003-alkaa-käyttämään-itseään-commit-raportti
|
||||
secrets: inherit
|
||||
|
||||
@@ -2,6 +2,65 @@
|
||||
|
||||
Reusable workflow -kirjasto Gitea Actionsille. Lisätietoja: [docs/](docs/)
|
||||
|
||||
## Provider-binding — miten consumer löytää providerin
|
||||
|
||||
Consumer kutsuu provideria `uses:`-viittauksella. Ei discoveryä — polku kovakoodataan consumerin
|
||||
`ci.yml`:ään. Gitea hakee workflow YAML:n **samalta Gitea-palvelimelta** annetusta reposta ja tagista.
|
||||
|
||||
### Polun muodostus
|
||||
|
||||
```
|
||||
{owner}/{repo}/.gitea/workflows/ci-engine.yml@{ref}
|
||||
```
|
||||
|
||||
| Osa | Mistä | Esimerkki (homelab) |
|
||||
|-----|-------|---------------------|
|
||||
| `owner` | Repopolun ensimmäinen osa — **käyttäjänimi tai org** | `niko` |
|
||||
| `repo` | Repon nimi | `gitea-ci-library` |
|
||||
| tiedosto | Providerin workflow | `.gitea/workflows/ci-engine.yml` |
|
||||
| `@ref` | Tag tai branch provider-repossa | `@v1` (tuotanto) |
|
||||
|
||||
**Owner ei ole org-pakotettu.** Homelabissa ei välttämättä ole organisaatiotasoa — silloin owner on
|
||||
käyttäjänimi (`niko/...`), ei keksitty `org/`-prefiksi.
|
||||
|
||||
Polku löytyy repostasi:
|
||||
|
||||
```bash
|
||||
git remote get-url origin
|
||||
# → ssh://git@gitea.app.keskikuja.site:30009/niko/gitea-ci-library.git
|
||||
# owner = niko, repo = gitea-ci-library
|
||||
```
|
||||
|
||||
Consumerin `ci.yml`:
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
call-engine:
|
||||
uses: niko/gitea-ci-library/.gitea/workflows/ci-engine.yml@v1
|
||||
secrets: inherit
|
||||
```
|
||||
|
||||
### Usea Gitea-palvelin
|
||||
|
||||
`uses:` resolvoi **vain sen Gitea-instanssin** repohakemistosta, jolla runner ajaa työn.
|
||||
Toisella palvelimella oleva repo ei näy — cross-server-viittaus ei toimi.
|
||||
|
||||
Jokaisella Gitea-palvelimella, jossa mikropalvelut ajavat CI:tä, **tämä kirjasto-repo pitää olla
|
||||
läsnä** samalla `owner/repo`-polulla:
|
||||
|
||||
```
|
||||
Gitea A (kehitys) Gitea B (tuotanto/homelab 2)
|
||||
niko/gitea-ci-library ←→ niko/gitea-ci-library (mirror tai push-mirror)
|
||||
↑ ↑
|
||||
consumer uses: consumer uses:
|
||||
niko/gitea-ci-library/... niko/gitea-ci-library/...
|
||||
```
|
||||
|
||||
Mirror pitää `ci-engine.yml`:n ja tagit (`v1`) saatavilla kulloisellakin palvelimella. Tämä korvaa
|
||||
provider-repon checkout-hackit workflowissa — binding hoituu Gitean natiivilla `uses:`-mekanismilla.
|
||||
|
||||
Periaatteet: [tmp/data-flow-design.md](tmp/data-flow-design.md)
|
||||
|
||||
## Main-haaran suojaus
|
||||
|
||||
Jokaisessa tätä kirjastoa käyttävässä repossa `main`-haara suojataan — koodi päätyy sinne vain PR:n kautta:
|
||||
@@ -39,7 +98,11 @@ Ilman disablointia käyttäjä voi valita merge-commitin PR:n merge-napista —
|
||||
|
||||
## Gitea Actions runner (K8s / Helm)
|
||||
|
||||
Act runner suorittaa Gitea Actions workflowt. Asennus Kubernetes-klusteriin Helm chartilla:
|
||||
Act runner suorittaa Gitea Actions workflowt. **IaC-lähde:** alla oleva Helm-snippet on
|
||||
klusterin totuus — muutokset vain snippetiin, sitten `helm upgrade --install` (ei käsin muokattuja
|
||||
arvoja klusterissa).
|
||||
|
||||
Asennus Kubernetes-klusteriin Helm chartilla:
|
||||
|
||||
### 1. Rekisteröintitoken
|
||||
|
||||
@@ -67,6 +130,7 @@ helm upgrade --install act-runner gitea/actions \
|
||||
--set giteaRootURL="$GITEA_URL" \
|
||||
--set existingSecret=act-runner-token \
|
||||
--set existingSecretKey=token \
|
||||
--set statefulset.dind.tag=29.5.2-dind \
|
||||
--set-string 'statefulset.runner.config=log:
|
||||
level: info
|
||||
cache:
|
||||
@@ -78,13 +142,28 @@ container:
|
||||
--create-namespace
|
||||
```
|
||||
|
||||
path escapes from parent -bugi korjattiin Docker 29.5.2:ssa. Tämän teko aikana default on 29.5.1 — juuri tämän alle jäävä versio.
|
||||
|
||||
Oletus-lokitaso on `debug` — suositeltu `info`. Näkee jobien aloitukset ja valmistumiset ilman konttikerrosten purkua (Downloading/Extracting-spämmiä). `debug` on tarpeen vain vianselvityksessä.
|
||||
|
||||
#### Docker (DinD)
|
||||
|
||||
Helm chart deployaa DinD:n init-sidecarina (`docker:dind` samassa podissa).
|
||||
`require_docker: true` kytkee jobit siihen — erillistä DinD-asennusta ei tarvita.
|
||||
|
||||
**DinD-tag pinottu:** `29.5.2-dind` (ei chart-oletusta). Docker 29.5.1 aiheuttaa act-runnerissa
|
||||
`path escapes from parent` -virheen job-kontin käynnistyksessä.
|
||||
|
||||
Maven/npm-ajot käyttävät vain workflow'n `container:`-imagea; DinD tarvitaan vasta Docker-buildissä.
|
||||
|
||||
### 3. Varmista
|
||||
|
||||
```bash
|
||||
kubectl get pods -n gitea-actions
|
||||
# → act-runner-runner-0 2/2 Running
|
||||
# → act-runner-runner-0 Running
|
||||
|
||||
kubectl exec -n gitea-actions act-runner-runner-0 -c dind -- docker version
|
||||
# → Server Version: 29.5.2 (tai uudempi)
|
||||
```
|
||||
|
||||
Gitean puolella runner ilmestyy Active-tilaan pienellä viiveellä:
|
||||
@@ -96,6 +175,9 @@ Site Admin → Actions → Runners (tai Org → Settings → Actions → Runner
|
||||
|
||||
Tämän jälkeen `.gitea/workflows/ci.yml` triggeröityy automaattisesti pushista.
|
||||
|
||||
Tarkista ennen ensimmäistä ajoa: [Provider-binding](#provider-binding--miten-consumer-löytää-providerin)
|
||||
(owner/repo, `@ref` pushattu, Repo → Settings → Actions enabled).
|
||||
|
||||
Lisätietoa runnerin toiminnasta, konteista ja DinD:stä: [docs/runner.md](docs/runner.md)
|
||||
|
||||
### Muuta
|
||||
@@ -105,4 +187,5 @@ Lisätietoa runnerin toiminnasta, konteista ja DinD:stä: [docs/runner.md](docs/
|
||||
| `giteaRootURL` | Gitea-palvelimen osoite (esim. `https://gitea.example.com`) |
|
||||
| `existingSecret` | Kubernetes secretin nimi, jossa token |
|
||||
| `existingSecretKey` | Avain secretin sisällä |
|
||||
| `statefulset.dind.tag` | DinD-image tag (`29.5.2-dind` minimi) |
|
||||
| `statefulset.runner.labels` | Mukautetut labelit |
|
||||
|
||||
+79
-7
@@ -6,6 +6,65 @@ raportista. Kuvaus ja perustelut: [docs/](docs/).
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Secretit
|
||||
|
||||
Luo secretit ennen Helm-asennusta: [docs/secrets.md](docs/secrets.md)
|
||||
|
||||
```bash
|
||||
# Avaa secrets.md ja suorita snipletit siellä
|
||||
# Palaa tähän jälkeen Helm-asennukseen
|
||||
```
|
||||
|
||||
### 2. Instanssin values-tiedosto
|
||||
|
||||
```bash
|
||||
# Muokkaa oma values-tiedosto (tai käytä dev-values.yaml)
|
||||
cp git-pages/dev-values.yaml my-values.yaml
|
||||
# Muuta: ingress.host, certificate.issuerRef.name, jne.
|
||||
```
|
||||
|
||||
### 3. Helm-asennus
|
||||
|
||||
```bash
|
||||
NS=git-pages
|
||||
VALUES=my-values.yaml
|
||||
|
||||
helm upgrade --install git-pages ./git-pages \
|
||||
-n "$NS" --create-namespace \
|
||||
-f "$VALUES"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Vie publish-token Gitea Actions-secretiin (per repo)
|
||||
|
||||
⚠️ **Tehtävä jokaiselle repoille**, joka julkaisee raportteja git-pagesiin.
|
||||
|
||||
```bash
|
||||
NS=git-pages
|
||||
REPO_OWNER="niko"
|
||||
REPO_NAME="gitea-ci-library"
|
||||
|
||||
# 1. Lue plaintext-token erillisestä secretistä
|
||||
TOKEN=$(kubectl get secret git-pages-publish-token -n "$NS" -o jsonpath='{.data.token}' | base64 -d)
|
||||
|
||||
# 2. Kopioi leikepöydälle
|
||||
echo -n "$TOKEN" | pbcopy # macOS
|
||||
# echo -n "$TOKEN" | xclip -sel clip # Linux
|
||||
|
||||
# 3. Avaa Gitea Actions secrets -sivu
|
||||
open "https://gitea.app.keskikuja.site/${REPO_OWNER}/${REPO_NAME}/settings/actions/secrets"
|
||||
# Linux: xdg-open "https://gitea.app.keskikuja.site/${REPO_OWNER}/${REPO_NAME}/settings/actions/secrets"
|
||||
```
|
||||
|
||||
**Gitea UI:ssa:** New Secret → Name: `GIT_PAGES_PUBLISH_TOKEN` → Value: **liitä leikepöydältä** → Save
|
||||
|
||||
> 💡 **Monelle repoille:** Toista vaiheet 3–4, tai katso [automatisointi](docs/secrets.md#automatisointi-useamman-repon-salaisuuden-lis%C3%A4%C3%A4miseen).
|
||||
|
||||
---
|
||||
|
||||
## Käyttöönotto
|
||||
|
||||
### 1. Secretit
|
||||
@@ -63,12 +122,25 @@ helm template git-pages ./git-pages -f "$VALUES"
|
||||
|
||||
## CI-julkaisu
|
||||
|
||||
Workflow tarvitsee vähintään:
|
||||
Julkaisu DNS-osoitteeseen BasicAuthilla:
|
||||
|
||||
| Muuttuja | Kuvaus |
|
||||
|----------|--------|
|
||||
| `PAGES_HOST` | Sama kuin `ingress.host` (julkinen lukulinkki) |
|
||||
| `GIT_PAGES_PUBLISH_URL` | `https://{ingress.host}` |
|
||||
| `GIT_PAGES_PUBLISH_TOKEN` | Gitea Actions secret — sama arvo kuin publish-auth-secretissa |
|
||||
```bash
|
||||
# Esimerkki: julkaise raportti
|
||||
curl -X PATCH https://ci-reports.helm-dev.keskikuja.site/owner/repo/commit/sha8/ \
|
||||
-H "Authorization: Basic $(echo -n "publish:$GIT_PAGES_PUBLISH_TOKEN" | base64)" \
|
||||
-H "Content-Type: application/x-tar" \
|
||||
--data-binary @raportti.tar
|
||||
```
|
||||
|
||||
Julkaisu repossa: `scripts/publish-git-pages.sh` (dogfood: `.gitea/workflows/ci-engine.yml`).
|
||||
**Vaaditut asetukset:**
|
||||
|
||||
| Missä | Arvo |
|
||||
|-------|------|
|
||||
| Gitea Actions secret | `GIT_PAGES_PUBLISH_TOKEN` (sama kuin K8s `git-pages-publish-token` `token`-avain) |
|
||||
| Scriptissä | `curl` käyttää BasicAuth-headeria yllä |
|
||||
|
||||
**K8s secretit (Traefik-yhteensopivuus):**
|
||||
- `git-pages-publish-auth` = htpasswd (Traefik BasicAuth middleware)
|
||||
- `git-pages-publish-token` = plaintext (luetaan Giteaan viedessä)
|
||||
|
||||
Tarkemmat secret-ohjeet: [docs/secrets.md](docs/secrets.md).
|
||||
|
||||
@@ -14,12 +14,13 @@ persistence:
|
||||
size: 5Gi
|
||||
|
||||
retention:
|
||||
enabled: true
|
||||
giteaApiUrl: https://gitea.app.keskikuja.site
|
||||
rules:
|
||||
branches:
|
||||
default:
|
||||
maxAgeDays: 90
|
||||
keepMin: 5
|
||||
branches:
|
||||
main:
|
||||
maxAgeDays: 365
|
||||
keepMin: 20
|
||||
|
||||
@@ -39,21 +39,22 @@ TLS-rajaukset ovat Kubernetes-kerroksessa (Traefik, cert-manager, Secretit).
|
||||
|
||||
## URL ja sisältö
|
||||
|
||||
Julkinen osoite = **selvä URL** + **Gitea-yhteensopiva polku**:
|
||||
Julkinen osoite peilaa suoraan Gitean commit-polun rakennetta, lisäten raportin nimen:
|
||||
|
||||
```
|
||||
https://pages.example.com/acme-corp/backend-api/reports/abc12345/index.html
|
||||
└─ selvä URL ─┘ └────────── Gitea-yhteensopiva polku ──────────┘
|
||||
https://ci-reports.helm-dev.keskikuja.site/niko/gitea-ci-library/commit/14cf2eaeed8a4033bc37c52b0b4c29f25b253ceb/cucumber/index.html
|
||||
└────────────── selvä URL ──────────────┘ └──────────────────────── Gitea-yhteensopiva polku ────────────────────────┘
|
||||
```
|
||||
|
||||
Levyllä (apex index-site):
|
||||
Levyllä (apex index-site) polku vastaa URL:ia:
|
||||
|
||||
```
|
||||
/app/data/
|
||||
{owner}/
|
||||
{repo}/
|
||||
reports/
|
||||
{sha8}/
|
||||
commit/
|
||||
{sha}/
|
||||
{raportin-nimi}/
|
||||
index.html
|
||||
.meta # branch, sha, published_at
|
||||
```
|
||||
|
||||
@@ -55,6 +55,20 @@ URL rakennetaan kahdesta erillisestä osasta, ei yhdestä sekavasta kaavasta:
|
||||
Polku vastaa Gitea Pages -käytäntöä (`/{owner}/{repo}/...`). Host on aina sama —
|
||||
ei `{owner}.pages...`-subdomainia.
|
||||
|
||||
**Konkreettinen esimerkki (nykyinen ympäristö):**
|
||||
|
||||
| Elementti | Arvo |
|
||||
|-----------|------|
|
||||
| **Gitea-instanssi** | `gitea.app.keskikuja.site` |
|
||||
| **Repo** | `niko/gitea-ci-library` |
|
||||
| **Haara** | `plan/0003-alkaa-käyttämään-itseään-commit-raportti` |
|
||||
| **Commit SHA** | `14cf2eaeed8a4033bc37c52b0b4c29f25b253ceb` |
|
||||
| **Raportin nimi** | `cucumber` (esim.) |
|
||||
| **Gitea commit -URL** | `https://gitea.app.keskikuja.site/niko/gitea-ci-library/commit/14cf2eaeed8a4033bc37c52b0b4c29f25b253ceb` |
|
||||
| **Raportin julkinen URL** | `https://ci-reports.helm-dev.keskikuja.site/niko/gitea-ci-library/commit/14cf2eaeed8a4033bc37c52b0b4c29f25b253ceb/cucumber/index.html` |
|
||||
|
||||
Tämä varmistaa, että CI-statuslinkki on suoraan luettavissa ilman domain-rewriteä: raportin polku peilaa täsmälleen Gitean commit-polun rakennetta (`/{owner}/{repo}/commit/{sha}/{raportin-nimi}/`). Koska yksi ajo tuottaa useita raportteja, raportin nimi erottaa ne toisistaan.
|
||||
|
||||
Julkaisija (CI tai muu asiakas) lähettää tar-arkiston PATCH/PUT:lla. Lukija hakee
|
||||
HTML:n GET:llä. Ei Gitea-git-integraatiota eikä `pages`-branchia.
|
||||
|
||||
@@ -78,15 +92,17 @@ Apex-juuri `/` on tyhjä tarkoituksella — ei landing-sivua.
|
||||
rewritea, ei per-owner subdomaineja, ei erillistä “julkaisu-URL vs. lukemis-URL” -kaavaa.
|
||||
Yksi TLS-sertifikaatti, yksi IngressRoute, yksi PVC.
|
||||
|
||||
### 2. Julkaisu ja luku erotettu
|
||||
### 2. Sovelluksen sisäinen security kytketty pois, Traefik hoitaa rajauksen
|
||||
|
||||
Julkaisu (PATCH/PUT) vaatii Traefik BasicAuthin. Luku (GET/HEAD) on erillinen reitti —
|
||||
katso [Luku-auth](#luku-auth) alla.
|
||||
`git-pages`-sovelluksen koko sisäinen security-mekanismi on kytketty pois päältä (`PAGES_INSECURE=1`). Kirjoitusoikeuden validointi tapahtuu yksinomaan Kubernetes-reunalla Traefik BasicAuth -middlewaren avulla. Sovellus palvelee sisältöä sokeana; klusteri päättää, kuka saa kirjoittaa.
|
||||
|
||||
**Miksi:** git-pages ajaa `PAGES_INSECURE=1` — sovellus ei validoi julkaisuoikeuksia.
|
||||
Kirjoitusoikeus on eksplisiittisesti Traefik Middlewaressä (`git-pages-publish-auth`).
|
||||
### 3. Julkaisu ja luku erotettu
|
||||
|
||||
### 3. Yksi publish-token, kaksi säilöä
|
||||
Julkaisu (PATCH/PUT) vaatii Traefik BasicAuthin. Luku (GET/HEAD) on erillinen reitti — katso [Luku-auth](#luku-auth) alla.
|
||||
|
||||
**Miksi:** Koska sovellus ei validoi julkaisuoikeuksia, kirjoitusoikeus on eksplisiittisesti eriytetty Traefik Middlewaressä (`git-pages-publish-auth`).
|
||||
|
||||
### 4. Yksi publish-token, kaksi säilöä
|
||||
|
||||
Sama plaintext-token: klusterin Secretissä htpasswd-hashina, julkaisijan secret-holvissa
|
||||
(esim. CI-alustan Actions-secret).
|
||||
@@ -94,14 +110,14 @@ Sama plaintext-token: klusterin Secretissä htpasswd-hashina, julkaisijan secret
|
||||
**Miksi:** Ei Gitea PAT:ia eikä `write:repository` -oikeutta. Token antaa vain
|
||||
julkaisuoikeuden tähän palveluun. Yksi arvo, kaksi paikkaa — ks. [secrets.md](secrets.md).
|
||||
|
||||
### 4. Secretit erillisessä hallinnassa
|
||||
### 5. Secretit erillisessä hallinnassa
|
||||
|
||||
`git-pages-publish-auth` luodaan ennen käyttöönottoa — ei osana sovelluksen konfiguraatiotiedostoja.
|
||||
|
||||
**Miksi:** Salaisuudet eivät kulje versionoiduissa arvoissa. Rotaatio ja SealedSecrets
|
||||
pysyvät operaattorin hallussa. Ks. [secrets.md](secrets.md).
|
||||
|
||||
### 5. Minimaalinen parametrisointi
|
||||
### 6. Minimaalinen parametrisointi
|
||||
|
||||
Instance-arvot (`host`, `issuer`, PVC) `{env}-values.yaml`:ssa. Resurssinimet,
|
||||
secret-nimet ja Traefik-wire kovakoodattu templatessa.
|
||||
|
||||
+207
-66
@@ -1,96 +1,237 @@
|
||||
# Secrets - Prerequisites
|
||||
# Secrets — git-pages
|
||||
|
||||
`git-pages` requires the following Kubernetes Secrets to exist before the cluster
|
||||
consumers can use them. These secrets are not managed by Helm — create them manually or
|
||||
via external secret management (SealedSecrets, External Secrets Operator, Vault).
|
||||
## Quick Start
|
||||
|
||||
TLS (`git-pages-tls`) is issued by cert-manager — not a manual prerequisite.
|
||||
### Vaihe 1: Secret-arkkitehtuuri
|
||||
|
||||
---
|
||||
Järjestelmässä on kaksi loogista salaista arvoa. Publish-token jaetaan kahteen K8s-secretiin (Traefik-yhteensopivuus):
|
||||
|
||||
## Secret Inventory
|
||||
| Looginen nimi | K8s | Gitea |
|
||||
|---|---|---|
|
||||
| `report_publish_api_token` (htpasswd) | `git-pages-publish-auth` (users) | - |
|
||||
| `report_publish_api_token` (plaintext) | `git-pages-publish-token` (token) | Actions Secret: `GIT_PAGES_PUBLISH_TOKEN` |
|
||||
| `reports_retention_read_token` | `git-pages-retention-gitea` (token) | PAT: `CI-REPORTS_READ_FOR_RETENTION` |
|
||||
|
||||
| Secret | Keys | Consumers | Required |
|
||||
|--------|------|-----------|----------|
|
||||
| `git-pages-publish-auth` | `users` | Traefik Middleware `git-pages-publish-auth` | Always |
|
||||
| `git-pages-retention-gitea` | `token` | CronJob `git-pages-retention` | Always |
|
||||
**Huomio:** Publish-token jaetaan kahteen secretiin, koska Traefik BasicAuth middleware vaatii single-key secretin sekä on muodossa, missä sitä ei saa takaisin. Jokainen repo mikä raportteja käyttää, tarvitsee selväkielisen arvon, joka on "ylimääräisessä" secretissä.
|
||||
|
||||
---
|
||||
### Vaihe 2: Luo Gitea PAT (retention)
|
||||
|
||||
## Tokens
|
||||
|
||||
### Publish token
|
||||
|
||||
Ei Gitea PAT. Satunnainen merkkijono (`GIT_PAGES_PUBLISH_TOKEN`), joka menee kahteen paikkaan:
|
||||
|
||||
| Paikka | Muoto |
|
||||
|--------|-------|
|
||||
| Kubernetes `git-pages-publish-auth` | htpasswd: `publish:<hash>` |
|
||||
| Gitea Actions secret `GIT_PAGES_PUBLISH_TOKEN` | sama plaintext |
|
||||
|
||||
CI käyttää sitä Traefik BasicAuthiin. Ei git-kirjoitusoikeutta.
|
||||
|
||||
**Vienti Giteaan:** Organization or Repository → Settings → Actions → Secrets →
|
||||
`GIT_PAGES_PUBLISH_TOKEN` = publish-tokenin plaintext (org secret, jos usea repo julkaisee).
|
||||
|
||||
### Retention token
|
||||
|
||||
Gitea PAT CronJobille. CronJob listaa branchit repokohtaisesti
|
||||
(`GET /api/v1/repos/{owner}/{repo}/branches`) ja poistaa raportit, joiden `.meta.branch`
|
||||
ei ole enää Giteassa.
|
||||
|
||||
Tarvitaan vain **`read:repository`**. Ei `write:repository`. Tokenin omistajan täytyy
|
||||
nähdä kaikki repot, joista raportteja on levyllä.
|
||||
|
||||
**Ei viedä Giteaan** — vain Kubernetes Secret `git-pages-retention-gitea`.
|
||||
|
||||
**PAT Giteassa (read only):**
|
||||
**Avaa Gitea browserissa:**
|
||||
|
||||
1. Kirjaudu Gitea-käyttäjällä, jolla on luku kaikkiin raporttirepoihin
|
||||
2. **Settings** → **Applications** → **Generate New Token**
|
||||
3. Token name: esim. `git-pages-retention`
|
||||
4. Scopes: valitse vain **`read:repository`** — älä valitse `write:repository` eikä muita
|
||||
5. **Generate Token** → kopioi token heti (näytetään vain kerran)
|
||||
6. Aseta shelliin: `export GITEA_RETENTION_TOKEN='gitea_pat_…'`
|
||||
3. Token name: `CI-REPORTS_READ_FOR_RETENTION`
|
||||
4. Scopes: valitse vain **`read:repository`**
|
||||
5. **Generate Token** → **kopioi token heti** (näytetään vain kerran)
|
||||
6. Tallenna token talteen (`GITEA_RETENTION_TOKEN`)
|
||||
|
||||
---
|
||||
### Vaihe 3: Generoi publish-token
|
||||
|
||||
## Create Secrets
|
||||
**Palaa terminaalille:**
|
||||
|
||||
```bash
|
||||
GITEA_RETENTION_TOKEN="<from Gitea>"
|
||||
|
||||
GIT_PAGES_PUBLISH_TOKEN="$(openssl rand -base64 24)"
|
||||
echo "Publish-token generoitu. Tallennetaan K8s-secretiin Vaiheessa 4."
|
||||
echo "$GIT_PAGES_PUBLISH_TOKEN"
|
||||
```
|
||||
|
||||
### Vaihe 4: Luo K8s secrets
|
||||
|
||||
```bash
|
||||
NS=git-pages
|
||||
|
||||
# Publish
|
||||
GIT_PAGES_PUBLISH_TOKEN="$(openssl rand -base64 24)"
|
||||
|
||||
# 1. Publish-auth: htpasswd (Traefik BasicAuth - vaatii single-key secretin)
|
||||
kubectl create secret generic git-pages-publish-auth \
|
||||
--from-literal=users="$(docker run --rm httpd:2-alpine htpasswd -nb publish "$GIT_PAGES_PUBLISH_TOKEN")" \
|
||||
-n $NS
|
||||
-n "$NS"
|
||||
|
||||
echo "Gitea Actions → GIT_PAGES_PUBLISH_TOKEN:"
|
||||
echo "$GIT_PAGES_PUBLISH_TOKEN"
|
||||
# 2. Publish-token: plaintext (luetaan README:stä Giteaan viedessä)
|
||||
kubectl create secret generic git-pages-publish-token \
|
||||
--from-literal=token="$GIT_PAGES_PUBLISH_TOKEN" \
|
||||
-n "$NS"
|
||||
|
||||
# Retention (PAT luotu yllä Giteassa)
|
||||
# 3. Retention (käyttää Vaiheessa 2 luotua PAT:ia)
|
||||
kubectl create secret generic git-pages-retention-gitea \
|
||||
--from-literal=token="$GITEA_RETENTION_TOKEN" \
|
||||
-n $NS
|
||||
-n "$NS"
|
||||
|
||||
kubectl get secrets -n $NS
|
||||
kubectl get secrets -n "$NS"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Secret Management (Production)
|
||||
### Seuraava: Helm-asennus
|
||||
|
||||
Secrets can be created manually with the snippets above, or migrated to a secret management
|
||||
solution. The `kubectl create` blocks are the rolling source — replace them with the target
|
||||
tool's equivalent when ready:
|
||||
Palaa takaisin [README.md](../README.md#käyttöönotto) ja jatka kohdasta "Instanssin values-tiedosto".
|
||||
|
||||
| Approach | Replaces `kubectl create` with |
|
||||
|----------|-------------------------------|
|
||||
| Manual rotation | Re-run the same snippets with new values |
|
||||
| SealedSecrets | `kubeseal` encrypted manifest |
|
||||
| External Secrets Operator | `ExternalSecret` CR pointing to the vault |
|
||||
| Vault / other | Vault agent / CSI driver injection |
|
||||
---
|
||||
|
||||
Structure of `docs/secrets.md` stays identical regardless of the chosen approach.
|
||||
## Secret Arkkitehtuuri
|
||||
|
||||
### Loogiset salaisuudet
|
||||
|
||||
| Looginen nimi | K8s | Gitea |
|
||||
|---|---|---|
|
||||
| `report_publish_api_token` | `git-pages-publish-auth` (htpasswd) | Actions Secret: `GIT_PAGES_PUBLISH_TOKEN` |
|
||||
| `reports_retention_read_token` | `git-pages-retention-gitea` (token) | PAT: `CI-REPORTS_READ_FOR_RETENTION` |
|
||||
|
||||
### Secret Reference Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Publish Flow"
|
||||
P1["Actions Secret<br/>GIT_PAGES_PUBLISH_TOKEN"]
|
||||
P2["K8s Secret<br/>git-pages-publish-auth"]
|
||||
P1 -->|token| TRAEFIK
|
||||
P2 -->|htpasswd| TRAEFIK
|
||||
TRAEFIK["Traefik BasicAuth"]
|
||||
end
|
||||
|
||||
subgraph "Retention Flow"
|
||||
R1["K8s Secret<br/>git-pages-retention-gitea"]
|
||||
R2["Gitea PAT<br/>CI-REPORTS_READ_FOR_RETENTION"]
|
||||
R1 -->|token| CJ["CronJob"]
|
||||
R2 -->|API auth| GITEA["Gitea API"]
|
||||
CJ -->|read branches| GITEA
|
||||
end
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Flow 1: Julkaisu (Publish)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Actions as Gitea Actions
|
||||
participant Traefik as Traefik
|
||||
participant K8sAuth as K8s Secret<br/>git-pages-publish-auth
|
||||
participant K8sToken as K8s Secret<br/>git-pages-publish-token
|
||||
participant GP as git-pages
|
||||
|
||||
Note over Actions: 1. Lue plaintext-token
|
||||
Actions->>K8sToken: lue token-avain
|
||||
K8sToken-->>Actions: plaintext token
|
||||
|
||||
Note over Actions: 2. Lähettää raportin
|
||||
Actions->>Traefik: PUT / + BasicAuth<br/>publish:TOKEN + repo-url
|
||||
Traefik->>K8sAuth: lue users (htpasswd)
|
||||
K8sAuth-->>Traefik: publish:$apr1$...
|
||||
alt Token match
|
||||
Traefik->>GP: välitä
|
||||
GP-->>Traefik: 200 OK
|
||||
Traefik-->>Actions: 200 OK
|
||||
else Token ei match
|
||||
Traefik-->>Actions: 401 Unauthorized
|
||||
end
|
||||
```
|
||||
|
||||
**Kaksi secretiä (Traefik-yhteensopivuus):**
|
||||
- `git-pages-publish-auth` = `users` (htpasswd, Traefik käyttää)
|
||||
- `git-pages-publish-token` = `token` (plaintext, luetaan Giteaan viedessä)
|
||||
|
||||
---
|
||||
|
||||
### Flow 2: Luku (Read)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Browser as Selain
|
||||
participant Traefik as Traefik
|
||||
participant GP as git-pages
|
||||
|
||||
Browser->>Traefik: GET /OWNER/REPO/commit/SHA/raportti/index.html
|
||||
Traefik->>GP: välitä (ei authia)
|
||||
GP-->>Traefik: HTML
|
||||
Traefik-->>Browser: HTML
|
||||
```
|
||||
|
||||
GET/HEAD-reitillä ei ole Middlewarea. Luku on julkinen, jos URL tunnetaan.
|
||||
|
||||
---
|
||||
|
||||
### Flow 3: Retention (Siivous)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant CronJob as CronJob<br/>git-pages-retention
|
||||
participant K8sSecret as K8s Secret<br/>git-pages-retention-gitea
|
||||
participant GiteaAPI as Gitea API
|
||||
participant PVC as PVC /app/data
|
||||
|
||||
Note over CronJob: 1. Lue PAT
|
||||
CronJob->>K8sSecret: lue token
|
||||
K8sSecret-->>CronJob: Gitea PAT
|
||||
|
||||
Note over CronJob: 2. Skaalaa deployment 0:aan
|
||||
CronJob->>PVC: lue .meta-tiedostot
|
||||
|
||||
Note over CronJob: 3. Kysy branch-lista
|
||||
CronJob->>GiteaAPI: GET /api/v1/repos/OWNER/REPO/branches
|
||||
GiteaAPI-->>CronJob: branch names
|
||||
|
||||
Note over CronJob: 4. Vertaa ja poista
|
||||
CronJob->>PVC: poista vanhentuneet
|
||||
|
||||
Note over CronJob: 5. Skaalaa deployment 1:een
|
||||
```
|
||||
|
||||
**Huomio:** Retention-PAT:in omistajalla on oltava lukuoikeus KAIKKIIN repoihin,
|
||||
joista raportteja on PVC:llä.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **"secret not found"** — luiko secretit ennen Helm-asennusta?
|
||||
- **"401 Unauthorized"** — onko Gitea Actions secret oikea?
|
||||
- **"found 2 elements for secret"** — Traefik vaatii single-key secretin. Varmista että `git-pages-publish-auth` sisältää vain `users`-avaimen.
|
||||
- **"token hukkuu"** — generoi uusi token (Vaihe 3) ja päivitä molemmat publish-secretit:
|
||||
```bash
|
||||
# 1. Generoi uusi
|
||||
GIT_PAGES_PUBLISH_TOKEN="$(openssl rand -base64 24)"
|
||||
|
||||
# 2. Päivitä K8s secrets (molemmat)
|
||||
NS=git-pages
|
||||
kubectl delete secret git-pages-publish-auth -n "$NS"
|
||||
kubectl delete secret git-pages-publish-token -n "$NS"
|
||||
|
||||
kubectl create secret generic git-pages-publish-auth \
|
||||
--from-literal=users="$(docker run --rm httpd:2-alpine htpasswd -nb publish "$GIT_PAGES_PUBLISH_TOKEN")" \
|
||||
-n "$NS"
|
||||
|
||||
kubectl create secret generic git-pages-publish-token \
|
||||
--from-literal=token="$GIT_PAGES_PUBLISH_TOKEN" \
|
||||
-n "$NS"
|
||||
|
||||
# 3. Päivitä Gitea Actions secret jokaisessa repoissa (luke README:stä)
|
||||
```
|
||||
|
||||
## Automatisointi: useamman repon salaisuuden lisääminen
|
||||
|
||||
Jos repoja on monta, voit käyttää Gitea API:ta (vaatii admin-tokenin):
|
||||
|
||||
```bash
|
||||
ADMIN_TOKEN="<gitea-admin-token>"
|
||||
NS=git-pages
|
||||
|
||||
# Lue plaintext-token erillisestä secretistä
|
||||
TOKEN=$(kubectl get secret git-pages-publish-token -n "$NS" -o jsonpath='{.data.token}' | base64 -d)
|
||||
|
||||
for repo in "owner/repo1" "owner/repo2" "owner/repo3"; do
|
||||
curl -X POST "https://gitea.example.com/api/v1/repos/$repo/actions/secrets" \
|
||||
-H "Authorization: token $ADMIN_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"name\":\"GIT_PAGES_PUBLISH_TOKEN\",\"data\":\"$TOKEN\"}"
|
||||
done
|
||||
```
|
||||
|
||||
Tai `tea` CLI:lla (Gitea:n virallinen CLI):
|
||||
|
||||
```bash
|
||||
tea actions secrets add --repo owner/repo1 GIT_PAGES_PUBLISH_TOKEN "$TOKEN"
|
||||
tea actions secrets add --repo owner/repo2 GIT_PAGES_PUBLISH_TOKEN "$TOKEN"
|
||||
```
|
||||
|
||||
@@ -1,34 +1,43 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
set -eo pipefail
|
||||
|
||||
DATA_ROOT="${DATA_ROOT:-/app/data}"
|
||||
PAGES_URL="${PAGES_URL:-http://localhost:3000}"
|
||||
PAGES_HOST="${PAGES_HOST:?PAGES_HOST is required}"
|
||||
CONFIG="${RETENTION_CONFIG:-/etc/retention/retention.json}"
|
||||
GITEA_API_URL="${GITEA_API_URL:?GITEA_API_URL is required}"
|
||||
GITEA_TOKEN="${GITEA_TOKEN:?GITEA_TOKEN is required}"
|
||||
GITEA_API_URL="${GITEA_API_URL:-}"
|
||||
GITEA_TOKEN="${GITEA_TOKEN:-}"
|
||||
|
||||
curl_with_host() {
|
||||
curl -sS -H "Host: ${PAGES_HOST}" "$@"
|
||||
}
|
||||
|
||||
[ -d "$DATA_ROOT" ] || { echo "ERROR: data root missing: $DATA_ROOT" >&2; exit 1; }
|
||||
[ -f "$CONFIG" ] || { echo "ERROR: config missing: $CONFIG" >&2; exit 1; }
|
||||
|
||||
default_max_age=$(jq -r '.default.maxAgeDays' "$CONFIG")
|
||||
default_keep_min=$(jq -r '.default.keepMin' "$CONFIG")
|
||||
BRANCH_CACHE=""
|
||||
branch_exists() {
|
||||
local owner="$1" repo="$2" branch="$3" key="${owner}/${repo}/${branch}"
|
||||
local status
|
||||
|
||||
gitea_get() {
|
||||
curl -fsS -H "Authorization: token ${GITEA_TOKEN}" \
|
||||
-H "Accept: application/json" "${GITEA_API_URL}${1}"
|
||||
[ -z "$GITEA_API_URL" ] && return 0
|
||||
[ -z "$GITEA_TOKEN" ] && return 0
|
||||
|
||||
if grep -q "^${key}$" <<< "$BRANCH_CACHE" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
status=$(curl -sS -o /dev/null -w "%{http_code}" \
|
||||
-H "Authorization: token ${GITEA_TOKEN}" \
|
||||
"${GITEA_API_URL}/api/v1/repos/${owner}/${repo}/branches/${branch}" 2>/dev/null || echo "000")
|
||||
|
||||
if [ "$status" = "200" ]; then
|
||||
BRANCH_CACHE="${BRANCH_CACHE}${key}"$'\n'
|
||||
return 0
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
branch_names_for_repo() {
|
||||
local owner="$1" repo="$2" page=1
|
||||
while true; do
|
||||
local resp count
|
||||
resp=$(gitea_get "/api/v1/repos/${owner}/${repo}/branches?limit=50&page=${page}")
|
||||
count=$(echo "$resp" | jq 'length')
|
||||
[ "$count" -eq 0 ] && break
|
||||
echo "$resp" | jq -r '.[].name'
|
||||
[ "$count" -lt 50 ] && break
|
||||
page=$((page + 1))
|
||||
done
|
||||
}
|
||||
default_max_age=$(jq -r '.branches.default.maxAgeDays // 90' "$CONFIG")
|
||||
default_keep_min=$(jq -r '.branches.default.keepMin // 5' "$CONFIG")
|
||||
|
||||
rule_max_age() {
|
||||
local branch="$1" v
|
||||
@@ -42,99 +51,147 @@ rule_keep_min() {
|
||||
[ -n "$v" ] && echo "$v" || echo "$default_keep_min"
|
||||
}
|
||||
|
||||
delete_report_dir() {
|
||||
echo "DELETE $1 ($2)"
|
||||
rm -rf "$1"
|
||||
}
|
||||
|
||||
age_days() {
|
||||
local meta="$1" published epoch_pub now
|
||||
published=$(jq -r '.published_at // empty' "$meta")
|
||||
if [ -z "$published" ]; then
|
||||
epoch_pub=$(stat -c %Y "$(dirname "$meta")" 2>/dev/null || stat -f %m "$(dirname "$meta")")
|
||||
else
|
||||
local published="$1" epoch_pub now
|
||||
epoch_pub=$(date -u -d "$published" +%s 2>/dev/null || echo 0)
|
||||
fi
|
||||
now=$(date -u +%s)
|
||||
[ "$epoch_pub" -eq 0 ] && echo 99999 && return
|
||||
now=$(date -u +%s)
|
||||
echo $(( (now - epoch_pub) / 86400 ))
|
||||
}
|
||||
|
||||
CACHE_DIR=$(mktemp -d)
|
||||
trap 'rm -rf "$CACHE_DIR"' EXIT
|
||||
|
||||
active_branches_file() {
|
||||
local owner="$1" repo="$2"
|
||||
local cache="${CACHE_DIR}/${owner}__${repo}.branches"
|
||||
if [ ! -f "$cache" ]; then
|
||||
branch_names_for_repo "$owner" "$repo" | sort -u > "$cache"
|
||||
fi
|
||||
echo "$cache"
|
||||
}
|
||||
|
||||
branch_active() {
|
||||
grep -Fxq "$2" "$1"
|
||||
}
|
||||
|
||||
parse_report_path() {
|
||||
local report_dir="$1" rel rest
|
||||
rel="${report_dir#"${DATA_ROOT}/"}"
|
||||
REPO_OWNER="${rel%%/*}"
|
||||
parse_path() {
|
||||
local rel="$1"
|
||||
OWNER="${rel%%/*}"
|
||||
rest="${rel#*/}"
|
||||
REPO_NAME="${rest%%/*}"
|
||||
REPO="${rest%%/*}"
|
||||
}
|
||||
|
||||
# Pass 1: deleted branches (always — no config)
|
||||
while IFS= read -r meta; do
|
||||
report_dir=$(dirname "$meta")
|
||||
parse_report_path "$report_dir"
|
||||
branch=$(jq -r '.branch // empty' "$meta")
|
||||
[ -n "$branch" ] || continue
|
||||
echo "Fetching manifest from ${PAGES_URL}/.git-pages/manifest.json"
|
||||
MANIFEST=$(curl_with_host "${PAGES_URL}/.git-pages/manifest.json")
|
||||
echo "Manifest loaded"
|
||||
|
||||
cache=$(active_branches_file "$REPO_OWNER" "$REPO_NAME")
|
||||
if ! branch_active "$cache" "$branch"; then
|
||||
delete_report_dir "$report_dir" "branch deleted: ${branch}"
|
||||
META_PATHS=$(echo "$MANIFEST" | jq -r '.contents | to_entries[] | select(.key | test("/reports/")) | select(.key | endswith("/.meta")) | .key' 2>/dev/null || true)
|
||||
|
||||
if [ -z "$META_PATHS" ]; then
|
||||
echo "No .meta files found under /reports/ — nothing to clean"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== Phase 1: collect reports ==="
|
||||
declare -a REPORTS
|
||||
while IFS= read -r meta_path; do
|
||||
report_dir=$(dirname "$meta_path")
|
||||
parse_path "$report_dir"
|
||||
meta_content=$(curl_with_host "${PAGES_URL}/${meta_path}" 2>/dev/null || true)
|
||||
[ -n "$meta_content" ] || { echo " WARN: could not fetch $meta_path"; continue; }
|
||||
|
||||
branch=$(echo "$meta_content" | jq -r '.branch // empty' 2>/dev/null || true)
|
||||
published=$(echo "$meta_content" | jq -r '.published_at // empty' 2>/dev/null || true)
|
||||
|
||||
[ -n "$branch" ] || { echo " WARN: no branch in $meta_path"; continue; }
|
||||
[ -n "$published" ] || { echo " WARN: no published_at in $meta_path"; continue; }
|
||||
|
||||
days=$(age_days "$published")
|
||||
REPORTS+=("${report_dir}|${OWNER}|${REPO}|${branch}|${days}")
|
||||
echo " ${OWNER}/${REPO} branch=${branch} age=${days}d"
|
||||
done <<< "$META_PATHS"
|
||||
|
||||
[ "${#REPORTS[@]}" -eq 0 ] && { echo "No actionable reports"; exit 0; }
|
||||
|
||||
echo ""
|
||||
echo "=== Phase 2: check branches in Gitea ==="
|
||||
declare -a TO_DELETE
|
||||
declare -a KEEP
|
||||
for entry in "${REPORTS[@]}"; do
|
||||
IFS='|' read -r dir owner repo branch days <<< "$entry"
|
||||
|
||||
if [ -n "$GITEA_API_URL" ] && [ -n "$GITEA_TOKEN" ]; then
|
||||
if branch_exists "$owner" "$repo" "$branch"; then
|
||||
echo " BRANCH EXISTS: ${owner}/${repo}/${branch}"
|
||||
KEEP+=("${dir}|${owner}|${repo}|${branch}|${days}")
|
||||
else
|
||||
echo " BRANCH DELETED: ${owner}/${repo}/${branch} -> DELETE"
|
||||
TO_DELETE+=("$dir")
|
||||
fi
|
||||
done < <(find "$DATA_ROOT" -path '*/reports/*/.meta' -type f 2>/dev/null)
|
||||
else
|
||||
KEEP+=("${dir}|${owner}|${repo}|${branch}|${days}")
|
||||
fi
|
||||
done
|
||||
|
||||
# Pass 2: age + keepMin for active branches (newest first)
|
||||
SURVIVORS="${CACHE_DIR}/survivors.tsv"
|
||||
: > "$SURVIVORS"
|
||||
|
||||
while IFS= read -r meta; do
|
||||
report_dir=$(dirname "$meta")
|
||||
[ -d "$report_dir" ] || continue
|
||||
parse_report_path "$report_dir"
|
||||
branch=$(jq -r '.branch // empty' "$meta")
|
||||
[ -n "$branch" ] || continue
|
||||
|
||||
cache=$(active_branches_file "$REPO_OWNER" "$REPO_NAME")
|
||||
branch_active "$cache" "$branch" || continue
|
||||
|
||||
printf '%s\t%s\t%s\t%s\n' "${REPO_OWNER}/${REPO_NAME}" "$branch" "$report_dir" "$(age_days "$meta")" >> "$SURVIVORS"
|
||||
done < <(find "$DATA_ROOT" -path '*/reports/*/.meta' -type f 2>/dev/null)
|
||||
|
||||
current_key=""
|
||||
kept=0
|
||||
|
||||
while IFS=$'\t' read -r repo_key branch report_dir days; do
|
||||
key="${repo_key}|${branch}"
|
||||
if [ "$key" != "$current_key" ]; then
|
||||
current_key="$key"
|
||||
kept=0
|
||||
echo ""
|
||||
echo "=== Phase 3: apply retention rules to remaining reports ==="
|
||||
if [ "${#KEEP[@]}" -gt 0 ]; then
|
||||
IFS=$'\n'
|
||||
for entry in $(printf '%s\n' "${KEEP[@]}" | sort -t'|' -k4,4 -k5,5rn); do
|
||||
IFS='|' read -r dir owner repo branch days <<< "$entry"
|
||||
max_age=$(rule_max_age "$branch")
|
||||
keep_min=$(rule_keep_min "$branch")
|
||||
fi
|
||||
|
||||
if [ "$days" -gt "$max_age" ]; then
|
||||
delete_report_dir "$report_dir" "older than ${max_age}d (branch ${branch})"
|
||||
echo " DELETE: ${dir} (age ${days}d > maxAge ${max_age}d, branch ${branch})"
|
||||
TO_DELETE+=("$dir")
|
||||
continue
|
||||
fi
|
||||
|
||||
kept=$((kept + 1))
|
||||
if [ "$kept" -gt "$keep_min" ]; then
|
||||
delete_report_dir "$report_dir" "exceeds keepMin ${keep_min} (branch ${branch})"
|
||||
key="${branch}"
|
||||
count="${BRANCH_COUNTS[$key]:-0}"
|
||||
count=$((count + 1))
|
||||
BRANCH_COUNTS["$key"]=$count
|
||||
if [ "$count" -gt "$keep_min" ]; then
|
||||
echo " DELETE: ${dir} (kept ${keep_min}/${count}, exceeds keepMin, branch ${branch})"
|
||||
TO_DELETE+=("$dir")
|
||||
fi
|
||||
done < <(sort -t $'\t' -k1,1 -k2,2 -k4,4n "$SURVIVORS")
|
||||
done
|
||||
unset IFS
|
||||
fi
|
||||
|
||||
echo "Retention cleanup finished."
|
||||
if [ "${#TO_DELETE[@]}" -eq 0 ]; then
|
||||
echo "Nothing to delete"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== Phase 4: whiteout deletion ==="
|
||||
echo "Creating whiteout tar for ${#TO_DELETE[@]} report(s)..."
|
||||
|
||||
WHITEOUT_TAR=$(mktemp)
|
||||
trap 'rm -f "$WHITEOUT_TAR"' EXIT
|
||||
|
||||
python3 -c "
|
||||
import tarfile, sys
|
||||
|
||||
tar = tarfile.open(name='${WHITEOUT_TAR}', mode='w')
|
||||
|
||||
dirs = set()
|
||||
for d in sys.argv[1:]:
|
||||
dirs.add(d.strip())
|
||||
|
||||
tarinfo = tarfile.TarInfo()
|
||||
tarinfo.type = tarfile.CHRTYPE
|
||||
tarinfo.devmajor = 0
|
||||
tarinfo.devminor = 0
|
||||
|
||||
for d in sorted(dirs, key=len, reverse=True):
|
||||
info = tarinfo
|
||||
info.name = d
|
||||
tar.addfile(info)
|
||||
|
||||
tar.close()
|
||||
" "${TO_DELETE[@]}"
|
||||
|
||||
echo "Patching ${PAGES_URL}/ with whiteout tar..."
|
||||
HTTP_CODE=$(curl_with_host -X PATCH "${PAGES_URL}/" \
|
||||
-H "Content-Type: application/x-tar" \
|
||||
-H "Atomic: no" \
|
||||
--data-binary @"${WHITEOUT_TAR}" \
|
||||
-w "%{http_code}" \
|
||||
-o /dev/null)
|
||||
|
||||
echo "HTTP $HTTP_CODE"
|
||||
if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "204" ]; then
|
||||
echo "Retention cleanup finished."
|
||||
else
|
||||
echo "ERROR: retention HTTP ${HTTP_CODE}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -57,6 +57,46 @@ spec:
|
||||
periodSeconds: 20
|
||||
resources:
|
||||
{{- toYaml .Values.resources | nindent 12 }}
|
||||
{{- if and .Values.persistence.enabled .Values.retention.enabled (eq .Values.retention.mode "sidecar") }}
|
||||
- name: retention
|
||||
image: "{{ .Values.retention.image.repository }}:{{ .Values.retention.image.tag }}"
|
||||
imagePullPolicy: {{ .Values.retention.image.pullPolicy }}
|
||||
securityContext:
|
||||
runAsUser: 0
|
||||
command:
|
||||
- bash
|
||||
- -c
|
||||
- |
|
||||
set -euo pipefail
|
||||
echo "Retention sidecar: installing deps..."
|
||||
apt-get update -qq
|
||||
apt-get install -y --no-install-recommends curl jq python3 >/dev/null
|
||||
echo "Retention sidecar: ready"
|
||||
while true; do
|
||||
/scripts/retention-cleanup.sh
|
||||
echo "Retention sidecar: next run in 24h"
|
||||
sleep 86400
|
||||
done
|
||||
env:
|
||||
- name: PAGES_URL
|
||||
value: http://localhost:3000
|
||||
- name: PAGES_HOST
|
||||
value: {{ .Values.ingress.host | quote }}
|
||||
- name: RETENTION_CONFIG
|
||||
value: /etc/retention/retention.json
|
||||
- name: GITEA_API_URL
|
||||
value: {{ .Values.retention.giteaApiUrl | quote }}
|
||||
- name: GITEA_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: git-pages-retention-gitea
|
||||
key: token
|
||||
volumeMounts:
|
||||
- name: retention-scripts
|
||||
mountPath: /scripts
|
||||
- name: retention-config
|
||||
mountPath: /etc/retention
|
||||
{{- end }}
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
@@ -66,3 +106,15 @@ spec:
|
||||
persistentVolumeClaim:
|
||||
claimName: {{ include "git-pages.fullname" . }}-data
|
||||
{{- end }}
|
||||
{{- if and .Values.persistence.enabled .Values.retention.enabled (eq .Values.retention.mode "sidecar") }}
|
||||
- name: retention-scripts
|
||||
configMap:
|
||||
name: git-pages-retention
|
||||
defaultMode: 0755
|
||||
- name: retention-config
|
||||
configMap:
|
||||
name: git-pages-retention
|
||||
items:
|
||||
- key: retention.json
|
||||
path: retention.json
|
||||
{{- end }}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
{{- if .Values.persistence.enabled }}
|
||||
{{- if and .Values.persistence.enabled .Values.retention.enabled }}
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
{{- if .Values.persistence.enabled }}
|
||||
{{- if and .Values.persistence.enabled .Values.retention.enabled (eq .Values.retention.mode "cronjob") }}
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
{{- if .Values.persistence.enabled }}
|
||||
{{- if and .Values.persistence.enabled .Values.retention.enabled (eq .Values.retention.mode "cronjob") }}
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
|
||||
@@ -43,13 +43,16 @@ publishAuth:
|
||||
htpasswdUsers: ""
|
||||
|
||||
retention:
|
||||
enabled: false
|
||||
mode: sidecar
|
||||
schedule: "0 3 * * *"
|
||||
image:
|
||||
repository: bitnami/kubectl
|
||||
tag: "1.31.4"
|
||||
repository: debian
|
||||
tag: bookworm-slim
|
||||
pullPolicy: IfNotPresent
|
||||
giteaApiUrl: ""
|
||||
rules:
|
||||
branches:
|
||||
default:
|
||||
maxAgeDays: 90
|
||||
keepMin: 5
|
||||
branches: {}
|
||||
|
||||
@@ -34,7 +34,7 @@ cp -a "$REPORT_DIR/." "$WORK/${OWNER}/${REPO}/reports/${SHA8}/"
|
||||
cat > "$WORK/${OWNER}/${REPO}/reports/${SHA8}/.meta" <<EOF
|
||||
{"branch":"${GITHUB_REF_NAME:-}","sha":"${GITHUB_SHA}","published_at":"$(date -u +%Y-%m-%dT%H:%M:%SZ)"}
|
||||
EOF
|
||||
tar -cf "$TAR" -C "$WORK" "$OWNER"
|
||||
find "$WORK/$OWNER" \( -type f -o -type l \) -print | sed "s|^${WORK}/||" | tar -cf "$TAR" -C "$WORK" -T -
|
||||
|
||||
publish() {
|
||||
local method="$1"
|
||||
|
||||
Reference in New Issue
Block a user