Compare commits

...

18 Commits

Author SHA1 Message Date
Sean Mousseau 6979c3bb32 fix: resume interrupted jobs after startup (not only at boot) (#297)
The previous middleware logic gated recovery behind a
once-per-process flag. After the first request handled the boot-time
recovery check, the resume codepath never fired again — even when
new interruptions appeared later in the process lifetime.

Symptom: a sync that started after server boot, crashed mid-flight
(deadlock retry hitting maxRetries, network blip, container
restart of an upstream service, etc.) and never reached the resume
codepath would sit at `inProgress=true, lastCheckpoint=never`
forever. The periodic detector kept finding the stuck row and
logging `Found 1 interrupted jobs:` on every poll (driven by the
health endpoint via `hasJobsNeedingRecovery`), but the resumer
(`resumeInterruptedJob`) was only invoked from `initializeRecovery`
which the middleware never re-called.

Fix: replace the one-shot gate with an in-flight latch
(`recoveryInFlight`) that's released in a `finally` block. Throttling
of actual recovery work is delegated to the existing 5-minute
`skipIfRecentAttempt` check inside `initializeRecovery()`, which is
the right place for it. `recoveryInitialized` is kept and only used
to control whether the "first run" log lines fire.

Secondary fix: `findInterruptedJobs` previously logged
`Found N interrupted jobs:` unconditionally on every call, including
from passive polls in `hasJobsNeedingRecovery` (which the health
endpoint and middleware probe call frequently). That produced one
log line per poll per stuck job for as long as the job stayed stuck.
Make logging opt-in via a `logFound` parameter, default off; the
active recovery cycle in `initializeRecovery()` opts in so the
operator-facing log still surfaces which jobs are being worked on.

Related: #268 (partially addressed). The PR #280 / v3.15.7 fix was
about a JS scoping bug that made the *initial* migrate call's
catch path crash before transitioning the repo to `failed`. That
was one cause of stuck `mirroring` state; this PR addresses the
follow-on issue that even when a job *is* correctly detectable as
interrupted, the post-startup recovery path never re-engages.

Adds `orchestrator-resume-after-startup.test.ts` using the
structural-source test pattern from
`gitea-mirror-failure-recovery.test.ts` so the four guarantees
(no static gate; in-flight latch released in finally; logging
opt-in; active path opts in) are enforced without heavy mocks.
2026-05-23 20:08:33 +05:30
Sean Mousseau b9f14e55e2 fix: prevent duplicate issues & PRs on retry-after-deadlock + Link-header pagination (#296)
mirrorGitRepoIssuesToGitea and mirrorGitRepoPullRequestsToGitea both
had two compounding bugs that produced duplicate Gitea issues/PR-as-
issue rows on every sync against any non-SQLite backend.

(1) Pagination on existing-issues / existing-PRs / per-issue-comments
was wrong.

The loops paginated with `limit=100` and broke on `pageX.length <
itemsPerPage`. Gitea caps response size at server-side
[api].MAX_RESPONSE_ITEMS (default 50), so the very first page
already looks "short" and pagination terminated after one page. The
existing-issue and existing-PR maps were built from only ~50 items
per repo, so every issue/PR past page 1 was treated as new on every
sync and re-created via the CREATE branch.

Naive removal of the short-page break (relying only on "break on
empty page") doesn't work either: for some Gitea endpoints Gitea
returns the same data on every page when the page is past the
actual end instead of returning [], so the loop runs forever.

Fix: use the Link header (RFC 5988). If `rel="next"` is absent,
terminate pagination. Applied to: existing-issues pre-fetch,
existing-PRs pre-fetch, and per-issue comments fetch.

(2) Retry-after-deadlock duplicated issues/PRs even when the map
was correct.

Gitea's CreateIssue handler commits the issue insert in one
transaction and then deadlocks on the subsequent addLabel /
UPDATE repository transaction. The issue row is committed and
visible, but the in-memory dedup maps are never refreshed between
retries — processWithRetry re-invokes the callback, sees
existingIssue === undefined from the stale map, and creates a fresh
duplicate via httpPost.

Reproduces deterministically on MySQL (Error 1213 / 40001) and
PostgreSQL (40P01); SQLite escapes because writes serialize globally.

Fix: defensive recheck via httpGet by [GH-ISSUE #N] (issues) or
[PR #N] (pull requests) before the create call, with PATCH
fall-through when found. Applied to the issues create path AND
both create paths in the PR mirror (enriched + basic-fallback).
Also cache freshly-created items into the dedup maps after a
successful create so subsequent retries of the same per-item
callback don't lose track of it.

Adds gitea-issue-dedup-on-retry.test.ts using the structural-source
test pattern from gitea-mirror-failure-recovery.test.ts so all
guarantees are enforced without heavy module mocks (8 tests).
2026-05-23 20:08:23 +05:30
github-actions[bot] 2582988f94 chore: sync version to 3.16.0 2026-05-19 07:31:20 +00:00
ARUNAVO RAY 4ea62a9f3d chore: bump and digest-pin Bun base image to 1.3.14 (#295)
Same hardening principle as #293 for GitHub Actions: pin to an immutable
identifier so a future tag move can't silently change what we build against.

- Bumps oven/bun from 1.3.13 to 1.3.14 (released 2026-05-13)
- Pins to multi-arch digest sha256:9dba1a1b...db6f rather than just the tag
- Applies to both base and runner stages
2026-05-19 12:50:37 +05:30
ARUNAVO RAY 1f60b2cf39 feat: add Change password and Change email to account dropdown (#292)
Adds an account menu under the avatar in the header with Change password and
Change email actions, each opening a small dialog. Calls Better Auth's existing
change-password / change-email endpoints — no new API routes.

- Enables `user.changeEmail` in Better Auth with `updateEmailWithoutVerification`
  since the app runs with email verification disabled and no email sender is
  wired up
- Hides Change password for SSO-only users (no `credential` provider account),
  fails open if the listAccounts probe errors
- Resolves discussion #291 ("Change user password/email")
2026-05-19 12:45:04 +05:30
ARUNAVO RAY a02865a1aa ci: pin third-party GitHub Actions to commit SHAs (#293)
Tags are mutable. A compromised maintainer (or a maintainer's compromised
machine) can force-move v-tags to point at malicious commits, and any workflow
using `@vN` picks up the malicious code on its next run — see the recent
`actions-cool/issues-helper` / `maintain-one-comment` incident exfiltrating
credentials from `Runner.Worker` memory.

This commit pins every third-party action in the two workflows that handle
secrets (GHCR push, Docker Hub login, Scout token) to immutable 40-char SHAs,
with a trailing comment naming the release version for readability. SHAs are
the latest released tag at time of pin.

The two DeterminateSystems actions were on `@main` — a *branch* ref that moves
on every push, materially worse than a tag — and are now pinned to the latest
release SHAs (v22 / v13).

First-party `actions/*` and `github/codeql-action` are left on tags for now;
they're a separate, lower-risk follow-up.
2026-05-19 12:44:23 +05:30
github-actions[bot] ad549dad9b chore: sync version to 3.15.12 2026-05-16 06:50:50 +00:00
ARUNAVO RAY 20103220d9 chore: prune npm overrides that are no longer load-bearing (#290)
Removed 5 overrides whose constraints have since been picked up naturally
by the transitive dep graph. Verified by removing each and confirming the
resolved version (and the dep tree) is identical to what the override
produced:

- defu ^6.1.7 → still resolves 6.1.7
- fast-xml-parser ^5.5.6 → still resolves 5.5.6
- node-forge ^1.3.3 → package not in tree at all (override was dead)
- rollup >=4.59.0 → still resolves 4.59.0
- svgo ^4.0.1 → still resolves 4.0.1

Kept overrides that are still doing real work:

- @esbuild-kit/esm-loader → npm:tsx@^4.21.0 — deliberate replacement shim
- @xmldom/xmldom ^0.8.13, devalue ^5.8.1, fast-uri ^3.1.2,
  fast-xml-builder ^1.1.7, kysely ^0.28.17 — active CVE pins (#289)
- lodash ^4.18.1 — pins to the newer 4.18.x line over the legacy 4.17.x
  that transitive deps still pull
- picomatch ^4.0.4 — without it, picomatch@2.3.2 is added as a duplicate
  copy via a transitive that asks for 2.x

Future drift would be caught by Dependabot + the weekly Docker Scout
scan; the overrides above remain because they currently affect the tree.
2026-05-16 12:03:53 +05:30
ARUNAVO RAY fe2c825244 chore: bump npm overrides to patch HIGH-severity CVEs (#289)
Patches 9 Docker Scout HIGH alerts surfaced by the weekly image scan:

- @xmldom/xmldom 0.8.12 → 0.8.13 (CVE-2026-41672/3/4/5)
- devalue 5.6.4 → 5.8.1 (CVE-2026-42570)
- kysely 0.28.16 → 0.28.17 (CVE-2026-44635)
- fast-uri (new override) → ^3.1.2 (CVE-2026-6321, CVE-2026-6322)
- fast-xml-builder (new override) → ^1.1.7 (CVE-2026-44665)

All five resolve to fixed versions after `bun install`. Tests and astro
build pass locally.

Remaining open Docker Scout alerts (git-lfs Go stdlib, gnutls28, nghttp2)
are base-image or upstream-binary issues, not addressable via npm.
2026-05-16 11:42:53 +05:30
github-actions[bot] 4b858a0251 chore: sync version to 3.15.11 2026-05-16 04:28:38 +00:00
Eduardo Riguetto (Kralot) 585d2ceb84 fix: reconcile metadata on every sync instead of once per repo (#287)
Previously, mirror of issues/pull-requests/labels/milestones was
guarded by !metadataState.components.<component>, so once a repo had
been mirrored the metadata path was permanently skipped with logs
like "Issues already mirrored; skipping to avoid duplicates".

This meant title changes, new comments, label updates and milestone
edits on the source GitHub repo were never propagated, even when the
user explicitly clicked Sync.

The underlying mirror* functions already handle idempotent updates:
issues and PRs are matched by [GH-ISSUE #N] / [GH-PR #N] markers in
the title and PATCHed in place, labels are deduped by name, milestones
by title. The releases path already runs unconditionally for the same
reason ("always allowed to rerun for updates"); aligning the other
metadata paths with it.

Touched: mirrorGithubRepoToGitea, mirrorGitHubRepoToGiteaOrg, and the
syncGiteaRepoEnhanced path in gitea-enhanced.ts. Updated the
"already-synced-repo" test to assert reconciliation runs on resync.
2026-05-16 09:40:48 +05:30
Eduardo Riguetto (Kralot) 7c1f24dc2f fix: include organization_member in /user/repos affiliation (#286)
Affiliation was set to "owner,collaborator", omitting repos owned
by orgs the user belongs to. As a result:

- main sync, scheduler, and cleanup never saw org repos
- orgs appeared empty unless manually re-added via /api/sync/organization
- restart archived previously-mirrored org repos as orphans

GitHub's API default is owner,collaborator,organization_member;
restoring it fixes both symptoms with no other code changes.
2026-05-16 09:40:40 +05:30
github-actions[bot] 680b374c84 chore: sync version to 3.15.10 2026-05-04 08:37:38 +00:00
ARUNAVO RAY 088467a57d feat: add option to exclude collaborator repos from import (closes #279) (#283)
GitHub's listForAuthenticatedUser defaults to returning every repo the
user has access to (owner + collaborator + organization_member), which
imports a lot of noise for users who only want their own repos.

Adds an `includeCollaboratorRepos` toggle, defaulting to true to preserve
existing behavior. When disabled, the affiliation filter scopes the API
call to "owner" only.

The cleanup service overrides the filter to always include collaborator
repos when computing the "what's still on GitHub" list. Without this,
toggling the option off would mark previously-mirrored collab repos as
orphaned and archive/delete them from Gitea.

Wired through the schema, both UI<->DB mappers, the env-config loader
(with new INCLUDE_COLLABORATOR_REPOS env var), and the settings UI.
2026-05-04 14:00:10 +05:30
github-actions[bot] adb436444e chore: sync version to 3.15.9 2026-05-04 04:50:49 +00:00
ARUNAVO RAY cc635485f0 feat: surface auto-mirror toggle in automation settings (refs #278) (#282)
The fix in v3.15.8 made scheduleConfig.autoMirror an independent trigger
in the scheduler, but it remained reachable only via the AUTO_MIRROR_REPOS
env var. This adds a UI checkbox under the Automatic Syncing section so
the option can be toggled per-config without touching the environment.

The toggle is conditional on scheduling being enabled (since auto-mirror
without a scheduler is meaningless) and is independent of the existing
"Auto-mirror new starred repositories" toggle in GitHub settings. Together
they cover the full owned/starred matrix that the scheduler already
supports.

Plumbing: config-mapper.ts now round-trips autoMirror through the UI/DB
boundary, and ScheduleConfig in types/config.ts gets the matching field.
No schema or migration change — autoMirror was already in the zod schema.
2026-05-04 10:14:09 +05:30
github-actions[bot] 6f343de5fd chore: sync version to 3.15.8 2026-05-04 03:49:44 +00:00
ARUNAVO RAY a18f262ca7 fix: make autoMirrorStarred actually trigger auto-mirror (fixes #278) (#281)
The "Auto-mirror new starred repositories" checkbox in the GitHub settings
was a filter layered on top of scheduleConfig.autoMirror, which itself is
only settable via the AUTO_MIRROR_REPOS env var (no UI). So users who
checked the box saw their starred repos auto-imported but never mirrored.

Treat autoMirror and autoMirrorStarred as independent triggers in the
scheduler: autoMirror covers owned (and self-starred) repos, autoMirrorStarred
covers repos starred from other owners. Either flag on its own is enough
to enter the auto-mirror phase, and the filter scopes the work accordingly.

Also normalize the owner comparison to lowercase since GitHub usernames are
case-insensitive — previously a self-starred repo whose stored owner casing
differed from the configured owner would be misclassified as a third-party
star.

Behavior change worth flagging in release notes: anyone who currently has
the starred checkbox on (broken state) will start getting starred repos
mirrored on upgrade. AUTO_MIRROR_REPOS=true users see no change.
2026-05-04 09:12:57 +05:30
31 changed files with 1469 additions and 281 deletions
+9 -9
View File
@@ -51,13 +51,13 @@ jobs:
ref: ${{ env.SHA }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0
with:
driver-opts: network=host
- name: Log into registry
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
uses: docker/login-action@v3
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
@@ -66,7 +66,7 @@ jobs:
# Login to Docker Hub for Docker Scout (optional - provides better vulnerability data)
# Add DOCKERHUB_USERNAME and DOCKERHUB_TOKEN secrets to enable this
- name: Log into Docker Hub
uses: docker/login-action@v3
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
continue-on-error: true
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
@@ -108,7 +108,7 @@ jobs:
# Extract metadata for Docker
- name: Extract Docker metadata
id: meta
uses: docker/metadata-action@v5
uses: docker/metadata-action@c299e40c65443455700f0fdfc63efafe5b349051 # v5.10.0
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE }}
labels: |
@@ -124,7 +124,7 @@ jobs:
# Build and push Docker image
- name: Build and push Docker image
id: build-and-push
uses: docker/build-push-action@v6
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6.19.2
with:
context: .
platforms: linux/amd64,linux/arm64
@@ -139,7 +139,7 @@ jobs:
# Load image locally for security scanning (PRs only)
- name: Load image for scanning
if: github.event_name == 'pull_request'
uses: docker/build-push-action@v6
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6.19.2
with:
context: .
platforms: linux/amd64
@@ -212,7 +212,7 @@ jobs:
# Docker Scout comprehensive security analysis
- name: Docker Scout - Vulnerability Analysis & Recommendations
uses: docker/scout-action@v1
uses: docker/scout-action@bacf462e8d090c09660de30a6ccc718035f961e3 # v1.20.4
if: github.event_name != 'pull_request'
with:
command: cves,recommendations
@@ -226,7 +226,7 @@ jobs:
# Docker Scout for Pull Requests (using local image)
- name: Docker Scout - Vulnerability Analysis (PR)
uses: docker/scout-action@v1
uses: docker/scout-action@bacf462e8d090c09660de30a6ccc718035f961e3 # v1.20.4
if: github.event_name == 'pull_request'
with:
command: cves,recommendations
@@ -240,7 +240,7 @@ jobs:
# Compare to latest for PRs and pushes
- name: Docker Scout - Compare to Latest
uses: docker/scout-action@v1
uses: docker/scout-action@bacf462e8d090c09660de30a6ccc718035f961e3 # v1.20.4
if: github.event_name == 'pull_request'
with:
command: compare
+2 -2
View File
@@ -38,10 +38,10 @@ jobs:
- uses: actions/checkout@v4
- name: Install Nix
uses: DeterminateSystems/nix-installer-action@main
uses: DeterminateSystems/nix-installer-action@ef8a148080ab6020fd15196c2084a2eea5ff2d25 # v22
- name: Setup Nix Cache
uses: DeterminateSystems/magic-nix-cache-action@main
uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 # v13
- name: Regenerate bun.nix from bun.lock
run: nix run --accept-flake-config github:nix-community/bun2nix -- -o bun.nix
+2 -2
View File
@@ -1,6 +1,6 @@
# syntax=docker/dockerfile:1.4
FROM oven/bun:1.3.13-debian AS base
FROM oven/bun:1.3.14-debian@sha256:9dba1a1b43ce28c9d7931bfc4eb00feb63b0114720a0277a8f939ae4dfc9db6f AS base
WORKDIR /app
RUN apt-get update && apt-get -y upgrade && apt-get install -y --no-install-recommends \
python3 make g++ gcc wget sqlite3 openssl ca-certificates \
@@ -49,7 +49,7 @@ RUN git clone --branch "v${GIT_LFS_VERSION}" --depth 1 https://github.com/git-lf
&& install -m 755 /tmp/git-lfs/bin/git-lfs /usr/local/bin/git-lfs
# ----------------------------
FROM oven/bun:1.3.13-debian AS runner
FROM oven/bun:1.3.14-debian@sha256:9dba1a1b43ce28c9d7931bfc4eb00feb63b0114720a0277a8f939ae4dfc9db6f AS runner
WORKDIR /app
RUN apt-get update && apt-get -y upgrade && apt-get install -y --no-install-recommends \
git wget sqlite3 openssl ca-certificates \
+14 -13
View File
@@ -82,16 +82,13 @@
},
"overrides": {
"@esbuild-kit/esm-loader": "npm:tsx@^4.21.0",
"@xmldom/xmldom": "^0.8.12",
"defu": "^6.1.7",
"devalue": "^5.6.4",
"fast-xml-parser": "^5.5.6",
"kysely": "^0.28.16",
"@xmldom/xmldom": "^0.8.13",
"devalue": "^5.8.1",
"fast-uri": "^3.1.2",
"fast-xml-builder": "^1.1.7",
"kysely": "^0.28.17",
"lodash": "^4.18.1",
"node-forge": "^1.3.3",
"picomatch": "^4.0.4",
"rollup": ">=4.59.0",
"svgo": "^4.0.1",
},
"packages": {
"@acemir/cssom": ["@acemir/cssom@0.9.31", "", {}, "sha512-ZnR3GSaH+/vJ0YlHau21FjfLYjMpYVIzTD8M8vIEQvIGxeOXyXdzCI140rrCY862p/C/BbzWsjc1dgnM9mkoTA=="],
@@ -726,7 +723,7 @@
"@xmldom/is-dom-node": ["@xmldom/is-dom-node@1.0.1", "", {}, "sha512-CJDxIgE5I0FH+ttq/Fxy6nRpxP70+e2O048EPe85J2use3XKdatVM7dDVvFNjQudd9B49NPoZ+8PG49zj4Er8Q=="],
"@xmldom/xmldom": ["@xmldom/xmldom@0.8.12", "", {}, "sha512-9k/gHF6n/pAi/9tqr3m3aqkuiNosYTurLLUtc7xQ9sxB/wm7WPygCv8GYa6mS0fLJEHhqMC1ATYhz++U/lRHqg=="],
"@xmldom/xmldom": ["@xmldom/xmldom@0.8.13", "", {}, "sha512-KRYzxepc14G/CEpEGc3Yn+JKaAeT63smlDr+vjB8jRfgTBBI9wRj/nkQEO+ucV8p8I9bfKLWp37uHgFrbntPvw=="],
"acorn": ["acorn@8.16.0", "", { "bin": { "acorn": "bin/acorn" } }, "sha512-UVJyE9MttOsBQIDKw1skb9nAwQuR5wuGD3+82K6JgJlm/Y+KI92oNsMNGZCYdDsVtRHSak0pcV5Dno5+4jh9sw=="],
@@ -876,7 +873,7 @@
"detect-node-es": ["detect-node-es@1.1.0", "", {}, "sha512-ypdmJU/TbBby2Dxibuv7ZLW3Bs1QEmM7nHjEANfohJLvE0XVujisn1qPJcZxg+qDucsr+bP6fLD1rPS3AhJ7EQ=="],
"devalue": ["devalue@5.6.4", "", {}, "sha512-Gp6rDldRsFh/7XuouDbxMH3Mx8GMCcgzIb1pDTvNyn8pZGQ22u+Wa+lGV9dQCltFQ7uVw0MhRyb8XDskNFOReA=="],
"devalue": ["devalue@5.8.1", "", {}, "sha512-4CXDYRBGqN+57wVJkuXBYmpAVUSg3L6JAQa/DFqm238G73E1wuyc/JhGQJzN7vUf/CMphYau2zXbfWzDR5aTEw=="],
"devlop": ["devlop@1.1.0", "", { "dependencies": { "dequal": "^2.0.0" } }, "sha512-RWmIqhcFf1lRYBvNmr7qTNuyCt/7/ns2jbpp1+PalgE/rDQcBT0fioSMUpJ93irlUhC5hrg4cYqe6U+0ImW0rA=="],
@@ -960,9 +957,9 @@
"fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="],
"fast-uri": ["fast-uri@3.1.0", "", {}, "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA=="],
"fast-uri": ["fast-uri@3.1.2", "", {}, "sha512-rVjf7ArG3LTk+FS6Yw81V1DLuZl1bRbNrev6Tmd/9RaroeeRRJhAt7jg/6YFxbvAQXUCavSoZhPPj6oOx+5KjQ=="],
"fast-xml-builder": ["fast-xml-builder@1.1.4", "", { "dependencies": { "path-expression-matcher": "^1.1.3" } }, "sha512-f2jhpN4Eccy0/Uz9csxh3Nu6q4ErKxf0XIsasomfOihuSUa3/xw6w8dnOtCDgEItQFJG8KyXPzQXzcODDrrbOg=="],
"fast-xml-builder": ["fast-xml-builder@1.2.0", "", { "dependencies": { "path-expression-matcher": "^1.5.0", "xml-naming": "^0.1.0" } }, "sha512-00aAWieqff+ZJhsXA4g1g7M8k+7AYoMUUHF+/zFb5U6Uv/P0Vl4QZo84/IcufzYalLuEj9928bXN9PbbFzMF0Q=="],
"fast-xml-parser": ["fast-xml-parser@5.5.6", "", { "dependencies": { "fast-xml-builder": "^1.1.4", "path-expression-matcher": "^1.1.3", "strnum": "^2.1.2" }, "bin": { "fxparser": "src/cli/cli.js" } }, "sha512-3+fdZyBRVg29n4rXP0joHthhcHdPUHaIC16cuyyd1iLsuaO6Vea36MPrxgAzbZna8lhvZeRL8Bc9GP56/J9xEw=="],
@@ -1090,7 +1087,7 @@
"kleur": ["kleur@4.1.5", "", {}, "sha512-o+NO+8WrRiQEE4/7nwRJhN1HWpVmJm511pBHUxPLtp0BUISzlBplORYSmTclCnJvQq2tKu/sgl3xVpkc7ZWuQQ=="],
"kysely": ["kysely@0.28.16", "", {}, "sha512-3i5pmOiZvMDj00qhrIVbH0AnioVTx22DMP7Vn5At4yJO46iy+FM8Y/g61ltenLVSo3fiO8h8Q3QOFgf/gQ72ww=="],
"kysely": ["kysely@0.28.17", "", {}, "sha512-nbD8lB9EB3wNdMhOCdx5Li8DxnLbvKByylRLcJ1h+4SkrowVeECAyZlyiKMThF7xFdRz0jSQ2MoJr+wXux2y0Q=="],
"lightningcss": ["lightningcss@1.32.0", "", { "dependencies": { "detect-libc": "^2.0.3" }, "optionalDependencies": { "lightningcss-android-arm64": "1.32.0", "lightningcss-darwin-arm64": "1.32.0", "lightningcss-darwin-x64": "1.32.0", "lightningcss-freebsd-x64": "1.32.0", "lightningcss-linux-arm-gnueabihf": "1.32.0", "lightningcss-linux-arm64-gnu": "1.32.0", "lightningcss-linux-arm64-musl": "1.32.0", "lightningcss-linux-x64-gnu": "1.32.0", "lightningcss-linux-x64-musl": "1.32.0", "lightningcss-win32-arm64-msvc": "1.32.0", "lightningcss-win32-x64-msvc": "1.32.0" } }, "sha512-NXYBzinNrblfraPGyrbPoD19C1h9lfI/1mzgWYvXUTe414Gz/X1FD2XBZSZM7rRTrMA8JL3OtAaGifrIKhQ5yQ=="],
@@ -1662,6 +1659,8 @@
"xml-name-validator": ["xml-name-validator@5.0.0", "", {}, "sha512-EvGK8EJ3DhaHfbRlETOWAS5pO9MZITeauHKJyb8wyajUfQUenkIg2MvLDTZ4T/TgIcm3HU0TFBgWWboAZ30UHg=="],
"xml-naming": ["xml-naming@0.1.0", "", {}, "sha512-k8KO9hrMyNk6tUWqUfkTEZbezRRpONVOzUTnc97VnCvyj6Tf9lyUR9EDAIeiVLv56jsMcoXEwjW8Kv5yPY52lw=="],
"xmlchars": ["xmlchars@2.2.0", "", {}, "sha512-JZnDKK8B0RCDw84FNdDAIpZK+JuJw+s7Lz8nksI7SIuU3UXJJslUthsi+uWBUYOwPFwW7W7PRLRfUKpxjtjFCw=="],
"xpath": ["xpath@0.0.34", "", {}, "sha512-FxF6+rkr1rNSQrhUNYrAFJpRXNzlDoMxeXN5qI84939ylEv3qqPFKa85Oxr6tDaJKqwW6KKyo2v26TSv3k6LeA=="],
@@ -1760,6 +1759,8 @@
"dom-serializer/entities": ["entities@4.5.0", "", {}, "sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw=="],
"fast-xml-builder/path-expression-matcher": ["path-expression-matcher@1.5.0", "", {}, "sha512-cbrerZV+6rvdQrrD+iGMcZFEiiSrbv9Tfdkvnusy6y0x0GKBXREFg/Y65GhIfm0tnLntThhzCnfKwp1WRjeCyQ=="],
"hast-util-from-html/parse5": ["parse5@7.3.0", "", { "dependencies": { "entities": "^6.0.0" } }, "sha512-IInvU7fabl34qmi9gY8XOVxhYyMyuH2xUNpb2q8/Y+7552KlejkRvqvD19nMoUW/uQGGbqNpA6Tufu5FL5BZgw=="],
"hast-util-raw/parse5": ["parse5@7.3.0", "", { "dependencies": { "entities": "^6.0.0" } }, "sha512-IInvU7fabl34qmi9gY8XOVxhYyMyuH2xUNpb2q8/Y+7552KlejkRvqvD19nMoUW/uQGGbqNpA6Tufu5FL5BZgw=="],
+1
View File
@@ -100,6 +100,7 @@ Standard GitHub Enterprise Cloud on `github.com` works with the default — no o
| `PRIVATE_REPOSITORIES` | Include private repositories | `false` | `true`, `false` |
| `PUBLIC_REPOSITORIES` | Include public repositories | `true` | `true`, `false` |
| `INCLUDE_ARCHIVED` | Include archived repositories | `false` | `true`, `false` |
| `INCLUDE_COLLABORATOR_REPOS` | Include repositories where you are a collaborator (not just owned). Set to `false` to limit imports to repos you own. | `true` | `true`, `false` |
| `SKIP_FORKS` | Skip forked repositories | `false` | `true`, `false` |
| `MIRROR_STARRED` | Mirror starred repositories | `false` | `true`, `false` |
| `MIRROR_STARRED_LISTS` | Optional comma-separated GitHub Star List names to mirror (only used when `MIRROR_STARRED=true`) | - | Comma-separated list names (empty = all starred repos) |
+7 -10
View File
@@ -1,7 +1,7 @@
{
"name": "gitea-mirror",
"type": "module",
"version": "3.15.7",
"version": "3.16.0",
"engines": {
"bun": ">=1.2.9"
},
@@ -45,16 +45,13 @@
},
"overrides": {
"@esbuild-kit/esm-loader": "npm:tsx@^4.21.0",
"@xmldom/xmldom": "^0.8.12",
"defu": "^6.1.7",
"devalue": "^5.6.4",
"fast-xml-parser": "^5.5.6",
"kysely": "^0.28.16",
"@xmldom/xmldom": "^0.8.13",
"devalue": "^5.8.1",
"fast-uri": "^3.1.2",
"fast-xml-builder": "^1.1.7",
"kysely": "^0.28.17",
"lodash": "^4.18.1",
"node-forge": "^1.3.3",
"picomatch": "^4.0.4",
"rollup": ">=4.59.0",
"svgo": "^4.0.1"
"picomatch": "^4.0.4"
},
"dependencies": {
"@astrojs/check": "^0.9.7",
+134
View File
@@ -0,0 +1,134 @@
import { useEffect, useState } from "react";
import { toast } from "sonner";
import { KeyRound, LogOut, Mail } from "lucide-react";
import { useAuth } from "@/hooks/useAuth";
import { authClient } from "@/lib/auth-client";
import { withBase } from "@/lib/base-path";
import { Avatar, AvatarFallback, AvatarImage } from "@/components/ui/avatar";
import { Button } from "@/components/ui/button";
import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuLabel,
DropdownMenuSeparator,
DropdownMenuTrigger,
} from "@/components/ui/dropdown-menu";
import { ChangePasswordDialog } from "./ChangePasswordDialog";
import { ChangeEmailDialog } from "./ChangeEmailDialog";
export function AccountMenu() {
const { user, logout, refreshUser } = useAuth();
const [hasPassword, setHasPassword] = useState<boolean | null>(null);
const [passwordOpen, setPasswordOpen] = useState(false);
const [emailOpen, setEmailOpen] = useState(false);
useEffect(() => {
if (!user) {
setHasPassword(null);
return;
}
let cancelled = false;
(async () => {
try {
const accounts = await authClient.listAccounts();
if (cancelled) return;
const list = Array.isArray(accounts) ? accounts : accounts?.data;
setHasPassword(
Array.isArray(list) && list.some((a) => a.providerId === "credential")
);
} catch {
// Fail open: if we can't check, show the option rather than locking the
// user out of changing their password.
if (!cancelled) setHasPassword(true);
}
})();
return () => {
cancelled = true;
};
}, [user?.id]);
if (!user) {
return (
<Button variant="outline" size="sm" asChild>
<a href={withBase("/login")}>Login</a>
</Button>
);
}
const handleLogout = async () => {
toast.success("Logged out successfully");
await new Promise((resolve) => setTimeout(resolve, 500));
logout();
};
return (
<>
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button
variant="outline"
size="lg"
className="relative h-10 w-10 rounded-full p-0"
>
<Avatar className="h-full w-full">
<AvatarImage src={user.image || ""} alt={user.name || user.email} />
<AvatarFallback>
{(user.name || user.email || "U").charAt(0).toUpperCase()}
</AvatarFallback>
</Avatar>
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end" className="w-60">
<DropdownMenuLabel className="font-normal">
<div className="flex flex-col gap-0.5">
{user.name && (
<span className="text-sm font-medium leading-none">
{user.name}
</span>
)}
<span className="text-xs leading-none text-muted-foreground truncate">
{user.email}
</span>
</div>
</DropdownMenuLabel>
<DropdownMenuSeparator />
{hasPassword && (
<DropdownMenuItem
onSelect={() => setPasswordOpen(true)}
className="cursor-pointer"
>
<KeyRound className="h-4 w-4 mr-2" />
Change password
</DropdownMenuItem>
)}
<DropdownMenuItem
onSelect={() => setEmailOpen(true)}
className="cursor-pointer"
>
<Mail className="h-4 w-4 mr-2" />
Change email
</DropdownMenuItem>
<DropdownMenuSeparator />
<DropdownMenuItem onSelect={handleLogout} className="cursor-pointer">
<LogOut className="h-4 w-4 mr-2" />
Logout
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
{hasPassword && (
<ChangePasswordDialog
open={passwordOpen}
onOpenChange={setPasswordOpen}
/>
)}
<ChangeEmailDialog
open={emailOpen}
onOpenChange={setEmailOpen}
currentEmail={user.email}
onUpdated={refreshUser}
/>
</>
);
}
+110
View File
@@ -0,0 +1,110 @@
import { useState } from "react";
import { toast } from "sonner";
import { authClient } from "@/lib/auth-client";
import { Button } from "@/components/ui/button";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from "@/components/ui/dialog";
interface ChangeEmailDialogProps {
open: boolean;
onOpenChange: (open: boolean) => void;
currentEmail: string;
onUpdated?: () => void;
}
export function ChangeEmailDialog({
open,
onOpenChange,
currentEmail,
onUpdated,
}: ChangeEmailDialogProps) {
const [newEmail, setNewEmail] = useState("");
const [isSubmitting, setIsSubmitting] = useState(false);
const reset = () => {
setNewEmail("");
setIsSubmitting(false);
};
const handleOpenChange = (next: boolean) => {
if (!next) reset();
onOpenChange(next);
};
const handleSubmit = async (e: React.FormEvent) => {
e.preventDefault();
const trimmed = newEmail.trim();
if (!trimmed) {
toast.error("Please enter a new email");
return;
}
if (trimmed.toLowerCase() === currentEmail.toLowerCase()) {
toast.error("New email must differ from current email");
return;
}
setIsSubmitting(true);
try {
const { error } = await authClient.changeEmail({ newEmail: trimmed });
if (error) {
toast.error(error.message || "Failed to change email");
return;
}
toast.success("Email updated.");
onUpdated?.();
handleOpenChange(false);
} catch (err) {
toast.error(err instanceof Error ? err.message : "Failed to change email");
} finally {
setIsSubmitting(false);
}
};
return (
<Dialog open={open} onOpenChange={handleOpenChange}>
<DialogContent className="sm:max-w-md">
<DialogHeader>
<DialogTitle>Change email</DialogTitle>
<DialogDescription>
Current: <span className="font-medium">{currentEmail}</span>
</DialogDescription>
</DialogHeader>
<form onSubmit={handleSubmit} className="space-y-4">
<div className="space-y-2">
<Label htmlFor="new-email">New email</Label>
<Input
id="new-email"
type="email"
autoComplete="email"
value={newEmail}
onChange={(e) => setNewEmail(e.target.value)}
disabled={isSubmitting}
required
/>
</div>
<DialogFooter>
<Button
type="button"
variant="outline"
onClick={() => handleOpenChange(false)}
disabled={isSubmitting}
>
Cancel
</Button>
<Button type="submit" disabled={isSubmitting}>
{isSubmitting ? "Updating..." : "Update email"}
</Button>
</DialogFooter>
</form>
</DialogContent>
</Dialog>
);
}
@@ -0,0 +1,158 @@
import { useState } from "react";
import { toast } from "sonner";
import { authClient } from "@/lib/auth-client";
import { Button } from "@/components/ui/button";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import { Checkbox } from "@/components/ui/checkbox";
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from "@/components/ui/dialog";
interface ChangePasswordDialogProps {
open: boolean;
onOpenChange: (open: boolean) => void;
}
export function ChangePasswordDialog({ open, onOpenChange }: ChangePasswordDialogProps) {
const [currentPassword, setCurrentPassword] = useState("");
const [newPassword, setNewPassword] = useState("");
const [confirmPassword, setConfirmPassword] = useState("");
const [revokeOtherSessions, setRevokeOtherSessions] = useState(true);
const [isSubmitting, setIsSubmitting] = useState(false);
const reset = () => {
setCurrentPassword("");
setNewPassword("");
setConfirmPassword("");
setRevokeOtherSessions(true);
setIsSubmitting(false);
};
const handleOpenChange = (next: boolean) => {
if (!next) reset();
onOpenChange(next);
};
const handleSubmit = async (e: React.FormEvent) => {
e.preventDefault();
if (!currentPassword || !newPassword) {
toast.error("Please fill in all fields");
return;
}
if (newPassword !== confirmPassword) {
toast.error("New passwords do not match");
return;
}
if (newPassword === currentPassword) {
toast.error("New password must differ from current password");
return;
}
setIsSubmitting(true);
try {
const { error } = await authClient.changePassword({
currentPassword,
newPassword,
revokeOtherSessions,
});
if (error) {
toast.error(error.message || "Failed to change password");
return;
}
toast.success(
revokeOtherSessions
? "Password updated. Other sessions signed out."
: "Password updated."
);
handleOpenChange(false);
} catch (err) {
toast.error(err instanceof Error ? err.message : "Failed to change password");
} finally {
setIsSubmitting(false);
}
};
return (
<Dialog open={open} onOpenChange={handleOpenChange}>
<DialogContent className="sm:max-w-md">
<DialogHeader>
<DialogTitle>Change password</DialogTitle>
<DialogDescription>
Enter your current password and a new one. You'll stay signed in on this device.
</DialogDescription>
</DialogHeader>
<form onSubmit={handleSubmit} className="space-y-4">
<div className="space-y-2">
<Label htmlFor="current-password">Current password</Label>
<Input
id="current-password"
type="password"
autoComplete="current-password"
value={currentPassword}
onChange={(e) => setCurrentPassword(e.target.value)}
disabled={isSubmitting}
required
/>
</div>
<div className="space-y-2">
<Label htmlFor="new-password">New password</Label>
<Input
id="new-password"
type="password"
autoComplete="new-password"
value={newPassword}
onChange={(e) => setNewPassword(e.target.value)}
disabled={isSubmitting}
required
minLength={8}
/>
</div>
<div className="space-y-2">
<Label htmlFor="confirm-password">Confirm new password</Label>
<Input
id="confirm-password"
type="password"
autoComplete="new-password"
value={confirmPassword}
onChange={(e) => setConfirmPassword(e.target.value)}
disabled={isSubmitting}
required
minLength={8}
/>
</div>
<div className="flex items-center gap-2">
<Checkbox
id="revoke-sessions"
checked={revokeOtherSessions}
onCheckedChange={(checked) => setRevokeOtherSessions(checked === true)}
disabled={isSubmitting}
/>
<Label htmlFor="revoke-sessions" className="text-sm font-normal cursor-pointer">
Sign out other devices
</Label>
</div>
<DialogFooter>
<Button
type="button"
variant="outline"
onClick={() => handleOpenChange(false)}
disabled={isSubmitting}
>
Cancel
</Button>
<Button type="submit" disabled={isSubmitting}>
{isSubmitting ? "Updating..." : "Update password"}
</Button>
</DialogFooter>
</form>
</DialogContent>
</Dialog>
);
}
@@ -269,6 +269,31 @@ export function AutomationSettings({
</div>
</div>
</div>
<div className="flex items-start space-x-3 pt-1">
<Checkbox
id="enable-auto-mirror-new"
checked={scheduleConfig.autoMirror ?? false}
className="mt-1.25"
onCheckedChange={(checked) =>
onScheduleChange({
...scheduleConfig,
autoMirror: !!checked,
})
}
/>
<div className="space-y-0.5 flex-1">
<Label
htmlFor="enable-auto-mirror-new"
className="text-sm font-normal cursor-pointer"
>
Auto-mirror new repositories
</Label>
<p className="text-xs text-muted-foreground">
Automatically mirror newly imported repositories on each scheduled sync. When off, new repos are imported for browsing but require a manual mirror click. (Starred repos have their own toggle in GitHub settings.)
</p>
</div>
</div>
</div>
)}
@@ -35,6 +35,7 @@ import {
HardDrive,
FileCode2,
Plus,
Users,
X
} from "lucide-react";
import type { GitHubConfig, MirrorOptions, AdvancedOptions, DuplicateNameStrategy } from "@/types/config";
@@ -244,6 +245,26 @@ export function GitHubMirrorSettings({
</div>
</div>
<div className="flex items-start space-x-3">
<Checkbox
id="collaborator-repos"
checked={githubConfig.includeCollaboratorRepos ?? true}
onCheckedChange={(checked) => handleGitHubChange('includeCollaboratorRepos', !!checked)}
/>
<div className="space-y-0.5 flex-1">
<Label
htmlFor="collaborator-repos"
className="text-sm font-normal cursor-pointer flex items-center gap-2"
>
<Users className="h-3.5 w-3.5" />
Include collaborator repositories
</Label>
<p className="text-xs text-muted-foreground">
Also mirror repos where you're a collaborator but not the owner. Turn off to limit imports to repos you own.
</p>
</div>
</div>
<div className="grid grid-cols-1 md:grid-cols-2 gap-4">
<div className="flex items-start space-x-3">
<Checkbox
+4 -43
View File
@@ -2,18 +2,11 @@ import { useAuth } from "@/hooks/useAuth";
import { Button } from "@/components/ui/button";
import { ModeToggle } from "@/components/theme/ModeToggle";
import { Avatar, AvatarFallback, AvatarImage } from "../ui/avatar";
import { toast } from "sonner";
import { Skeleton } from "@/components/ui/skeleton";
import { useLiveRefresh } from "@/hooks/useLiveRefresh";
import { useConfigStatus } from "@/hooks/useConfigStatus";
import { Menu, LogOut, PanelRightOpen, PanelRightClose } from "lucide-react";
import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuTrigger,
} from "@/components/ui/dropdown-menu";
import { PanelRightOpen, PanelRightClose } from "lucide-react";
import { AccountMenu } from "@/components/auth/AccountMenu";
import { withBase } from "@/lib/base-path";
interface HeaderProps {
@@ -26,7 +19,7 @@ interface HeaderProps {
}
export function Header({ currentPage, onNavigate, onMenuClick, onToggleCollapse, isSidebarCollapsed, isSidebarOpen }: HeaderProps) {
const { user, logout, isLoading } = useAuth();
const { isLoading } = useAuth();
const { isLiveEnabled, toggleLive } = useLiveRefresh();
const { isFullyConfigured, isLoading: configLoading } = useConfigStatus();
@@ -47,13 +40,6 @@ export function Header({ currentPage, onNavigate, onMenuClick, onToggleCollapse,
return isLiveEnabled ? 'Disable live refresh' : 'Enable live refresh';
};
const handleLogout = async () => {
toast.success("Logged out successfully");
// Small delay to show the toast before redirecting
await new Promise((resolve) => setTimeout(resolve, 500));
logout();
};
// Auth buttons skeleton loader
function AuthButtonsSkeleton() {
return (
@@ -141,32 +127,7 @@ export function Header({ currentPage, onNavigate, onMenuClick, onToggleCollapse,
<ModeToggle />
{isLoading ? (
<AuthButtonsSkeleton />
) : user ? (
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button variant="outline" size="lg" className="relative h-10 w-10 rounded-full p-0">
<Avatar className="h-full w-full">
<AvatarImage src={user.image || ""} alt={user.name || user.email} />
<AvatarFallback>
{(user.name || user.email || "U").charAt(0).toUpperCase()}
</AvatarFallback>
</Avatar>
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end" className="w-48">
<DropdownMenuItem onClick={handleLogout} className="cursor-pointer">
<LogOut className="h-4 w-4 mr-2" />
Logout
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
) : (
<Button variant="outline" size="sm" asChild>
<a href={withBase('/login')}>Login</a>
</Button>
)}
{isLoading ? <AuthButtonsSkeleton /> : <AccountMenu />}
</div>
</div>
</header>
+6
View File
@@ -145,6 +145,12 @@ export const auth = betterAuth({
input: false, // Don't show in signup form - we'll derive from email
}
},
changeEmail: {
enabled: true,
// Email verification isn't wired up (sendResetPassword is a TODO),
// so allow direct updates. Safe here because emails stay unverified.
updateEmailWithoutVerification: true,
},
},
// Plugins configuration
+1
View File
@@ -23,6 +23,7 @@ export const githubConfigSchema = z.object({
includeArchived: z.boolean().default(false),
includePrivate: z.boolean().default(true),
includePublic: z.boolean().default(true),
includeCollaboratorRepos: z.boolean().default(true),
includeOrganizations: z.array(z.string()).default([]),
starredReposOrg: z.string().optional(),
starredReposMode: z.enum(["dedicated-org", "preserve-owner"]).default("dedicated-org"),
+7
View File
@@ -15,6 +15,7 @@ interface EnvConfig {
type?: 'personal' | 'organization';
privateRepositories?: boolean;
publicRepositories?: boolean;
includeCollaboratorRepos?: boolean;
mirrorStarred?: boolean;
skipForks?: boolean;
includeArchived?: boolean;
@@ -111,6 +112,11 @@ function parseEnvConfig(): EnvConfig {
type: process.env.GITHUB_TYPE as 'personal' | 'organization',
privateRepositories: process.env.PRIVATE_REPOSITORIES === 'true',
publicRepositories: process.env.PUBLIC_REPOSITORIES === 'true',
// Tri-state parse so unset falls through to existingConfig / schema default (true).
includeCollaboratorRepos:
process.env.INCLUDE_COLLABORATOR_REPOS === 'true' ? true
: process.env.INCLUDE_COLLABORATOR_REPOS === 'false' ? false
: undefined,
mirrorStarred: process.env.MIRROR_STARRED === 'true',
skipForks: process.env.SKIP_FORKS === 'true',
includeArchived: process.env.INCLUDE_ARCHIVED === 'true',
@@ -270,6 +276,7 @@ export async function initializeConfigFromEnv(): Promise<void> {
includeArchived: envConfig.github.includeArchived ?? existingConfig?.[0]?.githubConfig?.includeArchived ?? false,
includePrivate: envConfig.github.privateRepositories ?? existingConfig?.[0]?.githubConfig?.includePrivate ?? false,
includePublic: envConfig.github.publicRepositories ?? existingConfig?.[0]?.githubConfig?.includePublic ?? true,
includeCollaboratorRepos: envConfig.github.includeCollaboratorRepos ?? existingConfig?.[0]?.githubConfig?.includeCollaboratorRepos ?? true,
includeOrganizations: envConfig.github.mirrorOrganizations ? [] : (existingConfig?.[0]?.githubConfig?.includeOrganizations ?? []),
starredReposOrg: envConfig.github.starredReposOrg || existingConfig?.[0]?.githubConfig?.starredReposOrg || 'starred',
starredReposMode: envConfig.github.starredReposMode || existingConfig?.[0]?.githubConfig?.starredReposMode || 'dedicated-org',
+7 -4
View File
@@ -848,12 +848,15 @@ describe("Enhanced Gitea Operations", () => {
}
);
// All metadata components were previously synced, so none should be called again
// Metadata reconciliation now runs on every sync (mirror* functions
// are idempotent and PATCH existing entries by marker/name/title).
// Releases are still skipped here because the flag is off in this config.
// Labels are still skipped because the issues path also handles labels.
expect(mockMirrorGitHubReleasesToGitea).not.toHaveBeenCalled();
expect(mockMirrorGitRepoIssuesToGitea).not.toHaveBeenCalled();
expect(mockMirrorGitRepoPullRequestsToGitea).not.toHaveBeenCalled();
expect(mockMirrorGitRepoIssuesToGitea).toHaveBeenCalledTimes(1);
expect(mockMirrorGitRepoPullRequestsToGitea).toHaveBeenCalledTimes(1);
expect(mockMirrorGitRepoLabelsToGitea).not.toHaveBeenCalled();
expect(mockMirrorGitRepoMilestonesToGitea).not.toHaveBeenCalled();
expect(mockMirrorGitRepoMilestonesToGitea).toHaveBeenCalledTimes(1);
});
});
+9 -39
View File
@@ -601,25 +601,23 @@ export async function syncGiteaRepoEnhanced({
return metadataOctokit;
};
// Reconcile metadata on every sync (matches the release path).
// The underlying mirror* functions are idempotent: issues/PRs are
// matched via [GH-ISSUE #N] / [GH-PR #N] markers and PATCHed in
// place, labels are deduped by name, milestones by title.
const shouldMirrorReleases =
!!config.giteaConfig?.mirrorReleases && !skipMetadataForStarred;
const shouldMirrorIssuesThisRun =
!!config.giteaConfig?.mirrorIssues &&
!skipMetadataForStarred &&
!metadataState.components.issues;
!!config.giteaConfig?.mirrorIssues && !skipMetadataForStarred;
const shouldMirrorPullRequests =
!!config.giteaConfig?.mirrorPullRequests &&
!skipMetadataForStarred &&
!metadataState.components.pullRequests;
!!config.giteaConfig?.mirrorPullRequests && !skipMetadataForStarred;
// Labels-only path; issues run already creates/reconciles labels.
const shouldMirrorLabels =
!!config.giteaConfig?.mirrorLabels &&
!skipMetadataForStarred &&
!shouldMirrorIssuesThisRun &&
!metadataState.components.labels;
!shouldMirrorIssuesThisRun;
const shouldMirrorMilestones =
!!config.giteaConfig?.mirrorMilestones &&
!skipMetadataForStarred &&
!metadataState.components.milestones;
!!config.giteaConfig?.mirrorMilestones && !skipMetadataForStarred;
if (shouldMirrorReleases) {
const octokit = ensureOctokit();
@@ -684,13 +682,6 @@ export async function syncGiteaRepoEnhanced({
);
}
}
} else if (
config.giteaConfig?.mirrorIssues &&
metadataState.components.issues
) {
console.log(
`[Sync] Issues already mirrored for ${repository.name}; skipping to avoid duplicates`
);
}
if (shouldMirrorPullRequests) {
@@ -721,13 +712,6 @@ export async function syncGiteaRepoEnhanced({
);
}
}
} else if (
config.giteaConfig?.mirrorPullRequests &&
metadataState.components.pullRequests
) {
console.log(
`[Sync] Pull requests already mirrored for ${repository.name}; skipping`
);
}
if (shouldMirrorLabels) {
@@ -760,13 +744,6 @@ export async function syncGiteaRepoEnhanced({
);
}
}
} else if (
config.giteaConfig?.mirrorLabels &&
metadataState.components.labels
) {
console.log(
`[Sync] Labels already mirrored for ${repository.name}; skipping`
);
}
if (shouldMirrorMilestones) {
@@ -799,13 +776,6 @@ export async function syncGiteaRepoEnhanced({
);
}
}
} else if (
config.giteaConfig?.mirrorMilestones &&
metadataState.components.milestones
) {
console.log(
`[Sync] Milestones already mirrored for ${repository.name}; skipping`
);
}
if (metadataUpdated) {
+282
View File
@@ -0,0 +1,282 @@
/**
* Regression test for duplicate-issue creation on retry-after-deadlock.
*
* `mirrorGitRepoIssuesToGitea` pre-fetches all existing Gitea issues
* into `giteaIssueByGitHubNumber` ONCE at function entry, then iterates
* per-issue via `processWithRetry`. Each iteration uses the cached map
* to decide CREATE vs PATCH.
*
* The bug: when Gitea's CreateIssue handler commits the issue insert
* in one transaction and then deadlocks on the addLabel / repository
* counter update in a second transaction, the issue row is committed
* and visible but the in-memory map is never refreshed between
* retries. So `processWithRetry` would call the callback again,
* `existingIssue` is still `undefined` from the stale map, and a fresh
* `httpPost` creates a duplicate.
*
* Reproduces deterministically on MySQL (1213/40001) and PostgreSQL
* (40P01). SQLite escapes because writes serialize globally.
*
* This test asserts on the *structure* of the source rather than
* invoking the function, because behavioral tests for the issue-mirror
* pipeline require heavy module mocks that pollute other test files
* (bun's mock.module is process-wide). See
* `gitea-mirror-failure-recovery.test.ts` for the same convention.
*
* The two structural guarantees this test enforces:
* (1) Before the create-issue httpPost call, the code performs a
* defensive recheck via httpGet that queries Gitea by
* `[GH-ISSUE #N]` title marker handles "previous attempt
* committed the issue then threw" scenarios.
* (2) After a successful httpPost create, the new issue is written
* back into `giteaIssueByGitHubNumber` handles "this attempt
* created the issue, but a later step in the same callback
* (e.g. comment sync) throws and triggers another retry"
* scenarios.
*/
import { describe, test, expect } from "bun:test";
import { readFileSync } from "node:fs";
import { join } from "node:path";
const SOURCE = readFileSync(join(import.meta.dir, "gitea.ts"), "utf8");
/**
* Locate the body of a function declaration by name. Walks from the
* declaration, balances parens to skip the parameter list (which can
* contain destructured object literals with their own braces), then
* finds the body's opening brace and its matching close.
*
* Same helper as in `gitea-mirror-failure-recovery.test.ts`; kept
* local to keep this test file self-contained.
*/
function extractFunctionBody(source: string, declarationStart: RegExp): string {
const match = source.match(declarationStart);
if (!match) {
throw new Error(`Could not locate declaration ${declarationStart}`);
}
let i = match.index! + match[0].length;
while (i < source.length && source[i] !== "(") i++;
if (source[i] !== "(") {
throw new Error(`No '(' after ${declarationStart}`);
}
let parenDepth = 0;
for (; i < source.length; i++) {
if (source[i] === "(") parenDepth++;
else if (source[i] === ")") {
parenDepth--;
if (parenDepth === 0) {
i++;
break;
}
}
}
while (i < source.length && source[i] !== "{") i++;
if (source[i] !== "{") {
throw new Error(`No body '{' for ${declarationStart}`);
}
let braceDepth = 0;
const startIdx = i;
for (; i < source.length; i++) {
if (source[i] === "{") braceDepth++;
else if (source[i] === "}") {
braceDepth--;
if (braceDepth === 0) {
return source.slice(startIdx, i + 1);
}
}
}
throw new Error(`Unterminated body for ${declarationStart}`);
}
describe("issue dedup on retry-after-deadlock", () => {
const body = extractFunctionBody(
SOURCE,
/export const mirrorGitRepoIssuesToGitea = async\b/
);
test("body contains the per-issue create branch we expect to guard", () => {
// Sanity: make sure the test is looking at the right code path.
// If these strings disappear due to a refactor, this test should
// fail loudly so a human reviews whether the dedup guarantees
// still hold in the new shape.
expect(
body.includes(
"giteaIssueByGitHubNumber.get(issue.number)"
),
"expected the per-issue lookup against giteaIssueByGitHubNumber"
).toBe(true);
expect(
body.match(/await httpPost\(\s*`\$\{config\.giteaConfig!\.url\}\/api\/v1\/repos\/\$\{giteaOwner\}\/\$\{repoName\}\/issues`/),
"expected the create-issue httpPost call"
).toBeTruthy();
});
test("defensive recheck via httpGet runs BEFORE the create httpPost", () => {
// The recheck must query Gitea by [GH-ISSUE #N] marker to catch
// the partial-commit case. Without it, a deadlock-after-insert
// returns 5xx, processWithRetry re-runs the callback, and the
// create call produces a duplicate row.
const recheckIdx = body.search(
/await httpGet\([^)]*\[GH-ISSUE #\$\{issue\.number\}\]/
);
expect(
recheckIdx,
"defensive recheck via httpGet using [GH-ISSUE #N] marker must exist"
).toBeGreaterThanOrEqual(0);
// The recheck must come before the create httpPost in source order.
// The first httpPost on the .../issues endpoint inside this
// function body is the create call; we anchor against it.
const createIdx = body.search(
/await httpPost\(\s*`\$\{config\.giteaConfig!\.url\}\/api\/v1\/repos\/\$\{giteaOwner\}\/\$\{repoName\}\/issues`/
);
expect(createIdx, "create httpPost call must exist").toBeGreaterThanOrEqual(0);
expect(
recheckIdx,
"the recheck must run BEFORE the create call so it can short-circuit on partial-commit duplicates"
).toBeLessThan(createIdx);
});
test("recheck hit short-circuits via PATCH and updates the cache", () => {
// When the recheck finds a hit (i.e. a previous failed attempt
// already created this issue), the code should:
// - cache the hit into giteaIssueByGitHubNumber so subsequent
// retries within this run also find it
// - go down the PATCH path (httpPatch) instead of httpPost
// - log a recognisable line so operators can spot recovery
expect(
body.includes(
"giteaIssueByGitHubNumber.set(issue.number, recheckHit)"
) ||
body.match(/giteaIssueByGitHubNumber\.set\(\s*issue\.number\s*,\s*recheckHit/),
"recheck hit must be written back into giteaIssueByGitHubNumber"
).toBeTruthy();
expect(
body.match(/Recovered orphan from prior failed attempt/i),
"a log line should make the recovery path visible in operator logs"
).toBeTruthy();
});
test("pre-fetch issues pagination uses Link header (not short-page heuristic)", () => {
// The previous `if (pageIssues.length < issuesPerPage) break;`
// heuristic was wrong in both directions:
// - Gitea caps response size at `[api].MAX_RESPONSE_ITEMS`
// (default 50), typically lower than `issuesPerPage` (100),
// so the very first page already looks "short" and
// pagination terminated after 50 items. Every issue past
// that was misclassified as new and duplicated on every sync.
// - Naive removal of that break, relying only on "break on
// empty page", can loop forever because some Gitea endpoints
// return the same data on every page when asked for a page
// past the actual end (instead of returning []).
//
// The correct fix is to use the Link header (RFC 5988): if
// `rel="next"` is absent, we're done.
//
// This test asserts:
// - the broken short-page check is gone
// - the existing-issues loop checks the Link header for next
const issuesPaginationRegion = body.substring(
body.indexOf("existingGiteaIssues.push"),
body.indexOf("issuesPage += 1") + 30
);
expect(
issuesPaginationRegion,
"issues pagination region should be present"
).not.toBe("");
expect(
/\bpageIssues\.length\s*<\s*issuesPerPage\b/.test(issuesPaginationRegion),
"the short-page break (pageIssues.length < issuesPerPage) must be removed"
).toBe(false);
expect(
/existingIssuesRes\.headers\.get\(\s*["']link["']\s*\)/.test(
issuesPaginationRegion
) && /rel="next"/.test(issuesPaginationRegion),
"the issues pagination loop must use the Link header (rel=\"next\") " +
"to decide whether to fetch the next page"
).toBe(true);
});
test("per-issue comments pagination also uses Link header (not short-page heuristic)", () => {
// Same correctness concerns as issues pagination above. The per-
// issue comments endpoint is subject to the same Gitea page-size
// cap, and naive empty-page detection has the same risk.
expect(
/\bpageComments\.length\s*<\s*commentsPerPage\b/.test(body),
"the short-page break (pageComments.length < commentsPerPage) must be removed"
).toBe(false);
// Look only at the comments-fetch region (not the whole file) so
// a future caller using a different response variable name in
// another place won't false-positive this assertion.
const commentsRegion = body.substring(
body.indexOf("existingComments.push"),
body.indexOf("commentsPage += 1") + 30
);
expect(
commentsRegion,
"comments pagination region should be present"
).not.toBe("");
expect(
/existingCommentsRes\.headers\.get\(\s*["']link["']\s*\)/.test(
commentsRegion
) && /rel="next"/.test(commentsRegion),
"the comments pagination loop must use the Link header (rel=\"next\") " +
"to decide whether to fetch the next page"
).toBe(true);
});
describe("PR mirror has the same guarantees", () => {
// mirrorGitRepoPullRequestsToGitea has parallel structure:
// - pre-fetches existing Gitea "issues that are mirrored PRs",
// keyed by `[PR #N]` marker in title
// - per-PR callback decides PATCH vs CREATE
// - same Gitea-side pagination cap and deadlock-after-commit
// risks apply
// The fix mirrors gitea-issues here.
const prBody = extractFunctionBody(
SOURCE,
/export async function mirrorGitRepoPullRequestsToGitea\b/
);
test("PR pre-fetch pagination uses Link header", () => {
expect(
/\bpageIssues\.length\s*<\s*prIssuesPerPage\b/.test(prBody),
"the short-page break (pageIssues.length < prIssuesPerPage) must be removed"
).toBe(false);
// The PR pre-fetch reuses the existingIssuesRes variable name
expect(
/existingIssuesRes\.headers\.get\(\s*["']link["']\s*\)/.test(prBody) &&
/rel="next"/.test(prBody),
"the PR pre-fetch loop must use the Link header (rel=\"next\")"
).toBe(true);
});
test("PR create path defensively rechecks via [PR #N] before httpPost", () => {
// Both the enriched and basic-fallback create paths must have
// a recheck so partial-commit retries don't duplicate the PR.
const rechecks =
prBody.match(/Recovered orphan from prior failed attempt for PR/g) ||
[];
expect(
rechecks.length,
"expected at least two 'Recovered orphan' log lines " +
"(one for the enriched create path, one for the basic-fallback path)"
).toBeGreaterThanOrEqual(2);
});
});
test("successful create caches the new issue into the dedup map", () => {
// Without this, a retry triggered by a *later* step in the same
// per-issue callback (e.g. comment sync throwing) would re-enter
// the create path on the next attempt — same root duplication
// pattern, different trigger.
expect(
body.match(
/giteaIssueByGitHubNumber\.set\(\s*issue\.number\s*,\s*createdIssue\.data\s*\)/
),
"after a successful create, the new issue must be stored in giteaIssueByGitHubNumber"
).toBeTruthy();
});
});
+188 -109
View File
@@ -898,14 +898,15 @@ export const mirrorGithubRepoToGitea = async ({
}
}
// Determine metadata operations to avoid duplicates
// Reconcile metadata on every sync (matches the release path above).
// The underlying mirror* functions are idempotent: issues/PRs are
// matched via [GH-ISSUE #N] / [GH-PR #N] markers and PATCHed in place,
// labels are deduped by name, milestones by title.
const shouldMirrorIssuesThisRun =
!!config.giteaConfig?.mirrorIssues &&
!skipMetadataForStarred &&
!metadataState.components.issues;
!!config.giteaConfig?.mirrorIssues && !skipMetadataForStarred;
console.log(
`[Metadata] Issue mirroring check: mirrorIssues=${config.giteaConfig?.mirrorIssues}, alreadyMirrored=${metadataState.components.issues}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorIssues=${shouldMirrorIssuesThisRun}`
`[Metadata] Issue mirroring check: mirrorIssues=${config.giteaConfig?.mirrorIssues}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorIssues=${shouldMirrorIssuesThisRun}`
);
if (shouldMirrorIssuesThisRun) {
@@ -931,19 +932,13 @@ export const mirrorGithubRepoToGitea = async ({
);
// Continue with other metadata operations even if issues fail
}
} else if (config.giteaConfig?.mirrorIssues && metadataState.components.issues) {
console.log(
`[Metadata] Issues already mirrored for ${repository.name}; skipping to avoid duplicates`
);
}
const shouldMirrorPullRequests =
!!config.giteaConfig?.mirrorPullRequests &&
!skipMetadataForStarred &&
!metadataState.components.pullRequests;
!!config.giteaConfig?.mirrorPullRequests && !skipMetadataForStarred;
console.log(
`[Metadata] Pull request mirroring check: mirrorPullRequests=${config.giteaConfig?.mirrorPullRequests}, alreadyMirrored=${metadataState.components.pullRequests}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorPullRequests=${shouldMirrorPullRequests}`
`[Metadata] Pull request mirroring check: mirrorPullRequests=${config.giteaConfig?.mirrorPullRequests}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorPullRequests=${shouldMirrorPullRequests}`
);
if (shouldMirrorPullRequests) {
@@ -968,23 +963,16 @@ export const mirrorGithubRepoToGitea = async ({
);
// Continue with other metadata operations even if PRs fail
}
} else if (
config.giteaConfig?.mirrorPullRequests &&
metadataState.components.pullRequests
) {
console.log(
`[Metadata] Pull requests already mirrored for ${repository.name}; skipping`
);
}
// Labels-only path; issues run above already creates/reconciles labels.
const shouldMirrorLabels =
!!config.giteaConfig?.mirrorLabels &&
!skipMetadataForStarred &&
!shouldMirrorIssuesThisRun &&
!metadataState.components.labels;
!shouldMirrorIssuesThisRun;
console.log(
`[Metadata] Label mirroring check: mirrorLabels=${config.giteaConfig?.mirrorLabels}, alreadyMirrored=${metadataState.components.labels}, issuesRunning=${shouldMirrorIssuesThisRun}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorLabels=${shouldMirrorLabels}`
`[Metadata] Label mirroring check: mirrorLabels=${config.giteaConfig?.mirrorLabels}, issuesRunning=${shouldMirrorIssuesThisRun}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorLabels=${shouldMirrorLabels}`
);
if (shouldMirrorLabels) {
@@ -1009,19 +997,13 @@ export const mirrorGithubRepoToGitea = async ({
);
// Continue with other metadata operations even if labels fail
}
} else if (config.giteaConfig?.mirrorLabels && metadataState.components.labels) {
console.log(
`[Metadata] Labels already mirrored for ${repository.name}; skipping`
);
}
const shouldMirrorMilestones =
!!config.giteaConfig?.mirrorMilestones &&
!skipMetadataForStarred &&
!metadataState.components.milestones;
!!config.giteaConfig?.mirrorMilestones && !skipMetadataForStarred;
console.log(
`[Metadata] Milestone mirroring check: mirrorMilestones=${config.giteaConfig?.mirrorMilestones}, alreadyMirrored=${metadataState.components.milestones}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorMilestones=${shouldMirrorMilestones}`
`[Metadata] Milestone mirroring check: mirrorMilestones=${config.giteaConfig?.mirrorMilestones}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorMilestones=${shouldMirrorMilestones}`
);
if (shouldMirrorMilestones) {
@@ -1046,13 +1028,6 @@ export const mirrorGithubRepoToGitea = async ({
);
// Continue with other metadata operations even if milestones fail
}
} else if (
config.giteaConfig?.mirrorMilestones &&
metadataState.components.milestones
) {
console.log(
`[Metadata] Milestones already mirrored for ${repository.name}; skipping`
);
}
if (metadataUpdated) {
@@ -1587,13 +1562,13 @@ export async function mirrorGitHubRepoToGiteaOrg({
}
}
// Reconcile metadata on every sync. See note in mirrorGithubRepoToGitea
// above. The underlying mirror* functions are idempotent.
const shouldMirrorIssuesThisRun =
!!config.giteaConfig?.mirrorIssues &&
!skipMetadataForStarred &&
!metadataState.components.issues;
!!config.giteaConfig?.mirrorIssues && !skipMetadataForStarred;
console.log(
`[Metadata] Issue mirroring check: mirrorIssues=${config.giteaConfig?.mirrorIssues}, alreadyMirrored=${metadataState.components.issues}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorIssues=${shouldMirrorIssuesThisRun}`
`[Metadata] Issue mirroring check: mirrorIssues=${config.giteaConfig?.mirrorIssues}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorIssues=${shouldMirrorIssuesThisRun}`
);
if (shouldMirrorIssuesThisRun) {
@@ -1619,22 +1594,13 @@ export async function mirrorGitHubRepoToGiteaOrg({
);
// Continue with other metadata operations even if issues fail
}
} else if (
config.giteaConfig?.mirrorIssues &&
metadataState.components.issues
) {
console.log(
`[Metadata] Issues already mirrored for ${repository.name}; skipping`
);
}
const shouldMirrorPullRequests =
!!config.giteaConfig?.mirrorPullRequests &&
!skipMetadataForStarred &&
!metadataState.components.pullRequests;
!!config.giteaConfig?.mirrorPullRequests && !skipMetadataForStarred;
console.log(
`[Metadata] Pull request mirroring check: mirrorPullRequests=${config.giteaConfig?.mirrorPullRequests}, alreadyMirrored=${metadataState.components.pullRequests}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorPullRequests=${shouldMirrorPullRequests}`
`[Metadata] Pull request mirroring check: mirrorPullRequests=${config.giteaConfig?.mirrorPullRequests}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorPullRequests=${shouldMirrorPullRequests}`
);
if (shouldMirrorPullRequests) {
@@ -1659,23 +1625,16 @@ export async function mirrorGitHubRepoToGiteaOrg({
);
// Continue with other metadata operations even if PRs fail
}
} else if (
config.giteaConfig?.mirrorPullRequests &&
metadataState.components.pullRequests
) {
console.log(
`[Metadata] Pull requests already mirrored for ${repository.name}; skipping`
);
}
// Labels-only path; issues run above already creates/reconciles labels.
const shouldMirrorLabels =
!!config.giteaConfig?.mirrorLabels &&
!skipMetadataForStarred &&
!shouldMirrorIssuesThisRun &&
!metadataState.components.labels;
!shouldMirrorIssuesThisRun;
console.log(
`[Metadata] Label mirroring check: mirrorLabels=${config.giteaConfig?.mirrorLabels}, alreadyMirrored=${metadataState.components.labels}, issuesRunning=${shouldMirrorIssuesThisRun}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorLabels=${shouldMirrorLabels}`
`[Metadata] Label mirroring check: mirrorLabels=${config.giteaConfig?.mirrorLabels}, issuesRunning=${shouldMirrorIssuesThisRun}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorLabels=${shouldMirrorLabels}`
);
if (shouldMirrorLabels) {
@@ -1700,22 +1659,13 @@ export async function mirrorGitHubRepoToGiteaOrg({
);
// Continue with other metadata operations even if labels fail
}
} else if (
config.giteaConfig?.mirrorLabels &&
metadataState.components.labels
) {
console.log(
`[Metadata] Labels already mirrored for ${repository.name}; skipping`
);
}
const shouldMirrorMilestones =
!!config.giteaConfig?.mirrorMilestones &&
!skipMetadataForStarred &&
!metadataState.components.milestones;
!!config.giteaConfig?.mirrorMilestones && !skipMetadataForStarred;
console.log(
`[Metadata] Milestone mirroring check: mirrorMilestones=${config.giteaConfig?.mirrorMilestones}, alreadyMirrored=${metadataState.components.milestones}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorMilestones=${shouldMirrorMilestones}`
`[Metadata] Milestone mirroring check: mirrorMilestones=${config.giteaConfig?.mirrorMilestones}, isStarred=${repository.isStarred}, starredCodeOnly=${config.githubConfig?.starredCodeOnly}, shouldMirrorMilestones=${shouldMirrorMilestones}`
);
if (shouldMirrorMilestones) {
@@ -1740,13 +1690,6 @@ export async function mirrorGitHubRepoToGiteaOrg({
);
// Continue with other metadata operations even if milestones fail
}
} else if (
config.giteaConfig?.mirrorMilestones &&
metadataState.components.milestones
) {
console.log(
`[Metadata] Milestones already mirrored for ${repository.name}; skipping`
);
}
if (metadataUpdated) {
@@ -2203,7 +2146,21 @@ export const mirrorGitRepoIssuesToGitea = async ({
if (!pageIssues.length) break;
existingGiteaIssues.push(...pageIssues);
if (pageIssues.length < issuesPerPage) break;
// Use the Link header (RFC 5988) to decide whether more pages
// exist. The old short-page-length heuristic was wrong in both
// directions:
// - Gitea caps response size at `[api].MAX_RESPONSE_ITEMS`
// (default 50), typically lower than `issuesPerPage` (100),
// so the very first page already looks "short" and
// pagination terminated after 50 items — every issue past
// that was misclassified as new and duplicated on every sync.
// - For some endpoints Gitea returns the same data on every
// page when the page is past the end, so a naive "break on
// empty" alone can loop forever if the server doesn't return
// []. Link header is the safe signal.
const linkHeader = existingIssuesRes.headers.get("link") || "";
if (!/\brel="next"/.test(linkHeader)) break;
issuesPage += 1;
}
@@ -2346,30 +2303,85 @@ export const mirrorGitRepoIssuesToGitea = async ({
}
);
} else {
const createdIssue = await httpPost(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues`,
issuePayload,
{
Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
}
);
targetIssueNumber = createdIssue.data.number;
// Defensive recheck before create: a previous retry attempt may
// have already created this issue and then thrown. The common
// trigger is Gitea's CreateIssue handler committing the issue
// insert in one transaction and then deadlocking on the
// addLabel / repository counter update in a second transaction.
// The issue row is committed and visible, but the in-memory
// giteaIssueByGitHubNumber map (built once at function entry)
// doesn't know about it, so without this check processWithRetry
// would create a duplicate every time the create returns 5xx
// after a partial commit.
//
// Reproduces deterministically on MySQL (Error 1213 / 40001)
// and PostgreSQL (40P01); SQLite escapes because writes
// serialize globally.
let recheckHit: any = null;
try {
const recheck = await httpGet(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues?state=all&type=issues&q=${encodeURIComponent(`[GH-ISSUE #${issue.number}]`)}`,
{
Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
}
);
const candidates = Array.isArray(recheck.data) ? recheck.data : [];
recheckHit = candidates.find(
(c: any) => extractGitHubIssueNumber(c.title) === issue.number
) ?? null;
} catch (_recheckErr) {
// Best-effort; fall through to create.
}
if (issue.state === "closed" && createdIssue.data.state !== "closed") {
try {
await httpPatch(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${targetIssueNumber}`,
{ state: "closed" },
{
Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
}
);
} catch (closeError) {
console.error(
`[Issues] Failed to close issue #${targetIssueNumber}: ${
closeError instanceof Error ? closeError.message : String(closeError)
}`
);
if (recheckHit) {
giteaIssueByGitHubNumber.set(issue.number, recheckHit);
existingIssue = recheckHit;
targetIssueNumber = recheckHit.number;
console.log(
`[Issues] Recovered orphan from prior failed attempt for #${issue.number}; switching to PATCH`
);
await httpPatch(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${targetIssueNumber}`,
{
title: issuePayload.title,
body: issuePayload.body,
state: issue.state === "closed" ? "closed" : "open",
labels: issuePayload.labels,
},
{
Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
}
);
} else {
const createdIssue = await httpPost(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues`,
issuePayload,
{
Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
}
);
targetIssueNumber = createdIssue.data.number;
// Cache the new issue immediately so a subsequent retry of
// this callback (e.g. triggered by a later step like comment
// sync failing) doesn't lose track of it.
giteaIssueByGitHubNumber.set(issue.number, createdIssue.data);
if (issue.state === "closed" && createdIssue.data.state !== "closed") {
try {
await httpPatch(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${targetIssueNumber}`,
{ state: "closed" },
{
Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
}
);
} catch (closeError) {
console.error(
`[Issues] Failed to close issue #${targetIssueNumber}: ${
closeError instanceof Error ? closeError.message : String(closeError)
}`
);
}
}
}
}
@@ -2412,7 +2424,13 @@ export const mirrorGitRepoIssuesToGitea = async ({
: [];
if (!pageComments.length) break;
existingComments.push(...pageComments);
if (pageComments.length < commentsPerPage) break;
// Use the Link header to decide whether more pages exist.
// See note on the existing-issues pagination above; the
// same Gitea behaviors (MAX_RESPONSE_ITEMS cap and
// repeated-data on out-of-bound pages) apply here.
const commentsLinkHeader =
existingCommentsRes.headers.get("link") || "";
if (!/\brel="next"/.test(commentsLinkHeader)) break;
commentsPage += 1;
}
const mirroredCommentIds = new Set<number>();
@@ -3040,7 +3058,12 @@ export async function mirrorGitRepoPullRequestsToGitea({
}
}
if (pageIssues.length < prIssuesPerPage) break;
// See note on the existing-issues pre-fetch above: rely on Link
// header (RFC 5988) rather than short-page heuristic. Gitea caps
// page size at MAX_RESPONSE_ITEMS (default 50), and some
// endpoints repeat data on out-of-bound pages instead of [].
const linkHeader = existingIssuesRes.headers.get("link") || "";
if (!/\brel="next"/.test(linkHeader)) break;
prIssuesPage += 1;
}
@@ -3141,7 +3164,36 @@ export async function mirrorGitRepoPullRequestsToGitea({
closed: pr.state === "closed" || pr.merged_at !== null,
};
const existingPrIssue = existingPrIssuesByNumber.get(pr.number);
let existingPrIssue = existingPrIssuesByNumber.get(pr.number);
// Defensive recheck (see same pattern in mirrorGitRepoIssuesToGitea):
// a previous attempt may have committed the PR-issue row and
// then thrown on the addLabel/repository-counter update. The
// pre-fetched map doesn't know about it, so without this check
// processWithRetry would create a duplicate every retry.
if (!existingPrIssue) {
try {
const recheck = await httpGet(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues?state=all&type=issues&q=${encodeURIComponent(`[PR #${pr.number}]`)}`,
{
Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
}
);
const candidates = Array.isArray(recheck.data) ? recheck.data : [];
const hit = candidates.find((c: any) => {
const m = String(c.title || "").match(/\[PR #(\d+)\]/i);
return m && Number.parseInt(m[1], 10) === pr.number;
});
if (hit) {
existingPrIssue = hit;
existingPrIssuesByNumber.set(pr.number, hit);
console.log(
`[Pull Requests] Recovered orphan from prior failed attempt for PR #${pr.number}; switching to PATCH`
);
}
} catch (_recheckErr) {
// Best-effort; fall through to create.
}
}
if (existingPrIssue) {
await httpPatch(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${existingPrIssue.number}`,
@@ -3202,7 +3254,34 @@ export async function mirrorGitRepoPullRequestsToGitea({
};
try {
const existingPrIssue = existingPrIssuesByNumber.get(pr.number);
let existingPrIssue = existingPrIssuesByNumber.get(pr.number);
// Defensive recheck — same pattern as the enriched create
// branch above. Without this, the basic-info fallback would
// dup on retry-after-deadlock just like the enriched path.
if (!existingPrIssue) {
try {
const recheck = await httpGet(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues?state=all&type=issues&q=${encodeURIComponent(`[PR #${pr.number}]`)}`,
{
Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
}
);
const candidates = Array.isArray(recheck.data) ? recheck.data : [];
const hit = candidates.find((c: any) => {
const m = String(c.title || "").match(/\[PR #(\d+)\]/i);
return m && Number.parseInt(m[1], 10) === pr.number;
});
if (hit) {
existingPrIssue = hit;
existingPrIssuesByNumber.set(pr.number, hit);
console.log(
`[Pull Requests] Recovered orphan from prior failed attempt for PR #${pr.number} (basic fallback); switching to PATCH`
);
}
} catch (_recheckErr) {
// Best-effort; fall through to create.
}
}
if (existingPrIssue) {
await httpPatch(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${existingPrIssue.number}`,
+100
View File
@@ -0,0 +1,100 @@
import { describe, expect, test, mock } from "bun:test";
import { getGithubRepositories } from "@/lib/github";
function makeRepo() {
return {
name: "demo",
full_name: "octo/demo",
html_url: "https://github.com/octo/demo",
clone_url: "https://github.com/octo/demo.git",
owner: { login: "octo", type: "User" },
private: false,
fork: false,
has_issues: true,
archived: false,
size: 1,
language: "TypeScript",
description: "",
default_branch: "main",
visibility: "public",
disabled: false,
created_at: "2024-01-01T00:00:00Z",
updated_at: "2024-01-02T00:00:00Z",
};
}
function makeOctokit() {
let captured: Record<string, unknown> | null = null;
const paginate = mock(async (_method: unknown, options?: Record<string, unknown>) => {
captured = options ?? null;
return [makeRepo()];
});
return {
octokit: {
paginate,
repos: { listForAuthenticatedUser: () => {} },
} as any,
getCaptured: () => captured,
};
}
describe("getGithubRepositories - affiliation", () => {
test("defaults to owner+collaborator+organization_member when field is unset (backward compat)", async () => {
const { octokit, getCaptured } = makeOctokit();
await getGithubRepositories({ octokit, config: { githubConfig: { owner: "octo" } as any } });
expect(getCaptured()?.affiliation).toBe("owner,collaborator,organization_member");
});
test("uses owner+organization_member when includeCollaboratorRepos is false", async () => {
const { octokit, getCaptured } = makeOctokit();
await getGithubRepositories({
octokit,
config: { githubConfig: { owner: "octo", includeCollaboratorRepos: false } as any },
});
expect(getCaptured()?.affiliation).toBe("owner,organization_member");
});
test("uses owner+collaborator+organization_member when includeCollaboratorRepos is true", async () => {
const { octokit, getCaptured } = makeOctokit();
await getGithubRepositories({
octokit,
config: { githubConfig: { owner: "octo", includeCollaboratorRepos: true } as any },
});
expect(getCaptured()?.affiliation).toBe("owner,collaborator,organization_member");
});
test("override forces owner+collaborator+organization_member regardless of config (used by cleanup)", async () => {
const { octokit, getCaptured } = makeOctokit();
await getGithubRepositories({
octokit,
config: { githubConfig: { owner: "octo", includeCollaboratorRepos: false } as any },
includeCollaboratorReposOverride: true,
});
expect(getCaptured()?.affiliation).toBe("owner,collaborator,organization_member");
});
test("always includes organization_member (regression guard for org-repo invisibility)", async () => {
const cases: Array<{ includeCollab?: boolean; override?: boolean }> = [
{},
{ includeCollab: true },
{ includeCollab: false },
{ override: true },
{ includeCollab: false, override: true },
];
for (const c of cases) {
const { octokit, getCaptured } = makeOctokit();
await getGithubRepositories({
octokit,
config: {
githubConfig: {
owner: "octo",
...(c.includeCollab !== undefined && { includeCollaboratorRepos: c.includeCollab }),
} as any,
},
...(c.override !== undefined && { includeCollaboratorReposOverride: c.override }),
});
const aff = String(getCaptured()?.affiliation ?? "");
expect(aff.split(",")).toContain("organization_member");
}
});
});
+18 -1
View File
@@ -235,14 +235,31 @@ export async function getGithubRepoCloneUrl({
export async function getGithubRepositories({
octokit,
config,
includeCollaboratorReposOverride,
}: {
octokit: Octokit;
config: Partial<Config>;
// Force-include collaborator repos regardless of user setting. Used by the
// cleanup service so we never mark a collab repo as orphaned just because
// the import filter is currently off.
includeCollaboratorReposOverride?: boolean;
}): Promise<GitRepo[]> {
try {
const includeCollab =
includeCollaboratorReposOverride ??
config.githubConfig?.includeCollaboratorRepos ??
true;
// Always include organization_member so repos owned by orgs the user
// belongs to are returned. Omitting it caused org repos to be invisible
// to the main sync, the scheduler, and the cleanup service (which then
// archived them on restart as if they had been deleted on GitHub).
const affiliation = includeCollab
? "owner,collaborator,organization_member"
: "owner,organization_member";
const repos = await octokit.paginate(
octokit.repos.listForAuthenticatedUser,
{ per_page: 100 },
{ per_page: 100, affiliation },
);
const skipForks = config.githubConfig?.skipForks ?? false;
+17 -4
View File
@@ -216,9 +216,21 @@ export async function updateMirrorJobProgress({
}
/**
* Finds interrupted jobs that need to be resumed with enhanced criteria
* Finds interrupted jobs that need to be resumed with enhanced criteria.
*
* `logFound` defaults to false because this function is polled from
* passive callers (`hasJobsNeedingRecovery` from the health endpoint
* and middleware checks). Logging on every poll produces log spam at
* one-line-per-poll-per-stuck-job for as long as a job stays stuck.
*
* Callers that intend to act on the result (i.e. immediately resume
* the returned jobs) should pass `logFound: true` so the surfacing
* still happens in the recovery flow.
*/
export async function findInterruptedJobs() {
export async function findInterruptedJobs(
options: { logFound?: boolean } = {}
) {
const { logFound = false } = options;
try {
// Find jobs that are marked as in-progress but haven't been updated recently
const cutoffTime = new Date();
@@ -243,8 +255,9 @@ export async function findInterruptedJobs() {
)
);
// Log details about found jobs for debugging
if (interruptedJobs.length > 0) {
// Log details about found jobs for debugging — opt-in to avoid
// spamming the log when called from periodic passive checks.
if (logFound && interruptedJobs.length > 0) {
console.log(`Found ${interruptedJobs.length} interrupted jobs:`);
interruptedJobs.forEach(job => {
const lastCheckpoint = job.lastCheckpoint ? new Date(job.lastCheckpoint).toISOString() : 'never';
@@ -0,0 +1,128 @@
/**
* Regression test for the "interrupted jobs never resume after
* startup" orchestration bug.
*
* Symptom (before this fix):
* - Server starts cleanly. Middleware runs initial recovery pass,
* finds no interrupted jobs, sets `recoveryAttempted = true` and
* `recoveryInitialized = true`.
* - User triggers a sync at T=N (well after startup). The sync
* creates a `mirrorJobs` row with `inProgress=true`.
* - The sync fails mid-flight (deadlock retry, network blip,
* container restart of an upstream service, etc.) and never
* reaches the resume codepath, so the row stays
* `inProgress=true` with no checkpoint.
* - `findInterruptedJobs` (called periodically from the health
* endpoint via `hasJobsNeedingRecovery`) detects it and logs
* `Found 1 interrupted jobs:` on every poll.
* - But the resumer (`resumeInterruptedJob`) is only invoked from
* `initializeRecovery`, which is gated behind the
* once-per-process `!recoveryAttempted` check in
* `src/middleware.ts`. That check is false after startup, so the
* resumer NEVER fires again. The job is stuck forever.
*
* Root cause: the middleware gate was symmetric "skip recovery if
* we've ever attempted it" — but it should have been "always
* re-evaluate; only the recovery routine's own 5-minute throttle
* (`skipIfRecentAttempt` inside `initializeRecovery`) prevents
* thrashing".
*
* Secondary issue: `findInterruptedJobs` logged unconditionally on
* every call, even from passive checks like `hasJobsNeedingRecovery`,
* producing log spam at one line per poll per stuck job.
*
* This test asserts on the *structure* of the source rather than
* invoking the middleware, because exercising the middleware path
* requires a full Astro request pipeline with heavy mocks. See
* `gitea-mirror-failure-recovery.test.ts` and
* `gitea-issue-dedup-on-retry.test.ts` for the same convention.
*/
import { describe, test, expect } from "bun:test";
import { readFileSync } from "node:fs";
import { join } from "node:path";
const MIDDLEWARE_SRC = readFileSync(
join(import.meta.dir, "../middleware.ts"),
"utf8"
);
const HELPERS_SRC = readFileSync(
join(import.meta.dir, "helpers.ts"),
"utf8"
);
const RECOVERY_SRC = readFileSync(
join(import.meta.dir, "recovery.ts"),
"utf8"
);
describe("orchestrator: resume interrupted jobs after startup", () => {
test("middleware no longer gates recovery behind once-per-process `recoveryAttempted`", () => {
// The old gate looked like:
// if (!recoveryInitialized && !recoveryAttempted) {
// recoveryAttempted = true;
// ...
// }
// Once both flags flipped on the first request, recovery never
// ran again — even if jobs got stuck mid-runtime.
expect(
/\brecoveryAttempted\b/.test(MIDDLEWARE_SRC),
"the `recoveryAttempted` once-per-process flag must be removed " +
"from middleware.ts so post-startup interruptions can recover"
).toBe(false);
});
test("middleware uses an in-flight latch (not a one-shot gate) for runtime safety", () => {
// The replacement uses `recoveryInFlight` as a per-request
// mutex — set true at the start, set false in `finally`. The
// actual throttle (5-minute "recent attempt") lives inside
// `initializeRecovery()` in recovery.ts, which is the right
// place for it.
expect(
/\brecoveryInFlight\b/.test(MIDDLEWARE_SRC),
"middleware should use `recoveryInFlight` as the in-flight latch"
).toBe(true);
expect(
/recoveryInFlight\s*=\s*false/.test(MIDDLEWARE_SRC) &&
/\bfinally\s*\{[\s\S]*?recoveryInFlight\s*=\s*false[\s\S]*?\}/.test(
MIDDLEWARE_SRC
),
"the in-flight latch must be released in a `finally` block " +
"so an exception during recovery doesn't permanently jam the latch"
).toBe(true);
});
test("findInterruptedJobs logging is opt-in (default off) to stop poll spam", () => {
// Active recovery callers (initializeRecovery) opt in by passing
// { logFound: true }; passive checks (hasJobsNeedingRecovery,
// health endpoint, etc.) default to silent.
expect(
/export async function findInterruptedJobs\(\s*options[^)]*\)/.test(
HELPERS_SRC
),
"findInterruptedJobs should accept an options object"
).toBe(true);
expect(
/logFound\s*=\s*false/.test(HELPERS_SRC),
"the `logFound` option should default to false " +
"so periodic passive checks (e.g. hasJobsNeedingRecovery) " +
"don't spam the log on every poll"
).toBe(true);
expect(
/if\s*\(\s*logFound\s*&&\s*interruptedJobs\.length\s*>\s*0\s*\)/.test(
HELPERS_SRC
),
"the `Found N interrupted jobs` log must be gated by `logFound`"
).toBe(true);
});
test("active recovery path opts in to per-job logging", () => {
// Without this, the actual recovery cycle would also be silent
// — operators need to see which jobs are being resumed.
expect(
/findInterruptedJobs\(\s*\{\s*logFound:\s*true\s*\}\s*\)/.test(
RECOVERY_SRC
),
"initializeRecovery() must pass { logFound: true } to findInterruptedJobs " +
"so the active recovery cycle still logs which jobs it's working on"
).toBe(true);
});
});
+3 -2
View File
@@ -121,8 +121,9 @@ export async function initializeRecovery(options: {
// Clean up stale jobs first
await cleanupStaleJobs();
// Find interrupted jobs
const interruptedJobs = await findInterruptedJobs();
// Find interrupted jobs (with per-job logging — this is the
// active recovery path that will immediately try to resume them)
const interruptedJobs = await findInterruptedJobs({ logFound: true });
if (interruptedJobs.length === 0) {
console.log('No interrupted jobs found.');
+5 -2
View File
@@ -33,9 +33,12 @@ async function identifyOrphanedRepositories(config: any): Promise<any[]> {
let githubApiAccessible = true;
try {
// Fetch GitHub data
// Fetch GitHub data. Always include collaborator repos here regardless
// of the user's import filter, otherwise repos previously mirrored as a
// collaborator would be flagged as orphaned and archived/deleted as soon
// as the user disables the filter.
const [basicAndForkedRepos, starredRepos] = await Promise.all([
getGithubRepositories({ octokit, config }),
getGithubRepositories({ octokit, config, includeCollaboratorReposOverride: true }),
config.githubConfig?.includeStarred
? getGithubStarredRepositories({ octokit, config })
: Promise.resolve([]),
+45
View File
@@ -58,6 +58,51 @@ describe("Scheduler Service - Ignored Repository Handling", () => {
expect(shouldMirrorRepository(oldSyncedRepo)).toBe(true);
});
test("auto-mirror filter respects autoMirror and autoMirrorStarred independently", () => {
// Mirrors the inline filter at scheduler-service.ts L228-233 / L609-614:
// a repo is "starred from another owner" iff isStarred && owner !== githubOwner.
// Such repos are gated by autoMirrorStarred; everything else is gated by autoMirror.
const githubOwner = "Alice".toLowerCase();
const filterRepos = (
repos: Array<{ name: string; isStarred: boolean; owner: string }>,
autoMirror: boolean,
autoMirrorStarred: boolean,
) =>
repos.filter(repo => {
const isStarredFromOther = repo.isStarred && repo.owner.toLowerCase() !== githubOwner;
return isStarredFromOther ? autoMirrorStarred : autoMirror;
});
// "ALICE" tests case-insensitive owner match — GitHub usernames are case-insensitive,
// so a self-starred repo stored with different casing must still count as owned.
const repos = [
{ name: "owned-repo", isStarred: false, owner: "alice" },
{ name: "self-starred", isStarred: true, owner: "ALICE" },
{ name: "starred-from-bob", isStarred: true, owner: "bob" },
];
// Both off: nothing mirrors
expect(filterRepos(repos, false, false).map(r => r.name)).toEqual([]);
// Only autoMirror: owned + self-starred, not third-party stars
expect(filterRepos(repos, true, false).map(r => r.name)).toEqual([
"owned-repo",
"self-starred",
]);
// Only autoMirrorStarred: just third-party stars (the bug fix — used to be empty)
expect(filterRepos(repos, false, true).map(r => r.name)).toEqual([
"starred-from-bob",
]);
// Both on: everything
expect(filterRepos(repos, true, true).map(r => r.name)).toEqual([
"owned-repo",
"self-starred",
"starred-from-bob",
]);
});
test("should validate all repository status enum values", () => {
const validStatuses = [
"imported",
+34 -30
View File
@@ -203,10 +203,14 @@ async function runScheduledSync(config: any): Promise<void> {
}
}
// Auto-mirror: Mirror imported/pending/failed repositories if enabled
if (scheduleConfig.autoMirror) {
// Auto-mirror: Mirror imported/pending/failed repositories if enabled.
// autoMirror covers owned repos; autoMirrorStarred covers starred repos from other owners.
// Either flag on its own is enough to enter this phase.
const autoMirrorOwned = !!scheduleConfig.autoMirror;
const autoMirrorStarred = !!config.githubConfig?.autoMirrorStarred;
if (autoMirrorOwned || autoMirrorStarred) {
try {
console.log(`[Scheduler] Auto-mirror enabled - checking for repositories to mirror for user ${userId}...`);
console.log(`[Scheduler] Auto-mirror enabled (owned=${autoMirrorOwned}, starred=${autoMirrorStarred}) - checking for repositories to mirror for user ${userId}...`);
let reposNeedingMirror = await db
.select()
.from(repositories)
@@ -221,17 +225,16 @@ async function runScheduledSync(config: any): Promise<void> {
)
);
// Filter out starred repos from auto-mirror when autoMirrorStarred is disabled
if (!config.githubConfig?.autoMirrorStarred) {
const githubOwner = config.githubConfig?.owner || '';
const beforeCount = reposNeedingMirror.length;
reposNeedingMirror = reposNeedingMirror.filter(
repo => !repo.isStarred || repo.owner === githubOwner
);
const skippedCount = beforeCount - reposNeedingMirror.length;
if (skippedCount > 0) {
console.log(`[Scheduler] Skipped ${skippedCount} starred repositories from auto-mirror (autoMirrorStarred is disabled)`);
}
const githubOwner = (config.githubConfig?.owner || '').toLowerCase();
const beforeCount = reposNeedingMirror.length;
reposNeedingMirror = reposNeedingMirror.filter(repo => {
// GitHub usernames are case-insensitive; lowercase both sides to avoid misclassifying self-starred repos.
const isStarredFromOther = repo.isStarred && repo.owner.toLowerCase() !== githubOwner;
return isStarredFromOther ? autoMirrorStarred : autoMirrorOwned;
});
const skippedCount = beforeCount - reposNeedingMirror.length;
if (skippedCount > 0) {
console.log(`[Scheduler] Skipped ${skippedCount} repositories from auto-mirror (autoMirror=${autoMirrorOwned}, autoMirrorStarred=${autoMirrorStarred})`);
}
if (reposNeedingMirror.length > 0) {
@@ -574,10 +577,12 @@ async function performInitialAutoStart(): Promise<void> {
continue;
}
// Step 2: Trigger mirror for all repositories that need mirroring
// Only auto-mirror if autoMirror is enabled in schedule config
if (!config.scheduleConfig?.autoMirror) {
console.log(`[Scheduler] Step 2: Skipping initial mirror - autoMirror is disabled for user ${config.userId}`);
// Step 2: Trigger mirror for all repositories that need mirroring.
// autoMirror covers owned repos; autoMirrorStarred covers starred repos from other owners.
const autoMirrorOwned = !!config.scheduleConfig?.autoMirror;
const autoMirrorStarred = !!config.githubConfig?.autoMirrorStarred;
if (!autoMirrorOwned && !autoMirrorStarred) {
console.log(`[Scheduler] Step 2: Skipping initial mirror - autoMirror and autoMirrorStarred are both disabled for user ${config.userId}`);
// Still update schedule config timestamps
const currentTime2 = new Date();
@@ -587,7 +592,7 @@ async function performInitialAutoStart(): Promise<void> {
continue;
}
console.log(`[Scheduler] Step 2: Triggering mirror for repositories that need mirroring...`);
console.log(`[Scheduler] Step 2: Triggering mirror for repositories that need mirroring (owned=${autoMirrorOwned}, starred=${autoMirrorStarred})...`);
let reposNeedingMirror = await db
.select()
.from(repositories)
@@ -602,17 +607,16 @@ async function performInitialAutoStart(): Promise<void> {
)
);
// Filter out starred repos from auto-mirror when autoMirrorStarred is disabled
if (!config.githubConfig?.autoMirrorStarred) {
const githubOwner = config.githubConfig?.owner || '';
const beforeCount = reposNeedingMirror.length;
reposNeedingMirror = reposNeedingMirror.filter(
repo => !repo.isStarred || repo.owner === githubOwner
);
const skippedCount = beforeCount - reposNeedingMirror.length;
if (skippedCount > 0) {
console.log(`[Scheduler] Skipped ${skippedCount} starred repositories from initial auto-mirror (autoMirrorStarred is disabled)`);
}
const githubOwner = (config.githubConfig?.owner || '').toLowerCase();
const beforeCount = reposNeedingMirror.length;
reposNeedingMirror = reposNeedingMirror.filter(repo => {
// GitHub usernames are case-insensitive; lowercase both sides to avoid misclassifying self-starred repos.
const isStarredFromOther = repo.isStarred && repo.owner.toLowerCase() !== githubOwner;
return isStarredFromOther ? autoMirrorStarred : autoMirrorOwned;
});
const skippedCount = beforeCount - reposNeedingMirror.length;
if (skippedCount > 0) {
console.log(`[Scheduler] Skipped ${skippedCount} repositories from initial auto-mirror (autoMirror=${autoMirrorOwned}, autoMirrorStarred=${autoMirrorStarred})`);
}
if (reposNeedingMirror.length > 0) {
+92 -2
View File
@@ -1,6 +1,53 @@
import { expect, test } from "bun:test";
import { mapDbScheduleToUi, mapUiScheduleToDb } from "./config-mapper";
import { scheduleConfigSchema } from "@/lib/db/schema";
import {
mapDbScheduleToUi,
mapDbToUiConfig,
mapUiScheduleToDb,
mapUiToDbConfig,
} from "./config-mapper";
import { githubConfigSchema, scheduleConfigSchema } from "@/lib/db/schema";
import type {
AdvancedOptions,
GitHubConfig,
GiteaConfig,
MirrorOptions,
} from "@/types/config";
function buildMinimalUiConfigs(overrides: { includeCollaboratorRepos?: boolean } = {}) {
const githubConfig: GitHubConfig = {
username: "octo",
token: "ghp_x",
privateRepositories: false,
mirrorStarred: false,
...overrides,
};
const giteaConfig: GiteaConfig = {
url: "https://gitea.example",
username: "octo",
token: "g_x",
organization: "github-mirrors",
visibility: "public",
starredReposOrg: "starred",
preserveOrgStructure: false,
};
const mirrorOptions: MirrorOptions = {
mirrorReleases: false,
mirrorLFS: false,
mirrorMetadata: false,
metadataComponents: {
issues: false,
pullRequests: false,
labels: false,
milestones: false,
wiki: false,
},
};
const advancedOptions: AdvancedOptions = {
skipForks: false,
starredCodeOnly: false,
};
return { githubConfig, giteaConfig, mirrorOptions, advancedOptions };
}
test("mapUiScheduleToDb - builds cron from start time + frequency", () => {
const existing = scheduleConfigSchema.parse({});
@@ -34,3 +81,46 @@ test("mapDbScheduleToUi - infers clock mode for generated cron", () => {
expect(mapped.startTime).toBe("22:15");
expect(mapped.timezone).toBe("Asia/Kolkata");
});
test("includeCollaboratorRepos round-trips through UI -> DB -> UI when true", () => {
const ui = buildMinimalUiConfigs({ includeCollaboratorRepos: true });
const db = mapUiToDbConfig(
ui.githubConfig,
ui.giteaConfig,
ui.mirrorOptions,
ui.advancedOptions,
);
expect(db.githubConfig.includeCollaboratorRepos).toBe(true);
const roundTripped = mapDbToUiConfig({ githubConfig: db.githubConfig, giteaConfig: db.giteaConfig });
expect(roundTripped.githubConfig.includeCollaboratorRepos).toBe(true);
});
test("includeCollaboratorRepos round-trips through UI -> DB -> UI when false", () => {
const ui = buildMinimalUiConfigs({ includeCollaboratorRepos: false });
const db = mapUiToDbConfig(
ui.githubConfig,
ui.giteaConfig,
ui.mirrorOptions,
ui.advancedOptions,
);
expect(db.githubConfig.includeCollaboratorRepos).toBe(false);
const roundTripped = mapDbToUiConfig({ githubConfig: db.githubConfig, giteaConfig: db.giteaConfig });
expect(roundTripped.githubConfig.includeCollaboratorRepos).toBe(false);
});
test("DB row missing includeCollaboratorRepos defaults to true on read", () => {
// Existing rows from before this field existed have no value stored.
const ui = mapDbToUiConfig({ githubConfig: { owner: "octo", token: "" } });
expect(ui.githubConfig.includeCollaboratorRepos).toBe(true);
});
test("githubConfigSchema parses includeCollaboratorRepos with true default", () => {
const parsed = githubConfigSchema.parse({
owner: "octo",
type: "personal",
token: "",
});
expect(parsed.includeCollaboratorRepos).toBe(true);
});
+5
View File
@@ -50,6 +50,7 @@ export function mapUiToDbConfig(
// Map checkbox fields with proper names
includeStarred: githubConfig.mirrorStarred,
includePrivate: githubConfig.privateRepositories,
includeCollaboratorRepos: githubConfig.includeCollaboratorRepos ?? true,
includeForks: !advancedOptions.skipForks, // Note: UI has skipForks, DB has includeForks
skipForks: advancedOptions.skipForks, // Add skipForks field
includeArchived: false, // Not in UI yet, default to false
@@ -142,6 +143,7 @@ export function mapDbToUiConfig(dbConfig: any): {
username: dbConfig.githubConfig?.owner || "", // Map owner to username
token: dbConfig.githubConfig?.token || "",
privateRepositories: dbConfig.githubConfig?.includePrivate || false, // Map includePrivate to privateRepositories
includeCollaboratorRepos: dbConfig.githubConfig?.includeCollaboratorRepos ?? true,
mirrorStarred: dbConfig.githubConfig?.includeStarred || false, // Map includeStarred to mirrorStarred
starredLists: normalizeStarredLists(dbConfig.githubConfig?.starredLists),
};
@@ -246,6 +248,7 @@ export function mapUiScheduleToDb(uiSchedule: any, existing?: DbScheduleConfig):
enabled: !!uiSchedule.enabled,
interval: intervalExpression,
timezone,
autoMirror: typeof uiSchedule.autoMirror === "boolean" ? uiSchedule.autoMirror : base.autoMirror,
nextRun: scheduleChanged ? undefined : base.nextRun,
} as DbScheduleConfig;
}
@@ -264,6 +267,7 @@ export function mapDbScheduleToUi(dbSchedule: DbScheduleConfig): any {
clockFrequencyHours: 24,
startTime: "22:00",
timezone: "UTC",
autoMirror: false,
lastRun: null,
nextRun: null,
};
@@ -296,6 +300,7 @@ export function mapDbScheduleToUi(dbSchedule: DbScheduleConfig): any {
clockFrequencyHours: parsedClockSchedule?.frequencyHours ?? 24,
startTime: parsedClockSchedule?.startTime ?? "22:00",
timezone: normalizeTimezone(dbSchedule.timezone || "UTC"),
autoMirror: dbSchedule.autoMirror ?? false,
lastRun: dbSchedule.lastRun || null,
nextRun: dbSchedule.nextRun || null,
};
+33 -9
View File
@@ -17,9 +17,16 @@ function prefixAstroInternalAssetPaths(html: string, basePath: string): string {
return html.replace(ASTRO_INTERNAL_ASSET_PATH_PATTERN, `$1${basePath}/$2`);
}
// Flag to track if recovery has been initialized
// Flag to track whether the *startup* recovery pass has run. This
// only gates the post-startup chain (cleanup service, scheduler,
// etc.) and the "startup script may not have run" log line — it does
// NOT gate subsequent recovery attempts, see below.
let recoveryInitialized = false;
let recoveryAttempted = false;
// Throttle for runtime recovery retries (separate from the
// initializeRecovery() 5-minute throttle inside recovery.ts, which is
// keyed on `lastRecoveryAttempt`). This prevents one middleware
// invocation from triggering recovery while another is in flight.
let recoveryInFlight = false;
let cleanupServiceStarted = false;
let schedulerServiceStarted = false;
let repositoryCleanupServiceStarted = false;
@@ -118,17 +125,30 @@ export const onRequest = defineMiddleware(async (context, next) => {
}
}
// Initialize recovery system only once when the server starts
// This is a fallback in case the startup script didn't run
if (!recoveryInitialized && !recoveryAttempted) {
recoveryAttempted = true;
// Run recovery if jobs need it.
//
// The previous implementation used a once-per-process gate, so
// any mid-runtime interruption (a sync that started after boot,
// crashed mid-flight, and never got back to the resume path)
// would sit at `in_progress=true` forever — the periodic detector
// kept finding it, but the resumer never re-fired. This block
// now re-evaluates on every request, gated by `recoveryInFlight`
// (per-process) plus the 5-minute throttle inside
// `initializeRecovery()` (which prevents thrashing if a resume
// cycle keeps failing).
if (!recoveryInFlight) {
recoveryInFlight = true;
try {
// Check if recovery is actually needed before attempting
const needsRecovery = await hasJobsNeedingRecovery();
if (needsRecovery) {
console.log('⚠️ Middleware detected jobs needing recovery (startup script may not have run)');
if (!recoveryInitialized) {
console.log('⚠️ Middleware detected jobs needing recovery (startup script may not have run)');
} else {
console.log('⚠️ Middleware detected jobs needing recovery mid-run (sync interrupted after startup)');
}
console.log('Attempting recovery from middleware...');
// Run recovery with a shorter timeout since this is during request handling
@@ -148,7 +168,9 @@ export const onRequest = defineMiddleware(async (context, next) => {
} else {
console.log('⚠️ Middleware recovery completed with some issues');
}
} else {
} else if (!recoveryInitialized) {
// Only log this on the first request; otherwise we'd spam
// it on every request.
console.log('✅ No recovery needed (startup script likely handled it)');
}
@@ -161,7 +183,9 @@ export const onRequest = defineMiddleware(async (context, next) => {
const status = getRecoveryStatus();
console.log('Recovery status:', status);
recoveryInitialized = true; // Mark as attempted to avoid retries
recoveryInitialized = true;
} finally {
recoveryInFlight = false;
}
}
+2
View File
@@ -36,6 +36,7 @@ export interface ScheduleConfig {
clockFrequencyHours?: number;
startTime?: string;
timezone?: string;
autoMirror?: boolean;
lastRun?: Date;
nextRun?: Date;
}
@@ -60,6 +61,7 @@ export interface GitHubConfig {
username: string;
token: string;
privateRepositories: boolean;
includeCollaboratorRepos?: boolean;
mirrorStarred: boolean;
starredLists?: string[];
starredDuplicateStrategy?: DuplicateNameStrategy;