Compare commits

...

7 Commits

Author SHA1 Message Date
Arunavo Ray 1b84c75a97 chore: bump version to 3.18.0 2026-06-13 08:17:20 +05:30
ARUNAVO RAY 0b6b6b76bf feat(github): add skipPersonalRepos toggle to mirror only org repos (#304) (#320)
- Add `skipPersonalRepos: z.boolean().default(false)` to githubConfigSchema
- Filter out user-owned repos in getGithubRepositories when flag is true
- Wire ONLY_MIRROR_ORGS env var to skipPersonalRepos in env-config-loader
- Add checkbox UI in GitHubMirrorSettings Filtering & Behavior section
- Round-trip skipPersonalRepos through config-mapper (UI ↔ DB)
- Add skipPersonalRepos to AdvancedOptions TypeScript type
- Mark include/exclude arrays in configSchema as unused/reserved
- Update ENVIRONMENT_VARIABLES.md to document ONLY_MIRROR_ORGS effect
2026-06-13 08:00:50 +05:30
ARUNAVO RAY 7610a614da fix: scheduler auto-start gate, backup clone URL, cancel-pending action, actionable 405 (#319)
* fix(scheduler): make enabled flag authoritative for auto-start

checkAutoStartConfiguration() and performInitialAutoStart() previously
used `scheduleEnabled || hasMirrorInterval`, allowing a configured
GITEA_MIRROR_INTERVAL to trigger boot-time auto-start even after the
user disabled scheduling via the UI toggle.

env-config-loader already writes scheduleConfig.enabled=true when
GITEA_MIRROR_INTERVAL is set at container startup, so the interval is
a timing detail, not an enable signal. The documented env-var contract
is preserved: GITEA_MIRROR_INTERVAL at boot → env-config-loader sets
enabled=true → auto-start fires. But a later UI disable now sticks.

Add a focused unit test for the gate logic.

* fix(backup): always derive clone URL from user-configured Gitea URL

The pre-sync backup preferred repoInfo.clone_url, which reflects
Gitea's ROOT_URL setting. In Tailscale MagicDNS deployments (and any
setup where ROOT_URL is an external address), this URL is unreachable
from the app itself, causing bundle backup to fail.

Always build the clone URL as:
  ${config.giteaConfig.url.trimEnd('/')}/${owner}/${repo}.git

This matches the URL the app already uses for all other Gitea API
calls and is guaranteed reachable.

* feat(jobs): cancel-pending endpoint + fix misleading Delete All copy

Add POST /api/job/cancel-pending that sets the current user's
repositories with status "imported" or "failed" to "ignored",
preventing the scheduler from re-queuing them. In-flight "mirroring"
rows are left alone. Returns the count and logs one activity entry.

Fix the "Delete All Activities" dialog to clearly state it only clears
the history log and does not stop pending work. Rename button/title to
"Clear History" so intent is unambiguous.

Add a "Stop Pending Mirrors" button (StopCircle icon, amber) in both
mobile and desktop activity log toolbars, with a confirmation dialog
explaining repos are set to Ignored and can be re-enabled from the
Repositories page.

* fix(sync): actionable 405 error for non-pull-mirror repos

Gitea returns HTTP 405 with an empty body when the target repository is
no longer a pull mirror — e.g. the mirror was auto-disabled by Gitea or
the repository lost its mirror state after a manual edit.

Previously this fell through to the generic error handler which stored
the raw HttpError message (often empty) giving the user no guidance.

Now a 405 response is caught alongside the existing 400 handler and
sets the repository to "failed" with an actionable error message:

  "Gitea reports this repository is not a pull mirror (HTTP 405).
  In Gitea check Settings → Mirror Settings; if the mirror section is
  missing, delete the repository in Gitea and re-mirror it from
  gitea-mirror."

The same message is written to the activity log for visibility in the
dashboard.
2026-06-13 08:00:47 +05:30
ARUNAVO RAY c28dcc209f fix(releases): stop delete/recreate cycle on permanent order mismatch (#310) (#318)
Root cause (Theory A): the `needsRecreation` check compared GitHub
published_at-based expected indices against Gitea's API order. Gitea mirror
repos sort releases by tag-commit date, which can permanently disagree with
published_at order (e.g. unaconfig_dart v0.1.0 published after v0.1.1 but
tagged before). This made `currentExpectedIdx < nextExpectedIdx` evaluate
true on every sync, triggering delete-all-and-recreate forever — spamming
Gitea's activity feed with "released X" events (#310).

Fix: replace the destructive order-check machinery with set-based
reconciliation via `classifyReleasesForReconciliation`. Releases are
created when missing in Gitea and skipped (or PATCH-updated if content
drifted) when already present. No deletions are ever triggered by ordering.
Retain the existing release-limit trimming (retention cleanup) unchanged.

Also removes the 1-second per-release delay that was only needed for the
creation-order dance, significantly speeding up initial mirrors.

Adds unit tests covering: normal ordered repos, the unaconfig_dart inversion
fixture, missing→create, present→skip, and edge cases.
2026-06-13 08:00:44 +05:30
ARUNAVO RAY 40ee3cbc44 fix(mirror): reuse existing same-source mirrors instead of creating suffixed duplicates (#315) (#317)
Starred (and other) repos duplicated on every re-mirror (starred/Repo,
Repo-owner, Repo-owner-1, ...) because the existence check only asked
"does a repo with this name exist?" and never "is the existing repo a
mirror of THIS same source?". The repo's own prior mirror counted as a
collision, so generateUniqueRepoName bumped to the next suffix each run,
repointing mirroredLocation at the newest copy. Under a single re-call,
3 concurrent/retried jobs each computed a DIFFERENT suffixed name, so the
location-based in-flight guard never matched and the race produced extra
copies.

Fix (source-identity aware):
- New shared helper src/lib/utils/mirror-source-match.ts:
  - normalizeCloneUrl / cloneUrlsMatch: credential-, .git-, slash- and
    host-case-insensitive clone URL comparison.
  - isMirrorOfSource: a Gitea repo is "ours" only if it is a mirror AND
    its original_url matches this repo's source.
  - findExistingMirror: resolves an existing same-source mirror via the
    recorded mirroredLocation first (survives strategy changes — #309),
    then the base candidate name.
  - classifyCandidateName: pure available/reusable/taken decision.
- gitea-enhanced: export GiteaRepoInfo and add original_url (Gitea's
  recorded migration source) for source matching.
- Both create paths (mirrorGithubRepoToGitea, mirrorGitHubRepoToGiteaOrg):
  run findExistingMirror BEFORE name generation; on a hit, reuse that
  location and route into the existing "already mirrored" handling rather
  than calling generateUniqueRepoName. Names now converge under
  concurrency so the in-flight guard becomes effective.
- generateUniqueRepoName is now source-aware: an occupied name held by a
  mirror of the SAME source is reused (no suffix); suffixing only happens
  on a genuine different-source collision, preserving #95/#236 behavior.
  The per-user DB claim check is retained so two users mirroring the same
  source into a shared org stay separated.
- Phantom-fork guard (#309): the existingRepoInfo.mirror branches now
  verify same-source before marking "mirrored"; on mismatch they fall
  through to unique-name generation and create a separate mirror.
- Scheduler: a `failed` repo whose mirroredLocation still resolves to a
  live same-source mirror is routed to syncGiteaRepo instead of re-create,
  breaking the failed-metadata re-create loop cheaply.
- Remove dead src/lib/starred-repos-handler.ts (zero importers across all
  git history); its correct base-name/.mirror reuse logic now lives in the
  shared helper.

Tests: src/lib/utils/mirror-source-match.test.ts (30 cases) covers URL
normalization, reuse at base name, reuse via mirroredLocation across a
strategy change, genuine different-source collision (suffix), phantom
fork, stale mirroredLocation fallback, per-user DB-claim separation, and
the suffix-vs-reuse classification. Full suite: 319 pass, 0 fail.
2026-06-13 08:00:41 +05:30
Arunavo Ray 699a5771f5 chore: bump version to 3.17.1 2026-06-05 18:42:11 +05:30
ARUNAVO RAY e862714d6a fix(db): self-heal sso_providers duplicate-column crash on upgrade (#312) (#313)
Migration 0013 runs as a single transaction that rebuilds `organizations`
and then `ALTER TABLE sso_providers ADD saml_config` / `ADD domain_verified`.
On instances where those columns were already created outside Drizzle (declared
in schema.ts and added via db:push / an SSO-register round-trip on an
intermediate build), the ADD throws "duplicate column name: saml_config". That
rolls back the entire 0013 transaction, so 0013 is never recorded in
`__drizzle_migrations` and is retried — failing identically — on every boot,
crash-looping the server.

Add a pre-migrate repair (mirroring the existing repairFailedMigrations() for
the 0009 case): when 0013 is unrecorded but the columns already exist, preserve
any real SAML provider config, drop the stranded columns so the canonical 0013
runs in full (organizations rebuild included), then restore the preserved
values once the columns are re-added. No-op on fresh installs, clean upgrades,
and already-migrated databases.

This lets affected instances recover automatically on the next boot after
upgrading — no manual SQLite surgery required.

- src/lib/db/migration-repairs.ts: repairDuplicateSsoColumns + restoreSsoDataAfter0013
- src/lib/db/index.ts: wire both around migrate()
- scripts/validate-migrations.ts: cover the broken-upgrade + data-preservation path
2026-06-05 18:41:46 +05:30
23 changed files with 1719 additions and 530 deletions
+1 -1
View File
@@ -114,7 +114,7 @@ Standard GitHub Enterprise Cloud on `github.com` works with the default — no o
|----------|-------------|---------|---------|
| `MIRROR_ORGANIZATIONS` | Mirror organization repositories | `false` | `true`, `false` |
| `PRESERVE_ORG_STRUCTURE` | Preserve GitHub organization structure in Gitea | `false` | `true`, `false` |
| `ONLY_MIRROR_ORGS` | Only mirror organization repos (skip personal) | `false` | `true`, `false` |
| `ONLY_MIRROR_ORGS` | Only mirror organization repos (skip personal); sets `skipPersonalRepos: true` in GitHub config | `false` | `true`, `false` |
| `MIRROR_STRATEGY` | Repository organization strategy | `preserve` | `preserve`, `single-org`, `flat-user`, `mixed` |
### Advanced Settings
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "gitea-mirror",
"type": "module",
"version": "3.17.0",
"version": "3.18.0",
"engines": {
"bun": ">=1.2.9"
},
+102 -1
View File
@@ -3,6 +3,10 @@
import { Database } from "bun:sqlite";
import { readFileSync } from "fs";
import path from "path";
import {
repairDuplicateSsoColumns,
restoreSsoDataAfter0013,
} from "../src/lib/db/migration-repairs";
type JournalEntry = {
idx: number;
@@ -290,6 +294,100 @@ function verify0013Migration(db: any) {
assert(normName.dflt_value === null, `Expected normalized_name to have no default, got ${normName.dflt_value}`);
}
const MIGRATION_0012_TIMESTAMP = 1774062000000;
const MIGRATION_0013_TIMESTAMP = 1780377747526;
/**
* Reproduce the issue #312 crash state — sso_providers already carries
* saml_config / domain_verified before migration 0013 runs (stranded on an
* intermediate build), with __drizzle_migrations recorded only through 0012 —
* and verify repairDuplicateSsoColumns()/restoreSsoDataAfter0013() let the
* canonical 0013 run while preserving real SAML provider data.
*/
function validateBroken0013Repair() {
const migration0013 = migrations.find((m) => m.entry.tag === "0013_slim_galactus");
if (!migration0013) return; // 0013 not present (shouldn't happen) — nothing to test.
const db = new Database(":memory:");
try {
runMigrations(db, migrations.slice(0, 13)); // 0000-0012
// A real upgraded instance has a __drizzle_migrations table recorded
// through 0012 but not 0013.
db.run(
"CREATE TABLE IF NOT EXISTS `__drizzle_migrations` (id INTEGER PRIMARY KEY AUTOINCREMENT, hash text NOT NULL, created_at numeric)",
);
db.run("INSERT INTO `__drizzle_migrations` (hash, created_at) VALUES ('through-0012', ?)", [
MIGRATION_0012_TIMESTAMP,
]);
// Stranded columns from the intermediate build.
db.run("ALTER TABLE sso_providers ADD saml_config text");
db.run("ALTER TABLE sso_providers ADD domain_verified integer DEFAULT true NOT NULL");
db.run("INSERT INTO users (id, email, username, name) VALUES ('u1', 'u1@example.com', 'u1', 'User One')");
const samlJson = '{"entryPoint":"https://idp.example.com/sso","cert":"ABC123"}';
db.run(
"INSERT INTO sso_providers (id, issuer, domain, oidc_config, user_id, provider_id, saml_config, domain_verified) VALUES ('oidc1', 'https://idp', 'a.com', '{}', 'u1', 'p-oidc', NULL, 1)",
);
db.run(
"INSERT INTO sso_providers (id, issuer, domain, oidc_config, user_id, provider_id, saml_config, domain_verified) VALUES ('saml1', 'https://idp', 'b.com', '{}', 'u1', 'p-saml', ?, 1)",
[samlJson],
);
db.run(
"INSERT INTO sso_providers (id, issuer, domain, oidc_config, user_id, provider_id, saml_config, domain_verified) VALUES ('unv1', 'https://idp', 'c.com', '{}', 'u1', 'p-unv', NULL, 0)",
);
const preserved = repairDuplicateSsoColumns(db);
const colsAfterRepair = (db.query("PRAGMA table_info(sso_providers)").all() as TableInfoRow[]).map(
(c) => c.name,
);
assert(!colsAfterRepair.includes("saml_config"), "Expected repair to drop stranded saml_config column");
assert(
!colsAfterRepair.includes("domain_verified"),
"Expected repair to drop stranded domain_verified column",
);
const preservedIds = preserved.map((r) => r.id).sort();
assert(
preservedIds.length === 2 && preservedIds[0] === "saml1" && preservedIds[1] === "unv1",
`Expected SAML + unverified rows to be preserved, got ${JSON.stringify(preservedIds)}`,
);
// The canonical 0013 must now run without a duplicate-column error.
runMigration(db, migration0013);
restoreSsoDataAfter0013(db, preserved);
const rows = db
.query("SELECT id, saml_config, domain_verified FROM sso_providers ORDER BY id")
.all() as Array<{ id: string; saml_config: string | null; domain_verified: number }>;
const byId = Object.fromEntries(rows.map((r) => [r.id, r]));
assert(byId.oidc1.saml_config === null, "Expected OIDC provider saml_config to remain NULL");
assert(byId.oidc1.domain_verified === 1, "Expected OIDC provider domain_verified default 1");
assert(byId.saml1.saml_config === samlJson, "Expected SAML provider config to be preserved");
assert(byId.saml1.domain_verified === 1, "Expected SAML provider domain_verified preserved as 1");
assert(byId.unv1.saml_config === null, "Expected unverified provider saml_config NULL");
assert(byId.unv1.domain_verified === 0, "Expected explicit domain_verified=0 to be preserved");
// Idempotency: 0013 is now applied, so a re-run of the repair is a no-op.
db.run("INSERT INTO `__drizzle_migrations` (hash, created_at) VALUES ('through-0013', ?)", [
MIGRATION_0013_TIMESTAMP,
]);
const secondPass = repairDuplicateSsoColumns(db);
assert(secondPass.length === 0, "Expected repair to no-op once migration 0013 is recorded");
const colsAfterSecondPass = (
db.query("PRAGMA table_info(sso_providers)").all() as TableInfoRow[]
).map((c) => c.name);
assert(
colsAfterSecondPass.includes("saml_config") && colsAfterSecondPass.includes("domain_verified"),
"Expected columns to remain intact on the no-op second pass",
);
} finally {
db.close();
}
}
const latestUpgradeFixtures: Record<string, UpgradeFixture> = {
"0009_nervous_tyger_tiger": {
seed: seedPre0009Database,
@@ -361,8 +459,11 @@ function validateMigrations() {
upgradeDb.close();
}
// Exercise the runtime repair for the issue #312 duplicate-column crash.
validateBroken0013Repair();
console.log(
`Validated ${migrations.length} migrations from scratch and upgrade path for ${latestMigration.entry.tag}.`,
`Validated ${migrations.length} migrations from scratch and upgrade path for ${latestMigration.entry.tag}, plus the #312 SSO-column repair.`,
);
}
+97 -7
View File
@@ -1,6 +1,6 @@
import { useCallback, useEffect, useState, useRef } from 'react';
import { Button } from '@/components/ui/button';
import { ChevronDown, Download, RefreshCw, Search, Trash2, Filter } from 'lucide-react';
import { ChevronDown, Download, RefreshCw, Search, Trash2, Filter, StopCircle } from 'lucide-react';
import {
DropdownMenu,
DropdownMenuContent,
@@ -85,6 +85,8 @@ export function ActivityLog() {
const [activities, setActivities] = useState<MirrorJobWithKey[]>([]);
const [isInitialLoading, setIsInitialLoading] = useState(false);
const [showCleanupDialog, setShowCleanupDialog] = useState(false);
const [showCancelPendingDialog, setShowCancelPendingDialog] = useState(false);
const [isCancelPendingLoading, setIsCancelPendingLoading] = useState(false);
// Ref to track if component is mounted to prevent state updates after unmount
const isMountedRef = useRef(true);
@@ -354,6 +356,40 @@ export function ActivityLog() {
setShowCleanupDialog(false);
};
const confirmCancelPending = async () => {
if (!user?.id) return;
try {
setIsCancelPendingLoading(true);
setShowCancelPendingDialog(false);
const response = await fetch(withBase('/api/job/cancel-pending'), {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
});
if (!response.ok) {
const errorData = await response.json().catch(() => ({ error: 'Unknown error occurred' }));
throw new Error(errorData.error || `HTTP ${response.status}: ${response.statusText}`);
}
const res = await response.json();
if (res.success) {
toast.success(res.message);
// Refresh to show the new activity log entry
await fetchActivities(false);
} else {
showErrorToast(res.error || 'Failed to cancel pending mirrors.', toast);
}
} catch (error) {
console.error('Error cancelling pending mirrors:', error);
showErrorToast(error, toast);
} finally {
setIsCancelPendingLoading(false);
}
};
// Check if any filters are active
const hasActiveFilters = !!(filter.status || filter.type || filter.name);
const activeFilterCount = [filter.status, filter.type, filter.name].filter(Boolean).length;
@@ -552,11 +588,22 @@ export function ActivityLog() {
<RefreshCw className="h-4 w-4" />
</Button>
<Button
variant="outline"
size="icon"
onClick={() => setShowCancelPendingDialog(true)}
title="Stop pending mirrors"
className="text-amber-600 hover:text-amber-600 h-10 w-10 shrink-0"
disabled={isCancelPendingLoading}
>
<StopCircle className="h-4 w-4" />
</Button>
<Button
variant="outline"
size="icon"
onClick={handleCleanupClick}
title="Delete all activities"
title="Clear activity history"
className="text-destructive hover:text-destructive h-10 w-10 shrink-0"
>
<Trash2 className="h-4 w-4" />
@@ -683,12 +730,24 @@ export function ActivityLog() {
<RefreshCw className="h-4 w-4" />
</Button>
{/* cleanup all activities */}
{/* stop pending mirrors */}
<Button
variant="outline"
size="icon"
onClick={() => setShowCancelPendingDialog(true)}
title="Stop pending mirrors"
className="text-amber-600 hover:text-amber-600 h-10 w-10"
disabled={isCancelPendingLoading}
>
<StopCircle className="h-4 w-4" />
</Button>
{/* clear activity history */}
<Button
variant="outline"
size="icon"
onClick={handleCleanupClick}
title="Delete all activities"
title="Clear activity history"
className="text-destructive hover:text-destructive h-10 w-10"
>
<Trash2 className="h-4 w-4" />
@@ -709,9 +768,12 @@ export function ActivityLog() {
<Dialog open={showCleanupDialog} onOpenChange={setShowCleanupDialog}>
<DialogContent>
<DialogHeader>
<DialogTitle>Delete All Activities</DialogTitle>
<DialogTitle>Clear Activity History</DialogTitle>
<DialogDescription>
Are you sure you want to delete ALL activities? This action cannot be undone and will remove all mirror jobs and events from the database.
This clears the activity <strong>history log</strong> (mirror job records and events) it does not stop
any pending or in-progress mirrors. Repositories keep their current status and the scheduler
will continue to pick up pending work. To stop pending mirrors, use the
&ldquo;Stop Pending Mirrors&rdquo; button instead.
</DialogDescription>
</DialogHeader>
<DialogFooter>
@@ -723,7 +785,35 @@ export function ActivityLog() {
onClick={confirmCleanup}
disabled={isInitialLoading}
>
{isInitialLoading ? 'Deleting...' : 'Delete All Activities'}
{isInitialLoading ? 'Clearing...' : 'Clear History'}
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
{/* cancel pending mirrors confirmation dialog */}
<Dialog open={showCancelPendingDialog} onOpenChange={setShowCancelPendingDialog}>
<DialogContent>
<DialogHeader>
<DialogTitle>Stop Pending Mirrors</DialogTitle>
<DialogDescription>
This sets all repositories with status <strong>Imported</strong> or <strong>Failed</strong> to{' '}
<strong>Ignored</strong>, preventing the scheduler from mirroring them automatically.
Repositories that are currently mirroring are not affected.{' '}
You can re-enable individual repositories from the Repositories page.
</DialogDescription>
</DialogHeader>
<DialogFooter>
<Button variant="outline" onClick={() => setShowCancelPendingDialog(false)}>
Cancel
</Button>
<Button
variant="default"
className="bg-amber-600 hover:bg-amber-700"
onClick={confirmCancelPending}
disabled={isCancelPendingLoading}
>
{isCancelPendingLoading ? 'Stopping...' : 'Stop Pending Mirrors'}
</Button>
</DialogFooter>
</DialogContent>
@@ -927,6 +927,26 @@ export function GitHubMirrorSettings({
</p>
</div>
</div>
<div className="flex items-start space-x-3">
<Checkbox
id="skip-personal-repos"
checked={advancedOptions.skipPersonalRepos ?? false}
onCheckedChange={(checked) => handleAdvancedChange('skipPersonalRepos', !!checked)}
/>
<div className="space-y-0.5 flex-1">
<Label
htmlFor="skip-personal-repos"
className="text-sm font-normal cursor-pointer flex items-center gap-2"
>
<Users className="h-3.5 w-3.5" />
Skip personal repositories (only mirror organization repos)
</Label>
<p className="text-xs text-muted-foreground">
Exclude repositories owned by your personal GitHub account; only mirror repos belonging to organizations
</p>
</div>
</div>
</div>
</div>
</div>
+9
View File
@@ -3,6 +3,7 @@ import { drizzle } from "drizzle-orm/bun-sqlite";
import fs from "fs";
import path from "path";
import { migrate } from "drizzle-orm/bun-sqlite/migrator";
import { repairDuplicateSsoColumns, restoreSsoDataAfter0013 } from "./migration-repairs";
// Skip database initialization in test environment
let db: ReturnType<typeof drizzle>;
@@ -95,9 +96,17 @@ if (process.env.NODE_ENV !== "test") {
// Fix any migrations that were recorded but actually failed (e.g. v3.13.0 bug)
repairFailedMigrations();
// Fix the v3.17.0 duplicate-column crash: reconcile stranded sso_providers
// columns so migration 0013 can run (see #312). Returns data to re-apply
// once 0013 has re-added the columns.
const preservedSsoData = repairDuplicateSsoColumns(sqlite);
// Run migrations using Drizzle migrate function
migrate(db, { migrationsFolder: "./drizzle" });
// Re-apply any SSO provider data preserved by the repair above.
restoreSsoDataAfter0013(sqlite, preservedSsoData);
console.log("✅ Database migrations completed successfully");
} catch (error) {
console.error("❌ Error running migrations:", error);
+117
View File
@@ -0,0 +1,117 @@
import type { Database } from "bun:sqlite";
/**
* Pre-migration repairs that reconcile a database into the exact shape Drizzle's
* migrator expects, so a previously-failed migration can complete on the next
* boot. These run BEFORE `migrate()` and are deliberately defensive: any failure
* is logged and swallowed so they never make a recoverable database worse.
*/
/** Migration 0013 journal timestamp (from drizzle/meta/_journal.json, idx 13). */
const MIGRATION_0013_TIMESTAMP = 1780377747526;
export type PreservedSsoRow = {
id: string;
saml_config?: string | null;
domain_verified?: number;
};
/**
* Repair the v3.17.0 (PR #307) "duplicate column name: saml_config" crash loop
* reported in issue #312.
*
* Some instances ended up with `sso_providers.saml_config` / `domain_verified`
* already present BEFORE migration 0013 ran the columns were declared in
* schema.ts and entered the DB via `db:push` or an SSO-register round-trip on an
* intermediate build, while `__drizzle_migrations` never recorded a 0013 row.
*
* Migration 0013 runs as a single transaction (organizations rebuild + the two
* `ALTER TABLE sso_providers ADD ...`). The ADD hits the pre-existing column,
* throws "duplicate column", and rolls back the ENTIRE transaction so 0013 is
* never recorded and is retried, failing identically, on every boot.
*
* This is the mirror image of the 0009 repair in index.ts (record present,
* column missing): here the column is present but the record is missing. We
* reconcile `sso_providers` back to its true pre-0013 shape so the canonical
* 0013 can run in full (the organizations rebuild MUST NOT be skipped),
* preserving any real SAML provider config across the drop/re-add.
*
* Returns the rows whose values must be re-applied by {@link restoreSsoDataAfter0013}
* once 0013 has re-added the columns. Returns an empty array when there is
* nothing to do (fresh install, clean upgrade, or genuine pre-0013 shape).
*/
export function repairDuplicateSsoColumns(sqlite: Database): PreservedSsoRow[] {
try {
const migrationsTableExists = sqlite
.query("SELECT name FROM sqlite_master WHERE type='table' AND name='__drizzle_migrations'")
.get();
// Fresh install — no migrations recorded yet, vanilla migrate() handles it.
if (!migrationsTableExists) return [];
// 0013 already recorded (clean upgrade / already healed) — nothing to do.
const alreadyApplied = sqlite
.query("SELECT 1 FROM __drizzle_migrations WHERE created_at >= ? LIMIT 1")
.get(MIGRATION_0013_TIMESTAMP);
if (alreadyApplied) return [];
const ssoExists = sqlite
.query("SELECT name FROM sqlite_master WHERE type='table' AND name='sso_providers'")
.get();
if (!ssoExists) return [];
const cols = sqlite.query("PRAGMA table_info(sso_providers)").all() as { name: string }[];
const hasSaml = cols.some((c) => c.name === "saml_config");
const hasDomainVerified = cols.some((c) => c.name === "domain_verified");
// Genuine pre-0013 shape — let migration 0013 add the columns as-is.
if (!hasSaml && !hasDomainVerified) return [];
console.log(
"🔧 Detected stranded SSO columns (migration 0013 not recorded). Reconciling sso_providers so 0013 can run...",
);
// Preserve any real data before dropping. SAML providers store JSON config
// in saml_config; domain_verified may have been explicitly set to false.
const selectCols = ["id"];
if (hasSaml) selectCols.push("saml_config");
if (hasDomainVerified) selectCols.push("domain_verified");
const preserved = sqlite
.query(`SELECT ${selectCols.join(", ")} FROM sso_providers`)
.all() as PreservedSsoRow[];
// SQLite >= 3.35 (bun:sqlite ships much newer) supports DROP COLUMN.
if (hasSaml) sqlite.run("ALTER TABLE sso_providers DROP COLUMN saml_config");
if (hasDomainVerified) sqlite.run("ALTER TABLE sso_providers DROP COLUMN domain_verified");
// Only rows whose values differ from the 0013 defaults (saml_config NULL,
// domain_verified true/1) need restoring after the columns are re-added.
return preserved.filter(
(r) => (hasSaml && r.saml_config != null) || (hasDomainVerified && r.domain_verified === 0),
);
} catch (error) {
console.warn("⚠️ SSO column repair check failed (non-fatal):", error);
return [];
}
}
/**
* Re-apply the SSO provider values preserved by {@link repairDuplicateSsoColumns}
* once migration 0013 has re-added saml_config / domain_verified with their
* defaults (saml_config NULL, domain_verified = 1). No-op when nothing was
* preserved (the common OIDC-only case).
*/
export function restoreSsoDataAfter0013(sqlite: Database, preserved: PreservedSsoRow[]): void {
if (preserved.length === 0) return;
try {
const stmt = sqlite.prepare(
"UPDATE sso_providers SET saml_config = ?, domain_verified = ? WHERE id = ?",
);
for (const r of preserved) {
stmt.run(r.saml_config ?? null, r.domain_verified ?? 1, r.id);
}
console.log(`✅ Restored ${preserved.length} preserved SSO provider value(s) after migration 0013.`);
} catch (error) {
console.warn("⚠️ Failed to restore preserved SSO data (non-fatal):", error);
}
}
+3
View File
@@ -34,6 +34,7 @@ export const githubConfigSchema = z.object({
autoMirrorStarred: z.boolean().default(false),
skipStarredIssues: z.boolean().optional(), // Deprecated: kept for backward compatibility, use starredCodeOnly instead
starredDuplicateStrategy: z.enum(["suffix", "prefix", "owner-org"]).default("suffix").optional(),
skipPersonalRepos: z.boolean().default(false),
});
export const backupStrategyEnum = z.enum([
@@ -156,7 +157,9 @@ export const configSchema = z.object({
isActive: z.boolean().default(true),
githubConfig: githubConfigSchema,
giteaConfig: giteaConfigSchema,
// Unused/reserved — stored for future glob support but not currently read
include: z.array(z.string()).default(["*"]),
// Unused/reserved — stored for future glob support but not currently read
exclude: z.array(z.string()).default([]),
scheduleConfig: scheduleConfigSchema,
cleanupConfig: cleanupConfigSchema,
+2
View File
@@ -285,6 +285,8 @@ export async function initializeConfigFromEnv(): Promise<void> {
starredCodeOnly: envConfig.github.starredCodeOnly ?? existingConfig?.[0]?.githubConfig?.starredCodeOnly ?? false,
autoMirrorStarred: envConfig.github.autoMirrorStarred ?? existingConfig?.[0]?.githubConfig?.autoMirrorStarred ?? false,
starredLists: envConfig.github.starredLists ?? existingConfig?.[0]?.githubConfig?.starredLists ?? [],
// ONLY_MIRROR_ORGS=true maps to skipPersonalRepos: true
skipPersonalRepos: envConfig.github.onlyMirrorOrgs ?? existingConfig?.[0]?.githubConfig?.skipPersonalRepos ?? false,
};
// Build Gitea config
+38 -4
View File
@@ -43,13 +43,18 @@ type SyncDependencies = {
/**
* Enhanced repository information including mirror status
*/
interface GiteaRepoInfo {
export interface GiteaRepoInfo {
id: number;
name: string;
owner: { login: string } | string;
mirror: boolean;
mirror_interval?: string;
clone_url?: string;
// Original migration source URL. Gitea/Forgejo populate this with the
// upstream clone address for migrated/mirrored repos, so it is the
// authoritative way to tell whether an existing mirror points at THIS
// GitHub source (vs. a same-named mirror of a different source).
original_url?: string;
private: boolean;
}
@@ -544,9 +549,12 @@ export async function syncGiteaRepoEnhanced({
// Create backup if strategy says so
if (shouldBackupForStrategy(backupStrategy, forcePushDetected)) {
const cloneUrl =
repoInfo.clone_url ||
`${config.giteaConfig.url.replace(/\/$/, "")}/${repoOwner}/${repoName}.git`;
// Always derive the clone URL from the user-configured Gitea URL rather
// than repoInfo.clone_url (which reflects Gitea's ROOT_URL and may be
// unreachable from the app — e.g. Tailscale MagicDNS deployments where
// ROOT_URL resolves externally but the app talks to Gitea on a private
// address).
const cloneUrl = `${config.giteaConfig.url.replace(/\/$/, "")}/${repoOwner}/${repoName}.git`;
try {
const backupResult = await createPreSyncBundleBackup({
@@ -886,6 +894,32 @@ export async function syncGiteaRepoEnhanced({
status: "failed",
});
}
} else if (syncError instanceof HttpError && syncError.status === 405) {
// Gitea returns HTTP 405 (with an empty body) when the repository is not
// a pull-mirror in its database — e.g. Gitea auto-disabled the mirror or
// the repo lost its mirror state after a manual edit.
const actionableMessage =
`Gitea reports this repository is not a pull mirror (HTTP 405). ` +
`In Gitea check Settings → Mirror Settings; if the mirror section is ` +
`missing, delete the repository in Gitea and re-mirror it from gitea-mirror.`;
await db
.update(repositories)
.set({
status: repoStatusEnum.parse("failed"),
updatedAt: new Date(),
errorMessage: actionableMessage,
})
.where(eq(repositories.id, repository.id!));
await createMirrorJob({
userId: config.userId,
repositoryId: repository.id,
repositoryName: repository.name,
message: `Sync failed: ${repository.name} is not a pull mirror in Gitea (HTTP 405)`,
details: actionableMessage,
status: "failed",
});
}
throw syncError;
}
+150
View File
@@ -0,0 +1,150 @@
/**
* Unit tests for release reconciliation logic regression for #310.
*
* Root-cause verdict: Theory A (not Theory B).
*
* The old `needsRecreation` check compared published_at-based expected indices
* against Gitea's API order (which mirrors sort by tag-commit-date, not
* published_at). For repos where published_at order permanently disagrees with
* tag-commit-date order, `currentExpectedIdx < nextExpectedIdx` evaluates true
* on every single sync, triggering delete-and-recreate forever.
*
* Discriminating evidence unaconfig_dart fixture:
* v0.1.1 published_at 2024-01-13T23:44 (earlier) expectedOrder index 0
* v0.1.0 published_at 2024-01-16T00:23 (later) expectedOrder index 1
* v0.1.0 tagged 2024-01-13T23:30 (earlier tag commit)
* v0.1.1 tagged 2024-01-13T23:42 (later tag commit)
* Gitea tag-commit order: [v0.1.1, v0.1.0] (v0.1.1 has newer commit)
* Old check: currentExpectedIdx(v0.1.1)=0 < nextExpectedIdx(v0.1.0)=1 TRUE
* needsRecreation fires on every sync, forever.
*
* Theory B (operator inversion) would fire for ALL repos with >1 release, but
* field evidence shows only ~2 of 150 repos are affected ruling it out.
*
* Fix: replaced `needsRecreation` machinery with set-based reconciliation via
* `classifyReleasesForReconciliation`. Releases are created when missing, skipped
* (or PATCH-updated if content drifted) when present. No deletions for ordering.
*/
import { describe, expect, it } from "bun:test";
import { classifyReleasesForReconciliation } from "@/lib/gitea";
describe("classifyReleasesForReconciliation", () => {
describe("normal repo — published_at order matches tag-commit order", () => {
it("creates releases missing in Gitea", () => {
const github = ["v1.0.0", "v1.1.0", "v1.2.0"];
const gitea: string[] = [];
const { toCreate, toSkip } = classifyReleasesForReconciliation(github, gitea);
expect(toCreate).toEqual(["v1.0.0", "v1.1.0", "v1.2.0"]);
expect(toSkip).toEqual([]);
});
it("skips releases already present in Gitea", () => {
const github = ["v1.0.0", "v1.1.0", "v1.2.0"];
const gitea = ["v1.0.0", "v1.1.0", "v1.2.0"];
const { toCreate, toSkip } = classifyReleasesForReconciliation(github, gitea);
expect(toCreate).toEqual([]);
expect(toSkip).toEqual(["v1.0.0", "v1.1.0", "v1.2.0"]);
});
it("creates missing releases while skipping existing ones", () => {
const github = ["v1.0.0", "v1.1.0", "v1.2.0"];
const gitea = ["v1.0.0", "v1.2.0"]; // v1.1.0 is missing
const { toCreate, toSkip } = classifyReleasesForReconciliation(github, gitea);
expect(toCreate).toEqual(["v1.1.0"]);
expect(toSkip).toEqual(["v1.0.0", "v1.2.0"]);
});
it("does NOT produce any deletions — order mismatches are ignored", () => {
// Even if Gitea has them in a different order, the function never suggests deletion
const github = ["v1.0.0", "v1.1.0"];
const gitea = ["v1.1.0", "v1.0.0"]; // reversed order from Gitea
const { toCreate, toSkip } = classifyReleasesForReconciliation(github, gitea);
expect(toCreate).toEqual([]);
expect(toSkip).toHaveLength(2);
expect(toSkip).toContain("v1.0.0");
expect(toSkip).toContain("v1.1.0");
});
});
describe("unaconfig_dart regression — published_at order disagrees with tag-commit order (#310)", () => {
// v0.1.0: tagged 2024-01-13T23:30, published_at 2024-01-16T00:23 (published AFTER v0.1.1)
// v0.1.1: tagged 2024-01-13T23:42, published_at 2024-01-13T23:44 (published BEFORE v0.1.0)
//
// Gitea display order (by tag-commit date): [v0.1.1, v0.1.0] (v0.1.1 tagged later)
// GitHub published_at order (oldest first): [v0.1.1, v0.1.0] (v0.1.1 published earlier)
// Wait — in this specific case the orders AGREE. The inversion scenario is:
// GitHub sorts descending (newest first): [v0.1.0, v0.1.1]
// Gitea API returns by tag-commit DESC: [v0.1.1, v0.1.0]
// Old expectedOrder (ascending published): v0.1.1→0, v0.1.0→1
// Check for [v0.1.1, v0.1.0]: current=v0.1.1(idx=0) < next=v0.1.0(idx=1) → TRUE every time
it("does NOT trigger recreation when published_at order and tag-commit order disagree", () => {
// Both releases already in Gitea (as they would be after first successful sync).
// Old code would fire needsRecreation=true here on every subsequent sync.
// New code: set-based check — both present → toCreate is empty → no deletions.
const github = ["v0.1.0", "v0.1.1"]; // GitHub API returns newest published_at first
const gitea = ["v0.1.1", "v0.1.0"]; // Gitea tag-commit order (v0.1.1 tagged later)
const { toCreate, toSkip } = classifyReleasesForReconciliation(github, gitea);
expect(toCreate).toEqual([]); // nothing to create
expect(toSkip).toHaveLength(2);
expect(toSkip).toContain("v0.1.0");
expect(toSkip).toContain("v0.1.1");
});
it("creates v0.1.0 and v0.1.1 when Gitea has no releases yet (first sync)", () => {
const github = ["v0.1.0", "v0.1.1"];
const gitea: string[] = [];
const { toCreate, toSkip } = classifyReleasesForReconciliation(github, gitea);
expect(toCreate).toEqual(["v0.1.0", "v0.1.1"]);
expect(toSkip).toEqual([]);
});
it("only creates the missing release when one of the two already exists", () => {
const github = ["v0.1.0", "v0.1.1"];
const gitea = ["v0.1.1"]; // only v0.1.1 was created so far
const { toCreate, toSkip } = classifyReleasesForReconciliation(github, gitea);
expect(toCreate).toEqual(["v0.1.0"]);
expect(toSkip).toEqual(["v0.1.1"]);
});
});
describe("edge cases", () => {
it("handles empty GitHub releases list", () => {
const { toCreate, toSkip } = classifyReleasesForReconciliation([], ["v1.0.0"]);
expect(toCreate).toEqual([]);
expect(toSkip).toEqual([]);
});
it("handles both lists empty", () => {
const { toCreate, toSkip } = classifyReleasesForReconciliation([], []);
expect(toCreate).toEqual([]);
expect(toSkip).toEqual([]);
});
it("ignores Gitea releases that are not in the GitHub set (orphans, handled by retention cleanup)", () => {
const github = ["v1.0.0"];
const gitea = ["v1.0.0", "v0.9.0"]; // v0.9.0 is an orphan not in GitHub's limited set
const { toCreate, toSkip } = classifyReleasesForReconciliation(github, gitea);
expect(toCreate).toEqual([]);
expect(toSkip).toEqual(["v1.0.0"]);
// v0.9.0 not mentioned in either output — handled by retention cleanup, not here
});
});
});
+275 -193
View File
@@ -579,7 +579,27 @@ export const mirrorGithubRepoToGitea = async ({
// Determine the actual repository name to use (handle duplicates for starred repos)
let targetRepoName = repository.name;
if (
// REUSE-FIRST (issues #315 / #309): before generating any (suffixed) name,
// check whether this exact source is already mirrored — either at the
// recorded mirroredLocation or at the base name. If so, reuse that location
// and route into the "already mirrored" handling below instead of creating
// a duplicate. This must run before generateUniqueRepoName so the names
// converge under concurrency (the in-flight guard then becomes effective).
const { findExistingMirror } = await import("./utils/mirror-source-match");
const existingMirror = await findExistingMirror({
repository,
config,
candidateOwner: repoOwner,
candidateName: repository.name,
});
if (existingMirror) {
repoOwner = existingMirror.owner;
targetRepoName = existingMirror.repoName;
console.log(
`Reusing existing same-source mirror for ${repository.fullName} at ${repoOwner}/${targetRepoName}`
);
} else if (
repository.isStarred &&
config.githubConfig &&
(config.githubConfig.starredReposMode || "dedicated-org") === "dedicated-org"
@@ -594,6 +614,7 @@ export const mirrorGithubRepoToGitea = async ({
githubOwner,
fullName: repository.fullName,
strategy: config.githubConfig.starredDuplicateStrategy,
sourceCloneUrl: repository.cloneUrl,
});
if (targetRepoName !== repository.name) {
@@ -643,45 +664,72 @@ export const mirrorGithubRepoToGitea = async ({
strategy: "delete", // Can be configured: "skip", "delete", or "rename"
});
} else if (existingRepoInfo?.mirror) {
console.log(
`Repository ${targetRepoName} already exists in Gitea under ${repoOwner}. Updating database status.`
);
// PHANTOM-FORK GUARD (#309): a mirror at this name is only "ours" if it
// mirrors THIS source. existingMirror short-circuits the check
// because findExistingMirror already confirmed the source match.
const { isMirrorOfSource } = await import("./utils/mirror-source-match");
const sameSource =
!!existingMirror ||
isMirrorOfSource(existingRepoInfo, repository.cloneUrl);
await syncRepositoryMetadataToGitea({
config,
octokit,
repository,
giteaOwner: repoOwner,
giteaRepoName: targetRepoName,
giteaToken: decryptedConfig.giteaConfig.token,
});
if (!sameSource) {
// A different source occupies this name. Treat as a genuine collision:
// generate a unique name and fall through to create a separate mirror.
console.warn(
`[Mirror] ${repoOwner}/${targetRepoName} is a mirror of a different source. ` +
`Generating a unique name for ${repository.fullName} to avoid overwriting it.`
);
targetRepoName = await generateUniqueRepoName({
config,
orgName: repoOwner,
baseName: repository.name,
githubOwner: repository.fullName.split("/")[0],
fullName: repository.fullName,
strategy: config.githubConfig?.starredDuplicateStrategy,
sourceCloneUrl: repository.cloneUrl,
});
// expectedLocation is recomputed below before the "mirroring" write.
} else {
console.log(
`Repository ${targetRepoName} already exists in Gitea under ${repoOwner}. Updating database status.`
);
// Update database to reflect that the repository is already mirrored
await db
.update(repositories)
.set({
status: repoStatusEnum.parse("mirrored"),
updatedAt: new Date(),
lastMirrored: new Date(),
errorMessage: null,
mirroredLocation: `${repoOwner}/${targetRepoName}`,
})
.where(eq(repositories.id, repository.id!));
await syncRepositoryMetadataToGitea({
config,
octokit,
repository,
giteaOwner: repoOwner,
giteaRepoName: targetRepoName,
giteaToken: decryptedConfig.giteaConfig.token,
});
// Append log for "mirrored" status
await createMirrorJob({
userId: config.userId,
repositoryId: repository.id,
repositoryName: repository.name,
message: `Repository ${repository.name} already exists in Gitea`,
details: `Repository ${repository.name} was found to already exist in Gitea under ${repoOwner} and database status was updated.`,
status: "mirrored",
});
// Update database to reflect that the repository is already mirrored
await db
.update(repositories)
.set({
status: repoStatusEnum.parse("mirrored"),
updatedAt: new Date(),
lastMirrored: new Date(),
errorMessage: null,
mirroredLocation: `${repoOwner}/${targetRepoName}`,
})
.where(eq(repositories.id, repository.id!));
console.log(
`Repository ${repository.name} database status updated to mirrored`
);
return;
// Append log for "mirrored" status
await createMirrorJob({
userId: config.userId,
repositoryId: repository.id,
repositoryName: repository.name,
message: `Repository ${repository.name} already exists in Gitea`,
details: `Repository ${repository.name} was found to already exist in Gitea under ${repoOwner} and database status was updated.`,
status: "mirrored",
});
console.log(
`Repository ${repository.name} database status updated to mirrored`
);
return;
}
} else {
console.warn(
`[Mirror] Repository ${repoOwner}/${targetRepoName} exists but mirror status could not be verified. Continuing with mirror creation flow.`
@@ -689,6 +737,10 @@ export const mirrorGithubRepoToGitea = async ({
}
}
// Recompute the target location in case a phantom-fork collision above
// forced a renamed target after the initial expectedLocation was derived.
const targetLocation = `${repoOwner}/${targetRepoName}`;
console.log(`Mirroring repository ${repository.name}`);
// DOUBLE-CHECK: Final idempotency check right before updating status
@@ -696,7 +748,7 @@ export const mirrorGithubRepoToGitea = async ({
const finalCheck = await isRepoCurrentlyMirroring({
config,
repoName: targetRepoName,
expectedLocation,
expectedLocation: targetLocation,
});
if (finalCheck) {
@@ -714,7 +766,7 @@ export const mirrorGithubRepoToGitea = async ({
.update(repositories)
.set({
status: repoStatusEnum.parse("mirroring"),
mirroredLocation: expectedLocation,
mirroredLocation: targetLocation,
updatedAt: new Date(),
})
.where(eq(repositories.id, repository.id!));
@@ -1177,6 +1229,14 @@ async function isMirroredLocationClaimedInDb({
* Checks both the Gitea instance (HTTP) and the local DB (mirroredLocation)
* to reduce collisions during concurrent batch mirroring.
*
* Source-aware (issues #315 / #309): when a candidate name is already occupied
* by a mirror of THIS SAME GitHub source, the name is REUSED rather than
* suffixed this is what previously caused starred repos to spawn `-owner`,
* `-owner-1`, duplicates on every re-mirror. Suffixing only happens on a
* genuine different-source collision (preserving the #95/#236 cross-owner
* behavior). The per-user DB claim check is retained so two users mirroring the
* same source into a shared org stay separated.
*
* NOTE: This function only checks availability it does NOT claim the name.
* The actual claim happens later when mirroredLocation is written at the
* status="mirroring" DB update, which is protected by a unique partial index
@@ -1189,6 +1249,7 @@ async function generateUniqueRepoName({
githubOwner,
fullName,
strategy,
sourceCloneUrl,
}: {
config: Partial<Config>;
orgName: string;
@@ -1196,6 +1257,10 @@ async function generateUniqueRepoName({
githubOwner: string;
fullName: string;
strategy?: string;
// Source GitHub clone URL, used to decide whether an occupied name belongs to
// THIS repo's mirror (reuse) or a different source (suffix). When omitted,
// behavior degrades to the legacy "any occupant collides" semantics.
sourceCloneUrl?: string;
}): Promise<string> {
if (!fullName?.includes("/")) {
throw new Error(
@@ -1206,33 +1271,55 @@ async function generateUniqueRepoName({
const duplicateStrategy = strategy || "suffix";
const userId = config.userId || "";
// Helper: check both Gitea and local DB for a candidate name
const isNameTaken = async (candidateName: string): Promise<boolean> => {
const { getGiteaRepoInfo } = await import("./gitea-enhanced");
const { classifyCandidateName } = await import("./utils/mirror-source-match");
// Resolve the I/O for a candidate name (Gitea existence, DB claim, repo info)
// and defer the available/reusable/taken decision to the pure, unit-tested
// classifyCandidateName helper.
const classifyName = async (candidateName: string) => {
const existsInGitea = await isRepoPresentInGitea({
config,
owner: orgName,
repoName: candidateName,
});
if (existsInGitea) return true;
// Also check local DB to catch concurrent batch operations
// where another repo claimed this location but hasn't created it in Gitea yet
// A DB claim by a DIFFERENT repo (concurrent batch) always blocks reuse.
let claimedByOther = false;
if (userId) {
const claimedInDb = await isMirroredLocationClaimedInDb({
claimedByOther = await isMirroredLocationClaimedInDb({
userId,
candidateLocation: `${orgName}/${candidateName}`,
excludeFullName: fullName,
});
if (claimedInDb) return true;
}
return false;
// Only fetch repo info when it can actually change the decision (existing,
// same-source candidate that is not DB-claimed by another repo).
const repoInfo =
existsInGitea && sourceCloneUrl && !claimedByOther
? await getGiteaRepoInfo({
config,
owner: orgName,
repoName: candidateName,
})
: null;
return classifyCandidateName({
existsInGitea,
claimedByOther,
repoInfo,
sourceCloneUrl,
});
};
// First check if base name is available
const baseExists = await isNameTaken(baseName);
if (!baseExists) {
// First check the base name — reuse it if it already holds our own mirror.
const baseClass = await classifyName(baseName);
if (baseClass === "available") {
return baseName;
}
if (baseClass === "reusable") {
console.log(`Reusing existing same-source mirror name: ${orgName}/${baseName}`);
return baseName;
}
@@ -1262,9 +1349,14 @@ async function generateUniqueRepoName({
break;
}
const exists = await isNameTaken(candidateName);
const candidateClass = await classifyName(candidateName);
if (!exists) {
if (candidateClass === "reusable") {
console.log(`Reusing existing same-source mirror name: ${orgName}/${candidateName}`);
return candidateName;
}
if (candidateClass === "available") {
console.log(`Found unique name for duplicate starred repo: ${candidateName}`);
return candidateName;
}
@@ -1314,8 +1406,29 @@ export async function mirrorGitHubRepoToGiteaOrg({
// Determine the actual repository name to use (handle duplicates for starred repos)
let targetRepoName = repository.name;
// The org we will record/reuse for. Stays === orgName on the create path
// (migration uses orgName + giteaOrgId); a reuse hit may repoint it to the
// recorded mirroredLocation's owner for the early-return DB update.
let targetOwner = orgName;
if (
// REUSE-FIRST (issues #315 / #309): reuse an existing same-source mirror
// before generating any suffixed name. See mirrorGithubRepoToGitea for the
// rationale. Routes a hit into the "already mirrored" handling below.
const { findExistingMirror } = await import("./utils/mirror-source-match");
const existingMirror = await findExistingMirror({
repository,
config,
candidateOwner: orgName,
candidateName: repository.name,
});
if (existingMirror) {
targetOwner = existingMirror.owner;
targetRepoName = existingMirror.repoName;
console.log(
`Reusing existing same-source mirror for ${repository.fullName} at ${targetOwner}/${targetRepoName}`
);
} else if (
repository.isStarred &&
config.githubConfig &&
(config.githubConfig.starredReposMode || "dedicated-org") === "dedicated-org"
@@ -1330,6 +1443,7 @@ export async function mirrorGitHubRepoToGiteaOrg({
githubOwner,
fullName: repository.fullName,
strategy: config.githubConfig.starredDuplicateStrategy,
sourceCloneUrl: repository.cloneUrl,
});
if (targetRepoName !== repository.name) {
@@ -1340,7 +1454,7 @@ export async function mirrorGitHubRepoToGiteaOrg({
}
// IDEMPOTENCY CHECK: Check if this repo is already being mirrored
const expectedLocation = `${orgName}/${targetRepoName}`;
const expectedLocation = `${targetOwner}/${targetRepoName}`;
const isCurrentlyMirroring = await isRepoCurrentlyMirroring({
config,
repoName: targetRepoName,
@@ -1358,7 +1472,7 @@ export async function mirrorGitHubRepoToGiteaOrg({
const isExisting = await isRepoPresentInGitea({
config,
owner: orgName,
owner: targetOwner,
repoName: targetRepoName,
});
@@ -1366,7 +1480,7 @@ export async function mirrorGitHubRepoToGiteaOrg({
const { getGiteaRepoInfo, handleExistingNonMirrorRepo } = await import("./gitea-enhanced");
const existingRepoInfo = await getGiteaRepoInfo({
config,
owner: orgName,
owner: targetOwner,
repoName: targetRepoName,
});
@@ -1379,52 +1493,83 @@ export async function mirrorGitHubRepoToGiteaOrg({
strategy: "delete", // Can be configured: "skip", "delete", or "rename"
});
} else if (existingRepoInfo?.mirror) {
console.log(
`Repository ${targetRepoName} already exists in Gitea organization ${orgName}. Updating database status.`
);
// PHANTOM-FORK GUARD (#309): only treat this as "ours" if it mirrors
// THIS source. existingMirror short-circuits because findExistingMirror already
// confirmed the source match.
const { isMirrorOfSource } = await import("./utils/mirror-source-match");
const sameSource =
!!existingMirror ||
isMirrorOfSource(existingRepoInfo, repository.cloneUrl);
await syncRepositoryMetadataToGitea({
config,
octokit,
repository,
giteaOwner: orgName,
giteaRepoName: targetRepoName,
giteaToken: decryptedConfig.giteaConfig.token,
});
if (!sameSource) {
// Different source occupies this name: generate a unique name and
// fall through to create a separate mirror under orgName/giteaOrgId.
console.warn(
`[Mirror] ${targetOwner}/${targetRepoName} is a mirror of a different source. ` +
`Generating a unique name for ${repository.fullName} to avoid overwriting it.`
);
targetOwner = orgName;
targetRepoName = await generateUniqueRepoName({
config,
orgName,
baseName: repository.name,
githubOwner: repository.fullName.split("/")[0],
fullName: repository.fullName,
strategy: config.githubConfig?.starredDuplicateStrategy,
sourceCloneUrl: repository.cloneUrl,
});
} else {
console.log(
`Repository ${targetRepoName} already exists in Gitea organization ${targetOwner}. Updating database status.`
);
// Update database to reflect that the repository is already mirrored
await db
.update(repositories)
.set({
status: repoStatusEnum.parse("mirrored"),
updatedAt: new Date(),
lastMirrored: new Date(),
errorMessage: null,
mirroredLocation: `${orgName}/${targetRepoName}`,
})
.where(eq(repositories.id, repository.id!));
await syncRepositoryMetadataToGitea({
config,
octokit,
repository,
giteaOwner: targetOwner,
giteaRepoName: targetRepoName,
giteaToken: decryptedConfig.giteaConfig.token,
});
// Create a mirror job log entry
await createMirrorJob({
userId: config.userId,
repositoryId: repository.id,
repositoryName: repository.name,
message: `Repository ${targetRepoName} already exists in Gitea organization ${orgName}`,
details: `Repository ${targetRepoName} was found to already exist in Gitea organization ${orgName} and database status was updated.`,
status: "mirrored",
});
// Update database to reflect that the repository is already mirrored
await db
.update(repositories)
.set({
status: repoStatusEnum.parse("mirrored"),
updatedAt: new Date(),
lastMirrored: new Date(),
errorMessage: null,
mirroredLocation: `${targetOwner}/${targetRepoName}`,
})
.where(eq(repositories.id, repository.id!));
console.log(
`Repository ${targetRepoName} database status updated to mirrored in organization ${orgName}`
);
return;
// Create a mirror job log entry
await createMirrorJob({
userId: config.userId,
repositoryId: repository.id,
repositoryName: repository.name,
message: `Repository ${targetRepoName} already exists in Gitea organization ${targetOwner}`,
details: `Repository ${targetRepoName} was found to already exist in Gitea organization ${targetOwner} and database status was updated.`,
status: "mirrored",
});
console.log(
`Repository ${targetRepoName} database status updated to mirrored in organization ${targetOwner}`
);
return;
}
} else {
console.warn(
`[Mirror] Repository ${orgName}/${targetRepoName} exists but mirror status could not be verified. Continuing with mirror creation flow.`
`[Mirror] Repository ${targetOwner}/${targetRepoName} exists but mirror status could not be verified. Continuing with mirror creation flow.`
);
}
}
// Recompute the target location in case a phantom-fork collision above
// forced a renamed target after the initial expectedLocation was derived.
const targetLocation = `${orgName}/${targetRepoName}`;
console.log(
`Mirroring repository ${repository.fullName} to organization ${orgName} as ${targetRepoName}`
);
@@ -1437,7 +1582,7 @@ export async function mirrorGitHubRepoToGiteaOrg({
const finalCheck = await isRepoCurrentlyMirroring({
config,
repoName: targetRepoName,
expectedLocation,
expectedLocation: targetLocation,
});
if (finalCheck) {
@@ -1455,7 +1600,7 @@ export async function mirrorGitHubRepoToGiteaOrg({
.update(repositories)
.set({
status: repoStatusEnum.parse("mirroring"),
mirroredLocation: expectedLocation,
mirroredLocation: targetLocation,
updatedAt: new Date(),
})
.where(eq(repositories.id, repository.id!));
@@ -2515,6 +2660,37 @@ export const mirrorGitRepoIssuesToGitea = async ({
);
};
/**
* Classify a set of GitHub releases against the set already present in Gitea.
*
* Returns:
* - `toCreate`: tag names that exist on GitHub but are missing from Gitea
* - `toSkip`: tag names that already exist in Gitea (will be handled by PATCH-if-content-changed)
*
* Deliberately does NOT return anything to delete based on ordering Gitea mirrors
* order releases by tag-commit date, which can permanently disagree with GitHub's
* published_at order (e.g. unaconfig_dart v0.1.0/v0.1.1 #310). Destroying and
* re-emitting releases for a cosmetic display-order difference is never worth it.
*/
export function classifyReleasesForReconciliation(
githubTagNames: string[],
giteaTagNames: string[]
): { toCreate: string[]; toSkip: string[] } {
const giteaSet = new Set(giteaTagNames);
const toCreate: string[] = [];
const toSkip: string[] = [];
for (const tag of githubTagNames) {
if (giteaSet.has(tag)) {
toSkip.push(tag);
} else {
toCreate.push(tag);
}
}
return { toCreate, toSkip };
}
export async function mirrorGitHubReleasesToGitea({
octokit,
repository,
@@ -2601,23 +2777,10 @@ export async function mirrorGitHubReleasesToGitea({
let mirroredCount = 0;
let skippedCount = 0;
const getReleaseTimestamp = (release: (typeof limitedReleases)[number]) => {
// Use published_at first (when the release was published on GitHub)
// Fall back to created_at (when the git tag was created) only if published_at is missing
// This matches GitHub's sorting behavior and handles cases where multiple tags
// point to the same commit but have different publish dates
const sourceDate = release.published_at ?? release.created_at ?? "";
const timestamp = sourceDate ? new Date(sourceDate).getTime() : 0;
return Number.isFinite(timestamp) ? timestamp : 0;
};
// Process releases in their GitHub API order (newest first by default)
const releasesToProcess = limitedReleases.slice();
// Capture the latest releases, then process them oldest-to-newest so Gitea mirrors keep chronological order
const releasesToProcess = limitedReleases
.slice()
.sort((a, b) => getReleaseTimestamp(b) - getReleaseTimestamp(a))
.sort((a, b) => getReleaseTimestamp(a) - getReleaseTimestamp(b));
console.log(`[Releases] Processing ${releasesToProcess.length} releases in chronological order (oldest to newest by published date)`);
console.log(`[Releases] Processing ${releasesToProcess.length} releases for ${repository.fullName}`);
releasesToProcess.forEach((rel, idx) => {
const publishedDate = new Date(rel.published_at || rel.created_at);
const createdDate = new Date(rel.created_at);
@@ -2627,85 +2790,10 @@ export async function mirrorGitHubReleasesToGitea({
console.log(`[Releases] ${idx + 1}. ${rel.tag_name} - ${dateInfo}`);
});
// Check if existing releases in Gitea are in the wrong order
// If so, we need to delete and recreate them to fix the ordering
let needsRecreation = false;
try {
const existingReleasesResponse = await httpGet(
`${config.giteaConfig.url}/api/v1/repos/${repoOwner}/${repoName}/releases?per_page=100`,
{
Authorization: `token ${decryptedConfig.giteaConfig.token}`,
}
).catch(() => null);
if (existingReleasesResponse && existingReleasesResponse.data && Array.isArray(existingReleasesResponse.data)) {
const existingReleases = existingReleasesResponse.data;
if (existingReleases.length > 0) {
console.log(`[Releases] Found ${existingReleases.length} existing releases in Gitea, checking chronological order...`);
// Create a map of tag_name to expected chronological index (0 = oldest, n = newest)
const expectedOrder = new Map<string, number>();
releasesToProcess.forEach((rel, idx) => {
expectedOrder.set(rel.tag_name, idx);
});
// Check if existing releases are in the correct order based on created_unix
// Gitea sorts by created_unix DESC, so newer releases should have higher created_unix values
const releasesThatShouldExist = existingReleases.filter(r => expectedOrder.has(r.tag_name));
if (releasesThatShouldExist.length > 1) {
for (let i = 0; i < releasesThatShouldExist.length - 1; i++) {
const current = releasesThatShouldExist[i];
const next = releasesThatShouldExist[i + 1];
const currentExpectedIdx = expectedOrder.get(current.tag_name)!;
const nextExpectedIdx = expectedOrder.get(next.tag_name)!;
// Since Gitea returns releases sorted by created_unix DESC:
// - Earlier releases in the list should have HIGHER expected indices (newer)
// - Later releases in the list should have LOWER expected indices (older)
if (currentExpectedIdx < nextExpectedIdx) {
console.log(`[Releases] ⚠️ Incorrect ordering detected: ${current.tag_name} (index ${currentExpectedIdx}) appears before ${next.tag_name} (index ${nextExpectedIdx})`);
needsRecreation = true;
break;
}
}
}
if (needsRecreation) {
console.log(`[Releases] ⚠️ Releases are in incorrect chronological order. Will delete and recreate all releases.`);
// Delete all existing releases that we're about to recreate
for (const existingRelease of releasesThatShouldExist) {
try {
console.log(`[Releases] Deleting incorrectly ordered release: ${existingRelease.tag_name}`);
await httpDelete(
`${config.giteaConfig.url}/api/v1/repos/${repoOwner}/${repoName}/releases/${existingRelease.id}`,
{
Authorization: `token ${decryptedConfig.giteaConfig.token}`,
}
);
} catch (deleteError) {
console.error(`[Releases] Failed to delete release ${existingRelease.tag_name}: ${deleteError instanceof Error ? deleteError.message : String(deleteError)}`);
}
}
console.log(`[Releases] ✅ Deleted ${releasesThatShouldExist.length} releases. Will recreate in correct chronological order.`);
} else {
console.log(`[Releases] ✅ Existing releases are in correct chronological order.`);
}
}
}
} catch (orderCheckError) {
console.warn(`[Releases] Could not verify release order: ${orderCheckError instanceof Error ? orderCheckError.message : String(orderCheckError)}`);
// Continue with normal processing
}
for (const release of releasesToProcess) {
try {
// Check if release already exists (skip check if we just deleted all releases)
const existingReleasesResponse = needsRecreation ? null : await httpGet(
// Always check if release already exists — reconcile by tag set, not by ordering
const existingReleasesResponse = await httpGet(
`${config.giteaConfig.url}/api/v1/repos/${repoOwner}/${repoName}/releases/tags/${release.tag_name}`,
{
Authorization: `token ${decryptedConfig.giteaConfig.token}`,
@@ -2842,12 +2930,6 @@ export async function mirrorGitHubReleasesToGitea({
mirroredCount++;
const noteInfo = originalReleaseNote ? ` with ${originalReleaseNote.length} character changelog` : " without changelog";
console.log(`[Releases] Successfully mirrored release: ${release.tag_name}${noteInfo}`);
// Add delay to ensure proper timestamp ordering in Gitea
// Gitea sorts releases by created_unix DESC, and all releases created in quick succession
// will have nearly identical timestamps. The 1-second delay ensures proper chronological order.
console.log(`[Releases] Waiting 1 second to ensure proper timestamp ordering in Gitea...`);
await new Promise(resolve => setTimeout(resolve, 1000));
} catch (error) {
console.error(`[Releases] Failed to mirror release ${release.tag_name}: ${error instanceof Error ? error.message : String(error)}`);
}
+75 -9
View File
@@ -1,15 +1,23 @@
import { describe, expect, test, mock } from "bun:test";
import { getGithubRepositories } from "@/lib/github";
function makeRepo() {
function makeRepo(overrides: Partial<{
name: string;
full_name: string;
ownerLogin: string;
ownerType: string;
fork: boolean;
}> = {}) {
const ownerLogin = overrides.ownerLogin ?? "octo";
const ownerType = overrides.ownerType ?? "User";
return {
name: "demo",
full_name: "octo/demo",
html_url: "https://github.com/octo/demo",
clone_url: "https://github.com/octo/demo.git",
owner: { login: "octo", type: "User" },
name: overrides.name ?? "demo",
full_name: overrides.full_name ?? `${ownerLogin}/${overrides.name ?? "demo"}`,
html_url: `https://github.com/${ownerLogin}/${overrides.name ?? "demo"}`,
clone_url: `https://github.com/${ownerLogin}/${overrides.name ?? "demo"}.git`,
owner: { login: ownerLogin, type: ownerType },
private: false,
fork: false,
fork: overrides.fork ?? false,
has_issues: true,
archived: false,
size: 1,
@@ -23,11 +31,11 @@ function makeRepo() {
};
}
function makeOctokit() {
function makeOctokit(reposToReturn?: ReturnType<typeof makeRepo>[]) {
let captured: Record<string, unknown> | null = null;
const paginate = mock(async (_method: unknown, options?: Record<string, unknown>) => {
captured = options ?? null;
return [makeRepo()];
return reposToReturn ?? [makeRepo()];
});
return {
octokit: {
@@ -98,3 +106,61 @@ describe("getGithubRepositories - affiliation", () => {
}
});
});
describe("getGithubRepositories - skipPersonalRepos", () => {
const personalRepo = makeRepo({ name: "my-lib", ownerLogin: "octo", ownerType: "User" });
const orgRepo = makeRepo({ name: "org-lib", ownerLogin: "my-org", ownerType: "Organization" });
const otherUserRepo = makeRepo({ name: "collab-lib", ownerLogin: "other-user", ownerType: "User" });
test("default false — keeps all repos including personal", async () => {
const { octokit } = makeOctokit([personalRepo, orgRepo]);
const repos = await getGithubRepositories({
octokit,
config: { githubConfig: { owner: "octo", skipPersonalRepos: false } as any },
});
expect(repos.map((r) => r.name)).toContain("my-lib");
expect(repos.map((r) => r.name)).toContain("org-lib");
});
test("skipPersonalRepos=true — drops repos owned by authenticated user", async () => {
const { octokit } = makeOctokit([personalRepo, orgRepo]);
const repos = await getGithubRepositories({
octokit,
config: { githubConfig: { owner: "octo", skipPersonalRepos: true } as any },
});
expect(repos.map((r) => r.name)).not.toContain("my-lib");
expect(repos.map((r) => r.name)).toContain("org-lib");
});
test("skipPersonalRepos=true — keeps repos owned by other users (collaborator repos)", async () => {
const { octokit } = makeOctokit([personalRepo, orgRepo, otherUserRepo]);
const repos = await getGithubRepositories({
octokit,
config: { githubConfig: { owner: "octo", skipPersonalRepos: true } as any },
});
expect(repos.map((r) => r.name)).not.toContain("my-lib");
expect(repos.map((r) => r.name)).toContain("org-lib");
expect(repos.map((r) => r.name)).toContain("collab-lib");
});
test("skipPersonalRepos=true with no owner configured — keeps all repos (safe fallback)", async () => {
const { octokit } = makeOctokit([personalRepo, orgRepo]);
const repos = await getGithubRepositories({
octokit,
config: { githubConfig: { owner: "", skipPersonalRepos: true } as any },
});
// Empty owner means we can't identify the user, so nothing should be dropped
expect(repos.map((r) => r.name)).toContain("my-lib");
expect(repos.map((r) => r.name)).toContain("org-lib");
});
test("skipPersonalRepos=true — unset (undefined) behaves like false", async () => {
const { octokit } = makeOctokit([personalRepo, orgRepo]);
const repos = await getGithubRepositories({
octokit,
config: { githubConfig: { owner: "octo" } as any },
});
expect(repos.map((r) => r.name)).toContain("my-lib");
expect(repos.map((r) => r.name)).toContain("org-lib");
});
});
+12 -1
View File
@@ -263,10 +263,21 @@ export async function getGithubRepositories({
);
const skipForks = config.githubConfig?.skipForks ?? false;
const skipPersonalRepos = config.githubConfig?.skipPersonalRepos ?? false;
// The authenticated user's login — used to identify personally-owned repos
const authenticatedUserLogin = config.githubConfig?.owner ?? "";
const filteredRepos = repos.filter((repo) => {
const isForkAllowed = !skipForks || !repo.fork;
return isForkAllowed;
// When skipPersonalRepos is true, drop repos owned by the authenticated user
// (owner.type === "User" and owner.login matches the configured GitHub username).
// Org repos have owner.type === "Organization" so they are always kept.
const isPersonalRepo =
skipPersonalRepos &&
authenticatedUserLogin.length > 0 &&
repo.owner.login === authenticatedUserLogin &&
repo.owner.type === "User";
return isForkAllowed && !isPersonalRepo;
});
return filteredRepos.map((repo) => ({
+22
View File
@@ -103,6 +103,28 @@ describe("Scheduler Service - Ignored Repository Handling", () => {
]);
});
test("auto-start gate: enabled=true → should start, enabled=false → should not start even with mirrorInterval", () => {
// Mirror the gate logic from checkAutoStartConfiguration / performInitialAutoStart.
// The enabled flag is the single authoritative signal; a configured
// mirrorInterval is a timing detail and must not bypass a disabled toggle.
const shouldAutoStart = (scheduleConfig?: { enabled?: boolean }) =>
scheduleConfig?.enabled === true;
expect(shouldAutoStart({ enabled: true })).toBe(true);
expect(shouldAutoStart({ enabled: false })).toBe(false);
expect(shouldAutoStart({})).toBe(false);
expect(shouldAutoStart(undefined)).toBe(false);
// Simulating: user disabled scheduling but has a mirrorInterval configured.
// The old code checked `scheduleEnabled || hasMirrorInterval`; the fix
// ensures only the enabled flag is checked.
const configWithIntervalButDisabled = {
scheduleConfig: { enabled: false },
giteaConfig: { mirrorInterval: "8h" },
};
expect(shouldAutoStart(configWithIntervalButDisabled.scheduleConfig)).toBe(false);
});
test("should validate all repository status enum values", () => {
const validStatuses = [
"imported",
+36 -10
View File
@@ -266,6 +266,27 @@ async function runScheduledSync(config: any): Promise<void> {
visibility: repositoryVisibilityEnum.parse(repo.visibility),
};
// A `failed` repo whose recorded location still resolves to a
// live same-source mirror (e.g. migrate succeeded but metadata
// failed) must be SYNCED, not re-created — otherwise the
// re-create loop spawns suffixed duplicates (#315). The create
// path also reuses now, but routing to sync here avoids a
// wasted migrate attempt and keeps recovery cheap.
if (repo.status === 'failed' && repository.mirroredLocation) {
const { findExistingMirror } = await import('@/lib/utils/mirror-source-match');
const existing = await findExistingMirror({
repository,
config,
candidateOwner: repository.mirroredLocation.split('/')[0] || '',
candidateName: repository.name,
});
if (existing) {
await syncGiteaRepo({ config, repository });
console.log(`[Scheduler] Re-synced failed repository with live mirror: ${repo.fullName}`);
return;
}
}
await mirrorGithubRepoToGitea({ octokit, repository, config });
console.log(`[Scheduler] Auto-mirrored repository: ${repo.fullName}`);
} catch (error) {
@@ -431,13 +452,16 @@ async function checkAutoStartConfiguration(): Promise<boolean> {
.where(eq(configs.isActive, true));
for (const config of activeConfigs) {
// Check if scheduling is enabled via environment
// Check if scheduling is enabled.
// Note: env-config-loader already sets scheduleConfig.enabled=true when
// GITEA_MIRROR_INTERVAL is set at startup, so the enabled flag is the
// single authoritative gate here. Checking hasMirrorInterval directly
// would allow a configured interval to trigger auto-start even after the
// user explicitly disabled scheduling via the UI.
const scheduleEnabled = config.scheduleConfig?.enabled === true;
const hasMirrorInterval = !!config.giteaConfig?.mirrorInterval;
// If either SCHEDULE_ENABLED=true or GITEA_MIRROR_INTERVAL is set, we should auto-start
if (scheduleEnabled || hasMirrorInterval) {
console.log(`[Scheduler] Auto-start conditions met for user ${config.userId} (scheduleEnabled=${scheduleEnabled}, hasMirrorInterval=${hasMirrorInterval})`);
if (scheduleEnabled) {
console.log(`[Scheduler] Auto-start conditions met for user ${config.userId} (scheduleEnabled=${scheduleEnabled})`);
return true;
}
}
@@ -472,10 +496,12 @@ async function performInitialAutoStart(): Promise<void> {
}
const scheduleEnabled = config.scheduleConfig?.enabled === true;
const hasMirrorInterval = !!config.giteaConfig?.mirrorInterval;
// Only process configs that have scheduling or mirror interval configured
if (!scheduleEnabled && !hasMirrorInterval) {
// Only process configs where scheduling is explicitly enabled.
// env-config-loader already sets enabled=true when GITEA_MIRROR_INTERVAL
// is present, so this single check covers both the UI toggle and the
// env-var boot path without letting a bare interval override a disabled toggle.
if (!scheduleEnabled) {
continue;
}
-303
View File
@@ -1,303 +0,0 @@
/**
* Enhanced handler for starred repositories with improved error handling
*/
import type { Config, Repository } from "./db/schema";
import { Octokit } from "@octokit/rest";
import { processWithRetry } from "./utils/concurrency";
import {
getOrCreateGiteaOrgEnhanced,
getGiteaRepoInfo,
handleExistingNonMirrorRepo,
createOrganizationsSequentially
} from "./gitea-enhanced";
import { mirrorGithubRepoToGitea } from "./gitea";
import { getMirrorStrategyConfig } from "./utils/mirror-strategies";
import { createMirrorJob } from "./helpers";
/**
* Process starred repositories with enhanced error handling
*/
export async function processStarredRepositories({
config,
repositories,
octokit,
}: {
config: Config;
repositories: Repository[];
octokit: Octokit;
}): Promise<void> {
if (!config.userId) {
throw new Error("User ID is required");
}
const strategyConfig = getMirrorStrategyConfig();
console.log(`Processing ${repositories.length} starred repositories`);
console.log(`Using strategy config:`, strategyConfig);
// Step 1: Pre-create organizations to avoid race conditions
if (strategyConfig.sequentialOrgCreation) {
await preCreateOrganizations({ config, repositories });
}
// Step 2: Process repositories with enhanced error handling
await processWithRetry(
repositories,
async (repository) => {
try {
await processStarredRepository({
config,
repository,
octokit,
strategyConfig,
});
return repository;
} catch (error) {
console.error(`Failed to process starred repository ${repository.name}:`, error);
throw error;
}
},
{
concurrencyLimit: strategyConfig.repoBatchSize,
maxRetries: 2,
retryDelay: 2000,
onProgress: (completed, total, result) => {
const percentComplete = Math.round((completed / total) * 100);
if (result) {
console.log(
`Processed starred repository "${result.name}" (${completed}/${total}, ${percentComplete}%)`
);
}
},
onRetry: (repo, error, attempt) => {
console.log(
`Retrying starred repository ${repo.name} (attempt ${attempt}): ${error.message}`
);
},
}
);
}
/**
* Pre-create all required organizations sequentially
*/
async function preCreateOrganizations({
config,
repositories,
}: {
config: Config;
repositories: Repository[];
}): Promise<void> {
// Get unique organization names
const orgNames = new Set<string>();
const starredReposMode = config.githubConfig?.starredReposMode || "dedicated-org";
if (starredReposMode === "preserve-owner") {
for (const repo of repositories) {
orgNames.add(repo.organization || repo.owner);
}
} else if (config.githubConfig?.starredReposOrg) {
orgNames.add(config.githubConfig.starredReposOrg);
} else {
orgNames.add("starred");
}
// Add any other organizations based on mirror strategy
for (const repo of repositories) {
if (repo.destinationOrg) {
orgNames.add(repo.destinationOrg);
}
}
console.log(`Pre-creating ${orgNames.size} organizations sequentially`);
// Create organizations sequentially
await createOrganizationsSequentially({
config,
orgNames: Array.from(orgNames),
});
}
/**
* Process a single starred repository with enhanced error handling
*/
async function processStarredRepository({
config,
repository,
octokit,
strategyConfig,
}: {
config: Config;
repository: Repository;
octokit: Octokit;
strategyConfig: ReturnType<typeof getMirrorStrategyConfig>;
}): Promise<void> {
const starredReposMode = config.githubConfig?.starredReposMode || "dedicated-org";
const starredOrg =
starredReposMode === "preserve-owner"
? repository.organization || repository.owner
: config.githubConfig?.starredReposOrg || "starred";
// Check if repository exists in Gitea
const existingRepo = await getGiteaRepoInfo({
config,
owner: starredOrg,
repoName: repository.name,
});
if (existingRepo) {
if (existingRepo.mirror) {
console.log(`Starred repository ${repository.name} already exists as a mirror`);
// Update database status
const { db, repositories: reposTable } = await import("./db");
const { eq } = await import("drizzle-orm");
const { repoStatusEnum } = await import("@/types/Repository");
await db
.update(reposTable)
.set({
status: repoStatusEnum.parse("mirrored"),
updatedAt: new Date(),
lastMirrored: new Date(),
errorMessage: null,
mirroredLocation: `${starredOrg}/${repository.name}`,
})
.where(eq(reposTable.id, repository.id!));
return;
} else {
// Repository exists but is not a mirror
console.warn(`Starred repository ${repository.name} exists but is not a mirror`);
await handleExistingNonMirrorRepo({
config,
repository,
repoInfo: existingRepo,
strategy: strategyConfig.nonMirrorStrategy,
});
// If we deleted it, continue to create the mirror
if (strategyConfig.nonMirrorStrategy !== "delete") {
return; // Skip if we're not deleting
}
}
}
// Create the mirror
try {
await mirrorGithubRepoToGitea({
octokit,
repository,
config,
});
} catch (error) {
// Enhanced error handling for specific scenarios
if (error instanceof Error) {
const errorMessage = error.message.toLowerCase();
if (errorMessage.includes("already exists")) {
// Handle race condition where repo was created by another process
console.log(`Repository ${repository.name} was created by another process`);
// Check if it's a mirror now
const recheck = await getGiteaRepoInfo({
config,
owner: starredOrg,
repoName: repository.name,
});
if (recheck && recheck.mirror) {
// It's now a mirror, update database
const { db, repositories: reposTable } = await import("./db");
const { eq } = await import("drizzle-orm");
const { repoStatusEnum } = await import("@/types/Repository");
await db
.update(reposTable)
.set({
status: repoStatusEnum.parse("mirrored"),
updatedAt: new Date(),
lastMirrored: new Date(),
errorMessage: null,
mirroredLocation: `${starredOrg}/${repository.name}`,
})
.where(eq(reposTable.id, repository.id!));
return;
}
}
}
throw error;
}
}
/**
* Sync all starred repositories
*/
export async function syncStarredRepositories({
config,
repositories,
}: {
config: Config;
repositories: Repository[];
}): Promise<void> {
const strategyConfig = getMirrorStrategyConfig();
console.log(`Syncing ${repositories.length} starred repositories`);
await processWithRetry(
repositories,
async (repository) => {
try {
// Import syncGiteaRepo
const { syncGiteaRepo } = await import("./gitea");
await syncGiteaRepo({
config,
repository,
});
return repository;
} catch (error) {
if (error instanceof Error && error.message.includes("not a mirror")) {
console.warn(`Repository ${repository.name} is not a mirror, handling...`);
const starredReposMode = config.githubConfig?.starredReposMode || "dedicated-org";
const starredOrg =
starredReposMode === "preserve-owner"
? repository.organization || repository.owner
: config.githubConfig?.starredReposOrg || "starred";
const repoInfo = await getGiteaRepoInfo({
config,
owner: starredOrg,
repoName: repository.name,
});
if (repoInfo) {
await handleExistingNonMirrorRepo({
config,
repository,
repoInfo,
strategy: strategyConfig.nonMirrorStrategy,
});
}
}
throw error;
}
},
{
concurrencyLimit: strategyConfig.repoBatchSize,
maxRetries: 1,
retryDelay: 1000,
onProgress: (completed, total) => {
const percentComplete = Math.round((completed / total) * 100);
console.log(`Sync progress: ${completed}/${total} (${percentComplete}%)`);
},
}
);
}
+34
View File
@@ -124,3 +124,37 @@ test("githubConfigSchema parses includeCollaboratorRepos with true default", ()
});
expect(parsed.includeCollaboratorRepos).toBe(true);
});
test("skipPersonalRepos defaults to false in githubConfigSchema", () => {
const parsed = githubConfigSchema.parse({
owner: "octo",
type: "personal",
token: "",
});
expect(parsed.skipPersonalRepos).toBe(false);
});
test("skipPersonalRepos round-trips UI -> DB -> UI when true", () => {
const ui = buildMinimalUiConfigs();
const advancedWithSkip: AdvancedOptions = { ...ui.advancedOptions, skipPersonalRepos: true };
const db = mapUiToDbConfig(ui.githubConfig, ui.giteaConfig, ui.mirrorOptions, advancedWithSkip);
expect(db.githubConfig.skipPersonalRepos).toBe(true);
const roundTripped = mapDbToUiConfig({ githubConfig: db.githubConfig, giteaConfig: db.giteaConfig });
expect(roundTripped.advancedOptions.skipPersonalRepos).toBe(true);
});
test("skipPersonalRepos round-trips UI -> DB -> UI when false", () => {
const ui = buildMinimalUiConfigs();
const advancedWithSkip: AdvancedOptions = { ...ui.advancedOptions, skipPersonalRepos: false };
const db = mapUiToDbConfig(ui.githubConfig, ui.giteaConfig, ui.mirrorOptions, advancedWithSkip);
expect(db.githubConfig.skipPersonalRepos).toBe(false);
const roundTripped = mapDbToUiConfig({ githubConfig: db.githubConfig, giteaConfig: db.giteaConfig });
expect(roundTripped.advancedOptions.skipPersonalRepos).toBe(false);
});
test("DB row missing skipPersonalRepos defaults to false on read", () => {
const ui = mapDbToUiConfig({ githubConfig: { owner: "octo", token: "" } });
expect(ui.advancedOptions.skipPersonalRepos).toBe(false);
});
+2
View File
@@ -71,6 +71,7 @@ export function mapUiToDbConfig(
// Advanced options
starredCodeOnly: advancedOptions.starredCodeOnly,
autoMirrorStarred: advancedOptions.autoMirrorStarred ?? false,
skipPersonalRepos: advancedOptions.skipPersonalRepos ?? false,
};
// Map Gitea config to match database schema
@@ -194,6 +195,7 @@ export function mapDbToUiConfig(dbConfig: any): {
// Support both old (skipStarredIssues) and new (starredCodeOnly) field names for backward compatibility
starredCodeOnly: dbConfig.githubConfig?.starredCodeOnly ?? (dbConfig.githubConfig as any)?.skipStarredIssues ?? false,
autoMirrorStarred: dbConfig.githubConfig?.autoMirrorStarred ?? false,
skipPersonalRepos: dbConfig.githubConfig?.skipPersonalRepos ?? false,
};
return {
+420
View File
@@ -0,0 +1,420 @@
import { describe, test, expect } from "bun:test";
import {
normalizeCloneUrl,
cloneUrlsMatch,
isMirrorOfSource,
classifyCandidateName,
findExistingMirror,
} from "./mirror-source-match";
import type { Repository } from "@/lib/db/schema";
import type { Config } from "@/types/config";
// Minimal Repository factory for tests. Only the fields read by the helper
// matter (cloneUrl, mirroredLocation, fullName, name).
function makeRepo(overrides: Partial<Repository> = {}): Repository {
return {
id: "repo-1",
userId: "user-1",
configId: "config-1",
name: "Update",
fullName: "NostalgiaForInfinity/Update",
url: "https://github.com/NostalgiaForInfinity/Update",
cloneUrl: "https://github.com/NostalgiaForInfinity/Update.git",
owner: "NostalgiaForInfinity",
organization: undefined,
mirroredLocation: "",
isPrivate: false,
isForked: false,
forkedFrom: undefined,
hasIssues: false,
isStarred: true,
isArchived: false,
size: 0,
hasLFS: false,
hasSubmodules: false,
language: undefined,
description: undefined,
defaultBranch: "main",
visibility: "public",
status: "imported",
lastMirrored: undefined,
errorMessage: undefined,
createdAt: new Date(),
updatedAt: new Date(),
...overrides,
} as unknown as Repository;
}
const config: Partial<Config> = {
userId: "user-1",
giteaConfig: { url: "https://gitea.example.com", token: "t" } as any,
};
describe("normalizeCloneUrl", () => {
test("strips trailing .git", () => {
expect(normalizeCloneUrl("https://github.com/a/b.git")).toBe(
"https://github.com/a/b"
);
});
test("strips embedded credentials", () => {
expect(normalizeCloneUrl("https://x-access-token:ghp_secret@github.com/a/b.git")).toBe(
"https://github.com/a/b"
);
});
test("strips trailing slash", () => {
expect(normalizeCloneUrl("https://github.com/a/b/")).toBe(
"https://github.com/a/b"
);
});
test("lowercases host (and value)", () => {
expect(normalizeCloneUrl("https://GitHub.com/a/b")).toBe(
"https://github.com/a/b"
);
});
test("returns empty string for blank/invalid input", () => {
expect(normalizeCloneUrl("")).toBe("");
expect(normalizeCloneUrl(null)).toBe("");
expect(normalizeCloneUrl(undefined)).toBe("");
});
test("handles scp-style git URLs via fallback", () => {
expect(normalizeCloneUrl("git@github.com:a/b.git")).toBe("git@github.com:a/b");
});
});
describe("cloneUrlsMatch", () => {
test("https vs token-embedded URL match", () => {
expect(
cloneUrlsMatch(
"https://github.com/a/b.git",
"https://x-access-token:tok@github.com/a/b.git"
)
).toBe(true);
});
test(".git suffix and trailing slash differences match", () => {
expect(
cloneUrlsMatch("https://github.com/a/b", "https://github.com/a/b.git/")
).toBe(true);
});
test("host case-insensitive match", () => {
expect(
cloneUrlsMatch("https://GITHUB.com/a/b", "https://github.com/a/b")
).toBe(true);
});
test("different repos do not match", () => {
expect(
cloneUrlsMatch("https://github.com/a/b", "https://github.com/c/d")
).toBe(false);
});
test("empty/unknown URL never matches", () => {
expect(cloneUrlsMatch("", "https://github.com/a/b")).toBe(false);
expect(cloneUrlsMatch("https://github.com/a/b", undefined)).toBe(false);
});
});
describe("isMirrorOfSource", () => {
test("true when mirror with matching original_url", () => {
expect(
isMirrorOfSource(
{ mirror: true, original_url: "https://github.com/a/b" } as any,
"https://github.com/a/b.git"
)
).toBe(true);
});
test("false when not a mirror", () => {
expect(
isMirrorOfSource(
{ mirror: false, original_url: "https://github.com/a/b" } as any,
"https://github.com/a/b"
)
).toBe(false);
});
test("false when original_url is for a different source (phantom fork)", () => {
expect(
isMirrorOfSource(
{ mirror: true, original_url: "https://github.com/other/repo" } as any,
"https://github.com/a/b"
)
).toBe(false);
});
test("false when original_url missing (cannot confirm)", () => {
expect(
isMirrorOfSource({ mirror: true } as any, "https://github.com/a/b")
).toBe(false);
});
test("false for null repoInfo", () => {
expect(isMirrorOfSource(null, "https://github.com/a/b")).toBe(false);
});
});
describe("findExistingMirror", () => {
test("reuses existing same-source mirror at base candidate name (#315)", async () => {
const repo = makeRepo();
const getRepoInfo = async ({ owner, repoName }: any) => {
if (owner === "starred" && repoName === "Update") {
return {
mirror: true,
original_url: "https://github.com/NostalgiaForInfinity/Update",
} as any;
}
return null;
};
const match = await findExistingMirror({
repository: repo,
config,
candidateOwner: "starred",
candidateName: "Update",
getRepoInfo,
});
expect(match).not.toBeNull();
expect(match!.owner).toBe("starred");
expect(match!.repoName).toBe("Update");
});
test("reuses via mirroredLocation even when base name differs (strategy change, #309)", async () => {
// Strategy changed; current candidate name would be "Update" under "starred",
// but the historical mirror lives at "myorg/Update-NostalgiaForInfinity".
const repo = makeRepo({
mirroredLocation: "myorg/Update-NostalgiaForInfinity",
});
const getRepoInfo = async ({ owner, repoName }: any) => {
if (owner === "myorg" && repoName === "Update-NostalgiaForInfinity") {
return {
mirror: true,
original_url: "https://github.com/NostalgiaForInfinity/Update",
} as any;
}
return null;
};
const match = await findExistingMirror({
repository: repo,
config,
candidateOwner: "starred",
candidateName: "Update",
getRepoInfo,
});
expect(match).not.toBeNull();
expect(match!.owner).toBe("myorg");
expect(match!.repoName).toBe("Update-NostalgiaForInfinity");
});
test("returns null on genuine different-source collision (regression guard #95/#236)", async () => {
const repo = makeRepo();
const getRepoInfo = async ({ owner, repoName }: any) => {
if (owner === "starred" && repoName === "Update") {
// Same name, but it mirrors a DIFFERENT source.
return {
mirror: true,
original_url: "https://github.com/someoneelse/Update",
} as any;
}
return null;
};
const match = await findExistingMirror({
repository: repo,
config,
candidateOwner: "starred",
candidateName: "Update",
getRepoInfo,
});
expect(match).toBeNull();
});
test("returns null for phantom fork (non-mirror at the name)", async () => {
const repo = makeRepo();
const getRepoInfo = async ({ owner, repoName }: any) => {
if (owner === "starred" && repoName === "Update") {
return { mirror: false, original_url: "" } as any;
}
return null;
};
const match = await findExistingMirror({
repository: repo,
config,
candidateOwner: "starred",
candidateName: "Update",
getRepoInfo,
});
expect(match).toBeNull();
});
test("falls back to fresh creation when mirroredLocation is stale (Gitea repo deleted)", async () => {
const repo = makeRepo({ mirroredLocation: "starred/Update" });
// Both the recorded location and the base candidate are gone.
const getRepoInfo = async () => null;
const match = await findExistingMirror({
repository: repo,
config,
candidateOwner: "starred",
candidateName: "Update",
getRepoInfo,
});
expect(match).toBeNull();
});
test("matches mirror even when original_url is token-embedded / .git-suffixed", async () => {
const repo = makeRepo();
const getRepoInfo = async ({ owner, repoName }: any) => {
if (owner === "starred" && repoName === "Update") {
return {
mirror: true,
original_url:
"https://x-access-token:tok@github.com/NostalgiaForInfinity/Update.git",
} as any;
}
return null;
};
const match = await findExistingMirror({
repository: repo,
config,
candidateOwner: "starred",
candidateName: "Update",
getRepoInfo,
});
expect(match).not.toBeNull();
});
test("skips a candidate whose lookup throws and still resolves a later candidate", async () => {
const repo = makeRepo({ mirroredLocation: "myorg/Update" });
const getRepoInfo = async ({ owner }: any) => {
if (owner === "myorg") {
throw new Error("network blip");
}
if (owner === "starred") {
return {
mirror: true,
original_url: "https://github.com/NostalgiaForInfinity/Update",
} as any;
}
return null;
};
const match = await findExistingMirror({
repository: repo,
config,
candidateOwner: "starred",
candidateName: "Update",
getRepoInfo,
});
expect(match).not.toBeNull();
expect(match!.owner).toBe("starred");
});
});
describe("classifyCandidateName — suffix vs reuse decision (#315/#309)", () => {
const SOURCE = "https://github.com/NostalgiaForInfinity/Update.git";
test("free name → available", () => {
expect(
classifyCandidateName({
existsInGitea: false,
claimedByOther: false,
repoInfo: null,
sourceCloneUrl: SOURCE,
})
).toBe("available");
});
test("name occupied by OUR same-source mirror → reusable (no suffix, #315)", () => {
expect(
classifyCandidateName({
existsInGitea: true,
claimedByOther: false,
repoInfo: {
mirror: true,
original_url: "https://github.com/NostalgiaForInfinity/Update",
} as any,
sourceCloneUrl: SOURCE,
})
).toBe("reusable");
});
test("name occupied by a DIFFERENT source → taken (suffix, regression #95/#236)", () => {
expect(
classifyCandidateName({
existsInGitea: true,
claimedByOther: false,
repoInfo: {
mirror: true,
original_url: "https://github.com/someoneelse/Update",
} as any,
sourceCloneUrl: SOURCE,
})
).toBe("taken");
});
test("name occupied by a NON-mirror → taken (phantom-fork guard, #309)", () => {
expect(
classifyCandidateName({
existsInGitea: true,
claimedByOther: false,
repoInfo: { mirror: false, original_url: "" } as any,
sourceCloneUrl: SOURCE,
})
).toBe("taken");
});
test("our same-source mirror but DB-claimed by ANOTHER repo → taken (per-user separation)", () => {
expect(
classifyCandidateName({
existsInGitea: true,
claimedByOther: true,
repoInfo: {
mirror: true,
original_url: "https://github.com/NostalgiaForInfinity/Update",
} as any,
sourceCloneUrl: SOURCE,
})
).toBe("taken");
});
test("free in Gitea but DB-claimed by another concurrent op → taken", () => {
expect(
classifyCandidateName({
existsInGitea: false,
claimedByOther: true,
repoInfo: null,
sourceCloneUrl: SOURCE,
})
).toBe("taken");
});
test("existing mirror but unknown source (no sourceCloneUrl) → taken", () => {
expect(
classifyCandidateName({
existsInGitea: true,
claimedByOther: false,
repoInfo: {
mirror: true,
original_url: "https://github.com/NostalgiaForInfinity/Update",
} as any,
sourceCloneUrl: undefined,
})
).toBe("taken");
});
});
+213
View File
@@ -0,0 +1,213 @@
import type { Config } from "@/types/config";
import type { Repository } from "@/lib/db/schema";
import type { GiteaRepoInfo } from "@/lib/gitea-enhanced";
/**
* Source-identity matching for mirror reuse.
*
* Starred (and other) repos were duplicating on every re-mirror because the
* existence check only asked "does a repo with this name exist?" never
* "is the existing repo a mirror of THIS same GitHub source?". This module
* answers the second question so callers can reuse an existing same-source
* mirror instead of generating a suffixed duplicate. See issues #315 / #309.
*/
/**
* Normalize a git clone URL for source-identity comparison.
* Strips embedded credentials, a trailing ".git", a trailing slash, and
* lowercases the host (hosts are case-insensitive; paths are not). Returns an
* empty string for blank/invalid input so callers can treat it as "unknown".
*/
export function normalizeCloneUrl(rawUrl?: string | null): string {
if (typeof rawUrl !== "string") return "";
let url = rawUrl.trim();
if (!url) return "";
try {
const parsed = new URL(url);
// Drop any embedded credentials (e.g. https://user:token@host/...).
parsed.username = "";
parsed.password = "";
const host = parsed.host.toLowerCase();
// Strip trailing slash(es) first so a ".git/" suffix still normalizes.
const path = parsed.pathname.replace(/\/+$/, "").replace(/\.git$/i, "");
return `${parsed.protocol}//${host}${path}`.toLowerCase();
} catch {
// Fall back to best-effort string normalization for non-standard URLs
// (e.g. scp-style git@host:owner/repo). Strip credentials before "@",
// drop ".git"/trailing slash, and lowercase the whole thing.
url = url.replace(/^([a-z]+:\/\/)[^@/]+@/i, "$1");
url = url.replace(/\/+$/, "").replace(/\.git$/i, "");
return url.toLowerCase();
}
}
/**
* Whether two clone URLs point at the same source repository, ignoring
* credentials, ".git" suffix, trailing slash, and host case.
*/
export function cloneUrlsMatch(a?: string | null, b?: string | null): boolean {
const normA = normalizeCloneUrl(a);
const normB = normalizeCloneUrl(b);
if (!normA || !normB) return false;
return normA === normB;
}
/**
* Whether an existing Gitea repo is a mirror of the given GitHub source.
* Uses Gitea's original_url (the recorded migration source) when present;
* if Gitea didn't expose original_url, we cannot positively confirm the
* source and return false (callers then treat the name as a genuine
* collision rather than risk mapping onto an unrelated repo #309).
*/
export function isMirrorOfSource(
repoInfo: GiteaRepoInfo | null,
sourceCloneUrl?: string | null
): boolean {
if (!repoInfo || !repoInfo.mirror) return false;
return cloneUrlsMatch(repoInfo.original_url, sourceCloneUrl);
}
export type CandidateNameClassification = "available" | "reusable" | "taken";
/**
* Classify a candidate mirror name for the suffix-vs-reuse decision in
* generateUniqueRepoName. Pure (all I/O is pre-resolved by the caller):
* - "available": free in Gitea and not DB-claimed by another repo use it
* - "reusable": occupied in Gitea by a mirror of THIS source, not DB-claimed
* by another repo reuse it (no suffix)
* - "taken": occupied by a different source / non-mirror, or DB-claimed by
* another repo must suffix
*
* A DB claim by a DIFFERENT repo always blocks reuse so two users mirroring the
* same source into a shared org stay separated.
*/
export function classifyCandidateName({
existsInGitea,
claimedByOther,
repoInfo,
sourceCloneUrl,
}: {
existsInGitea: boolean;
claimedByOther: boolean;
repoInfo: GiteaRepoInfo | null;
sourceCloneUrl?: string | null;
}): CandidateNameClassification {
if (existsInGitea) {
if (!claimedByOther && isMirrorOfSource(repoInfo, sourceCloneUrl)) {
return "reusable";
}
return "taken";
}
// Not in Gitea, but possibly claimed in the DB by a concurrent operation.
if (claimedByOther) return "taken";
return "available";
}
export interface ExistingMirrorMatch {
owner: string;
repoName: string;
repoInfo: GiteaRepoInfo;
}
/**
* Resolve an existing same-source mirror for a repository, if one exists.
*
* Resolution order (backward compatible):
* 1. The recorded repository.mirroredLocation if it still resolves to a
* live mirror of THIS source, reuse it even when the base candidate name
* differs from the current naming strategy (handles strategy changes #309).
* 2. The provided candidate owner/name if that resolves to a live mirror of
* THIS source, reuse it (handles the self-collision that drove suffixing #315).
*
* Returns null when no live same-source mirror is found (caller should create
* a fresh mirror, generating a unique name if the candidate name is taken by a
* DIFFERENT source).
*/
export async function findExistingMirror({
repository,
config,
candidateOwner,
candidateName,
getRepoInfo,
}: {
repository: Repository;
config: Partial<Config>;
candidateOwner: string;
candidateName: string;
// Injectable for testing; defaults to the real Gitea lookup.
getRepoInfo?: (args: {
config: Partial<Config>;
owner: string;
repoName: string;
}) => Promise<GiteaRepoInfo | null>;
}): Promise<ExistingMirrorMatch | null> {
const lookup =
getRepoInfo ??
(async (args: {
config: Partial<Config>;
owner: string;
repoName: string;
}) => {
const { getGiteaRepoInfo } = await import("@/lib/gitea-enhanced");
return getGiteaRepoInfo(args);
});
const sourceCloneUrl = repository.cloneUrl;
// Candidate locations to probe, in priority order. Dedupe so we don't issue
// the same HTTP lookup twice when mirroredLocation equals the candidate.
const candidates: Array<{ owner: string; repoName: string }> = [];
const seen = new Set<string>();
const pushCandidate = (owner?: string | null, repoName?: string | null) => {
const o = (owner || "").trim();
const r = (repoName || "").trim();
if (!o || !r) return;
const key = `${o}/${r}`.toLowerCase();
if (seen.has(key)) return;
seen.add(key);
candidates.push({ owner: o, repoName: r });
};
if (repository.mirroredLocation && repository.mirroredLocation.trim()) {
const slashIndex = repository.mirroredLocation.indexOf("/");
if (slashIndex > 0 && slashIndex < repository.mirroredLocation.length - 1) {
pushCandidate(
repository.mirroredLocation.slice(0, slashIndex),
repository.mirroredLocation.slice(slashIndex + 1)
);
}
}
pushCandidate(candidateOwner, candidateName);
for (const candidate of candidates) {
let repoInfo: GiteaRepoInfo | null;
try {
repoInfo = await lookup({
config,
owner: candidate.owner,
repoName: candidate.repoName,
});
} catch (error) {
// A failed lookup (network/auth) should not be mistaken for "no mirror";
// skip this candidate and let the caller fall back to its normal flow.
console.warn(
`[Mirror] Could not look up ${candidate.owner}/${candidate.repoName} while resolving existing mirror for ${repository.fullName}: ${
error instanceof Error ? error.message : String(error)
}`
);
continue;
}
if (isMirrorOfSource(repoInfo, sourceCloneUrl)) {
return {
owner: candidate.owner,
repoName: candidate.repoName,
repoInfo: repoInfo as GiteaRepoInfo,
};
}
}
return null;
}
+89
View File
@@ -0,0 +1,89 @@
import type { APIRoute } from "astro";
import { db, repositories } from "@/lib/db";
import { and, eq, inArray } from "drizzle-orm";
import { repoStatusEnum } from "@/types/Repository";
import { createMirrorJob } from "@/lib/helpers";
import { createSecureErrorResponse } from "@/lib/utils";
import { requireAuthenticatedUserId } from "@/lib/auth-guards";
/**
* POST /api/job/cancel-pending
*
* Sets this user's repositories that are waiting to be mirrored
* (status: "imported" or "failed") to "ignored", preventing the scheduler
* from picking them up. Repos with status "mirroring" or "syncing" are
* left alone because they have in-flight work that cannot be aborted here.
*
* Returns the count of affected repositories and logs one activity entry.
*/
export const POST: APIRoute = async ({ request, locals }) => {
try {
const authResult = await requireAuthenticatedUserId({ request, locals });
if ("response" in authResult) return authResult.response;
const userId = authResult.userId;
// Statuses that represent queued-but-not-started work.
// "imported" → repo was discovered, never mirrored
// "failed" → last mirror attempt failed; scheduler will retry
const cancelableStatuses = ["imported", "failed"] as const;
// Fetch repos to cancel so we can count them and log meaningful details.
const toCancel = await db
.select({ id: repositories.id })
.from(repositories)
.where(
and(
eq(repositories.userId, userId),
inArray(repositories.status, cancelableStatuses),
),
);
const cancelCount = toCancel.length;
if (cancelCount > 0) {
const ids = toCancel.map((r) => r.id);
await db
.update(repositories)
.set({
status: repoStatusEnum.parse("ignored"),
updatedAt: new Date(),
errorMessage: "Cancelled by user — set to ignored via Stop Pending Mirrors.",
})
.where(
and(
eq(repositories.userId, userId),
inArray(repositories.id, ids),
),
);
}
// Log a single activity summarising the bulk action.
await createMirrorJob({
userId,
message: `Stopped pending mirrors: ${cancelCount} repositor${cancelCount === 1 ? "y" : "ies"} set to Ignored`,
details:
cancelCount > 0
? `${cancelCount} repositor${cancelCount === 1 ? "y" : "ies"} with status "imported" or "failed" have been set to "ignored". ` +
`They can be re-enabled from the Repositories page.`
: `No repositories in a pending state were found for this user.`,
status: cancelCount > 0 ? "ignored" : "skipped",
skipDuplicateEvent: false,
skipNotification: true,
});
return new Response(
JSON.stringify({
success: true,
message:
cancelCount > 0
? `${cancelCount} repositor${cancelCount === 1 ? "y has" : "ies have"} been set to Ignored.`
: "No repositories in a pending state were found.",
cancelledCount: cancelCount,
}),
{ status: 200, headers: { "Content-Type": "application/json" } },
);
} catch (error) {
return createSecureErrorResponse(error, "cancel pending mirrors", 500);
}
};
+1
View File
@@ -86,6 +86,7 @@ export interface AdvancedOptions {
skipForks: boolean;
starredCodeOnly: boolean;
autoMirrorStarred?: boolean;
skipPersonalRepos?: boolean;
}
export interface SaveConfigApiRequest {