fix: resume interrupted jobs after startup (not only at boot) (#297)

The previous middleware logic gated recovery behind a
once-per-process flag. After the first request handled the boot-time
recovery check, the resume codepath never fired again — even when
new interruptions appeared later in the process lifetime.

Symptom: a sync that started after server boot, crashed mid-flight
(deadlock retry hitting maxRetries, network blip, container
restart of an upstream service, etc.) and never reached the resume
codepath would sit at `inProgress=true, lastCheckpoint=never`
forever. The periodic detector kept finding the stuck row and
logging `Found 1 interrupted jobs:` on every poll (driven by the
health endpoint via `hasJobsNeedingRecovery`), but the resumer
(`resumeInterruptedJob`) was only invoked from `initializeRecovery`
which the middleware never re-called.

Fix: replace the one-shot gate with an in-flight latch
(`recoveryInFlight`) that's released in a `finally` block. Throttling
of actual recovery work is delegated to the existing 5-minute
`skipIfRecentAttempt` check inside `initializeRecovery()`, which is
the right place for it. `recoveryInitialized` is kept and only used
to control whether the "first run" log lines fire.

Secondary fix: `findInterruptedJobs` previously logged
`Found N interrupted jobs:` unconditionally on every call, including
from passive polls in `hasJobsNeedingRecovery` (which the health
endpoint and middleware probe call frequently). That produced one
log line per poll per stuck job for as long as the job stayed stuck.
Make logging opt-in via a `logFound` parameter, default off; the
active recovery cycle in `initializeRecovery()` opts in so the
operator-facing log still surfaces which jobs are being worked on.

Related: #268 (partially addressed). The PR #280 / v3.15.7 fix was
about a JS scoping bug that made the *initial* migrate call's
catch path crash before transitioning the repo to `failed`. That
was one cause of stuck `mirroring` state; this PR addresses the
follow-on issue that even when a job *is* correctly detectable as
interrupted, the post-startup recovery path never re-engages.

Adds `orchestrator-resume-after-startup.test.ts` using the
structural-source test pattern from
`gitea-mirror-failure-recovery.test.ts` so the four guarantees
(no static gate; in-flight latch released in finally; logging
opt-in; active path opts in) are enforced without heavy mocks.

.github/workflows/docker-build.yml

@@ -51,13 +51,13 @@ jobs:
           ref: ${{ env.SHA }}
 
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0
         with:
           driver-opts: network=host
 
       - name: Log into registry
         if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
-        uses: docker/login-action@v3
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
         with:
           registry: ${{ env.REGISTRY }}
           username: ${{ github.actor }}
@@ -66,7 +66,7 @@ jobs:
       # Login to Docker Hub for Docker Scout (optional - provides better vulnerability data)
       # Add DOCKERHUB_USERNAME and DOCKERHUB_TOKEN secrets to enable this
       - name: Log into Docker Hub
-        uses: docker/login-action@v3
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
         continue-on-error: true
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
@@ -108,7 +108,7 @@ jobs:
       # Extract metadata for Docker
       - name: Extract Docker metadata
         id: meta
-        uses: docker/metadata-action@v5
+        uses: docker/metadata-action@c299e40c65443455700f0fdfc63efafe5b349051 # v5.10.0
         with:
           images: ${{ env.REGISTRY }}/${{ env.IMAGE }}
           labels: |
@@ -124,7 +124,7 @@ jobs:
       # Build and push Docker image
       - name: Build and push Docker image
         id: build-and-push
-        uses: docker/build-push-action@v6
+        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6.19.2
         with:
           context: .
           platforms: linux/amd64,linux/arm64
@@ -139,7 +139,7 @@ jobs:
       # Load image locally for security scanning (PRs only)
       - name: Load image for scanning
         if: github.event_name == 'pull_request'
-        uses: docker/build-push-action@v6
+        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6.19.2
         with:
           context: .
           platforms: linux/amd64
@@ -212,7 +212,7 @@ jobs:
 
       # Docker Scout comprehensive security analysis
       - name: Docker Scout - Vulnerability Analysis & Recommendations
-        uses: docker/scout-action@v1
+        uses: docker/scout-action@bacf462e8d090c09660de30a6ccc718035f961e3 # v1.20.4
         if: github.event_name != 'pull_request'
         with:
           command: cves,recommendations
@@ -226,7 +226,7 @@ jobs:
 
       # Docker Scout for Pull Requests (using local image)
       - name: Docker Scout - Vulnerability Analysis (PR)
-        uses: docker/scout-action@v1
+        uses: docker/scout-action@bacf462e8d090c09660de30a6ccc718035f961e3 # v1.20.4
         if: github.event_name == 'pull_request'
         with:
           command: cves,recommendations
@@ -240,7 +240,7 @@ jobs:
 
       # Compare to latest for PRs and pushes
       - name: Docker Scout - Compare to Latest
-        uses: docker/scout-action@v1
+        uses: docker/scout-action@bacf462e8d090c09660de30a6ccc718035f961e3 # v1.20.4
         if: github.event_name == 'pull_request'
         with:
           command: compare

.github/workflows/nix-build.yml

@@ -38,10 +38,10 @@ jobs:
       - uses: actions/checkout@v4
 
       - name: Install Nix
-        uses: DeterminateSystems/nix-installer-action@main
+        uses: DeterminateSystems/nix-installer-action@ef8a148080ab6020fd15196c2084a2eea5ff2d25 # v22
 
       - name: Setup Nix Cache
-        uses: DeterminateSystems/magic-nix-cache-action@main
+        uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 # v13
 
       - name: Regenerate bun.nix from bun.lock
         run: nix run --accept-flake-config github:nix-community/bun2nix -- -o bun.nix

Dockerfile

@@ -1,6 +1,6 @@
 # syntax=docker/dockerfile:1.4
 
-FROM oven/bun:1.3.13-debian AS base
+FROM oven/bun:1.3.14-debian@sha256:9dba1a1b43ce28c9d7931bfc4eb00feb63b0114720a0277a8f939ae4dfc9db6f AS base
 WORKDIR /app
 RUN apt-get update && apt-get -y upgrade && apt-get install -y --no-install-recommends \
   python3 make g++ gcc wget sqlite3 openssl ca-certificates \
@@ -49,7 +49,7 @@ RUN git clone --branch "v${GIT_LFS_VERSION}" --depth 1 https://github.com/git-lf
   && install -m 755 /tmp/git-lfs/bin/git-lfs /usr/local/bin/git-lfs
 
 # ----------------------------
-FROM oven/bun:1.3.13-debian AS runner
+FROM oven/bun:1.3.14-debian@sha256:9dba1a1b43ce28c9d7931bfc4eb00feb63b0114720a0277a8f939ae4dfc9db6f AS runner
 WORKDIR /app
 RUN apt-get update && apt-get -y upgrade && apt-get install -y --no-install-recommends \
   git wget sqlite3 openssl ca-certificates \

package.json

@@ -1,7 +1,7 @@
 {
   "name": "gitea-mirror",
   "type": "module",
-  "version": "3.15.11",
+  "version": "3.16.0",
   "engines": {
     "bun": ">=1.2.9"
   },

src/components/auth/AccountMenu.tsx

@@ -0,0 +1,134 @@
+import { useEffect, useState } from "react";
+import { toast } from "sonner";
+import { KeyRound, LogOut, Mail } from "lucide-react";
+import { useAuth } from "@/hooks/useAuth";
+import { authClient } from "@/lib/auth-client";
+import { withBase } from "@/lib/base-path";
+import { Avatar, AvatarFallback, AvatarImage } from "@/components/ui/avatar";
+import { Button } from "@/components/ui/button";
+import {
+  DropdownMenu,
+  DropdownMenuContent,
+  DropdownMenuItem,
+  DropdownMenuLabel,
+  DropdownMenuSeparator,
+  DropdownMenuTrigger,
+} from "@/components/ui/dropdown-menu";
+import { ChangePasswordDialog } from "./ChangePasswordDialog";
+import { ChangeEmailDialog } from "./ChangeEmailDialog";
+
+export function AccountMenu() {
+  const { user, logout, refreshUser } = useAuth();
+  const [hasPassword, setHasPassword] = useState<boolean | null>(null);
+  const [passwordOpen, setPasswordOpen] = useState(false);
+  const [emailOpen, setEmailOpen] = useState(false);
+
+  useEffect(() => {
+    if (!user) {
+      setHasPassword(null);
+      return;
+    }
+    let cancelled = false;
+    (async () => {
+      try {
+        const accounts = await authClient.listAccounts();
+        if (cancelled) return;
+        const list = Array.isArray(accounts) ? accounts : accounts?.data;
+        setHasPassword(
+          Array.isArray(list) && list.some((a) => a.providerId === "credential")
+        );
+      } catch {
+        // Fail open: if we can't check, show the option rather than locking the
+        // user out of changing their password.
+        if (!cancelled) setHasPassword(true);
+      }
+    })();
+    return () => {
+      cancelled = true;
+    };
+  }, [user?.id]);
+
+  if (!user) {
+    return (
+      <Button variant="outline" size="sm" asChild>
+        <a href={withBase("/login")}>Login</a>
+      </Button>
+    );
+  }
+
+  const handleLogout = async () => {
+    toast.success("Logged out successfully");
+    await new Promise((resolve) => setTimeout(resolve, 500));
+    logout();
+  };
+
+  return (
+    <>
+      <DropdownMenu>
+        <DropdownMenuTrigger asChild>
+          <Button
+            variant="outline"
+            size="lg"
+            className="relative h-10 w-10 rounded-full p-0"
+          >
+            <Avatar className="h-full w-full">
+              <AvatarImage src={user.image || ""} alt={user.name || user.email} />
+              <AvatarFallback>
+                {(user.name || user.email || "U").charAt(0).toUpperCase()}
+              </AvatarFallback>
+            </Avatar>
+          </Button>
+        </DropdownMenuTrigger>
+        <DropdownMenuContent align="end" className="w-60">
+          <DropdownMenuLabel className="font-normal">
+            <div className="flex flex-col gap-0.5">
+              {user.name && (
+                <span className="text-sm font-medium leading-none">
+                  {user.name}
+                </span>
+              )}
+              <span className="text-xs leading-none text-muted-foreground truncate">
+                {user.email}
+              </span>
+            </div>
+          </DropdownMenuLabel>
+          <DropdownMenuSeparator />
+          {hasPassword && (
+            <DropdownMenuItem
+              onSelect={() => setPasswordOpen(true)}
+              className="cursor-pointer"
+            >
+              <KeyRound className="h-4 w-4 mr-2" />
+              Change password
+            </DropdownMenuItem>
+          )}
+          <DropdownMenuItem
+            onSelect={() => setEmailOpen(true)}
+            className="cursor-pointer"
+          >
+            <Mail className="h-4 w-4 mr-2" />
+            Change email
+          </DropdownMenuItem>
+          <DropdownMenuSeparator />
+          <DropdownMenuItem onSelect={handleLogout} className="cursor-pointer">
+            <LogOut className="h-4 w-4 mr-2" />
+            Logout
+          </DropdownMenuItem>
+        </DropdownMenuContent>
+      </DropdownMenu>
+
+      {hasPassword && (
+        <ChangePasswordDialog
+          open={passwordOpen}
+          onOpenChange={setPasswordOpen}
+        />
+      )}
+      <ChangeEmailDialog
+        open={emailOpen}
+        onOpenChange={setEmailOpen}
+        currentEmail={user.email}
+        onUpdated={refreshUser}
+      />
+    </>
+  );
+}

src/components/auth/ChangeEmailDialog.tsx

@@ -0,0 +1,110 @@
+import { useState } from "react";
+import { toast } from "sonner";
+import { authClient } from "@/lib/auth-client";
+import { Button } from "@/components/ui/button";
+import { Input } from "@/components/ui/input";
+import { Label } from "@/components/ui/label";
+import {
+  Dialog,
+  DialogContent,
+  DialogDescription,
+  DialogFooter,
+  DialogHeader,
+  DialogTitle,
+} from "@/components/ui/dialog";
+
+interface ChangeEmailDialogProps {
+  open: boolean;
+  onOpenChange: (open: boolean) => void;
+  currentEmail: string;
+  onUpdated?: () => void;
+}
+
+export function ChangeEmailDialog({
+  open,
+  onOpenChange,
+  currentEmail,
+  onUpdated,
+}: ChangeEmailDialogProps) {
+  const [newEmail, setNewEmail] = useState("");
+  const [isSubmitting, setIsSubmitting] = useState(false);
+
+  const reset = () => {
+    setNewEmail("");
+    setIsSubmitting(false);
+  };
+
+  const handleOpenChange = (next: boolean) => {
+    if (!next) reset();
+    onOpenChange(next);
+  };
+
+  const handleSubmit = async (e: React.FormEvent) => {
+    e.preventDefault();
+    const trimmed = newEmail.trim();
+    if (!trimmed) {
+      toast.error("Please enter a new email");
+      return;
+    }
+    if (trimmed.toLowerCase() === currentEmail.toLowerCase()) {
+      toast.error("New email must differ from current email");
+      return;
+    }
+
+    setIsSubmitting(true);
+    try {
+      const { error } = await authClient.changeEmail({ newEmail: trimmed });
+      if (error) {
+        toast.error(error.message || "Failed to change email");
+        return;
+      }
+      toast.success("Email updated.");
+      onUpdated?.();
+      handleOpenChange(false);
+    } catch (err) {
+      toast.error(err instanceof Error ? err.message : "Failed to change email");
+    } finally {
+      setIsSubmitting(false);
+    }
+  };
+
+  return (
+    <Dialog open={open} onOpenChange={handleOpenChange}>
+      <DialogContent className="sm:max-w-md">
+        <DialogHeader>
+          <DialogTitle>Change email</DialogTitle>
+          <DialogDescription>
+            Current: <span className="font-medium">{currentEmail}</span>
+          </DialogDescription>
+        </DialogHeader>
+        <form onSubmit={handleSubmit} className="space-y-4">
+          <div className="space-y-2">
+            <Label htmlFor="new-email">New email</Label>
+            <Input
+              id="new-email"
+              type="email"
+              autoComplete="email"
+              value={newEmail}
+              onChange={(e) => setNewEmail(e.target.value)}
+              disabled={isSubmitting}
+              required
+            />
+          </div>
+          <DialogFooter>
+            <Button
+              type="button"
+              variant="outline"
+              onClick={() => handleOpenChange(false)}
+              disabled={isSubmitting}
+            >
+              Cancel
+            </Button>
+            <Button type="submit" disabled={isSubmitting}>
+              {isSubmitting ? "Updating..." : "Update email"}
+            </Button>
+          </DialogFooter>
+        </form>
+      </DialogContent>
+    </Dialog>
+  );
+}

src/components/auth/ChangePasswordDialog.tsx

@@ -0,0 +1,158 @@
+import { useState } from "react";
+import { toast } from "sonner";
+import { authClient } from "@/lib/auth-client";
+import { Button } from "@/components/ui/button";
+import { Input } from "@/components/ui/input";
+import { Label } from "@/components/ui/label";
+import { Checkbox } from "@/components/ui/checkbox";
+import {
+  Dialog,
+  DialogContent,
+  DialogDescription,
+  DialogFooter,
+  DialogHeader,
+  DialogTitle,
+} from "@/components/ui/dialog";
+
+interface ChangePasswordDialogProps {
+  open: boolean;
+  onOpenChange: (open: boolean) => void;
+}
+
+export function ChangePasswordDialog({ open, onOpenChange }: ChangePasswordDialogProps) {
+  const [currentPassword, setCurrentPassword] = useState("");
+  const [newPassword, setNewPassword] = useState("");
+  const [confirmPassword, setConfirmPassword] = useState("");
+  const [revokeOtherSessions, setRevokeOtherSessions] = useState(true);
+  const [isSubmitting, setIsSubmitting] = useState(false);
+
+  const reset = () => {
+    setCurrentPassword("");
+    setNewPassword("");
+    setConfirmPassword("");
+    setRevokeOtherSessions(true);
+    setIsSubmitting(false);
+  };
+
+  const handleOpenChange = (next: boolean) => {
+    if (!next) reset();
+    onOpenChange(next);
+  };
+
+  const handleSubmit = async (e: React.FormEvent) => {
+    e.preventDefault();
+
+    if (!currentPassword || !newPassword) {
+      toast.error("Please fill in all fields");
+      return;
+    }
+    if (newPassword !== confirmPassword) {
+      toast.error("New passwords do not match");
+      return;
+    }
+    if (newPassword === currentPassword) {
+      toast.error("New password must differ from current password");
+      return;
+    }
+
+    setIsSubmitting(true);
+    try {
+      const { error } = await authClient.changePassword({
+        currentPassword,
+        newPassword,
+        revokeOtherSessions,
+      });
+      if (error) {
+        toast.error(error.message || "Failed to change password");
+        return;
+      }
+      toast.success(
+        revokeOtherSessions
+          ? "Password updated. Other sessions signed out."
+          : "Password updated."
+      );
+      handleOpenChange(false);
+    } catch (err) {
+      toast.error(err instanceof Error ? err.message : "Failed to change password");
+    } finally {
+      setIsSubmitting(false);
+    }
+  };
+
+  return (
+    <Dialog open={open} onOpenChange={handleOpenChange}>
+      <DialogContent className="sm:max-w-md">
+        <DialogHeader>
+          <DialogTitle>Change password</DialogTitle>
+          <DialogDescription>
+            Enter your current password and a new one. You'll stay signed in on this device.
+          </DialogDescription>
+        </DialogHeader>
+        <form onSubmit={handleSubmit} className="space-y-4">
+          <div className="space-y-2">
+            <Label htmlFor="current-password">Current password</Label>
+            <Input
+              id="current-password"
+              type="password"
+              autoComplete="current-password"
+              value={currentPassword}
+              onChange={(e) => setCurrentPassword(e.target.value)}
+              disabled={isSubmitting}
+              required
+            />
+          </div>
+          <div className="space-y-2">
+            <Label htmlFor="new-password">New password</Label>
+            <Input
+              id="new-password"
+              type="password"
+              autoComplete="new-password"
+              value={newPassword}
+              onChange={(e) => setNewPassword(e.target.value)}
+              disabled={isSubmitting}
+              required
+              minLength={8}
+            />
+          </div>
+          <div className="space-y-2">
+            <Label htmlFor="confirm-password">Confirm new password</Label>
+            <Input
+              id="confirm-password"
+              type="password"
+              autoComplete="new-password"
+              value={confirmPassword}
+              onChange={(e) => setConfirmPassword(e.target.value)}
+              disabled={isSubmitting}
+              required
+              minLength={8}
+            />
+          </div>
+          <div className="flex items-center gap-2">
+            <Checkbox
+              id="revoke-sessions"
+              checked={revokeOtherSessions}
+              onCheckedChange={(checked) => setRevokeOtherSessions(checked === true)}
+              disabled={isSubmitting}
+            />
+            <Label htmlFor="revoke-sessions" className="text-sm font-normal cursor-pointer">
+              Sign out other devices
+            </Label>
+          </div>
+          <DialogFooter>
+            <Button
+              type="button"
+              variant="outline"
+              onClick={() => handleOpenChange(false)}
+              disabled={isSubmitting}
+            >
+              Cancel
+            </Button>
+            <Button type="submit" disabled={isSubmitting}>
+              {isSubmitting ? "Updating..." : "Update password"}
+            </Button>
+          </DialogFooter>
+        </form>
+      </DialogContent>
+    </Dialog>
+  );
+}

src/components/layout/Header.tsx

@@ -2,18 +2,11 @@ import { useAuth } from "@/hooks/useAuth";
 import { Button } from "@/components/ui/button";
 
 import { ModeToggle } from "@/components/theme/ModeToggle";
-import { Avatar, AvatarFallback, AvatarImage } from "../ui/avatar";
-import { toast } from "sonner";
 import { Skeleton } from "@/components/ui/skeleton";
 import { useLiveRefresh } from "@/hooks/useLiveRefresh";
 import { useConfigStatus } from "@/hooks/useConfigStatus";
-import { Menu, LogOut, PanelRightOpen, PanelRightClose } from "lucide-react";
-import {
-  DropdownMenu,
-  DropdownMenuContent,
-  DropdownMenuItem,
-  DropdownMenuTrigger,
-} from "@/components/ui/dropdown-menu";
+import { PanelRightOpen, PanelRightClose } from "lucide-react";
+import { AccountMenu } from "@/components/auth/AccountMenu";
 import { withBase } from "@/lib/base-path";
 
 interface HeaderProps {
@@ -26,7 +19,7 @@ interface HeaderProps {
 }
 
 export function Header({ currentPage, onNavigate, onMenuClick, onToggleCollapse, isSidebarCollapsed, isSidebarOpen }: HeaderProps) {
-  const { user, logout, isLoading } = useAuth();
+  const { isLoading } = useAuth();
   const { isLiveEnabled, toggleLive } = useLiveRefresh();
   const { isFullyConfigured, isLoading: configLoading } = useConfigStatus();
 
@@ -47,13 +40,6 @@ export function Header({ currentPage, onNavigate, onMenuClick, onToggleCollapse,
     return isLiveEnabled ? 'Disable live refresh' : 'Enable live refresh';
   };
 
-  const handleLogout = async () => {
-    toast.success("Logged out successfully");
-    // Small delay to show the toast before redirecting
-    await new Promise((resolve) => setTimeout(resolve, 500));
-    logout();
-  };
-
   // Auth buttons skeleton loader
   function AuthButtonsSkeleton() {
     return (
@@ -141,32 +127,7 @@ export function Header({ currentPage, onNavigate, onMenuClick, onToggleCollapse,
 
           <ModeToggle />
 
-          {isLoading ? (
-            <AuthButtonsSkeleton />
-          ) : user ? (
-            <DropdownMenu>
-              <DropdownMenuTrigger asChild>
-                <Button variant="outline" size="lg" className="relative h-10 w-10 rounded-full p-0">
-                  <Avatar className="h-full w-full">
-                    <AvatarImage src={user.image || ""} alt={user.name || user.email} />
-                    <AvatarFallback>
-                      {(user.name || user.email || "U").charAt(0).toUpperCase()}
-                    </AvatarFallback>
-                  </Avatar>
-                </Button>
-              </DropdownMenuTrigger>
-              <DropdownMenuContent align="end" className="w-48">
-                <DropdownMenuItem onClick={handleLogout} className="cursor-pointer">
-                  <LogOut className="h-4 w-4 mr-2" />
-                  Logout
-                </DropdownMenuItem>
-              </DropdownMenuContent>
-            </DropdownMenu>
-          ) : (
-            <Button variant="outline" size="sm" asChild>
-              <a href={withBase('/login')}>Login</a>
-            </Button>
-          )}
+          {isLoading ? <AuthButtonsSkeleton /> : <AccountMenu />}
         </div>
       </div>
     </header>

src/lib/auth.ts

@@ -145,6 +145,12 @@ export const auth = betterAuth({
         input: false, // Don't show in signup form - we'll derive from email
       }
     },
+    changeEmail: {
+      enabled: true,
+      // Email verification isn't wired up (sendResetPassword is a TODO),
+      // so allow direct updates. Safe here because emails stay unverified.
+      updateEmailWithoutVerification: true,
+    },
   },
 
   // Plugins configuration

src/lib/gitea-issue-dedup-on-retry.test.ts

@@ -0,0 +1,282 @@
+/**
+ * Regression test for duplicate-issue creation on retry-after-deadlock.
+ *
+ * `mirrorGitRepoIssuesToGitea` pre-fetches all existing Gitea issues
+ * into `giteaIssueByGitHubNumber` ONCE at function entry, then iterates
+ * per-issue via `processWithRetry`. Each iteration uses the cached map
+ * to decide CREATE vs PATCH.
+ *
+ * The bug: when Gitea's CreateIssue handler commits the issue insert
+ * in one transaction and then deadlocks on the addLabel / repository
+ * counter update in a second transaction, the issue row is committed
+ * and visible — but the in-memory map is never refreshed between
+ * retries. So `processWithRetry` would call the callback again,
+ * `existingIssue` is still `undefined` from the stale map, and a fresh
+ * `httpPost` creates a duplicate.
+ *
+ * Reproduces deterministically on MySQL (1213/40001) and PostgreSQL
+ * (40P01). SQLite escapes because writes serialize globally.
+ *
+ * This test asserts on the *structure* of the source rather than
+ * invoking the function, because behavioral tests for the issue-mirror
+ * pipeline require heavy module mocks that pollute other test files
+ * (bun's mock.module is process-wide). See
+ * `gitea-mirror-failure-recovery.test.ts` for the same convention.
+ *
+ * The two structural guarantees this test enforces:
+ *   (1) Before the create-issue httpPost call, the code performs a
+ *       defensive recheck via httpGet that queries Gitea by
+ *       `[GH-ISSUE #N]` title marker — handles "previous attempt
+ *       committed the issue then threw" scenarios.
+ *   (2) After a successful httpPost create, the new issue is written
+ *       back into `giteaIssueByGitHubNumber` — handles "this attempt
+ *       created the issue, but a later step in the same callback
+ *       (e.g. comment sync) throws and triggers another retry"
+ *       scenarios.
+ */
+import { describe, test, expect } from "bun:test";
+import { readFileSync } from "node:fs";
+import { join } from "node:path";
+
+const SOURCE = readFileSync(join(import.meta.dir, "gitea.ts"), "utf8");
+
+/**
+ * Locate the body of a function declaration by name. Walks from the
+ * declaration, balances parens to skip the parameter list (which can
+ * contain destructured object literals with their own braces), then
+ * finds the body's opening brace and its matching close.
+ *
+ * Same helper as in `gitea-mirror-failure-recovery.test.ts`; kept
+ * local to keep this test file self-contained.
+ */
+function extractFunctionBody(source: string, declarationStart: RegExp): string {
+  const match = source.match(declarationStart);
+  if (!match) {
+    throw new Error(`Could not locate declaration ${declarationStart}`);
+  }
+  let i = match.index! + match[0].length;
+  while (i < source.length && source[i] !== "(") i++;
+  if (source[i] !== "(") {
+    throw new Error(`No '(' after ${declarationStart}`);
+  }
+  let parenDepth = 0;
+  for (; i < source.length; i++) {
+    if (source[i] === "(") parenDepth++;
+    else if (source[i] === ")") {
+      parenDepth--;
+      if (parenDepth === 0) {
+        i++;
+        break;
+      }
+    }
+  }
+  while (i < source.length && source[i] !== "{") i++;
+  if (source[i] !== "{") {
+    throw new Error(`No body '{' for ${declarationStart}`);
+  }
+  let braceDepth = 0;
+  const startIdx = i;
+  for (; i < source.length; i++) {
+    if (source[i] === "{") braceDepth++;
+    else if (source[i] === "}") {
+      braceDepth--;
+      if (braceDepth === 0) {
+        return source.slice(startIdx, i + 1);
+      }
+    }
+  }
+  throw new Error(`Unterminated body for ${declarationStart}`);
+}
+
+describe("issue dedup on retry-after-deadlock", () => {
+  const body = extractFunctionBody(
+    SOURCE,
+    /export const mirrorGitRepoIssuesToGitea = async\b/
+  );
+
+  test("body contains the per-issue create branch we expect to guard", () => {
+    // Sanity: make sure the test is looking at the right code path.
+    // If these strings disappear due to a refactor, this test should
+    // fail loudly so a human reviews whether the dedup guarantees
+    // still hold in the new shape.
+    expect(
+      body.includes(
+        "giteaIssueByGitHubNumber.get(issue.number)"
+      ),
+      "expected the per-issue lookup against giteaIssueByGitHubNumber"
+    ).toBe(true);
+    expect(
+      body.match(/await httpPost\(\s*`\$\{config\.giteaConfig!\.url\}\/api\/v1\/repos\/\$\{giteaOwner\}\/\$\{repoName\}\/issues`/),
+      "expected the create-issue httpPost call"
+    ).toBeTruthy();
+  });
+
+  test("defensive recheck via httpGet runs BEFORE the create httpPost", () => {
+    // The recheck must query Gitea by [GH-ISSUE #N] marker to catch
+    // the partial-commit case. Without it, a deadlock-after-insert
+    // returns 5xx, processWithRetry re-runs the callback, and the
+    // create call produces a duplicate row.
+    const recheckIdx = body.search(
+      /await httpGet\([^)]*\[GH-ISSUE #\$\{issue\.number\}\]/
+    );
+    expect(
+      recheckIdx,
+      "defensive recheck via httpGet using [GH-ISSUE #N] marker must exist"
+    ).toBeGreaterThanOrEqual(0);
+
+    // The recheck must come before the create httpPost in source order.
+    // The first httpPost on the .../issues endpoint inside this
+    // function body is the create call; we anchor against it.
+    const createIdx = body.search(
+      /await httpPost\(\s*`\$\{config\.giteaConfig!\.url\}\/api\/v1\/repos\/\$\{giteaOwner\}\/\$\{repoName\}\/issues`/
+    );
+    expect(createIdx, "create httpPost call must exist").toBeGreaterThanOrEqual(0);
+
+    expect(
+      recheckIdx,
+      "the recheck must run BEFORE the create call so it can short-circuit on partial-commit duplicates"
+    ).toBeLessThan(createIdx);
+  });
+
+  test("recheck hit short-circuits via PATCH and updates the cache", () => {
+    // When the recheck finds a hit (i.e. a previous failed attempt
+    // already created this issue), the code should:
+    //   - cache the hit into giteaIssueByGitHubNumber so subsequent
+    //     retries within this run also find it
+    //   - go down the PATCH path (httpPatch) instead of httpPost
+    //   - log a recognisable line so operators can spot recovery
+    expect(
+      body.includes(
+        "giteaIssueByGitHubNumber.set(issue.number, recheckHit)"
+      ) ||
+        body.match(/giteaIssueByGitHubNumber\.set\(\s*issue\.number\s*,\s*recheckHit/),
+      "recheck hit must be written back into giteaIssueByGitHubNumber"
+    ).toBeTruthy();
+
+    expect(
+      body.match(/Recovered orphan from prior failed attempt/i),
+      "a log line should make the recovery path visible in operator logs"
+    ).toBeTruthy();
+  });
+
+  test("pre-fetch issues pagination uses Link header (not short-page heuristic)", () => {
+    // The previous `if (pageIssues.length < issuesPerPage) break;`
+    // heuristic was wrong in both directions:
+    //   - Gitea caps response size at `[api].MAX_RESPONSE_ITEMS`
+    //     (default 50), typically lower than `issuesPerPage` (100),
+    //     so the very first page already looks "short" and
+    //     pagination terminated after 50 items. Every issue past
+    //     that was misclassified as new and duplicated on every sync.
+    //   - Naive removal of that break, relying only on "break on
+    //     empty page", can loop forever because some Gitea endpoints
+    //     return the same data on every page when asked for a page
+    //     past the actual end (instead of returning []).
+    //
+    // The correct fix is to use the Link header (RFC 5988): if
+    // `rel="next"` is absent, we're done.
+    //
+    // This test asserts:
+    //   - the broken short-page check is gone
+    //   - the existing-issues loop checks the Link header for next
+    const issuesPaginationRegion = body.substring(
+      body.indexOf("existingGiteaIssues.push"),
+      body.indexOf("issuesPage += 1") + 30
+    );
+    expect(
+      issuesPaginationRegion,
+      "issues pagination region should be present"
+    ).not.toBe("");
+    expect(
+      /\bpageIssues\.length\s*<\s*issuesPerPage\b/.test(issuesPaginationRegion),
+      "the short-page break (pageIssues.length < issuesPerPage) must be removed"
+    ).toBe(false);
+    expect(
+      /existingIssuesRes\.headers\.get\(\s*["']link["']\s*\)/.test(
+        issuesPaginationRegion
+      ) && /rel="next"/.test(issuesPaginationRegion),
+      "the issues pagination loop must use the Link header (rel=\"next\") " +
+        "to decide whether to fetch the next page"
+    ).toBe(true);
+  });
+
+  test("per-issue comments pagination also uses Link header (not short-page heuristic)", () => {
+    // Same correctness concerns as issues pagination above. The per-
+    // issue comments endpoint is subject to the same Gitea page-size
+    // cap, and naive empty-page detection has the same risk.
+    expect(
+      /\bpageComments\.length\s*<\s*commentsPerPage\b/.test(body),
+      "the short-page break (pageComments.length < commentsPerPage) must be removed"
+    ).toBe(false);
+    // Look only at the comments-fetch region (not the whole file) so
+    // a future caller using a different response variable name in
+    // another place won't false-positive this assertion.
+    const commentsRegion = body.substring(
+      body.indexOf("existingComments.push"),
+      body.indexOf("commentsPage += 1") + 30
+    );
+    expect(
+      commentsRegion,
+      "comments pagination region should be present"
+    ).not.toBe("");
+    expect(
+      /existingCommentsRes\.headers\.get\(\s*["']link["']\s*\)/.test(
+        commentsRegion
+      ) && /rel="next"/.test(commentsRegion),
+      "the comments pagination loop must use the Link header (rel=\"next\") " +
+        "to decide whether to fetch the next page"
+    ).toBe(true);
+  });
+
+  describe("PR mirror has the same guarantees", () => {
+    // mirrorGitRepoPullRequestsToGitea has parallel structure:
+    // - pre-fetches existing Gitea "issues that are mirrored PRs",
+    //   keyed by `[PR #N]` marker in title
+    // - per-PR callback decides PATCH vs CREATE
+    // - same Gitea-side pagination cap and deadlock-after-commit
+    //   risks apply
+    // The fix mirrors gitea-issues here.
+    const prBody = extractFunctionBody(
+      SOURCE,
+      /export async function mirrorGitRepoPullRequestsToGitea\b/
+    );
+
+    test("PR pre-fetch pagination uses Link header", () => {
+      expect(
+        /\bpageIssues\.length\s*<\s*prIssuesPerPage\b/.test(prBody),
+        "the short-page break (pageIssues.length < prIssuesPerPage) must be removed"
+      ).toBe(false);
+      // The PR pre-fetch reuses the existingIssuesRes variable name
+      expect(
+        /existingIssuesRes\.headers\.get\(\s*["']link["']\s*\)/.test(prBody) &&
+          /rel="next"/.test(prBody),
+        "the PR pre-fetch loop must use the Link header (rel=\"next\")"
+      ).toBe(true);
+    });
+
+    test("PR create path defensively rechecks via [PR #N] before httpPost", () => {
+      // Both the enriched and basic-fallback create paths must have
+      // a recheck so partial-commit retries don't duplicate the PR.
+      const rechecks =
+        prBody.match(/Recovered orphan from prior failed attempt for PR/g) ||
+        [];
+      expect(
+        rechecks.length,
+        "expected at least two 'Recovered orphan' log lines " +
+          "(one for the enriched create path, one for the basic-fallback path)"
+      ).toBeGreaterThanOrEqual(2);
+    });
+  });
+
+  test("successful create caches the new issue into the dedup map", () => {
+    // Without this, a retry triggered by a *later* step in the same
+    // per-issue callback (e.g. comment sync throwing) would re-enter
+    // the create path on the next attempt — same root duplication
+    // pattern, different trigger.
+    expect(
+      body.match(
+        /giteaIssueByGitHubNumber\.set\(\s*issue\.number\s*,\s*createdIssue\.data\s*\)/
+      ),
+      "after a successful create, the new issue must be stored in giteaIssueByGitHubNumber"
+    ).toBeTruthy();
+  });
+});

src/lib/gitea.ts

@@ -2146,7 +2146,21 @@ export const mirrorGitRepoIssuesToGitea = async ({
     if (!pageIssues.length) break;
 
     existingGiteaIssues.push(...pageIssues);
-    if (pageIssues.length < issuesPerPage) break;
+
+    // Use the Link header (RFC 5988) to decide whether more pages
+    // exist. The old short-page-length heuristic was wrong in both
+    // directions:
+    //   - Gitea caps response size at `[api].MAX_RESPONSE_ITEMS`
+    //     (default 50), typically lower than `issuesPerPage` (100),
+    //     so the very first page already looks "short" and
+    //     pagination terminated after 50 items — every issue past
+    //     that was misclassified as new and duplicated on every sync.
+    //   - For some endpoints Gitea returns the same data on every
+    //     page when the page is past the end, so a naive "break on
+    //     empty" alone can loop forever if the server doesn't return
+    //     []. Link header is the safe signal.
+    const linkHeader = existingIssuesRes.headers.get("link") || "";
+    if (!/\brel="next"/.test(linkHeader)) break;
     issuesPage += 1;
   }
 
@@ -2289,30 +2303,85 @@ export const mirrorGitRepoIssuesToGitea = async ({
           }
         );
       } else {
-        const createdIssue = await httpPost(
-          `${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues`,
-          issuePayload,
-          {
-            Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
-          }
-        );
-        targetIssueNumber = createdIssue.data.number;
+        // Defensive recheck before create: a previous retry attempt may
+        // have already created this issue and then thrown. The common
+        // trigger is Gitea's CreateIssue handler committing the issue
+        // insert in one transaction and then deadlocking on the
+        // addLabel / repository counter update in a second transaction.
+        // The issue row is committed and visible, but the in-memory
+        // giteaIssueByGitHubNumber map (built once at function entry)
+        // doesn't know about it, so without this check processWithRetry
+        // would create a duplicate every time the create returns 5xx
+        // after a partial commit.
+        //
+        // Reproduces deterministically on MySQL (Error 1213 / 40001)
+        // and PostgreSQL (40P01); SQLite escapes because writes
+        // serialize globally.
+        let recheckHit: any = null;
+        try {
+          const recheck = await httpGet(
+            `${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues?state=all&type=issues&q=${encodeURIComponent(`[GH-ISSUE #${issue.number}]`)}`,
+            {
+              Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
+            }
+          );
+          const candidates = Array.isArray(recheck.data) ? recheck.data : [];
+          recheckHit = candidates.find(
+            (c: any) => extractGitHubIssueNumber(c.title) === issue.number
+          ) ?? null;
+        } catch (_recheckErr) {
+          // Best-effort; fall through to create.
+        }
 
-        if (issue.state === "closed" && createdIssue.data.state !== "closed") {
-          try {
-            await httpPatch(
-              `${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${targetIssueNumber}`,
-              { state: "closed" },
-              {
-                Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
-              }
-            );
-          } catch (closeError) {
-            console.error(
-              `[Issues] Failed to close issue #${targetIssueNumber}: ${
-                closeError instanceof Error ? closeError.message : String(closeError)
-              }`
-            );
+        if (recheckHit) {
+          giteaIssueByGitHubNumber.set(issue.number, recheckHit);
+          existingIssue = recheckHit;
+          targetIssueNumber = recheckHit.number;
+          console.log(
+            `[Issues] Recovered orphan from prior failed attempt for #${issue.number}; switching to PATCH`
+          );
+          await httpPatch(
+            `${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${targetIssueNumber}`,
+            {
+              title: issuePayload.title,
+              body: issuePayload.body,
+              state: issue.state === "closed" ? "closed" : "open",
+              labels: issuePayload.labels,
+            },
+            {
+              Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
+            }
+          );
+        } else {
+          const createdIssue = await httpPost(
+            `${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues`,
+            issuePayload,
+            {
+              Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
+            }
+          );
+          targetIssueNumber = createdIssue.data.number;
+          // Cache the new issue immediately so a subsequent retry of
+          // this callback (e.g. triggered by a later step like comment
+          // sync failing) doesn't lose track of it.
+          giteaIssueByGitHubNumber.set(issue.number, createdIssue.data);
+
+          if (issue.state === "closed" && createdIssue.data.state !== "closed") {
+            try {
+              await httpPatch(
+                `${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${targetIssueNumber}`,
+                { state: "closed" },
+                {
+                  Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
+                }
+              );
+            } catch (closeError) {
+              console.error(
+                `[Issues] Failed to close issue #${targetIssueNumber}: ${
+                  closeError instanceof Error ? closeError.message : String(closeError)
+                }`
+              );
+            }
           }
         }
       }
@@ -2355,7 +2424,13 @@ export const mirrorGitRepoIssuesToGitea = async ({
             : [];
           if (!pageComments.length) break;
           existingComments.push(...pageComments);
-          if (pageComments.length < commentsPerPage) break;
+          // Use the Link header to decide whether more pages exist.
+          // See note on the existing-issues pagination above; the
+          // same Gitea behaviors (MAX_RESPONSE_ITEMS cap and
+          // repeated-data on out-of-bound pages) apply here.
+          const commentsLinkHeader =
+            existingCommentsRes.headers.get("link") || "";
+          if (!/\brel="next"/.test(commentsLinkHeader)) break;
           commentsPage += 1;
         }
         const mirroredCommentIds = new Set<number>();
@@ -2983,7 +3058,12 @@ export async function mirrorGitRepoPullRequestsToGitea({
       }
     }
 
-    if (pageIssues.length < prIssuesPerPage) break;
+    // See note on the existing-issues pre-fetch above: rely on Link
+    // header (RFC 5988) rather than short-page heuristic. Gitea caps
+    // page size at MAX_RESPONSE_ITEMS (default 50), and some
+    // endpoints repeat data on out-of-bound pages instead of [].
+    const linkHeader = existingIssuesRes.headers.get("link") || "";
+    if (!/\brel="next"/.test(linkHeader)) break;
     prIssuesPage += 1;
   }
 
@@ -3084,7 +3164,36 @@ export async function mirrorGitRepoPullRequestsToGitea({
           closed: pr.state === "closed" || pr.merged_at !== null,
         };
 
-        const existingPrIssue = existingPrIssuesByNumber.get(pr.number);
+        let existingPrIssue = existingPrIssuesByNumber.get(pr.number);
+        // Defensive recheck (see same pattern in mirrorGitRepoIssuesToGitea):
+        // a previous attempt may have committed the PR-issue row and
+        // then thrown on the addLabel/repository-counter update. The
+        // pre-fetched map doesn't know about it, so without this check
+        // processWithRetry would create a duplicate every retry.
+        if (!existingPrIssue) {
+          try {
+            const recheck = await httpGet(
+              `${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues?state=all&type=issues&q=${encodeURIComponent(`[PR #${pr.number}]`)}`,
+              {
+                Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
+              }
+            );
+            const candidates = Array.isArray(recheck.data) ? recheck.data : [];
+            const hit = candidates.find((c: any) => {
+              const m = String(c.title || "").match(/\[PR #(\d+)\]/i);
+              return m && Number.parseInt(m[1], 10) === pr.number;
+            });
+            if (hit) {
+              existingPrIssue = hit;
+              existingPrIssuesByNumber.set(pr.number, hit);
+              console.log(
+                `[Pull Requests] Recovered orphan from prior failed attempt for PR #${pr.number}; switching to PATCH`
+              );
+            }
+          } catch (_recheckErr) {
+            // Best-effort; fall through to create.
+          }
+        }
         if (existingPrIssue) {
           await httpPatch(
             `${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${existingPrIssue.number}`,
@@ -3145,7 +3254,34 @@ export async function mirrorGitRepoPullRequestsToGitea({
         };
         
         try {
-          const existingPrIssue = existingPrIssuesByNumber.get(pr.number);
+          let existingPrIssue = existingPrIssuesByNumber.get(pr.number);
+          // Defensive recheck — same pattern as the enriched create
+          // branch above. Without this, the basic-info fallback would
+          // dup on retry-after-deadlock just like the enriched path.
+          if (!existingPrIssue) {
+            try {
+              const recheck = await httpGet(
+                `${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues?state=all&type=issues&q=${encodeURIComponent(`[PR #${pr.number}]`)}`,
+                {
+                  Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
+                }
+              );
+              const candidates = Array.isArray(recheck.data) ? recheck.data : [];
+              const hit = candidates.find((c: any) => {
+                const m = String(c.title || "").match(/\[PR #(\d+)\]/i);
+                return m && Number.parseInt(m[1], 10) === pr.number;
+              });
+              if (hit) {
+                existingPrIssue = hit;
+                existingPrIssuesByNumber.set(pr.number, hit);
+                console.log(
+                  `[Pull Requests] Recovered orphan from prior failed attempt for PR #${pr.number} (basic fallback); switching to PATCH`
+                );
+              }
+            } catch (_recheckErr) {
+              // Best-effort; fall through to create.
+            }
+          }
           if (existingPrIssue) {
             await httpPatch(
               `${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${existingPrIssue.number}`,

src/lib/helpers.ts

@@ -216,9 +216,21 @@ export async function updateMirrorJobProgress({
 }
 
 /**
- * Finds interrupted jobs that need to be resumed with enhanced criteria
+ * Finds interrupted jobs that need to be resumed with enhanced criteria.
+ *
+ * `logFound` defaults to false because this function is polled from
+ * passive callers (`hasJobsNeedingRecovery` from the health endpoint
+ * and middleware checks). Logging on every poll produces log spam at
+ * one-line-per-poll-per-stuck-job for as long as a job stays stuck.
+ *
+ * Callers that intend to act on the result (i.e. immediately resume
+ * the returned jobs) should pass `logFound: true` so the surfacing
+ * still happens in the recovery flow.
  */
-export async function findInterruptedJobs() {
+export async function findInterruptedJobs(
+  options: { logFound?: boolean } = {}
+) {
+  const { logFound = false } = options;
   try {
     // Find jobs that are marked as in-progress but haven't been updated recently
     const cutoffTime = new Date();
@@ -243,8 +255,9 @@ export async function findInterruptedJobs() {
         )
       );
 
-    // Log details about found jobs for debugging
-    if (interruptedJobs.length > 0) {
+    // Log details about found jobs for debugging — opt-in to avoid
+    // spamming the log when called from periodic passive checks.
+    if (logFound && interruptedJobs.length > 0) {
       console.log(`Found ${interruptedJobs.length} interrupted jobs:`);
       interruptedJobs.forEach(job => {
         const lastCheckpoint = job.lastCheckpoint ? new Date(job.lastCheckpoint).toISOString() : 'never';

src/lib/orchestrator-resume-after-startup.test.ts

@@ -0,0 +1,128 @@
+/**
+ * Regression test for the "interrupted jobs never resume after
+ * startup" orchestration bug.
+ *
+ * Symptom (before this fix):
+ *   - Server starts cleanly. Middleware runs initial recovery pass,
+ *     finds no interrupted jobs, sets `recoveryAttempted = true` and
+ *     `recoveryInitialized = true`.
+ *   - User triggers a sync at T=N (well after startup). The sync
+ *     creates a `mirrorJobs` row with `inProgress=true`.
+ *   - The sync fails mid-flight (deadlock retry, network blip,
+ *     container restart of an upstream service, etc.) and never
+ *     reaches the resume codepath, so the row stays
+ *     `inProgress=true` with no checkpoint.
+ *   - `findInterruptedJobs` (called periodically from the health
+ *     endpoint via `hasJobsNeedingRecovery`) detects it and logs
+ *     `Found 1 interrupted jobs:` on every poll.
+ *   - But the resumer (`resumeInterruptedJob`) is only invoked from
+ *     `initializeRecovery`, which is gated behind the
+ *     once-per-process `!recoveryAttempted` check in
+ *     `src/middleware.ts`. That check is false after startup, so the
+ *     resumer NEVER fires again. The job is stuck forever.
+ *
+ * Root cause: the middleware gate was symmetric — "skip recovery if
+ * we've ever attempted it" — but it should have been "always
+ * re-evaluate; only the recovery routine's own 5-minute throttle
+ * (`skipIfRecentAttempt` inside `initializeRecovery`) prevents
+ * thrashing".
+ *
+ * Secondary issue: `findInterruptedJobs` logged unconditionally on
+ * every call, even from passive checks like `hasJobsNeedingRecovery`,
+ * producing log spam at one line per poll per stuck job.
+ *
+ * This test asserts on the *structure* of the source rather than
+ * invoking the middleware, because exercising the middleware path
+ * requires a full Astro request pipeline with heavy mocks. See
+ * `gitea-mirror-failure-recovery.test.ts` and
+ * `gitea-issue-dedup-on-retry.test.ts` for the same convention.
+ */
+import { describe, test, expect } from "bun:test";
+import { readFileSync } from "node:fs";
+import { join } from "node:path";
+
+const MIDDLEWARE_SRC = readFileSync(
+  join(import.meta.dir, "../middleware.ts"),
+  "utf8"
+);
+const HELPERS_SRC = readFileSync(
+  join(import.meta.dir, "helpers.ts"),
+  "utf8"
+);
+const RECOVERY_SRC = readFileSync(
+  join(import.meta.dir, "recovery.ts"),
+  "utf8"
+);
+
+describe("orchestrator: resume interrupted jobs after startup", () => {
+  test("middleware no longer gates recovery behind once-per-process `recoveryAttempted`", () => {
+    // The old gate looked like:
+    //   if (!recoveryInitialized && !recoveryAttempted) {
+    //     recoveryAttempted = true;
+    //     ...
+    //   }
+    // Once both flags flipped on the first request, recovery never
+    // ran again — even if jobs got stuck mid-runtime.
+    expect(
+      /\brecoveryAttempted\b/.test(MIDDLEWARE_SRC),
+      "the `recoveryAttempted` once-per-process flag must be removed " +
+        "from middleware.ts so post-startup interruptions can recover"
+    ).toBe(false);
+  });
+
+  test("middleware uses an in-flight latch (not a one-shot gate) for runtime safety", () => {
+    // The replacement uses `recoveryInFlight` as a per-request
+    // mutex — set true at the start, set false in `finally`. The
+    // actual throttle (5-minute "recent attempt") lives inside
+    // `initializeRecovery()` in recovery.ts, which is the right
+    // place for it.
+    expect(
+      /\brecoveryInFlight\b/.test(MIDDLEWARE_SRC),
+      "middleware should use `recoveryInFlight` as the in-flight latch"
+    ).toBe(true);
+    expect(
+      /recoveryInFlight\s*=\s*false/.test(MIDDLEWARE_SRC) &&
+        /\bfinally\s*\{[\s\S]*?recoveryInFlight\s*=\s*false[\s\S]*?\}/.test(
+          MIDDLEWARE_SRC
+        ),
+      "the in-flight latch must be released in a `finally` block " +
+        "so an exception during recovery doesn't permanently jam the latch"
+    ).toBe(true);
+  });
+
+  test("findInterruptedJobs logging is opt-in (default off) to stop poll spam", () => {
+    // Active recovery callers (initializeRecovery) opt in by passing
+    // { logFound: true }; passive checks (hasJobsNeedingRecovery,
+    // health endpoint, etc.) default to silent.
+    expect(
+      /export async function findInterruptedJobs\(\s*options[^)]*\)/.test(
+        HELPERS_SRC
+      ),
+      "findInterruptedJobs should accept an options object"
+    ).toBe(true);
+    expect(
+      /logFound\s*=\s*false/.test(HELPERS_SRC),
+      "the `logFound` option should default to false " +
+        "so periodic passive checks (e.g. hasJobsNeedingRecovery) " +
+        "don't spam the log on every poll"
+    ).toBe(true);
+    expect(
+      /if\s*\(\s*logFound\s*&&\s*interruptedJobs\.length\s*>\s*0\s*\)/.test(
+        HELPERS_SRC
+      ),
+      "the `Found N interrupted jobs` log must be gated by `logFound`"
+    ).toBe(true);
+  });
+
+  test("active recovery path opts in to per-job logging", () => {
+    // Without this, the actual recovery cycle would also be silent
+    // — operators need to see which jobs are being resumed.
+    expect(
+      /findInterruptedJobs\(\s*\{\s*logFound:\s*true\s*\}\s*\)/.test(
+        RECOVERY_SRC
+      ),
+      "initializeRecovery() must pass { logFound: true } to findInterruptedJobs " +
+        "so the active recovery cycle still logs which jobs it's working on"
+    ).toBe(true);
+  });
+});

src/lib/recovery.ts

@@ -121,8 +121,9 @@ export async function initializeRecovery(options: {
       // Clean up stale jobs first
       await cleanupStaleJobs();
 
-      // Find interrupted jobs
-      const interruptedJobs = await findInterruptedJobs();
+      // Find interrupted jobs (with per-job logging — this is the
+      // active recovery path that will immediately try to resume them)
+      const interruptedJobs = await findInterruptedJobs({ logFound: true });
 
       if (interruptedJobs.length === 0) {
         console.log('No interrupted jobs found.');

src/middleware.ts

@@ -17,9 +17,16 @@ function prefixAstroInternalAssetPaths(html: string, basePath: string): string {
   return html.replace(ASTRO_INTERNAL_ASSET_PATH_PATTERN, `$1${basePath}/$2`);
 }
 
-// Flag to track if recovery has been initialized
+// Flag to track whether the *startup* recovery pass has run. This
+// only gates the post-startup chain (cleanup service, scheduler,
+// etc.) and the "startup script may not have run" log line — it does
+// NOT gate subsequent recovery attempts, see below.
 let recoveryInitialized = false;
-let recoveryAttempted = false;
+// Throttle for runtime recovery retries (separate from the
+// initializeRecovery() 5-minute throttle inside recovery.ts, which is
+// keyed on `lastRecoveryAttempt`). This prevents one middleware
+// invocation from triggering recovery while another is in flight.
+let recoveryInFlight = false;
 let cleanupServiceStarted = false;
 let schedulerServiceStarted = false;
 let repositoryCleanupServiceStarted = false;
@@ -118,17 +125,30 @@ export const onRequest = defineMiddleware(async (context, next) => {
     }
   }
 
-  // Initialize recovery system only once when the server starts
-  // This is a fallback in case the startup script didn't run
-  if (!recoveryInitialized && !recoveryAttempted) {
-    recoveryAttempted = true;
+  // Run recovery if jobs need it.
+  //
+  // The previous implementation used a once-per-process gate, so
+  // any mid-runtime interruption (a sync that started after boot,
+  // crashed mid-flight, and never got back to the resume path)
+  // would sit at `in_progress=true` forever — the periodic detector
+  // kept finding it, but the resumer never re-fired. This block
+  // now re-evaluates on every request, gated by `recoveryInFlight`
+  // (per-process) plus the 5-minute throttle inside
+  // `initializeRecovery()` (which prevents thrashing if a resume
+  // cycle keeps failing).
+  if (!recoveryInFlight) {
+    recoveryInFlight = true;
 
     try {
       // Check if recovery is actually needed before attempting
       const needsRecovery = await hasJobsNeedingRecovery();
 
       if (needsRecovery) {
-        console.log('⚠️  Middleware detected jobs needing recovery (startup script may not have run)');
+        if (!recoveryInitialized) {
+          console.log('⚠️  Middleware detected jobs needing recovery (startup script may not have run)');
+        } else {
+          console.log('⚠️  Middleware detected jobs needing recovery mid-run (sync interrupted after startup)');
+        }
         console.log('Attempting recovery from middleware...');
 
         // Run recovery with a shorter timeout since this is during request handling
@@ -148,7 +168,9 @@ export const onRequest = defineMiddleware(async (context, next) => {
         } else {
           console.log('⚠️  Middleware recovery completed with some issues');
         }
-      } else {
+      } else if (!recoveryInitialized) {
+        // Only log this on the first request; otherwise we'd spam
+        // it on every request.
         console.log('✅ No recovery needed (startup script likely handled it)');
       }
 
@@ -161,7 +183,9 @@ export const onRequest = defineMiddleware(async (context, next) => {
       const status = getRecoveryStatus();
       console.log('Recovery status:', status);
 
-      recoveryInitialized = true; // Mark as attempted to avoid retries
+      recoveryInitialized = true;
+    } finally {
+      recoveryInFlight = false;
     }
   }