Compare commits

..

3 Commits

Author SHA1 Message Date
Arunavo Ray c1712bc670 chore: bump version to 3.15.3 2026-04-20 13:03:28 +05:30
ARUNAVO RAY c4550196e9 fix: honor GH_API_URL across all Octokit call sites (#269) (#273)
* fix: honor GH_API_URL across all Octokit call sites

Six Octokit call sites constructed `new Octokit(...)` directly instead of
going through `createGitHubClient()`, so `GH_API_URL` (and the
`GITHUB_API_URL` fallback) only applied to the handful of flows that used
the helper. For GHES / GHEC-with-data-residency users this surfaced most
visibly as the "Test Connection" button hitting `api.github.com/user`
and failing with 401 even when `GH_API_URL` was set correctly (#269).

Route everything through `createGitHubClient()`:
- src/pages/api/github/test-connection.ts (the reported failure)
- src/pages/api/sync/repository.ts (public-repo sync)
- src/lib/gitea-enhanced.ts (force-push detection + metadata octokit)
- src/lib/scheduler-service.ts (auto-discovery, auto-mirror, auto-start)
- src/tests/test-metadata-mirroring.ts (dev harness, for consistency)

Side benefit: scheduler + sync paths now also get throttling, rate-limit
tracking, and the standard User-Agent, which they were missing.

`createGitHubClient`'s `token` parameter is made optional so the
public-repo sync path (`new Octokit()` with no auth) can keep working.

Fixes #269

* fix: address review findings

- scheduler: pass config.githubConfig?.owner (the real DB field) instead
  of ?.username, which doesn't exist on the DB row and was silently
  resolving to undefined — matches every other DB-reading call site.
- sync/repository.ts: revert to bare Octokit for the unauthenticated
  public-repo lookup to preserve fast-fail on the 60 req/hr limit.
  Still reads GH_API_URL / GITHUB_API_URL inline so GHES / GHEC
  data-residency users benefit. The throttling plugin's retry-with-
  backoff is wrong UX for a one-shot button click.
- github.ts: revert createGitHubClient token back to required (no
  remaining callers pass undefined after the above).
- gitea-enhanced.ts: make the leftover Octokit import type-only.
- test-connection.test.ts: replace mid-test mock.module re-call with a
  mutable stub reference — safer against ESM live-binding semantics.
2026-04-20 13:02:34 +05:30
Arunavo Ray 8cb8fd6fe1 docs: document GH_API_URL for GitHub Enterprise and SERVER_CERT_PATH/SERVER_KEY_PATH for native HTTPS
- README + env reference + .env.example now cover using GH_API_URL to
  target GitHub Enterprise Server or GHEC with data residency.
- Env reference + .env.example now cover SERVER_CERT_PATH and
  SERVER_KEY_PATH, which @astrojs/node reads at runtime to terminate
  TLS directly without a reverse proxy.

Closes #269
Closes #272
2026-04-20 09:30:08 +05:30
10 changed files with 151 additions and 76 deletions
+14
View File
@@ -46,6 +46,14 @@ BETTER_AUTH_URL=http://localhost:4321
PUBLIC_BETTER_AUTH_URL=http://localhost:4321
# BETTER_AUTH_TRUSTED_ORIGINS=
# ===========================================
# HTTPS / TLS (Optional)
# ===========================================
# Set BOTH to have the server terminate TLS directly (no reverse proxy needed).
# Leave unset when TLS is handled upstream by Nginx/Traefik/Caddy.
# SERVER_CERT_PATH=/etc/ssl/gitea-mirror/cert.pem
# SERVER_KEY_PATH=/etc/ssl/gitea-mirror/key.pem
# ===========================================
# DOCKER CONFIGURATION (Optional)
# ===========================================
@@ -65,6 +73,12 @@ DOCKER_TAG=latest
# GITHUB_TOKEN=your-github-personal-access-token
# GITHUB_TYPE=personal # Options: personal, organization
# GitHub Enterprise (GHES / GHEC with data residency)
# Leave unset for standard github.com. Examples:
# GHES (self-hosted): https://ghe.example.com/api/v3
# GHEC data residency: https://api.TENANT.ghe.com
# GH_API_URL=https://ghe.example.com/api/v3
# Repository Selection
# PRIVATE_REPOSITORIES=false
# PUBLIC_REPOSITORIES=true
+15
View File
@@ -29,6 +29,7 @@ First user signup becomes admin. Configure GitHub and Gitea/Forgejo through the
## ✨ Features
- 🔁 Mirror public, private, and starred GitHub repos to Gitea/Forgejo
- 🏛️ **GitHub Enterprise support** - Works with GHES and GHEC with data residency via `GH_API_URL`
- 🏢 Mirror entire organizations with flexible strategies
- 🎯 Custom destination control for repos and organizations
- 📦 **Git LFS support** - Mirror large files with Git LFS
@@ -296,6 +297,20 @@ CLEANUP_DRY_RUN=false # Set to true to test without changes
- **The Whole Point of Backups**: Your Gitea/Forgejo mirrors are preserved even when GitHub sources disappear - that's why you have backups!
- **Strongly Recommended**: Always use `CLEANUP_ORPHANED_REPO_ACTION=archive` (default) instead of `delete`
### GitHub Enterprise (GHES / GHEC with Data Residency)
Gitea Mirror works with non-`github.com` GitHub deployments. Point the client at your Enterprise API via the `GH_API_URL` environment variable:
```bash
# GitHub Enterprise Server (self-hosted)
GH_API_URL=https://ghe.example.com/api/v3
# GitHub Enterprise Cloud with data residency
GH_API_URL=https://api.TENANT.ghe.com
```
Standard GitHub Enterprise Cloud on `github.com` needs no override. Use a token issued by the target Enterprise instance for `GITHUB_TOKEN`.
## Troubleshooting
### Reverse Proxy Configuration
+40
View File
@@ -16,6 +16,7 @@ When environment variables are set:
## Table of Contents
- [Core Configuration](#core-configuration)
- [HTTPS / TLS](#https--tls)
- [GitHub Configuration](#github-configuration)
- [Gitea Configuration](#gitea-configuration)
- [Mirror Options](#mirror-options)
@@ -41,6 +42,30 @@ Essential application settings required for running Gitea Mirror.
| `BETTER_AUTH_TRUSTED_ORIGINS` | Trusted origins for authentication requests. Comma-separated list of URLs. Use this to specify additional access URLs (e.g., local IP + domain: `http://10.10.20.45:4321,https://gitea-mirror.mydomain.tld`), SSO providers, reverse proxies, etc. | - | No |
| `ENCRYPTION_SECRET` | Optional encryption key for tokens (generate with: `openssl rand -base64 48`) | - | No |
## HTTPS / TLS
Gitea Mirror can terminate TLS directly via the underlying `@astrojs/node` adapter — useful when you don't want a separate reverse proxy. When both variables below are set, the server starts as a real HTTPS listener instead of HTTP.
| Variable | Description | Default | Required |
|----------|-------------|---------|----------|
| `SERVER_CERT_PATH` | Absolute path to the TLS certificate (PEM). Set together with `SERVER_KEY_PATH` to enable HTTPS. | - | No |
| `SERVER_KEY_PATH` | Absolute path to the TLS private key (PEM). Set together with `SERVER_CERT_PATH` to enable HTTPS. | - | No |
**Example (systemd or `.env`):**
```bash
SERVER_CERT_PATH=/etc/ssl/gitea-mirror/cert.pem
SERVER_KEY_PATH=/etc/ssl/gitea-mirror/key.pem
PORT=443
BETTER_AUTH_URL=https://mirror.example.com
BETTER_AUTH_TRUSTED_ORIGINS=https://mirror.example.com
```
Notes:
- The process must have read access to both files. When binding to `PORT=443`, grant the binary the `CAP_NET_BIND_SERVICE` capability (or run as a user allowed to bind privileged ports) rather than running as root.
- If you already terminate TLS at a reverse proxy (nginx, Traefik, Caddy), leave these unset and let the proxy handle certificates.
- Works in Docker too — mount your certs and set both paths to locations inside the container.
## GitHub Configuration
Settings for connecting to and configuring GitHub repository sources.
@@ -52,6 +77,21 @@ Settings for connecting to and configuring GitHub repository sources.
| `GITHUB_USERNAME` | Your GitHub username | - | - |
| `GITHUB_TOKEN` | GitHub personal access token (requires repo and admin:org scopes) | - | - |
| `GITHUB_TYPE` | GitHub account type | `personal` | `personal`, `organization` |
| `GH_API_URL` | GitHub API base URL. Override this to point at GitHub Enterprise Server or Enterprise Cloud with data residency. | `https://api.github.com` | e.g. `https://ghe.example.com/api/v3`, `https://api.TENANT.ghe.com` |
### GitHub Enterprise (GHES / GHEC with data residency)
Set `GH_API_URL` to point Octokit at a non-`github.com` API endpoint:
```bash
# GitHub Enterprise Server (self-hosted)
GH_API_URL=https://ghe.example.com/api/v3
# GitHub Enterprise Cloud with data residency
GH_API_URL=https://api.TENANT.ghe.com
```
Standard GitHub Enterprise Cloud on `github.com` works with the default — no override needed. Use a personal access token issued by the target Enterprise instance for `GITHUB_TOKEN`.
### Repository Selection
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "gitea-mirror",
"type": "module",
"version": "3.15.2",
"version": "3.15.3",
"engines": {
"bun": ">=1.2.9"
},
+4 -5
View File
@@ -8,7 +8,8 @@
import type { Config } from "@/types/config";
import type { Repository } from "./db/schema";
import { Octokit } from "@octokit/rest";
import type { Octokit } from "@octokit/rest";
import { createGitHubClient } from "./github";
import { createMirrorJob } from "./helpers";
import { decryptConfigTokens } from "./utils/config-encryption";
import { httpPost, httpGet, httpPatch, HttpError } from "./http-client";
@@ -431,7 +432,7 @@ export async function syncGiteaRepoEnhanced({
try {
const decryptedGithubToken = decryptedConfig.githubConfig?.token;
if (decryptedGithubToken) {
const fpOctokit = new Octokit({ auth: decryptedGithubToken });
const fpOctokit = createGitHubClient(decryptedGithubToken);
const detectionResult = await detectForcePush({
giteaUrl: config.giteaConfig.url,
giteaToken: decryptedConfig.giteaConfig.token,
@@ -596,9 +597,7 @@ export async function syncGiteaRepoEnhanced({
if (!decryptedConfig.githubConfig?.token) {
return null;
}
metadataOctokit = new Octokit({
auth: decryptedConfig.githubConfig.token,
});
metadataOctokit = createGitHubClient(decryptedConfig.githubConfig.token);
return metadataOctokit;
};
+13 -15
View File
@@ -99,15 +99,14 @@ async function runScheduledSync(config: any): Promise<void> {
if (scheduleConfig.autoImport !== false) {
console.log(`[Scheduler] Checking for new GitHub repositories for user ${userId}...`);
try {
const { getGithubRepositories, getGithubStarredRepositories } = await import('@/lib/github');
const { getGithubRepositories, getGithubStarredRepositories, createGitHubClient } = await import('@/lib/github');
const { v4: uuidv4 } = await import('uuid');
const { getDecryptedGitHubToken } = await import('@/lib/utils/config-encryption');
// Create GitHub client
// Create GitHub client (honors GH_API_URL for GHES / GHEC data residency)
const decryptedToken = getDecryptedGitHubToken(config);
const { Octokit } = await import('@octokit/rest');
const octokit = new Octokit({ auth: decryptedToken });
const octokit = createGitHubClient(decryptedToken, userId, config.githubConfig?.owner);
// Fetch GitHub data
const [basicAndForkedRepos, starredRepos] = await Promise.all([
getGithubRepositories({ octokit, config }),
@@ -117,7 +116,7 @@ async function runScheduledSync(config: any): Promise<void> {
]);
const allGithubRepos = mergeGitReposPreferStarred(basicAndForkedRepos, starredRepos);
const mirrorableGithubRepos = allGithubRepos.filter(isMirrorableGitHubRepo);
// Check for new repositories
const existingRepos = await db
.select({ normalizedFullName: repositories.normalizedFullName })
@@ -238,10 +237,10 @@ async function runScheduledSync(config: any): Promise<void> {
if (reposNeedingMirror.length > 0) {
console.log(`[Scheduler] Found ${reposNeedingMirror.length} repositories that need initial mirroring`);
// Prepare Octokit client
// Prepare Octokit client (honors GH_API_URL for GHES / GHEC data residency)
const decryptedToken = getDecryptedGitHubToken(config);
const { Octokit } = await import('@octokit/rest');
const octokit = new Octokit({ auth: decryptedToken });
const { createGitHubClient } = await import('@/lib/github');
const octokit = createGitHubClient(decryptedToken, userId, config.githubConfig?.owner);
// Process repositories in batches
const batchSize = scheduleConfig.batchSize || 10;
@@ -482,13 +481,12 @@ async function performInitialAutoStart(): Promise<void> {
try {
// Step 1: Import repositories from GitHub
console.log(`[Scheduler] Step 1: Importing repositories from GitHub for user ${config.userId}...`);
const { getGithubRepositories, getGithubStarredRepositories } = await import('@/lib/github');
const { getGithubRepositories, getGithubStarredRepositories, createGitHubClient } = await import('@/lib/github');
const { v4: uuidv4 } = await import('uuid');
// Create GitHub client
// Create GitHub client (honors GH_API_URL for GHES / GHEC data residency)
const decryptedToken = getDecryptedGitHubToken(config);
const { Octokit } = await import('@octokit/rest');
const octokit = new Octokit({ auth: decryptedToken });
const octokit = createGitHubClient(decryptedToken, config.userId, config.githubConfig?.owner);
// Fetch GitHub data
const [basicAndForkedRepos, starredRepos] = await Promise.all([
+46 -44
View File
@@ -1,39 +1,51 @@
import { describe, test, expect, mock, beforeEach, afterEach } from "bun:test";
import { POST } from "./test-connection";
import { Octokit } from "@octokit/rest";
// Mock the Octokit class
mock.module("@octokit/rest", () => {
// createGitHubClient returns this stub. Tests mutate `getAuthenticatedImpl`
// to steer the behavior without re-calling mock.module (which is fragile
// once the route module has already captured a live binding).
let getAuthenticatedImpl: () => Promise<any> = () =>
Promise.resolve({
data: {
login: "testuser",
name: "Test User",
avatar_url: "https://example.com/avatar.png",
},
});
mock.module("@/lib/github", () => {
return {
Octokit: mock(function() {
return {
users: {
getAuthenticated: mock(() => Promise.resolve({
data: {
login: "testuser",
name: "Test User",
avatar_url: "https://example.com/avatar.png"
}
}))
}
};
})
createGitHubClient: mock(() => ({
users: {
getAuthenticated: mock(() => getAuthenticatedImpl()),
},
})),
};
});
import { POST } from "./test-connection";
describe("GitHub Test Connection API", () => {
// Mock console.error to prevent test output noise
let originalConsoleError: typeof console.error;
beforeEach(() => {
originalConsoleError = console.error;
console.error = mock(() => {});
// Reset to the success stub before each test so tests are independent
getAuthenticatedImpl = () =>
Promise.resolve({
data: {
login: "testuser",
name: "Test User",
avatar_url: "https://example.com/avatar.png",
},
});
});
afterEach(() => {
console.error = originalConsoleError;
});
test("returns 400 if token is missing", async () => {
const request = new Request("http://localhost/api/github/test-connection", {
method: "POST",
@@ -42,16 +54,16 @@ describe("GitHub Test Connection API", () => {
},
body: JSON.stringify({})
});
const response = await POST({ request } as any);
expect(response.status).toBe(400);
const data = await response.json();
expect(data.success).toBe(false);
expect(data.message).toBe("GitHub token is required");
});
test("returns 200 with user data on successful connection", async () => {
const request = new Request("http://localhost/api/github/test-connection", {
method: "POST",
@@ -62,11 +74,11 @@ describe("GitHub Test Connection API", () => {
token: "valid-token"
})
});
const response = await POST({ request } as any);
expect(response.status).toBe(200);
const data = await response.json();
expect(data.success).toBe(true);
expect(data.message).toBe("Successfully connected to GitHub as testuser");
@@ -76,7 +88,7 @@ describe("GitHub Test Connection API", () => {
avatar_url: "https://example.com/avatar.png"
});
});
test("returns 400 if username doesn't match authenticated user", async () => {
const request = new Request("http://localhost/api/github/test-connection", {
method: "POST",
@@ -88,29 +100,19 @@ describe("GitHub Test Connection API", () => {
username: "differentuser"
})
});
const response = await POST({ request } as any);
expect(response.status).toBe(400);
const data = await response.json();
expect(data.success).toBe(false);
expect(data.message).toBe("Token belongs to testuser, not differentuser");
});
test("handles authentication errors", async () => {
// Mock Octokit to throw an error
mock.module("@octokit/rest", () => {
return {
Octokit: mock(function() {
return {
users: {
getAuthenticated: mock(() => Promise.reject(new Error("Bad credentials")))
}
};
})
};
});
// Swap the stub to throw an auth error for this test only
getAuthenticatedImpl = () => Promise.reject(new Error("Bad credentials"));
const request = new Request("http://localhost/api/github/test-connection", {
method: "POST",
+5 -5
View File
@@ -1,5 +1,5 @@
import type { APIRoute } from "astro";
import { Octokit } from "@octokit/rest";
import { createGitHubClient } from "@/lib/github";
import { createSecureErrorResponse } from "@/lib/utils";
export const POST: APIRoute = async ({ request }) => {
@@ -22,10 +22,10 @@ export const POST: APIRoute = async ({ request }) => {
);
}
// Create an Octokit instance with the provided token
const octokit = new Octokit({
auth: token,
});
// Create an Octokit instance with the provided token.
// Uses createGitHubClient so GH_API_URL / GITHUB_API_URL routes the call
// to the correct endpoint for GHES / GHEC with data residency.
const octokit = createGitHubClient(token);
// Test the connection by fetching the authenticated user
const { data } = await octokit.users.getAuthenticated();
+10 -1
View File
@@ -88,7 +88,16 @@ export const POST: APIRoute = async ({ request, locals }) => {
const configId = config.id;
const octokit = new Octokit(); // No auth for public repos
// Unauthenticated one-shot lookup for public repos.
// Uses bare Octokit (not createGitHubClient) to preserve fast-fail on the
// 60 req/hr public rate limit — this endpoint is user-facing, we don't
// want the throttling plugin to wait multiple retry-after windows.
// Still respects GH_API_URL / GITHUB_API_URL for GHES / GHEC data residency.
const baseUrl =
process.env.GH_API_URL ||
process.env.GITHUB_API_URL ||
"https://api.github.com";
const octokit = new Octokit({ baseUrl });
const { data: repoData } = await octokit.rest.repos.get({
owner: trimmedOwner,
+3 -5
View File
@@ -11,7 +11,7 @@ import { validateGiteaAuth } from "@/lib/gitea-auth-validator";
import { getConfigsByUserId } from "@/lib/db/queries/configs";
import { db, users, repositories } from "@/lib/db";
import { eq } from "drizzle-orm";
import { Octokit } from "@octokit/rest";
import { createGitHubClient } from "@/lib/github";
import type { Repository } from "@/lib/db/schema";
async function testMetadataMirroringAuth() {
@@ -108,10 +108,8 @@ async function testMetadataMirroringAuth() {
console.log("\n🔄 Test 4: Testing metadata mirroring authentication...");
try {
// Create Octokit instance
const octokit = new Octokit({
auth: config.githubConfig.token,
});
// Create Octokit instance (honors GH_API_URL for GHES / GHEC data residency)
const octokit = createGitHubClient(config.githubConfig.token);
// Test by attempting to fetch labels (lightweight operation)
const { httpGet } = await import("@/lib/http-client");