mirror of
https://github.com/RayLabsHQ/gitea-mirror.git
synced 2026-08-20 22:39:40 +08:00
Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| c1712bc670 | |||
| c4550196e9 | |||
| 8cb8fd6fe1 |
@@ -46,6 +46,14 @@ BETTER_AUTH_URL=http://localhost:4321
|
||||
PUBLIC_BETTER_AUTH_URL=http://localhost:4321
|
||||
# BETTER_AUTH_TRUSTED_ORIGINS=
|
||||
|
||||
# ===========================================
|
||||
# HTTPS / TLS (Optional)
|
||||
# ===========================================
|
||||
# Set BOTH to have the server terminate TLS directly (no reverse proxy needed).
|
||||
# Leave unset when TLS is handled upstream by Nginx/Traefik/Caddy.
|
||||
# SERVER_CERT_PATH=/etc/ssl/gitea-mirror/cert.pem
|
||||
# SERVER_KEY_PATH=/etc/ssl/gitea-mirror/key.pem
|
||||
|
||||
# ===========================================
|
||||
# DOCKER CONFIGURATION (Optional)
|
||||
# ===========================================
|
||||
@@ -65,6 +73,12 @@ DOCKER_TAG=latest
|
||||
# GITHUB_TOKEN=your-github-personal-access-token
|
||||
# GITHUB_TYPE=personal # Options: personal, organization
|
||||
|
||||
# GitHub Enterprise (GHES / GHEC with data residency)
|
||||
# Leave unset for standard github.com. Examples:
|
||||
# GHES (self-hosted): https://ghe.example.com/api/v3
|
||||
# GHEC data residency: https://api.TENANT.ghe.com
|
||||
# GH_API_URL=https://ghe.example.com/api/v3
|
||||
|
||||
# Repository Selection
|
||||
# PRIVATE_REPOSITORIES=false
|
||||
# PUBLIC_REPOSITORIES=true
|
||||
|
||||
@@ -29,6 +29,7 @@ First user signup becomes admin. Configure GitHub and Gitea/Forgejo through the
|
||||
## ✨ Features
|
||||
|
||||
- 🔁 Mirror public, private, and starred GitHub repos to Gitea/Forgejo
|
||||
- 🏛️ **GitHub Enterprise support** - Works with GHES and GHEC with data residency via `GH_API_URL`
|
||||
- 🏢 Mirror entire organizations with flexible strategies
|
||||
- 🎯 Custom destination control for repos and organizations
|
||||
- 📦 **Git LFS support** - Mirror large files with Git LFS
|
||||
@@ -296,6 +297,20 @@ CLEANUP_DRY_RUN=false # Set to true to test without changes
|
||||
- **The Whole Point of Backups**: Your Gitea/Forgejo mirrors are preserved even when GitHub sources disappear - that's why you have backups!
|
||||
- **Strongly Recommended**: Always use `CLEANUP_ORPHANED_REPO_ACTION=archive` (default) instead of `delete`
|
||||
|
||||
### GitHub Enterprise (GHES / GHEC with Data Residency)
|
||||
|
||||
Gitea Mirror works with non-`github.com` GitHub deployments. Point the client at your Enterprise API via the `GH_API_URL` environment variable:
|
||||
|
||||
```bash
|
||||
# GitHub Enterprise Server (self-hosted)
|
||||
GH_API_URL=https://ghe.example.com/api/v3
|
||||
|
||||
# GitHub Enterprise Cloud with data residency
|
||||
GH_API_URL=https://api.TENANT.ghe.com
|
||||
```
|
||||
|
||||
Standard GitHub Enterprise Cloud on `github.com` needs no override. Use a token issued by the target Enterprise instance for `GITHUB_TOKEN`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Reverse Proxy Configuration
|
||||
|
||||
@@ -16,6 +16,7 @@ When environment variables are set:
|
||||
## Table of Contents
|
||||
|
||||
- [Core Configuration](#core-configuration)
|
||||
- [HTTPS / TLS](#https--tls)
|
||||
- [GitHub Configuration](#github-configuration)
|
||||
- [Gitea Configuration](#gitea-configuration)
|
||||
- [Mirror Options](#mirror-options)
|
||||
@@ -41,6 +42,30 @@ Essential application settings required for running Gitea Mirror.
|
||||
| `BETTER_AUTH_TRUSTED_ORIGINS` | Trusted origins for authentication requests. Comma-separated list of URLs. Use this to specify additional access URLs (e.g., local IP + domain: `http://10.10.20.45:4321,https://gitea-mirror.mydomain.tld`), SSO providers, reverse proxies, etc. | - | No |
|
||||
| `ENCRYPTION_SECRET` | Optional encryption key for tokens (generate with: `openssl rand -base64 48`) | - | No |
|
||||
|
||||
## HTTPS / TLS
|
||||
|
||||
Gitea Mirror can terminate TLS directly via the underlying `@astrojs/node` adapter — useful when you don't want a separate reverse proxy. When both variables below are set, the server starts as a real HTTPS listener instead of HTTP.
|
||||
|
||||
| Variable | Description | Default | Required |
|
||||
|----------|-------------|---------|----------|
|
||||
| `SERVER_CERT_PATH` | Absolute path to the TLS certificate (PEM). Set together with `SERVER_KEY_PATH` to enable HTTPS. | - | No |
|
||||
| `SERVER_KEY_PATH` | Absolute path to the TLS private key (PEM). Set together with `SERVER_CERT_PATH` to enable HTTPS. | - | No |
|
||||
|
||||
**Example (systemd or `.env`):**
|
||||
|
||||
```bash
|
||||
SERVER_CERT_PATH=/etc/ssl/gitea-mirror/cert.pem
|
||||
SERVER_KEY_PATH=/etc/ssl/gitea-mirror/key.pem
|
||||
PORT=443
|
||||
BETTER_AUTH_URL=https://mirror.example.com
|
||||
BETTER_AUTH_TRUSTED_ORIGINS=https://mirror.example.com
|
||||
```
|
||||
|
||||
Notes:
|
||||
- The process must have read access to both files. When binding to `PORT=443`, grant the binary the `CAP_NET_BIND_SERVICE` capability (or run as a user allowed to bind privileged ports) rather than running as root.
|
||||
- If you already terminate TLS at a reverse proxy (nginx, Traefik, Caddy), leave these unset and let the proxy handle certificates.
|
||||
- Works in Docker too — mount your certs and set both paths to locations inside the container.
|
||||
|
||||
## GitHub Configuration
|
||||
|
||||
Settings for connecting to and configuring GitHub repository sources.
|
||||
@@ -52,6 +77,21 @@ Settings for connecting to and configuring GitHub repository sources.
|
||||
| `GITHUB_USERNAME` | Your GitHub username | - | - |
|
||||
| `GITHUB_TOKEN` | GitHub personal access token (requires repo and admin:org scopes) | - | - |
|
||||
| `GITHUB_TYPE` | GitHub account type | `personal` | `personal`, `organization` |
|
||||
| `GH_API_URL` | GitHub API base URL. Override this to point at GitHub Enterprise Server or Enterprise Cloud with data residency. | `https://api.github.com` | e.g. `https://ghe.example.com/api/v3`, `https://api.TENANT.ghe.com` |
|
||||
|
||||
### GitHub Enterprise (GHES / GHEC with data residency)
|
||||
|
||||
Set `GH_API_URL` to point Octokit at a non-`github.com` API endpoint:
|
||||
|
||||
```bash
|
||||
# GitHub Enterprise Server (self-hosted)
|
||||
GH_API_URL=https://ghe.example.com/api/v3
|
||||
|
||||
# GitHub Enterprise Cloud with data residency
|
||||
GH_API_URL=https://api.TENANT.ghe.com
|
||||
```
|
||||
|
||||
Standard GitHub Enterprise Cloud on `github.com` works with the default — no override needed. Use a personal access token issued by the target Enterprise instance for `GITHUB_TOKEN`.
|
||||
|
||||
### Repository Selection
|
||||
|
||||
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "gitea-mirror",
|
||||
"type": "module",
|
||||
"version": "3.15.2",
|
||||
"version": "3.15.3",
|
||||
"engines": {
|
||||
"bun": ">=1.2.9"
|
||||
},
|
||||
|
||||
@@ -8,7 +8,8 @@
|
||||
|
||||
import type { Config } from "@/types/config";
|
||||
import type { Repository } from "./db/schema";
|
||||
import { Octokit } from "@octokit/rest";
|
||||
import type { Octokit } from "@octokit/rest";
|
||||
import { createGitHubClient } from "./github";
|
||||
import { createMirrorJob } from "./helpers";
|
||||
import { decryptConfigTokens } from "./utils/config-encryption";
|
||||
import { httpPost, httpGet, httpPatch, HttpError } from "./http-client";
|
||||
@@ -431,7 +432,7 @@ export async function syncGiteaRepoEnhanced({
|
||||
try {
|
||||
const decryptedGithubToken = decryptedConfig.githubConfig?.token;
|
||||
if (decryptedGithubToken) {
|
||||
const fpOctokit = new Octokit({ auth: decryptedGithubToken });
|
||||
const fpOctokit = createGitHubClient(decryptedGithubToken);
|
||||
const detectionResult = await detectForcePush({
|
||||
giteaUrl: config.giteaConfig.url,
|
||||
giteaToken: decryptedConfig.giteaConfig.token,
|
||||
@@ -596,9 +597,7 @@ export async function syncGiteaRepoEnhanced({
|
||||
if (!decryptedConfig.githubConfig?.token) {
|
||||
return null;
|
||||
}
|
||||
metadataOctokit = new Octokit({
|
||||
auth: decryptedConfig.githubConfig.token,
|
||||
});
|
||||
metadataOctokit = createGitHubClient(decryptedConfig.githubConfig.token);
|
||||
return metadataOctokit;
|
||||
};
|
||||
|
||||
|
||||
@@ -99,15 +99,14 @@ async function runScheduledSync(config: any): Promise<void> {
|
||||
if (scheduleConfig.autoImport !== false) {
|
||||
console.log(`[Scheduler] Checking for new GitHub repositories for user ${userId}...`);
|
||||
try {
|
||||
const { getGithubRepositories, getGithubStarredRepositories } = await import('@/lib/github');
|
||||
const { getGithubRepositories, getGithubStarredRepositories, createGitHubClient } = await import('@/lib/github');
|
||||
const { v4: uuidv4 } = await import('uuid');
|
||||
const { getDecryptedGitHubToken } = await import('@/lib/utils/config-encryption');
|
||||
|
||||
// Create GitHub client
|
||||
|
||||
// Create GitHub client (honors GH_API_URL for GHES / GHEC data residency)
|
||||
const decryptedToken = getDecryptedGitHubToken(config);
|
||||
const { Octokit } = await import('@octokit/rest');
|
||||
const octokit = new Octokit({ auth: decryptedToken });
|
||||
|
||||
const octokit = createGitHubClient(decryptedToken, userId, config.githubConfig?.owner);
|
||||
|
||||
// Fetch GitHub data
|
||||
const [basicAndForkedRepos, starredRepos] = await Promise.all([
|
||||
getGithubRepositories({ octokit, config }),
|
||||
@@ -117,7 +116,7 @@ async function runScheduledSync(config: any): Promise<void> {
|
||||
]);
|
||||
const allGithubRepos = mergeGitReposPreferStarred(basicAndForkedRepos, starredRepos);
|
||||
const mirrorableGithubRepos = allGithubRepos.filter(isMirrorableGitHubRepo);
|
||||
|
||||
|
||||
// Check for new repositories
|
||||
const existingRepos = await db
|
||||
.select({ normalizedFullName: repositories.normalizedFullName })
|
||||
@@ -238,10 +237,10 @@ async function runScheduledSync(config: any): Promise<void> {
|
||||
if (reposNeedingMirror.length > 0) {
|
||||
console.log(`[Scheduler] Found ${reposNeedingMirror.length} repositories that need initial mirroring`);
|
||||
|
||||
// Prepare Octokit client
|
||||
// Prepare Octokit client (honors GH_API_URL for GHES / GHEC data residency)
|
||||
const decryptedToken = getDecryptedGitHubToken(config);
|
||||
const { Octokit } = await import('@octokit/rest');
|
||||
const octokit = new Octokit({ auth: decryptedToken });
|
||||
const { createGitHubClient } = await import('@/lib/github');
|
||||
const octokit = createGitHubClient(decryptedToken, userId, config.githubConfig?.owner);
|
||||
|
||||
// Process repositories in batches
|
||||
const batchSize = scheduleConfig.batchSize || 10;
|
||||
@@ -482,13 +481,12 @@ async function performInitialAutoStart(): Promise<void> {
|
||||
try {
|
||||
// Step 1: Import repositories from GitHub
|
||||
console.log(`[Scheduler] Step 1: Importing repositories from GitHub for user ${config.userId}...`);
|
||||
const { getGithubRepositories, getGithubStarredRepositories } = await import('@/lib/github');
|
||||
const { getGithubRepositories, getGithubStarredRepositories, createGitHubClient } = await import('@/lib/github');
|
||||
const { v4: uuidv4 } = await import('uuid');
|
||||
|
||||
// Create GitHub client
|
||||
|
||||
// Create GitHub client (honors GH_API_URL for GHES / GHEC data residency)
|
||||
const decryptedToken = getDecryptedGitHubToken(config);
|
||||
const { Octokit } = await import('@octokit/rest');
|
||||
const octokit = new Octokit({ auth: decryptedToken });
|
||||
const octokit = createGitHubClient(decryptedToken, config.userId, config.githubConfig?.owner);
|
||||
|
||||
// Fetch GitHub data
|
||||
const [basicAndForkedRepos, starredRepos] = await Promise.all([
|
||||
|
||||
@@ -1,39 +1,51 @@
|
||||
import { describe, test, expect, mock, beforeEach, afterEach } from "bun:test";
|
||||
import { POST } from "./test-connection";
|
||||
import { Octokit } from "@octokit/rest";
|
||||
|
||||
// Mock the Octokit class
|
||||
mock.module("@octokit/rest", () => {
|
||||
// createGitHubClient returns this stub. Tests mutate `getAuthenticatedImpl`
|
||||
// to steer the behavior without re-calling mock.module (which is fragile
|
||||
// once the route module has already captured a live binding).
|
||||
let getAuthenticatedImpl: () => Promise<any> = () =>
|
||||
Promise.resolve({
|
||||
data: {
|
||||
login: "testuser",
|
||||
name: "Test User",
|
||||
avatar_url: "https://example.com/avatar.png",
|
||||
},
|
||||
});
|
||||
|
||||
mock.module("@/lib/github", () => {
|
||||
return {
|
||||
Octokit: mock(function() {
|
||||
return {
|
||||
users: {
|
||||
getAuthenticated: mock(() => Promise.resolve({
|
||||
data: {
|
||||
login: "testuser",
|
||||
name: "Test User",
|
||||
avatar_url: "https://example.com/avatar.png"
|
||||
}
|
||||
}))
|
||||
}
|
||||
};
|
||||
})
|
||||
createGitHubClient: mock(() => ({
|
||||
users: {
|
||||
getAuthenticated: mock(() => getAuthenticatedImpl()),
|
||||
},
|
||||
})),
|
||||
};
|
||||
});
|
||||
|
||||
import { POST } from "./test-connection";
|
||||
|
||||
describe("GitHub Test Connection API", () => {
|
||||
// Mock console.error to prevent test output noise
|
||||
let originalConsoleError: typeof console.error;
|
||||
|
||||
|
||||
beforeEach(() => {
|
||||
originalConsoleError = console.error;
|
||||
console.error = mock(() => {});
|
||||
// Reset to the success stub before each test so tests are independent
|
||||
getAuthenticatedImpl = () =>
|
||||
Promise.resolve({
|
||||
data: {
|
||||
login: "testuser",
|
||||
name: "Test User",
|
||||
avatar_url: "https://example.com/avatar.png",
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
afterEach(() => {
|
||||
console.error = originalConsoleError;
|
||||
});
|
||||
|
||||
|
||||
test("returns 400 if token is missing", async () => {
|
||||
const request = new Request("http://localhost/api/github/test-connection", {
|
||||
method: "POST",
|
||||
@@ -42,16 +54,16 @@ describe("GitHub Test Connection API", () => {
|
||||
},
|
||||
body: JSON.stringify({})
|
||||
});
|
||||
|
||||
|
||||
const response = await POST({ request } as any);
|
||||
|
||||
|
||||
expect(response.status).toBe(400);
|
||||
|
||||
|
||||
const data = await response.json();
|
||||
expect(data.success).toBe(false);
|
||||
expect(data.message).toBe("GitHub token is required");
|
||||
});
|
||||
|
||||
|
||||
test("returns 200 with user data on successful connection", async () => {
|
||||
const request = new Request("http://localhost/api/github/test-connection", {
|
||||
method: "POST",
|
||||
@@ -62,11 +74,11 @@ describe("GitHub Test Connection API", () => {
|
||||
token: "valid-token"
|
||||
})
|
||||
});
|
||||
|
||||
|
||||
const response = await POST({ request } as any);
|
||||
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
|
||||
|
||||
const data = await response.json();
|
||||
expect(data.success).toBe(true);
|
||||
expect(data.message).toBe("Successfully connected to GitHub as testuser");
|
||||
@@ -76,7 +88,7 @@ describe("GitHub Test Connection API", () => {
|
||||
avatar_url: "https://example.com/avatar.png"
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
test("returns 400 if username doesn't match authenticated user", async () => {
|
||||
const request = new Request("http://localhost/api/github/test-connection", {
|
||||
method: "POST",
|
||||
@@ -88,29 +100,19 @@ describe("GitHub Test Connection API", () => {
|
||||
username: "differentuser"
|
||||
})
|
||||
});
|
||||
|
||||
|
||||
const response = await POST({ request } as any);
|
||||
|
||||
|
||||
expect(response.status).toBe(400);
|
||||
|
||||
|
||||
const data = await response.json();
|
||||
expect(data.success).toBe(false);
|
||||
expect(data.message).toBe("Token belongs to testuser, not differentuser");
|
||||
});
|
||||
|
||||
|
||||
test("handles authentication errors", async () => {
|
||||
// Mock Octokit to throw an error
|
||||
mock.module("@octokit/rest", () => {
|
||||
return {
|
||||
Octokit: mock(function() {
|
||||
return {
|
||||
users: {
|
||||
getAuthenticated: mock(() => Promise.reject(new Error("Bad credentials")))
|
||||
}
|
||||
};
|
||||
})
|
||||
};
|
||||
});
|
||||
// Swap the stub to throw an auth error for this test only
|
||||
getAuthenticatedImpl = () => Promise.reject(new Error("Bad credentials"));
|
||||
|
||||
const request = new Request("http://localhost/api/github/test-connection", {
|
||||
method: "POST",
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import type { APIRoute } from "astro";
|
||||
import { Octokit } from "@octokit/rest";
|
||||
import { createGitHubClient } from "@/lib/github";
|
||||
import { createSecureErrorResponse } from "@/lib/utils";
|
||||
|
||||
export const POST: APIRoute = async ({ request }) => {
|
||||
@@ -22,10 +22,10 @@ export const POST: APIRoute = async ({ request }) => {
|
||||
);
|
||||
}
|
||||
|
||||
// Create an Octokit instance with the provided token
|
||||
const octokit = new Octokit({
|
||||
auth: token,
|
||||
});
|
||||
// Create an Octokit instance with the provided token.
|
||||
// Uses createGitHubClient so GH_API_URL / GITHUB_API_URL routes the call
|
||||
// to the correct endpoint for GHES / GHEC with data residency.
|
||||
const octokit = createGitHubClient(token);
|
||||
|
||||
// Test the connection by fetching the authenticated user
|
||||
const { data } = await octokit.users.getAuthenticated();
|
||||
|
||||
@@ -88,7 +88,16 @@ export const POST: APIRoute = async ({ request, locals }) => {
|
||||
|
||||
const configId = config.id;
|
||||
|
||||
const octokit = new Octokit(); // No auth for public repos
|
||||
// Unauthenticated one-shot lookup for public repos.
|
||||
// Uses bare Octokit (not createGitHubClient) to preserve fast-fail on the
|
||||
// 60 req/hr public rate limit — this endpoint is user-facing, we don't
|
||||
// want the throttling plugin to wait multiple retry-after windows.
|
||||
// Still respects GH_API_URL / GITHUB_API_URL for GHES / GHEC data residency.
|
||||
const baseUrl =
|
||||
process.env.GH_API_URL ||
|
||||
process.env.GITHUB_API_URL ||
|
||||
"https://api.github.com";
|
||||
const octokit = new Octokit({ baseUrl });
|
||||
|
||||
const { data: repoData } = await octokit.rest.repos.get({
|
||||
owner: trimmedOwner,
|
||||
|
||||
@@ -11,7 +11,7 @@ import { validateGiteaAuth } from "@/lib/gitea-auth-validator";
|
||||
import { getConfigsByUserId } from "@/lib/db/queries/configs";
|
||||
import { db, users, repositories } from "@/lib/db";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { Octokit } from "@octokit/rest";
|
||||
import { createGitHubClient } from "@/lib/github";
|
||||
import type { Repository } from "@/lib/db/schema";
|
||||
|
||||
async function testMetadataMirroringAuth() {
|
||||
@@ -108,10 +108,8 @@ async function testMetadataMirroringAuth() {
|
||||
console.log("\n🔄 Test 4: Testing metadata mirroring authentication...");
|
||||
|
||||
try {
|
||||
// Create Octokit instance
|
||||
const octokit = new Octokit({
|
||||
auth: config.githubConfig.token,
|
||||
});
|
||||
// Create Octokit instance (honors GH_API_URL for GHES / GHEC data residency)
|
||||
const octokit = createGitHubClient(config.githubConfig.token);
|
||||
|
||||
// Test by attempting to fetch labels (lightweight operation)
|
||||
const { httpGet } = await import("@/lib/http-client");
|
||||
|
||||
Reference in New Issue
Block a user