nils/papercrate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

deployment

Browse Source
This commit is contained in:
nils
2025-10-30 16:21:52 +01:00
parent 67fd787a14
commit 80d21ed0a1
5 changed files with 306 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DEVELOPMENT.md
+94
View File
@@ -0,0 +1,94 @@
# Development
This document collects runtime assumptions and workflows for local development,
integration testing, and infrastructure automation.
## Local Development
Use the provided `papercrate.tmux` to spin up the full stack in one tmux session:
```bash
tmux -f papercrate.tmux attach
```
This creates windows for the compose stack, frontend dev server, backend API, and
background worker using the repository-relative paths defined in the tmux file.
Detach with `Ctrl+b d` and reattach later with the same command.
## Backend Integration Tests
Integration tests require a running Postgres instance (and, optionally, Quickwit
for OCR indexing). The repository includes a lightweight compose file for local
runs:
```bash
docker compose -f docker-compose.test.yml up -d
export TEST_DATABASE_URL=postgres://papercrate:papercrate_test@localhost:5433/papercrate_test
# optional, enables Quickwit indexing jobs
export QUICKWIT_ENDPOINT=http://localhost:7280
export QUICKWIT_INDEX=documents
cargo test
```
Stop the database when you are done:
```bash
docker compose -f docker-compose.test.yml down
```
The compose service uses tmpfs storage, giving each test run a clean database.
## Runtime Dependencies
- `ocrmypdf` (optional but recommended): Used by the OCR worker to extract text
from PDFs when no embedded text layer is available. Ensure it is installed and
available on the worker hosts if OCR is desired.
- Quickwit (optional): The Quickwit indexer is used to ingest extracted text for
search. Set `QUICKWIT_ENDPOINT` and `QUICKWIT_INDEX` in the environment when
running workers if you want indexing jobs to run. The local compose file starts
a Quickwit instance on `http://localhost:7280` and seeds the `documents` index
automatically.
## Configuration
The backend reads its settings from environment variables. In particular:
- `DATABASE_URL` – connection string for the primary Postgres database (required).
- `DATABASE_MAX_POOL_SIZE` – optional override for the r2d2 connection pool size.
Defaults to `2`; increase it in staging/production to match expected concurrency.
On startup each binary logs the effective configuration with secrets redacted
(for example, the database password is masked). This makes it easier to confirm
runtime settings in staging without exposing credentials.
## Running Migrations in Kubernetes
The backend container image ships the `diesel` CLI, so schema migrations can be
executed as a short-lived Job (or Helm hook) before rolling out new pods. Example
manifest:
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: papercrate-migrate
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: migrate
image: ghcr.io/example/papercrate-backend:<TAG>
command: ["/usr/local/bin/diesel", "migration", "run"]
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: papercrate-db
key: DATABASE_URL
```
Run the Job manually (`kubectl apply -f migrate-job.yaml`) or configure it as a
Helm pre-install/pre-upgrade hook so migrations run automatically on each
deployment. Once the Job succeeds, deploy/update the backend `Deployment` as
usual.
README.md
+31 -63
View File
@@ -1,73 +1,41 @@
# Papercrate
## Local Development
![Papercrate document workspace](docs/screenshot.png)
Use the provided `papercrate.tmux` to spin up the full stack in one tmux session:
## Single-Host Deployment (Docker Compose)
For self-hosting (for example on a Raspberry Pi), use the production-oriented
`docker-compose.yml`. Define the required secrets in a `.env` file alongside the
compose file before starting the stack:
```bash
tmux -f papercrate.tmux attach
cat <<'EOF' > .env
POSTGRES_PASSWORD=change-me
MINIO_ROOT_PASSWORD=change-me-too
JWT_SECRET=generate-a-long-random-string
WEBAUTHN_RP_ID=papercrate.local
WEBAUTHN_ORIGIN=http://papercrate.local:8080
# Optional overrides
# CORS_ALLOWED_ORIGIN=http://papercrate.local:8080
# REFRESH_COOKIE_SECURE=true
EOF
docker compose up -d
```
This creates windows for the compose stack, frontend dev server, backend API, and background worker using the repository-relative paths defined in the tmux file. Detach with `Ctrl+b d` and reattach later with the same command.
**Important:** the WebAuthn settings must match the public URL clients will use.
The relying party (RP) identifier is the bare host name, while the origin must
include scheme and port. Adjust the values above if you serve Papercrate from a
different host, domain, or HTTPS endpoint—otherwise passkey registration and
login will fail.
## Backend Integration Tests
The compose file builds the backend and frontend images locally, then launches
Postgres, MinIO, Quickwit, the API, background worker, WebDAV endpoint, and the
SPA frontend. Once the containers report healthy, visit `http://<host>:8080`
and use the passkey signup flow to provision the first tenant/user. Upgrades are
as simple as `git pull` followed by `docker compose up -d`.
Integration tests require a running Postgres instance (and, optionally, Quickwit for OCR indexing). The repository includes a lightweight compose file for local runs:
---
```bash
docker compose -f docker-compose.test.yml up -d
export TEST_DATABASE_URL=postgres://papercrate:papercrate_test@localhost:5433/papercrate_test
# optional, enables Quickwit indexing jobs
export QUICKWIT_ENDPOINT=http://localhost:7280
export QUICKWIT_INDEX=documents
cargo test
```
Stop the database when you are done:
```bash
docker compose -f docker-compose.test.yml down
```
The compose service uses tmpfs storage, giving each test run a clean database.
## Runtime Dependencies
- `ocrmypdf` (optional but recommended): Used by the OCR worker to extract text from PDFs when no embedded text layer is available. Ensure it is installed and available on the worker hosts if OCR is desired.
- Quickwit (optional): The Quickwit indexer is used to ingest extracted text for search. Set `QUICKWIT_ENDPOINT` and `QUICKWIT_INDEX` in the environment when running workers if you want indexing jobs to run. The local compose file starts a Quickwit instance on `http://localhost:7280` and seeds the `documents` index automatically.
## Configuration
The backend reads its settings from environment variables (see `backend/.env` for local defaults). In particular:
- `DATABASE_URL` – connection string for the primary Postgres database (required).
- `DATABASE_MAX_POOL_SIZE` – optional override for the r2d2 connection pool size. Defaults to `2`; increase it in staging/production to match expected concurrency.
On startup each binary logs the effective configuration with secrets redacted (for example, the database password is masked). This makes it easier to confirm the runtime settings in staging without exposing credentials.
## Running Migrations in Kubernetes
The backend container image ships the `diesel` CLI, so schema migrations can be executed as a short-lived Job (or Helm hook) before rolling out new pods. Example manifest:
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: papercrate-migrate
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: migrate
image: ghcr.io/example/papercrate-backend:<TAG>
command: ["/usr/local/bin/diesel", "migration", "run"]
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: papercrate-db
key: DATABASE_URL
```
Run the Job manually (`kubectl apply -f migrate-job.yaml`) or configure it as a Helm pre-install/pre-upgrade hook so migrations run automatically on each deployment. Once the Job succeeds, deploy/update the backend `Deployment` as usual.
For development workflows (local stack, integration tests, migrations, and
configuration details) see [DEVELOPMENT.md](./DEVELOPMENT.md).
docker-compose.yml
+180
View File
@@ -0,0 +1,180 @@
version: "3.9"
x-app-env: &app-env
DATABASE_URL: postgres://papercrate:${POSTGRES_PASSWORD:?POSTGRES_PASSWORD must be set}@postgres:5432/papercrate
DATABASE_MAX_POOL_SIZE: ${DATABASE_MAX_POOL_SIZE:-8}
SERVER_HOST: 0.0.0.0
SERVER_PORT: 3000
WEBDAV_HOST: 0.0.0.0
WEBDAV_PORT: 3001
JWT_SECRET: ${JWT_SECRET:?JWT_SECRET must be set}
JWT_ISSUER: ${JWT_ISSUER:-papercrate}
JWT_AUDIENCE: ${JWT_AUDIENCE:-papercrate-clients}
JWT_EXPIRY_MINUTES: ${JWT_EXPIRY_MINUTES:-60}
DOWNLOAD_TOKEN_AUDIENCE: ${DOWNLOAD_TOKEN_AUDIENCE:-papercrate-download}
DOWNLOAD_TOKEN_EXPIRY_MINUTES: ${DOWNLOAD_TOKEN_EXPIRY_MINUTES:-60}
REFRESH_TOKEN_EXPIRY_DAYS: ${REFRESH_TOKEN_EXPIRY_DAYS:-30}
REFRESH_COOKIE_SECURE: ${REFRESH_COOKIE_SECURE:-false}
REFRESH_COOKIE_DOMAIN: ${REFRESH_COOKIE_DOMAIN:-}
CORS_ALLOWED_ORIGIN: ${CORS_ALLOWED_ORIGIN:-}
AWS_ENDPOINT_URL: http://minio:9000
AWS_ACCESS_KEY_ID: ${MINIO_ROOT_USER:-papercrate}
AWS_SECRET_ACCESS_KEY: ${MINIO_ROOT_PASSWORD:?MINIO_ROOT_PASSWORD must be set}
AWS_REGION: ${AWS_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-documents}
QUICKWIT_ENDPOINT: http://quickwit:7280
QUICKWIT_INDEX: ${QUICKWIT_INDEX:-documents}
WORKER_MAX_DOCUMENT_BYTES: ${WORKER_MAX_DOCUMENT_BYTES:-209715200}
UPLOAD_BODY_LIMIT_BYTES: ${UPLOAD_BODY_LIMIT_BYTES:-134217728}
WEBAUTHN_RP_ID: ${WEBAUTHN_RP_ID:-papercrate.local}
WEBAUTHN_ORIGIN: ${WEBAUTHN_ORIGIN:-https://papercrate.local}
WEBAUTHN_RP_NAME: ${WEBAUTHN_RP_NAME:-Papercrate}
RUST_LOG: ${RUST_LOG:-info}
services:
postgres:
image: postgres:16-alpine
restart: unless-stopped
environment:
POSTGRES_USER: papercrate
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD must be set}
POSTGRES_DB: papercrate
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U papercrate -d papercrate"]
interval: 10s
timeout: 5s
retries: 5
ports:
- "${POSTGRES_PORT:-5432}:5432"
minio:
image: minio/minio:latest
command: server /data --console-address ":9001"
restart: unless-stopped
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-papercrate}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?MINIO_ROOT_PASSWORD must be set}
volumes:
- minio_data:/data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
ports:
- "${MINIO_API_PORT:-9000}:9000"
- "${MINIO_CONSOLE_PORT:-9001}:9001"
createbuckets:
image: minio/mc:latest
depends_on:
minio:
condition: service_healthy
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-papercrate}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?MINIO_ROOT_PASSWORD must be set}
S3_BUCKET: ${S3_BUCKET:-documents}
entrypoint: >
/bin/sh -c "
set -e;
/usr/bin/mc alias set papercrate http://minio:9000 $${MINIO_ROOT_USER} $${MINIO_ROOT_PASSWORD};
/usr/bin/mc mb papercrate/$${S3_BUCKET} --ignore-existing;
exit 0;
"
restart: "no"
quickwit:
image: quickwit/quickwit:0.8.2
command: ["run"]
restart: unless-stopped
environment:
QW_ENABLE_API_AUTH: "false"
QW_DATA_DIR: /quickwit/data
volumes:
- quickwit_data:/quickwit/data
healthcheck:
test: ["CMD", "curl", "-sf", "http://127.0.0.1:7280/api/v1/version"]
interval: 10s
timeout: 5s
retries: 5
ports:
- "${QUICKWIT_PORT:-7280}:7280"
backend:
build:
context: ./backend
image: papercrate/backend:latest
depends_on:
postgres:
condition: service_healthy
minio:
condition: service_healthy
createbuckets:
condition: service_completed_successfully
quickwit:
condition: service_healthy
migrator:
condition: service_completed_successfully
environment:
<<: *app-env
ports:
- "${API_PORT:-3000}:3000"
healthcheck:
test: ["CMD", "curl", "-sf", "http://localhost:3000/api/health"]
interval: 10s
timeout: 5s
retries: 5
restart: unless-stopped
worker:
image: papercrate/backend:latest
depends_on:
backend:
condition: service_healthy
environment:
<<: *app-env
entrypoint: ["/usr/local/bin/papercrate-worker"]
restart: unless-stopped
webdav:
image: papercrate/backend:latest
depends_on:
backend:
condition: service_healthy
environment:
<<: *app-env
entrypoint: ["/usr/local/bin/papercrate-webdav"]
ports:
- "${WEBDAV_PORT:-3001}:3001"
restart: unless-stopped
frontend:
build:
context: ./frontend
image: papercrate/frontend:latest
environment:
API_PROXY_PASS: http://backend:3000
depends_on:
backend:
condition: service_healthy
ports:
- "${FRONTEND_PORT:-8080}:80"
restart: unless-stopped
migrator:
image: papercrate/backend:latest
depends_on:
postgres:
condition: service_healthy
environment:
<<: *app-env
entrypoint: ["/usr/local/bin/diesel"]
command: ["migration", "run"]
restart: "no"
volumes:
postgres_data:
minio_data:
quickwit_data:
docs/screenshot.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 798 KiB

papercrate.tmux
+1 -1
View File
@@ -1,5 +1,5 @@
# tmux session for Papercrate dev stack
new-session -d -s papercrate -n docker -c . 'docker compose up'
new-session -d -s papercrate -n docker -c . 'docker compose -f docker-composer.dev.yml up'
new-window -t papercrate:1 -n frontend -c ./frontend 'npm run dev'