mirror of https://gitlab.com/Zer0interrupt/recipe-manager.git synced 2026-11-01 23:24:11 -05:00

Chris 423334db4d Add structured logging (Pino), Docker log rotation, and OpenObserve + Fluent Bit observability stack

- logger.js: Pino JSON logger with console-style helpers and pino-http request logging (static assets skipped, headers not logged, secrets redacted)

- server.js: replace 89 console.* calls with log.info/warn/error; add request logging middleware

- docker-compose.yml: json-file rotation (10m x 5, compressed), LOG_LEVEL, 'forkfolio' log tag

- observability/: standalone OpenObserve + Fluent Bit stack routing tagged container logs to per-app streams

- .dockerignore: exclude observability stack from app image

2026-11-01 01:17:21 -05:00

2.6 KiB

Raw Permalink Blame History

Observability stack (OpenObserve + Fluent Bit)

A standalone, self-hosted log pipeline shared by every app on this Docker host.

ForkFolio (Pino JSON → stdout)
   └─► Docker json-file driver (rotated: 10 MB × 5, tag "forkfolio")
          └─► Fluent Bit (tails /var/lib/docker/containers, routes by tag)
                 └─► OpenObserve stream "forkfolio" (search, dashboards, alerts, retention)

First-time setup

cd observability
cp .env.example .env        # set a strong ZO_ROOT_USER_PASSWORD
docker compose up -d

Open http://localhost:5080 and sign in with the credentials from .env.
Recreate ForkFolio so its new logging options apply:
```
cd ..
docker compose up -d --build
```
In OpenObserve go to Logs → stream forkfolio. The stream appears after the first log line arrives.

Example queries (SQL mode)

-- All errors
SELECT * FROM "forkfolio" WHERE level = 'error' ORDER BY _timestamp DESC

-- Slow API requests (> 2s)
SELECT req_method, req_url, responsetime FROM "forkfolio" WHERE responsetime > 2000

-- Scraper activity
SELECT * FROM "forkfolio" WHERE msg LIKE '%[Puppeteer]%'

OpenObserve flattens nested fields, so req.url becomes req_url. Field names are lowercased.

Adding another app

Add to the app's docker-compose.yml:

logging:
  driver: json-file
  options:
    max-size: "10m"
    max-file: "5"
    tag: "myapp"

Copy the [OUTPUT] block in fluent-bit/fluent-bit.conf, change Match to app.myapp and the URI stream to /api/${ZO_ORG}/myapp/_json.
Run docker compose restart fluent-bit.

If you'd rather collect untagged containers too, uncomment the catch-all output, which sends them to a docker stream.

Retention and rotation

Layer	Setting	Default
Docker (local disk)	`max-size` × `max-file` in each app's compose file	10 MB × 5 ≈ 50 MB
OpenObserve (long-term)	`ZO_COMPACT_DATA_RETENTION_DAYS` in `.env`, or per stream in the UI	30 days

Notes

Log level: set LOG_LEVEL (debug, info, warn, error) in ForkFolio's compose file.
Docker Desktop (Windows/macOS): the /var/lib/docker/containers mount refers to the Docker VM, and works with the WSL2 backend. On a Linux server it works as-is.
Exposure: port 5080 is published on all interfaces. Put it behind a reverse proxy with TLS, or bind it to 127.0.0.1:5080:5080, if the host is reachable from the internet.
Data: stored in ./openobserve-data (git-ignored). Back it up if you want to keep log history.

2.6 KiB Raw Permalink Blame History Unescape Escape