# Observability stack (OpenObserve + Fluent Bit)

A standalone, self-hosted log pipeline shared by every app on this Docker host.

```
ForkFolio (Pino JSON → stdout)
   └─► Docker json-file driver (rotated: 10 MB × 5, tag "forkfolio")
          └─► Fluent Bit (tails /var/lib/docker/containers, routes by tag)
                 └─► OpenObserve stream "forkfolio" (search, dashboards, alerts, retention)
```

## First-time setup

```bash
cd observability
cp .env.example .env        # set a strong ZO_ROOT_USER_PASSWORD
docker compose up -d
```

1. Open **http://localhost:5080** and sign in with the credentials from `.env`.
2. Recreate ForkFolio so its new logging options apply:
   ```bash
   cd ..
   docker compose up -d --build
   ```
3. In OpenObserve go to **Logs → stream `forkfolio`**. The stream appears after the first log line arrives.

## Example queries (SQL mode)

```sql
-- All errors
SELECT * FROM "forkfolio" WHERE level = 'error' ORDER BY _timestamp DESC

-- Slow API requests (> 2s)
SELECT req_method, req_url, responsetime FROM "forkfolio" WHERE responsetime > 2000

-- Scraper activity
SELECT * FROM "forkfolio" WHERE msg LIKE '%[Puppeteer]%'
```

OpenObserve flattens nested fields, so `req.url` becomes `req_url`. Field names are lowercased.

## Adding another app

1. Add to the app's `docker-compose.yml`:
   ```yaml
   logging:
     driver: json-file
     options:
       max-size: "10m"
       max-file: "5"
       tag: "myapp"
   ```
2. Copy the `[OUTPUT]` block in `fluent-bit/fluent-bit.conf`, change `Match` to `app.myapp` and the URI stream to `/api/${ZO_ORG}/myapp/_json`.
3. Run `docker compose restart fluent-bit`.

If you'd rather collect untagged containers too, uncomment the catch-all output, which sends them to a `docker` stream.

## Retention and rotation

| Layer | Setting | Default |
|---|---|---|
| Docker (local disk) | `max-size` × `max-file` in each app's compose file | 10 MB × 5 ≈ 50 MB |
| OpenObserve (long-term) | `ZO_COMPACT_DATA_RETENTION_DAYS` in `.env`, or per stream in the UI | 30 days |

## Notes

- **Log level:** set `LOG_LEVEL` (`debug`, `info`, `warn`, `error`) in ForkFolio's compose file.
- **Docker Desktop (Windows/macOS):** the `/var/lib/docker/containers` mount refers to the Docker VM, and works with the WSL2 backend. On a Linux server it works as-is.
- **Exposure:** port 5080 is published on all interfaces. Put it behind a reverse proxy with TLS, or bind it to `127.0.0.1:5080:5080`, if the host is reachable from the internet.
- **Data:** stored in `./openobserve-data` (git-ignored). Back it up if you want to keep log history.