# BeeX Uptime Monitoring

## Purpose

BeeX uses an independent Uptime Kuma instance on the AI Worker ECS to monitor external availability. It is intentionally separated from the business ECS so alerts can still be sent when the app/API service is down.

## Instance

- ECS: `beex-id-ops-ai-01`
- Public IP: `147.139.167.175`
- Uptime Kuma entry: `http://147.139.167.175/`
- Public access protection: Nginx Basic Auth
- Credential file on ECS: `/opt/beex-uptime/.credentials`
- Kuma data directory on ECS: `/opt/beex-uptime/data`

Do not write passwords or webhook secrets into this repository. Read them from the ECS credential file when needed.

## Services

### Uptime Kuma

- Container: `beex-uptime-kuma`
- Compose file: `/opt/beex-uptime/docker-compose.yml`
- Internal port: `127.0.0.1:3001`
- Public reverse proxy: Nginx port `80`

Useful commands:

```bash
ssh root@147.139.167.175
cd /opt/beex-uptime
docker compose ps
docker compose logs -f uptime-kuma
```

### Feishu Alert Bridge

The Feishu bot webhook uses signature verification. Uptime Kuma's native Feishu provider does not store a signing secret, so BeeX runs a small local bridge.

- Service: `beex-uptime-feishu-bridge.service`
- Code: `/opt/beex-uptime/feishu-bridge/bridge.py`
- Env file: `/opt/beex-uptime/feishu-bridge/env`
- Listen address: Docker gateway `172.17.0.1:3050`
- Kuma webhook URL: `http://host.docker.internal:3050/uptime/feishu`

Useful commands:

```bash
ssh root@147.139.167.175
systemctl status beex-uptime-feishu-bridge.service
journalctl -u beex-uptime-feishu-bridge.service -f
```

The bridge also normalizes Uptime Kuma TLS certificate alerts. Kuma sends certificate expiry messages in the form:

```text
[Monitor Name][https://example.com/path] server certificate [example.com] will be expired in 21 days
```

The bridge must parse this into a Feishu message with a real monitor name, URL, certificate domain, remaining days, and a clear action. It should not forward these alerts as `Unknown monitor`. The source copy currently lives at `/Users/zwp/x/beex-infra/ai-worker/feishu-bridge/bridge.py`; after editing, deploy it to `/opt/beex-uptime/feishu-bridge/bridge.py` and restart `beex-uptime-feishu-bridge.service`.

## TLS Certificate Alert Runbook

When Uptime reports that an HTTPS certificate will expire:

1. Find the service machine from the monitor URL. For `api-id-test.beexofficial.com` and `admin-api-id-test.beexofficial.com`, the current test service ECS is `beex-id-test-service`.
2. Verify the public certificate:

```bash
echo | openssl s_client -servername api-id-test.beexofficial.com -connect api-id-test.beexofficial.com:443 2>/dev/null | openssl x509 -noout -subject -issuer -enddate
```

3. If the certificate is near expiry, renew with Certbot on the owning ECS and reload Nginx.
4. Verify `/actuator/health` or the monitored URL from outside the machine.
5. Confirm the Feishu alert is actionable. It should include `类型: TLS 证书临期`, `监控项`, `地址`, `证书域名`, and `剩余天数`.

## Current Monitors

| Monitor | URL | Interval |
| --- | --- | --- |
| BeeX API ID Test Health | `https://api-id-test.beexofficial.com/actuator/health` | 60s |
| BeeX Admin API ID Test Health | `https://admin-api-id-test.beexofficial.com/actuator/health` | 60s |
| BeeX H5 ID Test Home | `https://h5-id-test.beexofficial.com/` | 60s |
| BeeX Admin Console | `https://admin.beexofficial.com/` | 60s |
| BeeX PRD Docs | `https://prd.beexofficial.com/` | 60s |
| BeeX Official Site | `https://www.beexofficial.com/` | 120s |
| BeeX App Runtime Config ID Test | `/api/v1/app/runtime-config` | 60s |
| BeeX H5 Package Latest ID Test | `/api/v1/app/h5-package/latest` | 120s |

Each monitor retries twice before being marked down. Alerts are sent to the configured Feishu bot through `BeeX Feishu Alerts`.

## Bootstrap

The bootstrap script can be rerun safely:

```bash
ssh root@147.139.167.175
/opt/beex-uptime/api-venv/bin/python /opt/beex-uptime/bootstrap.py
```

It initializes the Kuma admin account when needed, creates or updates the Feishu webhook notification, and creates or updates the standard BeeX monitors.

## Next Extensions

Add business-level heartbeats after the backend exposes stable endpoints:

- TikTok order sync lag, for example latest synced order age must be under 6 hours.
- Shopee order sync lag.
- WhatsApp inbound webhook lag.
- WhatsApp outbound send failure rate.
- H5 package latest published version freshness.
- Xendit webhook receive health after production payment is enabled.

These should not replace SLS or BI. Uptime Kuma is for availability and missing-heartbeat alerts; SLS remains the place for debugging logs and event analysis.