OpenFlare/docs/en/design/observability-design.md

# Edge Observability and Business Traffic Statistics Refactor

You will learn: the problems this refactor solves (dashboard "OpenResty outbound" vs "Zone data provided" inconsistency, field and aggregation redundancy), and how the target architecture makes **the Agent report only facts and the Server interpret facts**, with access logs as the single source of truth for business traffic.

---

## 1. Goals

### 1.1 Problems to Solve

1. **Dual sources of truth**: business throughput comes from both access-log aggregation and OpenResty observability deltas, and the numbers never match long-term.
2. **Agent over-computes**: the edge pre-aggregates `TrafficReport`, throughput accumulation, and the control plane aggregates again — semantics are hard to evolve and reconcile.
3. **Field semantic overlap**: "OpenResty outbound" and "data provided" are the same business problem for users, but the system uses two field sets and two pipelines.
4. **Instant vs cumulative mixed**: 60-second window counts are treated as process cumulative values for 24h deltas, causing severe underestimation.
5. **UI induces wrong comparisons**: the dashboard and Zone page use similar "traffic/data" wording without declaring scope and caliber differences.

### 1.2 Refactor Goals

| Goal | Description |
| --- | --- |
| **Single business truth** | request count, data provided, UV, status distribution, Top domains etc. **only** derived from access logs (and Server-side rollups) |
| **Agent reports only facts** | detail logs + machine readings + health snapshots; **business UV/TopN/24h totals pre-aggregation is forbidden** |
| **Field convergence** | one business concept maps to one authoritative field; machine NIC and business delivery strictly separated by name |
| **Reconcilable** | global "data provided" ≈ sum of per-Zone "data provided" (difference only from unbound/unknown Hosts) |
| **Evolvable** | changing time windows, TopN, ownership rules only changes the Server, not the Agent |

### 1.3 Non-Goals (outside this design)

* Building a general log platform, full-log long-term archive, or search product.
* Replacing ClickHouse / removing the analytics DB dependency.
* Reworking Relay / OpenFlared host metric collection (principles align, but not in this round's protocol main path).
* Real-time streaming alert engine, APM tracing (the OpenTelemetry server side already exists and is orthogonal to this business traffic model).

---

## 2. Scope and Constraints

### 2.1 Product Constraints (inherited)

* Single-tenant, single globally active config; observability introduces no multi-tenant billing isolation.
* Access logs and time-series observability use the switchable log primary DB (ClickHouse by default; switchable to PostgreSQL/SQLite), see [Log Store Decoupling](./logstore.md).
* Agent has no inbound control, Pull model; during offline periods local OpenResty keeps serving, and observability can buffer locally and backfill.

### 2.2 Engineering Constraints

* Agent stays lightweight: parse log lines, read `/proc`, health checks; no business analysis.
* Control-plane API errors still use the unified envelope and `response.Abort*`.
* Access log field changes must update both the OpenResty `log_format` and the Agent parser simultaneously; Agent and control plane ship at the same version, no legacy protocol parsing.

---

## 3. Design Principles

### Principle P1: Agent Reports Facts, Server Interprets Facts

```text
Agent  = collection + reliable delivery (raw / near-raw)
Server = storage + aggregation + ownership + trends + reconciliation
```

**Allowed edge processing (collection)**

* Parsing JSON access.log lines into structured fields
* path length caps, dropping invalid lines, skipping observability-port's own requests
* Reading NIC/CPU/memory counters as **raw values**
* Batching, compression, offline buffering and retries

**Forbidden edge processing (business computation)**

* UV / Top domains / status histograms / window request_count as authoritative metrics
* Maintaining "business in/out cumulative" for the dashboard
* Zone / domain ownership stats, country distribution (country can be resolved at Server insert time)

### Principle P2: Single Truth for Business Traffic = Access Logs

| Business Question | Single Answer |
| --- | --- |
| How much data was provided | `sum(bytes_sent)` |
| How many requests | `count()` |
| How many unique visitors | `uniqExact(remote_addr)` (or product-defined hashing) |
| Status codes / Top domains | `group by` on logs |

### Principle P3: Three Metric Layers Never Mixed

| Layer | Name | Purpose | Typical Fields |
| --- | --- | --- | --- |
| L1 Business delivery | Business Traffic | user & Zone reconciliation, dashboard business trends | access log |
| L2 Edge health | Edge Health | is OpenResty alive, current connections | status, connections |
| L3 Host capacity | Host Capacity | capacity planning, is the machine saturated | CPU, memory, disk, **NIC** |

Never name L3 NIC or L2 instant counts as "data provided"; never draw L1 and L3 on the same summary card without labeling semantics.

### Principle P4: One Business Concept, One Field

* **Data provided** ≡ response body delivered ≡ "OpenResty outbound (business meaning)" in legacy copy → **keep only `bytes_sent` aggregation**
* **Data received** (optional) ≡ request-side volume → log `request_length` aggregation
* **Host outbound** ≡ `network_tx` delta, copy must include "host/NIC"

---

## 4. Pre-Refactor Problems (Baseline)

### 4.1 Pre-Refactor Data Flow (redundant)

```text
One HTTP request
  │
  ├─ access.log line
  │     → Agent tail → AccessLogs[]
  │     → CH of_node_access_logs
  │     → Zone "data provided" ✅
  │
  ├─ Lua shared dict window/cumulative counts
  │     → /openflare/observability
  │     → TrafficReport + OpenrestyObservation(rx/tx)
  │     → CH request_reports / obs_openresty
  │     → dashboard "OpenResty in/outbound" ❌ easily inconsistent with Zone
  │
  ├─ second access.log aggregation (fallback when observability endpoint fails)
  │     → yet another TrafficReport / throughput
  │
  └─ host network_rx/tx
        → Snapshot → "host" curve in network trends
```

### 4.2 Field Overlap

| User Perception | System Field A | System Field B | Problem |
| --- | --- | --- | --- |
| Outbound / provided | `openresty_tx_bytes` | `bytes_sent` | duplicate business semantics |
| Inbound | `openresty_rx_bytes` | `request_length` (log) | duplicate business semantics |
| Request count | `TrafficReport.request_count` | `count(access_logs)` | duplicate aggregation, window easily double-counted |
| Outbound (machine) | `network_tx_bytes` | (no business equivalent) | should be named separately, never reconciled with business |

### 4.3 Typical Failure Modes

1. Window counts treated as cumulative deltas → 24h business throughput severely underestimated.  
2. Hourly rollup `max−min` broken for resetting counters.  
3. Zone uses logs, dashboard uses observability → users think the system is wrong.  
4. Changing caliber requires syncing Lua, Agent state accumulation, Server deltas, and frontend copy.

---

## 5. Target Architecture

### 5.1 Target Data Flow

```mermaid
flowchart TB
  subgraph edge [Edge Node]
    OR[OpenResty]
    LOG[access.log]
    PROC[host /proc and disk]
    STUB[stub_status connections]
    AG[Agent]
    OR -->|log_format writes line| LOG
    LOG -->|tail incremental details only| AG
    PROC -->|reading snapshots| AG
    STUB -->|instant connections| AG
    OR -->|health probe| AG
  end

  subgraph server [Control-Plane Server]
    HB[Heartbeat / WS receive]
    CH[(ClickHouse)]
    AGG[Aggregation query layer]
    API[Admin API]
    HB --> CH
    CH --> AGG
    AGG --> API
  end

  subgraph ui [Admin Panel]
    DASH[Dashboard: global business trends]
    ZONE[Zone: filter by domain]
    NODE[Node: host resources + health]
  end

  AG -->|AccessLogs + HostSnapshot + Health| HB
  API --> DASH
  API --> ZONE
  API --> NODE
```

### 5.2 Responsibility Matrix

| Capability | Agent | Server | Frontend |
| --- | --- | --- | --- |
| Write access.log | OpenResty | — | — |
| Read and report details | ✅ | store | — |
| sum/count/uniq/TopN | ❌ | ✅ | display |
| Zone domain filtering | ❌ | ✅ | select Zone |
| Host CPU/memory/NIC | read raw values and report | delta/average | node/dashboard resource area |
| OpenResty connections | read instant and report | latest value | node health |
| Business 24h in/outbound | ❌ | log aggregation | uniformly called "data provided/received" |

---

## 6. Metrics and Field Model

### 6.1 Authoritative Field Table (target)

#### L1 Business Delivery (from access logs)

| Concept | Storage Field | Aggregation | Display Name |
| --- | --- | --- | --- |
| Request time | `logged_at` | window filter | — |
| Node | `node_id` | group | — |
| Client IP | `remote_addr` | `uniq` → UV | Unique visitors |
| Host | `host` | group / Zone mapping | Domain |
| Path | `path` | optional | — |
| Status code | `status_code` | group | Status distribution |
| **Data provided** | **`bytes_sent`** | **`sum`** | **Data provided** |
| **Data received** | **`request_length`** | **`sum`** | **Data received** (optional display) |
| Region | `region` (resolved & written by Server) | group | Source region |

> Note: the JSON key in the OpenResty `log_format` may keep the name `bytes_sent`; the value must come from **`$body_bytes_sent`** (consistent with production), representing response body delivered, i.e. "data provided".

#### L2 Edge Health (instant; no 24h business totals)

| Concept | Field | Description |
| --- | --- | --- |
| OpenResty health | `openresty_status` / message | existing |
| Current connections | `openresty_connections` | stub_status |
| (optional) rough recent-window QPS | node detail "right now" only, **never** authoritative 24h totals | if implemented must be labeled "instant" |

#### L3 Host Capacity

| Concept | Field | Display Name |
| --- | --- | --- |
| CPU / memory / disk usage | `host_metrics` | keep |
| NIC cumulative bytes | `network_rx_bytes` / `network_tx_bytes` | **Host NIC in/outbound** |
| Disk IO cumulative | `disk_read_bytes` / `disk_write_bytes` | Disk read/write |

### 6.2 Removed Fields (no compatibility layer)

| Original Field | Disposition | Reason |
| --- | --- | --- |
| `openresty_tx_bytes` / `openresty_rx_bytes` | **removed** | business bytes follow access logs |
| `TrafficReport` and TopN/window UV | **removed** | edge pre-aggregation |
| Agent state business lifetime accumulators | removed | violates P1 |
| Lua shared dict business throughput/window request counts | removed | not the delivery main path |

### 6.3 Naming Reference (frontend copy enforced)

| Forbidden Copy | Correct Copy | Data Source |
| --- | --- | --- |
| OpenResty outbound (business volume) | **Data provided** | `sum(bytes_sent)` |
| OpenResty inbound (business volume) | **Data received** | `sum(request_length)` |
| Network outbound (unspecified) | **Host NIC outbound** | `network_tx` delta |
| Two cards: data provided vs outbound | **keep only one business card** | logs |

---

## 7. Agent Design

### 7.1 Heartbeat Payload (target protocol)

Keep and strengthen:

```text
NodePayload
  identity / version / openresty_status / openresty_message  # latest state → PG
  profile                  # host overview (low frequency)
  host_metrics             # L3 resource readings (incl. NIC cumulative raw values)
  edge_health              # L2: status + connections (CH time series; message not in CH)
  access_logs[]            # L1 details (main path)
  health_events[]
  buffered[]               # buffered facts above, not reports
  waf_ip_group_checksums
```

Removed from the protocol (no compatibility layer):

```text
traffic_report
openresty_observation
snapshot / buffered_observability aliases
```

### 7.2 Access Log Reporting Requirements

Each detail at minimum contains:

| Field | Required | Note |
| --- | --- | --- |
| `logged_at_unix` | ✅ | request completion time |
| `remote_addr` | ✅ | UV |
| `host` | ✅ | Zone mapping |
| `path` | ✅ | may be truncated |
| `status_code` | ✅ | |
| `bytes_sent` | ✅ | body bytes, data provided |
| `request_length` | ✅ | data received |

Agent responsibilities:

1. Tail `access.log` by offset (reset offset on truncation/rotation, **only report new lines still present in the file**).  
2. Parse into structured form, batch into heartbeat / WS.  
3. Offline writes to local buffer, backfill by window once connected.  
4. **No sum/count/uniq on details.**

### 7.3 Host Snapshot

* Keep reporting NIC/disk **cumulative counter raw values** (not business pre-aggregation).  
* Server does non-negative deltas between adjacent samples → host trends.  
* This is unrelated to "data provided"; the UI must display it in a separate section.

### 7.4 OpenResty Local Observability

Converged state:

* Keep: health checks, `stub_status` current connections.  
* The main path no longer relies on `log.lua` shared dict business counts; `/openflare/observability` only returns health and connection snapshots, not business report sources.

### 7.5 Relationship with the Agent Design Doc

This design strengthens "pure data landing" in [Agent & Publish Model](./agent-design.md):

* Config and certificates: land and report applied state.  
* Observability: only carry facts, not business conclusions.

---

## 8. Server Design

### 8.1 Storage

| Input | Table | Description |
| --- | --- | --- |
| `access_logs[]` | `of_node_access_logs` | authoritative business details |
| `host_metrics` | `of_node_metric_snapshots` | L3; NIC/disk cumulative |
| `openresty_status` / `openresty_message` | **PG node table** | L2 **latest-state authority** (message only here) |
| `edge_health` | `of_node_edge_health` | L2 time series: status + connections (**no message**) |

GeoIP: continue resolving `remote_addr` → `region` in the Server insert path, not in the Agent.

### 8.2 Aggregation Layer (unified)

All business trends and Zone stats share the same query semantics:

```text
filter: logged_at ∈ [since, until]
optional: node_id / host IN (...)
metrics:
  request_count     = count()
  unique_visitors   = uniqExact(remote_addr)
  bytes_provided    = sum(bytes_sent)      -- data provided
  bytes_received    = sum(request_length)  -- data received
  series folded by hour/bucket
  distributions by status_code / host / region
```

Implementation locations:

* Zone: `GET .../zones/:id/stats` (existing, align field naming)  
* Dashboard: overview traffic / business network trends **switch to the same aggregation** (global, no host filter or Top filter)  
* Node detail: business volume = the same aggregation filtered by that `node_id`; host NIC still uses metric deltas  

### 8.3 Derived Rollups (optional performance path)

When detail queries over all nodes for 24h are too heavy, allow **Server-side** materialized views:

```text
of_access_log_hourly
  (hour, node_id, host, request_count, bytes_sent, bytes_received, ...)
```

Constraints:

* Derived only by CH from `of_node_access_logs`; **Agent is forbidden from writing this table directly**.  
* Zone / dashboard prefer reading the rollup, falling back to details (similar to the existing metric hourly policy).

### 8.4 Decommissioned Analytics Paths

| Path | After Migration |
| --- | --- |
| `BuildNetworkTrendPoints` delta on openresty_rx/tx | deleted, or keep only `network_*` host curves |
| `of_node_obs_openresty` throughput fields | stop writing; drop table or shrink columns after TTL expiry |
| `of_node_request_reports` + traffic hourly | business trends no longer depend on it; table can be deprecated wholesale |
| Dashboard compact openresty_tx series | change to bytes_provided series |

---

## 9. API and Frontend

### 9.1 Semantically Unified Response Fields

Business stats APIs should uniformly use:

```json
{
  "request_count": 0,
  "unique_visitors": 0,
  "bytes_provided": 0,
  "bytes_received": 0,
  "series": [
    {
      "bucket_started_at": "...",
      "request_count": 0,
      "unique_visitors": 0,
      "bytes_provided": 0,
      "bytes_received": 0
    }
  ]
}
```

API business byte fields use `bytes_provided` / `bytes_received` (access-log aggregation); no more openresty throughput aliases.

### 9.2 Dashboard

* **Business area**: request trend, data provided, data received (optional), status codes, Top domains, source regions — all L1.  
* **Resource area**: CPU/memory, **host NIC**, disk IO — all L3.  
* **Forbidden**: showing "OpenResty in/outbound" in the business area as a metric reconciled with Zone.

Suggest splitting or retitling "24-hour network and disk trends":

* "24-hour business traffic" → `bytes_provided` / `bytes_received` / requests  
* "24-hour host network and disk" → `network_*` / `disk_*`

### 9.3 Zone `/websites/:id`

* Keep cards like "total data provided".  
* Data and the dashboard business area use **the same aggregation function**, only `hosts = zone domain list`.  
* Docs and UI may note: the global dashboard includes all Hosts; this page is only this Zone.

### 9.4 Node Detail

* Business throughput: that node's `sum(bytes_sent)` etc.  
* OpenResty: health + current connections.  
* NIC: clearly "host".

---

## 10. OpenResty and Log Format

### 10.1 Keep

Existing JSON `log_format` core fields:

```text
ts, host, path, remote_addr, status, request_time,
bytes_sent (= $body_bytes_sent), request_length
```

### 10.2 Changes

* No longer rely on log-phase writes of business shared dict counts as control-plane input.  
* Observability-port requests continue not writing business stats (or `access_log off`).

### 10.3 Agent Parsing

* Protocol `NodeAccessLog` adds `request_length`.  
* Legacy log lines missing fields default to 0, not blocking the whole batch.

---

## 11. Upgrade and Migration (no compatibility layer)

### 11.1 Phase Review (shipped)

| Phase | Content |
| --- | --- |
| **M1–M5** | read path switches to access logs; protocol v2; stop pre-aggregation; edge_health + access_log_hourly; drop old tables and API compat fields |

### 11.2 Upgrade Strategy

* **Agent: destroy-and-recreate preferred**; binary replacement allowed.  
* On binary replacement: the local old observability buffer (including `snapshot` / `openresty_observation` / `traffic_report`) is **deleted wholesale**, rebuilt after running.  
* Server **does not** parse v1 fields, **does not** dual-read request_reports / openresty throughput.  
* Detail-missing periods: business charts are empty or partial; **never** impersonate data provided with NIC or removed openresty throughput.

### 11.3 Data Backfill

* Historical "data provided" follows access logs.  
* Before `of_access_log_hourly` is created, history is backfilled with goose SQL (ANTI JOIN to prevent duplicates).

### 11.4 Health-State Authority

* **Current state**: PG `openresty_status` / `openresty_message`.  
* **Time series**: log primary DB `of_node_edge_health` (status + connections; no message).

### 11.5 UV

* **Whole-window unique visitors**: `uniqExact(remote_addr)` (dashboard totals, Zone totals).  
* **Bucketed UV** (Zone curves): per-bucket uniq, **not summable across buckets**; UI must note it.  
* **Hourly trend path**: don't plot / fill per-hour UV (hourly table has no UV).

---

## 12. Storage and Capacity

* Business trends rely on details or hourly rollups; watch `of_node_access_logs` TTL and sampling.  
* If details are too large: prefer **Server-side rollup** rather than restoring Agent pre-aggregation.  
* For high-cardinality path scenarios, limit detail path length (existing); aggregation doesn't do global Top over full paths by default.

---

## 13. Risks and Trade-offs

| Risk | Mitigation |
| --- | --- |
| Large detail volume makes CH and heartbeat heavy | batching, compression, sampling policy evaluation; Server rollup; limit per-batch count |
| Brief log loss lowers business volume | local buffer and rotation handling; monitor access log collection lag |
| Users still compare "NIC outbound" with "data provided" | UI sections and copy enforce the "host" prefix |
| Old Agents stay online long-term | **no compatibility layer**; Agents must be upgraded/rebuilt |

**Why not keep Agent pre-aggregation as an optimization?**

* Saving bandwidth re-splits the truth, drifts calibers, and repeats this problem.  
* Optimization belongs in Server derived tables and queries, not edge business computation.

---

## 14. Key Decision Summary

| Decision | Choice | Rejected Alternative |
| --- | --- | --- |
| Business traffic truth | access logs | OpenResty dict / TrafficReport |
| Agent role | report only facts | edge UV/TopN/throughput accumulation |
| "Outbound" vs "provided" | merged into data provided | dual fields and dual pipelines long-term |
| NIC traffic | independent L3, separate copy | reconciled side-by-side with business outbound |
| Performance | CH rollup | Agent pre-aggregation |
| Migration | switch read path first, then slim Agent | drop details first, rely on pre-aggregation |

---

## 15. Doc and Code Mapping

| Area | Main Paths |
| --- | --- |
| Protocol | `pkg/protocol/agent.go` |
| Agent collection | `internal/apps/agent/observability/`, `heartbeat/` |
| OpenResty logging and Lua | `pkg/render/openresty/`, `internal/apps/agent/nginx/observability_assets.go` |
| Server storage | `internal/apps/openflare/agent/observability.go` |
| Log aggregation | `internal/repository/analytics/node_access_log*.go`, `internal/apps/openflare/zone/stats.go` |
| Dashboard | `internal/apps/openflare/dashboard/`, `internal/apps/openflare/observability/analytics.go` |
| Frontend | `frontend/app/(main)/page.tsx`, `components/dashboard/*`, `websites/.../zone-overview.tsx` |

**Recommended reading order:**

1. **[Observability Transport Model](./observability-transport-model.md)** (latest: what to send, where collected from, frequency, sample JSON)  
2. [Agent Reporting Protocol and Observability Data Model](./observability-data-model.md) (protocol fields and DDL)

---

## 16. Revision History

| Date | Notes |
| --- | --- |
| 2026-07-17 | initial draft: target architecture and migration phases for dual truth, Agent pre-aggregation, field redundancy |
| 2026-07-17 | added protocol/table-structure chapter links `observability-data-model.md` |