diff --git a/docs/superpowers/plans/2026-04-28-monitoring-retention-storage.md b/docs/superpowers/plans/2026-04-28-monitoring-retention-storage.md new file mode 100644 index 0000000..2020249 --- /dev/null +++ b/docs/superpowers/plans/2026-04-28-monitoring-retention-storage.md @@ -0,0 +1,598 @@ +# Monitoring Retention And Storage Display Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add configurable monitoring data retention and show database storage usage on the config page. + +**Architecture:** Store retention in `vite_config` as `monitor_retention_days`, parse it through a focused monitoring helper, and reuse it from existing cleanup loops. Add a repository storage-summary helper, expose it via an admin-only API, and render it in the existing React config page. + +**Tech Stack:** Go `net/http`, GORM, SQLite/PostgreSQL, Vite/React/TypeScript, existing shadcn bridge components. + +--- + +## File Structure + +- Create: `go-backend/internal/monitoring/retention.go` for retention constants, parsing, and validation. +- Test: `go-backend/internal/monitoring/retention_test.go`. +- Modify: `go-backend/internal/metrics/ingestion.go` and `go-backend/internal/metrics/ingestion_test.go` for config-driven cleanup. +- Modify: `go-backend/internal/http/handler/tunnel_quality_prober.go` so `tunnel_quality` uses the same retention and still prunes when probing is disabled. +- Create: `go-backend/internal/store/repo/repository_storage.go` and `go-backend/internal/store/repo/repository_storage_test.go` for database size summaries. +- Modify: `go-backend/internal/store/repo/repository.go` to keep the SQLite DB path on `Repository`. +- Create: `go-backend/internal/http/handler/storage.go` for the storage endpoint. +- Modify: `go-backend/internal/http/handler/handler.go` to register `/api/v1/system/storage` and validate `monitor_retention_days`. +- Modify: `go-backend/internal/http/middleware/auth.go` so `/api/v1/system/*` is admin-only. +- Create: `go-backend/tests/contract/storage_contract_test.go` for endpoint auth/shape coverage. +- Modify: `vite-frontend/src/api/types.ts`, `vite-frontend/src/api/index.ts`, and `vite-frontend/src/pages/config.tsx` for UI display. + +Implementation should not create git commits unless the user explicitly requests them. + +--- + +### Task 1: Add Retention Config Helper + +**Files:** +- Create: `go-backend/internal/monitoring/retention.go` +- Create: `go-backend/internal/monitoring/retention_test.go` +- Modify: `go-backend/internal/http/handler/handler.go` + +- [ ] **Step 1: Write the failing tests** + +Create `go-backend/internal/monitoring/retention_test.go`: + +```go +package monitoring + +import "testing" + +func TestMonitoringRetentionDaysFromConfigMap(t *testing.T) { + tests := []struct { + name string + cfg map[string]string + want int + }{ + {"missing uses default", nil, 7}, + {"valid custom", map[string]string{ConfigMonitorRetentionDays: "3"}, 3}, + {"trimmed custom", map[string]string{ConfigMonitorRetentionDays: " 30 "}, 30}, + {"invalid uses default", map[string]string{ConfigMonitorRetentionDays: "abc"}, 7}, + {"too small uses default", map[string]string{ConfigMonitorRetentionDays: "0"}, 7}, + {"too large uses default", map[string]string{ConfigMonitorRetentionDays: "3651"}, 7}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := MonitoringRetentionDaysFromConfigMap(tc.cfg); got != tc.want { + t.Fatalf("expected %d, got %d", tc.want, got) + } + }) + } +} + +func TestNormalizeMonitoringRetentionDays(t *testing.T) { + for _, value := range []string{"1", "7", "3650", " 30 "} { + if got, err := NormalizeMonitoringRetentionDays(value); err != nil || got == "" { + t.Fatalf("expected %q valid, got value=%q err=%v", value, got, err) + } + } + for _, value := range []string{"", "0", "-1", "3651", "abc", "1.5"} { + if got, err := NormalizeMonitoringRetentionDays(value); err == nil { + t.Fatalf("expected %q invalid, got value=%q", value, got) + } + } +} +``` + +- [ ] **Step 2: Run tests to verify failure** + +Run: `go test ./internal/monitoring -run 'TestMonitoringRetentionDaysFromConfigMap|TestNormalizeMonitoringRetentionDays' -count=1` + +Expected: FAIL with undefined `ConfigMonitorRetentionDays`, `MonitoringRetentionDaysFromConfigMap`, and `NormalizeMonitoringRetentionDays`. + +- [ ] **Step 3: Implement helper** + +Create `go-backend/internal/monitoring/retention.go`: + +```go +package monitoring + +import ( + "fmt" + "strconv" + "strings" +) + +const ( + ConfigMonitorRetentionDays = "monitor_retention_days" + DefaultMonitorRetentionDays = 7 + MinMonitorRetentionDays = 1 + MaxMonitorRetentionDays = 3650 +) + +func MonitoringRetentionDaysFromConfigMap(cfg map[string]string) int { + if cfg == nil { + return DefaultMonitorRetentionDays + } + days, err := parseMonitoringRetentionDays(cfg[ConfigMonitorRetentionDays]) + if err != nil { + return DefaultMonitorRetentionDays + } + return days +} + +func NormalizeMonitoringRetentionDays(value string) (string, error) { + days, err := parseMonitoringRetentionDays(value) + if err != nil { + return "", err + } + return strconv.Itoa(days), nil +} + +func parseMonitoringRetentionDays(value string) (int, error) { + trimmed := strings.TrimSpace(value) + if trimmed == "" { + return 0, fmt.Errorf("监控数据保留天数不能为空") + } + days, err := strconv.Atoi(trimmed) + if err != nil { + return 0, fmt.Errorf("监控数据保留天数必须是整数") + } + if days < MinMonitorRetentionDays || days > MaxMonitorRetentionDays { + return 0, fmt.Errorf("监控数据保留天数必须在 %d 到 %d 之间", MinMonitorRetentionDays, MaxMonitorRetentionDays) + } + return days, nil +} +``` + +- [ ] **Step 4: Validate config updates** + +In `go-backend/internal/http/handler/handler.go`, add this case to `normalizeAndValidateConfigValue`: + +```go + case monitoring.ConfigMonitorRetentionDays: + return monitoring.NormalizeMonitoringRetentionDays(value) +``` + +- [ ] **Step 5: Run tests** + +Run: `go test ./internal/monitoring ./internal/http/handler -run 'TestMonitoringRetention|TestNormalize|Test' -count=1` + +Expected: PASS or only unrelated pre-existing failures, which must be investigated before continuing. + +--- + +### Task 2: Use Retention Config In Cleanup + +**Files:** +- Modify: `go-backend/internal/metrics/ingestion.go` +- Modify: `go-backend/internal/metrics/ingestion_test.go` +- Modify: `go-backend/internal/http/handler/tunnel_quality_prober.go` + +- [ ] **Step 1: Write failing cleanup test** + +Append to `go-backend/internal/metrics/ingestion_test.go`, adding `go-backend/internal/store/model` to imports: + +```go +func TestPruneMetricsUsesConfiguredRetentionDays(t *testing.T) { + r, err := repo.Open(":memory:") + if err != nil { + t.Fatalf("open repo: %v", err) + } + defer r.Close() + + now := time.Now().UnixMilli() + if err := r.UpsertConfig("monitor_retention_days", "2", now); err != nil { + t.Fatalf("upsert retention config: %v", err) + } + + oldMetric := &model.NodeMetric{NodeID: 1, Timestamp: now - int64(3*24*time.Hour/time.Millisecond), CPUUsage: 10} + newMetric := &model.NodeMetric{NodeID: 1, Timestamp: now - int64(1*24*time.Hour/time.Millisecond), CPUUsage: 20} + if err := r.InsertNodeMetric(oldMetric); err != nil { + t.Fatalf("insert old metric: %v", err) + } + if err := r.InsertNodeMetric(newMetric); err != nil { + t.Fatalf("insert new metric: %v", err) + } + + svc := NewIngestionService(r) + svc.pruneMetricsAt(time.UnixMilli(now)) + + metrics, err := r.GetNodeMetrics(1, now-int64(4*24*time.Hour/time.Millisecond), now+1000) + if err != nil { + t.Fatalf("get node metrics: %v", err) + } + if len(metrics) != 1 || metrics[0].CPUUsage != 20 { + t.Fatalf("expected only newer metric to remain, got %#v", metrics) + } +} +``` + +- [ ] **Step 2: Run test to verify failure** + +Run: `go test ./internal/metrics -run TestPruneMetricsUsesConfiguredRetentionDays -count=1` + +Expected: FAIL with undefined `pruneMetricsAt`. + +- [ ] **Step 3: Implement config-driven prune** + +In `go-backend/internal/metrics/ingestion.go`, import `go-backend/internal/monitoring` and replace `pruneMetrics` with: + +```go +func (s *IngestionService) retentionDaysFromConfig() int { + if s == nil || s.repo == nil { + return monitoring.DefaultMonitorRetentionDays + } + cfg, err := s.repo.GetConfigsByNames([]string{monitoring.ConfigMonitorRetentionDays}) + if err != nil { + return monitoring.DefaultMonitorRetentionDays + } + return monitoring.MonitoringRetentionDaysFromConfigMap(cfg) +} + +func (s *IngestionService) pruneMetrics() { + s.pruneMetricsAt(time.Now()) +} + +func (s *IngestionService) pruneMetricsAt(now time.Time) { + cutoff := now.Add(-time.Duration(s.retentionDaysFromConfig()) * 24 * time.Hour).UnixMilli() + if s.repo == nil { + return + } + if err := s.repo.PruneNodeMetrics(cutoff); err != nil { + log.Printf("monitoring prune failed op=node_metric cutoff=%d err=%v", cutoff, err) + } + if err := s.repo.PruneTunnelMetrics(cutoff); err != nil { + log.Printf("monitoring prune failed op=tunnel_metric cutoff=%d err=%v", cutoff, err) + } + if err := s.repo.PruneServiceMonitorResults(cutoff); err != nil { + log.Printf("monitoring prune failed op=service_monitor_result cutoff=%d err=%v", cutoff, err) + } +} +``` + +Remove the unused `retentionDays` field from `IngestionService` and remove `svc.retentionDays = 1` from existing tests. + +- [ ] **Step 4: Update tunnel quality pruning** + +In `go-backend/internal/http/handler/tunnel_quality_prober.go`, import `go-backend/internal/monitoring`, remove `tunnelQualityRetention`, remove the `if !p.isEnabled() { return }` guard from `maybePrune`, and calculate cutoff with: + +```go +func (p *tunnelQualityProber) retentionDays() int { + if p == nil || p.handler == nil || p.handler.repo == nil { + return monitoring.DefaultMonitorRetentionDays + } + cfg, err := p.handler.repo.GetConfigsByNames([]string{monitoring.ConfigMonitorRetentionDays}) + if err != nil { + return monitoring.DefaultMonitorRetentionDays + } + return monitoring.MonitoringRetentionDaysFromConfigMap(cfg) +} +``` + +Then use: + +```go + cutoff := now - int64(time.Duration(p.retentionDays())*24*time.Hour/time.Millisecond) +``` + +- [ ] **Step 5: Run cleanup tests** + +Run: `go test ./internal/metrics ./internal/http/handler -run 'TestPruneMetrics|TestPruneMetricsUsesConfiguredRetentionDays|TunnelQuality' -count=1` + +Expected: PASS. + +--- + +### Task 3: Add Storage Summary Backend API + +**Files:** +- Modify: `go-backend/internal/store/repo/repository.go` +- Create: `go-backend/internal/store/repo/repository_storage.go` +- Create: `go-backend/internal/store/repo/repository_storage_test.go` +- Create: `go-backend/internal/http/handler/storage.go` +- Modify: `go-backend/internal/http/handler/handler.go` +- Modify: `go-backend/internal/http/middleware/auth.go` +- Create: `go-backend/tests/contract/storage_contract_test.go` + +- [ ] **Step 1: Write failing repository tests** + +Create `go-backend/internal/store/repo/repository_storage_test.go`: + +```go +package repo + +import ( + "path/filepath" + "testing" + + "go-backend/internal/store/model" +) + +func TestDatabaseStorageSummarySQLiteIncludesSize(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "storage.db") + r, err := Open(dbPath) + if err != nil { + t.Fatalf("open repo: %v", err) + } + defer r.Close() + + if err := r.InsertNodeMetric(&model.NodeMetric{NodeID: 1, Timestamp: 123, CPUUsage: 1}); err != nil { + t.Fatalf("insert metric: %v", err) + } + summary, err := r.DatabaseStorageSummary() + if err != nil { + t.Fatalf("storage summary: %v", err) + } + if summary.DBType != "sqlite" || summary.DatabaseSizeBytes <= 0 || summary.DatabaseSizeText == "" { + t.Fatalf("unexpected summary: %#v", summary) + } +} + +func TestFormatDatabaseSize(t *testing.T) { + for _, tc := range []struct{ bytes int64; want string }{{0, "0 B"}, {512, "512 B"}, {1024, "1.0 KB"}, {1024 * 1024, "1.0 MB"}} { + if got := formatDatabaseSize(tc.bytes); got != tc.want { + t.Fatalf("formatDatabaseSize(%d)=%q want %q", tc.bytes, got, tc.want) + } + } +} +``` + +- [ ] **Step 2: Run test to verify failure** + +Run: `go test ./internal/store/repo -run 'TestDatabaseStorageSummarySQLiteIncludesSize|TestFormatDatabaseSize' -count=1` + +Expected: FAIL with undefined `DatabaseStorageSummary` and `formatDatabaseSize`. + +- [ ] **Step 3: Implement repository helper** + +Modify `Repository` in `repository.go`: + +```go +type Repository struct { + db *gorm.DB + dbPath string +} +``` + +Return `&Repository{db: db, dbPath: path}` from `Open` and `&Repository{db: db}` from `OpenPostgres`. + +Create `go-backend/internal/store/repo/repository_storage.go`: + +```go +package repo + +import ( + "errors" + "fmt" + "os" +) + +type DatabaseStorageSummary struct { + DBType string `json:"dbType"` + DatabaseSizeBytes int64 `json:"databaseSizeBytes"` + DatabaseSizeText string `json:"databaseSizeText"` +} + +func (r *Repository) DatabaseStorageSummary() (DatabaseStorageSummary, error) { + if r == nil || r.db == nil { + return DatabaseStorageSummary{}, errors.New("repository not initialized") + } + switch r.db.Dialector.Name() { + case "sqlite": + size, err := sqliteDatabaseFileSize(r.dbPath) + if err != nil { return DatabaseStorageSummary{}, err } + return DatabaseStorageSummary{"sqlite", size, formatDatabaseSize(size)}, nil + case "postgres": + var size int64 + if err := r.db.Raw("SELECT pg_database_size(current_database())").Scan(&size).Error; err != nil { return DatabaseStorageSummary{}, err } + return DatabaseStorageSummary{"postgres", size, formatDatabaseSize(size)}, nil + default: + return DatabaseStorageSummary{}, fmt.Errorf("unsupported database dialect %q", r.db.Dialector.Name()) + } +} + +func sqliteDatabaseFileSize(path string) (int64, error) { + if path == "" || path == ":memory:" { return 0, nil } + var total int64 + for _, candidate := range []string{path, path + "-wal", path + "-shm"} { + info, err := os.Stat(candidate) + if err != nil { + if os.IsNotExist(err) { continue } + return 0, err + } + if !info.IsDir() { total += info.Size() } + } + return total, nil +} + +func formatDatabaseSize(bytes int64) string { + if bytes < 1024 { return fmt.Sprintf("%d B", bytes) } + units := []string{"KB", "MB", "GB", "TB"} + value := float64(bytes) / 1024 + for _, unit := range units { + if value < 1024 || unit == "TB" { return fmt.Sprintf("%.1f %s", value, unit) } + value /= 1024 + } + return fmt.Sprintf("%d B", bytes) +} +``` + +- [ ] **Step 4: Add API handler and route** + +Create `go-backend/internal/http/handler/storage.go`: + +```go +package handler + +import ( + "net/http" + + "go-backend/internal/http/response" +) + +func (h *Handler) storageSummary(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet && r.Method != http.MethodPost { + response.WriteJSON(w, response.ErrDefault("请求失败")) + return + } + if h == nil || h.repo == nil { + response.WriteJSON(w, response.Err(-2, "repository not initialized")) + return + } + summary, err := h.repo.DatabaseStorageSummary() + if err != nil { + response.WriteJSON(w, response.Err(-2, err.Error())) + return + } + response.WriteJSON(w, response.OK(summary)) +} +``` + +Register in `Handler.Register`: `mux.HandleFunc("/api/v1/system/storage", h.storageSummary)`. + +In `requiresAdmin`, add: + +```go + if strings.HasPrefix(path, "/api/v1/system/") { + return true + } +``` + +- [ ] **Step 5: Write contract test for auth and shape** + +Create `go-backend/tests/contract/storage_contract_test.go` with a test that sends GET `/api/v1/system/storage` as non-admin and expects `403`, then as admin and expects `code == 0`, `dbType`, numeric `databaseSizeBytes`, and `databaseSizeText`. + +- [ ] **Step 6: Run storage tests** + +Run: `go test ./internal/store/repo ./tests/contract -run 'TestDatabaseStorageSummarySQLiteIncludesSize|TestFormatDatabaseSize|TestStorageSummaryRequiresAdminAndReturnsSize' -count=1` + +Expected: PASS. + +--- + +### Task 4: Add Frontend Config UI + +**Files:** +- Modify: `vite-frontend/src/api/types.ts` +- Modify: `vite-frontend/src/api/index.ts` +- Modify: `vite-frontend/src/pages/config.tsx` + +- [ ] **Step 1: Add API type and function** + +In `types.ts` add: + +```ts +export interface StorageSummaryApiData { + dbType: string; + databaseSizeBytes: number; + databaseSizeText: string; +} +``` + +In `index.ts`, import `StorageSummaryApiData` and add: + +```ts +export const getStorageSummary = () => + Network.get("/system/storage"); +``` + +- [ ] **Step 2: Add retention config item** + +In `config.tsx`, add to `CONFIG_ITEMS` near monitoring: + +```ts + { + key: "monitor_retention_days", + label: "监控数据保留天数", + placeholder: "7", + description: + "统一清理节点指标、隧道流量、服务监控结果和隧道质量历史；默认 7 天。", + type: "input", + }, +``` + +Add `"monitor_retention_days"` to `getInitialConfigs()` keys. + +- [ ] **Step 3: Fetch and display database size** + +In `config.tsx`, add state: + +```ts +const [storageSummary, setStorageSummary] = useState("加载中..."); +``` + +Add a load effect: + +```ts +useEffect(() => { + let mounted = true; + getStorageSummary() + .then((response) => { + if (!mounted) return; + if (response.code === 0 && response.data?.databaseSizeText) { + setStorageSummary(response.data.databaseSizeText); + } else { + setStorageSummary("获取失败"); + } + }) + .catch(() => { + if (mounted) setStorageSummary("获取失败"); + }); + return () => { + mounted = false; + }; +}, []); +``` + +Render inside the basic settings card before the save button: + +```tsx + +
+

+ 数据库占用 +

+

+ 当前后端数据库文件/实例占用空间，仅用于容量参考。 +

+
+ {storageSummary} +
+
+``` + +- [ ] **Step 4: Build frontend** + +Run: `pnpm run build` from `vite-frontend`. + +Expected: TypeScript and Vite build pass. + +--- + +### Task 5: Final Verification + +**Files:** +- All files changed by previous tasks. + +- [ ] **Step 1: Run backend tests** + +Run: `go test ./...` from `go-backend`. + +Expected: PASS. + +- [ ] **Step 2: Run frontend build** + +Run: `pnpm run build` from `vite-frontend`. + +Expected: PASS. + +- [ ] **Step 3: Review diff** + +Run: `git diff --stat` and `git diff -- docs/superpowers/specs/2026-04-28-monitoring-retention-storage-design.md docs/superpowers/plans/2026-04-28-monitoring-retention-storage.md go-backend vite-frontend`. + +Expected: Diff is limited to retention config, storage summary, tests, and config UI. + +--- + +## Self-Review + +- Spec coverage: retention config, uniform cleanup, storage summary API, frontend display, validation, and verification are covered. +- Placeholder scan: no TBD/TODO placeholders; the one contract-test step describes exact assertions even though the surrounding helper functions already exist in contract tests. +- Type consistency: backend JSON fields match frontend `StorageSummaryApiData` exactly: `dbType`, `databaseSizeBytes`, `databaseSizeText`. diff --git a/docs/superpowers/specs/2026-04-28-monitoring-retention-storage-design.md b/docs/superpowers/specs/2026-04-28-monitoring-retention-storage-design.md new file mode 100644 index 0000000..eade714 --- /dev/null +++ b/docs/superpowers/specs/2026-04-28-monitoring-retention-storage-design.md @@ -0,0 +1,74 @@ +# Monitoring Retention And Storage Display Design + +## Goal + +Add an administrator-facing configuration for monitoring data retention and display the current database storage usage in the configuration page. + +## Scope + +- Add a single config key: `monitor_retention_days`. +- Default retention is `7` days. +- Apply the retention window uniformly to: + - `node_metric` + - `tunnel_metric` + - `service_monitor_result` + - `tunnel_quality` +- Show database usage on the config page as a read-only operational value. + +## Non-Goals + +- No per-table retention settings. +- No manual purge button. +- No database vacuum/compaction action. +- No frontend test framework changes. + +## Backend Design + +### Retention Config + +- Store `monitor_retention_days` in `vite_config`, consistent with existing site settings. +- Accept integer values from `1` through `3650`. +- Missing or invalid stored values fall back to `7` days. +- `normalizeAndValidateConfigValue` rejects invalid user-submitted values so bad config does not get saved through the API. + +### Cleanup Flow + +- `metrics.IngestionService.pruneMetrics()` reads `monitor_retention_days` from the repository each hourly cleanup cycle. +- The computed cutoff is used for `node_metric`, `tunnel_metric`, and `service_monitor_result`. +- `tunnel_quality` uses the same retention config. +- `tunnel_quality` cleanup must run even when real-time tunnel quality probing is disabled; disabling probing should stop new probe writes, not stop cleanup. + +### Database Storage API + +- Add an admin-only API endpoint for storage summary, for example `/api/v1/system/storage`. +- Response fields: + - `dbType`: `sqlite` or `postgres` + - `databaseSizeBytes`: raw byte count + - `databaseSizeText`: human-readable formatted size +- SQLite implementation reports the DB file size and includes `-wal` and `-shm` sidecar files when present. +- PostgreSQL implementation uses `pg_database_size(current_database())`. +- If size cannot be determined, return an API error rather than a misleading zero. + +## Frontend Design + +- Add `monitor_retention_days` to the config page. +- Label: `监控数据保留天数`. +- Description: `统一清理节点指标、隧道流量、服务监控结果和隧道质量历史；默认 7 天。` +- Use a regular numeric input through the existing config rendering path. +- Fetch database storage summary when the config page loads. +- Display a read-only card/row named `数据库占用` with `databaseSizeText`. +- If fetching fails, show `获取失败` and keep config editing usable. + +## Error Handling + +- Invalid retention values return a validation error on save. +- Cleanup logs individual prune failures and continues with other tables, matching existing monitoring cleanup behavior. +- Storage summary failures are non-blocking in the frontend. + +## Testing + +- Backend unit tests for retention config parsing and validation. +- Backend tests proving custom retention is used by monitoring cleanup. +- Backend API/repository test for SQLite storage size returning a non-negative byte count and formatted text. +- Run `go test ./...` in `go-backend`. +- Run `pnpm run build` in `vite-frontend`.