mirror of
https://github.com/Rain-kl/OpenFlare.git
synced 2026-10-05 15:26:36 +08:00
feat(access-logs): 上报并展示边缘缓存状态 cache_status
OpenResty 日志输出 $upstream_cache_status;Agent/协议/ClickHouse 贯通入库。 明细列表与详情按 HIT/MISS 等推导命中、回源、未缓存三态标签。
This commit is contained in:
@@ -28,6 +28,7 @@ sidebar: false
|
|||||||
- 边缘访问日志现支持上报并存储 User-Agent;概览新增设备类型饼图,以及浏览器、操作系统、User-Agent 排行。
|
- 边缘访问日志现支持上报并存储 User-Agent;概览新增设备类型饼图,以及浏览器、操作系统、User-Agent 排行。
|
||||||
- 访问日志明细支持查看单条请求详情,可查看 User-Agent、IP 访问趋势与分析,并快捷将 IP 加入或移出 IP 组。
|
- 访问日志明细支持查看单条请求详情,可查看 User-Agent、IP 访问趋势与分析,并快捷将 IP 加入或移出 IP 组。
|
||||||
- 访问日志页签状态改为 URL 参数记忆,筛选后不会再跳回概览。
|
- 访问日志页签状态改为 URL 参数记忆,筛选后不会再跳回概览。
|
||||||
|
- 访问日志支持边缘缓存状态(`cache_status`):明细列表与详情可查看命中 / 回源 / 未缓存。
|
||||||
- 优化访问日志概览的设备类型与状态码饼图布局,使设备类型与状态码饼图在 1024px(lg)以上屏幕保持双列并排展示,提升页面紧凑度。
|
- 优化访问日志概览的设备类型与状态码饼图布局,使设备类型与状态码饼图在 1024px(lg)以上屏幕保持双列并排展示,提升页面紧凑度。
|
||||||
- 修复访问日志概览在 ClickHouse 查询发生错误时静默吞掉报错的缺陷,引入错误日志输出,方便定位问题。
|
- 修复访问日志概览在 ClickHouse 查询发生错误时静默吞掉报错的缺陷,引入错误日志输出,方便定位问题。
|
||||||
|
|
||||||
|
|||||||
@@ -14390,6 +14390,10 @@ const docTemplate = `{
|
|||||||
"description": "body bytes = 已提供数据",
|
"description": "body bytes = 已提供数据",
|
||||||
"type": "integer"
|
"type": "integer"
|
||||||
},
|
},
|
||||||
|
"cache_status": {
|
||||||
|
"description": "$upstream_cache_status",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
"host": {
|
"host": {
|
||||||
"type": "string"
|
"type": "string"
|
||||||
},
|
},
|
||||||
@@ -16511,6 +16515,9 @@ const docTemplate = `{
|
|||||||
"observability.AccessLogView": {
|
"observability.AccessLogView": {
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"properties": {
|
"properties": {
|
||||||
|
"cache_status": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
"host": {
|
"host": {
|
||||||
"type": "string"
|
"type": "string"
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -0,0 +1,62 @@
|
|||||||
|
# 访问日志 cache_status 明细可见 — 实现计划
|
||||||
|
|
||||||
|
说明:对应设计 [observability-data-model.md §3.5.1](../design/observability-data-model.md)。第一期只做明细可见,不上报 upstream 地址。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 目标与背景
|
||||||
|
|
||||||
|
* **需求背景**:访问日志无法判断请求是否命中边缘缓存、是否回源。
|
||||||
|
* **开发范围 (Scope)**:
|
||||||
|
* **必做**:OpenResty 日志输出 `$upstream_cache_status`;Agent 上报;CH 入库;列表/详情展示三态标签。
|
||||||
|
* **Out of Scope**:命中率看板、hourly 维度、`upstream_addr`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 设计与决策
|
||||||
|
|
||||||
|
* **唯一字段**:`cache_status` string(原始值)。
|
||||||
|
* **UI 三态(不落库)**:
|
||||||
|
* 命中:`HIT` / `STALE` / `REVALIDATED` / `UPDATING`
|
||||||
|
* 回源:`MISS` / `EXPIRED`
|
||||||
|
* 未缓存:`BYPASS` / `-` / 空
|
||||||
|
* **数据流**:log_format → Agent parse → protocol → Server model → CH → API → 前端明细。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 修改清单
|
||||||
|
|
||||||
|
### 边缘 / 协议
|
||||||
|
|
||||||
|
* `pkg/render/openresty/types.go`、`internal/model/openflare_option.go`:`log_format` 增加 `cache_status`
|
||||||
|
* `internal/apps/agent/observability/traffic.go`:解析与映射
|
||||||
|
* `pkg/protocol/agent.go`:`NodeAccessLog.CacheStatus`
|
||||||
|
|
||||||
|
### Server / CH
|
||||||
|
|
||||||
|
* goose:`202607180005_access_log_cache_status.sql`
|
||||||
|
* `internal/model/analytics/node_access_log.go`、writer、list/scan、store 映射
|
||||||
|
* `internal/model/openflare_observability.go`、agent build records
|
||||||
|
* API `AccessLogView` + list 响应带 `cache_status`
|
||||||
|
|
||||||
|
### 前端
|
||||||
|
|
||||||
|
* types / 明细列表标签 / 详情字段
|
||||||
|
* 三态 helper:`resolveCacheOutcome(cache_status)`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 验证
|
||||||
|
|
||||||
|
* `go test ./internal/apps/agent/observability/ ./internal/repository/analytics/ ./internal/apps/openflare/agent/`
|
||||||
|
* `make swagger`(若 Handler 响应结构变更)
|
||||||
|
* `make code-check` / `make prettier`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 落地进度
|
||||||
|
|
||||||
|
* [x] log_format + protocol + agent parse
|
||||||
|
* [x] CH migration + 写入/读取
|
||||||
|
* [x] API + 前端明细展示
|
||||||
|
* [ ] 测试与提交
|
||||||
@@ -16,6 +16,7 @@
|
|||||||
* [Zone 与域名资源重构](./20260712-zone-domain-refactor.md):以 Zone 和正规化 Zone 域名替代托管域名及反代路由中的域名/证书冗余字段。
|
* [Zone 与域名资源重构](./20260712-zone-domain-refactor.md):以 Zone 和正规化 Zone 域名替代托管域名及反代路由中的域名/证书冗余字段。
|
||||||
* [WAF 可编排规则](./20260713-waf-orchestration.md):使用 React Flow 编辑 DAG 规则,发布时编译并由 OpenResty 纯内存执行。
|
* [WAF 可编排规则](./20260713-waf-orchestration.md):使用 React Flow 编辑 DAG 规则,发布时编译并由 OpenResty 纯内存执行。
|
||||||
* [边缘可观测与业务流量统计重构](./20260717-observability-redesign.md):访问日志为业务唯一真相;Agent 只上报明细与主机读数;收敛「出站/已提供」双字段。
|
* [边缘可观测与业务流量统计重构](./20260717-observability-redesign.md):访问日志为业务唯一真相;Agent 只上报明细与主机读数;收敛「出站/已提供」双字段。
|
||||||
|
* [访问日志 cache_status 明细可见](./20260718-access-log-cache-status.md):上报 `$upstream_cache_status`,明细展示命中/回源/未缓存三态。
|
||||||
|
|
||||||
## 使用建议
|
## 使用建议
|
||||||
|
|
||||||
|
|||||||
@@ -14383,6 +14383,10 @@
|
|||||||
"description": "body bytes = 已提供数据",
|
"description": "body bytes = 已提供数据",
|
||||||
"type": "integer"
|
"type": "integer"
|
||||||
},
|
},
|
||||||
|
"cache_status": {
|
||||||
|
"description": "$upstream_cache_status",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
"host": {
|
"host": {
|
||||||
"type": "string"
|
"type": "string"
|
||||||
},
|
},
|
||||||
@@ -16504,6 +16508,9 @@
|
|||||||
"observability.AccessLogView": {
|
"observability.AccessLogView": {
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"properties": {
|
"properties": {
|
||||||
|
"cache_status": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
"host": {
|
"host": {
|
||||||
"type": "string"
|
"type": "string"
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -657,6 +657,9 @@ definitions:
|
|||||||
bytes_sent:
|
bytes_sent:
|
||||||
description: body bytes = 已提供数据
|
description: body bytes = 已提供数据
|
||||||
type: integer
|
type: integer
|
||||||
|
cache_status:
|
||||||
|
description: $upstream_cache_status
|
||||||
|
type: string
|
||||||
host:
|
host:
|
||||||
type: string
|
type: string
|
||||||
logged_at_unix:
|
logged_at_unix:
|
||||||
@@ -2065,6 +2068,8 @@ definitions:
|
|||||||
type: object
|
type: object
|
||||||
observability.AccessLogView:
|
observability.AccessLogView:
|
||||||
properties:
|
properties:
|
||||||
|
cache_status:
|
||||||
|
type: string
|
||||||
host:
|
host:
|
||||||
type: string
|
type: string
|
||||||
id:
|
id:
|
||||||
|
|||||||
@@ -46,9 +46,11 @@ import { formatBytes, formatCompactNumber } from '@/lib/utils/metrics';
|
|||||||
|
|
||||||
import { buildIPGroupPayloadFromGroup } from '../../waf/components/helpers';
|
import { buildIPGroupPayloadFromGroup } from '../../waf/components/helpers';
|
||||||
import {
|
import {
|
||||||
|
cacheOutcomeLabel,
|
||||||
formatOverviewRangeHint,
|
formatOverviewRangeHint,
|
||||||
formatOverviewTrendLabel,
|
formatOverviewTrendLabel,
|
||||||
OVERVIEW_RANGE_OPTIONS,
|
OVERVIEW_RANGE_OPTIONS,
|
||||||
|
resolveCacheOutcome,
|
||||||
type OverviewRangeHours,
|
type OverviewRangeHours,
|
||||||
} from './access-log-utils';
|
} from './access-log-utils';
|
||||||
|
|
||||||
@@ -463,6 +465,21 @@ export function AccessLogDetailDialog({
|
|||||||
</Badge>
|
</Badge>
|
||||||
}
|
}
|
||||||
/>
|
/>
|
||||||
|
<DetailField
|
||||||
|
label='缓存'
|
||||||
|
value={
|
||||||
|
<div className='flex flex-wrap items-center gap-2'>
|
||||||
|
<Badge variant='outline' className='text-[10px]'>
|
||||||
|
{cacheOutcomeLabel(
|
||||||
|
resolveCacheOutcome(activeItem.cache_status),
|
||||||
|
)}
|
||||||
|
</Badge>
|
||||||
|
<span className='font-mono text-xs text-muted-foreground'>
|
||||||
|
{activeItem.cache_status || '—'}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
}
|
||||||
|
/>
|
||||||
<DetailField
|
<DetailField
|
||||||
label='路径'
|
label='路径'
|
||||||
value={activeItem.path || '—'}
|
value={activeItem.path || '—'}
|
||||||
|
|||||||
@@ -64,3 +64,34 @@ export function formatOverviewTrendLabel(value: string, hours: number) {
|
|||||||
}
|
}
|
||||||
return `${month}/${day} ${hour}:00`;
|
return `${month}/${day} ${hour}:00`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export type CacheOutcome = 'hit' | 'origin' | 'uncached';
|
||||||
|
|
||||||
|
export function resolveCacheOutcome(
|
||||||
|
cacheStatus: string | undefined | null,
|
||||||
|
): CacheOutcome {
|
||||||
|
const status = (cacheStatus ?? '').trim().toUpperCase();
|
||||||
|
if (
|
||||||
|
status === 'HIT' ||
|
||||||
|
status === 'STALE' ||
|
||||||
|
status === 'REVALIDATED' ||
|
||||||
|
status === 'UPDATING'
|
||||||
|
) {
|
||||||
|
return 'hit';
|
||||||
|
}
|
||||||
|
if (status === 'MISS' || status === 'EXPIRED') {
|
||||||
|
return 'origin';
|
||||||
|
}
|
||||||
|
return 'uncached';
|
||||||
|
}
|
||||||
|
|
||||||
|
export function cacheOutcomeLabel(outcome: CacheOutcome) {
|
||||||
|
switch (outcome) {
|
||||||
|
case 'hit':
|
||||||
|
return '命中';
|
||||||
|
case 'origin':
|
||||||
|
return '回源';
|
||||||
|
default:
|
||||||
|
return '未缓存';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -32,7 +32,25 @@ import type { AccessLogItem, AccessLogList } from '@/lib/services/openflare';
|
|||||||
import { formatDateTime } from '@/lib/utils';
|
import { formatDateTime } from '@/lib/utils';
|
||||||
|
|
||||||
import { AccessLogDetailDialog } from './access-log-detail-dialog';
|
import { AccessLogDetailDialog } from './access-log-detail-dialog';
|
||||||
import { DETAIL_SORT_OPTIONS } from './access-log-utils';
|
import {
|
||||||
|
cacheOutcomeLabel,
|
||||||
|
DETAIL_SORT_OPTIONS,
|
||||||
|
resolveCacheOutcome,
|
||||||
|
type CacheOutcome,
|
||||||
|
} from './access-log-utils';
|
||||||
|
|
||||||
|
function cacheOutcomeVariant(
|
||||||
|
outcome: CacheOutcome,
|
||||||
|
): 'default' | 'secondary' | 'outline' | 'destructive' {
|
||||||
|
switch (outcome) {
|
||||||
|
case 'hit':
|
||||||
|
return 'default';
|
||||||
|
case 'origin':
|
||||||
|
return 'secondary';
|
||||||
|
default:
|
||||||
|
return 'outline';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
function PaginationBar({
|
function PaginationBar({
|
||||||
page,
|
page,
|
||||||
@@ -136,53 +154,66 @@ export function DetailTab({
|
|||||||
<TableHead className='text-xs'>IP</TableHead>
|
<TableHead className='text-xs'>IP</TableHead>
|
||||||
<TableHead className='text-xs'>域名</TableHead>
|
<TableHead className='text-xs'>域名</TableHead>
|
||||||
<TableHead className='text-xs'>路径</TableHead>
|
<TableHead className='text-xs'>路径</TableHead>
|
||||||
|
<TableHead className='text-xs'>缓存</TableHead>
|
||||||
<TableHead className='text-xs'>状态码</TableHead>
|
<TableHead className='text-xs'>状态码</TableHead>
|
||||||
<TableHead className='w-12 text-center text-xs' />
|
<TableHead className='w-12 text-center text-xs' />
|
||||||
</TableRow>
|
</TableRow>
|
||||||
</TableHeader>
|
</TableHeader>
|
||||||
<TableBody>
|
<TableBody>
|
||||||
{(data?.items ?? []).map((item) => (
|
{(data?.items ?? []).map((item) => {
|
||||||
<TableRow key={item.id} className='border-dashed'>
|
const outcome = resolveCacheOutcome(item.cache_status);
|
||||||
<TableCell className='text-xs'>
|
return (
|
||||||
{formatDateTime(item.logged_at)}
|
<TableRow key={item.id} className='border-dashed'>
|
||||||
</TableCell>
|
<TableCell className='text-xs'>
|
||||||
<TableCell className='text-xs'>
|
{formatDateTime(item.logged_at)}
|
||||||
{item.node_name || item.node_id}
|
</TableCell>
|
||||||
</TableCell>
|
<TableCell className='text-xs'>
|
||||||
<TableCell className='text-xs font-mono'>
|
{item.node_name || item.node_id}
|
||||||
{item.remote_addr}
|
</TableCell>
|
||||||
</TableCell>
|
<TableCell className='text-xs font-mono'>
|
||||||
<TableCell className='text-xs'>{item.host}</TableCell>
|
{item.remote_addr}
|
||||||
<TableCell className='text-xs max-w-48 truncate'>
|
</TableCell>
|
||||||
{item.path}
|
<TableCell className='text-xs'>{item.host}</TableCell>
|
||||||
</TableCell>
|
<TableCell className='text-xs max-w-48 truncate'>
|
||||||
<TableCell>
|
{item.path}
|
||||||
<Badge variant='outline' className='text-[10px]'>
|
</TableCell>
|
||||||
{item.status_code}
|
<TableCell>
|
||||||
</Badge>
|
<Badge
|
||||||
</TableCell>
|
variant={cacheOutcomeVariant(outcome)}
|
||||||
<TableCell className='text-center'>
|
className='text-[10px]'
|
||||||
<Tooltip>
|
title={item.cache_status || undefined}
|
||||||
<TooltipTrigger asChild>
|
>
|
||||||
<Button
|
{cacheOutcomeLabel(outcome)}
|
||||||
variant='ghost'
|
</Badge>
|
||||||
size='icon'
|
</TableCell>
|
||||||
className='h-6 w-6 text-muted-foreground hover:text-foreground'
|
<TableCell>
|
||||||
onClick={() => {
|
<Badge variant='outline' className='text-[10px]'>
|
||||||
setSelected(item);
|
{item.status_code}
|
||||||
setDetailOpen(true);
|
</Badge>
|
||||||
}}
|
</TableCell>
|
||||||
>
|
<TableCell className='text-center'>
|
||||||
<Eye className='size-3' />
|
<Tooltip>
|
||||||
</Button>
|
<TooltipTrigger asChild>
|
||||||
</TooltipTrigger>
|
<Button
|
||||||
<TooltipContent side='top' className='text-xs'>
|
variant='ghost'
|
||||||
查看详情
|
size='icon'
|
||||||
</TooltipContent>
|
className='h-6 w-6 text-muted-foreground hover:text-foreground'
|
||||||
</Tooltip>
|
onClick={() => {
|
||||||
</TableCell>
|
setSelected(item);
|
||||||
</TableRow>
|
setDetailOpen(true);
|
||||||
))}
|
}}
|
||||||
|
>
|
||||||
|
<Eye className='size-3' />
|
||||||
|
</Button>
|
||||||
|
</TooltipTrigger>
|
||||||
|
<TooltipContent side='top' className='text-xs'>
|
||||||
|
查看详情
|
||||||
|
</TooltipContent>
|
||||||
|
</Tooltip>
|
||||||
|
</TableCell>
|
||||||
|
</TableRow>
|
||||||
|
);
|
||||||
|
})}
|
||||||
</TableBody>
|
</TableBody>
|
||||||
</Table>
|
</Table>
|
||||||
)}
|
)}
|
||||||
|
|||||||
@@ -544,6 +544,7 @@ export interface AccessLogItem {
|
|||||||
host: string;
|
host: string;
|
||||||
path: string;
|
path: string;
|
||||||
user_agent: string;
|
user_agent: string;
|
||||||
|
cache_status: string;
|
||||||
status_code: number;
|
status_code: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -23,6 +23,7 @@ type accessLogRecord struct {
|
|||||||
RemoteAddr string `json:"remote_addr"`
|
RemoteAddr string `json:"remote_addr"`
|
||||||
Path string `json:"path"`
|
Path string `json:"path"`
|
||||||
UserAgent string `json:"user_agent"`
|
UserAgent string `json:"user_agent"`
|
||||||
|
CacheStatus string `json:"cache_status"`
|
||||||
Status int `json:"status"`
|
Status int `json:"status"`
|
||||||
BytesSent int64 `json:"bytes_sent"`
|
BytesSent int64 `json:"bytes_sent"`
|
||||||
RequestLength int64 `json:"request_length"`
|
RequestLength int64 `json:"request_length"`
|
||||||
@@ -147,6 +148,7 @@ func (aggregate *trafficAggregate) consume(line []byte) {
|
|||||||
Host: strings.TrimSpace(record.Host),
|
Host: strings.TrimSpace(record.Host),
|
||||||
Path: normalizeAccessLogPath(record.Path),
|
Path: normalizeAccessLogPath(record.Path),
|
||||||
UserAgent: strings.TrimSpace(record.UserAgent),
|
UserAgent: strings.TrimSpace(record.UserAgent),
|
||||||
|
CacheStatus: normalizeCacheStatus(record.CacheStatus),
|
||||||
StatusCode: record.Status,
|
StatusCode: record.Status,
|
||||||
BytesSent: record.BytesSent,
|
BytesSent: record.BytesSent,
|
||||||
RequestLength: record.RequestLength,
|
RequestLength: record.RequestLength,
|
||||||
@@ -160,6 +162,7 @@ type parsedAccessLogRecord struct {
|
|||||||
RemoteAddr string
|
RemoteAddr string
|
||||||
Path string
|
Path string
|
||||||
UserAgent string
|
UserAgent string
|
||||||
|
CacheStatus string
|
||||||
Status int
|
Status int
|
||||||
BytesSent int64
|
BytesSent int64
|
||||||
RequestLength int64
|
RequestLength int64
|
||||||
@@ -193,6 +196,7 @@ func parseJSONAccessLogRecord(raw string) (parsedAccessLogRecord, bool) {
|
|||||||
RemoteAddr: strings.TrimSpace(record.RemoteAddr),
|
RemoteAddr: strings.TrimSpace(record.RemoteAddr),
|
||||||
Path: normalizeAccessLogPath(record.Path),
|
Path: normalizeAccessLogPath(record.Path),
|
||||||
UserAgent: strings.TrimSpace(record.UserAgent),
|
UserAgent: strings.TrimSpace(record.UserAgent),
|
||||||
|
CacheStatus: normalizeCacheStatus(record.CacheStatus),
|
||||||
Status: record.Status,
|
Status: record.Status,
|
||||||
BytesSent: record.BytesSent,
|
BytesSent: record.BytesSent,
|
||||||
RequestLength: record.RequestLength,
|
RequestLength: record.RequestLength,
|
||||||
@@ -200,6 +204,14 @@ func parseJSONAccessLogRecord(raw string) (parsedAccessLogRecord, bool) {
|
|||||||
}, true
|
}, true
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func normalizeCacheStatus(value string) string {
|
||||||
|
trimmed := strings.TrimSpace(value)
|
||||||
|
if trimmed == "-" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return trimmed
|
||||||
|
}
|
||||||
|
|
||||||
func parseCombinedAccessLogRecord(raw string) (parsedAccessLogRecord, bool) {
|
func parseCombinedAccessLogRecord(raw string) (parsedAccessLogRecord, bool) {
|
||||||
matches := combinedAccessLogPattern.FindStringSubmatch(raw)
|
matches := combinedAccessLogPattern.FindStringSubmatch(raw)
|
||||||
if len(matches) != combinedAccessLogMatchGroupCount {
|
if len(matches) != combinedAccessLogMatchGroupCount {
|
||||||
|
|||||||
@@ -14,8 +14,8 @@ func TestCollectAccessLogsReturnsFactsOnly(t *testing.T) {
|
|||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
logPath := filepath.Join(tempDir, "openflare_access.log")
|
logPath := filepath.Join(tempDir, "openflare_access.log")
|
||||||
content := []byte(
|
content := []byte(
|
||||||
"{\"ts\":\"2026-03-14T08:00:00Z\",\"host\":\"app.example.com\",\"path\":\"/login\",\"remote_addr\":\"10.0.0.1\",\"status\":200,\"request_length\":128,\"bytes_sent\":512,\"request_time\":0.015,\"user_agent\":\"Mozilla/5.0\"}\n" +
|
"{\"ts\":\"2026-03-14T08:00:00Z\",\"host\":\"app.example.com\",\"path\":\"/login\",\"remote_addr\":\"10.0.0.1\",\"status\":200,\"request_length\":128,\"bytes_sent\":512,\"request_time\":0.015,\"user_agent\":\"Mozilla/5.0\",\"cache_status\":\"HIT\"}\n" +
|
||||||
"{\"ts\":\"2026-03-14T08:00:05Z\",\"host\":\"api.example.com\",\"path\":\"/v1/ping\",\"remote_addr\":\"10.0.0.2\",\"status\":502,\"request_length\":64,\"bytes_sent\":256,\"request_time\":0.008,\"user_agent\":\"curl/8.0\"}\n",
|
"{\"ts\":\"2026-03-14T08:00:05Z\",\"host\":\"api.example.com\",\"path\":\"/v1/ping\",\"remote_addr\":\"10.0.0.2\",\"status\":502,\"request_length\":64,\"bytes_sent\":256,\"request_time\":0.008,\"user_agent\":\"curl/8.0\",\"cache_status\":\"MISS\"}\n",
|
||||||
)
|
)
|
||||||
if err := os.WriteFile(logPath, content, 0o644); err != nil {
|
if err := os.WriteFile(logPath, content, 0o644); err != nil {
|
||||||
t.Fatalf("WriteFile failed: %v", err)
|
t.Fatalf("WriteFile failed: %v", err)
|
||||||
@@ -38,6 +38,9 @@ func TestCollectAccessLogsReturnsFactsOnly(t *testing.T) {
|
|||||||
if accessLogs[0].UserAgent != "Mozilla/5.0" || accessLogs[1].UserAgent != "curl/8.0" {
|
if accessLogs[0].UserAgent != "Mozilla/5.0" || accessLogs[1].UserAgent != "curl/8.0" {
|
||||||
t.Fatalf("unexpected user agents: %+v", accessLogs)
|
t.Fatalf("unexpected user agents: %+v", accessLogs)
|
||||||
}
|
}
|
||||||
|
if accessLogs[0].CacheStatus != "HIT" || accessLogs[1].CacheStatus != "MISS" {
|
||||||
|
t.Fatalf("unexpected cache status: %+v", accessLogs)
|
||||||
|
}
|
||||||
|
|
||||||
snapshot, err := stateStore.Load()
|
snapshot, err := stateStore.Load()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -24,7 +24,8 @@ const (
|
|||||||
healthSeverityWarning = "warning"
|
healthSeverityWarning = "warning"
|
||||||
healthSeverityCritical = "critical"
|
healthSeverityCritical = "critical"
|
||||||
accessLogPathMaxLength = 100
|
accessLogPathMaxLength = 100
|
||||||
accessLogUserAgentMaxLength = 512
|
accessLogUserAgentMaxLength = 512
|
||||||
|
accessLogCacheStatusMaxLength = 32
|
||||||
healthEventMessageMaxLength = 4096
|
healthEventMessageMaxLength = 4096
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -210,6 +211,7 @@ func buildNodeAccessLogRecords(nodeID string, direct []NodeAccessLog, buffered [
|
|||||||
Host: strings.TrimSpace(item.Host),
|
Host: strings.TrimSpace(item.Host),
|
||||||
Path: truncateForDatabase(strings.TrimSpace(item.Path), accessLogPathMaxLength),
|
Path: truncateForDatabase(strings.TrimSpace(item.Path), accessLogPathMaxLength),
|
||||||
UserAgent: truncateForDatabase(strings.TrimSpace(item.UserAgent), accessLogUserAgentMaxLength),
|
UserAgent: truncateForDatabase(strings.TrimSpace(item.UserAgent), accessLogUserAgentMaxLength),
|
||||||
|
CacheStatus: truncateForDatabase(strings.TrimSpace(item.CacheStatus), accessLogCacheStatusMaxLength),
|
||||||
StatusCode: item.StatusCode,
|
StatusCode: item.StatusCode,
|
||||||
BytesSent: bytesSent,
|
BytesSent: bytesSent,
|
||||||
RequestLength: requestLength,
|
RequestLength: requestLength,
|
||||||
|
|||||||
@@ -46,16 +46,17 @@ type AccessLogQuery struct {
|
|||||||
|
|
||||||
// AccessLogView is a single access log row.
|
// AccessLogView is a single access log row.
|
||||||
type AccessLogView struct {
|
type AccessLogView struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
NodeID string `json:"node_id"`
|
NodeID string `json:"node_id"`
|
||||||
NodeName string `json:"node_name"`
|
NodeName string `json:"node_name"`
|
||||||
LoggedAt time.Time `json:"logged_at"`
|
LoggedAt time.Time `json:"logged_at"`
|
||||||
RemoteAddr string `json:"remote_addr"`
|
RemoteAddr string `json:"remote_addr"`
|
||||||
Region string `json:"region"`
|
Region string `json:"region"`
|
||||||
Host string `json:"host"`
|
Host string `json:"host"`
|
||||||
Path string `json:"path"`
|
Path string `json:"path"`
|
||||||
UserAgent string `json:"user_agent"`
|
UserAgent string `json:"user_agent"`
|
||||||
StatusCode int `json:"status_code"`
|
CacheStatus string `json:"cache_status"`
|
||||||
|
StatusCode int `json:"status_code"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// AccessLogList is a paginated access log response.
|
// AccessLogList is a paginated access log response.
|
||||||
@@ -476,16 +477,17 @@ func ListAccessLogs(ctx context.Context, input AccessLogQuery) (*AccessLogList,
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
views = append(views, AccessLogView{
|
views = append(views, AccessLogView{
|
||||||
ID: formatAccessLogID(item.ID),
|
ID: formatAccessLogID(item.ID),
|
||||||
NodeID: item.NodeID,
|
NodeID: item.NodeID,
|
||||||
NodeName: nodeNames[item.NodeID],
|
NodeName: nodeNames[item.NodeID],
|
||||||
LoggedAt: item.LoggedAt,
|
LoggedAt: item.LoggedAt,
|
||||||
RemoteAddr: item.RemoteAddr,
|
RemoteAddr: item.RemoteAddr,
|
||||||
Region: item.Region,
|
Region: item.Region,
|
||||||
Host: item.Host,
|
Host: item.Host,
|
||||||
Path: item.Path,
|
Path: item.Path,
|
||||||
UserAgent: item.UserAgent,
|
UserAgent: item.UserAgent,
|
||||||
StatusCode: item.StatusCode,
|
CacheStatus: item.CacheStatus,
|
||||||
|
StatusCode: item.StatusCode,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
return &AccessLogList{
|
return &AccessLogList{
|
||||||
|
|||||||
@@ -0,0 +1,6 @@
|
|||||||
|
-- +goose Up
|
||||||
|
ALTER TABLE of_node_access_logs
|
||||||
|
ADD COLUMN IF NOT EXISTS cache_status String DEFAULT '';
|
||||||
|
|
||||||
|
-- +goose Down
|
||||||
|
ALTER TABLE of_node_access_logs DROP COLUMN IF EXISTS cache_status;
|
||||||
@@ -10,7 +10,7 @@ import (
|
|||||||
|
|
||||||
const (
|
const (
|
||||||
nodeAccessLogTableName = "of_node_access_logs"
|
nodeAccessLogTableName = "of_node_access_logs"
|
||||||
nodeAccessLogInsertColumns = "id, node_id, logged_at, remote_addr, region, host, path, user_agent, status_code, bytes_sent, request_length, request_time_ms, created_at"
|
nodeAccessLogInsertColumns = "id, node_id, logged_at, remote_addr, region, host, path, user_agent, cache_status, status_code, bytes_sent, request_length, request_time_ms, created_at"
|
||||||
)
|
)
|
||||||
|
|
||||||
// NodeAccessLog stores OpenFlare edge node access records in ClickHouse.
|
// NodeAccessLog stores OpenFlare edge node access records in ClickHouse.
|
||||||
@@ -23,6 +23,7 @@ type NodeAccessLog struct {
|
|||||||
Host string `gorm:"column:host"`
|
Host string `gorm:"column:host"`
|
||||||
Path string `gorm:"column:path"`
|
Path string `gorm:"column:path"`
|
||||||
UserAgent string `gorm:"column:user_agent"`
|
UserAgent string `gorm:"column:user_agent"`
|
||||||
|
CacheStatus string `gorm:"column:cache_status"`
|
||||||
StatusCode int32 `gorm:"column:status_code"`
|
StatusCode int32 `gorm:"column:status_code"`
|
||||||
BytesSent uint64 `gorm:"column:bytes_sent"`
|
BytesSent uint64 `gorm:"column:bytes_sent"`
|
||||||
RequestLength uint64 `gorm:"column:request_length"`
|
RequestLength uint64 `gorm:"column:request_length"`
|
||||||
|
|||||||
@@ -285,6 +285,7 @@ func toAnalyticsNodeAccessLog(record *OpenFlareAccessLog) analyticsmodel.NodeAcc
|
|||||||
Host: record.Host,
|
Host: record.Host,
|
||||||
Path: record.Path,
|
Path: record.Path,
|
||||||
UserAgent: record.UserAgent,
|
UserAgent: record.UserAgent,
|
||||||
|
CacheStatus: record.CacheStatus,
|
||||||
StatusCode: openFlareAccessLogStatusCodeToInt32(record.StatusCode),
|
StatusCode: openFlareAccessLogStatusCodeToInt32(record.StatusCode),
|
||||||
BytesSent: bytesSent,
|
BytesSent: bytesSent,
|
||||||
RequestLength: requestLength,
|
RequestLength: requestLength,
|
||||||
@@ -317,6 +318,7 @@ func fromAnalyticsNodeAccessLogs(rows []analyticsmodel.NodeAccessLog) []*OpenFla
|
|||||||
Host: row.Host,
|
Host: row.Host,
|
||||||
Path: row.Path,
|
Path: row.Path,
|
||||||
UserAgent: row.UserAgent,
|
UserAgent: row.UserAgent,
|
||||||
|
CacheStatus: row.CacheStatus,
|
||||||
StatusCode: int(row.StatusCode),
|
StatusCode: int(row.StatusCode),
|
||||||
BytesSent: bytesSent,
|
BytesSent: bytesSent,
|
||||||
RequestLength: requestLength,
|
RequestLength: requestLength,
|
||||||
|
|||||||
@@ -48,6 +48,7 @@ type OpenFlareAccessLog struct {
|
|||||||
Host string `json:"host" gorm:"index;size:255"`
|
Host string `json:"host" gorm:"index;size:255"`
|
||||||
Path string `json:"path" gorm:"size:2048"`
|
Path string `json:"path" gorm:"size:2048"`
|
||||||
UserAgent string `json:"user_agent" gorm:"column:user_agent;size:512"`
|
UserAgent string `json:"user_agent" gorm:"column:user_agent;size:512"`
|
||||||
|
CacheStatus string `json:"cache_status" gorm:"column:cache_status;size:32"`
|
||||||
StatusCode int `json:"status_code" gorm:"index"`
|
StatusCode int `json:"status_code" gorm:"index"`
|
||||||
BytesSent int64 `json:"bytes_sent" gorm:"column:bytes_sent;not null;default:0"`
|
BytesSent int64 `json:"bytes_sent" gorm:"column:bytes_sent;not null;default:0"`
|
||||||
RequestLength int64 `json:"request_length" gorm:"column:request_length;not null;default:0"`
|
RequestLength int64 `json:"request_length" gorm:"column:request_length;not null;default:0"`
|
||||||
|
|||||||
@@ -38,7 +38,7 @@ http {
|
|||||||
fastcgi_temp_path __OPENFLARE_NGINX_CACHE_DIR__/fastcgi_temp;
|
fastcgi_temp_path __OPENFLARE_NGINX_CACHE_DIR__/fastcgi_temp;
|
||||||
uwsgi_temp_path __OPENFLARE_NGINX_CACHE_DIR__/uwsgi_temp;
|
uwsgi_temp_path __OPENFLARE_NGINX_CACHE_DIR__/uwsgi_temp;
|
||||||
scgi_temp_path __OPENFLARE_NGINX_CACHE_DIR__/scgi_temp;
|
scgi_temp_path __OPENFLARE_NGINX_CACHE_DIR__/scgi_temp;
|
||||||
{{OpenRestyConnectionUpgradeMap}}{{OpenRestyDefaultServerBlock}} log_format openflare_json escape=json '{"ts":"$time_iso8601","host":"$host","path":"$request_uri","remote_addr":"$remote_addr","status":$status,"request_time":$request_time,"bytes_sent":$body_bytes_sent,"request_length":$request_length,"user_agent":"$http_user_agent"}';
|
{{OpenRestyConnectionUpgradeMap}}{{OpenRestyDefaultServerBlock}} log_format openflare_json escape=json '{"ts":"$time_iso8601","host":"$host","path":"$request_uri","remote_addr":"$remote_addr","status":$status,"request_time":$request_time,"bytes_sent":$body_bytes_sent,"request_length":$request_length,"user_agent":"$http_user_agent","cache_status":"$upstream_cache_status"}';
|
||||||
access_log {{OpenRestyAccessLogPath}} openflare_json;
|
access_log {{OpenRestyAccessLogPath}} openflare_json;
|
||||||
sendfile on;
|
sendfile on;
|
||||||
tcp_nopush on;
|
tcp_nopush on;
|
||||||
|
|||||||
@@ -36,7 +36,7 @@ func ListNodeAccessLogs(ctx context.Context, filter NodeAccessLogFilter) ([]anal
|
|||||||
clause, args := buildNodeAccessLogFilterClause(filter)
|
clause, args := buildNodeAccessLogFilterClause(filter)
|
||||||
tableName := nodeAccessLogTableName()
|
tableName := nodeAccessLogTableName()
|
||||||
sql := fmt.Sprintf(`
|
sql := fmt.Sprintf(`
|
||||||
SELECT id, node_id, logged_at, remote_addr, region, host, path, user_agent, status_code, bytes_sent, request_length, request_time_ms, created_at
|
SELECT id, node_id, logged_at, remote_addr, region, host, path, user_agent, cache_status, status_code, bytes_sent, request_length, request_time_ms, created_at
|
||||||
FROM %s
|
FROM %s
|
||||||
WHERE %s
|
WHERE %s
|
||||||
ORDER BY %s`, tableName, clause, nodeAccessLogOrderClause(filter.SortBy, filter.SortOrder))
|
ORDER BY %s`, tableName, clause, nodeAccessLogOrderClause(filter.SortBy, filter.SortOrder))
|
||||||
@@ -69,6 +69,7 @@ func scanNodeAccessLogRows(rows driver.Rows) ([]analyticsmodel.NodeAccessLog, er
|
|||||||
&item.Host,
|
&item.Host,
|
||||||
&item.Path,
|
&item.Path,
|
||||||
&item.UserAgent,
|
&item.UserAgent,
|
||||||
|
&item.CacheStatus,
|
||||||
&item.StatusCode,
|
&item.StatusCode,
|
||||||
&item.BytesSent,
|
&item.BytesSent,
|
||||||
&item.RequestLength,
|
&item.RequestLength,
|
||||||
|
|||||||
@@ -49,9 +49,10 @@ func TestBatchInsertNodeAccessLogs_UsesModelBatchSQL(t *testing.T) {
|
|||||||
assert.True(t, mockBatch.sendCalled)
|
assert.True(t, mockBatch.sendCalled)
|
||||||
require.Len(t, mockBatch.rows, 1)
|
require.Len(t, mockBatch.rows, 1)
|
||||||
assert.Equal(t, "node-a", mockBatch.rows[0][1])
|
assert.Equal(t, "node-a", mockBatch.rows[0][1])
|
||||||
require.Len(t, mockBatch.rows[0], 13)
|
require.Len(t, mockBatch.rows[0], 14)
|
||||||
assert.Equal(t, "", mockBatch.rows[0][7]) // user_agent
|
assert.Equal(t, "", mockBatch.rows[0][7]) // user_agent
|
||||||
assert.Equal(t, uint64(2048), mockBatch.rows[0][9]) // bytes_sent
|
assert.Equal(t, "", mockBatch.rows[0][8]) // cache_status
|
||||||
assert.Equal(t, uint64(0), mockBatch.rows[0][10]) // request_length
|
assert.Equal(t, uint64(2048), mockBatch.rows[0][10]) // bytes_sent
|
||||||
assert.Equal(t, uint32(0), mockBatch.rows[0][11]) // request_time_ms
|
assert.Equal(t, uint64(0), mockBatch.rows[0][11]) // request_length
|
||||||
|
assert.Equal(t, uint32(0), mockBatch.rows[0][12]) // request_time_ms
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -47,6 +47,7 @@ func BatchInsertNodeAccessLogs(ctx context.Context, logs []analyticsmodel.NodeAc
|
|||||||
logItem.Host,
|
logItem.Host,
|
||||||
logItem.Path,
|
logItem.Path,
|
||||||
strings.TrimSpace(logItem.UserAgent),
|
strings.TrimSpace(logItem.UserAgent),
|
||||||
|
strings.TrimSpace(logItem.CacheStatus),
|
||||||
logItem.StatusCode,
|
logItem.StatusCode,
|
||||||
logItem.BytesSent,
|
logItem.BytesSent,
|
||||||
logItem.RequestLength,
|
logItem.RequestLength,
|
||||||
|
|||||||
@@ -139,6 +139,7 @@ type NodeAccessLog struct {
|
|||||||
Host string `json:"host"`
|
Host string `json:"host"`
|
||||||
Path string `json:"path"`
|
Path string `json:"path"`
|
||||||
UserAgent string `json:"user_agent,omitempty"`
|
UserAgent string `json:"user_agent,omitempty"`
|
||||||
|
CacheStatus string `json:"cache_status,omitempty"` // $upstream_cache_status
|
||||||
StatusCode int `json:"status_code"`
|
StatusCode int `json:"status_code"`
|
||||||
BytesSent int64 `json:"bytes_sent"` // body bytes = 已提供数据
|
BytesSent int64 `json:"bytes_sent"` // body bytes = 已提供数据
|
||||||
RequestLength int64 `json:"request_length"` // 接收数据
|
RequestLength int64 `json:"request_length"` // 接收数据
|
||||||
|
|||||||
@@ -60,7 +60,7 @@ http {
|
|||||||
fastcgi_temp_path __OPENFLARE_NGINX_CACHE_DIR__/fastcgi_temp;
|
fastcgi_temp_path __OPENFLARE_NGINX_CACHE_DIR__/fastcgi_temp;
|
||||||
uwsgi_temp_path __OPENFLARE_NGINX_CACHE_DIR__/uwsgi_temp;
|
uwsgi_temp_path __OPENFLARE_NGINX_CACHE_DIR__/uwsgi_temp;
|
||||||
scgi_temp_path __OPENFLARE_NGINX_CACHE_DIR__/scgi_temp;
|
scgi_temp_path __OPENFLARE_NGINX_CACHE_DIR__/scgi_temp;
|
||||||
{{OpenRestyConnectionUpgradeMap}}{{OpenRestyDefaultServerBlock}} log_format openflare_json escape=json '{"ts":"$time_iso8601","host":"$host","path":"$request_uri","remote_addr":"$remote_addr","status":$status,"request_time":$request_time,"bytes_sent":$body_bytes_sent,"request_length":$request_length,"user_agent":"$http_user_agent"}';
|
{{OpenRestyConnectionUpgradeMap}}{{OpenRestyDefaultServerBlock}} log_format openflare_json escape=json '{"ts":"$time_iso8601","host":"$host","path":"$request_uri","remote_addr":"$remote_addr","status":$status,"request_time":$request_time,"bytes_sent":$body_bytes_sent,"request_length":$request_length,"user_agent":"$http_user_agent","cache_status":"$upstream_cache_status"}';
|
||||||
access_log {{OpenRestyAccessLogPath}} openflare_json;
|
access_log {{OpenRestyAccessLogPath}} openflare_json;
|
||||||
sendfile on;
|
sendfile on;
|
||||||
tcp_nopush on;
|
tcp_nopush on;
|
||||||
|
|||||||
Reference in New Issue
Block a user