fix: market-sync 2h timeout 截断 + 加 daily-check 巡检
work #01 (2026-07-03):昨天 15:30 runall 中 kline_5min 全量 69min 把 2h 窗口吃满,share_snapshot / market_regime 被 systemd SIGTERM 截断; 今天又叠加 worker.py 未长驻,导致 10 个 task missed。 变更: - bin/systemd/market-sync.service: TimeoutStartSec 7200 → 14400 (4h) - bin/daily_sync_check.py: 新增每日巡检,读 PG dataset_registry, 对比 SCHEDULE 判定 ok/missed/failed/stuck/never_run,写 JSON 报告, 失败时 POST HMAC webhook - bin/systemd/market-sync-daily-check.{service,timer}: 23:00 daily 触发 daily_check(替代 session-only 的 /loop 方案) - config/daily_check_secrets.env.template: webhook 配置模板 - docs/works/2026-07-03-01-market-sync-timeout.md: 完整 work 记录 (根因 + 中期/长期改进建议) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,31 @@
|
||||
[Unit]
|
||||
Description=Market sync daily check — reads dataset_registry, writes report, alerts on failures
|
||||
Documentation=file:///home/gao/Development/quant_home/market_sync/bin/daily_sync_check.py
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=/home/gao/Development/quant_home/market_sync
|
||||
User=gao
|
||||
Group=gao
|
||||
|
||||
# 巡检:读 PG dataset_registry,对比今日预期,写 logs/daily_check_*.json
|
||||
# 失败时 POST webhook 到 FAILURE_WEBHOOK_URL(HMAC-SHA256 签名)
|
||||
ExecStart=/home/gao/Development/quant_home/market_sync/.venv/bin/python /home/gao/Development/quant_home/market_sync/bin/daily_sync_check.py
|
||||
|
||||
# 巡检应 < 30s 完成;超时 = PG 不通或代码 bug,不让卡死
|
||||
TimeoutStartSec=120
|
||||
|
||||
# 不要 Restart=(oneshot 失败就让 OnFailure= 发通知,别自动重跑)
|
||||
# 失败时 exit code = 1,可被 OnFailure= 钩住发通知
|
||||
|
||||
# 日志走 journald(journalctl -u market-sync-daily-check -f)
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=market-sync-daily-check
|
||||
|
||||
Environment=PYTHONUNBUFFERED=1
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,15 @@
|
||||
[Unit]
|
||||
Description=Schedule market sync daily check — Daily 23:00 Asia/Shanghai
|
||||
# market-sync-daily-check.service 是这个 timer 的执行单元
|
||||
# 23:00 触发:周一~五最后一个任务是 longhubang @ 22:00,留 1h buffer;
|
||||
# 周六 23:00 给 stock_node @ 11:30 留 11h+ buffer
|
||||
|
||||
[Timer]
|
||||
# 每天 23:00 触发;Persistent=true 确保关机/错过时下次开机补跑
|
||||
OnCalendar=*-*-* 23:00:00 Asia/Shanghai
|
||||
Persistent=true
|
||||
Unit=market-sync-daily-check.service
|
||||
# 不要 AccuracySec(默认 1min 漂移够用)
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -13,8 +13,10 @@ Group=gao
|
||||
# 入口脚本(runall_once.py + structured_watch.sh 二合一)
|
||||
ExecStart=/home/gao/Development/quant_home/market_sync/bin/market_sync_run.sh
|
||||
|
||||
# 硬上限 2h(实际 ~80min,留 buffer 给 retry / 慢任务)
|
||||
TimeoutStartSec=7200
|
||||
# 硬上限 4h(实际 ~80min + kline_5min 增量 70min, 总 ~2.5h, 留 buffer 给 retry / 慢任务)
|
||||
# 历史 bug(2026-07-02):2h 超时截断,share_snapshot / market_regime 没跑成
|
||||
# 详见 docs/works/2026-07-03-01-market-sync-timeout.md
|
||||
TimeoutStartSec=14400
|
||||
|
||||
# 不要 Restart=(oneshot 失败就让 OnFailure= 发通知,别自动重跑——
|
||||
# 重跑会撞外部 API 限流窗口,浪费 1.5h)
|
||||
|
||||
Reference in New Issue
Block a user