fix: market-sync 2h timeout 截断 + 加 daily-check 巡检

work #01 (2026-07-03):昨天 15:30 runall 中 kline_5min 全量 69min 把 2h
窗口吃满,share_snapshot / market_regime 被 systemd SIGTERM 截断;
今天又叠加 worker.py 未长驻,导致 10 个 task missed。

变更:
- bin/systemd/market-sync.service: TimeoutStartSec 7200 → 14400 (4h)
- bin/daily_sync_check.py: 新增每日巡检,读 PG dataset_registry,
  对比 SCHEDULE 判定 ok/missed/failed/stuck/never_run,写 JSON 报告,
  失败时 POST HMAC webhook
- bin/systemd/market-sync-daily-check.{service,timer}: 23:00 daily 触发
  daily_check(替代 session-only 的 /loop 方案)
- config/daily_check_secrets.env.template: webhook 配置模板
- docs/works/2026-07-03-01-market-sync-timeout.md: 完整 work 记录
  (根因 + 中期/长期改进建议)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
gao
2026-07-03 10:48:20 +08:00
parent 7a985dddd5
commit 295177027f
6 changed files with 542 additions and 2 deletions
@@ -0,0 +1,31 @@
[Unit]
Description=Market sync daily check — reads dataset_registry, writes report, alerts on failures
Documentation=file:///home/gao/Development/quant_home/market_sync/bin/daily_sync_check.py
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
WorkingDirectory=/home/gao/Development/quant_home/market_sync
User=gao
Group=gao
# 巡检:读 PG dataset_registry,对比今日预期,写 logs/daily_check_*.json
# 失败时 POST webhook 到 FAILURE_WEBHOOK_URLHMAC-SHA256 签名)
ExecStart=/home/gao/Development/quant_home/market_sync/.venv/bin/python /home/gao/Development/quant_home/market_sync/bin/daily_sync_check.py
# 巡检应 < 30s 完成;超时 = PG 不通或代码 bug,不让卡死
TimeoutStartSec=120
# 不要 Restart=oneshot 失败就让 OnFailure= 发通知,别自动重跑)
# 失败时 exit code = 1,可被 OnFailure= 钩住发通知
# 日志走 journaldjournalctl -u market-sync-daily-check -f
StandardOutput=journal
StandardError=journal
SyslogIdentifier=market-sync-daily-check
Environment=PYTHONUNBUFFERED=1
[Install]
WantedBy=multi-user.target
+15
View File
@@ -0,0 +1,15 @@
[Unit]
Description=Schedule market sync daily check — Daily 23:00 Asia/Shanghai
# market-sync-daily-check.service 是这个 timer 的执行单元
# 23:00 触发:周一~五最后一个任务是 longhubang @ 22:00,留 1h buffer
# 周六 23:00 给 stock_node @ 11:30 留 11h+ buffer
[Timer]
# 每天 23:00 触发;Persistent=true 确保关机/错过时下次开机补跑
OnCalendar=*-*-* 23:00:00 Asia/Shanghai
Persistent=true
Unit=market-sync-daily-check.service
# 不要 AccuracySec(默认 1min 漂移够用)
[Install]
WantedBy=timers.target
+4 -2
View File
@@ -13,8 +13,10 @@ Group=gao
# 入口脚本(runall_once.py + structured_watch.sh 二合一)
ExecStart=/home/gao/Development/quant_home/market_sync/bin/market_sync_run.sh
# 硬上限 2h(实际 ~80min留 buffer 给 retry / 慢任务)
TimeoutStartSec=7200
# 硬上限 4h(实际 ~80min + kline_5min 增量 70min, 总 ~2.5h, 留 buffer 给 retry / 慢任务)
# 历史 bug2026-07-02):2h 超时截断,share_snapshot / market_regime 没跑成
# 详见 docs/works/2026-07-03-01-market-sync-timeout.md
TimeoutStartSec=14400
# 不要 Restart=oneshot 失败就让 OnFailure= 发通知,别自动重跑——
# 重跑会撞外部 API 限流窗口,浪费 1.5h)