fix: market-sync 2h timeout 截断 + 加 daily-check 巡检

work #01 (2026-07-03):昨天 15:30 runall 中 kline_5min 全量 69min 把 2h
窗口吃满,share_snapshot / market_regime 被 systemd SIGTERM 截断;
今天又叠加 worker.py 未长驻,导致 10 个 task missed。

变更:
- bin/systemd/market-sync.service: TimeoutStartSec 7200 → 14400 (4h)
- bin/daily_sync_check.py: 新增每日巡检,读 PG dataset_registry,
  对比 SCHEDULE 判定 ok/missed/failed/stuck/never_run,写 JSON 报告,
  失败时 POST HMAC webhook
- bin/systemd/market-sync-daily-check.{service,timer}: 23:00 daily 触发
  daily_check(替代 session-only 的 /loop 方案)
- config/daily_check_secrets.env.template: webhook 配置模板
- docs/works/2026-07-03-01-market-sync-timeout.md: 完整 work 记录
  (根因 + 中期/长期改进建议)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
gao
2026-07-03 10:48:20 +08:00
parent 7a985dddd5
commit 295177027f
6 changed files with 542 additions and 2 deletions
@@ -0,0 +1,31 @@
[Unit]
Description=Market sync daily check — reads dataset_registry, writes report, alerts on failures
Documentation=file:///home/gao/Development/quant_home/market_sync/bin/daily_sync_check.py
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
WorkingDirectory=/home/gao/Development/quant_home/market_sync
User=gao
Group=gao
# 巡检:读 PG dataset_registry,对比今日预期,写 logs/daily_check_*.json
# 失败时 POST webhook 到 FAILURE_WEBHOOK_URLHMAC-SHA256 签名)
ExecStart=/home/gao/Development/quant_home/market_sync/.venv/bin/python /home/gao/Development/quant_home/market_sync/bin/daily_sync_check.py
# 巡检应 < 30s 完成;超时 = PG 不通或代码 bug,不让卡死
TimeoutStartSec=120
# 不要 Restart=oneshot 失败就让 OnFailure= 发通知,别自动重跑)
# 失败时 exit code = 1,可被 OnFailure= 钩住发通知
# 日志走 journaldjournalctl -u market-sync-daily-check -f
StandardOutput=journal
StandardError=journal
SyslogIdentifier=market-sync-daily-check
Environment=PYTHONUNBUFFERED=1
[Install]
WantedBy=multi-user.target