refactor: PG-only 迁移 + 龙虎榜/tick/moneyflow 同步 + stock_code 统一 + mairui 编码修复

## 1. PG-only 重构 - 删 app/core/db/connection.py + schema.py (MySQL 路径) - 新 app/core/db/{orm,models,pg_bootstrap}.py — SQLAlchemy 2.x ORM 一键建表 - 16 张业务表全在 market_data schema,原生 TIMESTAMPTZ / JSONB / Float / TEXT - requirements.txt 删 PyMySQL 路径,加 psycopg2 ## 2. 同步任务扩展(3 个新 task) - **tick_trade** (mairui hsrl/zbjy):当天逐笔交易,21:00 发布 - **moneyflow** (mairui hsstock/history/transaction):个股资金流,21:30 发布 - **longhubang** (akshare):龙虎榜聚合层 + 席位层(2 张新表) - 长虎榜放宽 akshare 政策:仅"无替代源 + 烟测通过"场景允许 - data_eastmoney 私有 API 不需要(akshare 烟测通过) - 3-timer 设计: - 15:30 market-sync.service (8 base tasks via runall_once) - 21:05 market-sync-tick.service (tick_trade) - 21:35 market-sync-moneyflow.service (moneyflow) - 22:00 market-sync-lhb.service (longhubang,新加) - bin/systemd/ 新增 tick / moneyflow / lhb 各 1 对 service+timer - bin/market_sync_*_run.sh wrapper 脚本(不做法定节假日过滤,fail-open) ## 3. stock_code 统一为带 SH/SZ/BJ 前缀 - 历史 bug:stocks.code 用 SH600519,但 kline/moneyflow/tick_trade/kline_5min /stock_sector_map/industry 6 张表用纯 6 位 600519,跨表 JOIN 全部 0 行 - 新增 to_hermes() 工具:6位 / 9位(mairui `000001.SZ` 格式)→ 统一 SH000001 - 5 个 task 改写:用 to_hermes(code6) 写入 stock_code - 一次性迁移 6 张表存量 154M 行(CASE WHEN 探测 + 去重 + 加前缀) - ORM: stock_sector_map.stock_code / industry.code String(6)→String(10) ## 4. Bug 修复 - **share table stock_code 格式**:之前写 6 位不带前缀,与 stocks 不一致 → 修 task_share_snapshot + 一次性 UPDATE 63,417 行加前缀 - **share_snapshot warning 状态错填 last_error**: → 加 mark_sync_warning() 走专用路径,不写 last_failure_at / last_error - **schedule config lastRun 不同步**: → 加 update_job_status_for_dataset(),SyncTask.run() 完成后自动镜像 → cli/runall 触发的 task 也能更新 schedule config ## 5. mairui UTF-8 编码修复 - 历史 bug:mairui.py:_fetch 用 latin-1 兜底解码,把所有 UTF-8 中文名 double-encoded 写入 stocks.name(如 `歌华有线` 变成 `æ\xad\x8cå\x8d\x8e...`) - 加 _decode_response():UTF-8 → GBK → latin-1 兜底 - 一次性修复 stocks.name 5,213 行: - 4,370 行 (encode('latin-1').decode('utf-8') 反向解码) - 616 行 (含 fullwidth Ａ,宽松 printable 检查) - 820 行 (mid-character 截断,重新从 mairui 拉) ## 6. 测试 - tests/test_smoke.py: TASKS 10→11, SYNC_DEFINITIONS 10→11 - tests/test_schema_models.py: 16→18 张表,新增 longhubang_daily/seat - pytest 11/11 passed ## 验证 - 6 张表 0 残留无前缀行 - stocks JOIN kline_stock / kline_5min / moneyflow / tick_trade / stock_sector_map:88-100% 命中 - 5,213 stocks.name 全部正确 UTF-8 中文 - pytest 11/11 passed
2026-07-01 22:00:58 +08:00
parent 05635b76b9
commit 7a985dddd5
45 changed files with 3037 additions and 958 deletions
@@ -19,6 +19,7 @@ from app.core.sync.registry import (
    mark_sync_progress,
    mark_sync_running,
    mark_sync_success,
+    mark_sync_warning,
 )
 from app.core.utils.logging import get_logger

@@ -79,15 +80,20 @@ class SyncTask:
                    error=result.get("error", message),
                    status="blocked",
                )
-            else:  # warning / partial
+            elif status == "warning":
+                # 部分失败:走专用 warning 路径,不写 last_failure_at / last_error
+                mark_sync_warning(self.dataset_id, message=message)
+            else:  # 其他未识别状态默认走 failed
                mark_sync_failed(
                    self.dataset_id,
                    message=message,
                    error=result.get("error", message),
-                    status="warning",
+                    status="failed",
                )
            result["elapsed_sec"] = elapsed
            logger.info(f"[{self.dataset_id}] 完成 {status}: {message} ({elapsed}s)")
+            # 同步 schedule config 的 lastRun（fix Bug 2）
+            self._maybe_mirror_schedule_status(trigger_source, status, message)
            return result
        except Exception as e:
            elapsed = round(time.time() - t0, 1)
@@ -139,3 +145,28 @@ class SyncTask:
            progress_total=total,
            current_step=current_step,
        )
+
+    def _maybe_mirror_schedule_status(
+        self, trigger_source: str, status: str, message: str
+    ) -> None:
+        """把非 schedule 触发的运行结果同步到 schedule config 表（fix Bug 2）。
+
+        背景：bin/runall_once.py（15:30 systemd 触发）和 python -m app.entrypoints.cli sync
+        （手动触发）走的都是 task.run(trigger_source="cli" or "manual")，不会经过
+        app.core.scheduler 的 update_job_status。结果：schedule config 表的 lastRun /
+        lastStatus 一直停在 scheduler 真正跑过的那天（实际是 6/15 之后就没动过）。
+        UI 看到的就是"schedule 视图"和"实际跑过"脱节。
+
+        解法：每次 task.run() 完成后，lazy import scheduler，找到这个 dataset_id 对应的
+        schedule config 并镜像 lastRun/lastStatus/lastMessage。
+
+        scheduler 触发的（trigger_source="schedule"）会自己 update_job_status，跳过即可。
+        """
+        if trigger_source == "schedule":
+            return
+        try:
+            from app.core.scheduler.scheduler import update_job_status_for_dataset
+            update_job_status_for_dataset(self.dataset_id, status, message)
+        except Exception as e:
+            # 镜像失败不影响主流程；只记 debug log
+            logger.debug(f"[_maybe_mirror_schedule_status] {self.dataset_id} 镜像失败: {e}")
@@ -26,8 +26,8 @@ SYNC_DEFINITIONS: list[dict[str, Any]] = [
        "dataset_id": "stock_basic",
        "name": "股票基础信息（含最新股本）",
        "description": "全市场 A 股代码 / 名称 / 交易所 / 上市状态 + 股本快照（雪球）",
-        "storage_uri": "MySQL market_data_sync_db.stocks",
-        "storage_layer": "mysql",
+        "storage_uri": "PG market_data.stocks",
+        "storage_layer": "pg",
        "management_role": "基础字典",
        "source": "Baostock query_stock_basic + 雪球 quote_detail",
        "sync_script": "app.tasks.task_stocks_basic:run",
@@ -38,8 +38,8 @@ SYNC_DEFINITIONS: list[dict[str, Any]] = [
        "dataset_id": "kline_daily",
        "name": "原始日K线数据",
        "description": "全市场 A 股日频 OHLCV，多源交叉校验（雪球/新浪/Baostock）",
-        "storage_uri": "MySQL market_data_sync_db.kline_stock",
-        "storage_layer": "mysql",
+        "storage_uri": "PG market_data.kline_stock",
+        "storage_layer": "pg",
        "management_role": "原始源",
        "source": "雪球主源 + 新浪/Baostock 复核",
        "sync_script": "app.tasks.task_kline_daily:run",
@@ -50,8 +50,8 @@ SYNC_DEFINITIONS: list[dict[str, Any]] = [
        "dataset_id": "kline_index",
        "name": "六大指数日线",
        "description": "上证 / 深证 / 创业板 / 沪深300 / 中证500 / 中证1000",
-        "storage_uri": "MySQL market_data_sync_db.kline_index + indices",
-        "storage_layer": "mysql",
+        "storage_uri": "PG market_data.kline_index + indices",
+        "storage_layer": "pg",
        "management_role": "原始源",
        "source": "新浪指数日K",
        "sync_script": "app.tasks.task_kline_index:run",
@@ -62,22 +62,34 @@ SYNC_DEFINITIONS: list[dict[str, Any]] = [
        "dataset_id": "kline_5min",
        "name": "5 分钟 K 线",
        "description": "全市场 A 股 5 分钟 K 线（mairui 源）",
-        "storage_uri": "MySQL market_data_sync_db.kline_5min",
-        "storage_layer": "mysql",
+        "storage_uri": "PG market_data.kline_5min",
+        "storage_layer": "pg",
        "management_role": "原始源",
        "source": "mairui stockMin",
        "sync_script": "app.tasks.task_kline_5min:run",
        "dependency_ids": ["stock_basic"],
        "sort_order": 40,
    },
+    {
+        "dataset_id": "tick_trade",
+        "name": "当天逐笔交易",
+        "description": "全市场 A 股当天逐笔成交（mairui hsrl/zbjy 源；每日 21:00 发布，仅当天数据）",
+        "storage_uri": "PG market_data.tick_trade",
+        "storage_layer": "pg",
+        "management_role": "原始源",
+        "source": "mairui hsrl/zbjy",
+        "sync_script": "app.tasks.task_tick_trade:run",
+        "dependency_ids": ["stock_basic"],
+        "sort_order": 45,
+    },
    {
        "dataset_id": "moneyflow",
        "name": "资金流",
-        "description": "个股资金流（主力/大/中/小单净额），mairui 源",
-        "storage_uri": "MySQL market_data_sync_db.moneyflow",
-        "storage_layer": "mysql",
+        "description": "个股资金流（主力/大/中/小单净额），mairui 源（每日 21:30 发布，由 market-sync-moneyflow.timer 21:35 触发）",
+        "storage_uri": "PG market_data.moneyflow",
+        "storage_layer": "pg",
        "management_role": "原始源",
-        "source": "mairui hsstock/history/transaction",
+        "source": "mairui hsstock/history/transaction (21:30 publish)",
        "sync_script": "app.tasks.task_moneyflow:run",
        "dependency_ids": ["stock_basic"],
        "sort_order": 50,
@@ -86,8 +98,8 @@ SYNC_DEFINITIONS: list[dict[str, Any]] = [
        "dataset_id": "industry_sector",
        "name": "股票-行业映射",
        "description": "股票-行业映射 + 行业字典（Baostock）",
-        "storage_uri": "MySQL market_data_sync_db.industry + sectors + stock_sector_map",
-        "storage_layer": "mysql",
+        "storage_uri": "PG market_data.industry + sectors + stock_sector_map",
+        "storage_layer": "pg",
        "management_role": "基础字典",
        "source": "Baostock query_stock_industry",
        "sync_script": "app.tasks.task_industry_sector:run",
@@ -98,8 +110,8 @@ SYNC_DEFINITIONS: list[dict[str, Any]] = [
        "dataset_id": "sector_features",
        "name": "行业聚合特征",
        "description": "由 kline_stock + industry 衍生：行业日收益 / 行业指数 close / EMA10-20-200 / score。纯本地计算，无外部 API。",
-        "storage_uri": "MySQL market_data_sync_db.sector_indices + sector_features_daily",
-        "storage_layer": "mysql",
+        "storage_uri": "PG market_data.sector_indices + sector_features_daily",
+        "storage_layer": "pg",
        "management_role": "衍生源",
        "source": "本地计算（kline_stock + industry）",
        "sync_script": "app.tasks.task_sector_features:run",
@@ -110,8 +122,8 @@ SYNC_DEFINITIONS: list[dict[str, Any]] = [
        "dataset_id": "share_snapshot",
        "name": "股本快照",
        "description": "全市场 A 股最新股本（雪球 quote_detail，单位亿股）",
-        "storage_uri": "MySQL market_data_sync_db.stocks.total_share/float_share + share 表",
-        "storage_layer": "mysql",
+        "storage_uri": "PG market_data.stocks.total_share/float_share + share 表",
+        "storage_layer": "pg",
        "management_role": "基础字典",
        "source": "雪球 quote_detail",
        "sync_script": "app.tasks.task_share_snapshot:run",
@@ -122,14 +134,26 @@ SYNC_DEFINITIONS: list[dict[str, Any]] = [
        "dataset_id": "market_regime",
        "name": "市场情绪 (Market Regime)",
        "description": "基于本地日K线聚合的 advance_ratio / 恐慌标记（衍生源）",
-        "storage_uri": "MySQL market_data_sync_db.market_regime_daily",
-        "storage_layer": "mysql",
+        "storage_uri": "PG market_data.market_regime_daily",
+        "storage_layer": "pg",
        "management_role": "衍生源",
        "source": "本地日K线聚合（无外部接口）",
        "sync_script": "app.tasks.task_market_regime:run",
        "dependency_ids": ["kline_daily"],
        "sort_order": 80,
    },
+    {
+        "dataset_id": "longhubang",
+        "name": "龙虎榜(聚合 + 席位)",
+        "description": "akshare 源（东方财富封装）;每日 22:00 timer 触发,新增聚合层 + 席位层两张表",
+        "storage_uri": "PG market_data.longhubang_daily + longhubang_seat",
+        "storage_layer": "pg",
+        "management_role": "原始源",
+        "source": "akshare stock_lhb_detail_em + stock_lhb_stock_detail_em (东方财富)",
+        "sync_script": "app.tasks.task_longhubang:run",
+        "dependency_ids": ["stock_basic"],
+        "sort_order": 85,
+    },
 ]


@@ -236,6 +260,26 @@ def mark_sync_failed(
    )


+def mark_sync_warning(dataset_id: str, *, message: str = "") -> None:
+    """部分失败的标记:成功 + 失败混合 → status=warning。
+
+    与 mark_sync_failed 的区别：warning 不写 last_failure_at / last_error，
+    也不触发 needs_resync=1（warning 状态下数据可能部分可用，不是硬错误）。
+    历史 bug（2026-07-01 修复）：SyncTask.run() 把 warning 走 mark_sync_failed 路径
+    导致 UI 误显示"失败"（last_failure_at 有值、last_error 非空）。
+    """
+    now = _now_ts()
+    db_ops.update_dataset_registry_state(
+        dataset_id,
+        status="warning",
+        finished_at=now,
+        last_success_at=now,  # 部分成功也算成功一刻，记到 last_success
+        message=message or "部分失败",
+        current_step="",
+        updated_at=now,
+    )
+
+
 def mark_sync_blocked(dataset_id: str, *, message: str) -> None:
    mark_sync_failed(dataset_id, message=message, error=message, status="blocked")