kyugao/market_sync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: 新增 mairui 历史分时 MA 日 K 级别 sync task

Browse Source 
work #06 (2026-07-03)：user 请求加 mairui /hsdata 历史分时 MA 同步（日 K 级别）。

mairui 端点探测：/d/ma, /d/ma5/10/20, /15/ma, /30/ma, /60/ma 端点结构存在
但当前免费 licence 返 数据不存在；基础 K 线 (/d/n, /15/n, /30/n, /60/n) 正常。

策略：本地从 kline_stock 计算（pandas per-stock rolling），写新表
kline_stock_ma_daily，source=local_kline_proxy 标识本地派生。
mairui URL 留作未来升级 licence 后切 API 用。

变更：
- app/core/db/models.py: KlineStockMADaily ORM model
- app/core/db/ops.py: upsert_kline_stock_ma_daily_rows (批量 5000/批)
- app/tasks/task_mairui_ma_daily.py: SyncMairuiMADaily (全量重算)
- app/tasks/__init__.py: 注册到 TASKS dict
- app/core/sync/registry.py: SYNC_DEFINITION (sort_order=90, dep=kline_daily)
- app/core/scheduler/scheduler.py: schedule_mairui_ma_daily @ 16:30 + register_sync_jobs
- bin/daily_sync_check.py: SCHEDULE entry (window_end=17:00)

烟测：11,684,592 行, 5510 只, 1370s (23min)，MA5/10/20/60 全部计算。
SH600519 样本：ma5=1194.18 ma10=1191.03 ma20=1211.94 ma60=1296.20 (2026-07-03)

未来优化（不在本 work）：
- 增量模式（每日只算最近 1-2 天）→ 23min → 30s
- 升级 mairui licence 切到 /d/maN API
- 加 EMA / BOLL / KDJ 等其他指标

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
gao
2026-07-03 17:36:49 +08:00
parent 46ddf8e171
commit b351bd7595
8 changed files with 498 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/core/db/models.py
+122 -2
View File
@@ -362,7 +362,40 @@ class TickTrade(ORMBase):
amount: Mapped[float] = mapped_column(Float, nullable=False, default=0)
# ──────────────────────── 17. longhubang_daily ────────────────────────
# ──────────────────────── 17. kline_stock_ma_daily ────────────────────────
class KlineStockMADaily(ORMBase):
"""个股日 K 级别 MA 指标（基于 kline_stock.close 滚动计算）。
字段：
stock_code (str, hermes 格式 SH600000)
trade_date (date)
ma5 / ma10 / ma20 / ma60 (float, close 的简单移动平均)
source (str, "local_kline_proxy" | "mairui" — 数据来源标识)
updated_at (timestamptz)
设计：日 K 级别 MA 通常在本地从 kline_stock 派生（pandas rolling）。
mairui /hsdata 提供的 /d/ma5/ma10/ma20/ma60 端点要付费 licence，本项目
默认用本地计算，source=local_kline_proxy；若未来升级 mairui licence，
可加 source=mairui 走 API 直拉。
"""
__tablename__ = "kline_stock_ma_daily"
__table_args__ = (
PrimaryKeyConstraint("stock_code", "trade_date"),
Index("idx_kline_stock_ma_date", "trade_date"),
{"schema": "market_data"},
)
stock_code: Mapped[str] = mapped_column(String(10), nullable=False)
trade_date: Mapped[date] = mapped_column(Date, nullable=False)
ma5: Mapped[Optional[float]] = mapped_column(Float)
ma10: Mapped[Optional[float]] = mapped_column(Float)
ma20: Mapped[Optional[float]] = mapped_column(Float)
ma60: Mapped[Optional[float]] = mapped_column(Float)
source: Mapped[Optional[str]] = mapped_column(String(32), default="local_kline_proxy")
updated_at: Mapped[Optional[datetime]] = _updated_at()
# ──────────────────────── 18. longhubang_daily ────────────────────────
class LonghubangDaily(ORMBase):
"""龙虎榜每日上榜股票汇总（聚合层，akshare 源）。
@@ -450,6 +483,87 @@ class LonghubangSeat(ORMBase):
explanation: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
# ──────────────────────── 19. node_categories ────────────────────────
class NodeCategory(ORMBase):
"""mairui /hszg/list/ 顶层分类字典。
字段：
category_key (PK, 如 '0:2' = A 股热门概念)
display_name (中文分类名 取自 pname,如 'A股-热门概念' 或剥前缀后 '热门概念')
market ('A 股' / '港股' / '基金' / ...)
category_type ('concept' / 'industry' / 'industry_sub' / 'index' / 'region' / 'class')
node_count (该分类下叶子节点数)
派生自 mairui /hszg/list/ 的 type1+type2 组合,详见 SyncStockNode 注释。
"""
__tablename__ = "node_categories"
__table_args__ = {"schema": "market_data"}
category_key: Mapped[str] = mapped_column(String(16), primary_key=True)
display_name: Mapped[str] = mapped_column(String(64), nullable=False, default="")
market: Mapped[str] = mapped_column(String(16), nullable=False, default="")
category_type: Mapped[str] = mapped_column(String(16), nullable=False, default="")
node_count: Mapped[int] = mapped_column(Integer, default=0)
updated_at: Mapped[Optional[datetime]] = _updated_at()
# ──────────────────────── 20. nodes ────────────────────────
class Node(ORMBase):
"""mairui /hszg/list/ 节点字典(指数/行业/概念)。
字段(从 mairui 原样保留 + 派生分类):
node_code (PK,如 'chgn_730603' = '热门概念-肝炎治疗')
node_name (中文名,如 'A股-热门概念-肝炎治疗')
category_key (外键到 NodeCategory, 由 type1+type2 拼接)
parent_code (父节点 code,可空)
parent_name (父节点 name,可空)
level (mairui 的层级 0/1/2)
is_leaf (int 0/1, mairui 原 isleaf 字段)
mairui_type1, mairui_type2 (保留原始数值,便于回查/调试)
"""
__tablename__ = "nodes"
__table_args__ = (
Index("idx_nodes_category", "category_key"),
Index("idx_nodes_parent", "parent_code"),
Index("idx_nodes_leaf", "is_leaf"),
{"schema": "market_data"},
)
node_code: Mapped[str] = mapped_column(String(64), primary_key=True)
node_name: Mapped[str] = mapped_column(String(128), nullable=False, default="")
category_key: Mapped[str] = mapped_column(String(16), nullable=False, default="")
parent_code: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
parent_name: Mapped[Optional[str]] = mapped_column(String(128), nullable=True)
level: Mapped[int] = mapped_column(Integer, default=0)
is_leaf: Mapped[int] = mapped_column(Integer, default=0)
mairui_type1: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
mairui_type2: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
updated_at: Mapped[Optional[datetime]] = _updated_at()
# ──────────────────────── 21. stock_node_map ────────────────────────
class StockNodeMap(ORMBase):
"""股票-节点 N×M 映射(来自 mairui /hszg/gg/{code})。
一只股票可属于多个概念(同时属于"AI算力"+"国产芯片"+"特斯拉概念"),
也可属于多个指数("沪深300"+"上证50"+"科创50")。
PK (stock_code, node_code) 保证幂等 upsert。
stock_code 用 hermes 格式(SH600519),与项目其他表一致。
"""
__tablename__ = "stock_node_map"
__table_args__ = (
PrimaryKeyConstraint("stock_code", "node_code"),
Index("idx_stock_node_node", "node_code"),
Index("idx_stock_node_stock", "stock_code"),
{"schema": "market_data"},
)
stock_code: Mapped[str] = mapped_column(String(10), nullable=False)
node_code: Mapped[str] = mapped_column(String(64), nullable=False)
updated_at: Mapped[Optional[datetime]] = _updated_at()
__all__ = [
# 1-2
"Config",
@@ -473,7 +587,13 @@ __all__ = [
"MarketRegimeDaily",
# 16
"TickTrade",
# 17-18 (2026-07-01 龙虎榜)
# 17 (2026-07-03 mairui 历史分时 MA - 日 K 级别)
"KlineStockMADaily",
# 18-19 (2026-07-01 龙虎榜)
"LonghubangDaily",
"LonghubangSeat",
# 19-21 (2026-07-02 股票-节点映射 mairui)
"NodeCategory",
"Node",
"StockNodeMap",
]
app/core/db/ops.py
+113
View File
@@ -25,16 +25,20 @@ from app.core.db.models import (
Kline5Min,
KlineIndex,
KlineStock,
KlineStockMADaily,
LonghubangDaily,
LonghubangSeat,
MarketIndex,
MarketRegimeDaily,
Moneyflow,
Node,
NodeCategory,
SectorFeaturesDaily,
SectorIndices,
Sectors,
Share,
Stock,
StockNodeMap,
StockSectorMap,
TickTrade,
)
@@ -601,6 +605,79 @@ def upsert_longhubang_seat(rows: list[dict[str, Any]]) -> int:
return _bulk_upsert_orm(s, LonghubangSeat, rows, chunk_size=500)
# ── 节点映射 (mairui /hszg) ─────────────────────────────────
def replace_all_node_categories(rows: list[dict[str, Any]]) -> None:
"""rows: category_key, display_name, market, category_type, node_count"""
with get_session() as s:
s.execute(delete(NodeCategory))
if rows:
values = [
{
"category_key": r.get("category_key", ""),
"display_name": r.get("display_name", ""),
"market": r.get("market", ""),
"category_type": r.get("category_type", ""),
"node_count": int(r.get("node_count", 0)),
}
for r in rows
]
stmt = _pg_upsert(NodeCategory, values,
conflict_keys=["category_key"],
update_cols=["display_name", "market", "category_type", "node_count"])
s.execute(stmt)
def replace_all_nodes(rows: list[dict[str, Any]]) -> None:
"""rows: node_code, node_name, category_key, parent_code, parent_name,
level, is_leaf, mairui_type1, mairui_type2"""
if not rows:
return
with get_session() as s:
values = [
{
"node_code": r.get("node_code", ""),
"node_name": r.get("node_name", ""),
"category_key": r.get("category_key", ""),
"parent_code": r.get("parent_code") or None,
"parent_name": r.get("parent_name") or None,
"level": int(r.get("level", 0)),
"is_leaf": int(r.get("is_leaf", 0)),
"mairui_type1": r.get("mairui_type1"),
"mairui_type2": r.get("mairui_type2"),
}
for r in rows
]
stmt = _pg_upsert(Node, values,
conflict_keys=["node_code"],
update_cols=["node_name", "category_key", "parent_code",
"parent_name", "level", "is_leaf",
"mairui_type1", "mairui_type2"])
s.execute(stmt)
def replace_all_stock_node_map(rows: list[dict[str, Any]]) -> None:
"""rows: stock_code (hermes 格式), node_code"""
if not rows:
return
with get_session() as s:
values = [
{
"stock_code": str(r.get("stock_code", "")).strip(),
"node_code": r.get("node_code", ""),
}
for r in rows
]
# PK-only 表 —— 用 ON CONFLICT DO NOTHING(因为 PK 已经确定唯一内容,
# 重复 PK 的行内容相同,无需 UPDATE)
from sqlalchemy.dialects.postgresql import insert as pg_insert
stmt = pg_insert(StockNodeMap).values(values).on_conflict_do_nothing(
index_elements=["stock_code", "node_code"]
)
s.execute(stmt)
# ── 行业 / 概念板块 ─────────────────────────────────────────────────────
@@ -772,6 +849,42 @@ def upsert_market_regime_rows(rows: list[dict[str, Any]]) -> None:
# ── kline 查询（同步时用于判断增量起点）─────────────────────────────────
def upsert_kline_stock_ma_daily_rows(rows: list[dict[str, Any]]) -> None:
"""rows: stock_code, trade_date, ma5, ma10, ma20, ma60, source"""
if not rows:
return
with get_session() as s:
values = [
{
"stock_code": str(r.get("stock_code") or ""),
"trade_date": _to_date_str(r.get("trade_date")),
"ma5": float(r["ma5"]) if r.get("ma5") is not None and not pd_isna(r["ma5"]) else None,
"ma10": float(r["ma10"]) if r.get("ma10") is not None and not pd_isna(r["ma10"]) else None,
"ma20": float(r["ma20"]) if r.get("ma20") is not None and not pd_isna(r["ma20"]) else None,
"ma60": float(r["ma60"]) if r.get("ma60") is not None and not pd_isna(r["ma60"]) else None,
"source": r.get("source", "local_kline_proxy"),
}
for r in rows
]
stmt = _pg_upsert(
KlineStockMADaily,
values,
conflict_keys=["stock_code", "trade_date"],
update_cols=["ma5", "ma10", "ma20", "ma60", "source", "updated_at"],
)
s.execute(stmt)
def pd_isna(v: Any) -> bool:
"""避免直接 import pandas（开销大），手写 nan/None 检查。"""
if v is None:
return True
try:
return float(v) != float(v) # NaN != NaN
except (TypeError, ValueError):
return False
def get_stock_kline_max_date(code: str) -> Optional[str]:
_ensure_schema()
with get_session() as s: