gao
7a985dddd5
## 1. PG-only 重构
- 删 app/core/db/connection.py + schema.py (MySQL 路径)
- 新 app/core/db/{orm,models,pg_bootstrap}.py — SQLAlchemy 2.x ORM 一键建表
- 16 张业务表全在 market_data schema,原生 TIMESTAMPTZ / JSONB / Float / TEXT
- requirements.txt 删 PyMySQL 路径,加 psycopg2
## 2. 同步任务扩展(3 个新 task)
- **tick_trade** (mairui hsrl/zbjy):当天逐笔交易,21:00 发布
- **moneyflow** (mairui hsstock/history/transaction):个股资金流,21:30 发布
- **longhubang** (akshare):龙虎榜聚合层 + 席位层(2 张新表)
- 长虎榜放宽 akshare 政策:仅"无替代源 + 烟测通过"场景允许
- data_eastmoney 私有 API 不需要(akshare 烟测通过)
- 3-timer 设计:
- 15:30 market-sync.service (8 base tasks via runall_once)
- 21:05 market-sync-tick.service (tick_trade)
- 21:35 market-sync-moneyflow.service (moneyflow)
- 22:00 market-sync-lhb.service (longhubang,新加)
- bin/systemd/ 新增 tick / moneyflow / lhb 各 1 对 service+timer
- bin/market_sync_*_run.sh wrapper 脚本(不做法定节假日过滤,fail-open)
## 3. stock_code 统一为带 SH/SZ/BJ 前缀
- 历史 bug:stocks.code 用 SH600519,但 kline/moneyflow/tick_trade/kline_5min
/stock_sector_map/industry 6 张表用纯 6 位 600519,跨表 JOIN 全部 0 行
- 新增 to_hermes() 工具:6位 / 9位(mairui `000001.SZ` 格式)→ 统一 SH000001
- 5 个 task 改写:用 to_hermes(code6) 写入 stock_code
- 一次性迁移 6 张表存量 154M 行(CASE WHEN 探测 + 去重 + 加前缀)
- ORM: stock_sector_map.stock_code / industry.code String(6)→String(10)
## 4. Bug 修复
- **share table stock_code 格式**:之前写 6 位不带前缀,与 stocks 不一致
→ 修 task_share_snapshot + 一次性 UPDATE 63,417 行加前缀
- **share_snapshot warning 状态错填 last_error**:
→ 加 mark_sync_warning() 走专用路径,不写 last_failure_at / last_error
- **schedule config lastRun 不同步**:
→ 加 update_job_status_for_dataset(),SyncTask.run() 完成后自动镜像
→ cli/runall 触发的 task 也能更新 schedule config
## 5. mairui UTF-8 编码修复
- 历史 bug:mairui.py:_fetch 用 latin-1 兜底解码,把所有 UTF-8 中文名
double-encoded 写入 stocks.name(如 `歌华有线` 变成 `æ\xad\x8cå\x8d\x8e...`)
- 加 _decode_response():UTF-8 → GBK → latin-1 兜底
- 一次性修复 stocks.name 5,213 行:
- 4,370 行 (encode('latin-1').decode('utf-8') 反向解码)
- 616 行 (含 fullwidth Ａ,宽松 printable 检查)
- 820 行 (mid-character 截断,重新从 mairui 拉)
## 6. 测试
- tests/test_smoke.py: TASKS 10→11, SYNC_DEFINITIONS 10→11
- tests/test_schema_models.py: 16→18 张表,新增 longhubang_daily/seat
- pytest 11/11 passed
## 验证
- 6 张表 0 残留无前缀行
- stocks JOIN kline_stock / kline_5min / moneyflow / tick_trade / stock_sector_map:88-100% 命中
- 5,213 stocks.name 全部正确 UTF-8 中文
- pytest 11/11 passed
82 lines
3.0 KiB
Bash
Executable File
82 lines
3.0 KiB
Bash
Executable File
#!/bin/bash
|
|
# 串行跑所有 6 个同步任务，后台模式，写到 logs/
|
|
# 用法：./run_all_sync.sh [STEP]
|
|
# STEP 为空 = 跑 1-6
|
|
# STEP=1 只跑 stock_basic，等等
|
|
|
|
set -e
|
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
|
PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
cd "$PROJECT_DIR"
|
|
|
|
mkdir -p logs
|
|
LOG_DIR="$PROJECT_DIR/logs"
|
|
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
|
RUN_LOG="$LOG_DIR/run_all_$TIMESTAMP.log"
|
|
|
|
# 读 token
|
|
if [ -f "$PROJECT_DIR/.env" ]; then
|
|
set -a
|
|
source "$PROJECT_DIR/.env"
|
|
set +a
|
|
fi
|
|
|
|
# 把 .env 里的 TRADING_HOLIDAYS 转成 -- 跳过
|
|
# XUEQIU_TOKEN 在 .env 里
|
|
|
|
source .venv/bin/activate
|
|
export PYTHONPATH="$PROJECT_DIR"
|
|
|
|
LOG_PREFIX="[$TIMESTAMP]"
|
|
|
|
run_step() {
|
|
local step_name="$1"
|
|
local task_id="$2"
|
|
local extra_args="$3"
|
|
local step_log="$LOG_DIR/step_${step_name}_$TIMESTAMP.log"
|
|
|
|
echo "$LOG_PREFIX === Step $step_name: $task_id ===" | tee -a "$RUN_LOG"
|
|
echo "$LOG_PREFIX log: $step_log" | tee -a "$RUN_LOG"
|
|
|
|
# 跑任务（前台模式，方便顺序执行 + 立即看到结果）
|
|
python -m app.entrypoints.cli sync "$task_id" $extra_args 2>&1 | tee "$step_log" | tail -10
|
|
local exit_code=${PIPESTATUS[0]}
|
|
|
|
if [ $exit_code -eq 0 ]; then
|
|
echo "$LOG_PREFIX ✓ Step $step_name OK" | tee -a "$RUN_LOG"
|
|
else
|
|
echo "$LOG_PREFIX ✗ Step $step_name FAIL (exit=$exit_code)" | tee -a "$RUN_LOG"
|
|
return $exit_code
|
|
fi
|
|
}
|
|
|
|
case "${1:-all}" in
|
|
1|stock_basic) run_step "1_stock_basic" "stock_basic" ;;
|
|
2|industry_sector) run_step "2_industry_sector" "industry_sector" ;;
|
|
3|kline_index) run_step "3_kline_index" "kline_index" ;;
|
|
4|kline_daily) run_step "4_kline_daily" "kline_daily" "--workers 20" ;;
|
|
5|share_snapshot) run_step "5_share_snapshot" "share_snapshot" "--workers 10" ;;
|
|
6|market_regime) run_step "6_market_regime" "market_regime" ;;
|
|
7|kline_5min) run_step "7_kline_5min" "kline_5min" "--workers 10" ;;
|
|
8|moneyflow) run_step "8_moneyflow" "moneyflow" "--workers 10" ;;
|
|
9|sector_features) run_step "9_sector_features" "sector_features" ;;
|
|
10|tick_trade) run_step "10_tick_trade" "tick_trade" "--workers 5 --force" ;;
|
|
all)
|
|
run_step "1_stock_basic" "stock_basic" || exit 1
|
|
run_step "2_industry_sector" "industry_sector" || exit 1
|
|
run_step "3_kline_index" "kline_index" || exit 1
|
|
run_step "4_kline_daily" "kline_daily" "--workers 20" || exit 1
|
|
run_step "5_share_snapshot" "share_snapshot" "--workers 10" || exit 1
|
|
run_step "6_market_regime" "market_regime" || exit 1
|
|
run_step "7_kline_5min" "kline_5min" "--workers 10" || exit 1
|
|
run_step "8_moneyflow" "moneyflow" "--workers 10" || exit 1
|
|
run_step "9_sector_features" "sector_features" || exit 1
|
|
run_step "10_tick_trade" "tick_trade" "--workers 5 --force" || exit 1
|
|
echo "$LOG_PREFIX ✓ 全部 10 步完成" | tee -a "$RUN_LOG"
|
|
;;
|
|
*)
|
|
echo "usage: $0 [1|2|3|4|5|6|7|8|9|10|all]"
|
|
exit 1
|
|
;;
|
|
esac
|