This commit is contained in:
2025-12-27 11:44:50 +08:00
commit ccd43fac1f
1193 changed files with 384161 additions and 0 deletions
@@ -0,0 +1,6 @@
ollama是新安装的,然后映射到ollama_home目录
ollama的service,添加环境变量:
Environment="OLLAMA_HOST=0.0.0.0" "OLLAMA_MODELS=/data/ollama_home"
openwebui是新安装的docker, 新建的容器,映射到openwebui的工作目录
docker create -p 3000:8080 -v /data/openwebui_home/data:/app/backend/data -e OLLAMA_BASE_URL=http://192.168.5.135:11434 -e HF_ENDPOINT=https://hf-mirror.com --name open-webui --restart always registry.cn-guangzhou.aliyuncs.com/gdcyy-open-library/openwebui
@@ -0,0 +1,164 @@
{
"edges":[
{
"fromNode":"e63c7ecc7d262aee",
"fromSide":"right",
"id":"7369442a7b5d66e0",
"styleAttributes":{},
"toNode":"887188af1d31da85",
"toSide":"left"
},
{
"fromNode":"51d753880be94a2d",
"fromSide":"right",
"id":"64a9fadce0614b9a",
"styleAttributes":{},
"toNode":"6ff39169b738088e",
"toSide":"left"
},
{
"fromNode":"51d753880be94a2d",
"fromSide":"right",
"id":"d3943efd8ccc1e2a",
"styleAttributes":{},
"toNode":"265cbe2e0a54ea5b",
"toSide":"left"
},
{
"fromNode":"51d753880be94a2d",
"fromSide":"right",
"id":"3d3aba2aa5c92a82",
"styleAttributes":{},
"toNode":"59a3bdfe32a41fa5",
"toSide":"left"
},
{
"fromNode":"51d753880be94a2d",
"fromSide":"right",
"id":"2ef4c8f1c60ed6d7",
"styleAttributes":{},
"toNode":"e63c7ecc7d262aee",
"toSide":"left"
},
{
"fromNode":"59a3bdfe32a41fa5",
"fromSide":"right",
"id":"4a09aa38feaec8fe",
"styleAttributes":{},
"toNode":"3615d3d907a16d43",
"toSide":"left"
},
{
"fromNode":"265cbe2e0a54ea5b",
"fromSide":"right",
"id":"aecab99420d9eec9",
"styleAttributes":{},
"toNode":"cbbc3892bb2bcadf",
"toSide":"left"
},
{
"fromNode":"cbbc3892bb2bcadf",
"fromSide":"right",
"id":"6a4472d4e0769b6a",
"styleAttributes":{},
"toNode":"f3468ff4204ff01b",
"toSide":"left"
}
],
"metadata":{
"frontmatter":{},
"version":"1.0-1.0"
},
"nodes":[
{
"height":60,
"id":"59a3bdfe32a41fa5",
"styleAttributes":{},
"text":"多用户",
"type":"text",
"width":140,
"x":-100,
"y":-260
},
{
"height":60,
"id":"e63c7ecc7d262aee",
"styleAttributes":{},
"text":"外接模型",
"type":"text",
"width":140,
"x":-100,
"y":-380
},
{
"height":60,
"id":"887188af1d31da85",
"styleAttributes":{},
"text":"连接Ollama",
"type":"text",
"width":145,
"x":120,
"y":-380
},
{
"height":60,
"id":"51d753880be94a2d",
"styleAttributes":{},
"text":"支持功能",
"type":"text",
"width":160,
"x":-400,
"y":-200
},
{
"height":60,
"id":"265cbe2e0a54ea5b",
"styleAttributes":{},
"text":"扩展",
"type":"text",
"width":140,
"x":-100,
"y":-140
},
{
"height":60,
"id":"6ff39169b738088e",
"styleAttributes":{},
"text":"OpenAI兼容接口\nAPIKey 1/用户",
"type":"text",
"width":182,
"x":-100,
"y":-20
},
{
"height":60,
"id":"3615d3d907a16d43",
"styleAttributes":{},
"text":"用户自主注册 后台创建",
"type":"text",
"width":260,
"x":120,
"y":-260
},
{
"height":60,
"id":"cbbc3892bb2bcadf",
"styleAttributes":{},
"text":"知识库 工具",
"type":"text",
"width":168,
"x":120,
"y":-140
},
{
"height":60,
"id":"f3468ff4204ff01b",
"styleAttributes":{},
"text":"开放|自有",
"type":"text",
"width":141,
"x":340,
"y":-140
}
]
}
@@ -0,0 +1,5 @@
``` bash
# 指定模型堂驻内存
$ curl http://127.0.0.1:11434/api/generate -d '{"model":"qwen3-coder:latest", "keep_alive": -1}'
$ curl http://127.0.0.1:11434/api/generate -d '{"model":"qwen2.5:14b", "keep_alive": -1}'
```
@@ -0,0 +1 @@
【本地部署最强OCR大模型olmOCR!支持结构化精准提取复杂PDF文件内容!完美识别中英文文档、模糊扫描件与复杂表格!本地部署与实际测试全过程!医疗法律行业必备-哔哩哔哩】 https://b23.tv/n4v6JDO
@@ -0,0 +1,8 @@
```yml
chatmodel:
  provider: "deepseek"  # 服务提供商(例如:azure/openai
  baseurl: "https://api.deepseek.com"  # API基础地址
  apikey: "sk-ffef52570c5f4834b11580d38552731f"  # API密钥(建议使用环境变量替代)
  model: "deepseek-reasoner"  # 使用的聊天模型名称
```
+11
View File
@@ -0,0 +1,11 @@
```bash
$ docker create --name open-webui -p 3000:8080 \
-v /data/openwebui_home/data:/app/backend/data \
-e OLLAMA_BASE_URL=http://192.168.3.231:11434 \
-e HF_ENDPOINT=https://hf-mirror.com \
--restart always ghcr.goops.top/open-webui/open-webui:main
```
页面刷新慢:因为开着openai的接口,需要在配置里禁掉。
![[OpenWebUI.canvas|OpenWebUI]]
File diff suppressed because it is too large Load Diff
+10
View File
@@ -0,0 +1,10 @@
---
opp_publish_id: "941389673032228864"
opp_publish_status: published
opp_publish_title: 向量数据库
---
Sqlite
* sqlite_vec
Postgres [[pg_vector && pgvectorscale使用说明]]
* pg-vector
* pg-vectorscale
@@ -0,0 +1,269 @@
{
"edges":[
{"fromNode":"258f55793c08ec9b","fromSide":"bottom","id":"391218fa7e78f94b","toNode":"8bb1d2063a8eee5c","toSide":"top"},
{"fromNode":"7046fe626e6a00c6","fromSide":"bottom","id":"8a7f1613db523f59","toNode":"8d8d274b7e0659bc","toSide":"top"},
{"fromEnd":"arrow","fromNode":"8bb1d2063a8eee5c","fromSide":"right","id":"74a5db9fc5d23a44","label":"匹配","toNode":"8d8d274b7e0659bc","toSide":"left"},
{"fromNode":"6ff6e60699c83ba0","fromSide":"bottom","id":"22ee5d1522146d24","toNode":"51895cca1d63c642","toSide":"top"},
{"fromNode":"04debfe2a686fbd9","fromSide":"bottom","id":"dbd020ffad92f19f","toNode":"ce9b2bf9eaf6f352","toSide":"top"},
{"fromNode":"8d705284d54c4bf8","fromSide":"right","id":"29bbf34481b2cf84","toNode":"9a1235e1dcfd91d5","toSide":"left"},
{"fromNode":"48e5dbe21e9dd690","fromSide":"right","id":"ed564faa07d5a3e0","toNode":"4691e870fa3a13a9","toSide":"left"},
{"fromNode":"e22849b89185cd61","fromSide":"right","id":"9760a31471da1233","toNode":"fadf39c3d47836a8","toSide":"left"},
{"fromNode":"1472033b4b40e3c9","fromSide":"bottom","id":"e0ffdf72232e96ab","toNode":"78a782c53235378b","toSide":"top"}
],
"nodes":[
{
"height":860,
"id":"d92ec6c2200d9b77",
"label":"特征向量的存储-向量数据库",
"styleAttributes":{},
"type":"group",
"width":1120,
"x":-340,
"y":1600
},
{
"height":400,
"id":"60797eca86c1e565",
"label":"场景举例",
"styleAttributes":{},
"type":"group",
"width":1120,
"x":-340,
"y":520
},
{
"height":300,
"id":"324a73f0cc613d91",
"label":"个性化推荐",
"styleAttributes":{},
"type":"group",
"width":1120,
"x":-340,
"y":140
},
{
"height":160,
"id":"1472033b4b40e3c9",
"label":"特征向量的提取-Embedding模型",
"styleAttributes":{},
"type":"group",
"width":1120,
"x":-340,
"y":980
},
{
"height":60,
"id":"258f55793c08ec9b",
"styleAttributes":{},
"text":"用户偏好",
"type":"text",
"width":140,
"x":-220,
"y":200
},
{
"height":60,
"id":"7046fe626e6a00c6",
"styleAttributes":{},
"text":"目标内容",
"type":"text",
"width":140,
"x":500,
"y":200
},
{
"height":621,
"id":"e22849b89185cd61",
"styleAttributes":{},
"text":"##### 多维向量\n* 维度:可以认为是大模型计算出来的基于语义上不同维度的值。\n* 例:广州这夏天,真是让我这东北人没法活。\n\t* 维度一:情感[-1, 1]:强烈得分 0.9 \n\t* 维度二:温度[-1, 1]:热得分 0.88\n\t* 维度三:色温[-1, 1]:暖色 得分 0.91\n\t* 维度四:科技[-1, 1]:不相关得分 0.001\n\t* 维度五:喜好[-1, 1]:厌恶得分 -0.82\n\n##### 向量的距离\n* 在二维空间里,向量的距离是平面上一个线段的距离\n\t* 这个距离越近,两个点的x,y的坐标就越近\n* 在三维空间里,向量的距离是立体空间上两个点的连线\n\t* 这个距离越近,两个点的x,y,z的坐标就越近\n* 在四维、五维,只要是空间,对应维度的向量,都有一个距离。距离越近,相近度越大",
"type":"text",
"width":640,
"x":-340,
"y":-540
},
{
"height":347,
"id":"fadf39c3d47836a8",
"styleAttributes":{},
"text":"##### 感受\n前例:\n* 广州这夏天,真是让我这东北人没法活。\n\n对比例子:\n1. 机房服务器的风扇是真的吵。\n2. 机房没了空调,热得没法呆人了。\n\n相对来说,2描述的场景和前例更加相似。",
"type":"text",
"width":400,
"x":370,
"y":-403
},
{
"height":60,
"id":"8bb1d2063a8eee5c",
"styleAttributes":{},
"text":"多维特征向量",
"type":"text",
"width":180,
"x":-240,
"y":320
},
{
"height":60,
"id":"8d8d274b7e0659bc",
"styleAttributes":{},
"text":"多维特征向量",
"type":"text",
"width":180,
"x":480,
"y":320
},
{
"height":192,
"id":"6ff6e60699c83ba0",
"styleAttributes":{},
"text":"#### 公益租赁平台 - AI中介\n个性化推荐房源:\n1. 用户当前的位置,推荐周边\n2. 用户搜索记录,小区,户型,价位,配套情况,交通偏好",
"type":"text",
"width":503,
"x":-310,
"y":540
},
{
"height":192,
"id":"04debfe2a686fbd9",
"styleAttributes":{},
"text":"#### 公益租赁平台 - 健康咨询\n个性化膳食推荐:\n1. 用户体质评测,匹配食材\n2. 用户搜索记录,食材偏好,料理方式偏好",
"type":"text",
"width":503,
"x":249,
"y":540
},
{
"height":100,
"id":"51895cca1d63c642",
"styleAttributes":{},
"text":"房屋属性:位置,周边,设施,价格,户型 ....",
"type":"text",
"width":503,
"x":-310,
"y":800
},
{
"height":126,
"id":"8d705284d54c4bf8",
"styleAttributes":{},
"text":"* BAAI/bge-m3\n* BAAI/bge-large-zh-v1.5\n* BAAI/bge-large-en-v1.5",
"type":"text",
"width":268,
"x":-312,
"y":1000
},
{
"height":329,
"id":"78a782c53235378b",
"styleAttributes":{},
"text":"\n| **度量方法** | **核心特性** | | **典型应用场景** | **案例参考** |\n| --------- | ------------------- | --- | ----------------------------------------------------- | --------- |\n| **L1距离** | 计算各维度绝对差之和,路径类似网格行走 | | 1. 特征选择(稀疏性)<br>2. 路径规划(网格状移动)<br>3. 鲁棒性要求高的数据分析 | 城市导航、异常检测 |\n| **L2距离** | 计算多维空间中的直线距离 | | 1. 图像特征匹配<br>2. 地理位置计算<br>3. 需要精确几何距离的场景 | 人脸识别、三维建模 |\n| **余弦相似度** | 计算向量夹角的余弦值,忽略向量长度 | | 1. 文本语义相似度<br>2. 用户兴趣推荐(忽略强度差异)<br>3. 高维稀疏数据(如NLP词向量) | 文档检索、广告推荐 |\n",
"type":"text",
"width":1120,
"x":-340,
"y":1200
},
{
"height":138,
"id":"bed4573cdc8d4dc7",
"styleAttributes":{},
"text":"传统数据库扩展\n* Sqlite-vec\n* postgres-vector",
"type":"text",
"width":210,
"x":-312,
"y":1630
},
{
"height":138,
"id":"44e6262f9df1ae6f",
"styleAttributes":{},
"text":"传统数据库扩展\n* Redis\n* MongoDB",
"type":"text",
"width":210,
"x":-38,
"y":1630
},
{
"height":138,
"id":"13bab0d5df168e4a",
"styleAttributes":{},
"text":"服务\n* Pinecone\n* Milvus",
"type":"text",
"width":216,
"x":520,
"y":1630
},
{
"height":138,
"id":"3389fe8228ace3e4",
"styleAttributes":{},
"text":"本地文件\n* chromaDB\n* lanceDB",
"type":"text",
"width":220,
"x":236,
"y":1630
},
{
"height":623,
"id":"019068aae556c2e1",
"styleAttributes":{},
"text":"```sql\ncreate virtual table vec_examples using vec0(\n sample_embedding float[5]\n);\ninsert into vec_examples(rowid, sample_embedding)\n values\n (1, '[-0.200, 0.250, 0.341, -0.211, 0.645'),\n (2, '[0.443, -0.501, 0.355, -0.771, 0.707'),\n (3, '[0.716, -0.927, 0.134, 0.052, -0.669'),\n (4, '[-0.710, 0.330, 0.656, 0.041, -0.990');\n-- KNN style query\nselect rowid,distance from vec_examples\nwhere sample_embedding match '[0.890, 0.544, 0.825, 0.961, 0.358]'\norder by distance\nlimit 2;\n\n/*\n┌───────┬──────────────────┐\n│ rowid │ distance │\n├───────┼──────────────────┤\n│ 2 │ 2.38687372207642 │\n│ 1 │ 2.38978505134583 │\n└───────┴──────────────────┘\n*/\n```\n",
"type":"text",
"width":1072,
"x":-312,
"y":1800
},
{
"height":100,
"id":"ce9b2bf9eaf6f352",
"styleAttributes":{},
"text":"* 膳食:适应人群,适应体质,所用食材,料理方式\n* 方案:面向人群,调养目标 ...",
"type":"text",
"width":503,
"x":249,
"y":800
},
{
"height":60,
"id":"9a1235e1dcfd91d5",
"styleAttributes":{},
"text":"1024",
"type":"text",
"width":100,
"x":-20,
"y":1033
},
{
"height":126,
"id":"3d30185fe259b33e",
"styleAttributes":{},
"text":"* 文本-Embedding\n* 图-Embedding\n* 声音-Embedding\n",
"type":"text",
"width":222,
"x":540,
"y":1000
},
{
"height":126,
"id":"48e5dbe21e9dd690",
"styleAttributes":{},
"text":"* text-embedding-ada-002\n* text-embedding-3-small\n* text-embedding-3-large",
"type":"text",
"width":283,
"x":108,
"y":1000
},
{
"height":60,
"id":"4691e870fa3a13a9",
"styleAttributes":{},
"text":"1536",
"type":"text",
"width":97,
"x":423,
"y":1033
}
],
"metadata":{
"version":"1.0-1.0",
"frontmatter":{}
}
}
+1
View File
@@ -0,0 +1 @@
单机单卡快速训练框架[unsloth](https://www.unsloth.ai/)
@@ -0,0 +1,8 @@
`sudo certbot certonly --text --agree-tos --email 917929050@qq.com --standalone -d app.scopdev.jtgdyyb.cn
使用DNS解析认证
`sudo certbot-auto certonly --manual --preferred-challenge dns -d www.xxx.com
证书存放地址
/etc/letsencrypt/live
+1
View File
@@ -0,0 +1 @@
https://github.com/Bronya0/Kafka-King
+549
View File
@@ -0,0 +1,549 @@
{
"edges":[
{
"id":"5311072484566988",
"styleAttributes":{"pathfindingMethod":"square"},
"toFloating":false,
"fromFloating":false,
"fromNode":"a088e85e91293434",
"fromSide":"bottom",
"toNode":"24c1312b4fc764d7",
"toSide":"top"
},
{
"id":"2f6ce6e78eebc535",
"styleAttributes":{"pathfindingMethod":"square"},
"toFloating":false,
"fromFloating":false,
"fromNode":"7e034ab4587534f8",
"fromSide":"bottom",
"toNode":"24c1312b4fc764d7",
"toSide":"top"
},
{
"id":"ca86ea0bc78e362d",
"styleAttributes":{},
"toFloating":false,
"fromNode":"bb32b85e68a3c5a6",
"fromSide":"bottom",
"toNode":"24c1312b4fc764d7",
"toSide":"top"
},
{
"id":"dcb49779f1396c86",
"styleAttributes":{"pathfindingMethod":"square"},
"toFloating":false,
"fromNode":"f9483b9c68b4ca95",
"fromSide":"bottom",
"toNode":"24c1312b4fc764d7",
"toSide":"top"
},
{
"id":"fa48c88118e041cd",
"styleAttributes":{"pathfindingMethod":"square"},
"toFloating":false,
"fromNode":"d541ad6e1481d9b3",
"fromSide":"bottom",
"toNode":"24c1312b4fc764d7",
"toSide":"top"
},
{
"id":"d203c1ec3edcb103",
"styleAttributes":{"pathfindingMethod":"square"},
"toFloating":false,
"fromFloating":false,
"fromNode":"4f62d8a7660f5155",
"fromSide":"bottom",
"toNode":"6f1d4aa0e394c3fc",
"toSide":"top"
},
{
"id":"50735bd49c8ebf40",
"styleAttributes":{"pathfindingMethod":"square"},
"toFloating":false,
"fromFloating":false,
"fromNode":"7d2f4e279def45da",
"fromSide":"bottom",
"toNode":"6f1d4aa0e394c3fc",
"toSide":"top"
},
{
"id":"1af01a2210ca6691",
"styleAttributes":{"pathfindingMethod":"square"},
"toFloating":false,
"fromNode":"6286d89761a9c395",
"fromSide":"bottom",
"toNode":"6f1d4aa0e394c3fc",
"toSide":"top"
},
{
"id":"021fff7e70488500",
"styleAttributes":{},
"toFloating":false,
"fromFloating":false,
"fromNode":"2f7f8d771e76ac3d",
"fromSide":"bottom",
"toNode":"28adebc518af8f41",
"toSide":"top",
"label":"ws://url/uri+token"
},
{
"id":"157a8b8b3d433ec7",
"styleAttributes":{},
"toFloating":false,
"fromNode":"566be66ace698d58",
"fromSide":"bottom",
"toNode":"78ea03c7259f97a0",
"toSide":"top"
},
{
"id":"f8c655b3c2eb9131",
"styleAttributes":{},
"toFloating":false,
"fromNode":"8fe0b6419e0b01b1",
"fromSide":"bottom",
"toNode":"e637376395b8424b",
"toSide":"top"
},
{
"id":"534d1591d591ab16",
"styleAttributes":{},
"toFloating":false,
"fromNode":"a7cd89568ff94edd",
"fromSide":"bottom",
"toNode":"aaecfc8d2cd8987a",
"toSide":"top"
},
{
"id":"88420432ef7d7cba",
"styleAttributes":{},
"toFloating":false,
"fromFloating":false,
"fromNode":"a7cd89568ff94edd",
"fromSide":"right",
"toNode":"2f7f8d771e76ac3d",
"toSide":"left",
"label":"token"
},
{
"id":"d47b295180f5154e",
"styleAttributes":{},
"toFloating":false,
"fromNode":"a5311e2ceb1af6ff",
"fromSide":"bottom",
"toNode":"5eef7d3ff512845b",
"toSide":"top"
},
{
"id":"11bfa021e47bee6a",
"styleAttributes":{"pathfindingMethod":"square"},
"toFloating":false,
"fromNode":"1d968add45109fc4",
"fromSide":"bottom",
"toNode":"778d251815154128",
"toSide":"left"
},
{
"id":"9ec97e188133650d",
"styleAttributes":{"pathfindingMethod":"square"},
"toFloating":false,
"fromFloating":false,
"fromNode":"778d251815154128",
"fromSide":"top",
"toNode":"1d968add45109fc4",
"toSide":"right"
},
{
"id":"8497a83162464f1f",
"styleAttributes":{"pathfindingMethod":"square"},
"toFloating":false,
"fromFloating":false,
"fromNode":"9bc530c3107092b1",
"fromSide":"bottom",
"toNode":"1d968add45109fc4",
"toSide":"left"
},
{
"id":"c9fbcb78c7de38b9",
"styleAttributes":{"pathfindingMethod":"square"},
"toFloating":false,
"fromNode":"9bc530c3107092b1",
"fromSide":"bottom",
"toNode":"e3b18695fa20f63e",
"toSide":"left"
},
{
"id":"4d8a4d511cdaee2b",
"styleAttributes":{},
"toFloating":false,
"fromNode":"5989527a0beba624",
"fromSide":"right",
"toNode":"9bc530c3107092b1",
"toSide":"left",
"fromEnd":"arrow",
"label":"Websocket长连接"
},
{
"id":"eec3fe18d6abba52",
"styleAttributes":{},
"toFloating":false,
"fromNode":"5989527a0beba624",
"fromSide":"bottom",
"toNode":"e59f51d6cc055628",
"toSide":"top"
},
{
"id":"5510d08807d7b4e5",
"styleAttributes":{"pathfindingMethod":"square"},
"toFloating":false,
"fromNode":"e59f51d6cc055628",
"fromSide":"bottom",
"toNode":"0160569168edd2f4",
"toSide":"top"
},
{
"id":"d18ce4da305bb2e7",
"styleAttributes":{"pathfindingMethod":"square"},
"toFloating":false,
"fromFloating":false,
"fromNode":"e59f51d6cc055628",
"fromSide":"bottom",
"toNode":"1d2ec3b7b7a40c8f",
"toSide":"top"
}
],
"metadata":{
"frontmatter":{},
"version":"1.0-1.0"
},
"nodes":[
{"id":"a1a0f1f7f1faf671","type":"group","x":1280,"y":100,"width":670,"height":640,"label":"MsgGateway服务"},
{"id":"e6cf0f9ce7031bb7","type":"group","x":-500,"y":-400,"width":760,"height":380,"label":"服务端"},
{"id":"32f5104e2e1236c4","type":"group","x":-500,"y":-820,"width":760,"height":320,"label":"SDK"},
{"id":"6f1d4aa0e394c3fc","type":"group","x":-480,"y":-220,"width":720,"height":180,"label":"下层服务"},
{"id":"a5311e2ceb1af6ff","type":"group","x":400,"y":-340,"width":560,"height":100,"label":"Auth服务"},
{"id":"96073594a2eb838c","type":"group","x":400,"y":-680,"width":560,"height":100,"label":"客户端SDK"},
{"id":"aaecfc8d2cd8987a","type":"group","x":400,"y":-500,"width":280,"height":100,"label":"APIService"},
{"id":"28adebc518af8f41","type":"group","x":700,"y":-500,"width":260,"height":100,"label":"MsgGateway"},
{
"id":"a088e85e91293434",
"styleAttributes":{},
"text":"Android",
"type":"text",
"x":-480,
"y":-800,
"width":140,
"height":60
},
{
"id":"2f7f8d771e76ac3d",
"type":"text",
"text":"WebSocket客户端",
"styleAttributes":{"textAlign":"center"},
"x":740,
"y":-660,
"width":180,
"height":60
},
{
"id":"7e034ab4587534f8",
"styleAttributes":{},
"text":"iOS",
"type":"text",
"x":-328,
"y":-800,
"width":114,
"height":60
},
{
"id":"f9483b9c68b4ca95",
"styleAttributes":{},
"text":"Browser",
"type":"text",
"x":-26,
"y":-800,
"width":126,
"height":60
},
{
"id":"bb32b85e68a3c5a6",
"styleAttributes":{},
"text":"MiniApp",
"type":"text",
"x":-184,
"y":-800,
"width":128,
"height":60
},
{
"id":"d541ad6e1481d9b3",
"styleAttributes":{},
"text":"Other",
"type":"text",
"x":126,
"y":-800,
"width":114,
"height":60
},
{
"id":"a7cd89568ff94edd",
"type":"text",
"text":"Http连接",
"styleAttributes":{"textAlign":"center"},
"x":440,
"y":-660,
"width":200,
"height":60
},
{
"id":"7e54ebcf5ca41a90",
"styleAttributes":{"textAlign":"center"},
"text":"长连接(WebSocket",
"type":"text",
"x":-230,
"y":-580,
"width":220,
"height":60
},
{
"id":"24c1312b4fc764d7",
"styleAttributes":{"textAlign":"center"},
"text":"Core SDK With Golang",
"type":"text",
"x":-480,
"y":-680,
"width":720,
"height":60
},
{
"id":"643fefa30582c725",
"styleAttributes":{"textAlign":"center"},
"text":"短连接(Http",
"type":"text",
"x":-480,
"y":-580,
"width":220,
"height":60
},
{
"id":"275361f8bd185882",
"styleAttributes":{"textAlign":"center"},
"text":"用户信息",
"type":"text",
"x":-440,
"y":-200,
"width":140,
"height":60
},
{
"id":"7d2f4e279def45da",
"styleAttributes":{"textAlign":"center"},
"text":"API服务\n微服务远程调用",
"type":"text",
"x":-480,
"y":-360,
"width":220,
"height":60
},
{
"id":"81daaa946bed5ffc",
"styleAttributes":{"textAlign":"center"},
"text":"朋友/群组",
"type":"text",
"x":-440,
"y":-120,
"width":140,
"height":60
},
{
"id":"3e6f641292cc4dd2",
"styleAttributes":{"textAlign":"center"},
"text":"会话信息",
"type":"text",
"x":-190,
"y":-120,
"width":140,
"height":60
},
{
"id":"4f62d8a7660f5155",
"styleAttributes":{"textAlign":"center"},
"text":"MsgGateway\nWebSocket长连接服务",
"type":"text",
"x":-230,
"y":-360,
"width":220,
"height":60
},
{
"id":"6286d89761a9c395",
"styleAttributes":{"textAlign":"center"},
"text":"其他服务",
"type":"text",
"x":20,
"y":-360,
"width":220,
"height":60
},
{
"id":"8fe0b6419e0b01b1",
"type":"text",
"text":"API服务",
"styleAttributes":{"textAlign":"center"},
"x":440,
"y":-480,
"width":200,
"height":60
},
{
"id":"566be66ace698d58",
"type":"text",
"text":"发起连接",
"styleAttributes":{"textAlign":"center"},
"x":740,
"y":-480,
"width":180,
"height":60
},
{
"id":"afadf110737841fe",
"styleAttributes":{"textAlign":"center"},
"text":"用户认证",
"type":"text",
"x":-190,
"y":-200,
"width":140,
"height":60
},
{
"id":"e6a84d96ead95498",
"styleAttributes":{"textAlign":"center"},
"text":"信息",
"type":"text",
"x":60,
"y":-200,
"width":140,
"height":60
},
{
"id":"782074c3ea044381",
"styleAttributes":{"textAlign":"center"},
"text":"三方",
"type":"text",
"x":60,
"y":-120,
"width":140,
"height":60
},
{
"id":"78ea03c7259f97a0",
"type":"text",
"text":"ParseToken\n解析Token",
"styleAttributes":{"textAlign":"center"},
"x":740,
"y":-320,
"width":180,
"height":60
},
{
"id":"e637376395b8424b",
"type":"text",
"text":"创建一个Token",
"styleAttributes":{"textAlign":"center"},
"x":440,
"y":-320,
"width":200,
"height":60
},
{
"id":"5eef7d3ff512845b",
"type":"text",
"text":"AuthDatabase\n基于Redis的缓存服务",
"styleAttributes":{"textAlign":"center"},
"x":440,
"y":-200,
"width":480,
"height":60
},
{
"id":"5989527a0beba624",
"type":"text",
"text":"客户端SDK",
"styleAttributes":{},
"x":300,
"y":120,
"width":260,
"height":60
},
{
"id":"e59f51d6cc055628",
"type":"text",
"text":"客户端类型",
"styleAttributes":{},
"x":361,
"y":210,
"width":139,
"height":60
},
{
"id":"0160569168edd2f4",
"type":"text",
"text":"GoBaseSDK-GOB",
"styleAttributes":{"textAlign":"center"},
"x":261,
"y":300,
"width":169,
"height":60
},
{
"id":"1d2ec3b7b7a40c8f",
"type":"text",
"text":"Others-JSON",
"styleAttributes":{"textAlign":"center"},
"x":431,
"y":300,
"width":169,
"height":60
},
{
"id":"9bc530c3107092b1",
"type":"text",
"text":"WebSocket Client",
"styleAttributes":{},
"x":1310,
"y":120,
"width":260,
"height":60
},
{
"id":"e3b18695fa20f63e",
"type":"text",
"text":"ticker drived heartbeat",
"styleAttributes":{},
"x":1500,
"y":240,
"width":240,
"height":60
},
{
"id":"1d968add45109fc4",
"type":"text",
"text":"readMessage",
"styleAttributes":{},
"x":1500,
"y":400,
"width":200,
"height":60
},
{
"id":"778d251815154128",
"type":"text",
"text":"处理消息",
"styleAttributes":{"textAlign":"center"},
"x":1720,
"y":520,
"width":200,
"height":60
}
]
}
@@ -0,0 +1,54 @@
---
opp_publish_id: "944175079304876032"
opp_publish_status: published
opp_publish_title: OpenIM源码分析之一---项目部署与结构
---
# 开始
部署一套体验环境,参考[官方指引](https://docs.openim.io/zh-Hans/guides/gettingStarted/dockerCompose),Docker方式当然最便捷的。不过我们的目的当然不仅是,体验一下发个消息,对于项目结构本身依然还是0认知,通过源码部署,依照[官方文档-源码部署](https://docs.openim.io/zh-Hans/guides/gettingStarted/imSourceCodeDeployment),完成部署也不是什么难事。那么让我们从这里开始,一起走进OpenIM项目。
# 部署结构
## 依赖服务安装
OpenIM已经准备好的依赖组件的Docker服务配置,位于项目[open-im-server](https://github.com/openimsdk/open-im-server)下。
`$ git clone https://github.com/openimsdk/open-im-server && cd open-im-server
| 组件类别 | OpenIM依赖以下组件 | 运维组件 | 测试组件 |
| ---- | ------------------------------------------ | ------------------------------------- | ------------------------ |
| 组件列表 | mongodb<br>redis<br>kafka<br>MinIO<br>etcd | prometheus<br>alertmanager<br>grafana | admin-front<br>web-front |
| 可选 | 必选 | 可选 | 可选 |
| 启动命令 | `$ docker compose up -d` | `$ docker compose --profile m up -d` | 跟随依赖组件启动 |
### 文档补充说明:
* [Minior的访问配置](https://docs.openim.io/zh-Hans/guides/gettingStarted/imSourceCodeDeployment#23-%E8%AE%BE%E7%BD%AE%E5%A4%96%E7%BD%91-ip-%E6%88%96%E5%9F%9F%E5%90%8D%E5%8F%82%E8%80%83-nginx-%E9%85%8D%E7%BD%AE)Minio主要用于存储用户对话过程中的文件,而这些请求是直接进minio的,因此需要配置域名,或用户可以访问的IP地址。若是内网,给Minio一个内网IP即可。端口号默认Docker启动Minio使用的端口,映射到了主体的10005,具体查看minio.yml配置确认。
* [Go的代理](https://docs.openim.io/zh-hans/guides/gettingstarted/imsourcecodedeployment#24-%EF%B8%8F-%E5%88%9D%E5%A7%8B%E5%8C%96)golang语言没限制,不过golang依赖库的机制对github有非常强的依赖,大部分的依赖库都在github上,源码运行,需要配置go依赖的代理。
## 核心服务
在上面的准备基础完成之后,使用openim的指定的编译工具,mage start就可以直接启动核心服务了,这里看原文,不赘述。
如果自己的系统需要IM,那么到此为止,便可以基于OpenIM,来进行二次开发了。
但如果你想直接上手试一下消息,就需要下面的AppServer演示服务接口。
## AppServer演示接口服务
核心服务中,有USER的逻辑,但是却没有账号体系,因此上面的web-front和admin-front项目,可以打开,但却没办法登录,这需要另外的实现账号机制的后端的API服务。这个服务即AppServer,需要另外部署。
服务获取:`$ git clone https://github.com/openimsdk/chat&& cd chat`
如果您是在本地部署,那么admin-front的默认端口应该是11002,浏览器打开http://localhost:11002即可。
系统初始的管理员账号与密码是:chatAdmin/chatAdmin。
# 核心服务结构
* 主服务:
* openim-api:对客户端曝露的核心服务,http协议,信息查询,与消息发送请求。
* openim-msggateway:对客户端曝露的核心服务,ws协议,处理客户端实时消息发送与监听。
* openim-cmdutil
* openim-crontask
* openim-msgtransfer
* openim-push
* 底层服务
* openim-rpc-auth
* openim-rpc-conversation
* openim-rpc-friend
* openim-rpc-group
* openim-rpc-msg
* openim-rpc-third
* openim-rpc-user
主服务与底层服务之间的关系
![|center|800](https://picbed-1253586264.cos.ap-guangzhou.myqcloud.com/uPic/2025062316-5Yrpqe.jpg)
# SDK结构
OpenIM的SDK支持多平台,核心使用golang编写,封装了对API服务请求,与长链接的处理逻辑。而各端的封装主要完成了客户端参数配置,与桥接Golang的部分。
![800](https://picbed-1253586264.cos.ap-guangzhou.myqcloud.com/uPic/2025062316-QpVmoW.jpg)
@@ -0,0 +1,42 @@
##### OpenIM五种系统消息类型
* TextMessagePing/Pone Message
* BinaryMessageChat Message)
* CloseMessage(客户端关闭连接)
* PingMessage
* PongMessage
##### BinaryMessageReq/Resp
```golang
type Req struct {
ReqIdentifier int32 `json:"reqIdentifier" validate:"required"`
Token string `json:"token"`
SendID string `json:"sendID" validate:"required"`
OperationID string `json:"operationID" validate:"required"`
MsgIncr string `json:"msgIncr" validate:"required"`
Data []byte `json:"data"`
}
type Resp struct {
ReqIdentifier int32 `json:"reqIdentifier"`
MsgIncr string `json:"msgIncr"`
OperationID string `json:"operationID"`
ErrCode int `json:"errCode"`
ErrMsg string `json:"errMsg"`
Data []byte `json:"data"`
}
```
##### ReqIdentifier
```golang
WSGetNewestSeq = 1001
WSPullMsgBySeqList = 1002
WSSendMsg = 1003 // 一般聊天消息
WSSendSignalMsg = 1004
WSPullMsg = 1005
WSGetConvMaxReadSeq = 1006
WsPullConvLastMessage = 1007
WSPushMsg = 2001
WSKickOnlineMsg = 2002
WsLogoutMsg = 2003
WsSetBackgroundStatus = 2004
WsSubUserOnlineStatus = 2005
WSDataError = 3001
```
@@ -0,0 +1,68 @@
# 概括:
客户端交互的接口都在API服务中以http接口方式实现,而消息收发都在msggateway中以websocket协议实现。以下API定义在OpenIMAPI服务中,OpenIMAPI本身只起到接口转接的作用,具体的逻辑定义在下面的RPC服务,user&auth中。
# 接口
* 用户接口
```golang
u := NewUserApi(user.NewUserClient(userConn), client, cfg.Discovery.RpcService)
{
userRouterGroup := r.Group("/user")
userRouterGroup.POST("/user_register", u.UserRegister)
userRouterGroup.POST("/update_user_info", u.UpdateUserInfo)
userRouterGroup.POST("/update_user_info_ex", u.UpdateUserInfoEx)
userRouterGroup.POST("/set_global_msg_recv_opt", u.SetGlobalRecvMessageOpt)
userRouterGroup.POST("/get_users_info", u.GetUsersPublicInfo)
userRouterGroup.POST("/get_all_users_uid", u.GetAllUsersID)
userRouterGroup.POST("/account_check", u.AccountCheck)
userRouterGroup.POST("/get_users", u.GetUsers)
userRouterGroup.POST("/get_users_online_status", u.GetUsersOnlineStatus)
userRouterGroup.POST("/get_users_online_token_detail", u.GetUsersOnlineTokenDetail)
userRouterGroup.POST("/subscribe_users_status", u.SubscriberStatus)
userRouterGroup.POST("/get_users_status", u.GetUserStatus)
userRouterGroup.POST("/get_subscribe_users_status", u.GetSubscribeUsersStatus)
userRouterGroup.POST("/process_user_command_add", u.ProcessUserCommandAdd)
userRouterGroup.POST("/process_user_command_delete", u.ProcessUserCommandDelete)
userRouterGroup.POST("/process_user_command_update", u.ProcessUserCommandUpdate)
userRouterGroup.POST("/process_user_command_get", u.ProcessUserCommandGet)
userRouterGroup.POST("/process_user_command_get_all", u.ProcessUserCommandGetAll)
userRouterGroup.POST("/add_notification_account", u.AddNotificationAccount)
userRouterGroup.POST("/update_notification_account", u.UpdateNotificationAccountInfo)
userRouterGroup.POST("/search_notification_account", u.SearchNotificationAccount)
}
```
* 认证相关
```golang
{
a := NewAuthApi(pbAuth.NewAuthClient(authConn))
authRouterGroup := r.Group("/auth")
authRouterGroup.POST("/get_admin_token", a.GetAdminToken)
authRouterGroup.POST("/get_user_token", a.GetUserToken)
authRouterGroup.POST("/parse_token", a.ParseToken)
authRouterGroup.POST("/force_logout", a.ForceLogout)
}
```
# 认证逻辑
![|600](https://picbed-1253586264.cos.ap-guangzhou.myqcloud.com/uPic/2025062409-cKhzvC.jpg)
## Token说明
TokenKey格式:`前缀:用户ID:登录平台` = `UID_PID_TOKEN_STATUS:4834513665:Web`
TokenValue说明:{Token:TokenFlag}
TokenFlag取值如下,也就是说,TokenValue将以JsonMap的形式保存,最里可以看到最多4种情况
* NormalToken = 0
* InValidToken = 1
* KickedToken = 2
* ExpiredToken = 3
## AuthDatabase
Auth作为一个底层RPC服务,在启动是,会实例化AuthDatabase,Auth服务实现的4四认证相关接口,便是依赖authDatabase实现认证数据的管理。
authDatabase实现的四个方法如下,即操作上面Token的方法:
```
GetTokensWithoutError(ctx context.Context, userID string, platformID int) (map[string]int, error)
CreateToken(ctx context.Context, userID string, platformID int) (string, error)
BatchSetTokenMapByUidPid(ctx context.Context, tokens []string) error
SetTokenMapByUidPid(ctx context.Context, userID string, platformID int, m map[string]int) error
```
# 三方系统整合逻辑
即替换AuthDatabase的Token逻辑,比如改为依赖三方系统提供的Token获取与校验接口。当前出租屋的逻辑倒是权益之计,需要前端系统主动设置token.可以解决问题,但逻辑有漏洞。
@@ -0,0 +1 @@
https://open.work.weixin.qq.com/help2/pc/19886