Repository navigation
feat: 高级应用/嵌入态模式下按数据源过滤术语与 SQL 示例检索 - #1431
Closed
yigemaserwang wants to merge 3 commits into
Closed
yigemaserwang wants to merge 3 commits into
yigemaserwang wants to merge 3 commits into
Conversation
In advanced-application mode (including DataEase embedding), the retrieved datasource object comes from the external system whose id lives in a different id space than core_datasource, so ds_id resolved to None and terminology/SQL example retrieval was filtered by the application anchor only. Add resolve_embedded_ds_id_by_fingerprint() which matches the external datasource to the local core_datasource by physical connection fingerprint (type, normalized host, port, database, schema), and keep ds_id alongside the application anchor when filtering terminology and SQL examples. Fallback on mismatch returns None, which degrades to the current application-anchor-only behavior (under-filtering, never mis-filtering).
…ion mode When both the advanced application anchor and a datasource are provided, append a datasource condition to the ILIKE path and switch the vector path to a new SQL constant that requires the term to be either general (specific_ds = false/null) or bound to the current datasource (datasource_ids @> jsonb_build_array(:ds)). The standalone-mode branch is untouched.
…ion mode Same anchor-and-datasource double filtering for SQL examples: the example must be bound to the current datasource (exact equality) when both anchors are present. ILIKE and vector paths aligned.
Contributor
|
主要与我们现有逻辑有冲突: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
关联 issue: #1430
改动内容(What)
3 个文件、3 个逻辑 commit:
apps/chat/task/llm.py—— 新增resolve_embedded_ds_id_by_fingerprint()(连接指纹身份翻译),并在调用术语/SQL 示例检索时保留ds_id,与高级应用锚并行传递apps/terminology/curd/terminology.py—— 新增向量检索 SQL 常量 + ILIKE 路径叠加数据源条件(仅高级应用分支)apps/data_training/curd/data_training.py—— SQL 示例侧同构双锚过滤不涉及 schema 变更、不涉及 API 变更,独立态行为不变。
为什么改(Why)
在高级应用模式(type=1,DataEase 嵌入所用模式)下,术语与 SQL 示例检索只按应用锚过滤——数据源维度被显式丢弃(
calculate_ds_id = None)。同一应用服务多个数据源时:/* ds_canary */"* 的术语,在用户查询 A 库时被注入——且大模型真的执行了该指令(经chat_logoperate=9/10 留痕与生成 SQL 双重验证)。根因横跨三处:
run_task丢弃外部数据源 id(SQLBot 与嵌入宿主两侧 id 空间不同)、filter_*_template对 type=1 强制calculate_ds_id = None、检索函数的高级应用分支无数据源条件。完整分析与证据:见关联 issue。实现方式(How)
三步,对应根因链:
resolve_embedded_ds_id_by_fingerprint()按连接指纹(type, host, port, database, schema)将外部数据源映射到本地core_datasource:localhost/::1/0.0.0.0→127.0.0.1;实际配置中两种写法并存)、database必须参与(同实例多库场景)、SSL 不参与匹配(宿主→SQLBot 的数据源传输 VO 无 SSL 字段,两侧无对齐值)。calculate_ds_id;检索函数签名本就同时接受这两个参数槽位。specific_ds = false/null)放行,绑定含当前数据源的放行(datasource_ids @> jsonb_build_array(:ds)),其余过滤。ILIKE 与向量两条召回路径同构处理;示例侧为单值等值(datasource = :ds)。新向量 SQL 常量沿用现有运行时 config 参数风格(:similarity、:top_count)。失效模式安全:指纹匹配未命中(如 SQLBot 侧未录入该库)时
ds_id保持None,检索退化为现状的仅应用锚过滤——只会"少过滤",不会"错过滤"。验证(Verification)
/* ds_canary */注释。检查单(Checklist)