michael/merlyn
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Patch 5: Replace silent truncation with explicit token limit in read-text-file

Browse Source 
Silent truncation of legal documents is unacceptable - Merlyn was reading
partial files without knowing it.

New behavior:
- Files under AGENT_MAX_FILE_TOKENS (default 500,000): return full content
- Files over limit: return explicit message with token count and options
- Paralegal controls any summarization decision
- Merlyn never silently truncates legal documents
This commit is contained in:
PQ32 Developer 2026-05-10 15:28:14 -07:00
parent 5bcef7d604
commit acbf66c5bf
1 changed files with 14 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
server/utils/agents/aibitat/plugins/filesystem/read-text-file.js
View File

@ -104,19 +104,23 @@ module.exports.FilesystemReadTextFile = {
this.super.introspect(`Successfully read ${filePath}`); this.super.introspect(`Successfully read ${filePath}`);
} }
const { content: finalContent, wasTruncated } = const { TokenManager } = require("../../../../helpers/tiktoken");
filesystem.truncateContentForContext( const tokenManager = new TokenManager(this.super.model);
content, const tokenCount = tokenManager.countFromString(content);
this.super, const maxFileTokens = Number(process.env.AGENT_MAX_FILE_TOKENS) || 500_000;
"[Content truncated - file exceeds context limit. Use head/tail parameters to read specific portions.]"
);
if (wasTruncated) { if (tokenCount > maxFileTokens) {
this.super.introspect( return [
`${this.caller}: File content was truncated to fit context limit` `File "${filePath}" contains ${tokenCount.toLocaleString()} tokens, which exceeds the limit of ${maxFileTokens.toLocaleString()} tokens.`,
); `To proceed, choose one of the following options:`,
`1. Read specific portions using the head or tail parameters`,
`2. Search for specific content using search-files`,
`3. Request a summary of the file`,
].join("\n");
} }
const finalContent = content;
const filename = path.basename(validPath); const filename = path.basename(validPath);
this.super.addCitation?.({ this.super.addCitation?.({
id: `fs-${Buffer.from(validPath).toString("base64url").slice(0, 32)}`, id: `fs-${Buffer.from(validPath).toString("base64url").slice(0, 32)}`,