* Support Gitee AI(LLM Provider)
* refactor(server): 重构 GiteeAI 模型窗口限制功能,暂时将窗口限制硬编码,计划使用外部 API 数据和缓存
* updates for Gitee AI
* use legacy lookup since gitee does not enable getting token context windows
* add more missing records
* reorder imports
---------
Co-authored-by: 方程 <fangcheng@oschina.cn>
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* feat: add base64 document attachment support to API chat endpoints
* remove parsed documents from api chat process
linting for simplicity
fix logic issues with flow of data through the pipeline
documentiation
* change var scope
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Add Default System Prompt Management
- Introduced a new route for fetching and updating the default system prompt in the backend.
- Added a new Admin page for managing the default system prompt, including a form for editing and saving changes.
- Updated the SettingsSidebar to include a link to the new Default System Prompt page.
- Implemented fetching of available system prompt variables for use in the prompt editor.
- Enhanced the ChatSettings and ChatPromptSettings components to support the new default system prompt functionality.
This commit lays the groundwork for improved management of system prompts across workspaces.
* Remove validation for system prompt in ChatSettings component
* Add comment for system prompt in workspaces model
* linting, simplify logic for default assumption
* dev build
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* paperless ngx data connector
* wip resync paperless ngx
* fix generateChunkSource for resyncing paperless ngx
* lint
* Refactor Paperless-NGX connector
Fix issue with date rendering in tooltip + extended width
Move tooltip details to be column for more space
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Add User-Agent header on the requests sent by Generic OpenAI providers.
* Moved getAnythingLLMUserAgent helper fn to server/endpoints/utils.js and changed fallback version string to "unknown"
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Added the ability to pass in metadata to the /document/upload/{folderName} endpoint
* Added the ability to pass in metadata to the /document/upload-link endpoint
* feat: added metadata to document/upload api endpoint
* simplify optional metadata in document dev api endpoints
* lint
* patch handling of metadata in dev api
* Linting, small comments
---------
Co-authored-by: jstawskigmi <jstawski@getmyinterns.org>
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Create parse endpoint in collector (#4212)
* create parse endpoint in collector
* revert cleanup temp util call
* lint
* remove unused cleanupTempDocuments function
* revert slug change
minor change for destinations
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Add parsed files table and parse server endpoints (#4222)
* add workspace_parsed_files table + parse endpoints/models
* remove dev api parse endpoint
* remove unneeded imports
* iterate over all files + remove unneeded update function + update telemetry debounce
* Upload UI/UX context window check + frontend alert (#4230)
* prompt user to embed if exceeds prompt window + handle embed + handle cancel
* add tokenCountEstimate to workspace_parsed_files + optimizations
* use util for path locations + use safeJsonParse
* add modal for user decision on overflow of context window
* lint
* dynamic fetching of provider/model combo + inject parsed documents
* remove unneeded comments
* popup ui for attaching/removing files + warning to embed + wip fetching states on update
* remove prop drilling, fetch files/limits directly in attach files popup
* rework ux of FE + BE optimizations
* fix ux of FE + BE optimizations
* Implement bidirectional sync for parsed file states
linting
small changes and comments
* move parse support to another endpoint file
simplify calls and loading of records
* button borders
* enable default users to upload parsed files but NOT embed
* delete cascade on user/workspace/thread deletion to remove parsedFileRecord
* enable bgworker with "always" jobs and optional document sync jobs
orphan document job: Will find any broken reference files to prevent overpollution of the storage folder. This will run 10s after boot and every 12hr after
* change run timeout for orphan job to 1m to allow settling before spawning a worker
* linting and cleanup pr
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* dev build
* fix tooltip hiding during embedding overflow files
* prevent crash log from ERRNO on parse files
* unused import
* update docs link
* Migrate parsed-files to GET endpoint
patch logic for grabbing models names from utils
better handling for undetermined context windows (null instead of Pos_INIFI)
UI placeholder for null context windows
* patch URL
---------
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
* WIP on mobile connections
todo: register devices
todo: data sync or connection
* improve connection flow and registration
add streaming from service
TODO: user scoping
* dev build mobile support
* fix path
* handle relative URLs
* handle localhost access in product
* add device de-register
* sync styles
* move UI to be out of the normal path since beta only
* Add user scoping to mobile connection requests
Remigrate DB for user associations
Implement temp token registration to prevent unauthorized device registration requests
cleanup middlewares
* fix multimodal chats via openai compat api
* lint
* add tests for multi-modal content in openai compat endpoint
* refactor to normalize how openai attachments are handled
* uncheck file
* rewrite tests, autodetect mime from dataurl, and spread attachments from prompt
* lint
* revert and fix tests
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* implement importing of agent flows from community hub
* auto enable flow on import
* remove unused blocks for docker
prevent importing or saving of agent flows that have unsupported blocks for version or platform
* dev build
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* allow custom headers in upload-link endpoint
* override loader.scrape to allow for passing of headers in langchain puppeteer
* lint
* Rename some variables
move positional args to named args
update documentation to reflect arg changes and funciton sigs
validate header object before attempting to end to forward to request
* update header validation for custom headers
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP MCP full compatibility layer
* implement MCP agent function wrapping and invocation methods
* Add `uvx` to docker bin for MCP executions
* dev build
* prune removed data
* Wrap MCP servers to lazy load items to not block the UI
Mobile bug fixes
* arm64 test build
* reset dev builder
* remove unused prop
* enable slash commands in dev api
* lint
* Remove ability to use default slash commands in API request
Add `reset` param to body that can reset chats according to the api chat execution parameters
Allow null `message` if `reset` is set in request.
Added early return for if message is null and reset is true
Enable chat to reset chat history and continue `message` execution
Added generic WorkspaceChat history reset function. Deprecated others
* update grep function comment
remove debug
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add bio to users table
* lint
* add bio field to edit user admin page
* fix bio saving on new user
* simplify updating localstorage user
* linting
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* feat: Add endpoint to retrieve documents by folder name
* isWithin Check on path to prevent path traversal
* feat: Add endpoint to upload documents to a specified folder
* refactor upload to folder endpoint + update jsdoc for swagger
* linting
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* feat: Add endpoint to retrieve documents by folder name
* isWithin Check on path to prevent path traversal
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Reranker WIP
* add cacheing and singleton loading
* Add field to workspaces for vectorSearchMode
Add UI for lancedb to change mode
update all search endpoints to pass in reranker prop if provider can use it
* update hint text
* When reranking, swap score to rerank score
* update optchain
