* PGVector support for vector db storage
* forgot files
* comments
* dev build
* Add ENV connection and table schema validations for vector table
add .reset call to drop embedding table when changing the AnythingLLM embedder
update instrutions
Add preCheck error reporting in UpdateENV
add timeout to pg connection
* update setup
* update README
* update doc
* feat: implement iam role auth for bedrock
* fix: make client refreshes properly when switching between iam_user and iam_role
* checkout agent flow
* fix aiprovider for bedrock in agent use
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Fixed two primary issues discovered while using AWS Bedrock with Anthropic Claude Sonnet models:
- Context Window defaults to 8192 maximum, which isn't correct
- Multimodal stopped working when removing langchain, which was transparently handling image_url to a format sonnet expects.
* Ran `yarn lint`
* Updated .env.example to have aws bedrock examples too
* Refactor for readability
move utils for AWS specific functionality to subfile
add token output max to ENV so setting persits
---------
Co-authored-by: Tristan Stahnke <tristan.stahnke+gpsec@guidepointsecurity.com>
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* allow custom headers in upload-link endpoint
* override loader.scrape to allow for passing of headers in langchain puppeteer
* lint
* Rename some variables
move positional args to named args
update documentation to reflect arg changes and funciton sigs
validate header object before attempting to end to forward to request
* update header validation for custom headers
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Enable bypass of ip limitations via ENV in collector startup
resolves#3625
connect #3626
* dev build
* bump dockerx build action
* enable runtime setting config of collector requests
* comments and linting for option passing
* unset
* unset
* update docs link
* linting and docs
* Update Azure AI options and model map with new model configurations
* linting
---------
Co-authored-by: Shinya Suzuki <shinya.s.825@gmail.com>
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* handling of citations in openRouter provider #3581
* Update pplx enrichToken function comment
Modify OR enrichToken to be generic handler function with optional params
handle _just_ Perplexity in-line citations since no other models support this functionality
* remove console log
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Update defaultModels.js
add gemma-3-27b-it to v1BetaModels
* Update defaultModels.js
20250330 model update
* Update defaultModels.js
remove text embedding
* Update name and inputTokenLimit modelMap.js
* Update gemini to load models from both endpoints
dedupe models
decide endpoint based on expieremental status from fetch
add util script for maintainers
reduce cache time on gemini models to 1 day
* remove comment
---------
Co-authored-by: DreamerC <dreamerwolf.tw@gmail.com>
* WIP MCP full compatibility layer
* implement MCP agent function wrapping and invocation methods
* Add `uvx` to docker bin for MCP executions
* dev build
* prune removed data
* Wrap MCP servers to lazy load items to not block the UI
Mobile bug fixes
* arm64 test build
* reset dev builder
* remove unused prop
* enable slash commands in dev api
* lint
* Remove ability to use default slash commands in API request
Add `reset` param to body that can reset chats according to the api chat execution parameters
Allow null `message` if `reset` is set in request.
Added early return for if message is null and reset is true
Enable chat to reset chat history and continue `message` execution
Added generic WorkspaceChat history reset function. Deprecated others
* update grep function comment
remove debug
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Add multilingual support for ocr mudule
* Add OCR langauge as server var that is passed into Collector
Support all valid tesseract language codes
Filter and parse only valid codes with fallbacks'
* persist TARGET_OCR_LANG
* update docker example env
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* feat: add new model provider PPIO
* fix: fix ppio model fetching
* fix: code lint
* reorder LLM
update interface for streaming and chats to use valid keys
linting
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Windows development environment variables support
* moved cross-env to dev dependencies
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* add bio to users table
* lint
* add bio field to edit user admin page
* fix bio saving on new user
* simplify updating localstorage user
* linting
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* feat: Add endpoint to retrieve documents by folder name
* isWithin Check on path to prevent path traversal
* feat: Add endpoint to upload documents to a specified folder
* refactor upload to folder endpoint + update jsdoc for swagger
* linting
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* add claude 3-7 sonnet
* made all the changes everywhere
* add 3-7-sonnet-latest model
* lint
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* feat: Add endpoint to retrieve documents by folder name
* isWithin Check on path to prevent path traversal
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
bug fixes for sanitizing Namespaces and handling chunk size limit of astradb collections in each doc
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Add `querySelectorAll` capability to web-scraping block
* patches and fallbacks
* fix styles of text in web scraping block
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* chore: rename Github to GitHub
Signed-off-by: Adam Setch <adam.setch@outlook.com>
* chore: rename Github to GitHub
Signed-off-by: Adam Setch <adam.setch@outlook.com>
* Undo some code changes for references
---------
Signed-off-by: Adam Setch <adam.setch@outlook.com>
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Add tokenizer improvments via Singleton class
linting
* dev build
* Estimation fallback when string exceeds a fixed byte size
* Add notice to tiktoken on backend
* feat: add support for voyage-3-large and voyage-code-3 embedding models
- Add voyage-3-large and voyage-code-3 to VoyageAiOptions dropdown
- Update getMaxEmbeddingLength to support 32k context for new models
- Update .env.example with new model options
* unset env example
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Reranker WIP
* add cacheing and singleton loading
* Add field to workspaces for vectorSearchMode
Add UI for lancedb to change mode
update all search endpoints to pass in reranker prop if provider can use it
* update hint text
* When reranking, swap score to rerank score
* update optchain
* Add support for Google Generative AI (Gemini) embedder
* Add missing example in docker
Fix UI key elements in options
Add Gemini to data handling section
Patch issues with chunk handling during embedding
* remove dupe in env
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Add support for gemini authenticated models endpoint
add customModels entry
add un-authed fallback to default listing
separate models by expiermental status
resolves#2866
* add back improved logic for apiVersion decision making
* add writible fields to dev api new workspace endpoint
* lint
* implement validations for workspace model
* update swagger comments
* simplify validations for workspace on frontend and API
* cleanup validations
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* wip remove all docs clear vector db on embedder/vector db change
* purge all cached docs and remove docs from workspaces on vectordb/embedder change
* lint
* remove unneeded console log
* remove reset vector stores endpoint and move to server side updateENV with postUpdate check
* reset embed module
* remove unused import
* simplify deletion process
rescoped document deletion to be more general for speed, everything needs to be reset anyway
fixed issue where unembedded docs not in any workspaces, but cached, were not removed
* add back missing readme file
update warning text modals
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Add vector search API endpoint
* Add missing import
* Modify the data that is returned
* Change similarityThreshold to scoreThreshold
As this is what is actually returned by the search
* Removing logging (oops!)
* chore: regen swagger docs for new endpoint
fix: update function to sanity check values to prevent crashes during search
---------
Co-authored-by: Scott Bowler <scott@dcsdigital.co.uk>
* wip hub connection page fe + backend
* lint
* implement backend for local hub items + placeholder endpoints to fetch hub app data
* fix hebrew translations
* revamp community integration flow
* change sidebar
* Auto import if id in URL param
remove preview in card screen and instead go to import flow
* get user's items + team items from hub + ui improvements to hub settings
* lint
* fix merge conflict
* refresh hook for community items
* add fallback for user items
* Disable bundle items by default on all instances
* remove translations (will complete later)
* loading skeleton
* Make community hub endpoints admin only
show visibility on items
combine import/apply for items to they are event logged for review
* improve middleware and import flow
* community hub ui updates
* Adjust importing process
* community hub to dev
* Add webscraper preload into imported plugins
* add runtime property to plugins
* Fix button status on imported skill change
show alert on skill change
Update markdown type and theme on import of agent skill
* update documentaion paths
* remove unused import
* linting
* review loading state
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* exposes `maxConcurrentChunks` parameter for the generic openai embedder through configuration. This allows setting a batch size for endpoints which don't support the default of 500
* Update new field to new UI
make getting to ensure proper type and format
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* togetherai llama 3.2 vision models support
* remove console log
* fix listing to reflect what is on the chart
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* feat: add new model provider: Novita AI
* feat: finished novita AI
* fix: code lint
* remove unneeded logging
* add back log for novita stream not self closing
* Clarify ENV vars for LLM/embedder seperation for future
Patch ENV check for workspace/agent provider
---------
Co-authored-by: Jason <ggbbddjm@gmail.com>
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* Fix incorrect json API description.
* small edits and validity checks
* remove console.logs
* unset and recheck changes
---------
Co-authored-by: Adam <phazei@gmail.com>
* thread creation additional params name and slug, with api
* typo fix
* Rebuild openai Swagger docs
Handle validations for fields to prevent invalid field inputs for .new
Enforce sluggification of `slug` to prevent breaking of URL structs
---------
Co-authored-by: abrakadobr <abrakadobr@gmail.com>
Adds support for only the llama3.2 vision models on groq. This comes with many conditionals and nuances to handle as Groqs vision implemention is quite bad right now
* Update OpenAI TTS config to allow a custom BaseURL
* uncheck config file
* break openai generic TTS into its own provider
* add space
* hide TTS on user msg
---------
Co-authored-by: Adam <phazei@gmail.com>
* support generic openai workspace model
* Update UI for free form input for some providers
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* set message limit per user
* remove old limit user messages + unused admin page
* fix daily message validation
* refactor message limit input
refactor canSendChat on user to a method on user model
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Refactor handleDefaultStreamResponseV2 function for better error handling
* run yarn lint
* small error handling changes
* update error handling flow and scope of vars
* add back space
---------
Co-authored-by: Roman <rrojaski@gmail.com>
* refactor agent to add fallback to workspace, then to chat provider/model
* commenting
update logic for bedrock and fireworks fallbacks
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* #2317 Fetch pinned documents once per folder to reduce the number of queries.
* Reorder the lines to keeps const declarations together.
* Add some comments to functions
move pinned document fetch for folder to function
move watched documents per-folder to also function the same
remove unused function in documents model
---------
Co-authored-by: Błażej Owczarczyk <blazeyy@gmail.com>
* Issue #1943: Add support for LLM provider - Fireworks AI
* Update UI selection boxes
Update base AI keys for future embedder support if needed
Add agent capabilites for FireworksAI
* class only return
---------
Co-authored-by: Aaron Van Doren <vandoren96+1@gmail.com>
* patch no text results for milvus chunks
* wrap addDocumentToNamespace in try catch for handling milvus errors
* lint
* revert milvus db changes
* add try catch to handle grpc error from milvus
* ollama: Switch from parallel to sequential chunk embedding
* throw error on empty embeddings
---------
Co-authored-by: John Blomberg <john.jb.blomberg@gmail.com>
* initial commit for chrome extension
* wip browser extension backend
* wip frontend browser extension settings
* fix typo for browserExtension route
* implement verification codes + frontend panel for browser extension keys
* reorganize + state management for all connection states
* implement embed to workspace
* add send page to anythingllm extension option + refactor
* refactor connection string auth + update context menus + organize background.js into models
* popup extension from main app and save if successful
* fix hebrew translation misspelling
* fetch custom logo inside chrome extension
* delete api keys on disconnect of extension
* use correct apiUrl constant in frontend + remove unneeded comments
* remove upload-link endpoint and send inner text html to raw text collector endpoint
* update readme
* fix readme link
* fix readme typo
* update readme
* handle deletion of browser keys with key id and DELETE endpoint
* move event string to constant
* remove tablename and writable fields from BrowserExtensionApiKey backend model
* add border-none to all buttons and inputs for desktop compatibility
* patch prisma injections
* update delete endpoints to delete keys by id
* remove unused prop
* add button to attempt browser extension connection + remove max active keys
* wip multi user mode support
* multi user mode support
* clean up backend + show created by in frotend browser extension page
* show multi user warning message on key creation + hide context menus when no workspaces
* show browser extension options to managers
* small backend changes and refactors
* extension cleanup
* rename submodule
* extension updates & docs
* dev docker build
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* Refactor api endpoint chat handler to its own function
remove legacy `chatWithWorkspace` and cleanup `index.js`
* Add `sessionId` in dev API to partition chats logically statelessly
* Enable agent context windows to be accurate per provider:model
* Refactor model mapping to external file
Add token count to document length instead of char-count
refernce promptWindowLimit from AIProvider in central location
* remove unused imports
* force lowercase and no space for new and
edit user modals
* edit account modal validations
* use pattern for form validation + remove validations from edit user
* revert comment deletions
* comment fix
* update validation message
* update regex
allow updating by block name changes to invalid names
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Revert "Patch ODBC support from missing binary/headers for node-odbc"
This reverts commit 9de6b1cc26.
* Revert "OBDC Support (#1933)"
This reverts commit cd597a361e.
* add possibility to connect to SQL Base by ODBC
---------
Co-authored-by: suchaudn <nicolas.suchaud@legrand.fr>
Co-authored-by: nicho2 <nicho2@laposte.net>
* Add support for GitLab repo collection as well as Github Repo collection
* Refactor for repo collectors to be more compact
---------
Co-authored-by: Emil Rofors <emirof@gmail.com>
* return 400 if user creation failed
The client might not check the error field unless they realize it's a 4xx error instead of 200 success.
* Update index.js
* use one-liner style
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Added Supported Models Free Tier - chat_models.txt
Need to fill in correct Parameter Count.
* Bump perplexity model
closes#1901closes#1900
---------
Co-authored-by: Tim-Hoekstra <135951177+Tim-Hoekstra@users.noreply.github.com>
* bump base image
* Bump mysql
* attestations
* attestations perms
* attestations perms
* fix permissions for attestetions for master push
* test cleanup
* repin base image
* revert base
* patch frontend-clean
* force resolve braces to fixed version
* rebump
* patch llm providers that have manual inputs for model pref
* refactor agent model fallback
update UI to show disabled providers to stop questions about provider limitations
* patch log on startup
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* implement delete message feature
* add new delete chat endpoint that just hides messages from ui and fix perms
* refactor: add delete-event hooks for delete so we dont need to pass so much
refactor: implmentation of delete endpoint PUT
* update animation
linting
remove element on animiation to prevent popin
* linting and cleanup imports
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* implement thread forking feature
* rename thread based on forked message
* refactor bulk message create for thread fork + bump prisma version
* revert prisma version bump
* add todo to bulkCreate function in workspace chats
* cast user input to expected type to prevent prisma injection
* refactor: update order of ops for thread fork
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* wip bg workers for live document sync
* Add ability to re-embed specific documents across many workspaces via background queue
bgworkser is gated behind expieremental system setting flag that needs to be explictly enabled
UI for watching/unwatching docments that are embedded.
TODO: UI to easily manage all bg tasks and see run results
TODO: UI to enable this feature and background endpoints to manage it
* create frontend views and paths
Move elements to correct experimental scope
* update migration to delete runs on removal of watched document
* Add watch support to YouTube transcripts (#1716)
* Add watch support to YouTube transcripts
refactor how sync is done for supported types
* Watch specific files in Confluence space (#1718)
Add failure-prune check for runs
* create tmp workflow modifications for beta image
* create tmp workflow modifications for beta image
* create tmp workflow modifications for beta image
* dual build
update copy of alert modals
* update job interval
* Add support for live-sync of Github files
* update copy for document sync feature
* hide Experimental features from UI
* update docs links
* [FEAT] Implement new settings menu for experimental features (#1735)
* implement new settings menu for experimental features
* remove unused context save bar
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* dont run job on boot
* unset workflow changes
* Add persistent encryption service
Relay key to collector so persistent encryption can be used
Encrypt any private data in chunkSources used for replay during resync jobs
* update jsDOC
* Linting and organization
* update modal copy for feature
---------
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
* WIP workspace threads developer api endpoints
* fix swagger docs
* fix broken endpoints and test on single/multi user modes
* support for chatting with workspace thread by userId
* update swagger json
* Update workspace.js
adds userId to the workspaceUsers function
* Update openapi.json
Updates the workspace/users API docs to display correct 200 response example, including userId, username, role and last update datetime.
* add user management endpoints to list all users via developer api
* rename users to user management in swagger docs
* linting
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Enhance API - add GET for users with access to workspace
Adds GET request endpoint for retrieving a list of users with permissions to access the specified workspace
* Update Swagger for users with access to workspace
Adds swagger docs for the endpoint used to retrieve users with access to workspace. "v1/admin/workspaces/:workspaceId/users"
* make thread name 'Thread' when new thread is created
* implement auto generated thread titles
* implement truncated prompt as thread name
* move rename of thread into workspaceThread function
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Updated apt-packages source for devcontainer
Switched the devcontainer's package source to a different repository to
align with updated dependencies and package availability. The previous
source from 'rocker-org' is replaced with 'devcontainers-contrib', which
may offer more recent or relevant development tools.
* Subject: Centralize prettier ignores and refine
config
Body:
Centralized all prettier ignore rules by removing individual
`.prettierignore` files in subprojects and updating the root
`.prettierignore` to include previously ignored patterns, ensuring
consistency across the workspace. Additionally, the prettier
configuration was refined by making the file pattern for `.config.js`
files consistent and adjusting quote styles for better readability. All
lint scripts across the project were updated to respect the centralized
ignore path, enhancing maintainability.
The consolidation simplifies the process of managing ignore rules as the
project scales, ensuring developers can focus on writing code without
worrying about divergent formatting standards. These changes also align
with introducing comprehensive linting across multiple environments to
keep the codebase clean and consistent.
This adjustment is a foundational step towards a more streamlined and
unified code base, making it easier for new contributors to adhere to
established coding standards and reducing the cognitive load associated
with managing multiple configuration files across the project.
* unset package json changes
---------
Co-authored-by: Francisco Bischoff <franzbischoff@gmail.com>
Co-authored-by: Francisco Bischoff <984592+franzbischoff@users.noreply.github.com>
* implement custom icon on login screen for single & multi user + custom app name feature
* hide field when not relevant
* set customApp name
* show original anythingllm login logo unless custom logo is set
* nit-picks
* remove console log
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add support for voyageai embedder
* remove unneeded import
* linting
* Add ENV examples
Update how chunks are processed for Voyage
use correct langchain import
Add data handling
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
