* add eslint config to server
* add break statements to switch case
* add support for browser globals and turn off empty catch blocks
* disable lines with useless try/catch wrappers
* format
* fix no-undef errors
* disbale lines violating no-unsafe-finally
* ignore syncStaticLists.mjs
* use proper null check for creatorId instead of unreachable nullish coalescing
* remove unneeded typescript eslint comment
* make no-unused-private-class-members a warning
* disable line for no-empty-objects
* add new lint script
* fix no-unused-vars violations
* make no-unsued-vars an error
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Migrate Astra to class (#4722)
migrate astra to class
* Migrate LanceDB to class (#4721)
migrate lancedb to class
* Migrate Pinecone to class (#4726)
migrate pinecone to class
* Migrate Zilliz to class (#4729)
migrate zilliz to class
* Migrate Weaviate to class (#4728)
migrate weaviate to class
* Migrate Qdrant to class (#4727)
migrate qdrant to class
* Migrate Milvus to class (#4725)
migrate milvus to class
* Migrate Chroma to class (#4723)
migrate chroma to class
* Migrate Chroma Cloud to class (#4724)
* migrate chroma to class
* migrate chroma cloud to class
* move limits to class field
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Migrate PGVector to class (#4730)
* migrate pgvector to class
* patch pgvector test
* convert connectionString, tableName, and validateConnection to static methods
* move instance properties to class fields
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Refactor Zilliz Cloud vector DB provider (#4749)
simplify zilliz implementation by using milvus as base class
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* VectorDatabase base class (#4738)
create generic VectorDatabase base class
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Extend VectorDatabase base class to all providers (#4755)
extend VectorDatabase base class to all providers
* patch lancedb import
* breakout name and add generic logger
* dev tag build
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* fix astra not showing similarity score in citations ui
* fix qdrant not showing similarity score in citations ui
* fix weaviate not showing similarity score in citations ui
* fix(server): correct Qdrant batching logic for large uploads
Closes#4544
* modify O(3N) process to O(N)
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Fix JSDOC for updateOrCreateCollection
* Add sanitizeForJsonb method to PGVector for safe JSONB handling
This new method recursively sanitizes values intended for JSONB storage, removing disallowed control characters and ensuring safe insertion into PostgreSQL. The method is integrated into the vector insertion process to sanitize metadata before database operations.
* Add unit tests for PGVector.sanitizeForJsonb method
This commit introduces a comprehensive test suite for the PGVector.sanitizeForJsonb method, ensuring it correctly handles various input types, including null, undefined, strings with disallowed control characters, objects, arrays, and Date objects. The tests verify that the method sanitizes inputs without mutating the original data structures.
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* add chroma cloud as new vector db provider
* update docker example env
* extend chroma class to chroma cloud
* update readme
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP on embedder selection
TODO: apply splitting and query prefixes (if applicable)
* wip on upsert
* Support base model
support nomic-text-embed-v1
support multilingual-e5-small
Add prefixing for both embedding and query for RAG tasks
Add chunking prefix to all vector dbs to apply prefix when possible
Show dropdown and auto-pull on new selection
* norm translations
* move supported models to constants
handle null seelction or invalid selection on dropdown
update comments
* dev
* patch text splitter maximums for now
* normalize translations
* add tests for splitter functionality
* normalize
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* PGVector support for vector db storage
* forgot files
* comments
* dev build
* Add ENV connection and table schema validations for vector table
add .reset call to drop embedding table when changing the AnythingLLM embedder
update instrutions
Add preCheck error reporting in UpdateENV
add timeout to pg connection
* update setup
* update README
* update doc
bug fixes for sanitizing Namespaces and handling chunk size limit of astradb collections in each doc
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* chore: rename Github to GitHub
Signed-off-by: Adam Setch <adam.setch@outlook.com>
* chore: rename Github to GitHub
Signed-off-by: Adam Setch <adam.setch@outlook.com>
* Undo some code changes for references
---------
Signed-off-by: Adam Setch <adam.setch@outlook.com>
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Reranker WIP
* add cacheing and singleton loading
* Add field to workspaces for vectorSearchMode
Add UI for lancedb to change mode
update all search endpoints to pass in reranker prop if provider can use it
* update hint text
* When reranking, swap score to rerank score
* update optchain
* patch no text results for milvus chunks
* wrap addDocumentToNamespace in try catch for handling milvus errors
* lint
* revert milvus db changes
* add try catch to handle grpc error from milvus
* wip bg workers for live document sync
* Add ability to re-embed specific documents across many workspaces via background queue
bgworkser is gated behind expieremental system setting flag that needs to be explictly enabled
UI for watching/unwatching docments that are embedded.
TODO: UI to easily manage all bg tasks and see run results
TODO: UI to enable this feature and background endpoints to manage it
* create frontend views and paths
Move elements to correct experimental scope
* update migration to delete runs on removal of watched document
* Add watch support to YouTube transcripts (#1716)
* Add watch support to YouTube transcripts
refactor how sync is done for supported types
* Watch specific files in Confluence space (#1718)
Add failure-prune check for runs
* create tmp workflow modifications for beta image
* create tmp workflow modifications for beta image
* create tmp workflow modifications for beta image
* dual build
update copy of alert modals
* update job interval
* Add support for live-sync of Github files
* update copy for document sync feature
* hide Experimental features from UI
* update docs links
* [FEAT] Implement new settings menu for experimental features (#1735)
* implement new settings menu for experimental features
* remove unused context save bar
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* dont run job on boot
* unset workflow changes
* Add persistent encryption service
Relay key to collector so persistent encryption can be used
Encrypt any private data in chunkSources used for replay during resync jobs
* update jsDOC
* Linting and organization
* update modal copy for feature
---------
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
* feature: Integrate Astra as vectorDBProvider
feature: Integrate Astra as vectorDBProvider
* Update .env.example
* Add env.example to docker example file
Update spellcheck fo Astra
Update Astra key for vector selection
Update order of AstraDB options
Resize Astra logo image to 330x330
Update methods of Astra to take in latest vectorDB params like TopN and more
Update Astra interface to support default methods and avoid crash errors from 404 collections
Update Astra interface to comply to max chunk insertion limitations
Update Astra interface to dynamically set dimensionality from chunk 0 size on creation
* reset workspaces
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* migrate pinecone package to latest version and migrate pinecone vectordb provider class
* remove pinecone environment name env variable and update docs to reflect removal & serverless support complete
* migrate query for pinecone db
* typo in log
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* create configurable topN per workspace
* Update TopN UI text
Fix fallbacks for all providers
Add SQLite CHECK to TOPN value
* merge with master
Update zilliz provider for variable TopN
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
