* add eslint config to server
* add break statements to switch case
* add support for browser globals and turn off empty catch blocks
* disable lines with useless try/catch wrappers
* format
* fix no-undef errors
* disbale lines violating no-unsafe-finally
* ignore syncStaticLists.mjs
* use proper null check for creatorId instead of unreachable nullish coalescing
* remove unneeded typescript eslint comment
* make no-unused-private-class-members a warning
* disable line for no-empty-objects
* add new lint script
* fix no-unused-vars violations
* make no-unsued-vars an error
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Migrate Astra to class (#4722)
migrate astra to class
* Migrate LanceDB to class (#4721)
migrate lancedb to class
* Migrate Pinecone to class (#4726)
migrate pinecone to class
* Migrate Zilliz to class (#4729)
migrate zilliz to class
* Migrate Weaviate to class (#4728)
migrate weaviate to class
* Migrate Qdrant to class (#4727)
migrate qdrant to class
* Migrate Milvus to class (#4725)
migrate milvus to class
* Migrate Chroma to class (#4723)
migrate chroma to class
* Migrate Chroma Cloud to class (#4724)
* migrate chroma to class
* migrate chroma cloud to class
* move limits to class field
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Migrate PGVector to class (#4730)
* migrate pgvector to class
* patch pgvector test
* convert connectionString, tableName, and validateConnection to static methods
* move instance properties to class fields
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Refactor Zilliz Cloud vector DB provider (#4749)
simplify zilliz implementation by using milvus as base class
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* VectorDatabase base class (#4738)
create generic VectorDatabase base class
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Extend VectorDatabase base class to all providers (#4755)
extend VectorDatabase base class to all providers
* patch lancedb import
* breakout name and add generic logger
* dev tag build
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* fix astra not showing similarity score in citations ui
* fix qdrant not showing similarity score in citations ui
* fix weaviate not showing similarity score in citations ui
* WIP on embedder selection
TODO: apply splitting and query prefixes (if applicable)
* wip on upsert
* Support base model
support nomic-text-embed-v1
support multilingual-e5-small
Add prefixing for both embedding and query for RAG tasks
Add chunking prefix to all vector dbs to apply prefix when possible
Show dropdown and auto-pull on new selection
* norm translations
* move supported models to constants
handle null seelction or invalid selection on dropdown
update comments
* dev
* patch text splitter maximums for now
* normalize translations
* add tests for splitter functionality
* normalize
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* wip bg workers for live document sync
* Add ability to re-embed specific documents across many workspaces via background queue
bgworkser is gated behind expieremental system setting flag that needs to be explictly enabled
UI for watching/unwatching docments that are embedded.
TODO: UI to easily manage all bg tasks and see run results
TODO: UI to enable this feature and background endpoints to manage it
* create frontend views and paths
Move elements to correct experimental scope
* update migration to delete runs on removal of watched document
* Add watch support to YouTube transcripts (#1716)
* Add watch support to YouTube transcripts
refactor how sync is done for supported types
* Watch specific files in Confluence space (#1718)
Add failure-prune check for runs
* create tmp workflow modifications for beta image
* create tmp workflow modifications for beta image
* create tmp workflow modifications for beta image
* dual build
update copy of alert modals
* update job interval
* Add support for live-sync of Github files
* update copy for document sync feature
* hide Experimental features from UI
* update docs links
* [FEAT] Implement new settings menu for experimental features (#1735)
* implement new settings menu for experimental features
* remove unused context save bar
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* dont run job on boot
* unset workflow changes
* Add persistent encryption service
Relay key to collector so persistent encryption can be used
Encrypt any private data in chunkSources used for replay during resync jobs
* update jsDOC
* Linting and organization
* update modal copy for feature
---------
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
* create configurable topN per workspace
* Update TopN UI text
Fix fallbacks for all providers
Add SQLite CHECK to TOPN value
* merge with master
Update zilliz provider for variable TopN
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP on continuous prompt window summary
* wip
* Move chat out of VDB
simplify chat interface
normalize LLM model interface
have compression abstraction
Cleanup compressor
TODO: Anthropic stuff
* Implement compression for Anythropic
Fix lancedb sources
* cleanup vectorDBs and check that lance, chroma, and pinecone are returning valid metadata sources
* Resolve Weaviate citation sources not working with schema
* comment cleanup
* WIP Anythropic support for chat, chat and query w/context
* Add onboarding support for Anthropic
* cleanup
* fix Anthropic answer parsing
move embedding selector to general util
* include score value in similarityResponse for weaviate
* include score value in si
milarityResponse for qdrant
* include score value in si
milarityResponse for pinecone
* include score value in similarityResponse for chroma
* include score value in similarityResponse for lancedb
* distance to similarity
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
Limit is due to POST body max size. Sufficiently large requests will abort automatically
We should report that error back on the frontend during embedding
Update vectordb providers to return on failed
* WIP converted all sqlite models into prisma calls
* modify db setup and fix ApiKey model calls in admin.js
* renaming function params to be consistent
* converted adminEndpoints to utilize prisma orm
* converted chatEndpoints to utilize prisma orm
* converted inviteEndpoints to utilize prisma orm
* converted systemEndpoints to utilize prisma orm
* converted workspaceEndpoints to utilize prisma orm
* converting sql queries to prisma calls
* fixed default param bug for orderBy and limit
* fixed typo for workspace chats
* fixed order of deletion to account for sql relations
* fix invite CRUD and workspace management CRUD
* fixed CRUD for api keys
* created prisma setup scripts/docs for understanding how to use prisma
* prisma dependency change
* removing unneeded console.logs
* removing unneeded sql escape function
* linting and creating migration script
* migration from depreciated sqlite script update
* removing unneeded migrations in prisma folder
* create backup of old sqlite db and use transactions to ensure all operations complete successfully
* adding migrations to gitignore
* updated PRISMA.md docs for info on how to use sqlite migration script
* comment changes
* adding back migrations folder to repo
* Reviewing SQL and prisma integraiton on fresh repo
* update inline key replacement
* ensure migration script executes and maps foreign_keys regardless of db ordering
* run migration endpoint
* support new prisma backend
* bump version
* change migration call
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
