* implement native embedder job queue
* persist embedding progress across renders
* add development worker timeouts
* change to static method
* native reranker
* remove useless return
* lint
* simplify
* make embedding worker timeout value configurable by admin
* add event emission for missing data
* lint
* remove onProgress callback argument
* make rerank to rerankDirect
* persists progress state across app reloads
* remove chunk level progress reporting
* remove unuse dvariable
* make NATIVE_RERANKING_WORKER_TIMEOUT user configurable
* remove dead code
* scope embedding progress per-user and clear stale state on SSE reconnect
* lint
* revert vector databases and embedding engines to call their original methods
* simplify rerank
* simplify progress fetching by removing updateProgressFromApi
* remove duplicate jsdoc
* replace sessionStorage persistence with server-side history replay for embedding progress
* fix old comment
* fix: ignore premature SSE all_complete when embedding hasn't started yet
The SSE connection opens before the embedding API call fires, so the
server sees no buffered history and immediately sends all_complete.
Firefox dispatches this eagerly enough that it closes the EventSource
before real progress events arrive, causing the progress UI to clear
and fall back to the loading spinner. Chrome's EventSource timing
masks the race.
Track slugs where startEmbedding was called but no real progress event
has arrived yet via awaitingProgressRef. Ignore the first all_complete
for those slugs and keep the connection open for the real events.
* reduce duplication with progress emissions
* remove dead code
* refactor: streamline embedding progress handling
Removed unnecessary tracking of slugs for premature all_complete events in the EmbeddingProgressProvider. Updated the server-side logic to avoid sending all_complete when no embedding is in progress, allowing the connection to remain open for real events. Adjusted the embedding initiation flow to ensure the server processes the job before the SSE connection opens, improving the reliability of progress updates.
* fix stale comment
* remove unused function
* fix event emissions for document creation failure
* refactor: move Reranking Worker Idle Timeout input to LanceDBOptions component
Extracted the Reranking Worker Idle Timeout input from GeneralEmbeddingPreference and integrated it into the LanceDBOptions component. This change enhances modularity and maintains a cleaner structure for the settings interface.
* lint
* remove unused hadHistory vars
* refactor workspace directory by hoisting component and converting into functions
* moved EmbeddingProgressProvider to wrap Document Manager Modal
* refactor embed progress SSE connection to use fetchEventSource instead of native EventSource API.
* refactor message handlng into a function and reduce duplication
* refactor: utilize writeResponseChunk for event emissions in document embedding progress SSE
* refactor: explicit in-proc embedding and rerank methods that are called by workers instead of process.send checks
* Abstract EmbeddingProgressBus and Worker Queue into modules
* remove error and toast messages on embed process result
* use safeJsonParse
* add chunk-level progress events with per-document progress bar in UI
* remove unused parameter
* rename all worker timeout references to use ttl | remove ttl updating from UI
* refactor: pass embedding context through job payload instead of global state
* lint
* add graceful shutdown for workers
* apply figma styles
* refactor embedding worker to use bree
* use existing WorkerQueue class as the management layer for jobs
* lint
* revert all reranking worker changes back to master state
Removes the reranking worker queue, rerankViaWorker/rerankInProcess
renames, and NATIVE_RERANKING_WORKER_TTL config so this branch
only contains the embedding worker job queue feature.
* remove breeManaged flag — WorkerQueue always spawns via Bree
* fix prompt embedding bug
* have embedTextInput call embedChunksInProcess
* add message field to `process.send()`
* remove nullish check and error throw
* remove bespoke graceful shutdown logix
* add spawnWorker method and asbtract redudant flows into helper methods
* remove unneeded comment
* remove recomputation of TTL value
* frontend cleanup and refactor
* wip on backend refactor
* backend overhaul
* small lint
* second pass
* add logging, update endpoint
* simple refactor
* add reporting to all embedder providers
* fix styles
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* add eslint config to server
* add break statements to switch case
* add support for browser globals and turn off empty catch blocks
* disable lines with useless try/catch wrappers
* format
* fix no-undef errors
* disbale lines violating no-unsafe-finally
* ignore syncStaticLists.mjs
* use proper null check for creatorId instead of unreachable nullish coalescing
* remove unneeded typescript eslint comment
* make no-unused-private-class-members a warning
* disable line for no-empty-objects
* add new lint script
* fix no-unused-vars violations
* make no-unsued-vars an error
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* #2317 Fetch pinned documents once per folder to reduce the number of queries.
* Reorder the lines to keeps const declarations together.
* Add some comments to functions
move pinned document fetch for folder to function
move watched documents per-folder to also function the same
remove unused function in documents model
---------
Co-authored-by: Błażej Owczarczyk <blazeyy@gmail.com>
* wip bg workers for live document sync
* Add ability to re-embed specific documents across many workspaces via background queue
bgworkser is gated behind expieremental system setting flag that needs to be explictly enabled
UI for watching/unwatching docments that are embedded.
TODO: UI to easily manage all bg tasks and see run results
TODO: UI to enable this feature and background endpoints to manage it
* create frontend views and paths
Move elements to correct experimental scope
* update migration to delete runs on removal of watched document
* Add watch support to YouTube transcripts (#1716)
* Add watch support to YouTube transcripts
refactor how sync is done for supported types
* Watch specific files in Confluence space (#1718)
Add failure-prune check for runs
* create tmp workflow modifications for beta image
* create tmp workflow modifications for beta image
* create tmp workflow modifications for beta image
* dual build
update copy of alert modals
* update job interval
* Add support for live-sync of Github files
* update copy for document sync feature
* hide Experimental features from UI
* update docs links
* [FEAT] Implement new settings menu for experimental features (#1735)
* implement new settings menu for experimental features
* remove unused context save bar
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* dont run job on boot
* unset workflow changes
* Add persistent encryption service
Relay key to collector so persistent encryption can be used
Encrypt any private data in chunkSources used for replay during resync jobs
* update jsDOC
* Linting and organization
* update modal copy for feature
---------
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
* Enable ability to do full-text query on documents
Show alert modal on first pin for client
Add ability to use pins in stream/chat/embed
* typo and copy update
* simplify spread of context and sources
* WIP event logging - new table for events and new settings view for viewing
* WIP add logging
* UI for log rows
* rename files to Logging to prevent getting gitignore
* add metadata for all logging events and colored badges in logs page
* remove unneeded comment
* cleanup namespace for logging
* clean up backend calls
* update logging to show to => from settings changes
* add logging for invitations, created, deleted, and accepted
* add logging for user created, updated, suspended, or removed
* add logging for workspace deleted
* add logging for chat logs exported
* add logging for API keys, LLM, embedder, vector db, embed chat, and reset button
* modify event logs
* update to event log types
* simplify rendering of event badges
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
Limit is due to POST body max size. Sufficiently large requests will abort automatically
We should report that error back on the frontend during embedding
Update vectordb providers to return on failed
* WIP converted all sqlite models into prisma calls
* modify db setup and fix ApiKey model calls in admin.js
* renaming function params to be consistent
* converted adminEndpoints to utilize prisma orm
* converted chatEndpoints to utilize prisma orm
* converted inviteEndpoints to utilize prisma orm
* converted systemEndpoints to utilize prisma orm
* converted workspaceEndpoints to utilize prisma orm
* converting sql queries to prisma calls
* fixed default param bug for orderBy and limit
* fixed typo for workspace chats
* fixed order of deletion to account for sql relations
* fix invite CRUD and workspace management CRUD
* fixed CRUD for api keys
* created prisma setup scripts/docs for understanding how to use prisma
* prisma dependency change
* removing unneeded console.logs
* removing unneeded sql escape function
* linting and creating migration script
* migration from depreciated sqlite script update
* removing unneeded migrations in prisma folder
* create backup of old sqlite db and use transactions to ensure all operations complete successfully
* adding migrations to gitignore
* updated PRISMA.md docs for info on how to use sqlite migration script
* comment changes
* adding back migrations folder to repo
* Reviewing SQL and prisma integraiton on fresh repo
* update inline key replacement
* ensure migration script executes and maps foreign_keys regardless of db ordering
* run migration endpoint
* support new prisma backend
* bump version
* change migration call
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* multi user wip
* WIP MUM features
* invitation mgmt
* suspend or unsuspend users
* workspace mangement
* manage chats
* manage chats
* add Support for admin system settings for users to delete workspaces and limit chats per user
* fix issue ith system var
update app to lazy load invite page
* cleanup and bug fixes
* wrong method
* update readme
* update readme
* update readme
* bump version to 0.1.0
* 1. Define LLM Temperature as a workspace setting
2. Implement rudimentry table migration code for both new and existing repos to bring tables up to date
3. Trigger for workspace on update to update timestamp
4. Always fallback temp to 0.7
5. Extract WorkspaceModal into Tabbed content
6. Remove workspace name UNIQUE constraint (cannot be migrated :()
7. Add slug +seed when existing slug is already take
8. Seperate name from slug so display names can be changed
* remove blocking test return
* Updates for Linux for frontend/server
* frontend/server docker
* updated Dockerfile for deps related to node vectordb
* updates for collector in docker
* docker deps for ODT processing
* ignore another collector dir
* storage mount improvements; run as UID
* fix pypandoc version typo
* permissions fixes
