* implement native embedder job queue
* persist embedding progress across renders
* add development worker timeouts
* change to static method
* native reranker
* remove useless return
* lint
* simplify
* make embedding worker timeout value configurable by admin
* add event emission for missing data
* lint
* remove onProgress callback argument
* make rerank to rerankDirect
* persists progress state across app reloads
* remove chunk level progress reporting
* remove unuse dvariable
* make NATIVE_RERANKING_WORKER_TIMEOUT user configurable
* remove dead code
* scope embedding progress per-user and clear stale state on SSE reconnect
* lint
* revert vector databases and embedding engines to call their original methods
* simplify rerank
* simplify progress fetching by removing updateProgressFromApi
* remove duplicate jsdoc
* replace sessionStorage persistence with server-side history replay for embedding progress
* fix old comment
* fix: ignore premature SSE all_complete when embedding hasn't started yet
The SSE connection opens before the embedding API call fires, so the
server sees no buffered history and immediately sends all_complete.
Firefox dispatches this eagerly enough that it closes the EventSource
before real progress events arrive, causing the progress UI to clear
and fall back to the loading spinner. Chrome's EventSource timing
masks the race.
Track slugs where startEmbedding was called but no real progress event
has arrived yet via awaitingProgressRef. Ignore the first all_complete
for those slugs and keep the connection open for the real events.
* reduce duplication with progress emissions
* remove dead code
* refactor: streamline embedding progress handling
Removed unnecessary tracking of slugs for premature all_complete events in the EmbeddingProgressProvider. Updated the server-side logic to avoid sending all_complete when no embedding is in progress, allowing the connection to remain open for real events. Adjusted the embedding initiation flow to ensure the server processes the job before the SSE connection opens, improving the reliability of progress updates.
* fix stale comment
* remove unused function
* fix event emissions for document creation failure
* refactor: move Reranking Worker Idle Timeout input to LanceDBOptions component
Extracted the Reranking Worker Idle Timeout input from GeneralEmbeddingPreference and integrated it into the LanceDBOptions component. This change enhances modularity and maintains a cleaner structure for the settings interface.
* lint
* remove unused hadHistory vars
* refactor workspace directory by hoisting component and converting into functions
* moved EmbeddingProgressProvider to wrap Document Manager Modal
* refactor embed progress SSE connection to use fetchEventSource instead of native EventSource API.
* refactor message handlng into a function and reduce duplication
* refactor: utilize writeResponseChunk for event emissions in document embedding progress SSE
* refactor: explicit in-proc embedding and rerank methods that are called by workers instead of process.send checks
* Abstract EmbeddingProgressBus and Worker Queue into modules
* remove error and toast messages on embed process result
* use safeJsonParse
* add chunk-level progress events with per-document progress bar in UI
* remove unused parameter
* rename all worker timeout references to use ttl | remove ttl updating from UI
* refactor: pass embedding context through job payload instead of global state
* lint
* add graceful shutdown for workers
* apply figma styles
* refactor embedding worker to use bree
* use existing WorkerQueue class as the management layer for jobs
* lint
* revert all reranking worker changes back to master state
Removes the reranking worker queue, rerankViaWorker/rerankInProcess
renames, and NATIVE_RERANKING_WORKER_TTL config so this branch
only contains the embedding worker job queue feature.
* remove breeManaged flag — WorkerQueue always spawns via Bree
* fix prompt embedding bug
* have embedTextInput call embedChunksInProcess
* add message field to `process.send()`
* remove nullish check and error throw
* remove bespoke graceful shutdown logix
* add spawnWorker method and asbtract redudant flows into helper methods
* remove unneeded comment
* remove recomputation of TTL value
* frontend cleanup and refactor
* wip on backend refactor
* backend overhaul
* small lint
* second pass
* add logging, update endpoint
* simple refactor
* add reporting to all embedder providers
* fix styles
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Add className property to various LLM and embedder classes to fix logging bug after minification
* Fix bug with this.log method by applying the missing private field symbol
* Add ENV to configure api request delay for generic open ai embedding engine
* yarn lint formatting
* refactor
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* exposes `maxConcurrentChunks` parameter for the generic openai embedder through configuration. This allows setting a batch size for endpoints which don't support the default of 500
* Update new field to new UI
make getting to ensure proper type and format
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
