41495cdabe
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Marcello Fitton
|
41495cdabe
|
feat: Scheduled Jobs (#5322)
* initialize * expand tool result text limit | add syntax highlighting and json formatting to tool result rendering * fix onError jsdoc * lint * fix unread icon * route protection * improve form handling for NewJobModal * safeJsonParse * remove unneeded comments * remove trycatch * add truncateText helper * add explicit fallback value tos safeJsonParse * add shared cron constant and helpers * reduce frontend indirection * use isLight to compute syntax highlighting theme * remove dead code * remove forJob and make job limit to 50 * create recomputeNextRunAt helper method * add comment about nextRunAt recomputation * add job queue and concurrency control to scheduled jobs * use p-queue * change default max concurrent value to 1 * add comment explaining internal scheduling system * add recomputeNextRunAt on boot * add generated documents to run details * Modify toolsOverride functionality where no tools selected means no tools are given to the agent add a select all/deselect all toggle button for easily selecting all tools in the cerate job form * create usePolling hook * add polling to scheduled jobs and scheduled job runs pages * add cron generation feature in job form * remove cron generation feature | add cron builder feature | add max active scheduled jobs limit * set MAX_ACTIVE to null * replace hour and minute input fields with input with type time * simplify * organize components * move components to bottom of page component * change Generated Documents to Generated Files * add i18n to cronstrue * add i18n * add type="button" to button elements * refactor fileSource retrieval logic * one scheduled job run can have status "running" * add protection of file retrieveal from scheduled job in multiuser mode * fix comments * make job status default to queued * add queued status * fix bug with result trace rendering * store timeout ref and clearTimeout once race settles * remove unneeded handlerPromise tracking * move imports to top level * refactor hardcoded paths to path resolve functions * implement new job form design * simplify * fix button styles * fix runJob bug * implement styles for scheduled jobs page * apply dark mode figma styles * delete unused translation key * implement light mode for new new job modal, run history, and run details * lint * fix light mode scroll bar in tool call card * adjust table header contrast * fix type in subtitle * kill workers when job is in-flight before deleting job * add border-none to buttons * change locale time to iso string * import BackgroundService module level | instatiate backgroundService singltone once and reuse across handlers * add p-queue, @breejs/later and cron-validate as core deps * parse cron expression to a builder state once * add theme to day buttons in cron builder * fix stale tools selection caption * flip popover when popover clips screen height * make ScheduleJob.trigger() await the run insertion | disable run now button if job is in flight * regen table * refactor generated file card * refactor frontend * remove logs * major refactor for tool picking, fix bree/later bug * combine action endpoints, move contine to method * fix unoptimized query with include + take + order * fix dangerous use, refactor job to utils * add copy content to text response * improve notification system subscription for browser * remove unused translations * prevent gen-file cleanup job from deleting active job file generated references * rich text copy * Scheduled Jobs: Translations (#5482) * add locales for scheduled jobs * i18n --------- Co-authored-by: Timothy Carambat <rambat1010@gmail.com> * add config flag with UI notice * update README * telemetry datapoints * Always use UTC on backend, convert to local in frontend * fix tz render * Add job killing * cleanup thinking text in job notifications and break out reasoning in response text. Also hide zero metrics since that is useless * Port generatedFile schema to the normalized workspace chat `outputs` file format so porting to thread is simple and implem between chats <> jobs is 1:1 * what the fuck * compiled bug * fixed thinking oddity in complied frontend * supress multi-toast * fix duration call * Revert "fix duration call" This reverts commit 0491bc71f4223e65ea4046561b15b268fefb8da2. * revert and reapply fix --------- Co-authored-by: Timothy Carambat <rambat1010@gmail.com> |
||
|
Marcello Fitton
|
42a41201a8
|
feat: Document Embedding Status Events | Refactor Document Embedding to Job Queue and Forked Process (#5254)
* implement native embedder job queue * persist embedding progress across renders * add development worker timeouts * change to static method * native reranker * remove useless return * lint * simplify * make embedding worker timeout value configurable by admin * add event emission for missing data * lint * remove onProgress callback argument * make rerank to rerankDirect * persists progress state across app reloads * remove chunk level progress reporting * remove unuse dvariable * make NATIVE_RERANKING_WORKER_TIMEOUT user configurable * remove dead code * scope embedding progress per-user and clear stale state on SSE reconnect * lint * revert vector databases and embedding engines to call their original methods * simplify rerank * simplify progress fetching by removing updateProgressFromApi * remove duplicate jsdoc * replace sessionStorage persistence with server-side history replay for embedding progress * fix old comment * fix: ignore premature SSE all_complete when embedding hasn't started yet The SSE connection opens before the embedding API call fires, so the server sees no buffered history and immediately sends all_complete. Firefox dispatches this eagerly enough that it closes the EventSource before real progress events arrive, causing the progress UI to clear and fall back to the loading spinner. Chrome's EventSource timing masks the race. Track slugs where startEmbedding was called but no real progress event has arrived yet via awaitingProgressRef. Ignore the first all_complete for those slugs and keep the connection open for the real events. * reduce duplication with progress emissions * remove dead code * refactor: streamline embedding progress handling Removed unnecessary tracking of slugs for premature all_complete events in the EmbeddingProgressProvider. Updated the server-side logic to avoid sending all_complete when no embedding is in progress, allowing the connection to remain open for real events. Adjusted the embedding initiation flow to ensure the server processes the job before the SSE connection opens, improving the reliability of progress updates. * fix stale comment * remove unused function * fix event emissions for document creation failure * refactor: move Reranking Worker Idle Timeout input to LanceDBOptions component Extracted the Reranking Worker Idle Timeout input from GeneralEmbeddingPreference and integrated it into the LanceDBOptions component. This change enhances modularity and maintains a cleaner structure for the settings interface. * lint * remove unused hadHistory vars * refactor workspace directory by hoisting component and converting into functions * moved EmbeddingProgressProvider to wrap Document Manager Modal * refactor embed progress SSE connection to use fetchEventSource instead of native EventSource API. * refactor message handlng into a function and reduce duplication * refactor: utilize writeResponseChunk for event emissions in document embedding progress SSE * refactor: explicit in-proc embedding and rerank methods that are called by workers instead of process.send checks * Abstract EmbeddingProgressBus and Worker Queue into modules * remove error and toast messages on embed process result * use safeJsonParse * add chunk-level progress events with per-document progress bar in UI * remove unused parameter * rename all worker timeout references to use ttl | remove ttl updating from UI * refactor: pass embedding context through job payload instead of global state * lint * add graceful shutdown for workers * apply figma styles * refactor embedding worker to use bree * use existing WorkerQueue class as the management layer for jobs * lint * revert all reranking worker changes back to master state Removes the reranking worker queue, rerankViaWorker/rerankInProcess renames, and NATIVE_RERANKING_WORKER_TTL config so this branch only contains the embedding worker job queue feature. * remove breeManaged flag — WorkerQueue always spawns via Bree * fix prompt embedding bug * have embedTextInput call embedChunksInProcess * add message field to `process.send()` * remove nullish check and error throw * remove bespoke graceful shutdown logix * add spawnWorker method and asbtract redudant flows into helper methods * remove unneeded comment * remove recomputation of TTL value * frontend cleanup and refactor * wip on backend refactor * backend overhaul * small lint * second pass * add logging, update endpoint * simple refactor * add reporting to all embedder providers * fix styles --------- Co-authored-by: Timothy Carambat <rambat1010@gmail.com> |
||
|
Timothy Carambat
|
7aaea7f514
|
File creation agent skills (#5280)
* Powerpoint File Creation (#5278) * wip * download card * UI for downloading * move to fs system with endpoint to pull files * refactor UI * final-pass * remove save-file-browser skill and refactor * remove fileDownload event * reset * reset file * reset timeout * persist toggle * Txt creation (#5279) * wip * download card * UI for downloading * move to fs system with endpoint to pull files * refactor UI * final-pass * remove save-file-browser skill and refactor * remove fileDownload event * reset * reset file * reset timeout * wip * persist toggle * add arbitrary text creation file * Add PDF document generation with markdown formatting (#5283) add support for branding in bottom right corner refactor core utils and frontend rendering * Xlsx document creation (#5284) add Excel doc & sheet creation * Basic docx creation (#5285) * Basic docx creation * add test theme support + styling and title pages * simplify skill selection * handle TG attachments * send documents over tg * lazy import * pin deps * fix lock * i18n for file creation (#5286) i18n for file-creation connect #5280 * theme overhaul * Add PPTX subagent for better results * forgot files * Add PPTX subagent for better results (#5287) * Add PPTX subagent for better results * forgot files * make sub-agent use proper tool calling if it can and better UI hints |
||
|
Sean Hatfield
|
192ca411f2
|
Telegram bot connector (#5190)
* wip telegram bot connector * encrypt bot token, reorg telegram bot modules, secure pairing codes * offload telegram chat to background worker, add @agent support with chart png rendering, reconnect ui * refactor telegram bot settings page into subcomponents * response.locals for mum, telemetry for connecting to telegram * simplify telegram command registration * improve telegram bot ux: rework switch/history/resume commands * add voice, photo, and TTS support to telegram bot with long message handling * lint * rename external_connectors to external_communication_connectors, add voice response mode, persist chat workspace/thread selection * lint * fix telegram bot connect/disconnect bugs, kill telegram bot on multiuser mode enable * add english translations * fix qr code in light mode * repatch migration * WIP checkpoint * pipeline overhaul for using response obj * format functions * fix comment block * remove conditional dumpENV + lint * remove .end() from sendStatus calls * patch broken streaming where streaming only first chunk * refactor * use Ephemeral handler now * show metrics and citations in real GUI * bugfixes * prevent MuM persistence, UI cleanup, styling for status * add new workspace flow in UI Add thread chat count fix 69 byte payload callback limit bug * handle pagination for workspaces, threads, and models * modularize commands and navigation * add /proof support for citation recall * handle backlog message spam * support abort of response streams * code cleanup * spam prevention * fix translations, update voice typing indicator, fix token bug * frontend refactor, update tips on /status and voice response improvements * collapse agent though blocks * support images * Fix mime issues with audio from other devices * fix config issue post server stop * persist image on agentic chats * 5189 i18n (#5245) * i18n translations connect #5189 * prune translations * fix errors * fix translation gaps --------- Co-authored-by: Timothy Carambat <rambat1010@gmail.com> |
||
|
Timothy Carambat
|
0fb33736da
|
Workspace Chat with documents overhaul (#4261)
* Create parse endpoint in collector (#4212) * create parse endpoint in collector * revert cleanup temp util call * lint * remove unused cleanupTempDocuments function * revert slug change minor change for destinations --------- Co-authored-by: timothycarambat <rambat1010@gmail.com> * Add parsed files table and parse server endpoints (#4222) * add workspace_parsed_files table + parse endpoints/models * remove dev api parse endpoint * remove unneeded imports * iterate over all files + remove unneeded update function + update telemetry debounce * Upload UI/UX context window check + frontend alert (#4230) * prompt user to embed if exceeds prompt window + handle embed + handle cancel * add tokenCountEstimate to workspace_parsed_files + optimizations * use util for path locations + use safeJsonParse * add modal for user decision on overflow of context window * lint * dynamic fetching of provider/model combo + inject parsed documents * remove unneeded comments * popup ui for attaching/removing files + warning to embed + wip fetching states on update * remove prop drilling, fetch files/limits directly in attach files popup * rework ux of FE + BE optimizations * fix ux of FE + BE optimizations * Implement bidirectional sync for parsed file states linting small changes and comments * move parse support to another endpoint file simplify calls and loading of records * button borders * enable default users to upload parsed files but NOT embed * delete cascade on user/workspace/thread deletion to remove parsedFileRecord * enable bgworker with "always" jobs and optional document sync jobs orphan document job: Will find any broken reference files to prevent overpollution of the storage folder. This will run 10s after boot and every 12hr after * change run timeout for orphan job to 1m to allow settling before spawning a worker * linting and cleanup pr --------- Co-authored-by: Timothy Carambat <rambat1010@gmail.com> * dev build * fix tooltip hiding during embedding overflow files * prevent crash log from ERRNO on parse files * unused import * update docs link * Migrate parsed-files to GET endpoint patch logic for grabbing models names from utils better handling for undetermined context windows (null instead of Pos_INIFI) UI placeholder for null context windows * patch URL --------- Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com> |
||
|
Timothy Carambat
|
29c9eeaa5c
|
Add winston logging for production (#1811)
|
||
|
Timothy Carambat
|
cb6b3340f0
|
Update bgworker to use fork instead of worker_thread (#1808)
|
||
|
Timothy Carambat
|
dc4ad6b5a9
|
[BETA] Live document sync (#1719)
* wip bg workers for live document sync * Add ability to re-embed specific documents across many workspaces via background queue bgworkser is gated behind expieremental system setting flag that needs to be explictly enabled UI for watching/unwatching docments that are embedded. TODO: UI to easily manage all bg tasks and see run results TODO: UI to enable this feature and background endpoints to manage it * create frontend views and paths Move elements to correct experimental scope * update migration to delete runs on removal of watched document * Add watch support to YouTube transcripts (#1716) * Add watch support to YouTube transcripts refactor how sync is done for supported types * Watch specific files in Confluence space (#1718) Add failure-prune check for runs * create tmp workflow modifications for beta image * create tmp workflow modifications for beta image * create tmp workflow modifications for beta image * dual build update copy of alert modals * update job interval * Add support for live-sync of Github files * update copy for document sync feature * hide Experimental features from UI * update docs links * [FEAT] Implement new settings menu for experimental features (#1735) * implement new settings menu for experimental features * remove unused context save bar --------- Co-authored-by: timothycarambat <rambat1010@gmail.com> * dont run job on boot * unset workflow changes * Add persistent encryption service Relay key to collector so persistent encryption can be used Encrypt any private data in chunkSources used for replay during resync jobs * update jsDOC * Linting and organization * update modal copy for feature --------- Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com> |