* Enhance YouTube transcript loading to include video metadata in parsed content when parseOnly is true
* extract to function
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Refactor DefaultChat component to display a simple welcome message instead of mock chat interface onboarding text
* Add last visited workspace functionality
- Implemented localStorage management for the last visited workspace in DefaultChat and WorkspaceChat components.
- Updated UserButton to clear last visited workspace on logout.
- Refactored workspace navigation to prioritize last visited workspace if available.
* Refactor workspace fetching & last visited workspace logic into one useEffect
* Enhance loading state in DefaultChat component with skeleton UI elements for improved user experience
* minor UI spacing changes
* refactor order for guard clauses for early exit conditions
* move last known to end of getWorkspace loader
* languages for PR #4542 (#4550)
* languages for PR #4542
* Drop welcome message keys from translations
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* fix(server): correct Qdrant batching logic for large uploads
Closes#4544
* modify O(3N) process to O(N)
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* fix: remove unnecessary toLowerCase in URL validation
* test: enhance URL validation tests to preserve case sensitivity and format
* test: update URL validation tests to ensure domain normalization to lowercase while preserving path case
* small formatting
* fix filenames when downloading live URI
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Migrate gemini agents away from `Untooled`
* disable agents for gemma models as they are not supported for tool calling
* Dev build
resolve#4452 via function name prefix and then stripping within provider
* Reimplement Cohere models
- Redo LLM implementation to grab models from endpoint and pre-filter
- Migrate embedding models to also grab from remote
- Add records for easy context window lookup'
* fix comment
* auto model context limit detection for ollama llm provider
* auto model context limit detection for lmstudio llm provider
* Patch Ollama to function and sync context windows like Foundry
* normalize how model context windows are cached from endpoint service
todo: move this into global utility class with MODEL_MAP
eager load models on boot to pre-cache them
add performance model improvements into ollama agent as well as apply n_ctx
* remove debug log
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* add microsoft foundry local llm and agent providers
* minor change to fix early stop token + overloading of context window
always use user defined window _unless_ it is larger than the models real contenxt window
cache the context windows when we can from the API (0.7.*)+
Unload model forcefully on model change to prevent resource hogging
* add back token preference since some models have very large windows and can crash a machine
normalize cases
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Add capability to web scraping feature for document creation to download and parse statically hosted files
* lint
* Remove unneeded comment
* Simplified process by using key of ACCEPTED_MIMES to validate the response content type, as a result unlocked all supported files
* Add TODO comments for future implementation of asDoc.js to handle standard MS Word files in constants.js
* Return captureAs argument to be exposed by scrapeGenericUrl and passed into getPageContent | Return explicit argument of captureAs into scrapeGenericUrl in processLink fn
* Return debug log for scrapeGenericUrl
* Change conditional to a guard clause.
* Add error handling, validation, and JSDOC to getContentType helper fn
* remove unneeded comments
* Simplify URL validation by reusing module
* Rename downloadFileToHotDir to downloadURIToFile and moved up to a global module | Add URL valuidation to downloadURIToFile
* refactor
* add support for webp
remove unused imports
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP agentic tool call streaming
- OpenAI
- Anthropic
- Azure OpenAI
* WIP rest of providers EXCLUDES Bedrock and GenericOpenAI
* patch untooled complete/streaming to use chatCallback provider from provider class and not assume OpenAI client struct
example: Ollama
* modify ollama to function with its own overrides
normalize completion/stream outputs across providers/untooled
* dev build
* fix message sanization for anthropic agent streaming
* wip fix anthropic agentic streaming sanitization
* patch gemini, webgenui, generic aibitat providers + disable providers unable to test
* refactor anthropic aibitat provider for empty message and tool call formatting
* Add frontend missing prop check
update Azure for streaming support
update Gemini to streamting support on gemini-* models
generic OpenAI disable streaming
verify localAI support
verify NVIDIA Nim support
* DPAIS, remove temp from call, support streaming'
* remove 0 temp to remove possibility of bad temp error/500s/400s
* Patch condition where model is non-streamable and no tools are present or called resulting in the provider `handleFunctionCallChat` being called - which returns a string.
This would then fail in Untooled.complete since response would be a string and not the expected `response.choices?.[0]?.message`
Modified this line to handle both conditions for stream/non-streaming and tool presence or lack thereof
* Allow generic Openai to be streamable since using untooled it should work fine
honor disabled streaming for provider where that concern may apply for regular chats
* rename function and more gemini-specific function to gemini provider
* add comments for readability
.complete on azure should be non-streaming as this is the sync response
* migrate CometAPI, but disable as we cannot test
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* Fix system prompt variable color logic by removing unused variable type from switch statement and adding new types.
* Add workspace id, name and user id as default system prompt variables
* Combine user and workspace variable evaluations into a single if statment, reducing redundant code.
* minor refactor
* add systemPromptVariable expandSystemPromptVariables test cases
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Fix JSDOC for updateOrCreateCollection
* Add sanitizeForJsonb method to PGVector for safe JSONB handling
This new method recursively sanitizes values intended for JSONB storage, removing disallowed control characters and ensuring safe insertion into PostgreSQL. The method is integrated into the vector insertion process to sanitize metadata before database operations.
* Add unit tests for PGVector.sanitizeForJsonb method
This commit introduces a comprehensive test suite for the PGVector.sanitizeForJsonb method, ensuring it correctly handles various input types, including null, undefined, strings with disallowed control characters, objects, arrays, and Date objects. The tests verify that the method sanitizes inputs without mutating the original data structures.
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Add HTTP request logging middleware for development mode
- Introduced httpLogger middleware to log HTTP requests and responses.
- Enabled logging only in development mode to assist with debugging.
* Update httpLogger middleware to disable time logging by default
* Add httpLogger middleware for development mode in collector service
* Refactor httpLogger middleware to rename timeLogs parameter to enableTimestamps for clarity
* Make HTTP Logger only mount in development and environment flag is enabled.
* Update .env.example to clarify HTTP Logger configuration comments
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* fix: youtube transcript collector not work well with non en or non asr caption
* stub YT test in Github actions
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Add className property to various LLM and embedder classes to fix logging bug after minification
* Fix bug with this.log method by applying the missing private field symbol
* fix: resolve Firefox search icon overlapping placeholder text
- Increase input left padding from pl-4 to pl-9 to provide clearance
- Remove redundant placeholder:pl-4 class
- Ensures 24px spacing between search icon and text content
* Update SearchBox component to adjust padding on focus state
---------
Co-authored-by: neha <neha@posthog.com>
Co-authored-by: angelplusultra <macfittondev@gmail.com>
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
Update DPAISOptions component and constants to use new OpenAI endpoint (#4432)
- Changed placeholder in DPAISOptions component to reflect updated endpoint.
- Updated DPAIS_COMMON_URLS in constants to include '/openai' in URLs.
