* Update common.js
Improving the Romanian translation and filling in missing elements.
* linting
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Added exa-search case to the search provider switch in web-browsing.js
* Added ExaSearchOptions component for API key input
* update
* Patch missing image crashing UI
Fix issue where ENV key did not exist or was saved on click
Update copy for provider
Add Docs for ENV keys for manual placements
update systemssettings for returning key saved to UI
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Create parse endpoint in collector (#4212)
* create parse endpoint in collector
* revert cleanup temp util call
* lint
* remove unused cleanupTempDocuments function
* revert slug change
minor change for destinations
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Add parsed files table and parse server endpoints (#4222)
* add workspace_parsed_files table + parse endpoints/models
* remove dev api parse endpoint
* remove unneeded imports
* iterate over all files + remove unneeded update function + update telemetry debounce
* Upload UI/UX context window check + frontend alert (#4230)
* prompt user to embed if exceeds prompt window + handle embed + handle cancel
* add tokenCountEstimate to workspace_parsed_files + optimizations
* use util for path locations + use safeJsonParse
* add modal for user decision on overflow of context window
* lint
* dynamic fetching of provider/model combo + inject parsed documents
* remove unneeded comments
* popup ui for attaching/removing files + warning to embed + wip fetching states on update
* remove prop drilling, fetch files/limits directly in attach files popup
* rework ux of FE + BE optimizations
* fix ux of FE + BE optimizations
* Implement bidirectional sync for parsed file states
linting
small changes and comments
* move parse support to another endpoint file
simplify calls and loading of records
* button borders
* enable default users to upload parsed files but NOT embed
* delete cascade on user/workspace/thread deletion to remove parsedFileRecord
* enable bgworker with "always" jobs and optional document sync jobs
orphan document job: Will find any broken reference files to prevent overpollution of the storage folder. This will run 10s after boot and every 12hr after
* change run timeout for orphan job to 1m to allow settling before spawning a worker
* linting and cleanup pr
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* dev build
* fix tooltip hiding during embedding overflow files
* prevent crash log from ERRNO on parse files
* unused import
* update docs link
* Migrate parsed-files to GET endpoint
patch logic for grabbing models names from utils
better handling for undetermined context windows (null instead of Pos_INIFI)
UI placeholder for null context windows
* patch URL
---------
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
* WIP on mobile connections
todo: register devices
todo: data sync or connection
* improve connection flow and registration
add streaming from service
TODO: user scoping
* dev build mobile support
* fix path
* handle relative URLs
* handle localhost access in product
* add device de-register
* sync styles
* move UI to be out of the normal path since beta only
* Add user scoping to mobile connection requests
Remigrate DB for user associations
Implement temp token registration to prevent unauthorized device registration requests
cleanup middlewares
* WIP on embedder selection
TODO: apply splitting and query prefixes (if applicable)
* wip on upsert
* Support base model
support nomic-text-embed-v1
support multilingual-e5-small
Add prefixing for both embedding and query for RAG tasks
Add chunking prefix to all vector dbs to apply prefix when possible
Show dropdown and auto-pull on new selection
* norm translations
* move supported models to constants
handle null seelction or invalid selection on dropdown
update comments
* dev
* patch text splitter maximums for now
* normalize translations
* add tests for splitter functionality
* normalize
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* docs(zh-tw, zh-cn): Complete missing Traditional and Simplified Chinese translations
This commit aims to complete all previously 'null' Traditional Chinese and Simplified Chinese translation keys in the project.
**Traditional Chinese (zh-tw):**
- Completed all 'null' descriptions in `/frontend/src/locales/zh_TW/common.js`.
**Simplified Chinese (zh-cn):**
- Completed all 'null' descriptions in `/frontend/src/locales/zh/common.js`.
---
此提交旨在補齊專案中所有原為 'null' 的繁體中文和簡體中文翻譯鍵值。
**繁體中文翻譯 (Traditional Chinese - zh-tw):**
- 完善了 `/frontend/src/locales/zh_TW/common.js` 裡所有 'null' 的說明。
**簡體中文翻譯 (Simplified Chinese - zh-cn):**
- 完善了 `/frontend/src/locales/zh/common.js` 裡所有 'null' 的說明。
* linting
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* configurable message limit for embed widget
* remove console log
* make field optional + add fallback
* rework validation logic
* lint
* remove field specific guard, it cannot be lte 0 like all other fields
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* feat: Add Korean (ko) locale translations
- Add comprehensive Korean translations for UI elements
- Include main page features, workspace settings, and chat components
- Support Korean localization for common user interactions
* updates, lint
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add hint for chats in query mode
* add and normalize translations
* improve semantic HTML for linking
* use existing util
* linting
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Enable UI/UX for model swapping in chat window
* forgot component
* patch useGetProviders hook to set loading on change of provider
* dev build
* normalize translations
* patch how model default is provided
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* implement importing of agent flows from community hub
* auto enable flow on import
* remove unused blocks for docker
prevent importing or saving of agent flows that have unsupported blocks for version or platform
* dev build
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* wip: create direct output switch on last block and send response to ui
* lint
* Return flow on direct output enabled
prevent new blocks below direct output block
Update executor/aibitat to handle skipping of handler outputs
* dev build
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* wip create skeleton for new embed chat ui/ux
* update ui for embed chats
* lint
* update sidebar/paths
* remove old embed pages
* patch broken link
* add created timestamp to differentiate embeds
update translation key to lowercase
add created at translation key
* update text colors
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Add keyboard shortcuts help feature
* Minor lint fixes
* Enhance keyboard shortcuts feature by updating help shortcut and adding keyboard shortcuts button in Quick Links. Include new translation for keyboard shortcuts in locale.
* Added documentation of keyboard shortcuts
* refactor keyboard shortcuts to not render on every page (like login)
limit shortcuts to admins
update locales
remove keyboard shortcut button - move to resources
remove readme entry
* move translation key
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* new ui for citations
* move onclick to parent
* lint
* change div to button
* update citations UI and parsing to inherit user text size and design specs
* new render animation for citations
fix bug with citation on score on modal
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* corrected toggle server toast text
* reordered code for readability
* reordered code for readability
* ran yarn lint
* reordered code for readability
* ran yarn lint
* PGVector support for vector db storage
* forgot files
* comments
* dev build
* Add ENV connection and table schema validations for vector table
add .reset call to drop embedding table when changing the AnythingLLM embedder
update instrutions
Add preCheck error reporting in UpdateENV
add timeout to pg connection
* update setup
* update README
* update doc
* feat: implement iam role auth for bedrock
* fix: make client refreshes properly when switching between iam_user and iam_role
* checkout agent flow
* fix aiprovider for bedrock in agent use
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Fixed two primary issues discovered while using AWS Bedrock with Anthropic Claude Sonnet models:
- Context Window defaults to 8192 maximum, which isn't correct
- Multimodal stopped working when removing langchain, which was transparently handling image_url to a format sonnet expects.
* Ran `yarn lint`
* Updated .env.example to have aws bedrock examples too
* Refactor for readability
move utils for AWS specific functionality to subfile
add token output max to ENV so setting persits
---------
Co-authored-by: Tristan Stahnke <tristan.stahnke+gpsec@guidepointsecurity.com>
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Update index.jsx
"Any existing instances of Bing Search APIs, with your LLM will be decommissioned completely, and the product will no longer be available for usage or new customer signups."
https://azure.microsoft.com/en-us/updates?id=483570
I checked by creating azure account myself to see it is not available. The alternative "Grounding with Bing Search" product only has one pricing available--Grounding with Bing Search ($35 per 1K transactions)
* Update index.jsx
"Any existing instances of Bing Search APIs, with your LLM will be decommissioned completely, and the product will no longer be available for usage or new customer signups."
https://azure.microsoft.com/en-us/updates?id=483570
I checked by creating azure account myself to see it is not available. The alternative "Grounding with Bing Search" product only has one pricing available--Grounding with Bing Search ($35 per 1K transactions)
* update copy
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* auto STT submit control
* simplified STT auto and TTS autoplay
* add chatId to piper elements
* change defaults
* add chat customization page for enabling/disabling new features + options on account modal
* add translation placeholders
* show keybind command
* dev build
* Account modal ui prevent page reload + show Tooltip
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* Translate main page landing
* normalize all languages
* Add a few translations, zh,tw,ja,es,de
* fix issues where translation lang change would not effect constants until full reload instead of on update
* dev
* fix noWorkspaceError key in translation usage
* migrate hash navigation to use query params for home page
* move from url param to query param
* uncheck file
* update paths to handle search options
update action name that is used in query param
update action handlers
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Update Azure AI options and model map with new model configurations
* linting
---------
Co-authored-by: Shinya Suzuki <shinya.s.825@gmail.com>
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* wip getting started checklist page
* lint
* add new ui skeleton for new home page/checklist
* make legacy home page appearance setting
* dynamic checklist rendering/close checklist
* make home page functional + update cta buttons in ws settings
* lint
* remove unneeded routes
* wip fixing checklist items (create ws and embed doc broken) + news section updates
* lint
* FINALLY fix all functionality & remove hook to simplify logic
* lint
* hide/show options based on user role/mum enabled
* add hover states and redo checklist ui
* remove welcome checklist page
* add validation to task completion
* polish behavior of checklist/fix roles on checklist items
* lint
* light mode/use tailwind color classes
* remove tutorials link
* Modify how legacy page works
* small UI updates
* remove unused paths
cleanup explore features
* revert save button changes
* conditionally render legacy default chat page when in multiuser
* remove role checks in checklist
* remove role checks in quick links
* remove unused hook
* dark mode new home page checklist ui updates
* Add news logging to repo for record keeping
update new module to pull from CDN
* simplify landing markup
* light mode styles
* remove border in light mode from merge conflict
* Update ignores
* slide up dismiss
* prevent checklist popin
* confetti?
* fix url hash on navigate
* watch for event changes for updating checklist
* useMemo and callback memory optimization
* move handlers to constants via params fwd
* dev
* update github text
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* WIP MCP full compatibility layer
* implement MCP agent function wrapping and invocation methods
* Add `uvx` to docker bin for MCP executions
* dev build
* prune removed data
* Wrap MCP servers to lazy load items to not block the UI
Mobile bug fixes
* arm64 test build
* reset dev builder
* remove unused prop
* added reordering of workspaces
* smooth ui updates on dnd + ui bug fixes
* pin react-beautiful-dnd version
* Move to localstorage solution to support single and multi-user
---------
Co-authored-by: Ron Metzger <ron.metzger@korn-recycling.de>
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* changed language support for following windows: chat, data-upload and account settings
* normalize translations
* linting, normalization with DE
---------
Co-authored-by: Ron Metzger <ron.metzger@korn-recycling.de>
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* feat: add new model provider PPIO
* fix: fix ppio model fetching
* fix: code lint
* reorder LLM
update interface for streaming and chats to use valid keys
linting
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Windows development environment variables support
* moved cross-env to dev dependencies
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* add bio to users table
* lint
* add bio field to edit user admin page
* fix bio saving on new user
* simplify updating localstorage user
* linting
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* feat: support user select message direction
* feat: optimizing the code
* feat: lint code
* fix: prevent localstorage read on every message component render
ui: refactor alignment UI selector for dark and light mode with simple styling
* docs: update jsdoc comment for hook
fix: apply chat alignment to homepage chat
* fix mobile styles of message chat alignment preference
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* add claude 3-7 sonnet
* made all the changes everywhere
* add 3-7-sonnet-latest model
* lint
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* feat(dataconnectors): support confluence personal access token
* fix: change select option
* linting
change name on accesstype field
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Add `querySelectorAll` capability to web-scraping block
* patches and fallbacks
* fix styles of text in web scraping block
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* chore: rename Github to GitHub
Signed-off-by: Adam Setch <adam.setch@outlook.com>
* chore: rename Github to GitHub
Signed-off-by: Adam Setch <adam.setch@outlook.com>
* Undo some code changes for references
---------
Signed-off-by: Adam Setch <adam.setch@outlook.com>
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* fix stop generation button ui
* keep action color theme and match rest of the UI
rescale button to be consistent with UI
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* feat: add support for voyage-3-large and voyage-code-3 embedding models
- Add voyage-3-large and voyage-code-3 to VoyageAiOptions dropdown
- Update getMaxEmbeddingLength to support 32k context for new models
- Update .env.example with new model options
* unset env example
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Reranker WIP
* add cacheing and singleton loading
* Add field to workspaces for vectorSearchMode
Add UI for lancedb to change mode
update all search endpoints to pass in reranker prop if provider can use it
* update hint text
* When reranking, swap score to rerank score
* update optchain
* Add support for Google Generative AI (Gemini) embedder
* Add missing example in docker
Fix UI key elements in options
Add Gemini to data handling section
Patch issues with chunk handling during embedding
* remove dupe in env
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* fix light mode table colors
* fix table styles from being overwritten in other sections that are not chat messages
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Add support for gemini authenticated models endpoint
add customModels entry
add un-authed fallback to default listing
separate models by expiermental status
resolves#2866
* add back improved logic for apiVersion decision making
* wip remove all docs clear vector db on embedder/vector db change
* purge all cached docs and remove docs from workspaces on vectordb/embedder change
* lint
* remove unneeded console log
* remove reset vector stores endpoint and move to server side updateENV with postUpdate check
* reset embed module
* remove unused import
* simplify deletion process
rescoped document deletion to be more general for speed, everything needs to be reset anyway
fixed issue where unembedded docs not in any workspaces, but cached, were not removed
* add back missing readme file
update warning text modals
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* redirect to default thread on thread deletion
* reset submodule
* handle redirect on bulk delete thread
* redirect on bulk thread delete only if deleted thread is selected
* redirect only on single delete of active thread
* lint
* patch threadSlug for clarity
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* add empty states to your account hub page + disconnect button for api key
* lint
* lint
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* wip hub connection page fe + backend
* lint
* implement backend for local hub items + placeholder endpoints to fetch hub app data
* fix hebrew translations
* revamp community integration flow
* change sidebar
* Auto import if id in URL param
remove preview in card screen and instead go to import flow
* get user's items + team items from hub + ui improvements to hub settings
* lint
* fix merge conflict
* refresh hook for community items
* add fallback for user items
* Disable bundle items by default on all instances
* remove translations (will complete later)
* loading skeleton
* Make community hub endpoints admin only
show visibility on items
combine import/apply for items to they are event logged for review
* improve middleware and import flow
* community hub ui updates
* Adjust importing process
* community hub to dev
* Add webscraper preload into imported plugins
* add runtime property to plugins
* Fix button status on imported skill change
show alert on skill change
Update markdown type and theme on import of agent skill
* update documentaion paths
* remove unused import
* linting
* review loading state
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* exposes `maxConcurrentChunks` parameter for the generic openai embedder through configuration. This allows setting a batch size for endpoints which don't support the default of 500
* Update new field to new UI
make getting to ensure proper type and format
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
