* Create parse endpoint in collector (#4212)
* create parse endpoint in collector
* revert cleanup temp util call
* lint
* remove unused cleanupTempDocuments function
* revert slug change
minor change for destinations
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Add parsed files table and parse server endpoints (#4222)
* add workspace_parsed_files table + parse endpoints/models
* remove dev api parse endpoint
* remove unneeded imports
* iterate over all files + remove unneeded update function + update telemetry debounce
* Upload UI/UX context window check + frontend alert (#4230)
* prompt user to embed if exceeds prompt window + handle embed + handle cancel
* add tokenCountEstimate to workspace_parsed_files + optimizations
* use util for path locations + use safeJsonParse
* add modal for user decision on overflow of context window
* lint
* dynamic fetching of provider/model combo + inject parsed documents
* remove unneeded comments
* popup ui for attaching/removing files + warning to embed + wip fetching states on update
* remove prop drilling, fetch files/limits directly in attach files popup
* rework ux of FE + BE optimizations
* fix ux of FE + BE optimizations
* Implement bidirectional sync for parsed file states
linting
small changes and comments
* move parse support to another endpoint file
simplify calls and loading of records
* button borders
* enable default users to upload parsed files but NOT embed
* delete cascade on user/workspace/thread deletion to remove parsedFileRecord
* enable bgworker with "always" jobs and optional document sync jobs
orphan document job: Will find any broken reference files to prevent overpollution of the storage folder. This will run 10s after boot and every 12hr after
* change run timeout for orphan job to 1m to allow settling before spawning a worker
* linting and cleanup pr
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* dev build
* fix tooltip hiding during embedding overflow files
* prevent crash log from ERRNO on parse files
* unused import
* update docs link
* Migrate parsed-files to GET endpoint
patch logic for grabbing models names from utils
better handling for undetermined context windows (null instead of Pos_INIFI)
UI placeholder for null context windows
* patch URL
---------
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
* WIP on mobile connections
todo: register devices
todo: data sync or connection
* improve connection flow and registration
add streaming from service
TODO: user scoping
* dev build mobile support
* fix path
* handle relative URLs
* handle localhost access in product
* add device de-register
* sync styles
* move UI to be out of the normal path since beta only
* Add user scoping to mobile connection requests
Remigrate DB for user associations
Implement temp token registration to prevent unauthorized device registration requests
cleanup middlewares
* WIP on embedder selection
TODO: apply splitting and query prefixes (if applicable)
* wip on upsert
* Support base model
support nomic-text-embed-v1
support multilingual-e5-small
Add prefixing for both embedding and query for RAG tasks
Add chunking prefix to all vector dbs to apply prefix when possible
Show dropdown and auto-pull on new selection
* norm translations
* move supported models to constants
handle null seelction or invalid selection on dropdown
update comments
* dev
* patch text splitter maximums for now
* normalize translations
* add tests for splitter functionality
* normalize
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* fix multimodal chats via openai compat api
* lint
* add tests for multi-modal content in openai compat endpoint
* refactor to normalize how openai attachments are handled
* uncheck file
* rewrite tests, autodetect mime from dataurl, and spread attachments from prompt
* lint
* revert and fix tests
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
Pull Request Type
[ ] ✨ feat
[x] 🐛 fix
[ ] ♻️ refactor
[ ] 💄 style
[ ] 🔨 chore
[ ] 📝 docs
Relevant Issues
N/A (No issue exists for this typo fix)
What is in this change?
This PR corrects a minor typo in an output message within a script.
Before:
echo "Copying to Sever Public"
After:
echo "Copying to Server Public"
Additional Information
This is a minor correction to improve the clarity and professionalism of the script's output. No functional changes are introduced.
Developer Validations
[ ] I ran yarn lint from the root of the repo & committed changes
[ ] Relevant documentation has been updated
[ ] I have tested my code functionality
[ ] Docker build succeeds locally
* docs(zh-tw, zh-cn): Complete missing Traditional and Simplified Chinese translations
This commit aims to complete all previously 'null' Traditional Chinese and Simplified Chinese translation keys in the project.
**Traditional Chinese (zh-tw):**
- Completed all 'null' descriptions in `/frontend/src/locales/zh_TW/common.js`.
**Simplified Chinese (zh-cn):**
- Completed all 'null' descriptions in `/frontend/src/locales/zh/common.js`.
---
此提交旨在補齊專案中所有原為 'null' 的繁體中文和簡體中文翻譯鍵值。
**繁體中文翻譯 (Traditional Chinese - zh-tw):**
- 完善了 `/frontend/src/locales/zh_TW/common.js` 裡所有 'null' 的說明。
**簡體中文翻譯 (Simplified Chinese - zh-cn):**
- 完善了 `/frontend/src/locales/zh/common.js` 裡所有 'null' 的說明。
* linting
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* configurable message limit for embed widget
* remove console log
* make field optional + add fallback
* rework validation logic
* lint
* remove field specific guard, it cannot be lte 0 like all other fields
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* feat: add support for custom table formatting in htmlToText conversion
* fix tables
* feat: improve plain text table formatting for AI readability
* fix options
* improve drupal wiki connector
* final fix
* adjust leading slash to match code
* linting
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
