merlyn/server/utils/helpers
Timothy Carambat 2c19dd09ed
Native Embedder model selection (incl: Multilingual support) (#3835)
* WIP on embedder selection
TODO: apply splitting and query prefixes (if applicable)

* wip on upsert

* Support base model
support nomic-text-embed-v1
support multilingual-e5-small
Add prefixing for both embedding and query for RAG tasks
Add chunking prefix to all vector dbs to apply prefix when possible
Show dropdown and auto-pull on new selection

* norm translations

* move supported models to constants
handle null seelction or invalid selection on dropdown
update comments

* dev

* patch text splitter maximums for now

* normalize translations

* add tests for splitter functionality

* normalize

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
2025-07-22 10:07:20 -07:00
..
admin patch admin pwd update 2024-02-06 14:39:56 -08:00
chat Handle BigInt in message response (#4110) 2025-07-10 12:33:34 -07:00
camelcase.js Add support for Weaviate VectorDB (#181) 2023-08-08 18:02:30 -07:00
customModels.js Native Embedder model selection (incl: Multilingual support) (#3835) 2025-07-22 10:07:20 -07:00
index.js Moonshot AI LLM & agent provider (#4178) 2025-07-22 09:56:51 -07:00
portAvailabilityChecker.js [FEAT] Check port access in docker before showing a default error (#961) 2024-04-02 10:34:50 -07:00
search.js Add ability to search workspace and threads (#4120) 2025-07-10 16:42:10 -07:00
tiktoken.js Add tokenizer improvments via Singleton class and estimation (#3072) 2025-01-30 17:55:03 -08:00
updateENV.js Native Embedder model selection (incl: Multilingual support) (#3835) 2025-07-22 10:07:20 -07:00