merlyn/server/utils/EmbeddingEngines
Timothy Carambat 2c19dd09ed
Native Embedder model selection (incl: Multilingual support) (#3835)
* WIP on embedder selection
TODO: apply splitting and query prefixes (if applicable)

* wip on upsert

* Support base model
support nomic-text-embed-v1
support multilingual-e5-small
Add prefixing for both embedding and query for RAG tasks
Add chunking prefix to all vector dbs to apply prefix when possible
Show dropdown and auto-pull on new selection

* norm translations

* move supported models to constants
handle null seelction or invalid selection on dropdown
update comments

* dev

* patch text splitter maximums for now

* normalize translations

* add tests for splitter functionality

* normalize

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
2025-07-22 10:07:20 -07:00
..
azureOpenAi Replace @azure/openai with openai, and update openai to version 4.95.1 (#3691) 2025-04-29 11:21:39 -07:00
cohere [FEAT] Cohere LLM and embedder support (#1233) 2024-05-02 10:35:50 -07:00
gemini Add gemini-embedding-exp-03-07 model support (#3767) 2025-05-05 09:37:01 -07:00
genericOpenAi Respect Max Chunk Len set - Generic OpenAI Embedder connection (#3838) 2025-05-16 11:18:24 -07:00
liteLLM [FEAT] Add LiteLLM embedding provider support (#1579) 2024-06-06 12:43:34 -07:00
lmstudio LMStudio 0 value embeddings patch (#4084) 2025-07-01 18:05:58 -07:00
localAi Bump openai package to latest (#1234) 2024-04-30 12:33:42 -07:00
mistral linting 2024-11-21 15:49:59 -08:00
native Native Embedder model selection (incl: Multilingual support) (#3835) 2025-07-22 10:07:20 -07:00
ollama Enable num_ctx to match defined chunk length in ollama embedder (#3129) 2025-02-05 15:46:39 -08:00
openAi Replace @azure/openai with openai, and update openai to version 4.95.1 (#3691) 2025-04-29 11:21:39 -07:00
voyageAi feat: add support for voyage-3-large and voyage-code-3 embedding models (#2955) 2025-01-08 17:21:30 -08:00