michael/merlyn
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,906 Commits 1 Branch 0 Tags 51 MiB
246edc38ce
Commit Graph

946 Commits

Author SHA1 Message Date
Timothy Carambat
a9d9f9cdfd
Add custom JWT TTL (#4234) 
resolves #4224
 2025-08-01 10:39:38 -07:00
Timothy Carambat
c218a0dfe3
Mobile sync support (#4173) 
* WIP on mobile connections
todo: register devices
todo: data sync or connection

* improve connection flow and registration
add streaming from service
TODO: user scoping

* dev build mobile support

* fix path

* handle relative URLs

* handle localhost access in product

* add device de-register

* sync styles

* move UI to be out of the normal path since beta only

* Add user scoping to mobile connection requests
Remigrate DB for user associations
Implement temp token registration to prevent unauthorized device registration requests
cleanup middlewares
 2025-07-31 12:28:03 -07:00
timothycarambat
4b7932f9e1 Fix agent iam_role implied agentic tool calling 
resolves #4193
contrib: @bechir2000
 2025-07-31 09:56:08 -07:00
Neha Prasad
ea5f66978b
fix: API export-chats endpoint function import error (#4220) 
fix: resolve prepareWorkspaceChatsForExport function not found error in API export-chats endpoint
 2025-07-30 12:19:27 -07:00
Timothy Carambat
a0af24624d
Update check for validFuncCall to only check required args and for undefined options (#4214) 
* Update check for `validFuncCall` to only check required args and for undefined options

* update

* remove unused helper class func
 2025-07-30 11:59:23 -07:00
Sean Hatfield
8c7923a283
Fix app version metrics endpoint (#4209) 
fix app version metrics endpoint
 2025-07-28 15:37:06 -07:00
Roman Wu
78cbb06c57
Modify the PostgreSQL SQL connector to support querying tables from schemas other than the default public schema. (#4202) 
* Add PostgreSQL schema support to SQL connector

- Add schema configuration option to PostgreSQL connector
- Update SQL queries to filter by specified schema
- Add schema input field to frontend connection modal (PostgreSQL only)
- Default to 'public' schema when no custom schema specified
- Add getQualifiedTableName() method for schema.table format

* Fix code formatting after linting

* Update Postgresql.js

* add back removal of ending curly brace

* Update Postgresql.js

* add back removal of ending curly brace (again?)

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-07-28 08:43:15 -07:00
Timothy Carambat
2c19dd09ed
Native Embedder model selection (incl: Multilingual support) (#3835) 
* WIP on embedder selection
TODO: apply splitting and query prefixes (if applicable)

* wip on upsert

* Support base model
support nomic-text-embed-v1
support multilingual-e5-small
Add prefixing for both embedding and query for RAG tasks
Add chunking prefix to all vector dbs to apply prefix when possible
Show dropdown and auto-pull on new selection

* norm translations

* move supported models to constants
handle null seelction or invalid selection on dropdown
update comments

* dev

* patch text splitter maximums for now

* normalize translations

* add tests for splitter functionality

* normalize

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
 2025-07-22 10:07:20 -07:00
Sean Hatfield
31a8ead823
Fix multimodal chats via openai compat api (#4135) 
* fix multimodal chats via openai compat api

* lint

* add tests for multi-modal content in openai compat endpoint

* refactor to normalize how openai attachments are handled

* uncheck file

* rewrite tests, autodetect mime from dataurl, and spread attachments from prompt

* lint

* revert and fix tests

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-07-22 09:57:32 -07:00
Sean Hatfield
6d6bd14622
Moonshot AI LLM & agent provider (#4178) 
* add moonshot ai LLM & agent provider

* fix moonshot agent calling

* handle attachments/fix moonshot llm provider

* update docs/example env

* add moonshot to onboarding privacy

* add moonshot to onboarding llm preference

* update privacy for moonshot ai

* update logo higher res

* remove caching and use modelmap
 2025-07-22 09:56:51 -07:00
Sean Hatfield
ea8ba2f7f2
Clear cached docs on text splitter changes (#4163) 
* clear cached docs on text splitter changes

* update warning message and title

* remove unsused translation keys

* wrap trycatch

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-07-17 11:03:18 -07:00
Timothy Carambat
043cb1d085
Add model field parameter for generic OpenAI TTS (#4171) 
resolves #4165
 2025-07-17 10:28:51 -07:00
timothycarambat
c535c69345 1.8.4 tag update 2025-07-16 10:40:39 -07:00
Sean Hatfield
49ea545d7f
SQL preflight connection validation (#4150) 
* wip sql connection string validation

* handle failed sql connections in frontend

* sql preflight connection validation on modal save

* revert unneeded be/fe changes

* linting, form updates

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-07-16 09:02:39 -07:00
Sean Hatfield
fe1dfb1fde
Add option to enable encryption on MSSQL Server db (#4134) 
* add option to enable encryption on mssql server dbs

* add tests for ConnectionStringParser
 2025-07-15 15:26:38 -07:00
Sean Hatfield
1cd0cc32b8
Fix chunking/snippet logs for clarity (#4129) 
update chunking/snippet logs for clarity
 2025-07-11 10:54:54 -07:00
Timothy Carambat
ea956f4d77
Add ability to search workspace and threads (#4120) 
* Add ability to search workspace and threads

* fix height

* styling

* update placeholder

* Translations PR (#4122)
 2025-07-10 16:42:10 -07:00
timothycarambat
ae2fa8805c resolves #4040 2025-07-10 13:20:39 -07:00
Sean Hatfield
49293e7dd7
Configurable message limit for embed chat widget (#4114) 
* configurable message limit for embed widget

* remove console log

* make field optional + add fallback

* rework validation logic

* lint

* remove field specific guard, it cannot be lte 0 like all other fields

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-07-10 12:45:53 -07:00
Sean Hatfield
5d60047dc7
Handle BigInt in message response (#4110) 
* wip handle bigints in message response

* extend bigint protoype to handle bigint stringification + add test

* unset unrelated file

* update tests, simplify implementation;

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-07-10 12:33:34 -07:00
Timothy Carambat
8001d9ddeb
update 1.8.3 tags for release (#4109) 
* update 1.8.3 tags for release

* whoops, botched news
 2025-07-09 12:17:56 -07:00
timothycarambat
8ea3c71b5e Merge branch 'master' of github.com:Mintplex-Labs/anything-llm 2025-07-08 14:29:54 -07:00
timothycarambat
db2466e179 persistence of the GENERIC_OPENAI_STREAMING_DISABLED setting 2025-07-08 14:29:47 -07:00
Sean Hatfield
4c16e7f7ab
Fix score reporting for Milvus, Zilliz, and Pinecone vector databases (#4106) 
* normalize milvus score reporting

* normalize pinecone score reporting

* normalize score reporting for zilliz
 2025-07-08 13:47:11 -07:00
Sean Hatfield
15b842f6ba
Patch tokenizer race condition bug (#4091) 
patch tokenizer race condition bug

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-07-07 13:25:30 -07:00
Fabio Nonato
0d7a7551b8
fix to support: feat2864 - using local credentials file with Amazon Bedrock (#3986) 
* fix: feat2864

* patch default case

---------

Co-authored-by: nonatofabio <fnp@amazon.com>
Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-07-02 09:15:23 -07:00
Sean Hatfield
f3839b355c
Allow static defined prompt variables to be accessed by any authorized user (#4086) 
* allow static defined prompt variables to be accessed by any authd user

* improve filtering logic and comments

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-07-02 08:09:42 -07:00
Sean Hatfield
fc55baf69a
LMStudio 0 value embeddings patch (#4084) 
patch lmstudio encoding_format to fix all embeddings as 0 value
 2025-07-01 18:05:58 -07:00
Sean Hatfield
07129e81f8
Add option to disable streaming via env for generic openai provider (#4079) 
* add option to disable streaming via env for generic openai provider

* move env check to streamingEnabled
 2025-07-01 12:47:46 -07:00
timothycarambat
d1d68af0f8 patch skipCache logic among all vector db providers 
resolves #3958
 2025-06-30 13:16:46 -07:00
Sean Hatfield
91e498229c
Publish slash commands to hub (#4019) 
* implement ui for publish system prompt to hub

* rework ui + add backend to upload to hub

* add success modal view + publish menu item + translations

* normalize translations

* refactor PublishEntityModal + add hook
for hub auth

* normalize translations

* fix ui for success screen

* refactor, auth checks, UI/UX, and naming conventions to be more clear

* move components to CommunityHub folder + small ui tweak

* wip publish agent flows to community hub

* rework translations/implement uploading agent flows

* normalize/restructure translations

* rename component/add jsdoc for consistency

* fix en translation

* remove comments/duplicate function from merge conf

* update styles of publish button in flow builder

* resolve ssr icon issue

* implement uploading slash commands to hub

* normalize translations

* show command, non editable

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-06-24 16:19:50 -07:00
Sean Hatfield
2d210aa90a
Publish agent flows to hub (#4001) 
* implement ui for publish system prompt to hub

* rework ui + add backend to upload to hub

* add success modal view + publish menu item + translations

* normalize translations

* refactor PublishEntityModal + add hook
for hub auth

* normalize translations

* fix ui for success screen

* refactor, auth checks, UI/UX, and naming conventions to be more clear

* move components to CommunityHub folder + small ui tweak

* wip publish agent flows to community hub

* rework translations/implement uploading agent flows

* normalize/restructure translations

* rename component/add jsdoc for consistency

* fix en translation

* remove comments/duplicate function from merge conf

* update styles of publish button in flow builder

* resolve ssr icon issue

* UI linting

* language diff

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-06-24 15:58:49 -07:00
Sean Hatfield
62d3803bf0
Fix prompt, model, and temperature overrides in embed chat widget (#4036) 
fix data-prompt attribute in embed chat widget

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-06-24 14:23:02 -07:00
Timothy Carambat
64d9fbc8f0
Show app version in system settings sidebar (#4044) 
* Add version tagging
resolves #4038
closes #4034
closes #4028

* add hook

* add build

* patch
 2025-06-24 13:56:12 -07:00
leopardracer
1e1694fcd0
Fix Typos in Comments and Documentation (#4015) 
* Update communityHub.js

* Update memory.js
 2025-06-24 09:06:59 -07:00
Sean Hatfield
96b532a0f4
Publish system prompts to hub (#3976) 
* implement ui for publish system prompt to hub

* rework ui + add backend to upload to hub

* add success modal view + publish menu item + translations

* normalize translations

* refactor PublishEntityModal + add hook
for hub auth

* normalize translations

* fix ui for success screen

* refactor, auth checks, UI/UX, and naming conventions to be more clear

* move components to CommunityHub folder + small ui tweak

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-06-16 09:59:38 -07:00
Maxim Evtush
7cef25822c
Fix typos in blog post example and save-file-browser plugin (#4009) 
* Update blog-post-coding.js

* Update save-file-browser.js

* Update blog-post-coding.js

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-06-16 09:19:44 -07:00
FT
e37f20f547
Fix Typo in Milvus Provider and Update Comment in Prisma Schema (#4003) 
* Update schema.prisma

* Update index.js
 2025-06-15 12:58:56 -07:00
fuder.eth
3c89ea987f
Fix Typos in MetaGenerator.js Documentation Comments (#3993) 
Update MetaGenerator.js

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-06-13 08:56:40 -07:00
kilavvy
1752bc261e
Fix Typos in Comments and Documentation (#3987) 
* Update markdown-katex.js

* Update PRISMA.md
 2025-06-12 13:59:38 -07:00
Timothy Carambat
2055c8accd
Flag to disable login UI and endpoints for credentialed auth (#3984) 
* Flag to disable login UI and endpoints for credentialed auth

* dev build

* fix translation key
 2025-06-11 12:46:40 -07:00
leopardracer
e779dcfeff
Fix Typo in Documentation and Metadata Field Name (#3979) 
* Update index.js

* Update index.js
 2025-06-10 09:27:44 -07:00
Timothy Carambat
c0d66e6c19
Enable UI/UX for model swapping in chat window (#3969) 
* Enable UI/UX for model swapping in chat window

* forgot component

* patch useGetProviders hook to set loading on change of provider

* dev build

* normalize translations

* patch how model default is provided

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
 2025-06-09 09:59:17 -07:00
Timothy Carambat
4eb951d40e
Fix model map staleness behavior or fallback (#3971) 
* Fix model map staleness behavior or fallback

* patch url

* fix log

* dev build
 2025-06-06 17:39:48 -07:00
Sean Hatfield
ef0928993e
Implement importing of agent flows from community hub (#3867) 
* implement importing of agent flows from community hub

* auto enable flow on import

* remove unused blocks for docker
prevent importing or saving of agent flows that have unsupported blocks for version or platform

* dev build

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-06-05 15:08:58 -07:00
Sean Hatfield
77f6262290
Direct output for agent flows (#3873) 
* wip: create direct output switch on last block and send response to ui

* lint

* Return flow on direct output enabled
prevent new blocks below direct output block
Update executor/aibitat to handle skipping of handler outputs

* dev build

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-06-05 14:46:44 -07:00
Timothy Carambat
271a682824
remove LLM Instruction inputVariable requirement (#3954) 2025-06-05 07:43:25 -07:00
Sean Hatfield
2938aafae7
JSON Parser from agent flow blocks (#3863) 
* implement json parser for agent flow to allow dot notation and array access

* lint

* patch parser for pathing on objects
add tests for cases

* Move webscraping deps to closure
update tests to not modify env since no longer needed
do not modify paths with spaces - could be text key with spaces

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-06-05 07:18:00 -07:00
Sean Hatfield
0c9817ad59
Patch apiChatHandler to include agent chats (#3905) 
patch apiChatHandler to include agent chats
 2025-06-03 14:39:55 -07:00
Paolo Calvi
9470fcce09
Fix/mcp environment inheritance (#3909) 
* Fix: MCP server environment inheritance across platforms

- Fix GUI applications not inheriting proper PATH/NODE_PATH
- Correct Docker NODE_PATH to point to modules directory
- Ensure base environment is always provided
- Preserve user environment variable overrides
- Resolves -32000 Connection closed errors on macOS/Linux GUI

* Fix: MCP server environment inheritance across platforms after linting

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-06-03 11:41:42 -07:00
Timothy Carambat
a0134de979
fix issue with files loading with watching enabled (#3930) 2025-06-01 14:53:48 -05:00
Timothy Carambat
a57536b715
Handle invalid response bodies for ContextWindowFinder (#3896) 
Handle invalid response bodies for contextwindowfinder
 2025-05-27 15:40:06 -07:00
Timothy Carambat
753e1e9574
Filter malformed post-processed docs from file picker (#3853) 
Filter malformed post-processed files from UI
 2025-05-19 13:07:24 -07:00
timothycarambat
7e5e0952c3 remove comment in file util 2025-05-19 12:46:45 -07:00
timothycarambat
cabcc16e49 fix folders accumulating many files 2025-05-16 11:51:03 -07:00
Timothy Carambat
0e1de19408
Respect Max Chunk Len set - Generic OpenAI Embedder connection (#3838) 
Respect Max Chunk Len set
resolves #3834
 2025-05-16 11:18:24 -07:00
timothycarambat
6c36443657 patch Cached tag in UI not showing from #3833 2025-05-15 21:49:01 -07:00
Timothy Carambat
3bbceb82aa
3416 file picker load times (#3833) 
* wip

* implment conditional streaming

* no bench
 2025-05-15 13:54:14 -07:00
Sean Hatfield
9371bcc71e
Optional summarization for web-scraping flow block (#3496) 
* remove summarization from web-scraping flow block

* add option to enable/disable summarization in web-scraping block

* change layout
summarize default to true (was the normal prior)

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-05-15 11:50:09 -07:00
timothycarambat
2450e49ac3 hoisting cleanup for format var 2025-05-14 16:25:17 -07:00
timothycarambat
dbd8afe1d2 Merge branch 'master' of github.com:Mintplex-Labs/anything-llm 2025-05-14 15:26:20 -07:00
timothycarambat
605910b76d forgot files for DPAIS 2025-05-14 15:26:14 -07:00
Timothy Carambat
378ceaecec
Support Dell Pro AI Studio provider (#3829) 2025-05-14 15:10:48 -07:00
Timothy Carambat
e80492606a
Automatic Context window detection (#3817) 
* Add context window finder from litellm maintained list
apply to all cloud providers, have client cache for 3 days

* linting
 2025-05-14 11:03:19 -07:00
omahs
946be93f08
Fix typos (#3826) 
* Fix typos

* language

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-05-14 09:30:28 -07:00
Sean Hatfield
2b274c62b7
Obsidian data connector (#3798) 
* add obsidian vault data connector

* lint

* add english translations

* normalize translations

* improve file parser and reader

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-05-12 13:45:27 -07:00
timothycarambat
492570dfed patch Azure image reading regressions 
resolves #3811
 2025-05-12 11:10:35 -07:00
Shixian Sheng
aef455d977
Fixed grammar and typos (#3802) 
* Update common.js

* Update README.md

* Update common.js

* Update README.md

* linting

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-05-12 09:42:50 -07:00
timothycarambat
d99ab52165 add error fix for pgvector 2025-05-09 14:20:52 -07:00
Timothy Carambat
e1b7f5820c
PGvector vector database support (#3788) 
* PGVector support for vector db storage

* forgot files

* comments

* dev build

* Add ENV connection and table schema validations for vector table
add .reset call to drop embedding table when changing the AnythingLLM embedder
update instrutions
Add preCheck error reporting in UpdateENV
add timeout to pg connection

* update setup

* update README

* update doc
 2025-05-09 12:27:11 -07:00
timothycarambat
f40559b180 fix null entry on new workspace 2025-05-08 08:34:37 -07:00
Sean Hatfield
10e65fc021
Chat prompt history menu (#3770) 
* wip system prompt history sidebar ui

* lint

* backend/frontend implementation for prompt history wip

* lint

* rework ui

* add delete menu and delete chat history by id

* lint

* ref for menu button

* reorganize components + light mode styles

* lint

* UI refactor

* backend refactor

* remove unused import

* add border-none to all buttons

* fix spacing on dots 3 icon button

* add window to confirm

* add english translations

* normalize translations

* lin

* patch store logic

* sticky header

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-05-08 08:29:46 -07:00
Timothy Carambat
6fc0a6a644
Enable workflow rule for package verification (#3778) 
enable workflow rule
 2025-05-07 12:51:14 -07:00
Timothy Carambat
f0104fe1a9
Merge commit from fork 2025-05-07 09:37:31 -07:00
Danny Steenman
5500fa2bc5
feat: support for iam roles for bedrock client (#2632) 
* feat: implement iam role auth for bedrock

* fix: make client refreshes properly when switching between iam_user and iam_role

* checkout agent flow

* fix aiprovider for bedrock in agent use

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-05-06 13:48:15 -07:00
Tristan Stahnke
b64a77f29f
Refactor AWS Bedrock Provider for Multi-modal Support & Correct Token Limits (#3714) 
* Fixed two primary issues discovered while using AWS Bedrock with Anthropic Claude Sonnet models:
- Context Window defaults to 8192 maximum, which isn't correct
- Multimodal stopped working when removing langchain, which was transparently handling image_url to a format sonnet expects.

* Ran `yarn lint`

* Updated .env.example to have aws bedrock examples too

* Refactor for readability
move utils for AWS specific functionality to subfile
add token output max to ENV so setting persits

---------

Co-authored-by: Tristan Stahnke <tristan.stahnke+gpsec@guidepointsecurity.com>
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-05-06 12:55:24 -07:00
timothycarambat
ae092b4d4a openai filter other niche models 2025-05-05 12:07:48 -07:00
Sean Hatfield
78e6117dd8
Filter image models from OpenAI (#3768) 
filter image models from openai llm provider
 2025-05-05 12:01:04 -07:00
Sebastián Darío Ramírez
4f23e44756
Add gemini-embedding-exp-03-07 model support (#3767) 
* adds gemini-embedding-exp-03-07 support

* unset random changes, make model map

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-05-05 09:37:01 -07:00
Sean Hatfield
8912d0f0fc
Add option to control KoboldCPP max response tokens (#3746) 
add option to control koboldcpp max response tokens
 2025-05-02 14:12:06 -07:00
ALEKSEY KAPUSTYANENKO
e93964b67b
#3707 added sse and streamable stransport support for mcp servers (#3747) 
* #3707 added sse and streamable stransport support for mcp servers

* #3707 yarn lint changes

* bump MCP SDK to latest

* option chain transport

* small refactor

---------

Co-authored-by: Aleksey Kapustyanenko <kapustyanenko@mercurydevelopment.com>
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-05-02 14:09:25 -07:00
timothycarambat
62627fca7f add preUpdate checks on updateENV 2025-04-29 14:44:23 -07:00
timothycarambat
15acb19a68 Telemetry offset disabled count 2025-04-29 14:10:59 -07:00
timothycarambat
dacd67124a Prune telemetry events 
Update telemetry for model tagging
Update readme on telemetry
 2025-04-29 13:54:00 -07:00
Timothy Carambat
f9ef5840cb
Remove @google/generative-ai from AnythingLLM (#3740) 
* Remove  from AnythingLLM
Migrate to full use of OpenAI Compatible endpoints'
resolves #3618

* lint
 2025-04-29 11:39:56 -07:00
Shinya Suzuki
cd900f9e4c
Replace @azure/openai with openai, and update openai to version 4.95.1 (#3691) 
* Replace @azure/openai to OpenAI lib

* Remove @azure/openai dependency and update openai to version 4.95.1

* linting

* update logging
fix translation dictionary error

* remove bad ENV key that DNE
linting
Patch Azure OpenAI
Migrate Azure Agent provider to use OpenAI Schema for tool calling performance

* unset

* migrate azure to use default OAI stream handler

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-04-29 11:21:39 -07:00
Sean Hatfield
610bdd4673
Allow custom headers in upload-link endpoint (#3695) 
* allow custom headers in upload-link endpoint

* override loader.scrape to allow for passing of headers in langchain puppeteer

* lint

* Rename some variables
move positional args to named args
update documentation to reflect arg changes and funciton sigs
validate header object before attempting to end to forward to request

* update header validation for custom headers

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-04-22 12:47:12 -07:00
Timothy Carambat
1601eb986c
Enable bypass of ip limitations via ENV in collector processing (#3652) 
* Enable bypass of ip limitations via ENV in collector startup
resolves #3625
connect #3626

* dev build

* bump dockerx build action

* enable runtime setting config of collector requests

* comments and linting for option passing

* unset

* unset

* update docs link

* linting and docs
 2025-04-21 11:10:41 -07:00
Timothy Carambat
fd4929b4d2
Feature/drupalwiki collector (#3693) 
* Implement DrupalWiki collector

* Add attachment downloading and processing functionality (#3)

* linting

* Linting
Add citation image
small refactors
add URL for citation identifier

---------

Co-authored-by: em <eugen.mayer@kontextwork.de>
Co-authored-by: rexjohannes <53578137+rexjohannes@users.noreply.github.com>
Co-authored-by: Eugen Mayer <136934+EugenMayer@users.noreply.github.com>
 2025-04-21 09:17:24 -07:00
Timothy Carambat
05ced11650
Add support for /v1/document/upload* endpoints to support auto-add to workspace (#3692) 2025-04-21 08:28:33 -07:00
Timothy Carambat
7afbc6d11b
enable MCP tool use via @agent in API calls (#3683) 
* enable MCP tool use via `@agent` in API calls

* relocate import
 2025-04-20 14:49:22 -07:00
Timothy Carambat
a76ee21703
Handle env override for PATH and other defaults when env is specific (#3676) 
* dev build
auto merge MCP PATH and NODE_PATH

* remove log
 2025-04-18 12:44:48 -07:00
timothycarambat
f176a294f5 remove hard-coded token limit check 
resolves #3674
 2025-04-18 12:05:45 -07:00
Shinya Suzuki
98c46c04e4
Update Azure AI options and model map with new model configurations (#3660) 
* Update Azure AI options and model map with new model configurations

* linting

---------

Co-authored-by: Shinya Suzuki <shinya.s.825@gmail.com>
Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-04-16 09:08:40 -07:00
timothycarambat
1d1fb817b0 linting 2025-04-15 12:51:08 -07:00
Michał Rudziński
be27299897
handling of citations in openRouter provider #3581 (#3620) 
* handling of citations in openRouter provider #3581

* Update pplx enrichToken function comment
Modify OR enrichToken to be generic handler function with optional params
handle _just_ Perplexity in-line citations since no other models support this functionality

* remove console log

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-04-15 10:57:09 -07:00
timothycarambat
1359151dd6 Merge branch 'master' of github.com:Mintplex-Labs/anything-llm 2025-04-14 13:54:50 -07:00
Sean Hatfield
15f524ca37
Fix customize logo bug (#3628) 
fix customize logo bug on first setup

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-04-14 11:51:04 -07:00
timothycarambat
9a14535cf4 update gitignore 2025-04-09 15:40:45 -07:00
Timothy Carambat
1b59295f89
Refactor Gemini to use OpenAI interface API (#3616) 
* Refactor Gemini to use OpenAI interface API

* add TODO

* handle errors better (gemini)

* remove unused code
 2025-04-07 17:18:31 -07:00
Timothy Carambat
699684e301
Disable Agent retry to prevent looping on crash (#3617) 2025-04-07 16:44:21 -07:00
Timothy Carambat
4ac900f645
Gemini model list sync (#3609) 
* Update defaultModels.js

add gemma-3-27b-it to v1BetaModels

* Update defaultModels.js

20250330 model update

* Update defaultModels.js

remove text embedding

* Update name and inputTokenLimit modelMap.js

* Update gemini to load models from both endpoints
dedupe models
decide endpoint based on expieremental status from fetch
add util script for maintainers
reduce cache time on gemini models to 1 day

* remove comment

---------

Co-authored-by: DreamerC <dreamerwolf.tw@gmail.com>
 2025-04-07 13:45:16 -07:00
timothycarambat
56eb74c664 optional "env" in MCP config 2025-04-06 13:41:17 -07:00
Sean Hatfield
59aa301e9a
Fix agent flow description bug (#3593) 
* fix agent flow description bug

* add jsdoc to AgentFlows class

* lint

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-04-03 17:30:08 -07:00
Sean Hatfield
2e947050df
Allow period in username (#3592) 
* allow period in username

* update validation hint

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-04-03 16:42:39 -07:00
Timothy Carambat
7a01298a8e
ModelContextProtocol (MCP) Full Compatibility (#3547) 
* WIP MCP full compatibility layer

* implement MCP agent function wrapping and invocation methods

* Add `uvx` to docker bin for MCP executions

* dev build

* prune removed data

* Wrap MCP servers to lazy load items to not block the UI
Mobile bug fixes

* arm64 test build

* reset dev builder

* remove unused prop
 2025-03-31 16:15:19 -05:00
Timothy Carambat
1c905b1105
Replace connectionString parser for sql-agent (#3560) 
resolves #3551
 2025-03-28 12:46:21 -07:00
Timothy Carambat
b6698ff89e
Persist images attached in API requests to the UI for rendering (#3546) 
Persist images attached in API requests to the UI for rendereing
resolves #3518
 2025-03-26 14:16:12 -07:00
Timothy Carambat
4d06884a36
add AgentFlow execution via API chat invocation (#3545) 
* add AgentFlow execution via API chat invocation

* reorder import
 2025-03-26 13:57:26 -07:00
Timothy Carambat
78c83383d8
Overhaul AWS Bedrock provider (#3537) 
* Patch AWS Bedrock provider for newer models and performance

* patch prompt constructor
 2025-03-25 15:58:16 -07:00
Shixian Sheng
dd701f9aa4
Update QDRANT_SETUP.md (#3530) 2025-03-25 12:45:18 -07:00
Sean Hatfield
f3ea21bcd1
Prompt variables (#3359) 
* wip prompt variables

* refactor backend + add popup suggestions menu to frontend

* use processString to replace all variables in system prompts

* update translations

* fix translations

* wip highlight variables

* revert accidental name change

* rename everything, remove translations

* Update prompt var UI and backend logic

* Update form handler
linting

* linting

* normalize all translation files for prompt variables

* prompt vars dev image

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-03-25 12:44:19 -07:00
Sean Hatfield
f0b7919a8f
Protected slash command check (#3524) 
* protected slash command check

* fix error handling on frontend

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-03-24 17:09:36 -07:00
Sean Hatfield
5785a705cf
Enable use of @agent in slash commands (#3508) 
* allow @agent in slash commands

* make prompt input focused on slash command click

* lint

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-03-24 11:35:48 -07:00
Sean Hatfield
19ba6bcc15
Enable slash commands in dev API chat/stream-chat (#3514) 
* enable slash commands in dev api

* lint

* Remove ability to use default slash commands in API request
Add `reset` param to body that can reset chats according to the api chat execution parameters
Allow null `message` if `reset` is set in request.
Added early return for if message is null and reset is true
Enable chat to reset chat history and continue `message` execution
Added generic WorkspaceChat history reset function. Deprecated others

* update grep function comment
remove debug

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-03-24 11:29:36 -07:00
timothycarambat
0d1a2f06aa remove dupe key 2025-03-20 11:16:40 -07:00
Sean Hatfield
688ade6d96
Bump LanceDB (#3015) 
* bump lancedb

* dev build

* normalize build

* wtf is going on with build now

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-03-20 10:00:38 -07:00
Timothy Carambat
b54ab7be05
CHORE: debounce telems to prevent spamming (#3505) 
debounce telems to prevent spamming
 2025-03-20 08:40:28 -07:00
Sean Hatfield
e01c0dff04
Fix embedding endpoint for OpenAI compatible API (#3467) 
* fix embedding endpoint for openai compatible api

* remove unused imports

* add backwards compat for embedding

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-03-17 18:10:24 -07:00
Sean Hatfield
f6239a39f8
fix chroma db + add similarity offset (#3458) 
* fix chroma db + add similarity offset

* patch chroma scoring

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-03-17 17:48:23 -07:00
timothycarambat
0d5e869f5c Merge branch 'master' of github.com:Mintplex-Labs/anything-llm 2025-03-17 17:47:55 -07:00
timothycarambat
0c34d62342 fix pkg order 2025-03-17 17:47:48 -07:00
Sean Hatfield
5658aac25c
Add remove folder developer API endpoint (#3489) 
* add remove folder api endpoint

* update purgeFolder function comment

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-03-17 17:05:21 -07:00
timothycarambat
0b1a0e0a1f Merge branch 'master' of github.com:Mintplex-Labs/anything-llm 2025-03-04 09:08:21 -08:00
timothycarambat
b4afb995d4 Show default deepseek models on wrong API key 
connect #3390
 2025-03-04 09:07:49 -08:00
Timothy Carambat
66b4bf2679
Add support for Anthropics /model endpoint (finally) (#3376) 
* Add support for Anthropics /model endpoint (finally)

* dev
 2025-02-28 13:29:43 -08:00
AbelDuan
df166eb64e
feat: Add multilingual support for ocr module (#3325) 
* Add multilingual support for ocr mudule

* Add OCR langauge as server var that is passed into Collector
Support all valid tesseract language codes
Filter and parse only valid codes with fallbacks'

* persist TARGET_OCR_LANG

* update docker example env

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-02-27 12:31:17 -08:00
cnJasonZ
2aeb4c2961
Add new model provider PPIO (#3211) 
* feat: add new model provider PPIO

* fix: fix ppio model fetching

* fix: code lint

* reorder LLM
update interface for streaming and chats to use valid keys
linting

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-02-27 10:53:00 -08:00
Kristofer Bourro
b07240deee
Windows development environment variables support (#3354) 
* Windows development environment variables support

* moved cross-env to dev dependencies

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-02-27 10:43:31 -08:00
Sean Hatfield
6dd1fdc546
Add bio field to user (#3346) 
* add bio to users table

* lint

* add bio field to edit user admin page

* fix bio saving on new user

* simplify updating localstorage user

* linting

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-02-26 15:23:24 -08:00
timothycarambat
fa454aa6ca patch openapi swagger docs 2025-02-26 12:30:26 -08:00
Shanmukeshwar
d005107e24
feat: Add endpoint to upload documents to a specified folder (#3276) 
* feat: Add endpoint to retrieve documents by folder name

* isWithin Check on path to prevent path traversal

* feat: Add endpoint to upload documents to a specified folder

* refactor upload to folder endpoint + update jsdoc for swagger

* linting

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
 2025-02-26 12:26:24 -08:00
Timothy Carambat
2f5ed6c705
fix: bug where in single user mode deletion of an api key deletes all (#3358) 
* fix: bug where in single user mode deletion of an api key deletes all
resolves #3352

* linting
 2025-02-26 12:17:00 -08:00
Skanda Kaashyap
d1354caccb
[FEAT] Add claude-3-7 (#3337) 
* add claude 3-7 sonnet

* made all the changes everywhere

* add 3-7-sonnet-latest model

* lint

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
 2025-02-25 12:52:17 -08:00
Sean Hatfield
c36df2c364
Fix garbled non English chars on document upload (#3301) 
update handleAPIFileUpload middleware to handle non english chars + update jsdoc
 2025-02-20 23:09:34 -08:00
Timothy Carambat
dec13beb6e
Update null-check function for aibitat runners (#3283) 
* Update nullcheck function for aibitat runners

resolves #3273
closes #3277

* lint
 2025-02-19 10:21:31 -08:00
Sean Hatfield
e53ec1474e
Normalize chat sorting defaults in developer API (#3270) 
* normalize sorting for workspace and workspace thread chats in dev api

* lint

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-02-18 20:50:22 -08:00
timothycarambat
12b43256a0 lint 2025-02-18 20:49:40 -08:00
timothycarambat
defeb96c38 patch swagger 2025-02-18 16:02:55 -08:00
Sushanth Srivatsa
3fd0fe8fc5
2749 ollama client auth token (#3005) 
* ollama auth token provision

* auth token provision

* ollama auth provision

* ollama auth token

* ollama auth provision

* token input field css fix

* Fix provider handler not using key
sensible fallback to not break existing installs
re-order of input fields
null-check for API key and header optional insert on request
linting

* apply header and auth to agent invocations

* upgrading to ollama 5.10 for passing headers to constructor

* rename Auth systemSetting key to be more descriptive
linting and copy

* remove untracked files + update gitignore

* remove debug

* patch lockfile

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-02-18 16:00:17 -08:00
Shanmukeshwar
eeaa6b0151
feat: Add endpoint to retrieve documents by folder name (#3258) 
* feat: Add endpoint to retrieve documents by folder name

* isWithin Check on path to prevent path traversal

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-02-18 14:01:19 -08:00
Sean Hatfield
354b66e09e
Add additional embed API endpoints (#3264) 
* wip embed dev api endpoints

* fix /v1/embed/new endpoint

* add delete endpoint to embeds

* linting

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-02-18 13:56:56 -08:00
Timothy Carambat
c6ff3a7765
fix: Patch agent flow to honor agent handler established provider (#3251) 
resolves #3248
 2025-02-17 14:44:23 -08:00
Hakeem Abbas
ca60ba827b
fix: sanitizeNamespace (#3246) 
bug fixes for sanitizing Namespaces and handling chunk size limit of astradb collections in each doc

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-02-17 13:54:32 -08:00
timothycarambat
e978c90ea0 Revert "disable PH people props" 
This reverts commit 6fd34e5476.
 2025-02-17 09:56:14 -08:00
timothycarambat
6fd34e5476 disable PH people props 2025-02-17 09:43:18 -08:00
Timothy Carambat
2a9066e83a
OCR PDFs as fallback during upload (#3204) 
* OCR PDFs as fallback in spawn thread

* wip

* build our own worker fanout and wrapper

* norm pkgs

* bump dev
 2025-02-14 11:57:31 -08:00
Timothy Carambat
b6d3a411b1
Add querySelectorAll capability to web-scraping block (#3186) 
* Add `querySelectorAll` capability to web-scraping block

* patches and fallbacks

* fix styles of text in web scraping block

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
 2025-02-13 16:11:15 -08:00
Adam Setch
ea8454ea79
chore: rename Gitlab to GitLab (#3200) 
* chore: rename `Gitlab` to `GitLab`

Signed-off-by: Adam Setch <adam.setch@outlook.com>

* undo code changes - breaks refs

---------

Signed-off-by: Adam Setch <adam.setch@outlook.com>
Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-02-13 12:13:37 -08:00
Adam Setch
d63438fa61
chore: rename Github to GitHub (#3199) 
* chore: rename Github to GitHub

Signed-off-by: Adam Setch <adam.setch@outlook.com>

* chore: rename Github to GitHub

Signed-off-by: Adam Setch <adam.setch@outlook.com>

* Undo some code changes for references

---------

Signed-off-by: Adam Setch <adam.setch@outlook.com>
Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-02-13 10:45:43 -08:00
Sean Hatfield
e5f3fb0892
Agent flow builder (#3077) 
* wip agent builder

* refactor structure for agent builder

* improve ui for add block menu and sidebar

* lint

* node ui improvement

* handle deleting variable in all nodes

* add headers and body to apiCall node

* lint

* Agent flow builder backend (#3078)

* wip agent builder backend

* save/load agent tasks

* lint

* refactor agent task to use uuids instead of names

* placeholder for run task

* update frontend sidebar + seperate backend to agent-tasks utils

* lint

* add deleting of agent tasks

* create AgentTasks class + wip load agent tasks into aibitat

* lint

* inject + call agent tasks

* wip call agent tasks

* add llm instruction + fix api calling blocks

* add ui + backend for editing/toggling agent tasks

* lint

* add back middlewares

* disable run task + add navigate to home on logo click

* implement normalizePath to prevent path traversal

* wip make api calling more consistent

* lint

* rename all references from task to flow

* patch load flow bug when on editing page

* remove unneeded files/comments

* lint

* fix delete endpoint + rename load flows

* add move block to ui + fix api-call backend + add telemetry

* lint

* add web scraping block

* only allow admin for agent builder

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* Move AgentFlowManager flows to static
simplify UI states
Handle LLM prompt flow when provided non-string

* delete/edit menu for agent flow panel + update flow icon

* lint

* fix open builder button hidden bug

* add tooltips to move up/down block buttons

* add tooltip to delete block

* truncate block description to fit on blocklist component

* light mode agent builder sidebar

* light mode api call block

* fix light mode styles for agent builder blocks

* agent flow fetch in UI

* sync delete flow

* agent flow ui/ux improvements

* remove unused AgentSidebar component

* comment out /run

* UI changes and updates for flow builder

* format flow panel info

* update link handling

* ui tweaks to header menu

* remove unused import

* update doc links
update block icons

* bump readme

* Patch code block header oddity
resolves #3117

* bump dev image

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-02-12 16:50:43 -08:00
Timothy Carambat
cc3d619061
Add handling to reasoning models for Generic OpenAI connector (#3183) 
* Add handling to resoning models for Generic OpenAI connector
resolves #3177

* linting
 2025-02-12 10:28:44 -08:00
Timothy Carambat
d90571fa33
Add improved logging and introspection to web-browsing (#3140) 2025-02-06 09:23:05 -08:00
Timothy Carambat
1bfd461719
Patch PPLX streaming for timeouts (#3130) 
Add in-text citations as well for PPLX token streaming
handle timeouts for stream/buffer hanging
 2025-02-05 16:35:22 -08:00
Timothy Carambat
df8d34d31e
Enable num_ctx to match defined chunk length in ollama embedder (#3129) 
* Enable `num_ctx` to match defined chunk length in ollama embedder

* remove console
 2025-02-05 15:46:39 -08:00
Timothy Carambat
e76baacec4
Add reasoning flag for azure models with "default" fallback (#3128) 2025-02-05 15:22:45 -08:00
timothycarambat
a5ee6121e2 Add patch for o# models on Azure 
connect #3023
Note: depends on user naming the deployment correctly.
 2025-02-05 11:34:03 -08:00
timothycarambat
c2c4f63643 bump cdn 2025-02-05 10:30:43 -08:00
timothycarambat
a4d5b2588f o3 model patch 2025-02-03 14:19:21 -08:00
Timothy Carambat
d1ca16f7f8
Add tokenizer improvments via Singleton class and estimation (#3072) 
* Add tokenizer improvments via Singleton class
linting

* dev build

* Estimation fallback when string exceeds a fixed byte size

* Add notice to tiktoken on backend
 2025-01-30 17:55:03 -08:00
Timothy Carambat
e1af72daa7
parse streams from reasoning_content from DeepSeek API (#3068) 2025-01-30 14:55:20 -08:00
Timothy Carambat
8eac83bb82
Enable include_reasoning in request (#3067) 2025-01-30 14:32:38 -08:00
timothycarambat
6192080635 Update NVIDIA branding 2025-01-28 10:27:31 -08:00
Timothy Carambat
6aa1854155
Add ability to disable default agent skills (#3041) 
* Add ability to disable default agent skills

* debug build
 2025-01-27 16:52:43 -08:00
Sean Hatfield
75790e7e90
Remove native LLM option (#3024) 
* remove native llm

* remove node-llama-cpp from dockerfile

* remove unneeded items from dockerfile

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-01-27 13:42:52 -08:00
Jason
c757c3fb5f
feat: update novita AI logo and default model (#3037) 2025-01-27 08:41:12 -08:00
Timothy Carambat
2ca22abc9c
Add Version to AzureOpenAI (#3023) 2025-01-24 13:41:37 -08:00
Sean Hatfield
48dcb22b25
Dynamic fetching of TogetherAI models (#3017) 
* implement dynamic fetching of togetherai models

* implement caching for togetherai models

* update gitignore for togetherai model caching

* Remove models.json from git tracking

* Remove .cached_at from git tracking

* lint

* revert unneeded change

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-01-24 11:06:59 -08:00
timothycarambat
273d116586 linting 2025-01-23 16:43:18 -08:00
Sean Hatfield
57f4f46a39
Bump perplexity models (#3014) 
* bump perplexity models

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2025-01-23 16:35:38 -08:00
Timothy Carambat
d35b37b6ec
wip agent ui animation (#2999) 
* wip agent ui animation

* WIP agent ui revision

* linting

* simplify css

* memoize agent responses

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
 2025-01-22 11:10:02 -08:00
Sean Hatfield
f8c72786df
Fix similarity score bug in lance/chroma dbs (#2986) 
* fix similarity score bug in lance/chroma dbs

* batch lower bound case

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-01-17 18:27:54 -08:00
Timothy Carambat
c4f75feb08
Support historical message image inputs/attachments for n+1 queries (#2919) 
* Support historical message image inputs/attachments for n+1 queries

* patch gemini

* OpenRouter vision support cleanup

* xai vision history support

* Mistral logging

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
 2025-01-16 13:49:06 -08:00
timothycarambat
a2264f46ff add index to response object for marshalling 
resolves #2956
 2025-01-14 14:43:00 -08:00
timothycarambat
4d5d8d4dec patch gemini embedding key preference 
resolves #2972
 2025-01-14 08:20:57 -08:00
Timothy Carambat
21af81085a
Add caching to Gemini /models (#2969) 
rename file typo
 2025-01-13 13:12:03 -08:00
ReindeerFlotilla
be886f7d61
feat: add support for voyage-3-large and voyage-code-3 embedding models (#2955) 
* feat: add support for voyage-3-large and voyage-code-3 embedding models

- Add voyage-3-large and voyage-code-3 to VoyageAiOptions dropdown
- Update getMaxEmbeddingLength to support 32k context for new models
- Update .env.example with new model options

* unset env example

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2025-01-08 17:21:30 -08:00
timothycarambat
6134c15ca4 Add fallback URL for reranking model 2025-01-07 15:09:54 -08:00
timothycarambat
e192364d8d Migrate CDN download URL from S3 bucket 2025-01-07 12:09:14 -08:00
Sean Hatfield
c6547ec04d
Update example ENV files (#2931) 
update example env files
 2025-01-02 20:09:42 -08:00
Timothy Carambat
ad01df8790
Reranker option for RAG (#2929) 
* Reranker WIP

* add cacheing and singleton loading

* Add field to workspaces for vectorSearchMode
Add UI for lancedb to change mode
update all search endpoints to pass in reranker prop if provider can use it

* update hint text

* When reranking, swap score to rerank score

* update optchain
 2025-01-02 14:27:52 -08:00
Timothy Carambat
bb5c3b7e0d
make similarityResponse object arguments and not positional (#2930) 
* make `similarityResponse` object arguments and not positional

* reuse client for qdrant
 2025-01-02 12:03:26 -08:00
timothycarambat
f417ffd5e0 resolves #2926 
floor ms value
 2025-01-01 15:54:44 -08:00
timothycarambat
d52249aff8 support attachments via thread API chat/stream-chat endpoints 2024-12-31 09:52:21 -08:00
Timothy Carambat
ef71b95cf9
Return timestamps in OpenAI compat endpoints in proper format (#2921) 2024-12-31 09:35:52 -08:00
Chaiwat Saithongcum
fa3079bbbf
Add support for Google Generative AI (Gemini) embedder (#2895) 
* Add support for Google Generative AI (Gemini) embedder

* Add missing example in docker
Fix UI key elements in options
Add Gemini to data handling section
Patch issues with chunk handling during embedding

* remove dupe in env

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2024-12-31 09:29:38 -08:00
Sean Hatfield
8d302c3f67
Patch custom models endpoint (#2903) 
* prevent non admin users from changing llm settings via custom-models endpoint

* permission middleware to JSDOC

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2024-12-30 14:58:26 -08:00
Sean Hatfield
696af19c45
Patch unauthorized access to other user's pfps (#2904) 
* patch unauthorized viewing of other user's pfps

* inline return responses

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2024-12-30 14:36:51 -08:00
Sean Hatfield
0b7bf68f2c
Normalize paths on files uploaded to prevent arbitrary file writes (#2905) 
* normalize paths on files uploaded to prevent arbitrary file writes

* force normalize path in string parse

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2024-12-30 14:29:10 -08:00
Sean Hatfield
99b6dedc8b
Fix typo in frontend for SearXNG agent setup (#2917) 
fix typo in frontend for SearXNG agent setup
 2024-12-30 14:11:20 -08:00
timothycarambat
4b2bb529c9 enable leftover mlock setting 2024-12-28 17:48:24 -08:00
Timothy Carambat
a51de73aaa
update ollama performance mode (#2874) 2024-12-18 11:21:35 -08:00
wolfganghuse
af703427c7
fix wrong metadata assignment in MilvusProvider (#2870) 
fixed wrong metadata assignment
 2024-12-18 10:33:18 -08:00
Timothy Carambat
d54b5dfc62
Remove fine-tuning flow (#2872) 
remove fine-tuning flow
 2024-12-18 10:24:02 -08:00
Timothy Carambat
b082c8e441
Add support for gemini authenticated models endpoint (#2868) 
* Add support for gemini authenticated models endpoint
add customModels entry
add un-authed fallback to default listing
separate models by expiermental status
resolves #2866

* add back improved logic for apiVersion decision making
 2024-12-17 15:20:26 -08:00
Sean Hatfield
f8885a43e4
Append writable fields to dev API new workspace endpoint (#2843) 
* add writible fields to dev api new workspace endpoint

* lint

* implement validations for workspace model

* update swagger comments

* simplify validations for workspace on frontend and API

* cleanup validations

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2024-12-16 15:12:20 -08:00
Timothy Carambat
dd7c4675d3
LLM performance metric tracking (#2825) 
* WIP performance metric tracking

* fix: patch UI trying to .toFixed() null metric
Anthropic tracking migraiton
cleanup logs

* Apipie implmentation, not tested

* Cleanup Anthropic notes, Add support for AzureOpenAI tracking

* bedrock token metric tracking

* Cohere support

* feat: improve default stream handler to track for provider who are actually OpenAI compliant in usage reporting
add deepseek support

* feat: Add FireworksAI tracking reporting
fix: improve handler when usage:null is reported (why?)

* Add token reporting for GenericOpenAI

* token reporting for koboldcpp + lmstudio

* lint

* support Groq token tracking

* HF token tracking

* token tracking for togetherai

* LiteLLM token tracking

* linting + Mitral token tracking support

* XAI token metric reporting

* native provider runner

* LocalAI token tracking

* Novita token tracking

* OpenRouter token tracking

* Apipie stream metrics

* textwebgenui token tracking

* perplexity token reporting

* ollama token reporting

* lint

* put back comment

* Rip out LC ollama wrapper and use official library

* patch images with new ollama lib

* improve ollama offline message

* fix image handling in ollama llm provider

* lint

* NVIDIA NIM token tracking

* update openai compatbility responses

* UI/UX show/hide metrics on click for user preference

* update bedrock client

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
 2024-12-16 14:31:17 -08:00
Sean Hatfield
ae510619f0
Purge cached docs and remove docs from all workspaces on vectorDB/embedder changes (#2819) 
* wip remove all docs clear vector db on embedder/vector db change

* purge all cached docs and remove docs from workspaces on vectordb/embedder change

* lint

* remove unneeded console log

* remove reset vector stores endpoint and move to server side updateENV with postUpdate check

* reset embed module

* remove unused import

* simplify deletion process
rescoped document deletion to be more general for speed, everything needs to be reset anyway
fixed issue where unembedded docs not in any workspaces, but cached, were not removed

* add back missing readme file
update warning text modals

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2024-12-16 12:16:20 -08:00
wolfganghuse
d145602d5a
Add attachments to GenericOpenAI prompt (#2831) 
* added attachments to genericopenai prompt

* add devnote

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2024-12-16 12:03:51 -08:00
Timothy Carambat
ff024286d0
Add new workspace user management endpoint (#2842) 
deprecate old endpoint which users workspace ID vs slug
resolves #2838
 2024-12-16 11:50:34 -08:00
Sean Hatfield
79cdcd4e64
Add thread name to workspaces get endpoint (#2822) 
add thread name to workspaces get endpoint

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2024-12-13 17:53:54 -08:00
Sean Hatfield
f651ca8628
APIPie LLM provider improvements (#2695) 
* fix apipie streaming/sort by chat models

* lint

* linting

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2024-12-13 14:18:02 -08:00
Timothy Carambat
a46eb2418a
add Gitlab to watchable documents (#2817) 2024-12-12 10:44:11 -08:00
Timothy Carambat
538078747d
Add vector search API endpoint (#2815) 
* Add vector search API endpoint

* Add missing import

* Modify the data that is returned

* Change similarityThreshold to scoreThreshold

As this is what is actually returned by the search

* Removing logging (oops!)

* chore: regen swagger docs for new endpoint
fix: update function to sanity check values to prevent crashes during search

---------

Co-authored-by: Scott Bowler <scott@dcsdigital.co.uk>
 2024-12-12 10:12:32 -08:00
timothycarambat
f8e91e1ffa patch gemini-2.0-key 2024-12-11 16:52:31 -08:00
timothycarambat
69b672b625 add gemini 1206 and gemini-2.0-flash exp models 
connect #2788
 2024-12-11 09:04:29 -08:00
Timothy Carambat
a69997a715
update chat model filters for openai (#2803) 2024-12-11 08:55:10 -08:00
timothycarambat
a0c5d898f0 patch docs with proper input for OpenAI compatible embeddings 
resolves #2801
 2024-12-11 08:15:52 -08:00
timothycarambat
4b09a06590 persist token window for NIM and not only on model change 2024-12-05 11:57:07 -08:00
Timothy Carambat
b2dd35fe15
Add Support for NVIDIA NIM (#2766) 
* Add Support for NVIDIA NIM

* update README

* linting
 2024-12-05 10:38:23 -08:00
Sean Hatfield
05c530221b
Community hub integration (#2555) 
* wip hub connection page fe + backend

* lint

* implement backend for local hub items + placeholder endpoints to fetch hub app data

* fix hebrew translations

* revamp community integration flow

* change sidebar

* Auto import if id in URL param
remove preview in card screen and instead go to import flow

* get user's items + team items from hub + ui improvements to hub settings

* lint

* fix merge conflict

* refresh hook for community items

* add fallback for user items

* Disable bundle items by default on all instances

* remove translations (will complete later)

* loading skeleton

* Make community hub endpoints admin only
show visibility on items
combine import/apply for items to they are event logged for review

* improve middleware and import flow

* community hub ui updates

* Adjust importing process

* community hub to dev

* Add webscraper preload into imported plugins

* add runtime property to plugins

* Fix button status on imported skill change
show alert on skill change
Update markdown type and theme on import of agent skill

* update documentaion paths

* remove unused import

* linting

* review loading state

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2024-11-26 09:59:43 -08:00
timothycarambat
62be0cd0c5 add gemini-exp-1121 to expiermental set 2024-11-22 09:36:44 -08:00
timothycarambat
e4f47c1abf linting 2024-11-21 15:49:59 -08:00
hdelossantos
304796ec59
feat: support setting maxConcurrentChunks for Generic OpenAI embedder (#2655) 
* exposes `maxConcurrentChunks` parameter for the generic openai embedder through configuration. This allows setting a batch size for endpoints which don't support the default of 500

* Update new field to new UI
make getting to ensure proper type and format

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2024-11-21 11:29:44 -08:00
Sean Hatfield
9f38b9337b
Mistral embedding engine support (#2667) 
* add mistral embedding engine support

* remove console log + fix data handling onboarding

* update data handling description

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2024-11-21 11:05:55 -08:00
timothycarambat
246152c024 Add gemini-exp-1121 
resolves #2657
 2024-11-21 11:02:43 -08:00
Sean Hatfield
55fc9cd6b1
TogetherAI Llama 3.2 vision models support (#2666) 
* togetherai llama 3.2 vision models support

* remove console log

* fix listing to reflect what is on the chart

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2024-11-21 10:42:42 -08:00
Timothy Carambat
26e2d8cc3b
Add more expiermental models from Gemini (#2663) 2024-11-20 09:52:33 -08:00
timothycarambat
af16332c41 remove dupe key in ModelMap 2024-11-19 20:20:28 -08:00
Sean Hatfield
727d802779
Light/dark mode UI overhaul (#2629) 
* Refactor workspace sidebar component styles (#2380)

rely on css for conditional styles

* New sidebar colors (#2381)

new sidebar colors

* Main container color update (#2382)

* Dark mode setup themes (#2411)

* setup generic tailwind theme + ability to add new themes

* add theme context

* use correct colors from design for sidebar + fix padding

* Settings sidebar UI updates (#2416)

settings sidebar ui updates

* fix sidebar resizing/truncate issue on hover

* Dark mode chat window (#2443)

* Support XLSX files (#2403)

* support xlsx files

* lint

* create seperate docs for each xlsx sheet

* lint

* use node-xlsx pkg for parsing xslx files

* lint

* update error handling

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* wip chat window

* ux+ux improvements and update new colors

* chat window dark mode

* remove comment

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* Dark mode welcome page (#2444)

* dark mode welcome page styles + refactor

* remove AI_BACKGROUND_COLOR and USER_BACKGROUND_COLOR constants

* Dark mode UI for admin and tools pages + mobile view improvements (#2454)

* dark mode ui for admin and tools pages + mobile view improvements

* lint

* ai provider pages + options darkmode ui

* placeholder generic class

* appearance settings styles

* ai providers mobile margin

* dark mode styles for agent skills + experimental features

* mobile styles on security settings

* fine tune flow ui dark mode

* workspace settings page

* lint

* Dark mode onboarding (#2461)

dark mode onboarding

* update all modals + normalize styles (#2471)

* lint

* Dark mode privacy & experimental pages (#2479)

* document watch + privacy pages ui + mobile modal darkmode

* lint

* Dark mode login screens (#2483)

* multi-user auth screen ui update

* dark mode login screen + recovery key modals

* remove unneeded import

* Workspace preset commands modals dark mode (#2484)

update workspace preset dark mode modal

* Document pinning modal ui update (#2490)

document pinning modal ui update

* Experimental agreement modal dark mode (#2491)

experimental agreement modal dark mode

* Serp options dark mode (#2492)

serp options dark mode

* field fixes

* attempt light mode wip

* setting sidebar

* Toasts and threads

* main page content and privacy page

* force rewrite for light

* add border for light mode rightside content

* more fixes

* wip

* wip

* wip light mode implementation

* wip dark light mode file picker

* document picker light mode ui

* slight ui tweaks

* light mode fine tuning flow

* light mode tweaks + qa fixes

* fix md rendering of light mode + tooltip fixes

* lint

* qa bug fixes

* Add developer hook for theme
move provider to outmost layer

* qa light mode bug fixes

* Linting and hotfixes for UI

* Light mode to dev

* accept invite light mode ui fix

* Fix onboarding inputs in dark mode

* fix close icons
last minute items

* patch z-index on tooltips

* patch light mode citations

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2024-11-18 15:40:18 -08:00
Sean Hatfield
e29f054706
Bump TogetherAI models (#2645) 
* bump together ai models

* Run post-bump command

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2024-11-18 13:08:26 -08:00
Sean Hatfield
27b07d46b3
Patch bad models endpoint path in LM Studio embedding engine (#2628) 
* patch bad models endpoint path in lm studio embedding engine

* convert to OpenAI wrapper compatibility

* add URL force parser/validation for LMStudio connections

* remove comment

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2024-11-13 12:34:42 -08:00
WinSun
e41a9beaae
Fix the problem of web-searching can not complete searching sessions when using bing search. (#2611) 
Fix the problem of web-searching can not complete searching sessions.
 2024-11-08 20:37:15 -08:00
timothycarambat
ec3b3010f8 Patch VoyageAI implementation from LC 
resolves #2593
 2024-11-06 11:43:41 -08:00
timothycarambat
5aa79128f7 bump Anthropic models 2024-11-06 08:14:08 -08:00
timothycarambat
5094330441 bump @anthropic-ai/sdk to 0.32.1 2024-11-05 10:29:47 -08:00
Sean Hatfield
a2eced0b43
DuckDuckGo web search agent skill support (#2584) 
* duckduckgo web search agent skill support

* move ddg to first option in dropdown menu

* lint

* add duckduckgo option stating no config required
 2024-11-04 17:01:11 -08:00
Timothy Carambat
04e29203a5
Add header static class for metadata assembly (#2567) 
* Add header static class for metadata assembly

* update comments

* patch header parsing for links
 2024-11-04 11:47:46 -08:00
Timothy Carambat
80565d79e0
2488 novita ai llm integration (#2582) 
* feat: add new model provider: Novita AI

* feat: finished novita AI

* fix: code lint

* remove unneeded logging

* add back log for novita stream not self closing

* Clarify ENV vars for LLM/embedder seperation for future
Patch ENV check for workspace/agent provider

---------

Co-authored-by: Jason <ggbbddjm@gmail.com>
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
 2024-11-04 11:34:29 -08:00
Sean Hatfield
e719d05027
Add optional limit and orderBy to dev api chat endpoint (#2559) 
add optional limit and orderBy to dev api chat endpoint
 2024-10-30 14:47:24 -07:00
Mr Simon C
ccde891aa2
Feature/update api example outputs (#2547) 
* /v1/workspace/{slug} actually returns an array - update example output

* update swagger example to show array output
 2024-10-29 16:50:19 -07:00
timothycarambat
4eeac2752b Add index leftover from prisma migration from PR #2553 2024-10-29 16:35:56 -07:00
Timothy Carambat
dd2756b570
add sessionToken validation connection auth for AWSbedrock (#2554) 2024-10-29 16:34:52 -07:00
Timothy Carambat
2c9cb28d5f
Simple SSO feature for login flows from external services (#2553) 
* Simple SSO feature for login flows from external services

* linting
 2024-10-29 15:30:53 -07:00
timothycarambat
3fe59a7cf5 patch bad reference for EphemeralAgentHandler 2024-10-29 10:02:46 -07:00
Timothy Carambat
40800631e3
Patch v1/document/upload filename charset encoding (#2535) 2024-10-25 12:03:19 -07:00
Timothy Carambat
72ba9f7f28
Add filtering to sessionID for workspace chats (#2531) 2024-10-24 15:10:50 -07:00
Timothy Carambat
5bc96bca88
Add Grok/XAI support for LLM & agents (#2517) 
* Add Grok/XAI support for LLM & agents

* forgot files
 2024-10-21 16:32:49 -07:00
Timothy Carambat
c734742189
Rework failover agent invocation selection for agent and ephemmeral agent (#2515) 
* Rework failover agent invocation selection for agent and ephemmeral agent

* update order of check

* update order of check

* lint
 2024-10-21 14:40:30 -07:00
Timothy Carambat
11b7ccda07
Fix tooling examples (#2513) 
* Fix incorrect json API description.

* small edits and validity checks

* remove console.logs

* unset and recheck changes

---------

Co-authored-by: Adam <phazei@gmail.com>
 2024-10-21 13:56:37 -07:00
Timothy Carambat
0524aadf58
Enable the ability to disable the chat history UI (#2501) 
* Enable the ability to disable the chat history UI

* forgot files
 2024-10-21 13:19:19 -07:00
Timothy Carambat
e71392d83f
Feature/thread creation slug name (#2512) 
* thread creation additional params name and slug, with api

* typo fix

* Rebuild openai Swagger docs
Handle validations for fields to prevent invalid field inputs for .new
Enforce sluggification of `slug` to prevent breaking of URL structs

---------

Co-authored-by: abrakadobr <abrakadobr@gmail.com>
 2024-10-21 13:09:55 -07:00
Timothy Carambat
446164d7b9
Add Groq vision preview support (#2511) 
Adds support for only the llama3.2 vision models on groq. This comes with many conditionals and nuances to handle as Groqs vision implemention is quite bad right now
 2024-10-21 12:37:39 -07:00
Timothy Carambat
7342839e77
Passthrough agentModel for LMStudio (#2499) 2024-10-18 11:44:48 -07:00
Timothy Carambat
93d7ce6d34
Handle Bedrock models that cannot use system prompts (#2489) 2024-10-16 12:31:04 -07:00
Timothy Carambat
3dc0f3f490
Tts open ai compatible endpoints (#2487) 
* Update OpenAI TTS config to allow a custom BaseURL

* uncheck config file

* break openai generic TTS into its own provider

* add space

* hide TTS on user msg

---------

Co-authored-by: Adam <phazei@gmail.com>
 2024-10-15 21:39:31 -07:00
Sean Hatfield
fa528e0cf3
OpenAI o1 model support (#2427) 
* support openai o1 models

* Prevent O1 use for agents
getter for isO1Model;

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2024-10-15 19:42:13 -07:00
Sean Hatfield
6674e5aab8
Support free-form input for workspace model for providers with no /models endpoint (#2397) 
* support generic openai workspace model

* Update UI for free form input for some providers

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
 2024-10-15 15:24:44 -07:00
Timothy Carambat
c3723ce2ff
Add backfilling on query for chat widget to improve UX (#2482) 2024-10-15 14:37:44 -07:00
Sean Hatfield
be6289d141
Daily message limit per user (#2417) 
* set message limit per user

* remove old limit user messages + unused admin page

* fix daily message validation

* refactor message limit input
refactor canSendChat on user to a method on user model

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
 2024-10-15 14:01:29 -07:00