merlyn/.github/workflows/dev-build.yaml
Timothy Carambat 0fb33736da
Workspace Chat with documents overhaul (#4261)
* Create parse endpoint in collector (#4212)

* create parse endpoint in collector

* revert cleanup temp util call

* lint

* remove unused cleanupTempDocuments function

* revert slug change
minor change for destinations

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* Add parsed files table and parse server endpoints (#4222)

* add workspace_parsed_files table + parse endpoints/models

* remove dev api parse endpoint

* remove unneeded imports

* iterate over all files + remove unneeded update function + update telemetry debounce

* Upload UI/UX context window check + frontend alert (#4230)

* prompt user to embed if exceeds prompt window + handle embed + handle cancel

* add tokenCountEstimate to workspace_parsed_files + optimizations

* use util for path locations + use safeJsonParse

* add modal for user decision on overflow of context window

* lint

* dynamic fetching of provider/model combo + inject parsed documents

* remove unneeded comments

* popup ui for attaching/removing files + warning to embed + wip fetching states on update

* remove prop drilling, fetch files/limits directly in attach files popup

* rework ux of FE + BE optimizations

* fix ux of FE + BE optimizations

* Implement bidirectional sync for parsed file states
linting
small changes and comments

* move parse support to another endpoint file
simplify calls and loading of records

* button borders

* enable default users to upload parsed files but NOT embed

* delete cascade on user/workspace/thread deletion to remove parsedFileRecord

* enable bgworker with "always" jobs and optional document sync jobs
orphan document job: Will find any broken reference files to prevent overpollution of the storage folder. This will run 10s after boot and every 12hr after

* change run timeout for orphan job to 1m to allow settling before spawning a worker

* linting and cleanup pr

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>

* dev build

* fix tooltip hiding during embedding overflow files

* prevent crash log from ERRNO on parse files

* unused import

* update docs link

* Migrate parsed-files to GET endpoint
patch logic for grabbing models names from utils
better handling for undetermined context windows (null instead of Pos_INIFI)
UI placeholder for null context windows

* patch URL

---------

Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
2025-08-11 09:26:19 -07:00

124 lines
4.7 KiB
YAML

name: AnythingLLM Development Docker image (amd64)
concurrency:
group: build-${{ github.ref }}
cancel-in-progress: true
on:
push:
branches: ['upload-ui-ux'] # put your current branch to create a build. Core team only.
paths-ignore:
- '**.md'
- 'cloud-deployments/*'
- 'images/**/*'
- '.vscode/**/*'
- '**/.env.example'
- '.github/ISSUE_TEMPLATE/**/*'
- '.devcontainer/**/*'
- 'embed/**/*' # Embed should be published to frontend (yarn build:publish) if any changes are introduced
- 'browser-extension/**/*' # Chrome extension is submodule
- 'server/utils/agents/aibitat/example/**/*' # Do not push new image for local dev testing of new aibitat images.
- 'extras/**/*' # Extra is just for news and other local content.
jobs:
push_multi_platform_to_registries:
name: Push Docker multi-platform image to multiple registries
runs-on: ubuntu-latest
permissions:
packages: write
contents: read
steps:
- name: Check out the repo
uses: actions/checkout@v4
- name: Check if DockerHub build needed
shell: bash
run: |
# Check if the secret for USERNAME is set (don't even check for the password)
if [[ -z "${{ secrets.DOCKER_USERNAME }}" ]]; then
echo "DockerHub build not needed"
echo "enabled=false" >> $GITHUB_OUTPUT
else
echo "DockerHub build needed"
echo "enabled=true" >> $GITHUB_OUTPUT
fi
id: dockerhub
# Uncomment this + add linux/arm64 to platforms if you want to build for arm64 as well
# - name: Set up QEMU
# uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
with:
version: v0.22.0
- name: Log in to Docker Hub
uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
# Only login to the Docker Hub if the repo is mintplex/anythingllm, to allow for forks to build on GHCR
if: steps.dockerhub.outputs.enabled == 'true'
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
with:
images: |
${{ steps.dockerhub.outputs.enabled == 'true' && 'mintplexlabs/anythingllm' || '' }}
tags: |
type=raw,value=dev
- name: Build and push multi-platform Docker image
uses: docker/build-push-action@v6
with:
context: .
file: ./docker/Dockerfile
push: true
sbom: true
provenance: mode=max
platforms: linux/amd64
# platforms: linux/amd64,linux/arm64
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
# For Docker scout there are some intermediary reported CVEs which exists outside
# of execution content or are unreachable by an attacker but exist in image.
# We create VEX files for these so they don't show in scout summary.
- name: Collect known and verified CVE exceptions
id: cve-list
run: |
# Collect CVEs from filenames in vex folder
CVE_NAMES=""
for file in ./docker/vex/*.vex.json; do
[ -e "$file" ] || continue
filename=$(basename "$file")
stripped_filename=${filename%.vex.json}
CVE_NAMES+=" $stripped_filename"
done
echo "CVE_EXCEPTIONS=$CVE_NAMES" >> $GITHUB_OUTPUT
shell: bash
# About VEX attestations https://docs.docker.com/scout/explore/exceptions/
# Justifications https://github.com/openvex/spec/blob/main/OPENVEX-SPEC.md#status-justifications
# Fixed to use v1.15.1 of scout-cli as v1.16.0 install script is broken
# https://github.com/docker/scout-cli
- name: Add VEX attestations
env:
CVE_EXCEPTIONS: ${{ steps.cve-list.outputs.CVE_EXCEPTIONS }}
run: |
echo $CVE_EXCEPTIONS
curl -sSfL https://raw.githubusercontent.com/docker/scout-cli/main/install.sh | sh -s --
for cve in $CVE_EXCEPTIONS; do
for tag in "${{ join(fromJSON(steps.meta.outputs.json).tags, ' ') }}"; do
echo "Attaching VEX exception $cve to $tag"
docker scout attestation add \
--file "./docker/vex/$cve.vex.json" \
--predicate-type https://openvex.dev/ns/v0.2.0 \
$tag
done
done
shell: bash