Skip to content

vectorstores/bedrockknowledgebases: improve document ingestion accordinto to AWS guidelines - #1566

Open
lucaronca wants to merge 10 commits into
tmc:mainfrom
lucaronca:improve-bedrock-ingestion
Open

lucaronca wants to merge 10 commits into
tmc:mainfrom
lucaronca:improve-bedrock-ingestion

Conversation

@lucaronca

@lucaronca lucaronca commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

On main, AddDocuments ingests the documents directly and then starts a sync of the whole data source in the same call, which the direct ingestion documentation says not to do. It sleeps between checks of that sync regardless of the context, gives up after about five minutes with "exceeded maximum number of retries" even when the documents were loaded and the sync is still running, and counts a completed sync as success even when some documents failed. There is also no way to upload many documents and sync the data source once.

AddDocuments and AddNamedDocuments now ingest the documents directly and wait until each one is INDEXED, and they return an error that lists the documents ending in any other status. A data source that was never synced gets its first sync, which AWS requires for S3 data sources, and a running sync makes them return ErrIngestionJobInProgress before anything is uploaded.

UploadDocuments, UploadNamedDocuments and StartSync cover bulk loads, where the documents are uploaded first and the data source is synced once. Every wait now lasts until the caller's context is done, and WithWaitUntilSearchable makes AddDocuments also wait until SimilaritySearch returns the documents, which on an OpenSearch Serverless knowledge base took about a minute after INDEXED.

Numeric metadata is returned as float32 again, since bedrockagentruntime v1.63 and later decode it as float64.

I tested the first sync, direct ingestion, ErrIngestionJobInProgress with a running sync, the cleanup after a failed ingestion, document names with spaces and accents, and WithWaitUntilSearchable against a real knowledge base. The package tests mock the AWS clients.

PR Checklist

  • Read the Contributing documentation.
  • Read the Code of conduct documentation.
  • Name your Pull Request title clearly, concisely, and prefixed with the name of the primarily affected package you changed according to Good commit messages (such as memory: add interfaces for X, Y or util: add whizzbang helpers).
  • Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
  • Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. Fixes #123).
  • Describes the source of new concepts.
  • References existing implementations as appropriate.
  • Contains test coverage for new functions.
  • Passes all golangci-lint checks.

@lucaronca lucaronca changed the title vectorstores/bedrockknowledgebases: ingest documents directly and add StartSync vectorstores/bedrockknowledgebases: improve document ingestion following aws guidelines Nov 2, 2026
@lucaronca
lucaronca marked this pull request as ready for review November 2, 2026 19:22
@lucaronca lucaronca changed the title vectorstores/bedrockknowledgebases: improve document ingestion following aws guidelines vectorstores/bedrockknowledgebases: improve document ingestion accordinto to AWS guidelines Nov 2, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant