Repository navigation
Share Kubernetes metadata watchers across kubernetes_logs sources #26703
Unanswered
anisimov-es
asked this question in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi Vector maintainers!
We've encountered significant Kubernetes API watch amplification when running multiple kubernetes_logs sources within the same Vector process in our production environment.
To address this, we implemented a process-local registry that allows sources to share Kubernetes metadata watchers instead of creating independent watchers for the same resources.
The optimization is already running in our production environment and has significantly reduced Kubernetes API load and Vector memory consumption.
We've submitted the implementation in #26567, but realized that the contribution guidelines recommend opening an issue first for large architectural changes.
Before proceeding further with the PR, we'd like to get feedback on the proposed architecture:
Is sharing Kubernetes metadata watchers across sources an approach you'd be willing to support upstream?
Are there architectural constraints or concerns we should consider?
Would you prefer a different design to address this problem?
We're happy to adjust the implementation based on your feedback.
Thanks!
All reactions