feat: support parallel flushes

[grobinson-grafana]

What this PR does / why we need it:

This pull request adds support for flushing in parallel (in the background), while we build another data object in the main thread. This prevents pipeline stalls and lets us run fewer consumers reducing TCO.

I have been very careful here to ensure we don't commit the wrong offsets, and avoid data loss.

1. I allow only one flush to run at a time, and as a result of this, I also guarantee that they run in order too.
2. If the first batch fails, and exceeds all its retries, I intentionally crash the process and let K8s restart it. This prevents data loss where the first batch fails, the second batch succeeds, and then we accidentally commit the offsets for the first batch.

Why crash? Well, stopping the pipeline and resetting the Kafka consumer is much more complex and error-prone. I want to keep it simple to start with and then add complexity if needed later on.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR