How do you troubleshoot alert delivery throttling when many jobs fail simultaneously?

This is a real operational failure mode reported by teams running StreamPark at scale: when something like a shared YARN Session cluster goes down, many jobs fail within seconds of each other, and the alert channels themselves — not StreamPark — become the bottleneck.

Chat-based robots like Feishu/Lark and WeChat Work group robots enforce rate limits on how often a webhook can be called. If StreamPark's alerting fires one HTTP call per failed job in a tight burst, several of those calls get throttled and silently dropped by the receiving platform, so only a subset of the expected alerts actually arrive — which is worse than no alerting, because it misleads whoever is triage since they don't realize how many jobs actually failed.

The fix teams have applied in production is to decouple job-failure detection from delivery: push each alert event onto an internal blocking queue instead of calling the webhook inline, and have a dedicated alert-sending thread drain that queue at a rate the channel's rate limit can actually sustain. This adds a short, bounded delay to alert delivery but guarantees every failure is eventually reported instead of a random subset being dropped under load.

Diagnosing this after the fact starts with comparing StreamPark's own failure count for the incident (visible per-application in the console) against how many alert messages actually landed in the channel; a gap between those two numbers is the signature of this exact throttling problem.