Ever had to pipe SQS to Kinesis without writing glue code? EventBridge Pipes handled filtering, batching, retries, and backpressure for me — on a 50k msgs/min stream — so it scaled horizontally with zero operational burden; curious what other cloud-native, no-code bridges people lean on for integration efficiency.
I lean on Kinesis Data Firehose as the no‑code batcher/fan‑in — set the buffer to 5 MB/10s while soaking, then bump to about 64 MB/120s to slash PUTs and smooth spikes; dynamic partitioning helps a ton… Small caveat: built‑in transforms can quietly throttle, so I keep them off unless absolutely needed. What is Amazon Data Firehose? - Amazon Data Firehose — have you compared Pipes vs Firehose when the sink is S3 or OpenSearch?
At ‘50k msgs/min’, a simple EventBridge Pipes input transformer to drop bulky fields before SQS→Kinesis kept each shard under the 1 MB/s write cap and cut PutRecords retries. Did you try that? Small caveat: the transform step is billed by EventBridge, so weigh it against the retry savings; docs: Amazon EventBridge.
Pipes to API Destinations with 50 rps cap + OAuth handled SaaS ‘backpressure’. Tried enrichment for header signing?
At about 50k msgs/min, I’ve had good results flipping the Kinesis stream to on‑demand so Pipes can surge without backpressure, and setting the target parameters to use a high‑cardinality field as the partition key to dodge hot shards. The combo felt “zero‑code” and cut our PutRecords errors to near zero; only caveat is on‑demand can be pricier if your rate is steady. @OP did you stay on provisioned shards or try on‑demand?