Apache HTTP localhost randomly taking 5 seconds on macOS Monterey but fast on HTTPS

I waited as long as I could but finally had to upgrade from Mojave.
I knew something would needlessly break. And of course it did.

Largish files (more than 100k) randomly take 5 seconds to load. It’s not always the same file, and not every time, and it only happens in HTTP Apache (tried port 80 and 8080). It works fine in Apache HTTPS/443 and Python’s SimpleHTTPServer on HTTP/80.

Things I have tried:

• It’s not DNS. It happens on http://127.0.0.1/large.jpg
• It’s not browser related, it happens on curl.
• It’s not IPv6. Turned it off, same deal.
• It’s not MTU, tried 1492, same thing (you can see my desperation by now)
• Already reinstalled it via brew reinstall httpd. Nothing.

Steps to reproduce:
i=1; while [ $i -lt 50 ]; do time curl http://127.0.0.1/large.jpg > /dev/null; ((i++)); done

Eventually it will stall for 5 seconds at 96%:
96 395k 96 382k

Another file:
i=1; while [ $i -lt 50 ]; do time curl http://127.0.0.1/jquery-latest.min.js > /dev/null; ((i++)); done

Eventually stalls at 85%
85 95786 85 81390

httpd -v
Server version: Apache/2.4.52 (Unix)
Server built: Dec 20 2021 13:37:44
Installed via brew

macOS Monterey 12.1 (21C52)

I’m loosing my mind, any help is much appreciated.
Thanks