gitproto: spool materialized push body to avoid mid-stream stall

main

Commit

Soph4mo ago

Buffer the receive-pack body (update-request header + pack) to a temp file in the materialized push path so the POST goes out in one continuous burst, instead of streaming the body incrementally as the pack encoder produces it.

The cause we're working around: go-git's encoder runs delta selection synchronously before writing any pack bytes, which on big repos can take tens of seconds while the HTTP request body sits idle waiting for the next chunk. CDN edges like Cloudflare's enforce an idle-write timeout on the request body and close the connection on a stall that long — surfacing as "use of closed network connection" mid-upload, with no server response. Spooling collapses "encode" and "write" into a single sequential phase, so once bytes start flowing they don't stop until the body is done. Bootstrap-relay didn't have this problem because source pack bytes flow steadily from the upstream upload-pack response — no internal stall, nothing to engineer around.

Side effect: the spooled body has a known length, so the POST goes out with Content-Length instead of Transfer-Encoding: chunked (matching upstream git's smart-HTTP transport behaviour), and req.GetBody lets Go's transport replay the body on transient connection failures. These are nice-to-haves; the stall-avoidance is the actual fix.

Scoped to materialized only. The materialized strategy already requires the full source object closure to be local before encoding begins, so spooling on upload doesn't change its fundamental shape. Relay paths (PushPack) keep streaming source bytes through to target with chunked encoding, and preserve the "streaming proxy" property git-sync is built around.

Implementation:

  • gitproto.SpooledBody is a temp-file-backed io.ReadCloser with a known size, constructed via NewSpooledBody(write func(io.Writer) error). PostRPCStreamBody type-asserts on it and wires up req.ContentLength / req.GetBody.

  • PushObjects (materialized) writes the encoded update-request followed by the pack into one SpooledBody and POSTs that. The previous io.Pipe + goroutine encoder is gone; materialized was never streaming end-to-end so the encode-while-upload overlap it provided wasn't load-bearing.

  • postReceivePack split out of sendReceivePack so the response decoding (sideband demux + report-status) is shared between streaming and spooled send paths.

The new TestPushObjectsBuffersBody asserts Content-Length is set and Transfer-Encoding is absent for materialized push. TestPushPackStartsHTTPBeforePackFullyRead is unchanged — relay keeps its streaming property.