Proposal: let a handler supply its own response body stream - see #509

[landrix]

This is a proposal, not a finished feature waiting to be merged. #509 has had no
answer yet, and the API shape is your call - I wrote this so there is something
concrete to react to, and I am happy to throw any part of it away.

The send loop is already stream based (ContentStream, ProcessBody,
ValidateRange, rfContentStreamNeedFree), so this adds an entry point rather
than new plumbing.

Note that this builds on the Range: overflow fix you merged as b61d9db: the
feature relies on its ProcessBody() abort, since a handler-supplied stream can
end short far more easily than a file can.

API

// THttpServerRequestAbstract
function SetOutStream(aStream: TStream; aOptions: THttpOutStreamOptions = [];
  const aContentType: RawUtf8 = ''; aContentLength: Int64 = -1): cardinal;

Returns HTTP_SUCCESS, like the other SetOut* methods, so a handler can write
result := Ctxt.SetOutStream(...).

• aContentLength = -1 computes aStream.Size - aStream.Position, like a file
  would use its size; that position is also the origin of any Range:
• hosOwned hands the stream to the server, which frees it once the response is
  sent or the request aborted
• hosNoRange declares up front that this stream serves no Range:, for a source
  which can only be read forward; it is also set automatically when the stream
  turns out not to be seekable
• a body of unknown length stays out of scope, as I wrote in the issue: the
  send loop always emits Content-Length: and has no chunked response encoding

How it is wired

The actual work is a new THttpRequestContext.ContentFromStream(), sitting right
next to ContentFromFile() and doing the same things: set ContentLength,
validate the range, seek, assign ContentStream, set rfContentStreamNeedFree,
return HTTP_SUCCESS or HTTP_RANGENOTSATISFIABLE. THttpServerRequest. ProcessOutStream is then a ~20 line wrapper called from SetupResponse, mirroring
ProcessStaticFile. Both socket server families share that path, so both stream it.

• ranges behave like a file: 206 with Content-Range:, 416 when unsatisfiable
• a stream which can not seek - a TStreamWithNoSeek such as TPipeStream, or
  one whose Position/Size raise - is served sequentially with
  Accept-Ranges: none, and a Range: request on it is ignored, so it answers
  200 with the whole body rather than a 206 without a Content-Range:
• servers which can not stream an in-process TStream - THttpApiServer, where
  http.sys sends from the kernel, and REST-over-WebSockets answers - call
  OutContentStreamToBuffer, which reads it into OutContent. A handler stays
  portable and only loses the memory benefit there, mirroring how OnBodyDownload
  falls back when it returns nil. Rejecting outright would have been the other
  option; I picked the one that does not break a handler moved between server
  classes. For http.sys this happens inside SendResponse, so that an
  OnBeforeRequest answering early cannot bypass it.

Ownership and error paths

The stream is owned by the request until SetupResponse hands it to the context,
and by the context afterwards. It is released on every path I could find:
ProcessDone after sending, Recycle on a reused THttpAsyncServer connection,
the new THttpServerRequestAbstract.Destroy if the connection dies first, a
second SetOutStream call, and the 416 rejection.
THttpAsyncServer.AsyncResponse discards a pending stream before assigning its
delayed content, otherwise the stale stream would win in SetupResponse.

Two cases are worth calling out, because I got both wrong at first:

• a handler which raises after SetOutStream had its body sent as the 500
  response, since CompressContentAndFinalizeHead uses ContentStream and never
  looks at the error page in OutContent. An error response now always discards a
  pending stream.
• a handler returning a non-200 status with its own stream body had that body
  range-processed - a Range: request could truncate a 404 body, or replace it
  with the generated 416 page. Range handling is now gated on StatusCodeIsSuccess.

Note: THttpServerRequest.Destroy previously did not call inherited (it was
void); it does now. An external descendant overriding Destroy without calling
inherited would leak - none exists in this repository.

Tests

TNetworkProtocols.DoHttpOutStream runs the same steps against both
THttpServer and THttpAsyncServer: a 3MB body from an owned stream, a range
request with its Content-Range: header, an unsatisfiable range, an explicit
shorter length, a replaced stream, a caller-owned stream which must survive, a
handler which raises after SetOutStream, a non-200 response with a range, a
stream supplied already positioned (with and without a range), a partially
consumed TStreamWithNoSeek, a stream whose Position raises without it being a
TStreamWithNoSeek, and one which does answer Position/Size but raises on the
actual Seek().

Those last two are deliberately separate, because they exercise different guards:
the first is caught in SetOutStream and downgrades the stream to
Accept-Ranges: none, the second is caught in ContentFromStream and answers 416.
Keeping them apart also keeps the test compiler independent - GetPosition is
virtual under FPC only, so a helper stream which raises on Seek(0, soCurrent)
alone would take one path under FPC and the other under Delphi.

Ownership is counted, not assumed: the test streams increment a counter in their
destructor, so a leak or a double free fails an assertion. Every claim was
verified by reverting the corresponding code and confirming the matching
assertions fail - on both server families separately.

What was measured, and what was not

I develop on ARM64, where x64 only runs under emulation, so everything needing a
native x64 result was run on separate hardware (Windows 11, i9-14900K, FPC 3.2.2
x86_64-win64, Delphi 13 dcc64 and dcc32).

• FPC 3.2.2, aarch64-linux and x86_64-win64: TNetworkProtocols green and stable
  over repeated runs.
• Delphi 13, Win64 and Win32, actually executed: HTTP and _SocketIO green
  over 30 runs each. Win32 matters here because the whole path is Int64 sized.
• THttpApiServer (http.sys), exercised directly rather than through the suite:
  a 1MB stream arrives byte identical, and a 3MB one - above
  HttpContentFromFileSizeInMemory - answers 500 with an empty body instead of a
  truncated one.
• REST-over-WebSockets, the same two cases through TWebSocketServerRest, same
  result. That is the path OutputToFrame gained, and it had never been executed
  before.
• Still not covered by a test: HTTP pipelining, Recycle cleanup of a pending
  stream, and a connection dying between the handler and SetupResponse. I traced
  them by reading the code and believe they are sound, but no assertion proves it.

That x64 round also caught a real defect, fixed in the current commit: a test
helper stream raised on every Seek(), including the harmless Seek(0, soCurrent)
that Delphi's TStream.Position uses, so /failseek took a different branch per
compiler - 30 out of 30 Delphi runs red, green under FPC throughout. The shipped
code was correct on both; the test expectation was not.

Open questions, where I would follow your preference

1. Non-seekable detection. Settled by your review: automatic detection
   stays, and hosNoRange is there for a handler which already knows.
2. The other backends. Settled: they buffer into OutContent.
3. 206 status selection. SetupResponse picks 206 from rfWantRange alone,
   before knowing whether a partial response actually happens. That never shows
   with files, since a file always validates the range or fails with 416; it
   surfaced here for the non-seekable and non-200 cases, which I handled by
   clearing rfWantRange. If you would rather have the status follow rfRange,
   that is a one line change in SetupResponse and both workarounds can go.
4. 416 without Content-Range: bytes */size. Unchanged from ContentFromFile,
   so this proposal does not alter it - mentioning it only because RFC 9110
   recommends the header.

My use case, for context: file contents stored as 4MB chunks across sharded
SQLite databases, so there is no file to point SetOutFile at, and a temporary
file per download would be pointless I/O.

[synopse]

1. About InheritsFrom(TStreamWithNoSeek) and booleans: why not just use a set?
   It sounds simple, readable and expandable. [] as default then proper options.
   For the two methods setting the TSTream.
   But keep the TStreamWithNoSeek detection anyway because it is as expected.

2. Buffering the response could make sense - only if it is not too big in size.
   Use the regular size limit for responses.

3. I am no RFC expert. But I guess 2xx means success and 4xx means error. Otherwise, the final code is less decisive.

4. I don't know. Perhaps in the future if someone complains about it.

[synopse]

make it inline

[synopse]

perhaps just these lines are enough:

 if ifOutContentStreamOwned in fInternalFlags then
    FreeAndNilSafe(fOutContentStream);

[synopse]

I guess a FastNewString + ReadAll() could make the trick since we have fOutContentStreamLength

[synopse]

only accept 200 is perhaps too restrictive

[synopse]

you may have inherited from TRawByteStringStream and override only Seek/Write

[synopse]

I like those tests. :)

[synopse]

I am very pleased with the current state.

Should we merge?

[landrix]

I'll let you know; I'm running a few more tests.

[landrix]

Both hunks I flagged as unverified are now exercised, on native x64 hardware
(Windows 11, i9-14900K, FPC 3.2.2 x86_64-win64, Delphi 13 dcc64 and dcc32).
I have updated the PR description accordingly.

http.sys and WebSockets, tested directly rather than through the suite, since
no test drives a SetOutStream body through either. Same two cases on both, with
HttpContentFromFileSizeInMemory at 2097152:

small status = 200
small length = 1048576 (expected 1048576)
small match  = TRUE
big status   = 500
big length   = 0

So the 1MB stream is buffered and arrives byte identical, and the 3MB one answers
500 with an empty body rather than a truncated one.

Delphi, executed for the first time rather than only compiled: HTTP and
_SocketIO green over 30 runs each, Win64 and Win32. Win32 was worth doing - the
whole path is Int64 sized.

That run caught a real defect, which is why the branch moved. My TRaiseSeekStream
raised on every Seek(), including the Seek(0, soCurrent) that Delphi's
TStream.Position uses because GetPosition is only virtual under FPC. So
/failseek was caught by SetOutStream under Delphi and by ContentFromStream
under FPC - 30 out of 30 Delphi runs red, green under FPC throughout. The shipped
code was right on both compilers, and defensively so under Delphi; only my test
expectation was wrong.

Fixed by making the helper raise on Position under both compilers, so
/failseek now asserts the SetOutStream guard, and by adding TNoJumpStream
and /failrange for a stream which knows its position but cannot jump - that one
asserts the ContentFromStream guard and its 416. Both were checked by reverting
the guarded line and confirming exactly those assertions go red, on both server
families.

One aside, unrelated to this PR: TWebSocketProtocol.SetEncryptKey returns
silently when aSettings = nil, leaving encryption off entirely. A client built
by hand that way talks plaintext to an encrypting server and every frame is
dropped with sprBadRequest. It cost me a while to find; a hint in the doc
comment might save the next person the same detour.

[landrix]

So, from my perspective, everything is ready for now.