Slim4 - output buffering large files & zip streaming

Hello,

I’ve been searching the docs and forums and haven’t been able to figure this one out. It’s probably something simple that I’m missing. TIA for any leads.

Upgrading Slim2 to Slim4. I was sending large files by clearing my output buffers and writing to the output response directly in chunks. Slim was no longer handling the response at that point. I was just manually writing the headers, then the pieces of the file, so it didn’t fill up in memory.

Doesn’t appear to be the Slim4 way to do things, however. Now I’m opening the file, doing a ->getBody() then doing a body->write() for the chunks. It works but I think it’s just piling up the chunks in memory and then sending the whole thing.

What is the proper Slim4 way to return large files in low-memory use ways?

Additionally, I am building a Zip and streaming it. Same issue. I used to just send the headers and use a ZipStreamer library to write direct to output. How can I do this in Slim4?

I’m using Psr\Http\Message\ResponseInterface as Response, and I presume there’s some sort of other ResponseInterface implementing object I should be using, but so far I can’t figure that out!

Any ideas? TIA.

To handle large responses, I would recommend using PHP streams. Slim will then send this stream in chunks to the client.

$stream = fopen('filename.zip', 'r');

$response = $response
    ->withHeader('Content-Type', 'application/zip')
    ->withHeader('Content-Disposition', 'attachment; filename="filename.zip"');

// nyholm\Psr7
return $response->withBody(\Nyholm\Psr7\Stream::create($stream));

With slim/psr7:

return $response->withBody(new \Slim\Http\Stream($stream));

Example with maennchen/ZipStream-PHP:

use Nyholm\Psr7\Stream;
use ZipStream\Option\Archive;
use ZipStream\ZipStream;
// ...

// Create ZIP file, only in-memory
$archive = new Archive();
$archive->setOutputStream(fopen('php://temp', 'r+'));

// Add files to ZIP file
$zip = new ZipStream(null, $archive);
$zip->addFile('test.txt', 'my file content');
$zip->finish();

$response = $response
    ->withHeader('Content-Type', 'application/zip')
    ->withHeader('Content-Disposition', 'attachment; filename="filename.zip"');

// nyholm\Psr7
return $response->withBody(Stream::create($archive->getOutputStream()));

You can find more example here

2 Likes

Hi Odan, Thank you, your tutorials rock, BTW.

I’ll give this a shot!

Edit: Tried it, worked great. For my chunk writing I’m now just writing to a stream then shipping the stream.

1 Like

Howdy Odan,

Working on this code some more, and have done some more testing. It appears to still be creating files on disk and output buffering, rather than directly streaming a created zip to output.

What I’m looking for is a way to build the zip while streaming it, not build a zip and then stream it.

In the past I’ve had to hijack the response from slim to do this, but I’m trying to accomplish this within slim’s response. Is that possible? Can slim provide a response that is built on the fly, rather than pre-built and stored?

Thank you,

Russell

This is possible because php://temp will use a temporary file once the amount of data stored hits a predefined limit (the default is 2 MB). php://memory instead, always store its data in memory.

$archive->setOutputStream(fopen('php://memory', 'r+'));

You could also try to change the maxmemory parameter: PHP streams

What I’m looking for is a way to build the zip while streaming it, not build a zip and then stream it.

As far as I know, the PSR-7 interface is not designed for this use case because only the response object should store the complete body content. But, php://output stream is a write-only stream that allows you to write to the output buffer mechanism in the same way as print and echo.

$archive->setOutputStream(fopen('php://output', 'r+'));

Hi Odan,

“As far as I know, the PSR-7 interface is not designed for this use case because only the response object should store the complete body content.”

Does this mean I need to hijack the response and NOT use the Slim4 response mechanism in order to actually build and stream a Zip on the fly?

If so, what happens to the Slim4 instance when I simply start outputting a stream that is being built on the fly? Do I need to do something to close it, or gracefully exit?

I’m not sure what you mean with this. Can you show me an example?

use the Slim4 response mechanism in order to actually build and stream a Zip on the fly?

Theoretically, it is not a Slim-4 specific response mechanism, but a response mechanism according to the PSR-7 standard.

The specification defines the response stream as follows:

1.3 Stream - HTTP messages consist of a start-line, headers, and a body.
The body of an HTTP message can be very small or extremely large.
Attempting to represent the body of a message as a string can easily
consume more memory than intended because the body must
be stored completely in memory.

This is what I mean with: “The PSR-7 interface is not designed for this use case”

If so, what happens to the Slim4 instance when I simply start outputting
a stream that is being built on the fly? Do I need to do something to close it, or gracefully exit?

Then the (Slim) Response Emitter is not able to send the HTTP response headers, because the output has already been started earlier. The response could be invalid then.

BUT, I found a working solution using a callback to produce the output (ZIP content) on the fly.

Here is the code for the callback-based stream implementation that works with Slim 4: CallbackStream.php

Usage example:

use App\Http;
use ZipStream\Option\Archive;
use ZipStream\ZipStream;
// ...

$callbackStream = new CallbackStream(function () {
    $archive = new Archive();

    // Flush ZIP file directly to output stream (php://output)
    $archive->setFlushOutput(true);
    $zip = new ZipStream(null, $archive);

    // Add files to ZIP file and stream it directly
    $zip->addFile('test.txt', 'my file content');
    $zip->finish();
});

return $response
    ->withHeader('Content-Type', 'application/zip')
    ->withHeader('Content-Disposition', 'attachment; filename="filename.zip"')
    ->withBody($callbackStream);

Howdy Odan!

Thank you, this has been very helpful. Looks like the modern incarnation of that CallbackStream response class is now found here:

I’m not sure what you mean with this. Can you show me an example?

I simply mean sending headers and response directly from the called function rather than returning the Response $response object back to the Slim App.

Theoretically, it is not a Slim-4 specific response mechanism, but a response mechanism according to the PSR-7 standard.

Thank you for explaining that. I’ve been using them interchangeably. I appreciate your clarification.

Then the (Slim) Response Emitter is not able to send the HTTP response headers, because the output has already been started earlier. The response could be invalid then.

OK, this partially answers my question. Does the response being invalid have any consequences to worry about? This may be moot now with the Callback method, I’m testing today.

Gah. Getting some errors that I don’t yet understand.

Laminas\Diactoros\Exception\UnreadableStreamException: Callback streams cannot read in /var/www/vendor/laminas/laminas-diactoros/src/Exception/UnreadableStreamException.php:28
Stack trace:
#0 /var/www/vendor/laminas/laminas-diactoros/src/CallbackStream.php(146): Laminas\Diactoros\Exception\UnreadableStreamException::forCallbackStream()
#1 /var/www/vendor/slim/slim/Slim/ResponseEmitter.php(124): Laminas\Diactoros\CallbackStream->read(4096)
#2 /var/www/vendor/slim/slim/Slim/ResponseEmitter.php(56): Slim\ResponseEmitter->emitBody(Object(Slim\Psr7\Response))
#3 /var/www/vendor/slim/slim/Slim/App.php(201): Slim\ResponseEmitter->emit(Object(Slim\Psr7\Response))
#4 /var/www/www/index.php(326): Slim\App->run()
#5 {main}

It may be that the laminas deal isn’t a good lead. Back to the example code.

It’s working using your Gist code. :slight_smile:


  /**
   * Execute the callback with output buffering.
   *
   * @return null|string Null on second and subsequent calls.
   */
  public function output()
  {
    if ($this->called) {
      return;
    }

    $this->called = true;

    ob_start();
    call_user_func($this->callback);
    return ob_get_clean();
  }

Question, do you think this means it will buffer the zip stream before sending it?

Good catch, this output buffering is not needed and should be removed.

I just fixed the CallbackStream file. See here:

This appears to be doing exactly what I wanted it to do, build the zip literally on the fly. You rock!

1 Like

I think this is a misreading of the standard. It’s the reasoning for using streams instead of strings, not a description of the streaming itself.