Reducing Memory Usage for Large JSON / Gzipped Responses

1

I’m sharing this mainly because I’m curious to find out how other people may have handled this particular sort of situation better (and if there aren’t better options then perhaps what I share below will be useful to you).

Over the summer I was working on a project idea and I was using Slim and also a few of the PSR-7 Middlewares available over here: https://github.com/oscarotero/psr7-middlewares

Including the Gzip one: https://github.com/oscarotero/psr7-middlewares/blob/master/src/Middleware/Gzip.php

The exact details are a little fuzzy at the moment, but the main thing I want to share is that during my development of this project, I was mainly issuing database queries and returning JSON responses containing the data retrieved from the database.

In most PHP applications this flow looks like this:

  1. Issue SQL query
  2. Collect rows into a PHP array
  3. Convert this array to JSON

Then with Slim I would pass that JSON into the response body (or simply passing in the PHP array using the withJson() method could be done instead). Then with the Gzip Middleware it would get compressed behind the scenes automatically.

The Gzipped response was fairly “small”…probably about 1.5 Megabytes worth of data. As JSON, the data was maybe about 35 MB. However on the PHP side, examining the peak and current memory usage values I could see that the peak memory usage was quite high (in the 150-200 Megabytes range, primarily due to the intermediate PHP array from Step 2 above that is created when the database results were retrieved).

Seeing this huge difference in size, and knowing that my intended result was simply to return the data as JSON anyway, I started looking for ways I could bypass the intermediate PHP array stage so I could go straight from issuing the SQL query to a gzipped response.

This resulted in me no longer being able to use that Gzip Middleware though unfortunately, but it did eventually result in this little composer package from me that I could use to go straight to a Gzipped JSON string as I’m retrieving the data from the database (it’s my first Packagist submission so hopefully others find some use for it):

This separate repo I created with some stats on how it can reduce memory usage (at the expense of increased processing time):

The main use case I’ve had for it has been for the situation I described above (trying to put a sizable amount of database result data into a small memory footprint), but it can probably have some other use cases too. I wish there was a way it could be made a bit faster (let’s say closer in speed to the built-in gzencode() function) but I’m not sure if that would be possible at this time.

So mainly if you have a project that needs to return a lot of data as JSON and you want to reduce its memory usage and you don’t really need to manipulate that data within your PHP code (you just need to return it) this little package might be helpful for you (and assuming the increase in time doesn’t cause any issues for your particular type of application).

use Orware\Compressed\CompressedString;

$compressedString = new CompressedString();

// You can pass in a PHP array or object and it will get automatically JSON encoded:
$list = [1, 2, 3, 4, 5, 6];
$compressedString->write($list);

// Gets the Decompressed String:
$decompressedString = $compressedString->getDecompressedContents();

// Writes the Decompressed String Directly to a file:
$compressedString->writeDecompressedContents('tests/files/appended_test_decompressed.txt');

// Writes the Compressed String Directly to a file:
$compressedString->writeCompressedContents('tests/files/appended_test_compressed.gz');

// Gets the Compressed String (although you can simply pass the object around instead, since the following would create a copy of the Gzip string):
$compressedString->getCompressedContents();
1 Like