addErrorMiddleware takes the following arguments:

 * @param bool $displayErrorDetails  Should be set to false in production
 * @param bool $logErrors Parameter is passed to the default ErrorHandler
 * @param bool $logErrorDetails Display error details in error log which can be
 *   replaced by a callable of your choice.
 * Note: This middleware should be added last. It will not handle any exceptions/errors
 * for middleware added after it.
$errorMiddleware = $app->addErrorMiddleware(true, true, true);

What’s not clear to me is…
By default, are the errors actually logged anywhere? If so, where?

Slim 4 uses the error_log function.

PHP stores error logs in someting like /var/log/apache2 if PHP is an apache2 module.
You can check the location of the log file using:

<?php phpinfo(); ?>