Hmm, no one? If anything is unclear, let me know.
So it boils down to if it is a request with Varnish it needs to assemble the routes urls from X-Orig-Host instead of HTTP_HOST. So I think I am going for overriding the UriFactory where the Uri is constructed.
Any opinions on that?