Perhaps @akrabat will chime in, but you can organize your dependencies however you would like. You are correct, action factories are essentially controllers that only have one method. The advantage of using a single method like that is that often not all of the controller’s methods will have the same dependencies so you can give the class/method only what it needs and nothing it will not use.
I couldn’t tell you if there is any strict difference between a service provider vs a service factory. I tend to think of service providers as being pretty simple to configure
new App\Some\Class() while factories needing a function or additional configuration to build. But I really don’t know for sure.
They are all entries in the container, so organize them in whatever manner makes the most sense to you and your app.