I don’t have a problem with there being a single labor pool. The real issue is the player has no information on what they’re doing and no control over them. It’s simply over-automated. I posted an idea here Laborer information panel that I think will not only help with the logistics but also let you direct them to what you want done.
As for laborers traveling all over, you would think the logistics algorithm would be simple. Find the resource in the storage closest to where it is to be delivered, and then choose a free laborer that’s closest to that storage to transport it. But I don’t think that’s what happens.