Exactly what I think.
The miller prioritizes it because it will spoil quicker. It makes sense … but makes all the production chain crumble.
If the miller took the grain from the granary, the grain wouldn’t be on the floor in the first place.
Anyhow, I added granaries to cope with the burst of grain production so my millers don’t have this issue again.
However, I do think that it’s something the devs should be aware of. It’s nice to take what will spoil first, but should be limited to what is currently in storage.
I had the same problem with the trading post, where the trader would go get some wood lying down in the forest rather than in the stockyard just next door (so not necessarily prioritizing spoilage, but just what is not in storage). It completely destroys the production chain.