Why there must be free will I am not denying the reality of physical mechanisms, far from it. But what you are doing is denying the reality of subjectivity because you cannot explain how consciousness interacts with matter. What I am saying is that, taken as a whole, the sum total of the world as we experience IS, in fact, an instantiation of the phenomenon of consciousness interacting with physical reality. Our material universe now is infinitely more complex than that of our essentially physically identical ancestors 100,000 years ago. And in measurable, physical terms. So the cumulative history of humanity is itself evidence of the influence of mind on matter. Consciousness defines itself experimentally via the success/failure of it's ongoing interactions with reality, and codifies these experiments as the product which is culture.
It's Idealism, Dilthey really brings these themes to the fore for me.