The Invalidity of Atheism
Yeah it's not a problem.
Logic itself isn't really a meaningful statement with no predication. Classical logic, which you may be referring to, asserts the law of non-contradiction and law of excluded middle (along with law of identity and maybe a few others depending on the specific language). Fuzzy logic denies those two former laws and we use it in electronics and there are tons of logic languages besides that.
I'll go a bit further and say everybody uses their logic language of choice, even if they don't know it or contradict themselves, so saying "logic is my epistemological choice" is trivial at best.
As for the assumption that the world is material, math cannot be material at least epistemologically. That is to say it may in fact be ontologically material but we have no way to approach that from this limit that physics necessitates math to do physics. This is an asymmetric relationship where we don't need physics to do math (as that would be circular).
In addition, modern math/logic is based on the principle of being more universally applicable than to material objects (as shown by frege's reasoning here).
Also physics is very much a philosophical endeavor and was called natural philosophy (as a group name with chemistry, biology etc) until a century and a half ago.
Also you can't even reject theism with atheism. You're using naturalism, materialism or whichever frameworks you're using. There's no way to get to an "atheist" position ontologically or epistemologically.