Yes, It's hard on everything. Buy your pieces as clean as you can afford.I wouldn't cheap out and get rougher stuff with a view to sorting it out. If you want the medullary rays it needs to be quartersawn. Here's a sample page on the type of cuts:
https://www.thefurnituremarket.co.uk/what-is-a-medullary-ray-in-oak-furniture
The other thing to think about is will you be filling it for a shiny finish or going for the rustic look where the grain is in relief; farmouse table style. Oak is open-pored compared to other woods. TBH, I think, unless you are going for the rustic look, it's going to be very hard work, for you and your equipment. It's my favourite looking wood but it takes muscle and patience to work with. Oak is more akin to working with metal than wood. White oak is 30% harder than black walnut. Notice that cherry is one below walnut and cherry is spot-on to hand sand -walnut should be as well. If you are going for the burr walnut look, rather than straight-grain, you have to sand with the grain with each little section... this is a very patient job as well.
https://en.wikipedia.org/wiki/Janka_hardness_test