I tore it down again - ugh - I took the brushes out and turned the motor by hand while listening to the bearings with a mechanic's stethoscope. Aggravatingly, the front ball bearing was rough. I drilled the rivets and replaced it... again. Good thing I had those in stock. But, there is still a roughness in the back roller bearing, when you put some radial force on it. Which, you know, this thing has two stretchy belts putting tons of radial force on it all the time.
I think electroluxxxx is right, roller bearings are just not smooth. It's a brand new bearing, it was not dry, and it's a Koyo bearing, not a cheapie like the other one.
Also, irritatingly, the way the brush holders are designed, the commutator is always sort of turning against the grain, which makes noise on its own. Add to that the noisy bearing, and tons of moving parts, and I suppose it's just a noisy machine and that's that.
I also balanced the fan blade, it's all chewed up. Didn't take much to get it right. I found that I had installed it wrong. I had three washers under it, it was supposed to be two under it and one on top, under the nut. Don't know how I messed that up. But still, it wasn't rubbing or anything.
Incidentally, I inspected one of those brushroll bearings a little closer. I want to say it's got rust pitting, but the reality of it is that it looks like it was either not machined to any tolerance or just not machined at all. It almost looks like it was cast or hot formed... I'm not sure what to call it. But it's evidently not machined with the same precision as you'd expect for a ball bearing.