I don't recall the exact amount of dust load, but I can remember Dyson advertising that after 10oz of dust (or whatever the amount was), that there was a significant drop in air flow for some competing bagged vacuums. Provided both vacuums are treated the same, there is no need to always test with empty bags. It certainly gives the bagged vacuums an advantage. Testing with empty bags provides information, but so does testing at various dirt loads. Put both test vacuums in canister mode and suck up 10 oz of dust and re-run the test. Also, provided both vacuums are tuned up and of similar age, the test can still be informative (but with increased variability). Many have argued that Consumer Reports should retest the machines after say 6 months of normal household use, and I agree with them.
Consider a vacuum like a Sebo with a geared belt. It's performance doesn't degrade as does a vacuum with a stretchable belt. Should the test reward the vacuum with the belt that stretches over time? If you are testing a vacuum that doesn't lose suction/airflow versus one that does, why is it only "fair" to test with an empty bag. The bagged vacuum may be superior for the first 8 oz of dirt, but the test doesn't tell you if the bagless is superior after that.
