Never too fussy, but ultimately you have to be satisfied with the end product. You will see things in a finished piece that non-turners will never notice. I usually sand through the first couple of grits, wet the piece with water to raise the grain and sand it back. Often helps with sanding out scratches but also gives a peek at what it will look like with a finish on it.