I run into this on end grain a lot, it just seems to require more time sanding than the face of the board. I'm no expert, but I'd go back through the grits of sanding on the end grain. to remove the saw marks.
It's looking like a nice box though, I'm always a fan of curly maple and walnut together.