I'm surprised the pencil rubbers work, but I have used them to clean up after using the pen rubber so I guess they do have some sanding property.
Though, when you say awhile, how long are we talking about - ad break, tv episode, movie, marathon evening, days, weeks...
As for the pen rubbers, they're a thing of the past here. Unless there is some specialty store that has them, supermarkets, stationers don't carry them anymore. I'm surprised they were still made after correction fluid (twink) came out. I only used them a couple of times and they were really bad on the paper surface.