I also see it drift in a dark room, everything is well for me when there's some light, so it seems enough for me.
The drifting is unpredictable though as you describe, you better be ready when it happens
You should probably also try to clean the pinhole while you're at it, I hear some people do...