This often has to do with the holes not being drilled directly opposite each other. I am in favor of drilling them together with double-sided tape before sanding them to the desired shape. You can also try to hit the pin with a hammer in the opposite direction of the deviation. So if it deviates to the right, tap the pin on the left at the back
On the rare occasion I make my own scales, I do exactly that for exactly that reason. I also sand the scales together: I draw the rough shape of the scales on the wood, including the location the pins should go, then tape the two pieces together, drill the holes, saw them in the rough shape and then start sanding. This way, they take on shape together, and the pinholes will stay in the same position to each other.