Since last session where it was very obvious there was some kind of issue, I've been trying to think of a solution. My initial thought is to simply have the rogue player announce their sneak attack or ranged double strike and then roll two d20 and if they get one success then it succeeds as a normal attack, however if both rolls are successful then the rogue has accomplished the special attack as they stated. I like this because it's simple but I realize that it doesn't reward stealth except as a damage bonus. It does however factor in AC which is supposed to fold in all those kind of abstractions that would be relevant such as target reflexes, situational awareness and toughness. It doesn't mean an automatic success (although it does give a big bonus to the basic attack success) but it scales up well as the character's attack bonuses increase and when opponents are tougher as well. It also kind of fits nicely with the missile double strike portion of the ability. I haven't thought of anything else that I like better yet either. Perhaps the there is a better way to organize the stealth roll and to hit roll in the existing rule so that the problem doesn't arise, but that doesn't mess with the statistics as much as this double roll mechanic will. But I'll take elegant over minor probability changes in most cases.
Also, back to problems with the original rule, what I meant by 'combat engagement' in the original text is not obvious. The intent was that once per opponent per combat the rogue would have one chance to pull off a surprising move that would confuse their foe. It was a way to bake in the tricky but make sure that in the long run it wasn't going to usurp the fighter role of going toe to toe. A rogue in combat with a number of foes could only surprise each particular opponent once. It should also encourage rogues to change targets.
This then might be a better rule for the section on surprise attack in the rogue description.
In combat a Rogue may attempt to perform a 'surprise attack' by rolling two d20 for their initial attack on an unsuspecting foe. If one die roll is successful then they succeed in their attack as normal, but if the second is also successful, they may either add their Subterfuge skill to the damage of their melee attack or perform an additional strike with a ranged weapon (provided they have an additional one available).I would like to put in something more concrete about the attack being novel or unseen or otherwise unexpected but I don't know how to do that without getting back to the problematic skill check. Also, how this interacts with the rules for critical hits and fumbles I leave up to your imagination for the moment.