If you are like me, then in building your RPGs, you have thought "I bet I could invent the perfect dice resolution mechanic."
Well, you are wrong. Not only by definition (no mechanic can be perfect), but also pragmatically. Until the next Pythagoras of dice comes along and figures out how to resolve RPG events in some enlightened way, we are pretty much stuck with our choice of the existing systems. There are a lot, and they yield unique, but also fundamentally similar probability distributions.
So, barring the search for a perfect dice system, why don't we try and layout everything the d20 can do (other than be rolled and have static modifiers added to it)? So next time, when you are making a system, you can explore your options within one die instead of trying to compare all the different systems.
Here we go...
Gambling: I'm going to list this one first to get it out of the way, because it can be applied to pretty much any of the following options, or implemented on its own as well. Simply, the players, and possibly GM, gamble on what the outcome of a roll will be. Maybe you could bet Hit Points, and get some kind of return in damage or outcome on a success. Maybe you could use the Price-is-Right method of guessing under, and being as close as possible, without going over. You could have a few different kinds of bets to choose from, like Craps, and each one has a certain probability for a certain return. Wanna bet odds for you attack roll? Okay, that's 50% probability, for 1d6 damage. Wanna bet only 18-20? Okay, that's 15%, for 2d8 damage. The options are countless here. Also, gambling is fun and reasonably stressful. That can be great for combat, at least.
Crits and Fumbles: These are almost universally used. Typically, a roll of 1 is a fumble and a roll of a 20 is a critical. You could always change which number means what. Maybe hitting the target number exactly is a critical hit, and a 20 is just a success as per normal roll. Maybe they are both crits? So now your crit chance is 10% for numbers 19 and lower, 5% for 20, and 10% for numbers 21 and above. Weird.
Expanded Crits and Fumbles: Check this out. You could also use this idea for a different kind of bonus. Forget the "GM's friend" of + or - 2 to adjust the roll for any situational things. Instead, expand the critical or fumble zone by 2 (so 18-20 crit, or 1-3 fumble). Now, taking a risk isn't about changing normal success and failure probabilities, it's about changing extraordinary success and/or failure probabilities.
Advantage & Disadvantage: If you don't know what this is, or haven't downloaded the Basic D&D Rules over at Wizards, you should. What I love about this mechanic is how self-contained it is. I suppose you could expand it to use 3d20s, if you were a hedonistic psychopath...
Evens & Odds: I first saw this in 13th Age, where certain character attacks/abilities can only be used on evens or odds. It is particularly interesting when you consider that some things can be used on even or odd misses, which is a great way of adding a silver lining to a failed attack roll. Damn, I missed the goblin...But hey! I can cleave into the next guy for 3 damage, because I rolled an even number! Huzzah!
Lucky Numbers: Again we return to the renowned Zak S. for his creative kung-fu mechanic that can totally be applied to anything your heart desires. It's like evens and odds.
Success+ (plus): I got this from a mechanic called "boosting" from Within the Ring of Fire. Essentially, the higher you roll in that system (which is an exploding 2d8 system), the more damage you deal with your attack, or the cooler stunts you can perform with skill checks, etc. You could do it as a 1:1 excess roll converts to damage thing. You could do it in increments that create greater and greater effects, like for every 3 points above the target number you can choose to do a stunt, which get stronger the more times you boost.
Failure- (minus): Same as above, but with failures and numbers below the target number.
Results fall within a range: This is a big change, but the idea is akin to this 2d6 system, but with a d20. You figure it out. Shouldn't be too hard.
Target is success, further is worse: In this one, you are aiming for the target number exactly, and each step away from it, above or below, is a little bit less successful.
Dice as Static Modifiers: My friend Max found a variant for 5e where the proficiency bonus is handled with a die that you roll in addition to the d20. So +2 proficiency would be 1d4, +3 would be 1d6, +4 1d8, +5 1d10, and +6 1d12. Granted, +12 is a bit high considering the original cap of +6, but you get the drift. You can do this easily for stats, weapons, classes (like a combat die, a magic die, a skill die, etc.).
Dice as Situational Modifiers: This one is new, as far as I can tell. Essentially, you use dice instead of the static bonus from a given situation. 1d4 for a moderate advantage, 1d6 for a good advantage, 1d8 for a serious advantage.
Dice as Modifier Resource: As above, but you get to add them on demand, instead of as a way of mechanizing an in-game advantage. Also, you only have so many per day, or per encounter, etc.
Reroll Numbers: This is from the way 13th Age handles two-weapon fighting. Simply, reroll all missed attacks on natural 2s. But you can expand on this indefinitely. Higher level fighters might get to reroll certain missed attacks.
Roll Under: Another big change, though you can implement it on only certain rolls to more easily describe improvement over time (roll under stats, for example). Goblin Punch's house system uses this mechanic really well.
Opposed Rolls, Higher Wins: Self explanatory.
Opposed Rolls, Difference is Result: As above, but you can add in the Success+ or Failure- stuff too.
Everything is save vs. 11: Why not flip a coin for every roll? Except don't flip a coin, because you can't adjust the probability of a coin flip. Instead, adjust the 50/50 chances with situational stuff, or Situational dice.
Everything is save vs. X: As above, but you choose a number other than 11 as your base for success. Make it 14 for a more difficult game, or 8 for an easier one, etc.
That's all I've got. If you've got anymore, leave them in the comments!