Nearly every person I've ever played D&D with is my close friend, and I love each of them very, very much. Having said that, I absolve myself of the hard feelings that might come with saying this: some of them are bad players.
By bad, I don't mean "they roll low all the time," a la Wil Wheaton, nor do I mean "they don't have an expert grasp of the rules," and in fact, those of my friends who understand the rules the least, and roll the lowest (on average) are often the best players. Rather, I mean they make uninteresting choices at nearly every opportunity, and count on the scaling within the game to make their character more effective, rather than counting on their own problem solving instincts.
There are several things that exacerbate this problem, such as lists, overly complex combat rules, and cookie-cutter encounters. Another mechanic that pours salt in the wound is unbounded numbers.
When bounded accuracy was announced as a major feature of D&D 5, I was very excited. I thought that bounded to-hit numbers, combined with a simple advantage/disadvantage mechanic, would result in a huge increase in player creativity at the table, particularly in combat.
But, I was wrong.
Things stayed the same, because damage continued to scale with level like it always did, and even worse, magic item bonuses are not calculated into the monster challenge rating math anymore, so most creatures will likely have less Hit Points than they should for the level of characters you throw them at.
That means monsters aren't all that hard to kill for characters of the appropriate level...
Which means just attacking and standing still is likely to win you the fight, all the math considered.
That's not my kind of game.
In my kind of game, players will likely die if they simply choose to attack a creature head-on, after level 3 or so. Goblins and dire rats and other such level 1 and 2 fodder shouldn't be a serious problem unless a character makes a bad mistake or gets totally surrounded. That's what intro levels are for: establishing a feel for the world, the game, and the ground rules of life-and-death. But once you enter larger monster territory, even just burly things like bugbears, the fear of god should be in the PCs if they don't have a plan that involves something more effective than "I swing my axe."
This means using the environment, taking creatures by surprise, enlisting further help from some townsfolk or mercenaries, or counting on a specific magic item to save your ass (if you're desperate).
A system can help you create combat that requires more creativity from your players, it just needs to be built keeping one thing in mind: characters should likely die in a fair fight against higher level creatures.
That's how you advance in a tabletop RPG: the character improves only enough to justify making the low level and high level monsters mechanically different, AND the PLAYERS get better at thinking about and approaching combat, and further analyze (in-game) how to increase their chances of survival.
A fighter character's sword skills should only become marginally more deadly through his tenure as an adventurer. But his combat skills should become MUCH more deadly, and this is only truly manifest via player improvement, not better numbers on a character sheet.
What you may notice is that this approach to designing a system would make random encounters particularly dangerous. While the discussion about the merits of random encounters is a whole different conversation, I will put this forward: good players should be able to get some kind of upper hand no matter what you throw at them, nor when. This doesn't mean they will win, or even survive, but it means that they will find something to work with.