Picture a class feature like a plank of wood.
If you put two or three planks of wood together, you can make a pretty kick-ass springboard to jump off from. You might not soar through the heavens, but you'll go somewhere.
If you put several planks of wood together, they start to look a lot like a box. A box that closes you in. A box that forces you to by new books just so you can open up a little door that leads...
...to another box.
I found myself trapped in these boxes for years, ecstatic whenever game developers would release a new splat book with a fighter variant or non-spell paladin build.
Now, the boxes aren't all bad. They keep you grounded so that you can come to understand that not everything you think up is a good or game-appropriate idea for a character. They are kinds like guard-rails at the bowling alley.
But now you're an adult, and it's time to take the training wheels off and achieve your full pen & paper RPG potential! YEAH BABY! WOOOOH!
So here's how you should set up your class system for your kick-ass RPG:
Step 1: Don't split hairs
Barbarians? Yeah, are they really all that different from fighters that have a temper problem and eschew heavier armor? The answer is no, they aren't. They still just hit stuff with big metal things and get hit with other big metal things. They both fall solidly into the category of "Warrior" as far as I'm concerned.
Step 2: Hit all the right notes (hint, there aren't that many)
Great, now you've got your main classes, like Warrior, specified. Next, give it what every member of that category must have, and then stop. Uses big weapons? Sounds right to me. Good at fighting with those weapons? You betcha. Especially strong and athletic? Now you're talking. You've got a sweet little, WIDELY INTERPRETABLE class now. That's good. Stop. Don't add anything else. Really. Stop. Please.
Step 3: Keep things simple, even if that means adding more classes
Paladins. They could be a special warrior variant, or prestige class, or paragon path, or special feat-tree style multiclass variant option splat nonsense. OR...
You can just make a paladin class. So it uses weapons like a warrior. No big deal, the fact that it can stitch peoples' wounds up by laying on hands is awesome and unique. As is their immunity to disease. Fantastic. Stop. No more. What? A divine mount? Shut up. Take that fluff out of my sight and let the DM give horses to whomever he/she goddam pleases.
Step 4: Fill in the blanks
Now, make any other classes that your system calls for. This requires you to have the rest of your system done. You need to know it inside and out, and you need to know what mechanics you are willing to let classes specialize in, and which you are not. For example: letting Fighters specialize (a.k.a. have a monopoly on) feats in 3.5 was a mistake. It slowly made feats a combat-centric, fighter-only mechanic. Not good. Rouges as skill monkeys? Also not good. Why can't my fighter be a great mason? [some mumbled jargon about balance and such]. Bull. Fighters can't have great skills, not even in masonry and blacksmithing, because you gave rogues a monopoly on ALL skills, and you know it.
The point is: to have unique, solid, simple classes you need to draw some serious lines in the sand between your mechanics. Skills need to be entirely devoid of combat-related things. Yes, your knowledge of demons may help in a fight, but if the main purpose of any skill is to help you fight better, it might as well be a new class, because CLASSES should be about COMBAT. And feats? scrap em. As soon as you add a mechanic to your game that literally covers all the other mechanics and more, you've broken your own game. It has become a mockery unto itself. Spells? Spells are feats for Wizards. They get all the best ones, and all the other Wizard-wannabe classes get the scraps left over. Not cool. Magic should be available to every class, and your magic-users should just get MORE magic each sessions/day, not SPECIAL SECRET magic.
Let me return to that comment about classes and combat. Consider this: a leader class. Bard, Warlord, etc. Once you make a class that is designed to be a leader, every other class is, by definition, not a leader. Players will immediately hone in on that, and if someone is playing a barbarian and they want to lead the hordes on conquest, they will believe that they need levels of warlord to do so effectively. "What? How can I lead the hordes? I don't have Inspiring Word or Wolf Pack Tactics. The other leaders will run circles around me." When you stop making classes about what you did during the fight with the goblins and instead focus on what you did leading up to the fight with the goblins, you are telling the players that their character can shine only in combat, or only not in combat.
Every class should give you something to do in combat, and every character should be able to pursue any talents outside of combat, regardless of class. Bam. Game balanced.
See what I'm getting at? In a word:
Adding a class to your game should give that class a MONOPOLY on one part of your game, plus a couple other, MINOR, THEMATICALLY GENERAL bonuses. Everything else should be one-size-fits-all.
For example: When I added paladins to my game, and their main class feature was being able to heal peoples' wounds, I promised that I would never allow a healing spell or class feature anywhere else. If I did, it would disenfranchise the paladin class. I also promised not to try and make paladins as good at fighting as warriors. Yes, even against demons and whatnot.
Okay, phew. That's good for today.