I LOVE SPELL LISTS AND SO CAN YOU!

Magic systems have what software engineers call a "user interface" (UI) and "system interface" or "back end".

Here's a computer example to help you understand UI vs. back end:

• The ATM at your local bank is a machine in the wall that has slots for money, slots for debit cards, some buttons, a screen, and some other stuff. When you go get money from the ATM, you are interacting with the UI.
• While you are blissfully pushing buttons and getting cash, there is other software running in the background that checks your pin number against the account it read off of your debit card. It then checks the withdrawal you are making against the balance in said account. These processes comprise the back end.

Now, here is an example using the magic system everyone loves to hate, Vancian magic:

• The wizard wakes up early in the morning and prepares spells from his spellbook. These spells each require some amount of (non-fungible) mental space and magical aptitude to prepare, therefore limiting him to a certain number of each level of spell. Once the spell is cast, it is gone from memory and cannot be used until it is prepared again. This is Vancian's UI.
• The player, meanwhile, reads a spell list on her character sheet and chooses which spells to prepare for the day by assigning them to the appropriate spell slots. If her character gets to learn new spells at this juncture, she reads those spells off of a larger spell list in some rulebook and chooses which ones are recorded onto her character sheet, and therefore her character's spellbook. This is Vancian's back end.

Critiques of D&D-style magic always seem to rail against one of these two halves of the system, either the UI or the back end. 

"Ewwwww," they say. "My wizard sucks so much when he doesn't have time to prepare spells. That's sooooo lame." This is a UI complaint.

"Uuuuuuugh," they spout. "Knowing all the spells available to me from the list is so banal. Where's the mystery?" This is a backend complaint.

The first claim is something I think everyone agrees on: vancian magic/spell slot magic/casting-overused-spells is boring as written. It's predictable in combat, and sitting at the table deciding what you will prepare for the day is always a drain on time and immersion.

Yeehaw! That UI can't have a spell-list back end...can it?

But you can add interesting spell components to solve the first problem, and crazy-awesome familiars or wacky spell-dream preparation mini-games to solve the second one.

That still leaves us with the list, however. A big list of everything a wizard can ever do. What usually follows in these critiques is some half-functional attempt at a magic-words system or a rune-magic system or something even more convoluted to increase "fun" and "spontaneity."

If you play D&D in any way even remotely resembling the way I play it, however, then you know these systems will either be trivially different from spell lists and have no impact on the feel of the game, or they will be so imbalanced or time-consuming that you'd be better served just shooting yourself in the foot.

This is not an opinion, this is a law of design: the more subsystems each individual piece of a game must fit into, the more confined you are to the design of those features.

Here's your damn spontaneous wizard!

For a case in point, see D&D 4E. Everything had to be a power, and therefore had to be used in combat, and therefore had to do damage and be useable a very specific number of times per unit-time. It's the Cocktail Weenie Approach. It has to stay on a toothpick. You can do a lot of things with that, but you also can't do a lot of things with that.

What is the single subsystem required to use a spell-list back end? Spell levels.

That's it. You can make anything you want into a spell in a spell-list system, and it will work so long as you assign it the correct level for your play style. Low magic games can move spells up a level or two, high magic games can move spells down a level or two.

Don't want PCs to be able to heal with magic? Take healing spells out.

Want cool blood magic stuff? Add spells with special blood-magic effects, no need for a new class or subsystem of magic.

Take a page out of the 5e book and let some spells be cast at higher levels than they are first listed to make them more powerful and useful in later play.

Make something a ritual by lowering the level and having it take 4 hours to do.

Make one spell a permanent and at-will invocation for a warlock-type character by forcing them to take a graft, or make a pact with a demon, or lose some max-HP, or lose some CON or STR to learn it. Allow wizards to become their spellbooks with magical give-and-take.

Okay okay okay...wait...I've got a knock... two comprehend languages, and... a ray of enfeeblement left.

You can't beat spell-lists and individual spell modules as the most open and creative magic system back end out there. You just can't. Every other kind of "build-it-as-you-go" back end for spellcasting has a mathematically provable limit to the number of spells you can make, and will always have entire "dead zones" of magic effects that just aren't doable with the given selection of runes, power words, range-damage-shape point allocations, etc.

So if you want a more interesting magic system than D&D Vancian as written, here's what you do:

1. First and foremost, scrap all the over-used spells that bore you or (in your opinion) ruin the game. You don't owe your players any access to particular spells SO LONG AS you tell them ahead of time that you will be changing the spell selection.
2. Now, put in a bunch of awesome spells and spell-features (like casting at higher levels or using blood to amplify their effect). Boom. Bad ass.
3. For the love of god, don't just have your wizard pour over their imaginary spellbook for 20 minutes of out-of-game time to prepare spells. Make it interesting! Use familiars instead of spellbooks. Use dice rolls to randomly dole out spells for each level. Something! I'll post other ways to do this in the future.
4. Give your magic-user players something to shake up their usual spells during the game. My vote is for letting them use any crazy component they find as part of a spell if it sounds like it might make sense. Nightmare ashes in a fireball? Hell yeah. Bat guano in your consecrate spell? Um... no, I don't think that does anything.
5. Be generous. Magic is awesome and scary, whether you are in a low magic or high magic system/setting. Let fireballs ignite clothing and blow up buildings. Let magical flying be fast and acrobatic. If you want magic to be cool, you need to let it be dangerous and unpredictable in a narrative way, not a bunch of mechanical ways.