"Some players, Mr. DM, just want to watch the world burn."
- Someone afraid of sandbox games
"Players don't panic when things go according to plan. Even if the plan is horrifying."
- Someone afraid of railroad games
Oh that unanswerable question: railroad, or sandbox? So short, and so mystifying. Within its seemingly binary choice are paradoxes and contradictions enough to fill entire college courses. See: Free Will vs. Determinism.
But let's boil the idea down to this: every DM wants their players to feel as though they are making choices that impact the world, but how do you do that without needing to literally create an entire world spontaneously?
Some will argue that you make a little "sandbox" and let the players do whatever they want in that little area. Easy. My response is this: even generating a tiny area that is a "free-roam" zone for the players is an unimaginably huge task. The idea doesn't ever scale down. PCs can cover a lot of ground in one session. You can't pre-generate all those possible paths, even in a very limited sandbox. And if you're thinking, "you don't generate everything the PCs can do, just the things they are likely to do" I say, welcome to a railroad game....
Meanwhile, if your argument is that you create some basic details and improvise the rest, you've entered into a similar paradox. If you magically incorporate your details into any path the PCs take, then you are running something VERY close to a railroad game. Best described as the "Quantum Ogre" effect. On the other hand, if you are willing to entirely abandon what you have prepared for the sake of maintaining player agency, then why prep at all? What are the odds that your players will choose exactly what you have prepared for them without railroading them at all?
To further confuse things: For every player that will purposefully avoid a pre-built story for something they think will be more exciting, there is a player who will do the opposite: ignore any hook or moment of inspired DM roleplaying for the never-ending slog through what they see as the "story."
The only way to appease both of these player types is to give the entire decision process over to the players, and then hold them to their choices. Your game must have cause and effect relationships, or the "watch the world burn" types will have no boundaries. Just so, the game needs to have very regular moments of choice, or the "go with the plan no matter what" types will quash everyone's creativity.
Okay, enough theory. Let's talk about putting this into practice.
Step 1: Prepare shit. There are no two-ways around it. You have to prepare sessions if you are going to give them any substance. Dynamic combats with interesting monsters, unique NPCs with believable spots in your world, competent antagonists, and many other parts of great D&D only thrive in the pre-game creative process. Maybe your random tables, acting, and improv are so good you don't really need to prep. If that's the case, I hate you.
Step 2: Accept that session 1 is railroad-y. Every D&D party has to suspend disbelief for long enough to accept that they are together, that they are at least allies in some respect, and that they will continue to work together until further notice. This is not an easy narrative step when your players are supposed to be brimming with "free will" and "agency." As the DM, you have to put your foot down and tell the PCs, "Welcome to session 1. Your characters are are all gathered in ______ for _______ and are planning to do _________. Go." Get the game going, regardless of whether every player is 100% into the mini-plot you have for the first session.
Step 3: Everything in the world has space for a hook. You have to bake hooks right into the body of your game world. Every corner should conceal another hook. But you need to expand your definition of what a hook is. Magic items are hooks. (What is it? Who can tell us? Where are they?) Information is a hook. Overheard conversations. Letters. Gossip. Everything. (The baron's raising an army? Why? Where? Someone mentioned the God of Socks. Who is that? Where can we learn about such a deity?) People are hooks. The dark spells cultists use are hooks. Statues are hooks. Paintings are hooks. EVERYTHING CAN BE A HOOK. Write a sentence for each good one you think up, and remind your players that these hidden treasures of adventure are everywhere if they pay attention and ask questions.
Step 4: Pace your sessions correctly. This will end up being a whole other blog post, but the short version is this: be smart about how long to spend on things around the table, and when to initiate them. Big combat after 4 hours of roleplaying session? Probably a poor idea, especially if it's getting late. Remember: you can always stretch things out with good roleplaying and description. You cannot, however, speed things up arbitrarily. When players focus on something, the time management is under their control. Prepare enough content to fill 50-75% of your session, then stretch it as needed.
Step 5: Finish your sessions at a choice, and have the players make it before ending. This makes #4 doubly important, because step 5 depends of successfully executing step 4. Each session should contain minimal "choice" points at the beginning and middle, because the more your players can veer off the path mid-session, the harder it is to make that session tight and satisfying. BUT, a major choice about what to do next that is placed at the end of a session allow players to tell you exactly what to prepare for next time. This choice point doesn't have to be "multiple choice" either. It can be "open response." So long as you have a week to prep what the PCs choose to do, they can do anything they want. Now THAT is agency.
Hooks are not choice points unless the players have nothing more urgent to do. Hooks can be kept in the back pocket of the party until some adventure time is freed up. This maximizes player agency, because it allows the players to choose the next part of their story organically, based on what most interests them about your game world. Free yourself from the trap of the escalating railroad campaign! You are better than that!
But let's be real: at the end of the day, even this style looks like a railroad. Pre-planned sessions that the PCs can't really change much once they are prepped by the DM.
But railroads can take you anywhere. Railroads (in the sense of pre-prepared sessions and stories) only become constricting when the players get an urge to jump off mid-way between stations. Solution: put in more stations. One at the end of every session. Keep the sessions focused and let them flourish into more depth instead of more plot if you have extra time in the moment. Your players will naturally want to follow mini-stories and plots to their immediate ends. Keeping players focused on a plot for one session is not an impossible task.
Trying to milk that one major player choice for three or four sessions of material is asking for trouble.