I'm going to claim that there are four qualities that make a good magic system:

  1. Depth. It should give rise to complex usage and far-reaching implications through relatively simple mechanics. A parallel can be drawn here to designing a strategy game.

    A really good example of this (and honestly just the best magic system I've ever seen overall) is the one in the Mistborn books. In these books, some people are born with Allomancy, which allows them to consume small bits of metal and then "burn" it for magical power. Each metal gives you a different effect, and they come organized in three dichotomies: internal versus external, mental versus physical, and pushing versus pulling. So for example, the internal physical pushing metal, pewter, makes the Allomancer's body much stronger and more agile while burning it. The external mental pulling metal, zinc, magnifies the emotions of others, which can be used to manipulate crowds and individuals. The external physical pushing metal, steel, allows you to push metal away from you - or, if the metal object is heavier than you, pull yourself toward it. This creates so much depth of interaction when we see Allomancers carrying around sacks of coins to use as projectiles and specialized soldiers called "hazekillers" that use wooden shields and weapons and wear no metal in order to better combat Allomancers.

    You can see that Allomancy has a consistent theme: metal is power. This is built on later in the books when The Atium stash is revealed to be Ruin's body. Additionally, since the powers are organized in a rigid system of dichotomies it doesn't feel like eight separate things magic can do. It feels more like three ideas from which eight powers follow. Contrast a system like in my novel Pillars of Life, where besides the four energies and the sixteen elements spawning from them and the four states of matter, magic also has several miscellaneous powers. These ones (at least the ones besides telekinesis and destruction) feel tacked on to what was already a complete system. Sure, they add room for me to do cool stuff with the plot, but it would be better if I could do that with more compact rules. Pillars of Life magic also has no unified theme.

  2. Transparency. The reader should be told how the magic works, so that they can appreciate how cool it is, and so that it feels "fair" when you use the magic to solve a problem.

    A really good example of a system that messes up in this area is (I bet you know what I'm going to say) Harry Potter. These books are all about magic and yet we never learn how it works. Instead of having an inelegant list of powers like in Pillars of Life, we just don't have a list of powers. Instead there are an unknown but massive number of "spells" and whenever the plot needs something, the author makes up a spell that can do it. This has the effect that the reader can't possibly predict what's going to happen because they simply don't know what's possible in this world. But trying to imagine what might happen next is part of the fun of fiction. If something happens that you didn't expect because you didn't decipher the clues, that's a good story. If something happens that you couldn't have expected because you didn't know it was possible, that's just lame, especially if the POV character did know it was possible.

    There's also quite a few instances of magic that don't involve incantations, spells or wands, such as potion making, divination (logically impossible), portable time travel devices (logically impossible), and probably others that I don't remember because it's been so many years. These things feel completely separate from the wand, spell, and incantation-based magic that wizards use directly. It's even worse than the tacked on powers in Pillars of Life.

    And I have more criticisms. We never learn what the process of learning a spell involves (which is pretty important considering the story is about students at a magic academy). We also never learn what makes some wizards more "powerful" than others, or even what it means to be more "powerful" than another person in this magic system. How are we supposed to appreciate how cool the magic is if we don't even know how it works? Bottom line: Harry Potter is the epitome of a bad magic system. It has every flaw and no strength.

    Actually, there is an exception to this. Final Fantasy 13 is a good example. In this game, magic exists and we don't know how it works, but it isn't used as a plot device - instead we're quickly given the impression that it's really only useful for combat (Lightning's gravCon device is a piece of technology as far as I'm concerned), and because of that combined with the combat system being an abstraction of what's actually happening in the story, we don't really need to know how it works. But if you're writing a novel and not a JRPG, odds are you intend your magic to be more than a combat asset, so you'll need to explain it.

  3. Balance. Magic should be powerful enough that it plays a large role in the plot, but shouldn't be so powerful that it completely outmodes technology and makes mages invincible.

    This one doesn't need too much of an explanation. If magic isn't powerful enough to be a central plot device, why is it even in the story? (Although I've never actually seen a magic system with that flaw I thought I'd mention it for completeness.) And if it's so powerful that it's the only plot device, then every conflict turns into a brute force contest to see who has the most magic. I think Dragon Ball is a good example of this. I haven't actually seen the anime but I've played Dragon Ball FighterZ, which shows me that characters can fly, shoot massive beams of destructive energy out of their hands, and when a fight ends, it's often by someone being launched away with so much force that they crash straight through multiple buildings. It's hard to take it seriously.

  4. Two-sidedness. The best magic systems, instead of giving great power at small cost or small power at small cost, give great power at great cost. A parallel can be drawn here to desiging moral dilemmas.

  5. Power is fun, but it's more fun when it has an interesting cost, limitation, or risk associated - especially since if you don't consciously do this, magic is likely to end up incredibly overpowered (like it did in Pillars of Life). In one story I created once (it's scrapped now), magic can get very powerful, but it costs wakefulness. The really cool thing about it is that the cost depends only the magnitude of the effect you achieve and not on how powerful you are. This means that when the characters were first gaining magic and it was weak, the effect was so small they didn't even notice, but I had planned that later in the story, when they became titans of magical power that could take on armies, fighting for just a few minutes would sap all of their energy and they would need to sleep for several hours to recover. I'm sure you can see that that's more interesting than just letting them use magic endlessly for free. It also served the plot in at least one way - it stopped the heroes from being invincible later in the story.

    Another example, from a story I have planned but probably won't be getting around to for a while, is built on the premise that each person lives many lives in many different worlds, and what's happening when you sleep is you're essentially signing out of one world and into another. People don't know about this because their memories are compartmentalized. Still, that's what dreams are - your memories from other worlds leaking a bit. So when you use magic, the barriers keeping your memories separate get a lot weaker, and you start to remember your other lives. This seems like a good thing... until you get so deep into it that you can no longer tell the difference between your different worlds. You remember things that didn't happen, try to apply knowledge of other worlds to your present one, and, from the outside, it just looks like you're going insane. You effectively are.

The next I want to talk about is stock powers: abilities that are present in a lot of magic systems in some form or other.

I guess another thing I might as well talk about while I'm here is stock costs of using magic.

Here's another question you'll have to answer when you design a magic system: how do you get magic? If it has a cost like consumption of a rare substance, it might be okay to say everyone has it innately, but in most cases I expect you'll want to limit magic to certain people.

I'll probably come back and expand this article later when I think of more to say, but for now, I guess I'm done. I hope my ideas help you to write a good novel.