I'm going to claim that there are four qualities that make a good magic system:
Depth. It should give rise to complex usage and far-reaching implications through relatively simple mechanics. A parallel can be drawn here to designing a strategy game.
A really good example of this (and honestly just the best magic system I've ever seen overall) is the one in the Mistborn books. In these books, some people are born with Allomancy, which allows them to consume small bits of metal and then "burn" it for magical power. Each metal gives you a different effect, and they come organized in three dichotomies: internal versus external, mental versus physical, and pushing versus pulling. So for example, the internal physical pushing metal, pewter, makes the Allomancer's body much stronger and more agile while burning it. The external mental pulling metal, zinc, magnifies the emotions of others, which can be used to manipulate crowds and individuals. The external physical pushing metal, steel, allows you to push metal away from you - or, if the metal object is heavier than you, pull yourself toward it. This creates so much depth of interaction when we see Allomancers carrying around sacks of coins to use as projectiles and specialized soldiers called "hazekillers" that use wooden shields and weapons and wear no metal in order to better combat Allomancers.
You can see that Allomancy has a consistent theme: metal is power. This is built on later in the books when The Atium stash is revealed to be Ruin's body. Additionally, since the powers are organized in a rigid system of dichotomies it doesn't feel like eight separate things magic can do. It feels more like three ideas from which eight powers follow. Contrast a system like in my novel Pillars of Life, where besides the four energies and the sixteen elements spawning from them and the four states of matter, magic also has several miscellaneous powers. These ones (at least the ones besides telekinesis and destruction) feel tacked on to what was already a complete system. Sure, they add room for me to do cool stuff with the plot, but it would be better if I could do that with more compact rules. Pillars of Life magic also has no unified theme.
Transparency. The reader should be told how the magic works, so that they can appreciate how cool it is, and so that it feels "fair" when you use the magic to solve a problem.
A really good example of a system that messes up in this area is (I bet you know what I'm going to say) Harry Potter. These books are all about magic and yet we never learn how it works. Instead of having an inelegant list of powers like in Pillars of Life, we just don't have a list of powers. Instead there are an unknown but massive number of "spells" and whenever the plot needs something, the author makes up a spell that can do it. This has the effect that the reader can't possibly predict what's going to happen because they simply don't know what's possible in this world. But trying to imagine what might happen next is part of the fun of fiction. If something happens that you didn't expect because you didn't decipher the clues, that's a good story. If something happens that you couldn't have expected because you didn't know it was possible, that's just lame, especially if the POV character did know it was possible.
There's also quite a few instances of magic that don't involve incantations, spells or wands, such as potion making, divination (logically impossible), portable time travel devices (logically impossible), and probably others that I don't remember because it's been so many years. These things feel completely separate from the wand, spell, and incantation-based magic that wizards use directly. It's even worse than the tacked on powers in Pillars of Life.
And I have more criticisms. We never learn what the process of learning a spell involves (which is pretty important considering the story is about students at a magic academy). We also never learn what makes some wizards more "powerful" than others, or even what it means to be more "powerful" than another person in this magic system. How are we supposed to appreciate how cool the magic is if we don't even know how it works? Bottom line: Harry Potter is the epitome of a bad magic system. It has every flaw and no strength.
Actually, there is an exception to this. Final Fantasy 13 is a good example. In this game, magic exists and we don't know how it works, but it isn't used as a plot device - instead we're quickly given the impression that it's really only useful for combat (Lightning's gravCon device is a piece of technology as far as I'm concerned), and because of that combined with the combat system being an abstraction of what's actually happening in the story, we don't really need to know how it works. But if you're writing a novel and not a JRPG, odds are you intend your magic to be more than a combat asset, so you'll need to explain it.
Balance. Magic should be powerful enough that it plays a large role in the plot, but shouldn't be so powerful that it completely outmodes technology and makes mages invincible.
This one doesn't need too much of an explanation. If magic isn't powerful enough to be a central plot device, why is it even in the story? (Although I've never actually seen a magic system with that flaw I thought I'd mention it for completeness.) And if it's so powerful that it's the only plot device, then every conflict turns into a brute force contest to see who has the most magic. I think Dragon Ball is a good example of this. I haven't actually seen the anime but I've played Dragon Ball FighterZ, which shows me that characters can fly, shoot massive beams of destructive energy out of their hands, and when a fight ends, it's often by someone being launched away with so much force that they crash straight through multiple buildings. It's hard to take it seriously.
Two-sidedness. The best magic systems, instead of giving great power at small cost or small power at small cost, give great power at great cost. A parallel can be drawn here to desiging moral dilemmas.
Power is fun, but it's more fun when it has an interesting cost, limitation, or risk associated - especially since if you don't consciously do this, magic is likely to end up incredibly overpowered (like it did in Pillars of Life). In one story I created once (it's scrapped now), magic can get very powerful, but it costs wakefulness. The really cool thing about it is that the cost depends only the magnitude of the effect you achieve and not on how powerful you are. This means that when the characters were first gaining magic and it was weak, the effect was so small they didn't even notice, but I had planned that later in the story, when they became titans of magical power that could take on armies, fighting for just a few minutes would sap all of their energy and they would need to sleep for several hours to recover. I'm sure you can see that that's more interesting than just letting them use magic endlessly for free. It also served the plot in at least one way - it stopped the heroes from being invincible later in the story.
Another example, from a story I have planned but probably won't be getting around to for a while, is built on the premise that each person lives many lives in many different worlds, and what's happening when you sleep is you're essentially signing out of one world and into another. People don't know about this because their memories are compartmentalized. Still, that's what dreams are - your memories from other worlds leaking a bit. So when you use magic, the barriers keeping your memories separate get a lot weaker, and you start to remember your other lives. This seems like a good thing... until you get so deep into it that you can no longer tell the difference between your different worlds. You remember things that didn't happen, try to apply knowledge of other worlds to your present one, and, from the outside, it just looks like you're going insane. You effectively are.
The next I want to talk about is stock powers: abilities that are present in a lot of magic systems in some form or other.
Telekinesis. This is one of the most obvious and straightforward powers of magic: the ability to apply force to an object remotely. Although the power itself is unoriginal and uncreative, there's still a lot of cool stuff you can do with this. Pillars of Life uses a simple restriction to make things a lot more interesting: you can never telekinet a living person's body. Not even your own. So even though powerful mages fly, they have to do so by standing on wooden platforms with their ankles strapped in and telekineting those. Isn't that a lot better than giving them outright flight capabilites like a lot of superheroes have? It means if they're ever caught without their platforms they might have trouble getting off the ground. It also means no force push (you can't circumvent this by telekineting someone's clothes). Another restriction Pillars of Life uses is that you can't telekinet something already being telekineted by someone else, no matter how much more powerful you are. That solves the problem of why you don't just push back any projectiles a weaker mage throws at you. It's also taken advantage of for a clever plan when the protagonist kills a much more powerful mage by collapsing the ceiling on him.
Time travel. Bug off, okay? It's logically impossible and breaks morality. Never use this one.
Conjuration (of substances, not creatures). What I've learned from designing a lot of magic systems is that conjuration is completely busto. All it does is create problems for your world. In Pillars of Life, any prime element can be conjured, and the only cost is the expendable, recharging mana, so there's no way to stop a reasonably powerful mage from erecting rock castles overnight. If that sounds silly and stupid, it is. The only reason it didn't break Pillars of Life's plot or world is because the bad mages took over and made magic illegal, so that kind of power is hard to come by. I have a sequel planned... and I'm dreading having to deal with these problems in a world where magic isn't illegal.
That doesn't mean there's no way to make this work. I think conjuration just needs a real cost. One idea I've had (this couldn't be implemented in Pillars of Life, but might be in later novels I write) is that it takes from your max mana to keep conjured things in existence. This solves the problem of magic making construction work easy as pie. It raises another problem though: it's exploitable. What if you could take advantage of that by just letting go of keeping stuff in existence to get your max mana back when other people were depending on your conjured substances? You'll probably have to answer that if you want to use this fix for conjuration. One of the more generally applicable costs below might be a better solution.
Some sort of obvious attack. This is extremely common (at least in RPGs), and can come in the form of fireballs, lightning bolts, or just straight up "magic blast". While it's the pinnacle of uncreative, that doesn't mean it's never a good idea. One way I could think of to make this more interesting is to say that shooting a fireball makes the caster extremely cold. Not only does this prevent the caster from just incinerating all their enemies, it gives the power a sort of theme: not creating energy, but transferring energy. Themes are good, especially when they double as limitations.
Healing. This is an incredibly useful thing to have plot-wise in a fantasy setting, since without it injuries are rather permanent, which leaves you with two options: either the heroes miraculously never get injured (which will be unbelievable if the story contains a lot of combat), or no single hero ever has to fight more than a few times throughout the story, which imposes a lot of restrictions on what you can do with the plot. Having magic be able to heal seems like an easy solution to all those problems. But doesn't it have drawbacks too? For one thing, if magical healing is easily accessible then there are going to be some consequences for your world: doctors don't exist, nobody is sick, etc, and also, sometimes you want to give a character a lasting injury that prevents them from partaking in combat, at least for a time, which you can't exactly do anymore. The solution is simply to make healing a little bit harder to access. Give it some sort of significant cost, like the costs given below, or maybe you have to hurt someone else who consents in order to heal.
I guess another thing I might as well talk about while I'm here is stock costs of using magic.
Recharging mana. This is a pretty lame one. I used it in Pillars of Life because I have a background in fantasy RPGs and I hadn't really learned to be creative back then. The main issue with this is that it isn't really a cost. Since mana is useful for nothing else, and especially since it recharges, it doesn't serve as a drawback to magic, just a limitation. It's still better than letting your characters use magic indefinitely though.
The consumption of a special substance. Allomancy is an example of this. It's better than just using recharging mana, but it still doesn't really do much to limit the characters' reliance on magic unless the substance is hard to come by (which it's not in Mistborn except for Atium). Even if it is, affluent people can likely still get as much of it as they need. That isn't strictly a bad thing though - odds are the villain is more affluent than the hero, and you want the villain to be powerful.
Pain. This is kind of generic but I've never seen it done. It has the property of creating a moral dimension: badguys will want to avoid using magic themselves, more so than goodguys. This is probably a bad thing for the story since you want the villain to be more powerful than the hero. Still, I think this is a decent idea.
Memory loss. Here's a more serious cost. Everyone will think twice before using magic now. It surprises me that I also can't remember seeing this used anywhere, although I did sort of use it myself in Pillars of Life (for just one of the tacked on powers).
Here's another question you'll have to answer when you design a magic system: how do you get magic? If it has a cost like consumption of a rare substance, it might be okay to say everyone has it innately, but in most cases I expect you'll want to limit magic to certain people.
It's genetic. Some people are born with it and others aren't. I really dislike this one. I can't help but feel bad for the people born without it. Depending on the magic system, being eternally locked out of it is like being disabled in real life. Non-magical characters usually end up getting dwarfed by the magic-wielding hero and villain. And although I'm not disabled in real life and don't know anyone who is, I imagine that's disheartening for them. Even Mistborn does this. In the end of the second Mistborn book, the most important non-magical character is even magically made a Mistborn so that he can join his wife in kicking ass in book three. That's even more insulting. A non-magical person was useless, so they just made him magical, compounding the idea that disabled people in real life are useless.
One way to avoid this problem would be to have being magical not be a strictly good thing. Obviously, if being magical simply gives you the option to use magic, then it will always be a strict positive. But if there are some consequences to simply being magical even without using your magic, that could get really interesting, as well as give non-magical characters a way to still be useful.
Study. This is fine in theory, but the problem is that you have to answer what "studying" to become a mage involves. Harry Potter, again, is the perfect example of failure in this area: a book about students at a magic academy that doesn't even try to answer this question.
I've never seen this one done well, and never tried it myself either. I'd like to at some point but I just don't have any ideas.
Endowment from supernatural beings. This is a lot like the "it's genetic" answer but without the problem, since no one is eternally locked out of it. I use it in at least one WIP.
An experience. Actually, Mistborn uses this one too: even if you're born an Allomancer, you have to "snap" - which involves a near-death experience - to unlock your powers.
This seems like the most useful and versatile of the solutions here. It allows you to limit magic, not just to a random subset of people like the genetic answer does or to people with specific ancestry, but to people with a certain common thread. For example, maybe the only people who can use magic are those who have killed someone. This means every magic user in your story will either be a villain or a rare good person who's had the fortune not only of knowing someone worth killing, but of being in a position to be able to kill them. This lends itself nicely to the villains being far more numerous and powerful than the heroes. There's also the obvious and juicy opportunity to explore someone who wasn't evil, but killed an innocent person to get magic so they could save more innocents (explore the agonizing process of picking someone out! Do you kill the kind grandparent who's too old to be much more than a burden, or the young adult whose business is doing a lot to help people, but whose motivations are mostly selfish? A good person would likely face huge amounts of self-doubt and retrospective guilt over such a decision.), or someone who killed a somewhat evil but not death-worthy person to get magic for self-serving uses, and is now outside the restrictive storytelling limitations of white-and-black morality. In fact there would probably be a lot of such people in such a world. Creating a natural source of conflict for the plot :)
That's just one example of what you can do with this. It's a broad solution with a huge design space. Not only that, but this option allows you to give magic to whoever you want throughout the story without needing to send them to magic school for years or come up with an explanation of how you can go from being a muggle to being a wizard. Obviously it won't be right for every story, but it's definitely a good option with a lot of potential.
I'll probably come back and expand this article later when I think of more to say, but for now, I guess I'm done. I hope my ideas help you to write a good novel.