Back in the days of me trying to philosophize about storytelling separately from psychology, I thought there were four prime traits for a good magic system. I don't stand by the old philosophy any more, but I think I said enough worthwhile things that I won't scrap the article. Here are the three traits I listed that I still think are, while not true prime criteria, useful to think about:
Depth. It should give rise to complex usage and far-reaching implications through relatively simple mechanics. A parallel can be drawn here to game design.
A really good example of depth (and honestly just the best magic system I've ever seen overall) is the Mistborn books. Some people are born with Allomancy, which allows them to consume small bits of metal and then "burn" it for magical power. Each metal gives you a different effect, and they come organized in three dichotomies: internal versus external, mental versus physical, and pushing versus pulling. For example, the internal physical pushing metal, pewter, makes the Allomancer's body much stronger and more agile while burning it. The external mental pulling metal, zinc, magnifies the emotions of others, which can be used to manipulate crowds and individuals. The external physical pushing metal, steel, allows you to push metal away from you - or, if the metal object is heavier than you, pull yourself toward it. This creates so much depth of interaction when we see Allomancers carrying around sacks of coins to use as projectiles and specialized soldiers called "hazekillers" that use wooden shields and weapons and wear no metal in order to better combat Allomancers.
Allomancy has a consistent theme: metal is power. This is built on later in the books when the Atium stash is revealed to be Ruin's body. And since the powers are organized in a rigid system of dichotomies, it doesn't feel like eight separate things magic can do. It feels more like three ideas from which eight powers follow. Contrast a system like in my novel Pillars of Life, where besides the four energies and the sixteen elements spawning from them and the four states of matter, magic has several miscellaneous powers. These ones, at least besides telekinesis and destruction, feel tacked on to what was already a complete system. Sure, they add room for me to do cool stuff with the plot, but they still made the system less elegant. Pillars of Life magic is so ugly and arbitrary compared to Allomancy.
Transparency. The reader should be told how the magic works, so that they can appreciate how cool it is, and so that it feels fair when you use the magic to solve a problem.
A really good example of a system that fails horribly in this area is (I bet you know what I'm going to say) Harry Potter. These books are all about magic and yet we never learn how it works. Instead of having an inelegant list of powers like in Pillars of Life, we just don't have a list of powers. Instead there are an unknown but massive number of "spells" and whenever the plot needs something, the author makes up a spell that can do it. This stops us from predicting what's going to happen because we just don't know what's possible in this world. But trying to imagine what might happen next is part of the fun of fiction. If something happens that you didn't expect because you didn't decipher the clues, that's a good story. If something happens that you couldn't have expected because you didn't know it was possible, that's just lame, especially if the POV character did know it was possible.
There's also quite a few instances of magic that don't involve incantations, spells or wands, such as potion making, divination (logically impossible), portable time travel devices (logically impossible), and probably others that I don't remember because it's been so many years. These things feel completely separate from the wand, spell, and incantation-based magic that wizards use directly, like rowling had to include every trite form of magic in one story. It's even worse than the tacked on powers in Pillars of Life.
We never even learn what the process of learning a spell involves, which is pretty important considering the story is about students at a magic academy. We also never learn what makes some wizards more "powerful" than others, or even what it means to be more "powerful" than another person in this magic system. How are we supposed to appreciate how cool the magic is if we don't even know how it works?
There is an exception to the rule of transparency. Final Fantasy 13 is a good example. In this game, magic exists and we don't know how it works, but it isn't used as a plot device - instead we're quickly given the impression that it's really only useful for combat (Lightning's gravCon device is a piece of technology as far as I'm concerned), and because of that combined with the combat system being an abstraction of what's actually happening in the story, we don't really need to know how it works. But if you're writing a novel and not a JRPG, odds are you intend your magic to be more than a combat asset, so you'll need to explain it.
Two-sidedness. The best magic systems, instead of giving great power at small cost or small power at small cost, give great power at great cost. A parallel can be drawn here to desiging moral dilemmas.
Power is fun, but it's more fun when it has an interesting cost, limitation, or risk associated. In one now-scrapped story I created, magic can get very powerful, but it costs wakefulness, and the cost depends only the magnitude of the effect you achieve and not on how powerful you are. So when the characters were first gaining magic and it was weak, the effect was so small they didn't even notice, but I had planned that later in the story, when they became much more powerful, fighting for just a few minutes would sap all of their energy and they would need to sleep for several hours to recover. Much more interesting than just letting them use magic endlessly for free. It also served the plot in at least one way: it stopped the heroes from being invincible later in the story.
Another example, from a story I have planned but probably won't be getting around to for a while, is built on the premise that each person lives many lives in many different worlds, and what's happening when you sleep is you're essentially signing out of one world and into another. People don't know about this because their memories are compartmentalized. Still, that's what dreams are - your memories from other worlds leaking a bit. So when you use magic, the barriers keeping your memories separate get a lot weaker, and you start to remember your other lives. This seems like a good thing... until you get so deep into it that you can no longer tell the difference between your different worlds. You remember things that didn't happen, try to apply knowledge of other worlds to your present one, and, from the outside, it just looks like you're going insane. You effectively are.
The next I want to talk about is stock powers: common or obvious abilities that are useful to think aobut as starting points.
Telekinesis. This is one of the most obvious and straightforward powers of magic: the ability to apply force to an object remotely. Although the power itself is unoriginal and uncreative, there's still a lot of cool stuff you can do with this. Pillars of Life uses a simple restriction to make things a lot more interesting: you can never telekinet a living person's body, not even your own. So even though powerful mages fly, they have to do so by standing on wooden platforms with their ankles strapped in and telekineting those. It means if they're ever caught without their platforms they might have trouble getting off the ground. It also means no Force-push (you can't circumvent this by telekineting someone's clothes). Another restriction Pillars of Life uses is that you can't telekinet something already being telekineted by someone else, no matter how much more powerful you are. That solves the problem of why you don't just push back any projectiles a weaker mage throws at you. It's also taken advantage of for a clever plan when the protagonist kills a much more powerful mage by collapsing the ceiling on him.
Time travel. Bug off, okay? It's logically impossible. Never use this one.
Conjuration (of substances, not creatures). What I've learned from designing a lot of magic systems - most of which died before the stories ever saw daylight, and partly for this reason - is that if not handled carefully, conjuration is completely busto. In Pillars of Life, any prime element can be conjured, and the only cost is the expendable, recharging mana, so there's no way to stop a reasonably powerful mage from erecting rock castles overnight. If that sounds silly and stupid, it is. The only reason it didn't break Pillars of Life is because the bad mages took over and made magic illegal, so that kind of power is hard to come by. I used to have a sequel planned, and dreaded having to deal with those problems in a world where magic wasn't monopolized by the villains.
That doesn't mean there's no way to make this work. I think conjuration just needs a real cost. One idea I've had (this couldn't be implemented in Pillars of Life, but might be in later novels I write) is that it takes from your max mana to keep conjured things in existence. This solves the problem of magic making construction work easy as pie. But (this may be good or bad) it's exploitable. What if you could take advantage of that by just letting go of keeping stuff in existence to get your max mana back when other people were depending on your conjured substances?
Some sort of obvious attack. This is extremely common in RPG magic systems (and RPGs are legitimate storytelling in every way), and can come in the form of fireballs, lightning bolts, or straight up "magic blast". While it's the pinnacle of uncreative, that doesn't mean it's inherently bad. Not all stories need a super creative magic system. One way I could think of to make this more interesting is to say that shooting a fireball makes the caster extremely cold. Not only does this prevent the caster from just incinerating all their enemies, it gives the power a sort of theme: not creating energy, but transferring energy. It would be interesting to build on that if there were other powers in the magic system. (Hm... I might've gotten inspiration from Mistborn for this...)
Healing. This is an incredibly useful thing to have plot-wise in a fantasy setting. Without it, injuries are rather permanent, which leaves you with two options: either the heroes miraculously never get injured (which will be unbelievable if the story contains a lot of combat), or no single hero ever has to fight more than once or twice throughout the story, which crosses off a lot of possible stories. Having magic be able to heal seems like an easy solution, but it brings with it a lot of its own issues. For one thing, if magical healing is easily accessible, it changes the world dramatically: doctors probably won't exist, nobody is sick, etc, and also, sometimes you want to give a character a lasting injury, which becomes difficult to do.
As with other unruly powers, healing can work fine if it's harder to access. Maybe you can heal injuries but it shortens the patient's lifespan by several years, or maybe you have to hurt someone else who consents in order to heal.
I guess another thing I might as well talk about while I'm here is stock costs of using magic.
Recharging mana. This is a pretty lame one. I used it in Pillars of Life because I have a background in fantasy RPGs and I hadn't really learned to be creative back then. The main issue with this is that it isn't really a cost. Since mana is useful for nothing else, and especially since it recharges, it doesn't serve as a drawback to magic, just a limitation. It's still better than letting your characters use magic indefinitely though.
The consumption of a special substance. Allomancy is an example of this. It's better than just using recharging mana, but it still doesn't really do much to limit the characters' reliance on magic unless the substance is hard to come by (which it's not in Mistborn except for Atium). Even if it is, affluent people can likely still get as much of it as they need. That isn't strictly a bad thing though - odds are the villain is more affluent than the hero, and you want the villain to be powerful.
Pain. This is kind of generic but I've never seen it done. It has the property of creating a moral dimension: badguys will want to avoid using magic themselves, more so than goodguys. This is probably a bad thing for the story since you want the villain to be more powerful than the hero. It's also a little questionable because the audience can't imagine it; describing pain well is not easy.
Memory loss. Here's a more serious cost. Everyone will think twice before using magic now. It surprises me that I also can't remember seeing this used anywhere, although I did sort of use it myself in Pillars of Life (for just one of the tacked on powers).
Here's another question you'll have to answer when you design a magic system: how do you get magic? If it has a cost like consumption of a rare substance, it might be okay to say everyone has it innately, but in most cases I expect you'll want to limit magic to certain people.
It's genetic. Some people are born with it and others aren't. I really dislike this one. I can't help but feel bad for the people born without it. Depending on the magic system, being eternally locked out of it is like being disabled in real life. Non-magical characters usually end up getting dwarfed by the magic-wielding hero and villain. And although I'm not disabled in real life and don't know anyone who is, I imagine that's disheartening for them. Even Mistborn does this; almost all of the heroes are rare Allomancers, and it's still the three with the most powers who drive the plot. (Two of them are POV, but I think that's because the magic system makes it inevitable that they'll dominate the story, and not vice versa.) In the end of the second Mistborn book, the most important non-magical character is even magically made a Mistborn so that he can join his wife in kicking ass in book three. That's even more insulting. A non-magical person was useless, so they just made him magical, compounding the idea that disabled people in real life are useless.
One way to avoid this problem would be to have being magical not be a strictly good thing. Obviously, if being magical just gives you the option to use magic, then it will always be a strict positive (outside of rare "kill yourself to avoid capture and torture" type of situations). But if there are some consequences to simply being magical even without using your magic, that can get really interesting, as well as give non-magical characters a way to still be useful. (Pillars of Life technically had one, that it makes you easier to find with scanning, but it didn't really matter because magic was too overpowered for it to ever be undesirable.)
Study. This is fine in theory, but the problem is that you have to answer what "studying" to become a mage involves. Harry Potter, again, is the perfect example of failure in this area: a book about students at a magic academy that doesn't even try to answer this question. I've never seen this one done well, and never tried it myself either. I'd like to at some point but I just don't have any ideas.
Endowment from supernatural beings. This is a lot like the "it's genetic" answer but without the problem, since no one is eternally locked out of it. I used it in at least one planned story that I'll probably never write (it was weak in every other way).
An experience. Actually, Mistborn uses this one too: even if you're born an Allomancer, you have to "snap" - which involves a near-death experience - to unlock your powers.
This seems like the most useful and versatile of the solutions here. It allows you to limit magic, not just to a random subset of people like the genetic answer does or to people with specific ancestry, but to people with a certain common thread. For example, maybe the only people who can use magic are those who have killed someone. This means every magic user in your story will either be a villain or a rare good person who's had the fortune not only of knowing someone worth killing, but of being in a position to be able to kill them. This lends itself nicely to the villains being far more numerous and powerful than the heroes. There's also the obvious and juicy opportunity to explore someone who wasn't evil, but killed an innocent person to get magic so they could save more innocents explore the agonizing process of picking someone out! Do you kill the kind grandparent who's too old to be much more than a burden, or the young adult whose business is doing a lot to help people, but whose motivations are mostly selfish? A good person would likely face huge amounts of self-doubt and retrospective guilt over such a decision. and is now outside the restrictive storytelling limitations of white-and-black morality. In fact there would probably be a lot of such people in such a world.
That's just one example of what you can do with this. It's a broad solution with a huge design space. Not only that, but this option allows you to give magic to whoever you want throughout the story without needing to send them to magic school for years or come up with an explanation of how you can go from being a muggle to being a wizard. Obviously it won't be right for every story, but it's definitely a good option with a lot of potential.
I'll probably come back and redo this article later when I have better philosophies, but for now, I guess I'm done. I hope my ideas help you to write a good novel.