Let me reiterate that I don't claim to be a good writer; I'm only claiming to know a few general principles and not to be good at applying them. You'll notice reading my work that much of the dialog you probably consider bad is bad because it doesn't follow this advice. The reason I don't follow my own advice is because following my advice is hard, and hard things are not easy to do.
With that out of the way, here are a few principles:
The best dialogue is that where each line is witty and not exactly what the audience expected.¶
This is the principle behind the system in Mass Effect where the dialog option labels are paraphrased from what Shepard actually says. I don't think it's unreasonable to think it was still a bad idea overall, but it would have been boring to have the player select a dialog line and then listen to Shepard repeat it verbatim, and the paraphrasing solves this problem.
For a concrete example, I'll talk about some edits I made throughout the development process to my abandoned novel The Pygon Hybrid. An except from chapter 1 reads:
I hear footsteps coming up the stairs. It's Brev and three other cadets I recognize: Logan, Stef, and Sally. They move immediately but casually to surround me.
"Well let's hear it then," I say.
"Telra..." Brev says. "You know how you're the only cadet who's better than me?"
"I'm surprised you're not too embarrassed to say it out loud. What of it?"
Originally, Telra's first line was written as "What?" and her second line as "Yeah... What of it?". Both of those lines are bad because they are exactly the obvious thing to say in that situation. They're the most nondescript possible way of communicating those thoughts. And that's bad. It's not entertaining to read because of that. Also you never want your protagonist to come off as a blank-slate, unremarkable. So I changed it to what it is now.
The rewritten first line shows a certain confidence. She's trying to show them that she's not intimidated and make them feel like she's one step ahead of them by assuming the obvious and prompting them in a way that implies they're being excessively indirect. The rewritten second line is a massive improvement. It gives Telra so much more personality as it shows her rubbing in her superiority (competence being the highest ideal to her) and being a little bit of an asshole.
Once again, I make no claims that my novels are good or that the majority of the dialogue is well-written, but here's one thing I definitely did right, and I hope you can learn from it.
The best dialogue is tailored to the character and the world.¶
I tried to do this a little bit in Pillars of Life (you can tell me how it came out) with Hobart having awful grammar and universally using "ye" as a second person pronoun. Yildirim has a noticeably more formal way of speaking where he avoids contractions unless it sounds particularly awkward without, and more frequently chooses "however" over "but", etc. This reflects the dignified, stolid personality that makes sense given his background and role in the forces of good. Mitilda, the youngest of the heroes at 10 years old, uses childish insults such as "jerkface" and "snotball". Most distinctively, she hasn't outgrown the typical 10-year-old unwillingness to accept that you've been pronouncing a word wrong. She says "telekinese" instead of "telekinet", and continues doing this to the end of the story, even calling out other characters for saying it "wrong".
I don't have the feeling that I did a good job though. If I had to guess, I'd say Hobart and Yildirim's speech patterns came out sounding shoehorned and unrealistic.
Another thing you'll notice as a difference between my two novels is the degree of profanity. Pillars of Life takes place in a pseudo-medieval fantasy setting, so I thought it suited the world to limit all characters to low-end profanity at the most. The protagonist uses "drat" where I would likely go as high as "shit", and the strongest that ever comes out is a few instances of "damn". The Pygon Hybrid, on the other hand, has almost every character routinely saying "fuck", because it takes place in an urban setting where the POV character is what you might call a "delinquent", and most of her allies have emotional issues.
The best dialogue is tailored to the characters' relationship and the situation.¶
By far the most common form of this flaw is overly formal dialog. Real people - at least ones who know each other well - speak as if communication is their ideal rather than some meaningless idea of "proper" speech. In particular, they use contractions. Contractions are a fact of how the English language is spoken. It can make sense for a character to use less of them in certain situations or depending on their personality but no real person avoids them the way some characters in fiction do.
Other forms of formal dialog that are usually excessive include:
"however" over "but" or "(al)though"
"perhaps" over "maybe"
"suppose" over "guess"
"that" when introducing a subordinate clause
"begin" over "start"
"may" over "might"
unnecessarily using the adverb suffix with adjectives that can act as adverbs already: quick/fast/slow etc
Obviously, when the character is trying to sound formal or grandiose, the stuff in that list is usually a good thing. But most often they aren't.
Another facet of informal speech - although this one is rather extreme - is omitting the pronoun subject: "Tried. Didn't work." instead of "I tried. It didn't work".
The best dialog is written by the characters, not by the plot.¶
One of the most common forms of bad dialog - and maybe the hardest to avoid - is dialog that gives the appearance of intentionally moving toward a specific destination that isn't either of the characters' goals. The more you show that the story wouldn't go the way it is if it weren't a story, the harder it gets to enjoy the story.
The eternal struggle of writing good dialog without clobbering the plot is to make the characters' flow of thoughts believably end up at the desired point. This will always be hard no matter how well you know your character or whatnot.
The writer has a critical tool here: uncertainty. People will have some idea of what makes sense for a given person to say in a given situation, but there's almost always some wiggle room: more than one thought might make sense as following the previous one, or two different wordings might both sound good and be in character which could affect how the other character feels in response, or the character might have two thoughts and choose to voice them in either order, and the response to whichever is voiced first might affect how it's believable for the second part of the conversation to go. The writer can do whatever they want inside these areas of uncertainty.
There's also the offscreen timing advantage: the writer can do whatever isn't unbelievable, so unless the audience knows where every character is at all times, it usually works just fine to have another character interrupt or enter a conversation to cut it off at exactly the point the author wanted it at. If this happens too often it might come off as contrived but it's fine for use here and there.
"Dialog tags are garbage," Yujiri angrily criticized.¶
I guess I should also talk a little about dialog tags. I honestly hate the conventional ways of doing this and I wish we could just write '(character): "(line)"'. I think the only reason it's not a good idea to do that is that readers don't expect it, so they notice it, contradicting the objective of unobtrusiveness. I still want to try it someday and prove it can work. Maybe in my next project. But for now, I'm using in-prose tags, and I expect you'll want to too, so I'll talk about them.
There are a lot of different common ways of formatting these. I remember when I was little I thought of "(line), said (name)" as the default format (Harry Potter used it). But I hate this format. It sounds old-fashioned and like you're trying to be poetic or something (especially when you write in first person: "(line), said I" sounds awful). More recently it seems like "(name) said" has become the standard over "said (name)", and I like that more, but I still hate the idea of putting dialog tags at the end of the line. Putting it at the beginning is a strict improvement in terms of clarity, and while readers don't often get confused about who's saying what, it can be really bad when it happens.
Sometimes people do put it at the beginning. Like the method I mentioned above, this is only awkward because of how non-standard it is. But I've seen some authors use it occasionally and it landed fine. The way this works best is by using a narrative statement instead of a plain tag. For example, instead of 'John said, (line)', write 'John stopped typing for a minute. (line)'. These little descriptions can add a lot to conversations. In fact they're pretty much strictly better than pure dialog tags if you can think of a good one. Of course, the downside is that since they're adding information they have to be tailored to the situation, which takes creativity. And creativity is hard ;)
Sometimes authors put the dialog tag in the middle of the line to achieve some of the benefit of having it at the beginning without it being quite as awkward. I don't have a problem with this - I do it myself sometimes - but I would ask that you put it in a place where there's supposed to be a pause, because it makes the reader imagine the character pausing over the tag.
Another common thing other writers say is that you shouldn't use dialog tags if they're not necessary. That's very true. Since tags are treated as part of the prose, they get read as such by the reader, and so they can be obtrusive if you use them on every line unnecessarily. Generally, in a two-person dialog you should only need a tag every fourth line or so. (The only reason you need them at all is incase the reader loses track of who spoke last.)
Also, I know I'm the one-millionth person to say this but I should talk about verb choice. Remember, since the goal is to provide clarity without being obtrusive, you should usually stick to "said" as the verb. Said is so nondescript and standard that it becomes somewhat invisible. Anything more obscure risks making it sound like you're having fun seeing how many synonyms you can come up with. And if you're writing your dialog right, more specialized verbs won't be necessary in the vast majority of cases. A verb like "agreed", "pointed out", "fumed", or similar should be evident from the content of the line itself. If the character is "yelling", the exclamation mark usually suffices to signify that. Unnecessary redundancy is bad. At best it wastes time; at worst it comes off as insulting. Here's a principle: Don't repeat yourself, or say again what you have said before. (A major exception is whispering - this one usually needs to be stated, especially since it's not mutually exclusive with the exclamation mark. "Respond" and "ask" are also okay sometimes since using said in those cases can make it sound like it's supposed to not be a response or a question.)