Ellipsis and dashes are often confused and misused in dialogue, especially by new writers unsure when to use each one. But if you slow down and think about it, they aren't that hard to figure out.
Dashes:
-Indicate an interrupted sentence
-Are read quickly
-Should be formatted as an em dash.
If you use word, simply type two en dashes (--) and put a quotation mark at the end of it. No spaces. It will automatically turn into an em dash. If you're not in word, you can hold alt, type 0151, and release alt. That's what I'm doing for this post.
Ellipsis:
-Indicate a trailed-off sentence
-Are read slowly
-Should always be 3 periods long. (It's often called a "dot dot dot" for a reason)
-Can also be used when a character is overhearing one side of a conversation, like a phone call.
Example:
Let's imagine a character named Josie is hesitantly exploring a haunted house with an adventurous character named Molly. Molly reaches for a door and Josie is sure something terrible is behind it. Notice the difference between the following sentences.
"I don't think you should..."
"I don't think you should—"
The first indicates that Josie has nothing more to say than the words there, or she got to scared to finish the sentence, or realized Molly wasn't listening to her. The sentence reads slowly at the end, with the reader's eyes lingering on "should."
In the second, it is clear that Josie had more to that sentence, perhaps "I don't think you should go in there." and something forced her to stop speaking. Maybe Molly opened the door and something jumped out at them. Maybe Molly told her to shut up. Which might look like this:
"I don't think you should—"
"Shut up," Molly said. "I'm going in."
Don't put any dialogue tag after a dash. You want readers to go from the end of that sentence to the next one with no pause.
Now, what about overhearing half a conversation? This is usually a character listening in to a phone call. Formatting these situations can be tricky, because you want readers to know what's going on, but you don't want a string of quotes from the person the character can hear.
I'll use an example from my own writing. This is some extra content I did for Chaos in G Major from Sam's POV that takes place in a hospital before the start of the novel. Sam, Juliet, and Olivia's dad is speaking.
“Hey Jules… You’re heading out?... Yes, we’re both here… Fourth floor, Mom knows where the room is… Sam’s awake, Olivia isn’t yet… That’s a good idea. I know Sam hasn’t eaten anything yet.”
Even though we can't hear what Juliet is saying, the reader can figure out she's said they're leaving, asked where to find her dad and Sam, and offered to bring food. If I'd formatted each of their dad's line as it's own quote, it'd be long and obnoxious to read. This way, it's all one line of dialogue, which is the way Sam is hearing it, with pauses when Juliet speaks.
You'll also notice that this isn't going to sound quite like a normal phone conversation. You need to make the statements of the heard speaker repeat enough of the unheard speaker that the reader knows what's happening. In real life, this conversation would likely be shorter and make no sense to Sam overhearing it. But dialogue isn't like real life speech.
Exercise 1:
Write a conversation between two characters who are constantly interrupting each other or trailing off their sentences. Get used to the formatting of each. How does trailing off feel different from an interruption? When you read the scene aloud, does it still sound right?
Exercise 2:
Practice writing a phone conversation the character can only hear one side of. How can you tell the reader what the unheard speaking is saying without making the heard speaker repeat everything? Read it aloud. Does it sound stiff?
If you're struggling, try writing out both sides of the conversation, then deleting the side that wouldn't be heard. Does it still make sense? What can you tweak so it does?