One good way to tell if the OC's a Mary Sue is to ask what would happen if you took her (or him) out of the story, and if there's a place for her (or him) in the world. Some OCs are like NPCs in a video game: they're the barrista at the coffee shop, and you could replace them with pretty much anyone without it making a difference. Others have to be who they are, but they still fit into the world-- like the murder victim's beloved sister who's secretly the culprit. Tony DiNozzo flirting with the barrista for half a scene doesn't make her a Sue; DiNozzo falling in love with the sister during the case, forgetting all about his other friends, and neglecting his job for her because they're Just That Perfect for each other probably does. (Actually, even that could make for good fic, if you made it about her psychological manipulation of him and made it clear that this was NOT a healthy relationship. But I digress.)

Personally, I think a lot of the fun of shipping canon characters is watching the author try to justify why these two, of all people, are together. Some use their interactions in canon to help that along, others don't.

1) Extremely Out Of Character characters
-this adresses your 'non-canon couple' question: if they can still act in character and get together, yeah. If one of them suddenly acts like he's blind/deaf/really stupid and/or one of them does things that they would never do in the real series, then it's OOC

2) Lack of SPaG: Spelling, Punctuation, and Grammar
-This just makes things hard to read. Plus, it makes us think that the author didn't work hard on their fic, or they don't care about their readers, or they expect us to do the work for them by filling it all in ourselves.

On the subject of OCs.

It is extremely difficult to get a good OC. A Mary Sue is a Mary Sue for several reasons.

#1: how other characters react to her. If the characters ALL love her no matter WHAT she does, and they're all fighting over who gets to escort her to the dance, etc., you can guess it's a Mary Sue.

#2: How does her presence affect the other characters? If she outdoes everyone in their own areas of expertise, always get the right side in an argument, etc. that's a Mary Sue.

#3: Is she perfect in every way? Does she have no flaws? Is she Superman without Kryptonite? She's a Sue.

#4: Is her general story well-written? If an author has no plot and tends to ramble on about what color the trim of her clothes is, you guessed it- it's a Sue.

Look for ways that a fic differs from canon. For instance, if a fic has Legolas as a weepy wimp who can barely raise a bow, let alone actually kill an orc with it, then that's a definite clue that it's a badfic.

If characters are being bashed or random canon events are taken out of context and used to justify the author's view of the character as evil/bad/a slut/etc., then that's another clue.

Shipping two characters who could be a couple doesn't make it a badfic, but there should be reasons for it. If an author has a couple who hate each other in canon and in the fic they're completely devoted to each other with no explanation as to how they resolved their differences, then it's probably a badfic.

The plot needs to make sense and it shouldn't be implausible at any stage. If a character is warped OOC so the plot can proceed, then yes, it's most probably a badfic. Another example is that there's a few fics in the Redwall fandom where a self-insert is dropped into Mossflower to help save Redwall from some imminent crisis, but given that Redwall is a sound, very strong building full of beasts who are trained, skilled warriors, there usually isn't a good reason as to why they'd need an unskilled kid from another world.

Also, if skills are being pulled out of a hat so that the plot can proceed or the characters can be even more awesome, then it's most likely a badfic. Any mention of a character 'just knowing' how to do something makes the fic implausible.

Hope that helps.

One telltale symptom of a bad story in general is if it makes excuses for events happening rather than having events in logical order or displayed as following cause/effect.

Most badfic has a superficial goal, like shipping a pairing or showing a character is AWESOME. They'll make excuses to get to this goal, or ignore basic facts and logic to attain it...

Excuse: "She murdered all of the people who DARED touch her man! It was OK and she didn't get arrested because those girls were evil."

Excuse: "Aragorn only acts the way he does around her because she's SO PRETTY!"

Excuse: "She's faster than Sonic the Hedgehog because she's half hedgehog, half cheetah."

etc. All of these things are excuses that commonly happen in bad fanfic...

A shipping pairing COULD be bad, if it makes shallow excuses for why they're together,("Sora and Riku are a couple because eating a paoupu fruit makes you GAY! And because I say so!") but another story with the same couple can be plausible and well-done if the two characters have real reasons to be together ("Sora and Riku have been through so much together, and after everything is over they discover their bond is deeper than they thought, etc.") that make sense and further them as characters.

Rather than excuses for mush, fluff, or fan-wank.

For the record, just because the pairing isn't plausible doesn't mean it's bad. See crackfic. Part of the appeal there is that it's intentionally so insane you can't help but laugh.

Well, first off, there are many definitions of "bad" fanfic. When choosing a fanfic to do a mission on, I ask myself some of the following questions: Is the plot believable? Do the characters act like they do in the source material? Are there any moments that make me go "Wait, what?"? These, among others, help me in determining whether or not it's mission-worthy.

As for an OC, no, just because there's an OC in the story does not make that character a Mary Sue. It's how the writer handles that character that defines whether or not it's a Sue/Stu. Again, some questions to ask: Does this character make sense within the universe of the story? If it's a setting where there are any kind of powers and/or abilities available, and that character has power/abilities, do those powers make sense for that character to have? Do they overshadow powers possessed by canon characters? Do they upstage canon characters constantly?

And pairings, well, I suppose that's a little more subjective. For example, if the canon source material plays with the fact that a certain pair could be romantically linked-- like say, Catherine and Warrick from CSI --then it's plausible that the pairing could take place. It doesn't make it a badfic.

So, I guess what I'm saying here is this: if it's plausible within the realm and rules of the canon material's universe, and doesn't throw anyone out of character, then no, it isn't a badfic. If if does, however, then you've got some problems. Does that help?