The dictionary gives several definitions for the word canon. With regards to Star Trek, most people go with the biblical definition:
A rule, standard or unit of measurement by which a writing was determined to be inspired or not; and the compilation of those writings considered inspired.
In the case of the Bible, all writings are considered to be inspired by God though they are written by several different people. In the case of Star Trek, all writings are considered to be inspired by Gene Roddenberry, the Great Bird of the Galaxy, though they are written by several different people.
There was much disagreement among scholars as to which writings were inspired by God and thus suitable for inclusion in the canon of the Bible, so it comes as no surprise that fans of Star Trek can't completely agree on which writings should be included in the canon of Star Trek. The official Star Trek web site even admits as much.
Traditionally, the realm of Trek canon has been defined as everything you see on television and in the theater, but it's not that simple. Gene Roddenberry himself stated that the fifth feature film, The Final Frontier, wasn't canon. Has not the Great Bird of the Galaxy spoken?
The Animated Series is officially considered to be outside of the canon, but it wasn't for lack of inspiration by Gene Roddenberry. The Animated Series was very much inspired by Gene, but the copyright to the episodes was retained by the writers. This is not how canon should be determined. But with the release of the Animated Series on DVD, it looks as though the copyright issues have been cleared up, paving the way for future inclusion of the Animated Series into the canon.
To understand canon, you need to start thinking of Star Trek as a full-fledged universe, into which we have only been given hour-long glimpses once a week. Much more happens in that universe which we have not been made privy to. Using our knowledge of how our universe works, we can reconcile the discrepancies and fill in the gaps based on what has been revealed to us and a bit of good old Vulcan logic.
The people who made Star Trek are humans just like everyone else, and as such are susceptible to making mistakes. When two facts contradict each other, the reliability of each must be weighed against the other. Also consider the plausibility of each fact: if intruders invade on Deck 29 of a ship that only has 25 decks, it's more plausible that the deck number was incorrect than that the ship was refitted since the last movie to include four more decks. Just as Gene discarded The Final Frontier from the canon, it may be necessary to discard other episodes and feature films from the canon. Certainly the precedent has been set for it, and the Voyager episode "Threshold" is a prime candidate for ejection from the canon.
When things are unclear, people behind-the-scenes will sometimes elaborate on them. Once in a while they will even tell us things we wouldn't otherwise know from on-screen, or they will tell us that what we saw on-screen was a mistake and will give us the correct information. Those who don't understand canon easily dismiss such things, and to their own disadvantage. If it logically makes sense in the context of the Star Trek universe, it's canon.
Another fantastic source of information is the numerous books that have been published. I'm not talking about novels, those are a separate issue. I'm referring to things like the Star Trek Encyclopedia, or the TNG Technical Manual. It's the stuff too boring to show on television, and it's more valuable than gold-pressed latinum.
While in most cases the books will support what is seen and heard on-screen, there are other cases where they will conflict. When there is a confliction, we must consider ALL the evidence, and make a decision based on logic. In most cases, the screen will be correct, but not always. Every once in a while, the book will be correct, and to recognize those times is what it means to truly understand canon.
But for each conflict between what we see on-screen and what we see in print, there are several gaps between what we see on-screen that are filled by what we see in print. Those portions of the books which don't contradict on-screen evidence or behind-the-scenes sources are, in my personal opinion, just as worthy of being considered canonical, especially when the book in question was written by someone who works behind-the-scenes and their regular job is to come up with this stuff for the episodes and feature films.
So now, what about the novels? I've read more than fifty TNG novels. Some of them I've really enjoyed, and some of them were quite stupid. Most prefer to just enjoy the novels for their literary value and not take them into consideration while discerning canon, with the possible exception of novels based on episodes because they provide a deeper insight into the characters and may also provide additional information.
Two novels by Jeri Taylor were officially considered to be part of the canon during the production of Voyager because they contained background information on many of the Voyager characters. However, this information didn't make their way into any episodes, leaving a shadow of doubt cast over whether an exception is still made for them.
Another branch of the Star Trek franchise is the games. We need to tread very lightly here, because the companies that are contracted to develop these games make a lot of sacrifices in the interest of game play. They have an even greater disregard for canon than the episode writers. In some cases, they genuinely try to make a good Star Trek game. In other cases, it's just an excuse to make another first-person shooter that will appeal to the Star Trek audience. More recently, game developers are attempting to appeal to a wider audience, and alienating some Star Trek fans in the process.
It could be argued that many of the novels, and to a lesser extent games, are just as inspired by Gene Roddenberry's vision as most of the episodes and feature films, and with a precedent already set for including novels in the canon, it's not all that far-fetched. I expect, though, that such a proposal would meet with much opposition from those who adhere to the traditional definition of canon. As a compromise, I propose taking a page from George Lucas' book and putting the novels, comics, and games in a category called the Expanded Universe. The Expanded Universe concept recognizes that that these works are still officially sanctioned and at least partially inspired by Gene's vision, without making them equally canonical with more reliable sources.
The events of Enterprise take place in an altered timeline, the result of numerous time travel mishaps in the previous five Star Trek series. This was confirmed in the last episode by Daniels, "None of this should be happening."
The producers would not even acknowledge that any continuity mistakes had been made. The most convenient resolution would be to reject the entire series, but there are some elements which could be considered canon: the species they encounter, the planets they visit, and some of the people and events could peacefully coexist with the history of the future as we know it. It's quite possible that the transporter was invented by Emory Erickson, but there was definitely no Earth Starfleet - prior to 2161, space exploration was carried out by the United Earth Space Probe Agency.
There is much more that could be said, but my intent was not to build a comprehensive list of which facts are canon and which are not; that would be an entire project in itself. My intent is to improve people's understanding of what should be considered canon and what should not. The key to discerning canon is a full understanding of what Gene Roddenberry's vision for Star Trek was. It is because the writers lost sight of this vision that we now find ourselves at this low-point in the Star Trek franchise.