I find this all really interesting, in part because of my work as an academic. When my students write academic papers, we require that they cite only peer-reviewed journals. (Several of which, by the way, have moved to online rather than print publication, especially in the sciences.) The assumption is that those have gone through a stringent editorial process by experts, and that recent articles are likely to be as accurate as the current state of human knowledge allows. We're training the students to eventually be experts themselves, or at least to know when they're reading something written by an expert. We privilege accuracy above all in an academic setting -- we want the truth, no matter how long it takes them to find it (or how tedious the search.)
Wikipedia, on the other hand, is a balancing act. They absolutely want accuracy. They could require the same level of reliability as academia does, but that would severely hamstring their breadth and depth, since it takes a great deal of effort and time for something to be published in a peer-reviewed journal. Even without the time limitations of print, you need to corral busy experts and get them to actually review the darn thing. I don't think anyone wants Wikipedia to go to that mode -- we have print encyclopedias for that. :-)
So they have to allow some more flex in what they consider a reputable source. And this is where it gets tricky. Let's look at some sources:
- Books/journals/magazines/newspapers: I gather that Wikipedia generally accepts these. A fine initial starting point, but this category includes everything from The Wall Street Journal to Locus (print or online) to the late-lamented Emerald City (a single editor online publication, very reliable) to Joe's F/SF Magazine (does not exist). Now, Joe may have scads of dough and have printed off a million beautiful color copies of his magazine, but there's no way to judge his reliability aside from that. Maybe Joe fact-checked every single item in his magazine, and he's just obsessive enough with his research that he's the first to announce that TOR has decided to switch to published sf/f romances exclusively, breaking the story just before Locus. Or maybe he secretly writes sf/f romances and is pissed off that TOR keeps rejecting his 400K-word novels, and this is his way of getting revenge. Probably his reliability is somewhat lower than all of the previous sources, but there's no way to tell -- and just because he 'publishes' it doesn't actually make it reliable.
- Personal sites/blogs/etc.: My site, Sepia Mutiny, The Whatever, Making Light, etc. Single-author, or multi-author, but generally without anyone editing any individual post aside from the initial author. Reliability here is dependent on the expertness of the author, plus their care in checking their facts. I am reasonably expert in some areas. I am moderately good about checking my facts. I would expect that Jed would be much more careful about checking his facts than me, just because of the kind of person he is. At least with these, there's some degree of accountability -- you the reader can generally trust that if you read something on Mary Anne Mohanraj's blog, Mary Anne actually wrote it (unless it's explicitly stated otherwise). But there's still the question of knowledgability/expertness. If the news about Tor publishing only sf/f romances appeared on Making Light, I'd believe it. I'd be startled, maybe even shocked, but I'd believe it. If it appeared on Jed's site, much as I love him, I'd probably check someplace else before I was sure he'd gotten his facts right. But if it was a note about something at Google on Jed's site, I'd trust that to be accurate. You have to take into account the nature of the information, and what specific areas the source is expert in. There was a period of some years when my "Alternative Sexualities in SF/F Booklist" was, I believe, the most comprehensive and accurate listing available in the world -- and it was only available online, on my site. I never bothered to publish it, because it was very accessible as it was. Would I have let my students cite it? Nope. But I, personally, would consider it appropriate as a source for Wikipedia.
- Postings to forums, mailing lists, etc., whether industry-specific or not: SFF.net, SAWNET, the Strange Horizons forums. Some of these may be private and require membership in an organization to post to it, which does provide a mild degree of 'expertness' and 'accountability' -- but not that much. They're easy to break into, easy to impersonate an actual expert. And many of them only require casual registration, or are completely open to the public. I'd consider these the least reliable sources, personally, even if a host of experts do congregate there, simply because there's so little control over making sure that the person posting is who they say they are. But at the same time, such a forum may well be where the most accurate and up-to-date information on a specific subject can be had.
A final postscript: I do think it's a bit hard to think about this clearly, as a professional in various fields, because ego so quickly gets involved. I want Wikipedia to consider me an expert source; I want my word, based on fifteen years in publishing and academia, to be sufficient. But the truth is that I should only be considered a reputable source in certain areas, and only when it's clear that I'm putting facts out there, as opposed to opinions, ideas, pet theories, etc. For a Wikipedia editor, even if I and my journal become a reputable expert source, it's always going to be a judgement call as to whether any particular thing I say is reliable enough to cite. But my point is, that's always the case, and the trustworthiness of any source must be evaluated. That's what we're trying to teach our students when we make them start with peer-reviewed journals alone.