MetaFic

Hi!

I found some "new" (old) interesting ones, a method to separate the wheat from the chaff, and some statistics.

First, the stories:
http://www.fanfiction.net/s/4437151/1/Harrys_New_Home
http://www.fanfiction.net/s/3736151/1/Better_Be_Slytherin

Second, I started to only look at stories that had more than a few hundred reviews apiece, because I noticed that people tend not to say anything if they really don't like what they read. Wanting systematic results, I used a mass downloader to save all pages listing Harry Potter stories on my hard drive. It took a little less than two hours (a bit more to tweak the parameters right) but then I had 11'153 pages on my drive (or a few counts short of 300'000 stories, with only a few repeats due to the site updating during my download), taking 370 MB of (uncompressed) space.
So far, so good.
A few hours of programming left me with a bit of software able to filter this, and now I am the proud owner of a tabulated text file that lists all Harry Potter stories (excluding the Crossovers - I have the raw data but not the tabulated file - yet); actually, I limited it to 10000 words stories or larger because Excel can't open files with more than 65'535 lines.

Third, some statistics from these 38'500 stories (total word count: 1'275 million):
- the largest one: "Resolving a Misunderstanding" - 160 chapters - 888'000 words! Only 90% of the whole original series...
- the most reviews: "Harry Potter and the Nightmares of Futures Past" - 9043
- the longest writing time: a few - quite short ones - at 2'700 days! Some people are patient. Or obstinate...

There are over 600 stories with more than 1'000 reviews apiece, totalising 88 million words...
Should I upload the file? With more stories? (starting at 5'000 words?)

Have a nice day,
Pascal

PS:
I was just wondering how many words are in all the reviews...

First, artina, it's a useful idea, but the lack of whitespace is death. There's an edit link (or there should be) on your original post -- would you mind injecting whitespace to help out my eyes? :)

Second, ilovecats, I'd suggest a few heuristics to make your analysis more useful. Off the cuff ideas to realize that sheer number of reviews is pretty pointless:

- For any review that has more than 6 exclamation marks in total, more than 3 exclamation or question marks in a row, or any individual word comprising more than 25 characters, count the review as not existing

- For any review of less than 20 words, count it as not existing; for any review of less than 40 words, weight it with k=0.5

- For any story that has multiple reviews by the same reader, if the average review in that story is of total word length less than 60 words, after applying the prior rules replace their total contribution to the count for that story as "ceiling( reviews_by_reader / 10 )" -- or perhaps instead of 10, "ceiling(total_chapters / 10)"

- For any story that has an average chapter length of less than 2,000 words, take the log2( review_count ) as the actual review count after applying the prior rules . . . or perhaps some other similar function. The crudest form might be "total_reviews / total_chapters".

Also, switch to "gnumeric" and you can use bigger spreadsheets. Alternately, blow off spreadsheets (to the horror of Sovran) and just manipulate it all via your scripts as .csv, SQL database, or the like. Write a separate tool to collect data and dump it into a secondary format that you can make pretty pictures from with a spreadsheet, ploticus, gnuplot, . . .

I wonder if OOo's Calc can do bigger spreadsheets? Might be a setting somewhere.

Hi Sovran!

Interesting ways of refining the search. Have a few problems:
- download time
- space on my hard drive (I would not be surprised if the reviews total more words than all the stories)
- the exponentially growing number of parameters to be checked (and the corresponding software complexity)
- my lack of a time-turner to pack more working hours in a day

BTW, the last proposal (the "crude" method) is the reason why I wanted to have a tabulated list...

When do we get more of "Meaning of One" and "The Contract"?

Well, that was Josh who had all the crafty technical thoughts. But hello. =)

MoO and The Contract are both waiting for me to have more time. There's one chapter of MoO in the beta process, and the next one is outlined and ready to be drafted, which means I need several hours of uninterrupted time to get started. The Contract needs less actual time, because the scenes are so short, but it requires a certain mindset.

Dave

Dave, you can write a few chapters of Echoes and I'll write a few chapters of MOO. Then we can really be cross-referenced. How's that sound?

RE: the review count . . .

Based on that comment, I was trying to point out the fallacy of just looking at review counts. Most stories have a "dedicated drones" group who review every chapter with a profound statement such as "wow, yer da best, update!!!!!!!" or some such. There is an extremely high correlation between chapter count and review count, and that correlation has zero to do with the quality of writing or the quality of reviews. I can point to examples at SIYE, PS, FF, etc.