Mock The Movie: Expect No Mercy transcript

Again, I managed to forget to start my scrape bot to pull tweets from Mentions directly. CA7746 bailed me out of a bit of a jam by reparsing the raw HTML of Twitter, a trick I’ve done once already but have evidently lost the code for. I was going to rewrite that parser tonight, but CA7746 has evidently spared me the difficulty.

My usual scrape bot, which pulls from @-mentions from the account proper, could only grab the last 200 statuses — a limitation of the API, it seems. Either I haven’t figured out how to paginate through the results properly, or it simply won’t let me do so the same way as paginating through a direct search for @MockTM would. I might rebuild the engine to grab transcripts from @MockTM searches, though that would mean we wouldn’t be able to limit the tweets pulled to only those people @MockTM has followed. That would mean letting potential spam in.

In case there’s anything spammy above the double-dash (haven’t had time to reread it all), let me know and I’ll pull it out.

[Read more...]