You may have heard that Scienceblogs is being shuttered at the end of this month, which is a real shame. I have a massive pile of data over there, and I’d like to bring it over here.
So I used the export tool to move all the Sciblogs Pharyngula data to an xml file. It was quick, too quick. I ended up with a 245mb xml file, which seems too small to contain all of the images, text, and comments. But OK, text is small, maybe it’s all there.
Then I go to my FtB dashboard to import the file. It tells me there’s a 10mb file limit! Wait, what? That makes no sense.
I stopped right there. I know that when Scienceblogs made their upgrade to WordPress, way back when, they made an utter botch of it, losing about 2/3 of the comments and messing up all the internal links. I don’t want to wreck this site, too.
Is there a simple solution to this dilemma?
Is there a complex solution that I can execute to do this job?
If there is a complex solution that I would probably screw up, are there any pros who can help me out? And how much will it cost?
The file size limit is configurable in your php.ini file.
Do make sure to do this on a clone instead of your live site and put it through its paces before you publish the results to the world!
Probably no images in that….
WordPress expert here.
What you need to do, while this site is currently active, is to generate a WordPress Export file with Tools > Export in the admin section.
Your blog is so big a single export file may “choke” during its creation. In that case, split the export up into type and a time-span until the export works.
Next get a new WordPress host set up and use Tools > Import > WordPress to to import your export file(s). When you do, check the “Download and import file attachments” option and it’ll connect to the existing site and grab all the files and store them on the new site in the appropriate place.
That should be it. If you need any help, feel free to email me and I’d be happy to assist.
Er, I should mention: The big file problem and 10MB limit is fixable with an app I have that can split WordPress Exports into whatever size is needed.
Your new host should probably handle larger than 10MB though. It’s completely host-dependent.
How do I tell if the export “works”? There was no error message.
Actually, I’m pretty sure it failed. The xml file ends part way through a comment.
I don’t suppose you have
physical access to the hardware(I guess it’s in a New Jersey webhosting facility or someplace far away?) or, at the very least, remote terminal access and administrative control to stop/restart services, dump a database, tar and rsync an archive of the file tree somewhere?If you are on a linux machine, you may be able to at least get all files using wget:
wget -r -np http://www.scienceblogs.com/pharyngula
-r: rekursiv
-np: no parent
This should copy the whole filetree, at least it worked (some time ago) fpr me on German scienceblogs.
Protip: Don’t use wordpress.
You should get the sciblogs people to export your data as csv/json directly from the database. That way you can be sure you have all of it.
I have previously manually split a WordPress export file to make it small enough to import. It’s fiddly, but do-able. But obviously you need a successful export file to begin with!
As @Amatueur mentioned, the sum total of all the site in its current form would be a tar file of the WordPress folder and a dump of the (probably MySQL) database. I’d be trying to snapshot those so you can be sure nothing gets left behind once the shutters go up.
Once you have that you can at least be confident you have all the data in one form or another. I would have thought that was something you could reasonably request from Scienceblogs.
After that it’s a matter of how much you want to get your hands dirty or if you can hand it to a trusted associate. For example you could fire up a virtual machine on your pc running Linux, Apache, MySQL and php, import your data snapshot and have a local version of the site to play with.
I seem to recall you have an IT background so I’m sure you’re aware none if this will happen without a few facepalms and much head-banging.
I seem to remember when I archived my little blog with the export option on WP, it generated a multitude of files not just one.
Also, WordPress offers a “guided transfer” service for 129 bucks?
Your php installation has a maximum upload size set for files, 10MiB is the default I believe.
You can change that in php.ini, have whoever administers your system make the appropriate change and possibly restart the webserver and you should be fine.
I’m no expert, but it occurs to me that a lot of the posts from ~2011-present were duplicated on Sb and here. So there will no doubt be a lot of post name collisions. I was thinking how this should be handled (merge? duplicate? hopefully not overwrite!), and I am now wondering if it might not be less of a hassle to create a 2nd freethoughtblogs account (pharyngula-sb, or sb-pharyngula, or whatever), and import the Sb backup to that account.
Experts, what say you?
Hi Doc ,your think you have problems ,my car has a whining sound coming from the front end .
At first i thought it was my twin brother ,but no. gearbox bearing on the way out .
£1600 for a recon box and new clutch .
These are the times you would like a god botherer to knock on your door .
As for your piffling little problems ,didn’t understand of word of what you posted .
There’s also the point that the comment import from the Movable Type version of Pharyngula was badly munged, and never actually completed. In many (most?) comments, Unicode characters show up in WordPress Sb as “?”. Also, some typoed HTML was/is in some of the imported comments, so entire comment threads show up as all italics or bold after a certain point, or have other formatting problems.
It would be nice if the original Movable Type post-and-comment database could be made available for a cleaned-up import.
It would also be nice if I had a flying autopiloting car.
Not an WP expert, but have created XML files before. I imagine that WP, uses PHP scripts to create XML files. Been there, done that. These files are created in memory, so there’ll be a limit to their size. Also a PHP script may die after (usually) 30 seconds seconds of CPU time. Can you export a few month’s’ worth at a time, to make the output files smaller?
I suggest pinging Orac. He seems to have found a way to do this in pieces – although it seems somewhat labour intensive, as he has warned of going dark on blogging till done:
http://scienceblogs.com/insolence/2017/10/17/a-change-is-gonna-come/
I will shed a tear for its passing, long as it’s been since I frequented the place. It was one of the first three sites I commented on ever, and one of the others went defunct a few years ago as well.
I’ve administered several WordPress sites and performed a number of import/exports. Feel free to get in touch if you need some help.