SUNScholar/Import on the new server
Back to Export & Import
Contents
- 1 Copy the exports tarball archive to the new server
- 2 Create the communities and collections on the new server
- 3 Create the folders
- 4 Extract the exports tarball archive on the new server
- 5 Create the import script
- 6 Create import config file
- 7 Create import collections file
- 8 Do the import
- 9 Test the exports for correctness if imports fail
Copy the exports tarball archive to the new server
Copy the tarball (exports.tgz) of items exported on the old machine to the home folder on the new machine. Try the following command to do the copy from one Ubuntu machine to another Ubuntu machine.
Open a terminal on the machine where the items were exported and type:
scp dspace@%old-host%:/home/exports.tgz dspace@%new-host%:/home/
Replace %old-host% with host name of your old server. Replace %new-host% with host name of your new machine. It is assumed that the "dspace" user account exists on the new machine.
If the above method does not work, then do the transfer manually using a portable disk or any other method that works for you. You could even try to use WinSCP.
Create the communities and collections on the new server
Now manually create the same communties and collections that were on the old machine, on your new machine. Take note of the collection id's created on the new machine and check them against the old collections id's on the old machine using a spreadsheet, for example:
old-collection-id = new-collection-id
Print a copy of the spreadsheet to use for the following procedures.
Also save a list of submitter email addresses (ePersons) per new collection. You might think of defining one submitter email address (ePerson) to use for the imports only.
Create the folders
Login to the new server and become the root user as follows:
sudo -i
Make an imports folder as follows:
mkdir /home/imports
Make a script folder as follows:
mkdir /root/scripts
Extract the exports tarball archive on the new server
Run the following command to untar the tarball contents into the new server's /home/exports folder.
tar -C /home -xzvf /home/exports.tgz
Check that the /home/exports folder contains the items exported from the old server by typing the following:
ls /home/exports
A list of folders should be displayed that represents your exported collections by collection id. If they are not there then check up on what went wrong with the copy and extraction of the exports tarball archive !
Create the import script
We create a symlink to hold the "imports" folder.
cd /home
ln -s exports imports
Open the import script for editing as follows:
nano /root/scripts/collection-import
Copy and paste the following into the editor:
#!/bin/bash # Setup the configs. . config # Check for an exports folder if [ ! -d $IMPORT ]; then clear echo "Have you extracted the collections to the correct folder." exit 1 fi # Go thru the collections for i in `cat collections` ; do echo $i > /tmp/map ECID=`cat /tmp/map | awk -F',' '{ print $1 }'` ICID=`cat /tmp/map | awk -F',' '{ print $2 }'` echo "Importing old collection:$ECID to new collection:$ICID" >> $IMPORT/import.log if [ ! -e $IMPORT/$ECID/$ECID-to-$ICID-items-map-list.csv ] ; then $DSHOME/bin/dspace import -a -e $PERSON -s $IMPORT/$ECID -c $HANDLE/$ICID -m $IMPORT/$ECID/$ECID-to-$ICID-items-map-list.csv else echo "$IMPORT/$ECID collection already imported" >> $IMPORT/import.log clear fi done
- NANO Editor Help
CTL+O | = Save the file and then press Enter |
CTL+X | = Exit "nano" |
CTL+K | = Delete line |
CTL+U | = Undelete line |
CTL+W | = Search for %%string%% |
CTL+\ | = Search for %%string%% and replace with $$string$$ |
CTL+C | = Show line numbers |
More info = http://en.wikipedia.org/wiki/Nano_(text_editor)
Please note:
- Check the DC metadata fields in the destination server and ensure that any customisation from the source server is also updated on the destination server.
- The "-w" adds the item to the collections workflow. This is for movement of assets not migration.
Now make the script executeable as follows:
chmod 0775 /root/scripts/collection-import
Create import config file
Make a config file as follows:
nano /root/scripts/config
Copy and paste the following into editor.
DSHOME="$HOME" HANDLE="123456789" IMPORT="$HOME/imports" PERSON="submitter@myrepo.ac.za"
Modify the above to suit your new site.
- NANO Editor Help
CTL+O | = Save the file and then press Enter |
CTL+X | = Exit "nano" |
CTL+K | = Delete line |
CTL+U | = Undelete line |
CTL+W | = Search for %%string%% |
CTL+\ | = Search for %%string%% and replace with $$string$$ |
CTL+C | = Show line numbers |
More info = http://en.wikipedia.org/wiki/Nano_(text_editor)
Create import collections file
Make collections file as follows:
nano /root/scripts/collections
Copy and paste the following into the editor.
12,34 34,43 23,56 56,78 21,99
Change the above to reflect the collection handle ID's you want to import using the spreadsheet that maps new collection ID's to old collection ID's. The first item per line in the file is the exported collection id and the second item after the comma is the new import collection id.
- NANO Editor Help
CTL+O | = Save the file and then press Enter |
CTL+X | = Exit "nano" |
CTL+K | = Delete line |
CTL+U | = Undelete line |
CTL+W | = Search for %%string%% |
CTL+\ | = Search for %%string%% and replace with $$string$$ |
CTL+C | = Show line numbers |
More info = http://en.wikipedia.org/wiki/Nano_(text_editor)
Do the import
Run the script as follows:
/root/scripts/collection-import
To watch the log file as the imports happen, type the following after opening another terminal:
tail -f /home/imports/import.log
Test the exports for correctness if imports fail
The imports will fail if the exports are not correct. Use the following to setup a script to check the exports for correctness on the new server. This script checks most things but not all bitstream types only PDF files.
Create export check script
Open the export check script for editing as follows:
nano /root/scripts/collection-check
Copy and paste the following to open editor:
#!/bin/bash # Setup the configs. . config # Clean out the old logs. rm /root/test.log # Go thru the collections for i in `cat collections` ; do echo $i > /tmp/map ECID=`cat /tmp/map | awk -F',' '{ print $1 }'` ICID=`cat /tmp/map | awk -F',' '{ print $2 }'` echo "Checking items in collection:$ECID" LIST=`ls $IMPORT/$ECID` for i in $LIST ; do ITM=$i cd $IMPORT/$ECID/$ITM if [ ! -e dublin_core.xml ] ; then echo "Item: $ITM, No dublin core." >> /root/test.log fi if [ ! -e handle ] ; then echo "Item: $ITM, No handle." >> /root/test.log fi if [ ! -e contents ] ; then echo "Item: $ITM, No contents file." >> /root/test.log fi if [ ! -e license.txt ] ; then echo "Item: $ITM, No license." >> /root/test.log fi ls $IMPORT/$ECID/$ITM | grep pdf > /tmp/pdf.log if [ $? -ne 0 ] ; then echo "Item: $ITM, No PDF file." >> /root/test.log fi done cat /root/test.log | mail -s "Exported collections test log" $PERSON echo "Check complete and report emailed to dspace admin user" done
- NANO Editor Help
CTL+O | = Save the file and then press Enter |
CTL+X | = Exit "nano" |
CTL+K | = Delete line |
CTL+U | = Undelete line |
CTL+W | = Search for %%string%% |
CTL+\ | = Search for %%string%% and replace with $$string$$ |
CTL+C | = Show line numbers |
More info = http://en.wikipedia.org/wiki/Nano_(text_editor)
Now make the script executeable as follows:
chmod 0775 /root/scripts/collection-check
Do the exports check
Type the following to do the check:
/root/scripts/collection-check
Now check your email, you should have one with details of the test log.
If there are errors, fix them on the old server. Delete the exports on the old server and do a new export after everything is fixed. Bascially start the export and import over again. At least you now have the scripts to do it over again.