SUNScholar/Export and Import Artifacts

From Libopedia
Revision as of 15:48, 14 May 2010 by Hgibson (talk | contribs) (→‎Create the import script)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Introduction

The functionality to export communities, whole collections and items only became available with Dspace version 1.5.2. For older versions of Dspace please check the following procedures and instructions.

Determining Collection and Item ID's

Collection ID's are the ones that appear in the browser URL when you hover over the collection with a mouse. Check the URL for the full handle. The collection ID is shown after handle prefix, ie. If you have not registered with CNRI, the number that appears after 123456789/. See the screenshot below. Use the same procedure to determine an item ID.

400px

Export on the old server

Create the folders

Become the root user by typing: 

sudo -i

Make an exports folder as follows: 

mkdir /home/exports

Make a scripts folder as follows: 

mkdir /root/scripts

Create export script

Make an export script file as follows: 

nano /root/scripts/collection-export

Copy and paste the following to the file open with the editor. 

#!/bin/bash
# Define the configs.
. config

# Create the folders
if [ ! -d $EXPORT ]; then
    echo "Please make an export folder available."
    exit 1
fi

# Do the exports.
for i in `cat collections`; do
     ECID=$i
    if [ ! -d $EXPORT/$ECID ]; then
	mkdir $EXPORT/$ECID
        echo "Exporting collection no: $ECID" >> $EXPORT/export.log
	$DSHOME/bin/export -m -d $EXPORT/$ECID -t COLLECTION -n 100 -i $HANDLE/$ECID
    else
        echo "Collection:$ECID already exported." >> $EXPORT/export.log	
    fi
done

# Give all permissions
chmod -R 0777 $EXPORT
chown -R root.root $EXPORT

Please note: The -m means export without metadata. This is for migration not movement of assets. Only available from DSpace version 1.6.0.

Save the file and exit the editor.

Now make the file executeable as follows: 

chmod 0755 /root/scripts/collection-export

Create export config file

Make a config file as follows: 

nano /root/scripts/config

Copy and paste the following into editor. 

HANDLE="123456789"
EXPORT="/home/exports"
DSHOME="/home/dspace"
PERSON="admin@myrepo.ac.za"

Modify the above to suit your site and then save and exit the file.

Create export collections file

Make collections file as follows: 

nano /root/scripts/collections

Copy and paste the following into the editor. 

12
34
23
56
21

Change the above to reflect the collection handle ID's you want to export. See the procedure outlined at the top of this page to determine the collection ID's.

Save the file and exit.

Do the export

Run the script as follows: 

/root/scripts/collection-export

To watch the log file as the exports happen, type the following after opening another terminal: 

tail -f /home/exports/export.log

Create an exports tarball archive

After the exports are complete, make a tarball backup as follows: 

cd /home

tar -czvf exports.tgz exports/

Now copy the tarball (exports.tgz) to a portable disk or remote backup device.

Import on the new server

Copy the exports tarball archive to the new server

Copy the tarball (exports.tgz) of items exported on the old machine to the home folder on the new machine. Try the following command to do the copy from one Ubuntu machine to another Ubuntu machine. Open a terminal on the machine where the items were exported and type: 

scp dspace@%old-host%:/home/exports.tgz dspace@%new-host%:/home/

Replace %old-host% with host name of your old server. Replace %new-host% with host name of your new machine. It is assumed that the "dspace" user account exists on the new machine.

If the above method does not work, then do the transfer manually using a portable disk or any other method that works for you. You could even try to use WinSCP.

Create the communities and collections on the new server

Now manually create the same communties and collections that were on the old machine, on your new machine. Take note of the collection id's created on the new machine (see the screenshot and instructions above) and check them against the old collections id's on the old machine using a spreadsheet, for example: 

old-collection-id = new-collection-id

Print a copy of the spreadsheet to use for the following procedures.

Also save a list of submitter email addresses (ePersons) per new collection. You might think of defining one submitter email address (ePerson) to use for the imports only.

Create the folders

Login to the new server and become the root user as follows: 

sudo -i

Make an imports folder as follows: 

mkdir /home/imports

Make a script folder as follows: 

mkdir /root/scripts

Extract the exports tarball archive on the new server

Run the following command to untar the tarball contents into the new server's /home/exports folder. 

tar -C /home -xzvf /home/exports.tgz

Check that the /home/exports folder contains the items exported from the old server by typing the following: 

ls /home/exports

A list of folders should be displayed that represents your exported collections by collection id. If they are not there then check up on what went wrong with the copy and extraction of the exports tarball archive !

Create the import script

We create a symlink to hold the "imports" folder. 

cd /home

ln -s exports imports

Open the import script for editing as follows: 

nano /root/scripts/collection-import

Copy and paste the following into the editor: 

#!/bin/bash

# Setup the configs.
. config

# Check for an exports folder
if [ ! -d $IMPORT ]; then
    clear
    echo "Have you extracted the collections to the correct folder."
    exit 1
fi

# Go thru the collections
for i in `cat collections` ; do
    echo $i > /tmp/map
    ECID=`cat /tmp/map | awk -F',' '{ print $1 }'` 
    ICID=`cat /tmp/map | awk -F',' '{ print $2 }'`
    echo "Importing old collection:$ECID to new collection:$ICID" >> $IMPORT/import.log
    if [ ! -e $IMPORT/$ECID/$ECID-to-$ICID-items-map-list.csv ] ; then
        $DSHOME/bin/import -a -w -e $PERSON -s $IMPORT/$ECID -c $HANDLE/$ICID -m $IMPORT/$ECID/$ECID-to-$ICID-items-map-list.csv
        else
	echo "$IMPORT/$ECID collection already imported" >> $IMPORT/import.log
        clear
    fi
done

Please note:

  1. Check the DC metadata fields in the destination server and ensure that any customisation from the source server is also updated on the destination server.
  2. The -w means add to the collections workflow. This is for movement of assets not migration.

Save the file and exit the editor.

Now make the script executeable as follows: 

chmod 0775 /root/scripts/collection-import

Create import config file

Make a config file as follows: 

nano /root/scripts/config

Copy and paste the following into editor. 

HANDLE="123456789"
IMPORT="/home/imports"
DSHOME="/home/dspace"
PERSON="submitter@myrepo.ac.za"

Modify the above to suit your new site and then save and exit the file.

Create import collections file

Make collections file as follows: 

nano /root/scripts/collections

Copy and paste the following into the editor. 

12,34
34,43
23,56
56,78
21,99

Change the above to reflect the collection handle ID's you want to import using the spreadsheet that maps new collection ID's to old collection ID's. The first item per line in the file is the exported collection id and the second item after the comma is the new import collection id.

Save the file and exit.

Do the import

Run the script as follows: 

/root/scripts/collection-import

To watch the log file as the imports happen, type the following after opening another terminal: 

tail -f /home/imports/import.log

Test the exports for correctness if imports fail

The imports will fail if the exports are not correct. Use the following to setup a script to check the exports for correctness on the new server.

Create export check script

Open the export check script for editing as follows: 

nano /root/scripts/collection-check

Copy and paste the following to open editor: 

#!/bin/bash

# Setup the configs.
. config

# Clean out the old logs.
rm /root/test.log

# Go thru the collections
for i in `cat collections` ; do
    echo $i > /tmp/map
    ECID=`cat /tmp/map | awk -F',' '{ print $1 }'` 
    ICID=`cat /tmp/map | awk -F',' '{ print $2 }'`
    echo "Checking items in collection:$ECID"
    LIST=`ls $IMPORT/$ECID`
    
    for i in $LIST ; do
	ITM=$i
	cd  $IMPORT/$ECID/$ITM
	if [ ! -e dublin_core.xml ] ; then
	echo "Item: $ITM, No dublin core." >> /root/test.log
	fi
	if [ ! -e handle ] ; then
	echo "Item: $ITM, No handle." >> /root/test.log
	fi
	if [ ! -e contents ] ; then
	echo "Item: $ITM, No contents file." >> /root/test.log
	fi
	if [ ! -e license.txt ] ; then
	echo "Item: $ITM, No license." >> /root/test.log
	fi
	ls $IMPORT/$ECID/$ITM | grep pdf > /tmp/pdf.log
	if [ $? -ne 0 ] ; then
	echo "Item: $ITM, No PDF file." >> /root/test.log
	fi
    done
    cat /root/test.log | mail -s "Exported collections test log" $PERSON
    echo "Check complete and report emailed to dspace admin user"
done

Save the file and exit the editor.

Now make the script executeable as follows: 

chmod 0775 /root/scripts/collection-check

Do the exports check

Type the following to do the check: 

/root/scripts/collection-check

Now check your email, you should have one with details of the test log.

If there are errors, fix them on the old server. Delete the exports on the old server and do a new export after everything is fixed. Bascially start the export and import over again. At least you now have the scripts to do it over again.

Help Info

/home/dspace/bin/export

It takes the following parameters: 

-d,--dest     destination collection directory
-h,--help     help
-i,--id       collection/item ID
-n,--number   sequence number to begin exporting items with
-m,--migrate  strips out any per-repository-specifics
-t,--type     type: COLLECTION or ITEM

/home/dspace/bin/import

It takes the following parameters: 

-s,--source        source collection directory
-c,--collection    destination collection handle
-e,--eperson       email of authorised person
-m,--mapfile       collection/item map file name 
-a,--add           add items
-d,--delete        delete items listed in mapfile
-r,--replace       replace items listed in mapfile 
-w,--workflow      send submission through collection's workflow
-p,--template      apply template
-t,--test          test run - do not actually import items
-R,--resume        resume a failed import (add only)
-h,--help          help

References

Command Line Help

Go to: http://www.ubuntu.sun.ac.za/wiki/index.php/SelfHelp for more help about the command line programs used in this procedure.

Retrieved from "http://wiki.lib.sun.ac.za/index.php?title=SUNScholar/Export_and_Import_Artifacts&oldid=2615"