Difference between revisions of "SUNScholar/Daily Admin"

From Libopedia
Jump to navigation Jump to search
 
(46 intermediate revisions by one other user not shown)
Line 4: Line 4:
 
</center>
 
</center>
  
==Step 1. Login==
+
==Introduction==
http://wiki.lib.sun.ac.za/index.php/SUNScholar/Prepare_Ubuntu/S01
+
<font color="red">'''Just after installation it is critically important that you enable daily automated tasks for your digital archive.'''</font>
Click on the link above to find out how to login to the server and then return here.
 
  
==Step 2. Create "dspace" user crontab==
+
In order to send out subscription emails, update search, browse, full-text indexes and do general daily housekeeping on the system, a regular maintenance script must be run automatically daily. On a Unix/Linux based system this is easy to accomplish with use of the "crontab" facility.
The "dspace" user has to perform tasks automatically at regular intervals, such as sending out subscription emails.
 
  
On a Linux/Unix system this is easy to accomplish using the "cron" functionality.
+
==Requirements==
 +
<font color="red">'''[[SUNScholar/Install_DSpace/S03#Step_3.2|Click here]] to setup the "source" folder first.'''</font>
 +
==Instructions==
  
===Sample crontab for DSpace => 3.2===
+
===[[SUNScholar/Daily Admin/6.X|For DSpace 6.X]]===
Refer to original at: '''https://github.com/DSpace/demo.dspace.org/blob/master/scripts/linux/crontab'''
+
===[[SUNScholar/Daily Admin/5.X|For DSpace 5.X]]===
 +
===[[SUNScholar/Daily Admin/4.X|For DSpace 4.X]]===
 +
===[[SUNScholar/Daily Admin/3.X|For DSpace 3.X]]===
  
Edit the crontab, by typing the following in a terminal:
+
==YouTube Video==
 
+
<html5media width="560" height="315">https://www.youtube.com/watch?v=jH8tI0byvFA</html5media>
su - dspace
 
 
 
crontab -e
 
If asked to select an editor, choose '''nano'''
 
 
 
;Instructions
 
# Delete all the contents of the file using '''CTL+k''' repeatedly
 
# Copy and paste the following into the open file.
 
# Save the file by typing '''CTL+o'''
 
# Exit the file by typing '''CTL+x'''
 
<pre>
 
####################################
 
# Initialize Environment Variables #
 
####################################
 
# Deliver cron email to system administrator
 
MAILTO="%myadmin@repo.ac.za%"
 
 
 
#Ensure that HOME is set properly for our service
 
HOME=/home/dspace
 
 
 
#Add Java to PATH (for all DSpace cron jobs)
 
#Also add all major 'bin' directories
 
PATH=/usr/bin:/bin:/usr/local/bin
 
 
 
#Specify default Java options (for all DSpace cron jobs)
 
JAVA_OPTS=-Xmx512M -Xms512M -Dfile.encoding=UTF-8
 
 
 
##########
 
# Hourly #
 
##########
 
# Regenerate System Sitemaps every 8 hours
 
# at 3 minutes past the hour
 
3  0,8,16 * * * $HOME/bin/dspace generate-sitemaps > /dev/null
 
 
 
# Update item counts every hour
 
@hourly   $HOME/bin/dspace itemcounter > /dev/null
 
 
 
###########
 
# Nightly #
 
###########
 
# 12:30AM
 
# Run DSpace statistical analysis tools (12months takes approx 40secs)
 
30 0 * * * $HOME/bin/dspace stat-general  > /dev/null
 
35 0 * * * $HOME/bin/dspace stat-monthly  > /dev/null
 
 
 
# 1:00AM
 
# Generate DSpace statistical analysis reports
 
00 1 * * * $HOME/bin/dspace stat-report-general  > /dev/null
 
05 1 * * * $HOME/bin/dspace stat-report-monthly  > /dev/null
 
 
 
# 2:00AM
 
# Run the DSpace media filter scripts and save all output to a log file
 
# (Creates image thumbnails, indexes full text, and
 
# performs any automated format conversions)
 
0 2 * * * $HOME/bin/dspace filter-media > $HOME/log/media-filter.log 2>&1
 
 
 
# 3:00AM
 
# Update simple, advanced search and browse indexes
 
0 3 * * *  $HOME/bin/dspace index-update > /dev/null
 
 
 
# 5:00AM
 
# Check for items to release from embargo in DSpace. 
 
0 5 * * * $HOME/bin/dspace embargo-lifter > $HOME/log/embargo-release.log 2>&1
 
 
 
# 6:00AM
 
# Run XOAI incremental import (and optimization)
 
0 6 * * * $HOME/bin/dspace oai import -o > /dev/null
 
 
 
# 6:30AM
 
# Optimise Discovery Solr Index
 
30 6 * * * $HOME/bin/dspace update-discovery-index -o > /dev/null
 
# Cleanup Web Spiders from DSpace Statistics Solr Index
 
# -i deletes all spiders matched by IP address, DNS name or Agent name
 
30 6 * * * $HOME/bin/dspace stats-util -i > /dev/null
 
 
 
# 7:00AM
 
# Optimize DSpace Statistics Solr Index
 
0 7 * * * $HOME/bin/dspace stats-util -o > /dev/null
 
 
 
# 8:00AM
 
# Send out DSpace subscription emails
 
# (This alerts users of newly deposited items of interest)
 
0 8 * * * $HOME/bin/dspace sub-daily > /dev/null
 
 
 
##########
 
# Weekly #
 
##########
 
 
 
###########
 
# Monthly #
 
###########
 
# 12:01AM
 
# Remove all DSpace log files which are more than 30 days old
 
# on the first of every month
 
01 0 1 * * find $HOME/log/*.log.* -mtime +30 -exec rm {} \;
 
 
 
# 1:00AM
 
# Completely remove any deleted bitstreams in DSpace
 
# on the first of every month
 
0 1 1 * * $HOME/bin/dspace cleanup > /dev/null
 
 
 
# 2:00AM
 
# Delete any ~/dspace/config/*/*.old files more than 30 days old (created by "ant update")
 
0 2 1 * * find $HOME/config -name "*-*-*.old" -mtime +30 -exec rm {} \;
 
# Delete any ~/dspace/*.bak-*-*/ directories more than 30 days old (created by "ant update")
 
0 2 1 * * find $HOME/*.bak-*-* -maxdepth 0 -type d -mtime +30 -exec rm -rf {} \;
 
</pre>
 
 
 
{{NANO}}
 
 
 
===Sample crontab for DSpace <= 1.8.2 (superseded, see above)===
 
Edit the crontab, by typing the following in a terminal:
 
 
 
su - dspace
 
 
 
crontab -e
 
If asked to select an editor, choose '''nano'''
 
 
 
Remove all the contents by pressing '''CTL+k''' repeatedly, then add the following lines using the copy and paste functions of firefox and nano:
 
 
 
'''Replace %myadmin@repo.ac.za% with the email address of the person to receive system admin email.'''
 
<pre>
 
# m h  dom mon dow  command
 
MAILTO="%myadmin@repo.ac.za%"
 
 
 
### Setup embargoes before enabling this ###
 
### After embargoes are setup, remove the hash to enable embargo checking ###
 
### See wiki customisation page for setup details ###
 
### http://wiki.lib.sun.ac.za/index.php/SUNScholar/Embargo_Systems ###
 
## Check for lifted embargos ##
 
#05 0 * * *      /home/dspace/bin/dspace embargo-lifter -a -v
 
##The above option is made available with DSpace 3.2 only
 
#10 0 * * *      /home/dspace/bin/dspace embargo-lifter -l -v
 
#15 0 * * *      /home/dspace/bin/dspace embargo-lifter -c -v
 
 
 
### See wiki customisation page for setup details ###
 
### http://wiki.lib.sun.ac.za/index.php/SUNScholar/Browse_Indexes ###
 
## Update item search and browse indexes ##
 
0 3 * * *      /home/dspace/bin/dspace index-update
 
 
 
### See wiki customisation page for setup details ###
 
### http://wiki.lib.sun.ac.za/index.php/SUNScholar/SOLR_Statistics ##
 
## SOLR stats update ##
 
@daily          /home/dspace/bin/dspace stats-util -b -r 
 
@weekly        /home/dspace/bin/dspace stats-util -o
 
@monthly        /home/dspace/bin/dspace stats-util -u -m -i -f
 
 
 
### See wiki customisation page for setup details ###
 
### http://wiki.lib.sun.ac.za/index.php/SUNScholar/SOLR_Discovery ###
 
## SOLR discovery update ##
 
@daily          /home/dspace/bin/dspace update-discovery-index
 
@weekly        /home/dspace/bin/dspace update-discovery-index -o
 
@monthly        /home/dspace/bin/dspace update-discovery-index -c
 
 
 
## Generate sitemaps for Google indexing ##
 
0 6 * * *      /home/dspace/bin/dspace generate-sitemaps
 
 
 
## Item counts ##
 
@hourly /home/dspace/bin/dspace itemcounter
 
 
 
## Send out subscription e-mails ##
 
0 1 * * *      /home/dspace/bin/dspace sub-daily
 
 
## Run the media filter ##
 
## This is setup to run weekly to reduce server load ##
 
## It can be setup to run nightly if you want ##
 
@weekly         /home/dspace/bin/dspace filter-media
 
 
 
## Bitstream checks ##
 
@weekly        /home/dspace/bin/dspace checker-emailer -a
 
@monthly        /home/dspace/bin/dspace checker -p -l
 
@monthly  /home/dspace/bin/dspace cleanup -v
 
 
 
### The following are not needed if using the new SOLR stats above ###
 
## Run stat analyses ##
 
#10 1 * * *      /home/dspace/bin/dspace stat-general
 
#20 1 * * *      /home/dspace/bin/dspace stat-monthly
 
 
 
## Run stat analyses reports ##
 
#10 2 * * *      /home/dspace/bin/dspace stat-report-general
 
#20 2 * * *      /home/dspace/bin/dspace stat-report-monthly
 
</pre>
 
 
 
{{NANO}}
 
 
 
==Step 3. Create "postgres" user crontab==
 
PostgreSQL also benefits from regular 'vacuuming', which optimizes the indexes and clears out any deleted data.
 
 
 
Type as follows in a terminal:
 
 
 
sudo -i
 
Enter the "dspace" user password. Then type:
 
su - postgres
 
 
 
crontab -e
 
Copy and paste the following:
 
20 4 * * * vacuumdb --analyze dspace > /dev/null 2>&1
 
 
 
{{NANO}}
 
  
 
==References==
 
==References==
 
*https://github.com/DSpace/demo.dspace.org/blob/master/scripts/linux/crontab
 
*https://github.com/DSpace/demo.dspace.org/blob/master/scripts/linux/crontab
*https://wiki.duraspace.org/display/DSDOC4x/Scheduled+Tasks+via+Cron
+
*http://www.crontab-generator.org
*https://wiki.duraspace.org/display/DSDOC3x/Installation#Installation-'cron'Jobs
 
*https://wiki.duraspace.org/display/DSDOC18/Installation#Installation-'cron'Jobs
 
*https://wiki.duraspace.org/display/DSDOC17/Installation#Installation-'cron'Jobs
 
 
*http://en.wikipedia.org/wiki/Cron
 
*http://en.wikipedia.org/wiki/Cron
 
*http://adminschoice.com/crontab-quick-reference
 
*http://adminschoice.com/crontab-quick-reference
 +
*http://www.yourownlinux.com/2014/04/schedule-your-jobs-in-linux-with-cron-examples-and-tutorial.html
 +
*http://ask.xmodulo.com/add-cron-job-linux.html
 +
*http://teknixx.com/working-with-cron-jobs-on-linux
 +
*http://www.cyberciti.biz/faq/disable-the-mail-alert-by-crontab-command
 +
*http://techarena51.com/index.php/a-beginners-guide-to-bash-scripting
 +
*http://www.linuxandubuntu.com/home/linux-utility-understand-linux-crontab-or-cron-expressions-to-run-command-at-specific-interval
 +
{{Tomcat7}}
 +
 +
[[Category:System Administration]]
 +
[[Category:Installation]]
 +
__NOTOC__

Latest revision as of 14:38, 30 October 2019

Back to After Installation Tasks

Introduction

Just after installation it is critically important that you enable daily automated tasks for your digital archive.

In order to send out subscription emails, update search, browse, full-text indexes and do general daily housekeeping on the system, a regular maintenance script must be run automatically daily. On a Unix/Linux based system this is easy to accomplish with use of the "crontab" facility.

Requirements

Click here to setup the "source" folder first.

Instructions

For DSpace 6.X

For DSpace 5.X

For DSpace 4.X

For DSpace 3.X

YouTube Video

References

When using the Ubuntu 16.04 LTS server change all instances of tomcat to tomcat8.
When using the Ubuntu 14.04 LTS server change all instances of tomcat to tomcat7.
When using the Ubuntu 12.04 LTS server change all instances of tomcat to tomcat6.