SUNScholar/Elastic Statistics/Import Logs

BACK TO ELASTIC STATISTICS

Introduction
To get a full statistical history after enabling Elastic Statistics, requires the import of old log data. Below are details to do just that.

PLEASE NOTE:
When importing records, try not to duplicate records.

For example:

If you enabled Elastic Statistics a month previously, then do not import the logs for the past month, because you will duplicate records from the period when you first enabled Elastic Statistics.

Step 1 - Export old log data to suitable import format
mkdir $HOME/scripts
 * Create export script:

nano $HOME/scripts/stats-export
 * Copy and paste the following, then save the file:
 * 1) !/bin/sh

cd $HOME/log ITEM=`ls dspace.log.*`
 * 1) Uncomment the following to debug ##
 * 2) echo $ITEM

for i in $ITEM ; do	echo "###################################" echo "Exporting stats for log file:... $i" $HOME/bin/dspace stats-log-converter -n -i $i -o $i.export done

chmod 0775 $HOME/scripts/stats-export $HOME/scripts/stats-export
 * Make the script executable:
 * Convert the logs:

PLEASE NOTE:

 * For a busy web site, the imports will take a long time. You will be updating a long history of visits in a very short space of time on the server. Please be patient, the import may take hours, maybe even days!
 * You can speed up the imports on an Ubuntu server by installing a name server caching daemon, type: sudo apt-get install nscd before you start the imports.
 * The following can be used to run the import script in the background in a "detached mode".
 * https://en.wikipedia.org/wiki/Nohup
 * https://en.wikipedia.org/wiki/GNU_Screen
 * https://en.wikipedia.org/wiki/Tmux
 * If the import should fail, then check the "stats-import.log" file to see which imports succeeded. Then remove the import files that have been processed and begin the import again. Repeat this process until all the import files have been processed.

Procedure
mkdir $HOME/scripts
 * Create import script:

nano $HOME/scripts/stats-import
 * Copy and paste the following, then save the file:
 * 1) !/bin/sh

cd $HOME/log ITEM=`ls *.export`
 * 1) Uncomment the following to debug ##
 * 2) echo $ITEM

for i in $ITEM ; do	echo "###################################" > stats-import.log echo "Importing stats for log file:... $i" > stats-import.log $HOME/bin/dspace stats-log-importer -v -i $i echo "Import of:... $i completed. Resting for a while!" > stats-import.log sleep 3 done chmod 0755 $HOME/scripts/stats-import $HOME/scripts/stats-import
 * Make the script executable:
 * Import the converted log files

Export Utility

 * $HOME/bin/dspace stats-log-converter

usage: ClassicDSpaceLogConverter -h,--help       help -i,--in   source file ('-' or omit for standard input) -m,--multiple   treat the input file as having a wildcard ending -n,--newformat  process new format log lines (1.6+) -o,--out  destination file or directory ('-' or omit for standard                  output) -v,--verbose    display verbose output (useful for debugging)

ClassicDSpaceLogConverter -i infilename -o outfilename -v (for verbose output)

Import Utility

 * $HOME/bin/dspace stats-log-importer-elasticsearch

usage: StatisticsImporterElasticSearch -h,--help      help -i,--in  the input file ('-' or omit for standard input) -m,--multiple  treat the input file as having a wildcard ending -s,--skipdns   skip performing reverse DNS lookups on IP addresses -v,--verbose   display verbose output (useful for debugging)