SUNScholar/Elastic Statistics/Import Logs

BACK TO ELASTIC STATISTICS

Introduction

To get a full statistical history after enabling Elastic Statistics, requires the import of old log data. Below are details to do just that.

Step 1 - Export old log data to suitable import format

• Create export script:

mkdir $HOME/scripts

nano $HOME/scripts/export-stats

• Copy and paste the following, then save the file:

#!/bin/sh

cd /home/dspace/log
ITEM=`ls dspace.log.*`
#echo $ITEM

for i in $ITEM ; do
	echo "###################################"
	echo "Exporting stats for log file:... $i"
	/home/dspace/bin/dspace stats-log-converter -n -v -i $i -o $i.export
done

---

NANO Editor Help

CTL+O	= Save the file and then press Enter
CTL+X	= Exit "nano"
CTL+K	= Delete line
CTL+U	= Undelete line
CTL+W	= Search for %%string%%
CTL+\	= Search for %%string%% and replace with $$string$$
CTL+C	= Show line numbers

More info = http://en.wikipedia.org/wiki/Nano_(text_editor)

---

• Make the script executable:

chmod 0775 $HOME/scripts/export-stats

• Convert the logs:

$HOME/scripts/export-stats

Step 2 - Import prepared log data to Elastic Statistics

• Create import script:

mkdir $HOME/scripts

nano $HOME/scripts/import-stats

• Copy and paste the following, then save the file:

#!/bin/sh

cd /home/dspace/log
ITEM=`ls *.export`
#echo $ITEM

for i in $ITEM ; do
	echo "###################################"
	echo "Importing stats for log file:... $i"
	/home/dspace/bin/dspace stats-log-importer -v -i $i
done

• Make the script executable:

chmod 0755 $HOME/scripts/import-stats

• Import the converted log files

$HOME/scripts/import-stats

References

Export Utility

/home/dspace/bin/dspace stats-log-converter

usage: ClassicDSpaceLogConverter
       
 -h,--help        help
 -i,--in <arg>    source file ('-' or omit for standard input)
 -m,--multiple    treat the input file as having a wildcard ending
 -n,--newformat   process new format log lines (1.6+)
 -o,--out <arg>   destination file or directory ('-' or omit for standard
                  output)
 -v,--verbose     display verbose output (useful for debugging)

	ClassicDSpaceLogConverter -i infilename -o outfilename -v (for verbose output)

Import Utility

/home/dspace/bin/dspace stats-log-importer-elasticsearch

usage: StatisticsImporterElasticSearch
       
 -h,--help       help
 -i,--in <arg>   the input file ('-' or omit for standard input)
 -m,--multiple   treat the input file as having a wildcard ending
 -s,--skipdns    skip performing reverse DNS lookups on IP addresses
 -v,--verbose    display verbose output (useful for debugging)