SUNScholar/Elastic Statistics/Import Logs

BACK TO ELASTIC STATISTICS

Introduction

To get a full statistical history after enabling Elastic Statistics, requires the import of old log data. Below are details to do just that.

Step 1 - Export old log data to suitable import format

I prepared the following bash script to do the conversion.

#!/bin/sh

cd /home/dspace/log
ITEM=`ls dspace.log.*`
#echo $ITEM

for i in $ITEM ; do
	echo "###################################"
	echo "Exporting stats for log file:... $i"
	/home/dspace/bin/dspace stats-log-converter -n -v -i $i -o $i.export
done

After running the script you will have a lot of log files with the .export extension. This takes quite a while with a lot of log files. Be patient.

Step 2 - Import prepared log data to Elastic Statistics

I prepared the following bash script to do the import.

#!/bin/sh

cd /home/dspace/log
ITEM=`ls *.export`
#echo $ITEM

for i in $ITEM ; do
	echo "###################################"
	echo "Importing stats for log file:... $i"
	/home/dspace/bin/dspace stats-log-importer -v -i $i
done

This takes quite a while with a lot of log files. Be patient.

References

Export Utility

/home/dspace/bin/dspace stats-log-converter

usage: ClassicDSpaceLogConverter
       
 -h,--help        help
 -i,--in <arg>    source file ('-' or omit for standard input)
 -m,--multiple    treat the input file as having a wildcard ending
 -n,--newformat   process new format log lines (1.6+)
 -o,--out <arg>   destination file or directory ('-' or omit for standard
                  output)
 -v,--verbose     display verbose output (useful for debugging)

	ClassicDSpaceLogConverter -i infilename -o outfilename -v (for verbose output)

Import Utility

/home/dspace/bin/dspace stats-log-importer-elasticsearch

usage: StatisticsImporterElasticSearch
       
 -h,--help       help
 -i,--in <arg>   the input file ('-' or omit for standard input)
 -m,--multiple   treat the input file as having a wildcard ending
 -s,--skipdns    skip performing reverse DNS lookups on IP addresses
 -v,--verbose    display verbose output (useful for debugging)