Difference between revisions of "SUNScholar/Optimisations/Logs"

From Libopedia
Jump to navigation Jump to search
 
(27 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
<center>
 +
'''[[SUNScholar/Optimisations|Back to Optimisations]]'''
 +
</center>
 +
==Default Log Levels==
 +
With the release of DSpace 4.2, it is now possible to define default log levels.
 +
 +
Modif the the '''$HOME/source/build.properties''' file. See example below.
 +
<pre>
 +
#####################
 +
# LOGLEVEL SETTINGS #
 +
#####################
 +
loglevel.other = INFO
 +
# loglevel.other: Log level for other third-party tools/APIs used by DSpace
 +
# Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
 +
loglevel.dspace = INFO
 +
# loglevel.dspace: Log level for all DSpace-specific code (org.dspace.*)
 +
# Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
 +
</pre>
 +
 
==Tomcat Java web applications==
 
==Tomcat Java web applications==
 
Changed all instances of "INFO" to "ERROR" in the following config file to reduce disk I/O and CPU load.
 
Changed all instances of "INFO" to "ERROR" in the following config file to reduce disk I/O and CPU load.
−
  nano /home/dspace/config/log4j.properties
+
  nano $HOME/<u>[[SUNScholar/Install_DSpace/S03#Step_3.2|source]]</u>/dspace/config/log4j.properties
  
−
Then copied the file to dspace-src config folder.
+
See example below:
 +
<pre>
 +
###########################################################################
 +
#
 +
# log4j.properties
 +
#
 +
# Version: $Revision$
 +
#
 +
# Date: $Date$
 +
#
 +
###########################################################################
 +
 
 +
# This is the log4j configuration file for DSpace. Log4j can be configured to
 +
# write files that rotate every day or every month.
 +
#
 +
# Text surrounded by ${} symbols is replaced with the corresponding
 +
# property from dspace.cfg by Ant when DSpace is deployed.  For example:
 +
#
 +
# dspace.url
 +
#
 +
# would be replaced with the dspace.url property in dspace.cfg.
 +
#
 +
# Additional properties expanded by Maven during the DSpace assembly
 +
# process:
 +
#
 +
# loglevel.dspace (currently set to: ${loglevel.dspace})
 +
#  Log level for all DSpace-specific code (org.dspace.*)
 +
#  Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
 +
#
 +
# loglevel.other (currently set to: ${loglevel.other})
 +
#  Log level for other third-party tools/APIs used by DSpace
 +
#  Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
 +
 
 +
###########################################################################
 +
# A1 is the name of the appender for most DSpace activity.
 +
###########################################################################
 +
# The root category is the default setting for all non-DSpace code.
 +
# Change this from INFO to DEBUG to see extra logging created by non-DSpace
 +
# code.
 +
log4j.rootCategory=${loglevel.other}, A1
 +
# This line sets the logging level for DSpace code. Set this to DEBUG to see
 +
# extra detailed logging for DSpace code.
 +
log4j.logger.org.dspace=${loglevel.dspace}, A1
 +
# Do not change this line
 +
log4j.additivity.org.dspace=false
 +
# The name of the file appender
 +
log4j.appender.A1=org.dspace.app.util.DailyFileAppender
 +
# The filename of the log file created. A date stamp is appended to this
 +
log4j.appender.A1.File=${log.dir}/dspace.log
 +
# Set this to yyyy-MM-DD for daily log files, or yyyy-MM for monthly files
 +
log4j.appender.A1.DatePattern=yyyy-MM-dd
 +
# The number of log files to keep, or 0 to keep them all
 +
log4j.appender.A1.MaxLogs=0
 +
# A1 uses PatternLayout.
 +
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
 +
log4j.appender.A1.layout.ConversionPattern=%d %-5p %c @ %m%n
 +
 
 +
 
 +
###########################################################################
 +
# A2 is the name of the appender for the Checker
 +
###########################################################################
 +
# This line sets the logging level for the checksum checker log file.
 +
# Set this to DEBUG to see extra detailed logging.
 +
log4j.logger.org.dspace.checker=INFO, A2
 +
# Do not change this line
 +
log4j.additivity.org.dspace.checker=false
 +
# The name of the file appender
 +
log4j.appender.A2=org.dspace.app.util.DailyFileAppender
 +
# The filename of the log file created. A date stamp is appended to this
 +
log4j.appender.A2.File=${log.dir}/checker.log
 +
# Set this to yyyy-MM-DD for daily log files, or yyyy-MM for monthly files
 +
log4j.appender.A2.DatePattern=yyyy-MM-dd
 +
# The number of log files to keep, or 0 to keep them all
 +
log4j.appender.A2.MaxLogs=0
 +
# A2 uses PatternLayout.
 +
log4j.appender.A2.layout=org.apache.log4j.PatternLayout
 +
log4j.appender.A2.layout.ConversionPattern=%m%n
 +
 
 +
 
 +
###########################################################################
 +
# A3 is the name of the appender for Cocoon (XMLUI only)
 +
###########################################################################
 +
# These lines sets the logging level for the cocoon log file.
 +
# Set these to DEBUG to see extra detailed logging.
 +
log4j.logger.org.apache.cocoon=INFO, A3
 +
log4j.logger.cocoon=INFO, A3
 +
log4j.logger.org.springframework=INFO, A3
 +
# Do not change these lines
 +
log4j.additivity.org.apache.cocoon=false
 +
log4j.additivity.cocoon=false
 +
log4j.additivity.org.springframework=false
 +
# The name of the file appender
 +
log4j.appender.A3=org.dspace.app.util.DailyFileAppender
 +
# The filename of the log file created. A date stamp is appended to this
 +
log4j.appender.A3.File=${log.dir}/cocoon.log
 +
# Set this to yyyy-MM-DD for daily log files, or yyyy-MM for monthly files
 +
log4j.appender.A3.DatePattern=yyyy-MM-dd
 +
# The number of log files to keep, or 0 to keep them all
 +
log4j.appender.A3.MaxLogs=14
 +
# A2 uses PatternLayout.
 +
log4j.appender.A3.layout=org.apache.log4j.PatternLayout
 +
log4j.appender.A3.layout.ConversionPattern=%d %-5p %c %x - %m%n
 +
 
 +
 
 +
###########################################################################
 +
# A4 is the name of the appender for Solr
 +
###########################################################################
 +
log4j.logger.org.apache.solr=ERROR, A4
 +
log4j.additivity.org.apache.solr=false
 +
log4j.appender.A4=org.dspace.app.util.DailyFileAppender
 +
log4j.appender.A4.File=${log.dir}/solr.log
 +
log4j.appender.A4.DatePattern=yyyy-MM-dd
 +
log4j.appender.A4.MaxLogs=14
 +
log4j.appender.A4.layout=org.apache.log4j.PatternLayout
 +
log4j.appender.A4.layout.ConversionPattern=%d %-5p %c %x - %m%n
 +
 
 +
 
 +
###########################################################################
 +
# Other settings
 +
###########################################################################
 +
 
 +
# Block passwords from being exposed in Axis logs.
 +
# (DEBUG exposes passwords in Basic Auth)
 +
log4j.logger.org.apache.axis.handlers.http.HTTPAuthHandler=INFO
 +
 
 +
# Block services logging except on exceptions
 +
log4j.logger.org.dspace.kernel=ERROR
 +
log4j.logger.org.dspace.services=ERROR
 +
log4j.logger.org.dspace.servicemanager=ERROR
 +
log4j.logger.org.dspace.providers=ERROR
 +
log4j.logger.org.dspace.utils=ERROR
 +
</pre>
 +
 
 +
Then [[SUNScholar/Rebuild_DSpace|rebuild DSpace]].
 +
 
 +
 
 +
''In addition check the log settings for Tomcat in /etc/default/tomcat7.''
  
 
==SOLR==
 
==SOLR==
Line 9: Line 164:
 
'''Please note:''' The procedure below is not stable, so I installed PSI Probe using the Tomcat manager web interface and now I set the SOLR log level via the web gui after each re-build. See: https://code.google.com/p/psi-probe
 
'''Please note:''' The procedure below is not stable, so I installed PSI Probe using the Tomcat manager web interface and now I set the SOLR log level via the web gui after each re-build. See: https://code.google.com/p/psi-probe
  
−
See: http://lucidworks.lucidimagination.com/display/solr/Configuring+Logging and https://jira.duraspace.org/browse/DS-1460
+
See: http://lucidworks.lucidimagination.com/display/solr/Configuring+Logging and https://jira.duraspace.org/browse/DS-1460 and https://jira.duraspace.org/browse/DS-1946
  
 
Create the following file:
 
Create the following file:
−
  nano /home/dspace/dspace-1.8.2-src-release/dspace/modules/solr/src/main/webapp/WEB-INF/classes/logging.properties
+
  nano $HOME/<u>[[SUNScholar/Install_DSpace/S03#Step_3.2|source]]</u>/dspace/modules/solr/src/main/webapp/WEB-INF/classes/logging.properties
 
Add the following to the file:
 
Add the following to the file:
 
<pre>
 
<pre>
Line 21: Line 176:
 
{{NANO}}
 
{{NANO}}
  
−
Save the file and [[SUNScholar/Rebuild Dspace|rebuild]].
+
Save the file and [[SUNScholar/Rebuild DSpace|rebuild]].
 +
 
 +
''Also see: http://skybert.wordpress.com/2009/07/22/how-to-get-solr-to-log-to-a-log-file''
 +
 
 +
;See news from @atmire below.
 +
----
 +
Limiting log output for SOLR
 +
 
 +
Adding SOLR logging propertiesWe extensively covered the growth of log files and the different log levels in last month's article. In addition to the dspace logfiles contained in /dspace/log, some DSpace components write directly into the logfiles of your application container, Tomcat. One culprit in particular is SOLR. SOLR is the key component responsible for statistics, Discovery and most recently, OAI-PMH. The default logging level for SOLR is INFO, meaning that it will write to the Tomcat catalina.out logfile for each request it receives. To override this behavior add a file named '''logging.properties''' to the solr module found in '''$dspace.src$/dspace/modules/solr/src/main/resources'''.
 +
 
 +
Add the following instruction in this empty file:
 +
org.apache.solr.level = WARNING
 +
----
 +
 
 
===Log file permissions===
 
===Log file permissions===
 
I setup a root cron crontab as follows.
 
I setup a root cron crontab as follows.
Line 27: Line 195:
 
  sudo -i
 
  sudo -i
  
−
  mkdir /root/scripts
+
  crontab -e
 +
Add the following line.
 +
@midnight  chmod 0777 -R $HOME/log
  
−
nano /root/scripts/fix-log-perms
+
Save the crontab edit.
−
Copy and paste the following.
+
{{NANO}}
−
<pre>
+
See: http://wiki.lib.sun.ac.za/index.php/SUNScholar/Install_DSpace/S06#Step_6.3:_Setup_Tomcat_server_permissions for reference about file permissions.
−
#!/bin/bash
+
[[Category:System Administration]]
−
 
+
[[Category:Optimisations]]
−
chmod 0777 -R /home/dspace/logs
 
−
</pre>
 
−
Make it executeable.
 
−
chmod 0755 /root/scripts/fix-log-perms
 
−
 
 
−
'''[[SUNScholar/Optimisations|Back to Optimisations]]'''
 

Latest revision as of 18:31, 24 August 2016

Back to Optimisations

Default Log Levels

With the release of DSpace 4.2, it is now possible to define default log levels.

Modif the the $HOME/source/build.properties file. See example below. 

#####################
# LOGLEVEL SETTINGS #
#####################
loglevel.other = INFO
# loglevel.other: Log level for other third-party tools/APIs used by DSpace
# Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
loglevel.dspace = INFO
# loglevel.dspace: Log level for all DSpace-specific code (org.dspace.*)
# Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL

Tomcat Java web applications

Changed all instances of "INFO" to "ERROR" in the following config file to reduce disk I/O and CPU load. 

nano $HOME/source/dspace/config/log4j.properties

See example below: 

###########################################################################
#
# log4j.properties
#
# Version: $Revision$
#
# Date: $Date$
#
###########################################################################

# This is the log4j configuration file for DSpace. Log4j can be configured to
# write files that rotate every day or every month.
#
# Text surrounded by ${} symbols is replaced with the corresponding
# property from dspace.cfg by Ant when DSpace is deployed.  For example:
#
# dspace.url
#
# would be replaced with the dspace.url property in dspace.cfg.
#
# Additional properties expanded by Maven during the DSpace assembly
# process:
# 
# loglevel.dspace (currently set to: ${loglevel.dspace})
#   Log level for all DSpace-specific code (org.dspace.*)
#   Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
#
# loglevel.other (currently set to: ${loglevel.other})
#   Log level for other third-party tools/APIs used by DSpace
#   Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL

###########################################################################
# A1 is the name of the appender for most DSpace activity.
###########################################################################
# The root category is the default setting for all non-DSpace code.
# Change this from INFO to DEBUG to see extra logging created by non-DSpace
# code.
log4j.rootCategory=${loglevel.other}, A1
# This line sets the logging level for DSpace code. Set this to DEBUG to see
# extra detailed logging for DSpace code.
log4j.logger.org.dspace=${loglevel.dspace}, A1
# Do not change this line
log4j.additivity.org.dspace=false
# The name of the file appender
log4j.appender.A1=org.dspace.app.util.DailyFileAppender
# The filename of the log file created. A date stamp is appended to this
log4j.appender.A1.File=${log.dir}/dspace.log
# Set this to yyyy-MM-DD for daily log files, or yyyy-MM for monthly files
log4j.appender.A1.DatePattern=yyyy-MM-dd
# The number of log files to keep, or 0 to keep them all
log4j.appender.A1.MaxLogs=0
# A1 uses PatternLayout.
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d %-5p %c @ %m%n


###########################################################################
# A2 is the name of the appender for the Checker
###########################################################################
# This line sets the logging level for the checksum checker log file.
# Set this to DEBUG to see extra detailed logging.
log4j.logger.org.dspace.checker=INFO, A2
# Do not change this line
log4j.additivity.org.dspace.checker=false
# The name of the file appender
log4j.appender.A2=org.dspace.app.util.DailyFileAppender
# The filename of the log file created. A date stamp is appended to this
log4j.appender.A2.File=${log.dir}/checker.log
# Set this to yyyy-MM-DD for daily log files, or yyyy-MM for monthly files
log4j.appender.A2.DatePattern=yyyy-MM-dd
# The number of log files to keep, or 0 to keep them all
log4j.appender.A2.MaxLogs=0
# A2 uses PatternLayout.
log4j.appender.A2.layout=org.apache.log4j.PatternLayout
log4j.appender.A2.layout.ConversionPattern=%m%n


###########################################################################
# A3 is the name of the appender for Cocoon (XMLUI only)
###########################################################################
# These lines sets the logging level for the cocoon log file.
# Set these to DEBUG to see extra detailed logging.
log4j.logger.org.apache.cocoon=INFO, A3
log4j.logger.cocoon=INFO, A3
log4j.logger.org.springframework=INFO, A3
# Do not change these lines
log4j.additivity.org.apache.cocoon=false
log4j.additivity.cocoon=false
log4j.additivity.org.springframework=false
# The name of the file appender
log4j.appender.A3=org.dspace.app.util.DailyFileAppender
# The filename of the log file created. A date stamp is appended to this
log4j.appender.A3.File=${log.dir}/cocoon.log
# Set this to yyyy-MM-DD for daily log files, or yyyy-MM for monthly files
log4j.appender.A3.DatePattern=yyyy-MM-dd
# The number of log files to keep, or 0 to keep them all
log4j.appender.A3.MaxLogs=14
# A2 uses PatternLayout.
log4j.appender.A3.layout=org.apache.log4j.PatternLayout
log4j.appender.A3.layout.ConversionPattern=%d %-5p %c %x - %m%n


###########################################################################
# A4 is the name of the appender for Solr
###########################################################################
log4j.logger.org.apache.solr=ERROR, A4
log4j.additivity.org.apache.solr=false
log4j.appender.A4=org.dspace.app.util.DailyFileAppender
log4j.appender.A4.File=${log.dir}/solr.log
log4j.appender.A4.DatePattern=yyyy-MM-dd
log4j.appender.A4.MaxLogs=14
log4j.appender.A4.layout=org.apache.log4j.PatternLayout
log4j.appender.A4.layout.ConversionPattern=%d %-5p %c %x - %m%n


###########################################################################
# Other settings
###########################################################################

# Block passwords from being exposed in Axis logs.
# (DEBUG exposes passwords in Basic Auth)
log4j.logger.org.apache.axis.handlers.http.HTTPAuthHandler=INFO

# Block services logging except on exceptions
log4j.logger.org.dspace.kernel=ERROR
log4j.logger.org.dspace.services=ERROR
log4j.logger.org.dspace.servicemanager=ERROR
log4j.logger.org.dspace.providers=ERROR
log4j.logger.org.dspace.utils=ERROR

Then rebuild DSpace.


In addition check the log settings for Tomcat in /etc/default/tomcat7.

SOLR

Log level

Please note: The procedure below is not stable, so I installed PSI Probe using the Tomcat manager web interface and now I set the SOLR log level via the web gui after each re-build. See: https://code.google.com/p/psi-probe

See: http://lucidworks.lucidimagination.com/display/solr/Configuring+Logging and https://jira.duraspace.org/browse/DS-1460 and https://jira.duraspace.org/browse/DS-1946

Create the following file: 

nano $HOME/source/dspace/modules/solr/src/main/webapp/WEB-INF/classes/logging.properties

Add the following to the file: 

 org.apache.commons.digester.Digester.level = WARNING
 org.apache.solr.level = SEVERE
NANO Editor Help
CTL+O = Save the file and then press Enter
CTL+X = Exit "nano"
CTL+K = Delete line
CTL+U = Undelete line
CTL+W = Search for %%string%%
CTL+\ = Search for %%string%% and replace with $$string$$
CTL+C = Show line numbers

More info = http://en.wikipedia.org/wiki/Nano_(text_editor)

Save the file and rebuild.

Also see: http://skybert.wordpress.com/2009/07/22/how-to-get-solr-to-log-to-a-log-file

See news from @atmire below.

Limiting log output for SOLR

Adding SOLR logging propertiesWe extensively covered the growth of log files and the different log levels in last month's article. In addition to the dspace logfiles contained in /dspace/log, some DSpace components write directly into the logfiles of your application container, Tomcat. One culprit in particular is SOLR. SOLR is the key component responsible for statistics, Discovery and most recently, OAI-PMH. The default logging level for SOLR is INFO, meaning that it will write to the Tomcat catalina.out logfile for each request it receives. To override this behavior add a file named logging.properties to the solr module found in $dspace.src$/dspace/modules/solr/src/main/resources.

Add the following instruction in this empty file: 

org.apache.solr.level = WARNING

Log file permissions

I setup a root cron crontab as follows. 

sudo -i

crontab -e

Add the following line. 

@midnight   chmod 0777 -R $HOME/log

Save the crontab edit.

NANO Editor Help
CTL+O = Save the file and then press Enter
CTL+X = Exit "nano"
CTL+K = Delete line
CTL+U = Undelete line
CTL+W = Search for %%string%%
CTL+\ = Search for %%string%% and replace with $$string$$
CTL+C = Show line numbers

More info = http://en.wikipedia.org/wiki/Nano_(text_editor)

See: http://wiki.lib.sun.ac.za/index.php/SUNScholar/Install_DSpace/S06#Step_6.3:_Setup_Tomcat_server_permissions for reference about file permissions.

Retrieved from "http://wiki.lib.sun.ac.za/index.php?title=SUNScholar/Optimisations/Logs&oldid=44325"