SUNScholar/Optimisations/Logs

Back to Optimisations

Default Log Levels
With the release of DSpace 4.2, it is now possible to define default log levels.

Modif the the $HOME/source/build.properties file. See example below. loglevel.other = INFO loglevel.dspace = INFO
 * 1) LOGLEVEL SETTINGS #
 * 1) LOGLEVEL SETTINGS #
 * 1) loglevel.other: Log level for other third-party tools/APIs used by DSpace
 * 2) Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
 * 1) loglevel.dspace: Log level for all DSpace-specific code (org.dspace.*)
 * 2) Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL

Tomcat Java web applications
Changed all instances of "INFO" to "ERROR" in the following config file to reduce disk I/O and CPU load. nano $HOME/ source /dspace/config/log4j.properties

See example below:
 * 1) log4j.properties
 * 2) Version: $Revision$
 * 3) Date: $Date$
 * 1) Version: $Revision$
 * 2) Date: $Date$
 * 1) Date: $Date$
 * 1) Date: $Date$


 * 1) This is the log4j configuration file for DSpace. Log4j can be configured to
 * 2) write files that rotate every day or every month.
 * 3) Text surrounded by ${} symbols is replaced with the corresponding
 * 4) property from dspace.cfg by Ant when DSpace is deployed.  For example:
 * 5) dspace.url
 * 6) would be replaced with the dspace.url property in dspace.cfg.
 * 7) Additional properties expanded by Maven during the DSpace assembly
 * 8) process:
 * 9) loglevel.dspace (currently set to: ${loglevel.dspace})
 * 10)   Log level for all DSpace-specific code (org.dspace.*)
 * 11)   Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
 * 12) loglevel.other (currently set to: ${loglevel.other})
 * 13)   Log level for other third-party tools/APIs used by DSpace
 * 14)   Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
 * 1)   Log level for all DSpace-specific code (org.dspace.*)
 * 2)   Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
 * 3) loglevel.other (currently set to: ${loglevel.other})
 * 4)   Log level for other third-party tools/APIs used by DSpace
 * 5)   Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
 * 1)   Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL

log4j.rootCategory=${loglevel.other}, A1 log4j.logger.org.dspace=${loglevel.dspace}, A1 log4j.additivity.org.dspace=false log4j.appender.A1=org.dspace.app.util.DailyFileAppender log4j.appender.A1.File=${log.dir}/dspace.log log4j.appender.A1.DatePattern=yyyy-MM-dd log4j.appender.A1.MaxLogs=0 log4j.appender.A1.layout=org.apache.log4j.PatternLayout log4j.appender.A1.layout.ConversionPattern=%d %-5p %c @ %m%n
 * 1) A1 is the name of the appender for most DSpace activity.
 * 2) The root category is the default setting for all non-DSpace code.
 * 3) Change this from INFO to DEBUG to see extra logging created by non-DSpace
 * 4) code.
 * 1) Change this from INFO to DEBUG to see extra logging created by non-DSpace
 * 2) code.
 * 1) This line sets the logging level for DSpace code. Set this to DEBUG to see
 * 2) extra detailed logging for DSpace code.
 * 1) Do not change this line
 * 1) The name of the file appender
 * 1) The filename of the log file created. A date stamp is appended to this
 * 1) Set this to yyyy-MM-DD for daily log files, or yyyy-MM for monthly files
 * 1) The number of log files to keep, or 0 to keep them all
 * 1) A1 uses PatternLayout.

log4j.logger.org.dspace.checker=INFO, A2 log4j.additivity.org.dspace.checker=false log4j.appender.A2=org.dspace.app.util.DailyFileAppender log4j.appender.A2.File=${log.dir}/checker.log log4j.appender.A2.DatePattern=yyyy-MM-dd log4j.appender.A2.MaxLogs=0 log4j.appender.A2.layout=org.apache.log4j.PatternLayout log4j.appender.A2.layout.ConversionPattern=%m%n
 * 1) A2 is the name of the appender for the Checker
 * 2) This line sets the logging level for the checksum checker log file.
 * 3) Set this to DEBUG to see extra detailed logging.
 * 1) This line sets the logging level for the checksum checker log file.
 * 2) Set this to DEBUG to see extra detailed logging.
 * 1) Do not change this line
 * 1) The name of the file appender
 * 1) The filename of the log file created. A date stamp is appended to this
 * 1) Set this to yyyy-MM-DD for daily log files, or yyyy-MM for monthly files
 * 1) The number of log files to keep, or 0 to keep them all
 * 1) A2 uses PatternLayout.

log4j.logger.org.apache.cocoon=INFO, A3 log4j.logger.cocoon=INFO, A3 log4j.logger.org.springframework=INFO, A3 log4j.additivity.org.apache.cocoon=false log4j.additivity.cocoon=false log4j.additivity.org.springframework=false log4j.appender.A3=org.dspace.app.util.DailyFileAppender log4j.appender.A3.File=${log.dir}/cocoon.log log4j.appender.A3.DatePattern=yyyy-MM-dd log4j.appender.A3.MaxLogs=14 log4j.appender.A3.layout=org.apache.log4j.PatternLayout log4j.appender.A3.layout.ConversionPattern=%d %-5p %c %x - %m%n
 * 1) A3 is the name of the appender for Cocoon (XMLUI only)
 * 2) These lines sets the logging level for the cocoon log file.
 * 3) Set these to DEBUG to see extra detailed logging.
 * 1) These lines sets the logging level for the cocoon log file.
 * 2) Set these to DEBUG to see extra detailed logging.
 * 1) Do not change these lines
 * 1) The name of the file appender
 * 1) The filename of the log file created. A date stamp is appended to this
 * 1) Set this to yyyy-MM-DD for daily log files, or yyyy-MM for monthly files
 * 1) The number of log files to keep, or 0 to keep them all
 * 1) A2 uses PatternLayout.

log4j.logger.org.apache.solr=ERROR, A4 log4j.additivity.org.apache.solr=false log4j.appender.A4=org.dspace.app.util.DailyFileAppender log4j.appender.A4.File=${log.dir}/solr.log log4j.appender.A4.DatePattern=yyyy-MM-dd log4j.appender.A4.MaxLogs=14 log4j.appender.A4.layout=org.apache.log4j.PatternLayout log4j.appender.A4.layout.ConversionPattern=%d %-5p %c %x - %m%n
 * 1) A4 is the name of the appender for Solr
 * 1) A4 is the name of the appender for Solr


 * 1) Other settings
 * 1) Other settings

log4j.logger.org.apache.axis.handlers.http.HTTPAuthHandler=INFO
 * 1) Block passwords from being exposed in Axis logs.
 * 2) (DEBUG exposes passwords in Basic Auth)

log4j.logger.org.dspace.kernel=ERROR log4j.logger.org.dspace.services=ERROR log4j.logger.org.dspace.servicemanager=ERROR log4j.logger.org.dspace.providers=ERROR log4j.logger.org.dspace.utils=ERROR
 * 1) Block services logging except on exceptions

Then rebuild DSpace.

In addition check the log settings for Tomcat in /etc/default/tomcat7.

Log level
Please note: The procedure below is not stable, so I installed PSI Probe using the Tomcat manager web interface and now I set the SOLR log level via the web gui after each re-build. See: https://code.google.com/p/psi-probe

See: http://lucidworks.lucidimagination.com/display/solr/Configuring+Logging and https://jira.duraspace.org/browse/DS-1460 and https://jira.duraspace.org/browse/DS-1946

Create the following file: nano $HOME/ source /dspace/modules/solr/src/main/webapp/WEB-INF/classes/logging.properties Add the following to the file: org.apache.commons.digester.Digester.level = WARNING org.apache.solr.level = SEVERE

Save the file and rebuild.

Also see: http://skybert.wordpress.com/2009/07/22/how-to-get-solr-to-log-to-a-log-file


 * See news from @atmire below.

Limiting log output for SOLR

Adding SOLR logging propertiesWe extensively covered the growth of log files and the different log levels in last month's article. In addition to the dspace logfiles contained in /dspace/log, some DSpace components write directly into the logfiles of your application container, Tomcat. One culprit in particular is SOLR. SOLR is the key component responsible for statistics, Discovery and most recently, OAI-PMH. The default logging level for SOLR is INFO, meaning that it will write to the Tomcat catalina.out logfile for each request it receives. To override this behavior add a file named logging.properties to the solr module found in $dspace.src$/dspace/modules/solr/src/main/resources.

Add the following instruction in this empty file: org.apache.solr.level = WARNING

Log file permissions
I setup a root cron crontab as follows.

sudo -i

crontab -e Add the following line. @midnight  chmod 0777 -R $HOME/log

Save the crontab edit.

See: http://wiki.lib.sun.ac.za/index.php/SUNScholar/Install_DSpace/S06#Step_6.3:_Setup_Tomcat_server_permissions for reference about file permissions.