SUNScholar/Repository Website Metrics

Back to Web Analytics

Robots

See: https://github.com/DSpace/DSpace/pull/498 and https://github.com/DSpace/DSpace/blob/master/dspace-xmlui/src/main/webapp/static/robots.txt

See below for an example http://scholar.sun.ac.za/robots.txt file.

User-agent: *
# Disable access to Discovery search and filters
Disallow: /discover
Disallow: /search-filter
 
# This should be the FULL URL to your HTML Sitemap. 
# Make sure to replace "[dspace.url]" with the value of your 'dspace.url' setting in your dspace.cfg file.
Sitemap: http://scholar.sun.ac.za/htmlmap
 
# If you have configured DSpace (Solr-based) Statistics to be publicly accessible,
# then you likely do not want this content to be indexed
# Disallow: /displaystats
 
# Uncomment the following line ONLY if sitemaps.org or HTML sitemaps are used
# and you have verified that your site is being indexed correctly.
# Disallow: /browse
 
# You also may wish to disallow access to the following paths, in order
# to stop web spiders from accessing user-based content:
# Disallow: /advanced-search
# Disallow: /contact
# Disallow: /feedback
# Disallow: /forgot
# Disallow: /login
# Disallow: /register
# Disallow: /search

HTML Head Tag Metadata

If you have heavily customized your metadata fields away from Dublin Core, you can modify the crosswalk that generates these elements by modifying:

[dspace]/config/crosswalks/xhtml-head-item.properties