SUNScholar/Repository Website Metrics/Robots

BACK TO REPOSITORY WEBSITE METRICS

Introduction

A proper robots.txt file is essential in ensuring that web robots (Google, Bing etc..) have the right permissions to index your repository web site.

For an example visit: http://scholar.sun.ac.za/robots.txt.

Procedure

Step 1

To enable a "robots.txt" file for the XMLUI, type the following:

mkdir $HOME/source/dspace/modules/xmlui/src/main/webapp/static

nano $HOME/source/dspace/modules/xmlui/src/main/webapp/static/robots.txt

See example below.

Step 2

Then rebuild DSpace to activate.

References

• https://github.com/DSpace/DSpace/pull/498
• https://github.com/DSpace/DSpace/blob/master/dspace-xmlui/src/main/webapp/static/robots.txt