Opened 8 years ago

Closed 8 years ago

#908 closed enhancement (fixed)

Prevent crawlers from indexing the vlo search pages

Reported by: Willem Elbers Owned by: Sander Maijers
Priority: major Milestone:
Component: System administration Version:
Keywords: lvps92-51-161-129.dedicated.hosteurope.de Cc: Willem Elbers, Dieter Van Uytvanck

Description

Crawlers are currently indexing the vlo search pages. These are not really usefull and cause a high load. Adjust the robots.txt to disallow indexing of the search pages as follows:

Disallow: /search
Disallow: /search/

Change History (4)

comment:1 Changed 8 years ago by DefaultCC Plugin

Cc: Willem Elbers Dieter Van Uytvanck added

comment:2 Changed 8 years ago by Willem Elbers

Adjusted robots.txt. Twan is testing.

Last edited 8 years ago by Willem Elbers (previous) (diff)

comment:3 Changed 8 years ago by Willem Elbers

Adjusted with user agent:

Sitemap: https://vlo.clarin.eu/sitemap/vlo-index.xml
User-agent: *
Disallow: /search
Disallow: /search/

comment:4 Changed 8 years ago by Willem Elbers

Resolution: → fixed
Status: new → closed
Note: See TracTickets for help on using tickets.