Generate sitemap

[Twan Goosen]

Create a sitemap that provides links to all record pages for Google and other search engines. There are some options, either have a statically generated site map on (or after) import, or generating (parts of) the sitemap on the fly at request. In both cases, one option would be to create a sitemap per collection or per provider and link to these from a sitemap index file.

Some useful information can be found at:

• ​http://www.sitemaps.org/protocol.html#index
• ​http://googlewebmastercentral.blogspot.nl/2006/10/multiple-sitemaps-in-same-directory.html
• ​https://www.quora.com/If-I-have-a-website-with-millions-of-unique-pages-should-I-submit-a-partial-sitemap-to-Google

[go.sugimoto@oeaw.ac.at]

As of 6th of January, the current state of Google indexing:

• Only c 37000 has been indexed, while we now have c 870000 in production. The indexing has started since September 6th 2015, if you happened to remember what you have configured around that date. There is no doubt that we should create sitemaps and submit them as soon as possible to increase discoverability and attract more traffic.

• Test sitemaps are created by ACDH. We have one URL per record. A sitemap index contains links to dozens of sitemaps, which describe up to 50000 URLs per sitemap (Google's requirement).

• We have a message from Google console about the extremely high number of URLs. It seems Googlebot had problem with it when crawling in December. It seems the problem is that Googlebot encountered a lot of jsessonid for the same all-record view. (Example: ​https://vlo.clarin.eu/search;jsessionid=DCFFD161E06B619B1F5457FAC6F163B5?36418-1.IBehaviorListener.0-options-collapse) The simplest prevention would be to submit our sitemap. It is also possible to avoid those URLs to be indexed, by configuring robots.txt, as a second precaution.

• If time permits, we might want to also discuss the change of record URI/L in the long run.

[Twan Goosen]

Davor wrote a sitemap generation tool for the VLO, currently available at ​https://github.com/acdh-oeaw/vlo-sitemap-gen (to be integrated with the VLO code base)

[Twan Goosen]

Milestone renamed

[Twan Goosen]

Milestone renamed

[Twan Goosen]

​Available and ​in use as of VLO 3.4.0

[Kai Zimmer]

As of 3rd of June 2016, searching for 'site:vlo.clarin.eu', Google states it indexed 44.200 documents, but when i go to the last page (adding all duplicates), there are only 54 (!) documents available. It seems Google doesn't find the content attractive...

[Kai Zimmer]

some suggestions:

add to robots.txt
Disallow: /wicket
Disallow: /*jsessionid

for sitemaps

• focus on one document type (index page OR cmdi) to lower the number of total URLs
• add a changefreq-attribute (probably monthly)
• resubmit the new sitemap via Google Search console

[Twan Goosen]

Thanks Kai, for your analysis and suggestions. Indeed, there seem to be issues with the way Google sees or understands the VLO and our previous attempts to tackle these have not been very successful. The latest release of the VLO at least produces proper 404s for records that do not exist anymore, which should lead to a cleaner index but not necessarily more results.

I have some hopes that the fix described in ​GitHub issue #4 will make things more attractive for Google.

I'm not sure if added disallowing '/wicket' and ' /*jsessionid' will actually have the desired effect. I will remove the parameter logic that I configured via webmaster tools as they probably don't do much good either.

The ​sitemap gets re-read and processed by Google frequently, as I have been able to confirm based on the data in the webmaster tools. Google does appear to be quite selective in the number of records it picks up, however. Possibly it could help to split the sitemap up into smaller pages although there is no indication or documentation that suggest that this will actually help.

I'm closing this ticket as there are no concrete actions remaining other than those mentioned here and in the GitHub ticket.