Ticket	Priority	Summary	Owner	Created	Modified
#907	minor	support multi-file upload again	André Moreira	8 years ago	6 years ago
#938	major	Replace clarin_datatable custom code by standard feeds module	Dieter Van Uytvanck	8 years ago	6 years ago
#958	minor	Files missing	Dieter Van Uytvanck	8 years ago	6 years ago
#965	minor	Clean up Drupal redirects	Dieter Van Uytvanck	8 years ago	6 years ago
#967	major	User involvement group pages on new website	Dieter Van Uytvanck	8 years ago	6 years ago

1 2 3 4 5 →

1. Overview

Currently the www.clarin.eu website is deployed in a high available (HA) setup. Two instances of both the apache webserver with drupal installation and mariadb database are running behind a reverse proxy, based on apache with mod_proxy, as decsribed in the figure below.

Requests are normally handled by the primary (live) instance.
If the primary instance is unavailable, the hot standby will take over. This is a read-only version of the website.
- Failure is active polled by using mod_proxy_hceck.
A full backup is created every day and imported into the hot stanby instance.

1.1. Docker images

Each component is run as a docker container based on a docker image. The following images are used:

Component	Docker Image	Repository
website (primary and secondary)	registry.gitlab.com/clarin-eric/alpine-httpd-drupal	gitlab
database (primary and secondary)	mariadb	docker hub
proxy	registry.gitlab.com/clarin-eric/alpine-httpd-proxy	gitlab

1.2. Servers

The container config for each server is managed via a docker compose setup:

Server	Stage	Repository
transip-vps3	development	gitlab
transip-vps4	production	gitlab

2. Conversion from legacy state to basic Drupal

This should be organized in the following stages.

2.1. Enable Bootstrap module

## Enable Bootstrap
drush.phar pm-enable 'jquery_update' 'bootstrap'

## The frontpage view was disabled in the legacy state, reenable it.
drush.phar views-enable 'frontpage'

2.2. Disable legacy/redundant extensions

## Must be updated to dev version first, since otherwise it won't uninstall due to bugs.
drush.phar pm-enable 'empty_page-7.x-1.x-dev'
## IMPORTANT: Quit endless loop here!

## Alpha.
drush.phar pm-disable 'uuid_features'

## ‘Minimally maintained’.
drush.phar pm-disable 'smart_trim'

## 7.x-3.0-rc5.
## drush.phar pm-disable 'admin_menu' 'admin_menu_toolbar'

## Apparently not in use.
drush.phar pm-disable 'entityreference_prepopulate' 'fences' 'colorbox'

## Seems unused, only used for twitter_pull_bean?
drush.phar pm-disable 'bean_admin_ui' 'bean' 'bean_entitycache' 'bean_uuid'

## Functionality seems unrequired.
drush.phar pm-disable 'imagecache_actions' 'imagecache_canvasactions' 'empty_page' 'timeago'

## Superfluous functionality and/or security risk.
drush.phar pm-disable 'backup_migrate' 'environment_indicator'  'admin_devel'

## The following modules could be directly traced to Jurriaan. They have cyclic dependencies and define a number of features and views.
drush.phar pm-disable 'bean_twitter_pull' 'twitter_pull' 'openscience_blog' 'openscience_events' 'openscience_groups' \
                      'context' 'context_ui' 'context_menu_block' 'context_og' 'clarin_eric_config' 'clarineric' 'clarin_eric_config' \
                      'openscience_news' 'openscience_showcases' 'lrt_showcases' 'openscience_ui' 'openscience' 'openscience_governance'

## Update Drupal core
drush.phar upc --no-backup                      

## Make sure that the remaining extensions are up-to-date.
drush.phar pm-update --no-backup

Remove residual Drupal views by visiting:

2.3. Enhance functionality before testing

Some Website Trac tickets (at the top of this page) are feature requests/enhancements. Ideally these can be added without much risk and then included in the acceptance tests.

https://trac.clarin.eu/ticket/938

drush.phar cache-clear 'all'

## Enable CKEditor for rich text edition (WYSIWYG).
drush.phar pm-disable 'wysiwyg'
drush.phar pm-enable 'ckeditor'

## Enable smtp module so that Centre registration form, which depends on e-mail functionality, works.

drush.phar pm-enable 'smtp'

## Enable flexslide for the caroussel on the front page with featured items.

drush.phar pm-enable flexslider
drush.phar pm-enable flexslider_views 

### Install the required flexslider libraries
### WARNING: undeclared dependency.

mkdir /var/www/localhost/htdocs/sites/all/libraries/flexslider
cd -- $_

wget https://github.com/woothemes/FlexSlider/raw/master/jquery.flexslider.js
wget https://github.com/woothemes/FlexSlider/raw/master/flexslider.css
wget https://github.com/woothemes/FlexSlider/raw/master/jquery.flexslider-min.js

mkdir /var/www/localhost/htdocs/sites/all/libraries/flexslider/fonts
cd -- $_
wget https://github.com/woothemes/FlexSlider/raw/version/2.2/fonts/flexslider-icon.woff

-> Disable Drupal queue

2.4. Content restructuring tasks

2.4.1. Menus and structure

Restructure content so that excess hierarchy is flattened.

Each page title should exactly match the related menu item's title, to reduce confusion. This becomes more important as the navigation on the website changes with the new CLARIN Horizon theme. For example, here this rule was broken. In case the page title is considered too long for the menu, shorten it to a single, brief title.

Two children that are not truly siblings.
Has a single child node: poor, deep structure.
Not really clear what should be listed here as ‘Services’ and what entries are merely one software package. The ranking/importance of items is not clear (and as it stands based on current items and order, is bad). Sander suggests that this page be more general and that it refers to /applications.

2.4.2. Language use

The website has quite some pages and text with insufficient editorial quality. The basic rule should be that pages aren't published before someone who is good at that has reviewed the editorial quality.

Example: Every year, the CLARIN - the Common Language Resources and Technology Infrastructure - holds an annual meeting. 2013 the meeting was held in Prague, information on the event can be found here.

Keep sentences short. ‘Information on the event can be found here’ -> ‘annual meeting’ (= link).
Do not redeclare abbreviations inconsistently. Ideally, do not declare them at all outside the Glossary and/or About pages.
Check whether you are writing redundantly. E.g., ‘annual meetings’ are always ‘every year’.
‘. 2013 the’: missing words. Edit your text carefully (certainly if the text is published).

2.4.3. Misc

Image HTML tags used to reference attachment files. Use something else, now it renders as broken images.
Decide which language variety and terms to use, and use them consistently.
1. Center -> centre
2. movies, films -> clips
3. federated login / federated identity / federation / ... ?
Decide on table style(s), describe the chosen style(s) in the CLARIN Style guide, and apply style(s) to all tables across the website by means of the CLARIN Horizon theme, not by separately styling each table. Currently, various style-influencing and noise attributes are in the HTML code for many pages.
1. Tables should have table headings. If not, is a table appropriate?
2. Unnecessary final column, should be merged with first column.
Too drafty. Diagrams should be properly rendered, realize that it is not an (internal) Trac page.
Simplify/clean up hard-to-read, excess, noisy HTML code. E.g. in:
Are page summaries (top text field in node editing view) really needed? They are being generated automatically if left empty. If manually specified, what is the policy for keeping them in sync with the content?
Find in page contents (HTML) improper absolute links (e.g. http://www.clarin.eu/content/...) and replace them with proper relative links (/node/...).

2.5. Acceptance tests

Adding a basic page.
Uploading multiple files in one go.
Writing a blog post.
Filling out the Centre Registry form.
TODO: ...

2.5.1. Criteria

Does the rich text editing work as expected?

2.6. Theming

We apply a Drupal Bootstrap subtheme to dev-www designed after the new CLARIN house style, named ‘CLARIN Horizon’.

curl -LSs 'https://github.com/drupalprojects/bootstrap/archive/7.x-3.5.tar.gz' | tar -x -z -p -v -f -
## Use the Less-based starter kit.
cp -apr -- 'bootstrap-7.x-3.5/starterkits/less' 'CLARIN_Horizon/'
cd -- 'CLARIN_Horizon/'
mv -- 'less.subtheme' 'CLARIN_Horizon.info'
## Edit subtheme properties.
nano 'CLARIN_Horizon.info'
## Set graphics.
mv -f -T -- ... favicon.ico
mv -f -T -- ... logo.png

git clone 'git@github.com:twbs/bootstrap.git'
cd -- 'bootstrap/'
git fetch --tags
git checkout 'v3.3.6'
cd -- '-'

## Install Less and NodeJS, e.g. with `sudo pacman -Syu nodejs-less-2.7.1-1` on Arch Linux.
sudo npm install -g 'less-plugin-clean-css'

## Transpile.
lessc 'less/style.less' --clean-css='--s0' > 'css/style.css'

# On dev-www host
sudo docker cp 'CLARIN_Horizon.tar.gz' 'clarinericwebsitedev3_httpd_1:/var/www/localhost/htdocs/sites/all/theme/'

# sh into clarinericwebsitedev3_httpd_1 as root, then ...
tar -f 'CLARIN_Horizon.tar.gz' -x -v -z
chown -Rc '0:www-data' 'CLARIN_Horizon/'

-> Enable and set as default theme ‘CLARIN_Horizon’
CKeditor: stop filtering content.
1. safe_html -> table[width|cellspacing|class].
2. ‘Rules for Class Names’ -> table.
3. ‘Rules for URLs used within inline styles’ -> /*. (???)
-> ‘bordered table’

2.7. Acceptance tests

2.8. Conversion of www.clarin.eu

Based on the final, validated steps, we set www.clarin.eu as read-only for a regular maintenance, and apply the steps to the www.clarin.eu instance.

2.9. Thorough cleanup of `dev-www`, then `www`

## Jurriaan's modules all have cyclic dependencies, so cannot be uninstalled.
## IMPORTANT: The following command
drush.phar pm-uninstall $(drush.phar pm-list --status='disabled' --format=list --no-core)
## May cause Drush defects reported as:
## Error: require_once(): Failed opening required '/var/www/localhost/htdocs/sites/all/modules/contrib/menu_block/plugins/content_types/menu_tree/menu_tree.inc'
## (include_path='.:/usr/share/pear') in /var/www/localhost/htdocs/sites/all/modules/contrib/ctools/includes/plugins.inc, line 477

## Uninstalling extensions using Drush does not result in actual removal.
## To forcefully remove ‘disabled, uninstalled’ modules/themes.
drush.phar pm-list --status='disabled,not installed' --no-core --pipe | \
    xargs -n 1 -I '{}' drush.phar drupal-directory --local-only --component='path' '{}' | \
        xargs -n 1 -I '{}' rm -frv -- '{}'

## Clean up informal dependencies, e.g. in 'sites/all/libraries/'
# rm -r 'sites/all/libraries/timeago/'

drush.phar cache-clear 'all'

## Remove security sensitive and redundant files from Drupal installation.
rm -- 'install.php' 'CHANGELOG.txt' 'INSTALL.txt' 'INSTALL.mysql.txt' \
      'INSTALL.pgsql.txt' 'LICENSE.txt' 'MAINTAINERS.txt' 'UPGRADE.txt' \
      'COPYRIGHT.txt' 'README.txt' 'INSTALL.sqlite.txt'

## Remove residual Drupal Blocks.
## See: https://dev-www-3.clarin.eu/admin/structure/block-types

## TODO: Remove residual Drupal Features. Cannot be done with Drush or the Drupal web interface alone.
...

## Remove LRT Showcase content type: if adding a content node, this content type is picked.

2.10. Acceptance test

(details to be added)

working frontpage
event list
news + blog list
contact page
views/static pages for: committees (governance), showcases, featured resources
intranet pages (drupal groups) reachable and usable
redirect /user/register to CLARIN IdP

2.11. Finishing touch

2.11.1. Enhancement: do not use CDN.

IMPORTANT: breaks the website without alternative source other than CDN. Make independent from `jsdeliver`: -> Advanced

2.11.2. Clean up content types

Not necessary: Merge nodes of LRT Showcase type with Showcase type.

Although these node have a similar (technical/internal) name (due to a misconception earlier on), they are fundamentally different: the showcases are more elaborate descriptions of how a CLARIN tool or resource can be used to answer a specific question. The featured resources (LRT Showcase) are simple examples of resources/tools. Many of them have a searchbox included. They should not be merged.

Last modified 5 years ago Last modified on 07/26/19 15:58:46

Attachments (1)

clarin website HA setup.png (26.6 KB) - added by Willem Elbers 7 years ago.

Download all attachments as: .zip

Download in other formats:

Plain Text

Context Navigation

This page is outdated and refers to the previous version of our Drupal setup!!

While this remains to be updated you can find up-to-date information about our setup on its ​GitLab project page

=== Start of old content ==

1. Important links

1.1. Working documents

2. Development workflow

3. Tickets

Results (1 - 5 of 23)