Changeset 5039

Timestamp:

04/24/14 09:59:48 (10 years ago)

Author:

yana.panchenko@uni-tuebingen.de

Message:

Language selection changed, other UI minor changes, code refactoring, scanned data location change:

sort languages by the name for the UI (before was by code)
display only those languages for selection, for which there is a corpus that has this language as the only one
create Language class to keep its data: language name and its codes in three ISO standards
separate search-result rendering into separate class
store start-end record offset in search-result
separate corpus-list rendering into a base class extended by live or cached rendering implementations
unite all scan requests into one method inside ScanCrawler?
add method in WebAppListener? for running in cache read only mode with scan crawl not scheduled (eg for testing)
separate reusable Aggregator ZK component creation into one class (ZKComp)
create SearchResultContent? class for extracting textual content of search result in defferent representations
typo in about page corrected, info in help page modified
add tooltips to prev/next buttons in search results page
move location to store scan data on server to /data/...

Location:

SRUAggregator/trunk/src/main

Files:

• java/eu/clarin/sru/fcs/aggregator/app/Aggregator.java (modified) (10 diffs)
• java/eu/clarin/sru/fcs/aggregator/app/SearchOptions.java (modified) (6 diffs)
• java/eu/clarin/sru/fcs/aggregator/app/SearchResults.java (modified) (27 diffs)
• java/eu/clarin/sru/fcs/aggregator/app/WebAppListener.java (modified) (6 diffs)
• java/eu/clarin/sru/fcs/aggregator/cache/ScanCacheFiled.java (modified) (1 diff)
• java/eu/clarin/sru/fcs/aggregator/cache/ScanCrawler.java (modified) (4 diffs)
• java/eu/clarin/sru/fcs/aggregator/sopt/CorpusModelLive.java (modified) (5 diffs)
• java/eu/clarin/sru/fcs/aggregator/sopt/CorpusRenderer.java (copied) (copied from SRUAggregator/trunk/src/main/java/eu/clarin/sru/fcs/aggregator/sopt/CorpusRendererCached.java) (11 diffs)
• java/eu/clarin/sru/fcs/aggregator/sopt/CorpusRendererCached.java (modified) (4 diffs)
• java/eu/clarin/sru/fcs/aggregator/sopt/CorpusRendererI.java (deleted)
• java/eu/clarin/sru/fcs/aggregator/sopt/CorpusRendererLive.java (modified) (4 diffs)
• java/eu/clarin/sru/fcs/aggregator/sopt/Language.java (added)
• java/eu/clarin/sru/fcs/aggregator/sopt/Languages.java (modified) (3 diffs)
• java/eu/clarin/sru/fcs/aggregator/sresult/SearchResult.java (modified) (2 diffs)
• java/eu/clarin/sru/fcs/aggregator/sresult/SearchResultContent.java (added)
• java/eu/clarin/sru/fcs/aggregator/sresult/SearchResultGroupRenderer.java (added)
• java/eu/clarin/sru/fcs/aggregator/sresult/SearchResultRecordRenderer.java (modified) (3 diffs)
• java/eu/clarin/sru/fcs/aggregator/util/ZKComp.java (added)
• webapp/about-fcs.html (modified) (1 diff)
• webapp/help-fcs.html (modified) (3 diffs)
• webapp/index.zul (modified) (2 diffs)

SRUAggregator/trunk/src/main/java/eu/clarin/sru/fcs/aggregator/app/Aggregator.java

@@ -71 +71 @@
     @Wire
     A nextButton;
+    @Wire
+    Label tooltipPrevText;
+    @Wire
+    Label tooltipNextText;
     
     @Wire
@@ -111 +115 @@
             int maxRecords = searchOptionsComposer.getMaxRecords();
             String searchLang = searchOptionsComposer.getSearchLang();
+            //searchOffset = new int[]{1, 0};
             searchOffset = new int[]{1, 0};
-            searchResultsComposer.executeSearch(selectedCorpora, maxRecords, searchString.getText(), searchOffset, searchLang);
+            searchOffset[0] = searchOffset[0] + searchOffset[1];
+            searchOffset[1] = maxRecords;
+            searchResultsComposer.executeSearch(selectedCorpora, searchOffset[0], maxRecords, searchString.getText(), searchLang);
             if (searchLang.equals("anylang")) {
                 this.weblichtTcf.setVisible(false);
@@ -182 +189 @@
             Executions.getCurrent().sendRedirect(WEBLICHT_URL
                 + url, "_blank");
-        } else {
-            Messagebox.show("Sorry, drop-off/WebLicht error!");
         }
     }
@@ -193 +198 @@
             Executions.getCurrent().sendRedirect(WEBLICHT_URL
                 + url, "_blank");
-        } else {
-            Messagebox.show("Sorry, drop-off/WebLicht error!");
         }
     }
@@ -253 +256 @@
     @Listen("onClick = #srLabel")
     public void onClickSearchResult(Event ev) {
+        setupPrevNextSearchTooltips();
         this.pagesVisibility.openSearchResult();
         if (this.searchResultsComposer.hasSearchInProgress()) {
@@ -261 +265 @@
             this.controlsVisibility.enableControls2();
         }
+        
     }
 
@@ -280 +285 @@
             int maxRecords = searchOptionsComposer.getMaxRecords();
             String searchLang = searchOptionsComposer.getSearchLang();
-            searchOffset[0] = searchOffset[0] - 2 * searchOffset[1];
+            //searchOffset[0] = searchOffset[0] - searchOffset[1];
+            searchOffset[0] = searchOffset[0] - maxRecords;
             if (searchOffset[0] < 1) {
                 searchOffset[0] = 1;
-                searchOffset[1] = 0;
-            }
-            searchResultsComposer.executeSearch(selectedCorpora, maxRecords, searchString.getText(), searchOffset, searchLang);
+            }
+            searchOffset[1] = maxRecords;
+            searchResultsComposer.executeSearch(selectedCorpora, searchOffset[0], maxRecords, searchString.getText(), searchLang);
             if (searchLang.equals("anylang")) {
                 this.weblichtTcf.setVisible(false);
@@ -294 +300 @@
         }
     }
+
 
     @Listen("onClick = #nextButton")
@@ -312 +319 @@
             int maxRecords = searchOptionsComposer.getMaxRecords();
             String searchLang = searchOptionsComposer.getSearchLang();
-            searchResultsComposer.executeSearch(selectedCorpora, maxRecords, searchString.getText(), searchOffset, searchLang);
+            searchOffset[0] = searchOffset[0] + searchOffset[1];
+            searchOffset[1] = maxRecords;
+            searchResultsComposer.executeSearch(selectedCorpora, searchOffset[0], maxRecords, searchString.getText(), searchLang);
             if (searchLang.equals("anylang")) {
                 this.weblichtTcf.setVisible(false);
@@ -341 +350 @@
         LOGGER.log(Level.INFO, "Received parameter: operation[{0}], ", operationString);
     }
+
+    private void setupPrevNextSearchTooltips() {
+        int startHit = searchOffset[0] - searchOptionsComposer.getMaxRecords();
+        if (startHit < 1) {
+            startHit = 1;
+        }
+        int endHit = searchOffset[0] - 1;
+        tooltipPrevText.setValue("hits " + 
+                    startHit + "-" + endHit);
+        startHit = searchOffset[0] + searchOffset[1];
+        endHit = startHit + searchOptionsComposer.getMaxRecords() - 1;
+        tooltipNextText.setValue("hits " + 
+                    startHit + "-" + endHit);
+    }
 }

SRUAggregator/trunk/src/main/java/eu/clarin/sru/fcs/aggregator/app/SearchOptions.java

@@ -16 +16 @@
 import eu.clarin.sru.fcs.aggregator.sopt.CorpusModelLive;
 import eu.clarin.sru.fcs.aggregator.sopt.CorpusRendererCached;
-import eu.clarin.sru.fcs.aggregator.sopt.CorpusRendererI;
 import eu.clarin.sru.fcs.aggregator.sopt.CorpusRendererLive;
 import eu.clarin.sru.fcs.aggregator.sopt.Languages;
 import eu.clarin.sru.fcs.aggregator.cache.ScanCache;
+import eu.clarin.sru.fcs.aggregator.sopt.CorpusRenderer;
+import eu.clarin.sru.fcs.aggregator.sopt.Language;
 import eu.clarin.sru.fcs.aggregator.util.SRUCQL;
 import java.lang.reflect.Type;
@@ -73 +74 @@
     
     private CorpusModelI corporaModel;
-    private CorpusRendererI corpusRenderer;
+    private CorpusRenderer corpusRenderer;
     
     private boolean liveMode = false;
@@ -135 +136 @@
             }
         }
+        onSelectAll(null);
     }
 
@@ -151 +153 @@
         Treechildren openTreeItems = tree.getTreechildren();
         for (Treeitem openItem : openTreeItems.getItems()) {
-            corpusRenderer.updateItem(openItem, true);
+            if (openItem.isVisible()) {
+                corpusRenderer.updateItem(openItem, true);
+            }
         }
     }
@@ -161 +165 @@
             
             DefaultTreeNode<Corpus> node = (DefaultTreeNode<Corpus>) openItem.getValue();
-            Corpus data = node.getData();
-            System.out.println(data);
+            //Corpus data = node.getData();
+            //System.out.println(data);
             
             corpusRenderer.updateItem(openItem, false);
@@ -227 +231 @@
             DefaultTreeNode<Corpus> node = item.getValue();
             Corpus corpus = node.getData();
-            for (String langCode : corpus.getLanguages()) {
-                availableLangs.add(langCode);
-            }
-        }
-
-        List<String> sortedAvailableLanguages = new ArrayList<String>(availableLangs.size());
-        sortedAvailableLanguages.addAll(availableLangs);
-        Collections.sort(sortedAvailableLanguages);
-        for (String langCode : sortedAvailableLanguages) {
-            String name = this.languages.nameForCode(langCode);
-            if (name != null) {
-                Comboitem item = this.languageSelect.appendItem(name);
-                item.setValue(langCode);
-            }
+            // offer for selection only those langauges
+            // that are unique for a corpus
+            if (corpus.getLanguages().size() == 1) {
+                availableLangs.add(corpus.getLanguages().iterator().next());
+            }
+            //for (String langCode : corpus.getLanguages()) {
+            //    availableLangs.add(langCode);
+            //}
+        }
+
+        //List<String> sortedAvailableLanguages = new ArrayList<String>(availableLangs.size());
+        //sortedAvailableLanguages.addAll(availableLangs);
+        //Collections.sort(sortedAvailableLanguages);
+        List<Language> sortedLangs = new ArrayList<Language>(availableLangs.size());
+        for (String langCode : availableLangs) {
+            Language lang = this.languages.langForCode(langCode);
+            if (lang != null) {
+                sortedLangs.add(lang);
+            }
+        }
+        Collections.sort(sortedLangs);
+        for (Language lang : sortedLangs) {
+                Comboitem item = this.languageSelect.appendItem(lang.getNameEn());
+                item.setValue(lang.getCode());
         }
     }

SRUAggregator/trunk/src/main/java/eu/clarin/sru/fcs/aggregator/app/SearchResults.java

@@ -21 +21 @@
 import org.zkoss.zk.ui.select.annotation.Wire;
 import org.zkoss.zul.Groupbox;
-import org.zkoss.zul.Label;
 import org.zkoss.zul.Messagebox;
 import eu.clarin.sru.client.SRUClientException;
-import eu.clarin.sru.client.SRURecord;
 import eu.clarin.sru.client.SRUSearchRetrieveRequest;
 import eu.clarin.sru.client.SRUSearchRetrieveResponse;
@@ -31 +29 @@
 import eu.clarin.sru.fcs.aggregator.sopt.Corpus;
 import eu.clarin.sru.fcs.aggregator.sopt.Languages;
-import eu.clarin.sru.fcs.aggregator.sresult.Kwic;
 import eu.clarin.sru.fcs.aggregator.sresult.SearchResult;
-import eu.clarin.sru.fcs.aggregator.sresult.SearchResultRecordRenderer;
+import eu.clarin.sru.fcs.aggregator.sresult.SearchResultContent;
+import eu.clarin.sru.fcs.aggregator.sresult.SearchResultGroupRenderer;
 import eu.clarin.sru.fcs.aggregator.util.SRUCQL;
-import eu.clarin.weblicht.wlfxb.io.WLDObjector;
-import eu.clarin.weblicht.wlfxb.io.WLFormatException;
-import eu.clarin.weblicht.wlfxb.md.xb.MetaData;
-import eu.clarin.weblicht.wlfxb.tc.api.MatchedCorpus;
-import eu.clarin.weblicht.wlfxb.tc.api.Token;
-import eu.clarin.weblicht.wlfxb.tc.xb.TextCorpusStored;
-import eu.clarin.weblicht.wlfxb.xb.WLData;
-import java.io.ByteArrayOutputStream;
 import java.io.IOException;
-import java.io.UnsupportedEncodingException;
 import java.text.SimpleDateFormat;
 import java.util.Date;
-import java.util.HashSet;
 import java.util.Random;
 import java.util.Timer;
@@ -53 +41 @@
 import java.util.concurrent.Future;
 import java.util.concurrent.atomic.AtomicBoolean;
-import opennlp.tools.tokenize.TokenizerME;
-import opennlp.tools.tokenize.TokenizerModel;
-import org.apache.poi.ss.usermodel.Cell;
-import org.apache.poi.ss.usermodel.CellStyle;
-import org.apache.poi.ss.usermodel.Font;
-import org.apache.poi.ss.usermodel.Row;
-import org.apache.poi.ss.usermodel.Sheet;
-import org.apache.poi.xssf.streaming.SXSSFWorkbook;
 import org.zkoss.zhtml.Filedownload;
 import org.zkoss.zk.ui.event.EventListener;
-import org.zkoss.zul.Column;
-import org.zkoss.zul.Columns;
-import org.zkoss.zul.Grid;
-import org.zkoss.zul.ListModel;
-import org.zkoss.zul.SimpleListModel;
 import org.zkoss.zul.Vlayout;
 
@@ -92 +67 @@
     private AtomicBoolean hasResults = new AtomicBoolean(false);
     private AtomicBoolean searchInProgress = new AtomicBoolean(false);
-    private int[] searchOffset;
+    //private int[] searchOffset = new int[2];
+    private int startRecord;
     private int maxRecords;
     private static final String WSPACE_SERVER_URL = "http://egi-cloud21.zam.kfa-juelich.de";
@@ -104 +80 @@
     private static final String DROP_OFF_URL = "http://ws1-clarind.esc.rzg.mpg.de/drop-off/storage/";
 
+    private Groupbox firstNoResultsGb = null;
+    private final Object firstNoResultsGbLoc = new Object();
+    
+    private SearchResultGroupRenderer srItemRenderer;
+    private SearchResultContent srExporter;
+    
+    private static final String SEARCH_RESULTS_ENCODING = "UTF-8";
 
     @Override
@@ -118 +101 @@
         Set<SearchResults> activeControllers = (Set<SearchResults>) Executions.getCurrent().getDesktop().getWebApp().getAttribute(WebAppListener.ACTIVE_SEARCH_CONTROLLERS);
         activeControllers.add(this);
+        
+        srItemRenderer = new SearchResultGroupRenderer();
+        srExporter = new SearchResultContent();
     }
 
@@ -126 +112 @@
         this.controlsVisibility.disableControls2();
         resultsBox.getChildren().clear();
+        synchronized (firstNoResultsGbLoc) {
+        firstNoResultsGb = null;
+        }
         this.searchInProgress.set(false);
         this.hasResults.set(false);
     }
 
-    void executeSearch(Map<String, Set<Corpus>> selectedCorpora, int maxRecords, String searchString, int[] searchOffset, String searchLanguage) {
+    void executeSearch(Map<String, Set<Corpus>> selectedCorpora, int startRecord, int maxRecords, String searchString, String searchLanguage) {
 
         this.controlsVisibility.disableControls1();
@@ -142 +131 @@
         this.hasResults.set(false);
         this.searchInProgress.set(true);
-        this.searchOffset = searchOffset;
+        this.startRecord = startRecord;
 
         // terminate previous search requests and corresponding response processing
@@ -156 +145 @@
         resultsProcessed = new ArrayList<SearchResult>();
         resultsUnprocessed = new ArrayList<SearchResult>();
-
+        synchronized (firstNoResultsGbLoc) {
+        firstNoResultsGb = null;
+        }
+        
         // set up timer, so that when it takes too much time to get the response to cancell the request
         timer = new Timer();
@@ -173 +165 @@
 
     private SearchResult executeSearch(Corpus corpus, String searchString) {
-        SearchResult resultsItem = new SearchResult(corpus, searchString);
+        SearchResult resultsItem = new SearchResult(corpus, searchString, startRecord, startRecord + maxRecords - 1);
         LOGGER.log(Level.INFO, "Executing search for {0} query={1} maxRecords={2}",
                 new Object[]{corpus.toString(), searchString, maxRecords});
@@ -183 +175 @@
         //searchRequest.setQuery("%22" + searchString + "%22");
         searchRequest.setQuery("\"" + searchString + "\"");
-        searchRequest.setStartRecord(searchOffset[0] + searchOffset[1]);
+        //searchRequest.setStartRecord(searchOffset[0] + searchOffset[1]);
+        searchRequest.setStartRecord(this.startRecord);
         if (resultsItem.hasCorpusHandler()) {
             searchRequest.setExtraRequestData(SRUCQL.SEARCH_CORPUS_HANDLE_PARAMETER, resultsItem.getCorpus().getHandle());
@@ -233 +226 @@
     }
 
+
     private class UpdateResultsThread extends Thread {
 
@@ -242 +236 @@
                 if (!resultsItem.isWaitingForResponse()) {
                     resultsItem.consumeResponse();
-                    Executions.schedule(resultsBox.getDesktop(), new SearchResults.ResponseListener(resultsItem, currentRequestId), new Event("onDummy"));
+                    Executions.schedule(resultsBox.getDesktop(), new SearchResults.ResponseListener(srItemRenderer, resultsItem, currentRequestId), new Event("onDummy"));
                     // this alternative to Executions.schedule() - sinchronious update - 
                     // doesn't work: if interrupted here (exception thrown), then
@@ -273 +267 @@
         int requestId;
         SearchResult resultsItem;
-
-        public ResponseListener(SearchResult resultsItem, int requestId) {
+        SearchResultGroupRenderer srRenderer;
+
+        public ResponseListener(SearchResultGroupRenderer srRenderer, SearchResult resultsItem, int requestId) {
+            this.srRenderer = srRenderer;
             this.resultsItem = resultsItem;
             this.requestId = requestId;
@@ -283 +279 @@
 
             // create groupbox with search results item
-            Groupbox groupbox = createRecordsGroup(resultsItem);
-
-            // appand this search result only if it
+            //Groupbox groupbox = createRecordsGroup(resultsItem);
+            Groupbox groupbox = this.srRenderer.createRecordsGroup(resultsItem);
+
+            // append this search result only if it
             // is a result of the current request
             if (requestId == currentRequestId) {
-                resultsBox.appendChild(groupbox);
+                // if the item has results, append it before
+                // the item that has no resuls (if exists):
+                if (groupbox.getAttribute(SearchResultGroupRenderer.NO_RESULTS) == null) {
+                    resultsBox.insertBefore(groupbox, firstNoResultsGb);
+                // if there are no results in this item,
+                // append it a the end:
+                } else {
+                    resultsBox.appendChild(groupbox);
+                    synchronized (firstNoResultsGbLoc) {
+                        if (firstNoResultsGb == null) {
+                            firstNoResultsGb = groupbox;
+                        }
+                    }
+                }
+
             }
 
@@ -295 +306 @@
             if (requestId != currentRequestId) {
                 groupbox.detach();
+                synchronized (firstNoResultsGbLoc) {
+                    firstNoResultsGb = null;
+                }
             }
 
@@ -302 +316 @@
                 if (last) {
                     controlsVisibility.disableProgressMeter();
-                    searchOffset[0] = searchOffset[0] + searchOffset[1];
-                    searchOffset[1] = maxRecords;
                     pagesVisibility.openSearchResult();
 
-                    if (searchOffset[0] > 1) {
+                    if (startRecord > 1) {
                         controlsVisibility.enablePrevButton();
                     }
@@ -321 +333 @@
     }
 
-    private Groupbox createRecordsGroup(SearchResult resultsItem) {
-
-        Groupbox recordsGroup = new Groupbox();
-        recordsGroup.setMold("3d");
-        recordsGroup.setSclass("ccsLightBlue");
-        recordsGroup.setContentStyle("border:0;");
-        recordsGroup.setStyle("margin:10px;10px;10px;10px;");
-        recordsGroup.setClosable(true);
-        // create title
-        StringBuilder sb = new StringBuilder();
-        if (resultsItem.hasCorpusHandler()) {
-            if (resultsItem.getCorpus().getDisplayName() != null) {
-//                sb.append(" ");
-                sb.append(resultsItem.getCorpus().getDisplayName());
-                sb.append(", ");
-            }
-        }
-        sb.append(resultsItem.getCorpus().getInstitution().getName());
-        recordsGroup.setTitle(sb.toString());
-
-        // populate it with records grid or failure message
-        if (resultsItem.getResponse() == null) { // there was an error in response
-            recordsGroup.appendChild(new Label("Sorry, the search failed!"));
-        } else if (resultsItem.getResponse().hasRecords()) { // the response was fine and there >=1 records
-            Grid grid = new Grid();
-//            grid.setWidth("100%");
-//            grid.setMold("paging");
-//            grid.setPageSize(10);
-            Columns columns = new Columns();
-            Column c;
-
-            c = new Column();
-            //c.setLabel("Left");
-            columns.appendChild(c);
-            //c.setHflex("2");
-            c = new Column();
-            //c.setLabel("Hit");
-            c.setHflex("min");
-            //c.setHflex("1");
-            columns.appendChild(c);
-            c = new Column();
-            //c.setHflex("2");
-            //c.setLabel("Right");
-            // info column
-            columns.appendChild(c);
-            c = new Column();
-            c.setHflex("min");
-            columns.appendChild(c);
-            grid.appendChild(columns);
-
-            List<SRURecord> sruRecords = resultsItem.getResponse().getRecords();
-            ListModel lmodel = new SimpleListModel(sruRecords);
-            grid.setModel(lmodel);
-            grid.setRowRenderer(new SearchResultRecordRenderer(resultsItem));
-            recordsGroup.appendChild(grid);
-            grid.setStyle("margin:10px;border:0px;");
-        } else { // the response was fine, but there are no records
-            recordsGroup.appendChild(new Label("no results"));
-        }
-        return recordsGroup;
-    }
+
     
     public void exportTCF() {
-            byte[] bytes = getExportTokenizedTCF();
+            byte[] bytes = srExporter.getExportTokenizedTCF(resultsProcessed, searchLanguage, languages);
             if (bytes != null) {
                 Filedownload.save(bytes, "text/tcf+xml", "ClarinDFederatedContentSearch.xml");
@@ -391 +343 @@
     
     public void exportText() {
-            String text = getExportText().toString();
+            String text = srExporter.getExportText(resultsProcessed);
             if (text != null) {
                 Filedownload.save(text, "text/plain", "ClarinDFederatedContentSearch.txt");
@@ -400 +352 @@
     void exportExcel() {
         
-        byte[] bytes = getExportExcel();
+        byte[] bytes = srExporter.getExportExcel(resultsProcessed);
             if (bytes != null) {
                 Filedownload.save(bytes, "text/tcf+xml", "ClarinDFederatedContentSearch.xls");
@@ -406 +358 @@
     }
 
-    private byte[] getExportExcel() {
-        
-        boolean noResult = true;
-        SXSSFWorkbook workbook = null;
-        ByteArrayOutputStream excelStream = new ByteArrayOutputStream();
-        if (resultsProcessed != null && !resultsProcessed.isEmpty()) {
-            try {
-                String[] headers = new String[] {
-                    "LEFT CONTEXT", "KEYWORD", "RIGHT CONTEXT", "PID", "REFERENCE"};
-            
-                workbook = new SXSSFWorkbook();
-                Sheet sheet = workbook.createSheet();
-
-                Font boldFont = workbook.createFont();
-                boldFont.setBoldweight(Font.BOLDWEIGHT_BOLD);
-
-                // Header
-                CellStyle headerStyle = workbook.createCellStyle();
-                headerStyle.setFont(boldFont);
-
-                Row row = sheet.createRow(0);
-
-                for (int j = 0; j < headers.length; ++j) {
-                    Cell cell = row.createCell(j, Cell.CELL_TYPE_STRING);
-                    cell.setCellValue(headers[j]);
-                    cell.setCellStyle(headerStyle);
-                }
-
-                // Body
-                Cell cell;
-                for (int k = 0; k < resultsProcessed.size(); k++) {
-                    SearchResult result = resultsProcessed.get(k);
-                    List<Kwic> kwics = result.getKwics();
-                    for (int i = 0; i < kwics.size(); i++) {
-                        Kwic kwic = kwics.get(i);
-                        row = sheet.createRow(k + i + 1);
-                        cell = row.createCell(0, Cell.CELL_TYPE_STRING);
-                        cell.setCellValue(kwic.getLeft());
-                        cell = row.createCell(1, Cell.CELL_TYPE_STRING);
-                        cell.setCellValue(kwic.getKeyword());
-                        cell = row.createCell(2, Cell.CELL_TYPE_STRING);
-                        cell.setCellValue(kwic.getRight());
-                        if (kwic.getPid() != null) {
-                            cell = row.createCell(3, Cell.CELL_TYPE_STRING);
-                            cell.setCellValue(kwic.getPid());
-                        }
-                        if (kwic.getReference() != null) {
-                            cell = row.createCell(3, Cell.CELL_TYPE_STRING);
-                            cell.setCellValue(kwic.getReference());
-                        }
-                        noResult = false;
-                    }
-                }
-                workbook.write(excelStream);
-            } catch (IOException ex) {
-                // should not happen
-                Logger.getLogger(SearchResults.class.getName()).log(Level.SEVERE, null, ex);
-            } finally {
-                if (workbook != null) {
-                    workbook.dispose();
-                }
-            }
-        }
-        if (noResult) {
-            Messagebox.show("Nothing to export!");
-            return null;
-        } else {
-            return excelStream.toByteArray();
-        }
-        
-    }
+  
 
     void exportPWText(String user, String pass) {
         byte[] bytes = null;
         try {
-            bytes = getExportText().toString().getBytes("UTF-8");
-        } catch (UnsupportedEncodingException ex) {
+            String text = srExporter.getExportText(resultsProcessed);
+            if (text != null) {
+                bytes = text.getBytes(SEARCH_RESULTS_ENCODING);
+            }
+        } catch (Exception ex) {
             Logger.getLogger(SearchResults.class.getName()).log(Level.SEVERE, null, ex);
         }
@@ -491 +376 @@
     
     String useWebLichtOnText() {
-        byte[] bytes = null;
         String url = null;
         try {
-            bytes = getExportText().toString().getBytes("UTF-8");
-        } catch (UnsupportedEncodingException ex) {
+            String text = srExporter.getExportText(resultsProcessed);
+            if (text != null) {
+                byte[] bytes = text.getBytes(SEARCH_RESULTS_ENCODING);
+                url = uploadToDropOff(bytes, "text/plan",".txt");
+            }
+        } catch (Exception ex) {
             Logger.getLogger(SearchResults.class.getName()).log(Level.SEVERE, null, ex);
-        }
-        if (bytes != null) {
-            url = uploadToDropOff(bytes, "text/plan",".txt");
         }
         return url;
@@ -506 +391 @@
     String useWebLichtOnToks() {
         String url = null;
-        byte[] bytes = getExportTokenizedTCF();
+        byte[] bytes = srExporter.getExportTokenizedTCF(resultsProcessed, searchLanguage, languages);
         if (bytes != null) {
             url = uploadToDropOff(bytes, "text/tcf+xml",".tcf");
@@ -514 +399 @@
 
     void exportPWExcel(String user, String pass) {
-        byte[] bytes = getExportExcel();
+        byte[] bytes = srExporter.getExportExcel(resultsProcessed);
             if (bytes != null) {
                 uploadToPW(user, pass, bytes, "application/vnd.ms-excel",".xls");
@@ -522 +407 @@
    public void exportPWTCF(String user, String pass) {
        
-       byte[] bytes = getExportTokenizedTCF();
+       byte[] bytes = srExporter.getExportTokenizedTCF(resultsProcessed, searchLanguage, languages);
             if (bytes != null) {
                 uploadToPW(user, pass, bytes, "text/tcf+xml",".tcf");
         }
     }
-    
-    private byte[] getExportTCF() {
-        StringBuilder text = new StringBuilder();
-        Set<String> resultsLangs = new HashSet<String>();
-        if (resultsProcessed != null && !resultsProcessed.isEmpty()) {
-            for (SearchResult result : resultsProcessed) {
-                resultsLangs.addAll(result.getCorpus().getLanguages());
-                for (Kwic kwic : result.getKwics()) {
-                    text.append(kwic.getLeft());
-                    text.append(" ");
-                    text.append(kwic.getKeyword());
-                    text.append(" ");
-                    text.append(kwic.getRight());
-                    text.append("\n");
-                }
-            }
-
-        }
-        ByteArrayOutputStream os = null;
-        if (text.length() == 0) {
-            Messagebox.show("Nothing to export!");
-        } else {
-            WLData data;
-            MetaData md = new MetaData();
-            String resultsLang = "unknown";
-            if (resultsLangs.size() == 1) {
-                resultsLang = resultsLangs.iterator().next();
-                String code2 = languages.code2ForCode(resultsLang);
-                if (code2 != null) {
-                    resultsLang = code2;
-                }
-            } else if (!searchLanguage.equals("anylang")) {
-                String code2 = languages.code2ForCode(searchLanguage);
-                if (code2 == null) {
-                    resultsLang = searchLanguage;
-                } else {
-                    resultsLang = code2;
-                }
-            }
-            TextCorpusStored tc = new TextCorpusStored(resultsLang);
-            tc.createTextLayer().addText(text.toString());
-            data = new WLData(md, tc);
-            os = new ByteArrayOutputStream();
-            try {
-                WLDObjector.write(data, os);
-            } catch (WLFormatException ex) {
-                LOGGER.log(Level.SEVERE, "Error exporting TCF {0} {1}", new String[]{ex.getClass().getName(), ex.getMessage()});
-                Messagebox.show("Sorry, export error!");
-            }
-        }
-        if (os == null) {
-            return null;
-        } else {
-            return os.toByteArray();
-        }
-    }
-    
-    
-        private byte[] getExportTokenizedTCF() {
-        StringBuilder text = new StringBuilder();
-        Set<String> resultsLangs = new HashSet<String>();
-        if (resultsProcessed != null && !resultsProcessed.isEmpty()) {
-            for (SearchResult result : resultsProcessed) {
-                resultsLangs.addAll(result.getCorpus().getLanguages());
-                for (Kwic kwic : result.getKwics()) {
-                    text.append(kwic.getLeft());
-                    text.append(" ");
-                    text.append(kwic.getKeyword());
-                    text.append(" ");
-                    text.append(kwic.getRight());
-                    text.append("\n");
-                }
-            }
-
-        }
-        ByteArrayOutputStream os = null;
-        if (text.length() == 0) {
-            Messagebox.show("Nothing to export!");
-        } else {
-            WLData data;
-            MetaData md = new MetaData();
-            String resultsLang = "unknown";
-            if (resultsLangs.size() == 1) {
-                resultsLang = resultsLangs.iterator().next();
-                String code2 = languages.code2ForCode(resultsLang);
-                if (code2 != null) {
-                    resultsLang = code2;
-                }
-            } else if (!searchLanguage.equals("anylang")) {
-                String code2 = languages.code2ForCode(searchLanguage);
-                if (code2 == null) {
-                    resultsLang = searchLanguage;
-                } else {
-                    resultsLang = code2;
-                }
-            }
-            TextCorpusStored tc = new TextCorpusStored(resultsLang);
-            tc.createTextLayer().addText(text.toString());
-            addTokensSentencesMatches(tc);
-            data = new WLData(md, tc);
-            os = new ByteArrayOutputStream();
-            try {
-                WLDObjector.write(data, os);
-            } catch (WLFormatException ex) {
-                LOGGER.log(Level.SEVERE, "Error exporting TCF {0} {1}", new String[]{ex.getClass().getName(), ex.getMessage()});
-                Messagebox.show("Sorry, export error!");
-            }
-        }
-        if (os == null) {
-            return null;
-        } else {
-            return os.toByteArray();
-        }
-    }
-    
-        
-    private void addTokensSentencesMatches(TextCorpusStored tc) {
-        
-        TokenizerModel model = (TokenizerModel) Executions.getCurrent().getDesktop().getWebApp().getAttribute(WebAppListener.DE_TOK_MODEL);
-        
-        if (model == null || !tc.getLanguage().equals("de")) {
-            return;
-        }
-        TokenizerME tokenizer = new TokenizerME(model);
-        
-        if (resultsProcessed != null && !resultsProcessed.isEmpty()) {
-            tc.createTokensLayer();
-            tc.createSentencesLayer();
-            tc.createMatchesLayer("FCS", resultsProcessed.get(0).getSearchString());
-            for (SearchResult result : resultsProcessed) {
-                MatchedCorpus mCorpus = tc.getMatchesLayer().addCorpus(result.getCorpus().getDisplayName(), result.getCorpus().getHandle());
-                for (Kwic kwic : result.getKwics()) {
-                    List<Token> tokens = new ArrayList<Token>();
-                    addToTcfTokens(tokens, tc, tokenizer.tokenize(kwic.getLeft()));
-                    String[] target = tokenizer.tokenize(kwic.getKeyword());
-                    List<Token> targetTokens = addToTcfTokens(tokens, tc, target);
-                    addToTcfTokens(tokens, tc, tokenizer.tokenize(kwic.getRight()));
-                    tc.getSentencesLayer().addSentence(tokens);
-                    List<String> pidAndRef = new ArrayList<String>();
-                    if (kwic.getPid() != null) {
-                        pidAndRef.add(kwic.getPid());
-                    }
-                    if (kwic.getReference() != null) {
-                        pidAndRef.add(kwic.getReference());
-                    }
-                    tc.getMatchesLayer().addItem(mCorpus, targetTokens, pidAndRef);
-                }
-            }
-        }
-    }
-    
-    
-    private List<Token> addToTcfTokens(List<Token> tokens, TextCorpusStored tc, String[] tokenStrings) {
-        List<Token> addedTokens = new ArrayList<Token>(tokenStrings.length);
-        for (String tokenString : tokenStrings) {
-            Token token = tc.getTokensLayer().addToken(tokenString);
-            addedTokens.add(token);
-            tokens.add(token);
-        }
-        return addedTokens;
-    }
-    
-    private CharSequence getExportText() {
-        StringBuilder text = new StringBuilder();
-        //Set<String> resultsLangs = new HashSet<String>();
-        if (resultsProcessed != null && !resultsProcessed.isEmpty()) {
-            for (SearchResult result : resultsProcessed) {
-                //resultsLangs.addAll(result.getCorpus().getLanguages());
-                for (Kwic kwic : result.getKwics()) {
-                    text.append(kwic.getLeft());
-                    text.append(" ");
-                    text.append(kwic.getKeyword());
-                    text.append(" ");
-                    text.append(kwic.getRight());
-                    text.append("\n");
-                }
-            }
-
-        }
-        if (text.length() == 0) {
-            Messagebox.show("Nothing to export!");
-            return null;
-        } else {
-            return text;
-        }
-    }
+
     
     public void exportCSV() {
-        String csv = getExportCSV(";");
+        String csv = srExporter.getExportCSV(resultsProcessed, ";");
         if (csv != null) {
             Filedownload.save(csv.toString(), "text/plain", "ClarinDFederatedContentSearch.csv");
@@ -723 +423 @@
     
     public void exportPWCSV(String user, String pass) {
-        String csv = getExportCSV(";");
+        //String csv = getExportCSV(";");
+        String csv = srExporter.getExportCSV(resultsProcessed, ";");
         if (csv != null) {
             uploadToPW(user, pass, csv.getBytes(), "text/csv",".csv");
         }
     }
-
-    private String getExportCSV(String separator) {
-
-        boolean noResult = true;
-        StringBuilder csv = new StringBuilder();
-        if (resultsProcessed != null && !resultsProcessed.isEmpty()) {
-            String[] headers = new String[] {
-                    "LEFT CONTEXT", "KEYWORD", "RIGHT CONTEXT", "PID", "REFERENCE"};
-            for (String header : headers) {
-                csv.append("\""); csv.append(header); csv.append("\"");
-                csv.append(separator);
-            }
-            csv.append("\n");
-
-            for (SearchResult result : resultsProcessed) {
-                for (Kwic kwic : result.getKwics()) {
-                    csv.append("\""); csv.append(escapeQuotes(kwic.getLeft())); csv.append("\"");
-                    csv.append(separator);
-                    csv.append("\""); csv.append(escapeQuotes(kwic.getKeyword())); csv.append("\"");
-                    csv.append(separator);
-                    csv.append("\""); csv.append(escapeQuotes(kwic.getRight())); csv.append("\"");
-                    csv.append(separator);
-                    csv.append("\"");
-                    if (kwic.getPid() != null) {
-                        csv.append(escapeQuotes(kwic.getPid()));
-                    }
-                    csv.append("\"");
-                    csv.append(separator);
-                    csv.append("\"");
-                    if (kwic.getReference() != null) {
-                        csv.append(escapeQuotes(kwic.getReference()));
-                    }
-                    csv.append("\"");
-                    csv.append("\n");
-                    noResult = false;
-                }
-            }
-        }
-        if (noResult) {
-            Messagebox.show("Nothing to export!");
-            return null;
-        } else {
-            return csv.toString();
-        }
-    }
-    
-    private CharSequence escapeQuotes(String text) {
-        StringBuilder sb = new StringBuilder();
-        for (int i = 0; i < text.length(); i++) {
-            char ch = text.charAt(i);
-            if (ch == '"') {
-                sb.append('"');
-            }
-            sb.append(ch);
-        }
-        return sb;
-    }
-    
-    private void uploadToPW(String user, String pass, byte[] bytes, String mimeType, String fileExtention) {
+    
+        private void uploadToPW(String user, String pass, byte[] bytes, String mimeType, String fileExtention) {
         try {
                 Sardine sardine = SardineFactory.begin();
@@ -850 +494 @@
     }
 
+
     private void setUpSRUVersion() {
         String[] paramValue = Executions.getCurrent().getParameterMap().get(SRUCQL.VERSION);

SRUAggregator/trunk/src/main/java/eu/clarin/sru/fcs/aggregator/app/WebAppListener.java

@@ -51 +51 @@
     private static final String SCAN_DIR_NAME = "scan";
     private static final TimeUnit CACHE_UPDATE_INTERVAL_UNIT = TimeUnit.HOURS;
-    private static final int CACHE_UPDATE_INTERVAL = 5;
-    private static final int CACHE_MAX_DEPTH = 2;
+    private static final int CACHE_UPDATE_INTERVAL = 6;
+    private static final int CACHE_MAX_DEPTH = 3;
     private static final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
     public static final String CORPUS_CRAWLER = "CORPUS_CRAWLER";
@@ -72 +72 @@
         
         setUpScanCache(webapp);
+        //setUpScanCacheForReadOnly(webapp);
         
         setUpTokenizers(webapp);
@@ -91 +92 @@
 
     private String getScanDirectory() {
-        File aggregatorDir = new File(System.getProperty("user.home"), "/." + AGGREGATOR_DIR_NAME);
+        //File aggregatorDir = new File(System.getProperty("user.home"), "/." + AGGREGATOR_DIR_NAME);
         //File aggregatorDir = new File("/var/www", "/." + AGGREGATOR_DIR_NAME);
+        File aggregatorDir = new File("/data/fcsAggregator");
         
         if (!aggregatorDir.exists()) {
-            aggregatorDir.mkdir();
+            if (!aggregatorDir.mkdir()) {
+                LOGGER.info("Scan directory does not exist and cannot be created: " 
+                        + aggregatorDir.getAbsolutePath());
+            }
         }
         File scanDir = new File(aggregatorDir, SCAN_DIR_NAME);
@@ -105 +110 @@
         return scanPath;
     }
+    
+    /**
+     * Use this method instead of setUpScanCache() method if it is necessary
+     * to run the application without scan data crawl, given that the scan 
+     * data was crawled before and was stored as cache under appropriate
+     * location (useful when testing or when smth is wrong with the endpoint 
+     * scan responses).
+     * 
+     * @param webapp 
+     */
+    private void setUpScanCacheForReadOnly(WebApp webapp) {
+        ScanCacheFiled scanCacheFiled = new ScanCacheFiled(getScanDirectory());
+        ScanCache scanCache;
+        LOGGER.info("Start cache read");
+        try {
+            scanCache = scanCacheFiled.read();
+            LOGGER.info("Finished cache read, number of root corpora: " + scanCache.getRootCorpora().size());
+        } catch (Exception e) {
+            LOGGER.log(Level.SEVERE, "Error while reading the scan cache!", e);
+            scanCache = new SimpleInMemScanCache();
+        }
+        webapp.setAttribute(CORPUS_CACHE, scanCache);
+    }
 
     private void setUpScanCache(WebApp webapp) {
@@ -111 +139 @@
         CenterRegistryI centerRegistry = new CenterRegistryLive();
         SRUThreadedClient sruScanClient = (SRUThreadedClient) webapp.getAttribute(WebAppListener.SHARED_SRU_CLIENT);
-        EndpointUrlFilter filter = new EndpointUrlFilter();
-        //filter.urlShouldContainAnyOf("leipzig", ".mpi.nl");
+        //EndpointUrlFilter filter = new EndpointUrlFilter();
+        //filter.urlShouldContainAnyOf("leipzig");
         //filter.urlShouldContainAnyOf("uni-tuebingen.de");
-        filter.urlShouldContainAnyOf("uni-tuebingen.de", ".mpi.nl");
+        //filter.urlShouldContainAnyOf("uni-tuebingen.de", ".mpi.nl");
         //filter.urlShouldContainAnyOf("dspin.dwds.de", "lindat.");
-        ScanCrawler scanCrawler = new ScanCrawler(centerRegistry, sruScanClient, filter, CACHE_MAX_DEPTH);
-        //ScanCrawler scanCrawler = new ScanCrawler(centerRegistry, sruScanClient, null, CACHE_MAX_DEPTH);
+        //ScanCrawler scanCrawler = new ScanCrawler(centerRegistry, sruScanClient, filter, CACHE_MAX_DEPTH);
+        ScanCrawler scanCrawler = new ScanCrawler(centerRegistry, sruScanClient, null, CACHE_MAX_DEPTH);
         ScanCache scanCache;
 
@@ -191 +219 @@
         webapp.setAttribute(DE_TOK_MODEL, model);
     }
+
 }

SRUAggregator/trunk/src/main/java/eu/clarin/sru/fcs/aggregator/cache/ScanCacheFiled.java

@@ -22 +22 @@
  * Utility for reading/writing scan data (endpoints descriptions) from/to
  * ScanCache from/to local files.
+ * 
+ * The data saved in a folder into files has the following format:
+ * inst.txt lists all the centers and the endpoints that have CQL compliant 
+ * resources, with each CQL endpoint top resources assigned a number.
+ * Number that represents a resource corresponds to a folder name, inside which 
+ * resource info is stored in a file corpus.txt. If a resource has sub-resources, 
+ * the folder contains sub-folders, also named by numbers, with their 
+ * corresponding corpus.txt files, and so on. 
+ * The corpus.txt file contains each piece of info on a separate line. Line 1 
+ * is resource endpoint, line 2 - handle, line 3 - display name, line 4 - landing 
+ * page, 5 - desription, 6 - languages (multiple languages are separated by | 
+ * separator), 7 - number of records.
  *
  * @author yanapanchenko

SRUAggregator/trunk/src/main/java/eu/clarin/sru/fcs/aggregator/cache/ScanCrawler.java

@@ -1 +1 @@
 package eu.clarin.sru.fcs.aggregator.cache;
 
+import eu.clarin.sru.client.SRUClientException;
 import eu.clarin.sru.client.SRUScanRequest;
 import eu.clarin.sru.client.SRUScanResponse;
@@ -10 +11 @@
 import eu.clarin.sru.fcs.aggregator.sopt.Institution;
 import eu.clarin.sru.fcs.aggregator.util.SRUCQL;
+import java.util.ArrayList;
 import java.util.List;
+import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
@@ -72 +75 @@
     }
 
-    // TODO: ask Oliver to add API support for the extra info in the 
+
+
+    private void addCorpora(SRUThreadedClient sruScanClient, String endpointUrl,
+            Institution institution, int depth, Corpus parentCorpus, ScanCache cache) {
+        
+
+        depth++;
+        if (depth > maxDepth) {
+            return;
+        }
+
+
+            List<Corpus> childrenCorpora = doScan(sruScanClient, endpointUrl, institution, parentCorpus);
+            
+            //if (childrenCorpora.isEmpty()) {
+                for (Corpus c : childrenCorpora) {
+                    // don't add corpus that introduces cyclic references
+                    // as of March 2014, there are 2 such endpoints...
+                    if (cache.getCorpus(c.getHandle())!= null) {
+                        LOGGER.warning("Cyclic reference in corpus " + c.getHandle() + " of endpoint " + endpointUrl);
+                        continue;
+                    }
+                    //Corpus c = createCorpus(institution, endpointUrl, term);
+                    //Corpus c = new Corpus(institution, endpointUrl);
+//                    c.setHandle(term.getValue());
+//                    c.setDisplayName(term.getDisplayTerm());
+//                    if (term.getNumberOfRecords() > 0) {
+//                        c.setNumberOfRecords(term.getNumberOfRecords());
+//                    }
+//                    addExtraInfo(c, term);
+                    cache.addCorpus(c, parentCorpus);
+                    // how not to query default corpus???
+                    addCorpora(sruScanClient, c.getEndpointUrl(), c.getInstitution(),
+                            depth, c, cache);
+                }
+                //} else if () {
+                // TODO if diagnistics came back, try simple scan without the 
+                // SRUCQLscan.RESOURCE_INFO_PARAMETER
+//            } 
+//else {
+//                if (parentCorpus == null) { // means root
+                    // create default root corpus:
+//                    Corpus c = new Corpus(institution, endpointUrl);
+//                    cache.addCorpus(c);
+//                }
+//            }
+            
+            
+//            SRUScanResponse response = doScan(sruScanClient, endpointUrl, parentCorpus);
+//            
+//            
+//            if (response != null && response.hasTerms()) {
+//                for (SRUTerm term : response.getTerms()) {
+//                    // don't add corpus that introduces cyclic references
+//                    // as of March 2014, there are 2 such endpoints...
+//                    if (cache.getCorpus(term.getValue())!= null) {
+//                        LOGGER.warning("Cyclic reference in corpus " + term.getValue() + " of endpoint " + endpointUrl);
+//                        continue;
+//                    }
+//                    Corpus c = createCorpus(institution, endpointUrl, term);
+//                    //Corpus c = new Corpus(institution, endpointUrl);
+////                    c.setHandle(term.getValue());
+////                    c.setDisplayName(term.getDisplayTerm());
+////                    if (term.getNumberOfRecords() > 0) {
+////                        c.setNumberOfRecords(term.getNumberOfRecords());
+////                    }
+////                    addExtraInfo(c, term);
+//                    cache.addCorpus(c, parentCorpus);
+//                    addCorpora(sruScanClient, c.getEndpointUrl(), c.getInstitution(),
+//                            depth, c, cache);
+//                }
+//                //} else if () {
+//                // TODO if diagnistics came back, try simple scan without the 
+//                // SRUCQLscan.RESOURCE_INFO_PARAMETER
+//            } else {
+//                if (parentCorpus == null) { // means root
+//                    // create default root corpus:
+//                    Corpus c = new Corpus(institution, endpointUrl);
+//                    cache.addCorpus(c);
+//                }
+//            }
+        
+    }
+
+    private static String normalizeHandle(Corpus corpus, boolean root) {
+        if (root) {
+            return Corpus.ROOT_HANDLE;
+        }
+        String handle = corpus.getHandle();
+        if (Corpus.HANDLE_WITH_SPECIAL_CHARS.matcher(handle).matches()) {
+            //resourceValue = "%22" + resourceValue + "%22";
+            handle = "\"" + handle + "\"";
+        }
+        return handle;
+    }
+
+//    public static SRUScanResponse doScan(SRUThreadedClient sruScanClient, 
+//            String endpointUrl, Corpus parentCorpus) {
+//        
+//        Future<SRUScanResponse> corporaResponse = null;
+//        SRUScanResponse response = null;
+//        try {
+//                    SRUScanRequest corporaRequest = new SRUScanRequest(endpointUrl);
+//            StringBuilder scanClause = new StringBuilder(SRUCQL.SCAN_RESOURCE_PARAMETER);
+//            scanClause.append("=");
+//            String normalizedHandle = normalizeHandle(parentCorpus, parentCorpus == null);
+//            scanClause.append(normalizedHandle);
+//            corporaRequest.setScanClause(scanClause.toString());
+//            corporaRequest.setExtraRequestData(SRUCQL.SCAN_RESOURCE_INFO_PARAMETER, 
+//                    SRUCQL.SCAN_RESOURCE_INFO_PARAMETER_DEFAULT_VALUE);
+//            corporaResponse = sruScanClient.scan(corporaRequest);
+//            Thread.sleep(5000);
+//            response = corporaResponse.get(600, TimeUnit.SECONDS);
+//            return response;
+//        }    catch (TimeoutException ex) {
+//            LOGGER.log(Level.SEVERE, "Timeout scanning corpora {0} at {1} {2} {3}",
+//                    new String[]{Corpus.ROOT_HANDLE, endpointUrl, ex.getClass().getName(), ex.getMessage()});
+//        } catch (Exception ex) {
+//            LOGGER.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
+//                    new String[]{Corpus.ROOT_HANDLE, endpointUrl, ex.getClass().getName(), ex.getMessage()});
+//        } finally {
+//            if (corporaResponse != null && !corporaResponse.isDone()) {
+//                corporaResponse.cancel(true);
+//            }
+//        }
+//        
+//        return response;
+//    }
+    
+    
+        public static List<Corpus> doScan(SRUThreadedClient sruScanClient, 
+            String endpointUrl, Institution institution, Corpus parentCorpus) {
+        
+        List<Corpus> corpora = new ArrayList<Corpus>();
+        Future<SRUScanResponse> corporaResponse = null;
+        SRUScanResponse response = null;
+        try {
+                    SRUScanRequest corporaRequest = new SRUScanRequest(endpointUrl);
+            StringBuilder scanClause = new StringBuilder(SRUCQL.SCAN_RESOURCE_PARAMETER);
+            scanClause.append("=");
+            String normalizedHandle = normalizeHandle(parentCorpus, parentCorpus == null);
+            scanClause.append(normalizedHandle);
+            corporaRequest.setScanClause(scanClause.toString());
+            corporaRequest.setExtraRequestData(SRUCQL.SCAN_RESOURCE_INFO_PARAMETER, 
+                    SRUCQL.SCAN_RESOURCE_INFO_PARAMETER_DEFAULT_VALUE);
+            corporaResponse = sruScanClient.scan(corporaRequest);
+            Thread.sleep(5000);
+            response = corporaResponse.get(600, TimeUnit.SECONDS);
+        }    catch (TimeoutException ex) {
+            LOGGER.log(Level.SEVERE, "Timeout scanning corpora {0} at {1} {2} {3}",
+                    new String[]{Corpus.ROOT_HANDLE, endpointUrl, ex.getClass().getName(), ex.getMessage()});
+        } catch (Exception ex) {
+            LOGGER.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
+                    new String[]{Corpus.ROOT_HANDLE, endpointUrl, ex.getClass().getName(), ex.getMessage()});
+        } finally {
+            if (corporaResponse != null && !corporaResponse.isDone()) {
+                corporaResponse.cancel(true);
+            }
+        }
+        
+        
+        if (response != null && response.hasTerms()) {
+                for (SRUTerm term : response.getTerms()) {
+                    // don't add corpus that introduces cyclic references
+                    // as of March 2014, there are 2 such endpoints...
+                    //if (cache.getCorpus(term.getValue())!= null) {
+                    //    LOGGER.warning("Cyclic reference in corpus " + term.getValue() + " of endpoint " + endpointUrl);
+                    //    continue;
+                    //}
+                    Corpus c = createCorpus(institution, endpointUrl, term);
+                    corpora.add(c);
+                }
+                //} else if () {
+                // TODO if diagnistics came back, try simple scan without the 
+                // SRUCQLscan.RESOURCE_INFO_PARAMETER
+            } else {
+                if (parentCorpus == null) { // means root
+                    // create default root corpus:
+                    Corpus c = new Corpus(institution, endpointUrl);
+                    corpora.add(c);
+                }
+            }
+        return corpora;
+    }
+
+    private static Corpus createCorpus(Institution institution, String endpointUrl, SRUTerm term) {
+                    Corpus c = new Corpus(institution, endpointUrl);
+                    c.setHandle(term.getValue());
+                    c.setDisplayName(term.getDisplayTerm());
+                    if (term.getNumberOfRecords() > 0) {
+                        c.setNumberOfRecords(term.getNumberOfRecords());
+                    }
+                    addExtraInfo(c, term);
+                    return c;
+    }
+    
+        // TODO: ask Oliver to add API support for the extra info in the 
     // SRU client/server libraries, so that it's not necessary to work
     // with DocumentFragment
-    private void addExtraInfo(Corpus c, SRUTerm term) {
+    private static void addExtraInfo(Corpus c, SRUTerm term) {
 
         DocumentFragment extraInfo = term.getExtraTermData();
@@ -116 +315 @@
         }
     }
-
-    private void addCorpora(SRUThreadedClient sruScanClient, String endpointUrl,
-            Institution institution, int depth, Corpus parentCorpus, ScanCache cache) {
-        
-        Future<SRUScanResponse> corporaResponse = null;
-
-        depth++;
-        if (depth > maxDepth) {
-            return;
-        }
-        try {
-            boolean root = false;
-            if (parentCorpus == null) {
-                root = true;
-            }
-
-            SRUScanRequest corporaRequest = new SRUScanRequest(endpointUrl);
-            StringBuilder scanClause = new StringBuilder(SRUCQL.SCAN_RESOURCE_PARAMETER);
-            scanClause.append("=");
-            String normalizedHandle = normalizeHandle(parentCorpus, root);
-            scanClause.append(normalizedHandle);
-            corporaRequest.setScanClause(scanClause.toString());
-            corporaRequest.setExtraRequestData(SRUCQL.SCAN_RESOURCE_INFO_PARAMETER, 
-                    SRUCQL.SCAN_RESOURCE_INFO_PARAMETER_DEFAULT_VALUE);
-            corporaResponse = sruScanClient.scan(corporaRequest);
-            Thread.sleep(5000);
-            SRUScanResponse response = corporaResponse.get(600, TimeUnit.SECONDS);
-            if (response != null && response.hasTerms()) {
-                for (SRUTerm term : response.getTerms()) {
-                    // don't add corpus that introduces cyclic references
-                    // as of March 2014, there are 2 such endpoints...
-                    if (cache.getCorpus(term.getValue())!= null) {
-                        LOGGER.warning("Cyclic reference in corpus " + term.getValue() + " of endpoint " + endpointUrl);
-                        continue;
-                    }
-                    Corpus c = new Corpus(institution, endpointUrl);
-                    c.setHandle(term.getValue());
-                    c.setDisplayName(term.getDisplayTerm());
-                    if (term.getNumberOfRecords() > 0) {
-                        c.setNumberOfRecords(term.getNumberOfRecords());
-                    }
-                    addExtraInfo(c, term);
-                    cache.addCorpus(c, parentCorpus);
-                    addCorpora(sruScanClient, c.getEndpointUrl(), c.getInstitution(),
-                            depth, c, cache);
-                }
-                //} else if () {
-                // TODO if diagnistics came back, try simple scan without the 
-                // SRUCQLscan.RESOURCE_INFO_PARAMETER
-            } else {
-                if (root) {
-                    // create default root corpus:
-                    Corpus c = new Corpus(institution, endpointUrl);
-                    cache.addCorpus(c);
-                }
-            }
-        } catch (TimeoutException ex) {
-            LOGGER.log(Level.SEVERE, "Timeout scanning corpora {0} at {1} {2} {3}",
-                    new String[]{Corpus.ROOT_HANDLE, endpointUrl, ex.getClass().getName(), ex.getMessage()});
-        } catch (Exception ex) {
-            LOGGER.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
-                    new String[]{Corpus.ROOT_HANDLE, endpointUrl, ex.getClass().getName(), ex.getMessage()});
-        } finally {
-            if (corporaResponse != null && !corporaResponse.isDone()) {
-                corporaResponse.cancel(true);
-            }
-        }
-    }
-
-    private String normalizeHandle(Corpus corpus, boolean root) {
-        if (root) {
-            return Corpus.ROOT_HANDLE;
-        }
-        String handle = corpus.getHandle();
-        if (Corpus.HANDLE_WITH_SPECIAL_CHARS.matcher(handle).matches()) {
-            //resourceValue = "%22" + resourceValue + "%22";
-            handle = "\"" + handle + "\"";
-        }
-        return handle;
-    }
 }

SRUAggregator/trunk/src/main/java/eu/clarin/sru/fcs/aggregator/sopt/CorpusModelLive.java

@@ -7 +7 @@
 import eu.clarin.sru.client.SRUThreadedClient;
 import eu.clarin.sru.fcs.aggregator.app.WebAppListener;
+import eu.clarin.sru.fcs.aggregator.cache.ScanCrawler;
 import static eu.clarin.sru.fcs.aggregator.sopt.Corpus.ROOT_HANDLE;
 import eu.clarin.sru.fcs.aggregator.util.SRUCQL;
@@ -136 +137 @@
 
     public void loadChildren(Treeitem item) {
+         logger.info("LOADING CHILDREN");
         // get first child
         Treeitem childTreeitem = (Treeitem) item.getTreechildren().getChildren().get(0);
@@ -144 +146 @@
         Corpus childCorpus = childNodeValue.getData();
         if (childCorpus.isTemporary()) {
+            logger.info("REPLACING TEMP WITH REAL");
             // remove temporary node
             remove(openedNodeValue, childTreeitem.getIndex());
@@ -186 +189 @@
 
     private void initRootChildren(CenterRegistryI startingPoint) {
+        logger.info("INITIALIZING ROOT CHILDREN");
         for (Institution instit : startingPoint.getCQLInstitutions()) {
             for (Endpoint endp : instit.getEndpoints()) {
-                try {
+                //try {
                     //TODO: temp for testing, this 3 lines are to be removed:
                     //if (//!endp.getUrl().contains("uni-leipzig.de") && 
@@ -196 +200 @@
                     //    continue;
                     //}
-
-                    Future<SRUScanResponse> corporaResponse = null;
-                    SRUScanRequest corporaRequest = new SRUScanRequest(endp.getUrl());
-                    StringBuilder scanClause = new StringBuilder(SRUCQL.SCAN_RESOURCE_PARAMETER);
-                    scanClause.append("=");
-                    scanClause.append(ROOT_HANDLE);
-                    corporaRequest.setScanClause(scanClause.toString());
-                    corporaRequest.setExtraRequestData(SRUCQL.SCAN_RESOURCE_INFO_PARAMETER, "true");
-                    corporaResponse = sruClient.scan(corporaRequest);
-                    SRUScanResponse response = corporaResponse.get(200, TimeUnit.SECONDS);
-                    if (response != null && response.hasTerms()) {
-                        for (SRUTerm term : response.getTerms()) {
-                            Corpus c = new Corpus(instit, endp.getUrl());
-                            c.setHandle(term.getValue());
-                            c.setDisplayName(term.getDisplayTerm());
-                            c.setNumberOfRecords(term.getNumberOfRecords());
-                            addExtraInfo(c, term);
-                            DefaultTreeNode<Corpus> rootChild = createNodeWithTempChildren(c);
-                            super.getRoot().add(rootChild);
-                        }
-                    } else {
-                        Corpus endpCorpus = new Corpus(endp.getInstitution(), endp.getUrl());
-                        DefaultTreeNode<Corpus> rootChild = createNodeWithTempChildren(endpCorpus);
+                    
+                    List<Corpus> rootCorpora = ScanCrawler.doScan(sruClient, endp.getUrl(), instit, null);
+                    for (Corpus c : rootCorpora) {
+                        DefaultTreeNode<Corpus> rootChild = createNodeWithTempChildren(c);
                         super.getRoot().add(rootChild);
                     }
-                } catch (SRUClientException ex) {
-                    logger.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
-                            new String[]{ROOT_HANDLE, endp.getUrl(), ex.getClass().getName(), ex.getMessage()});
-                } catch (InterruptedException ex) {
-                    logger.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
-                            new String[]{ROOT_HANDLE, endp.getUrl(), ex.getClass().getName(), ex.getMessage()});
-                } catch (ExecutionException ex) {
-                    logger.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
-                            new String[]{ROOT_HANDLE, endp.getUrl(), ex.getClass().getName(), ex.getMessage()});
-                } catch (TimeoutException ex) {
-                    logger.log(Level.SEVERE, "Timeout scanning corpora {0} at {1} {2} {3}",
-                            new String[]{ROOT_HANDLE, endp.getUrl(), ex.getClass().getName(), ex.getMessage()});
-                }
-            }
-
-        }
-    }
-
+//                    Future<SRUScanResponse> corporaResponse = null;
+//                    SRUScanRequest corporaRequest = new SRUScanRequest(endp.getUrl());
+//                    StringBuilder scanClause = new StringBuilder(SRUCQL.SCAN_RESOURCE_PARAMETER);
+//                    scanClause.append("=");
+//                    scanClause.append(ROOT_HANDLE);
+//                    corporaRequest.setScanClause(scanClause.toString());
+//                    corporaRequest.setExtraRequestData(SRUCQL.SCAN_RESOURCE_INFO_PARAMETER, "true");
+//                    corporaResponse = sruClient.scan(corporaRequest);
+//                    SRUScanResponse response = corporaResponse.get(200, TimeUnit.SECONDS);
+//                    if (response != null && response.hasTerms()) {
+//                        for (SRUTerm term : response.getTerms()) {
+//                            Corpus c = new Corpus(instit, endp.getUrl());
+//                            c.setHandle(term.getValue());
+//                            c.setDisplayName(term.getDisplayTerm());
+//                            c.setNumberOfRecords(term.getNumberOfRecords());
+//                            addExtraInfo(c, term);
+//                            DefaultTreeNode<Corpus> rootChild = createNodeWithTempChildren(c);
+//                            super.getRoot().add(rootChild);
+//                        }
+//                    } else {
+//                        Corpus endpCorpus = new Corpus(endp.getInstitution(), endp.getUrl());
+//                        DefaultTreeNode<Corpus> rootChild = createNodeWithTempChildren(endpCorpus);
+//                        super.getRoot().add(rootChild);
+//                    }
+//                } catch (SRUClientException ex) {
+//                    logger.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
+//                            new String[]{ROOT_HANDLE, endp.getUrl(), ex.getClass().getName(), ex.getMessage()});
+//                } catch (InterruptedException ex) {
+//                    logger.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
+//                            new String[]{ROOT_HANDLE, endp.getUrl(), ex.getClass().getName(), ex.getMessage()});
+//                } catch (ExecutionException ex) {
+//                    logger.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
+//                            new String[]{ROOT_HANDLE, endp.getUrl(), ex.getClass().getName(), ex.getMessage()});
+//                } catch (TimeoutException ex) {
+//                    logger.log(Level.SEVERE, "Timeout scanning corpora {0} at {1} {2} {3}",
+//                            new String[]{ROOT_HANDLE, endp.getUrl(), ex.getClass().getName(), ex.getMessage()});
+//                }
+            }
+
+        }
+    }
+    
     private Iterable<Corpus> getSubcorpora(Corpus corpus) {
-
-        ArrayList<Corpus> subCorpora = new ArrayList<Corpus>();
-        try {
-            SRUScanRequest corporaRequest = new SRUScanRequest(corpus.getEndpointUrl());
-            StringBuilder scanClause = new StringBuilder(SRUCQL.SCAN_RESOURCE_PARAMETER);
-            scanClause.append("=");
-            String resourceValue = corpus.getHandle();
-            if (corpus.getHandle() == null) {
-                resourceValue = ROOT_HANDLE;
-            }
-            if (Corpus.HANDLE_WITH_SPECIAL_CHARS.matcher(resourceValue).matches()) {
-                resourceValue = "%22" + resourceValue + "%22";
-            }
-            scanClause.append(resourceValue);
-            corporaRequest.setScanClause(scanClause.toString());
-            //!!!TODO request doesn't work for scan with resource handle???
-            //corporaRequest.setExtraRequestData(SRUCQLscan.RESOURCE_INFO_PARAMETER, "true");
-            Future<SRUScanResponse> corporaResponse = sruClient.scan(corporaRequest);
-            SRUScanResponse response = corporaResponse.get(200, TimeUnit.SECONDS);
-            if (response != null && response.hasTerms()) {
-            for (SRUTerm term : response.getTerms()) {
-                Corpus c = new Corpus(corpus.getInstitution(), corpus.getEndpointUrl());
-                c.setHandle(term.getValue());
-                c.setDisplayName(term.getDisplayTerm());
-                c.setNumberOfRecords(term.getNumberOfRecords());
-                addExtraInfo(c, term);
-                subCorpora.add(c);
-            }
-            System.out.println("Found " + subCorpora.size() + " children");
-        }
-        } catch (SRUClientException ex) {
-            logger.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
-                    new String[]{corpus.getHandle(), corpus.getEndpointUrl(), ex.getClass().getName(), ex.getMessage()});
-        } catch (InterruptedException ex) {
-            logger.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
-                    new String[]{corpus.getHandle(), corpus.getEndpointUrl(), ex.getClass().getName(), ex.getMessage()});
-        } catch (ExecutionException ex) {
-            logger.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
-                    new String[]{corpus.getHandle(), corpus.getEndpointUrl(), ex.getClass().getName(), ex.getMessage()});
-        } catch (TimeoutException ex) {
-            logger.log(Level.SEVERE, "Timeout scanning corpora {0} at {1} {2} {3}",
-                    new String[]{corpus.getHandle(), corpus.getEndpointUrl(), ex.getClass().getName(), ex.getMessage()});
-        }
-
-        return subCorpora;
-    }
+        //get rid of institution and endpoint? get get this info from corpus
+        return ScanCrawler.doScan(sruClient, corpus.getEndpointUrl(), corpus.getInstitution(), corpus);
+    }
+
+//    private Iterable<Corpus> getSubcorpora(Corpus corpus) {
+//        logger.info("OBTAINING SUBCORPORA");
+//        ArrayList<Corpus> subCorpora = new ArrayList<Corpus>();
+//        try {
+//            
+//            
+//            SRUScanRequest corporaRequest = new SRUScanRequest(corpus.getEndpointUrl());
+//            StringBuilder scanClause = new StringBuilder(SRUCQL.SCAN_RESOURCE_PARAMETER);
+//            scanClause.append("=");
+//            String resourceValue = corpus.getHandle();
+//            if (corpus.getHandle() == null) {
+//                resourceValue = ROOT_HANDLE;
+//            }
+//            if (Corpus.HANDLE_WITH_SPECIAL_CHARS.matcher(resourceValue).matches()) {
+//                resourceValue = "%22" + resourceValue + "%22";
+//            }
+//            scanClause.append(resourceValue);
+//            corporaRequest.setScanClause(scanClause.toString());
+//            //!!!TODO request doesn't work for scan with resource handle???
+//            //corporaRequest.setExtraRequestData(SRUCQLscan.RESOURCE_INFO_PARAMETER, "true");
+//            Future<SRUScanResponse> corporaResponse = sruClient.scan(corporaRequest);
+//            logger.info("BEFORE get " + resourceValue);
+//            SRUScanResponse response = corporaResponse.get(200, TimeUnit.SECONDS);
+//            logger.info("AFTER get " + resourceValue);
+//            if (response != null && response.hasTerms()) {
+//            for (SRUTerm term : response.getTerms()) {
+//                Corpus c = new Corpus(corpus.getInstitution(), corpus.getEndpointUrl());
+//                c.setHandle(term.getValue());
+//                c.setDisplayName(term.getDisplayTerm());
+//                c.setNumberOfRecords(term.getNumberOfRecords());
+//                addExtraInfo(c, term);
+//                subCorpora.add(c);
+//            }
+//            logger.info("OBTAINED SUBCORPORA " + subCorpora.size());
+//        }
+//        } catch (SRUClientException ex) {
+//            logger.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
+//                    new String[]{corpus.getHandle(), corpus.getEndpointUrl(), ex.getClass().getName(), ex.getMessage()});
+//        } catch (InterruptedException ex) {
+//            logger.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
+//                    new String[]{corpus.getHandle(), corpus.getEndpointUrl(), ex.getClass().getName(), ex.getMessage()});
+//        } catch (ExecutionException ex) {
+//            logger.log(Level.SEVERE, "Error accessing corpora {0} at {1} {2} {3}",
+//                    new String[]{corpus.getHandle(), corpus.getEndpointUrl(), ex.getClass().getName(), ex.getMessage()});
+//        } catch (TimeoutException ex) {
+//            logger.log(Level.SEVERE, "Timeout scanning corpora {0} at {1} {2} {3}",
+//                    new String[]{corpus.getHandle(), corpus.getEndpointUrl(), ex.getClass().getName(), ex.getMessage()});
+//        }
+//        
+//        return subCorpora;
+//    }
 
     private void addExtraInfo(Corpus c, SRUTerm term) {

SRUAggregator/trunk/src/main/java/eu/clarin/sru/fcs/aggregator/sopt/CorpusRenderer.java

@@ -2 +2 @@
 
 import eu.clarin.sru.fcs.aggregator.app.WebAppListener;
+import eu.clarin.sru.fcs.aggregator.util.ZKComp;
 import org.zkoss.zk.ui.Component;
 import org.zkoss.zk.ui.Executions;
 import org.zkoss.zk.ui.event.Event;
 import org.zkoss.zk.ui.event.EventListener;
-import org.zkoss.zk.ui.event.Events;
 import org.zkoss.zul.A;
 import org.zkoss.zul.Checkbox;
@@ -15 +15 @@
 import org.zkoss.zul.Treechildren;
 import org.zkoss.zul.Treeitem;
-import org.zkoss.zul.TreeitemRenderer;
 import org.zkoss.zul.Treerow;
 
 /**
+ * Provides methods to render a corpus tree displaying such information as
+ * corpus name, institution, description, landing page, number of records,
+ * language. Provides functionality for selecting/deselecting a resource in the
+ * rendered corpus tree.
  *
  * @author Yana Panchenko
- */
-/**
- * The structure of tree
- *
+ * 
+ * The structure of tree:
  * <pre>
  * &lt;treeitem>
@@ -36 +37 @@
  * </pre>
  */
-public class CorpusRendererCached implements TreeitemRenderer<DefaultTreeNode<Corpus>>, CorpusRendererI  {
-
-    private Languages languages;
-    private CorpusModelCached model;
-
-    public CorpusRendererCached(CorpusModelCached model) {
-        languages = (Languages) Executions.getCurrent().getDesktop().getWebApp().getAttribute(WebAppListener.LANGUAGES);
+public abstract class CorpusRenderer {
+
+    CorpusModelI model;
+    Languages languages;
+
+    public CorpusRenderer(CorpusModelI model) {
         this.model = model;
-    }
-
-    @Override
-    public void render(Treeitem treeItem, DefaultTreeNode<Corpus> treeNode, int index) throws Exception {
-
-        Treerow dataRow = new Treerow();
-        dataRow.setParent(treeItem);
-        treeItem.setValue(treeNode);
-        treeItem.setOpen(false);
-        Corpus data = treeNode.getData();
-        addCorpusDataIntoRow(dataRow, data);
-
-        treeItem.addEventListener(Events.ON_OPEN, new EventListener<Event>() {
-            @Override
-            public void onEvent(Event event) throws Exception {
-
-                Treeitem openedTreeitem = (Treeitem) event.getTarget();
-                //if (openedTreeitem.isOpen()) {
-                //    if (model.hasChildren(openedTreeitem)) {
-                //        model.loadChildren(openedTreeitem);
-                //        openedTreeitem.setOpen(model.hasChildren(openedTreeitem));
-                //    } else {
-                //        openedTreeitem.setOpen(false);
-                //    }
-                //}
-
-                if (model.isCorpusSelected((DefaultTreeNode<Corpus>) openedTreeitem.getValue())) {
-                    selectChildren(openedTreeitem);
-                }
-            }
-        });
-
-    }
-
-    //TODO check if it's working only on open children???
-    public void selectChildren(Treeitem openedTreeitem) {
-
-        for (Component comp : openedTreeitem.getChildren()) {
-            if (comp instanceof Treechildren) {
-                Treechildren item = (Treechildren) comp;
-                for (Treeitem childItem : item.getItems()) {
-                    updateItem(childItem, true);
-                }
-                break;
-            }
-        }
-
+        this.languages = (Languages) Executions.getCurrent().getDesktop().getWebApp().getAttribute(WebAppListener.LANGUAGES);
     }
 
@@ -114 +68 @@
     }
 
-    private void updateParentItem(Treeitem item, boolean checked) {
-
-        // if item becomes unselected, unselect it parent
-        if (!checked) {
-            Treeitem parent = item.getParentItem();
-            if (parent != null) {
-                for (Component comp : parent.getChildren()) {
-                    // update the item row
-                    if (comp instanceof Treerow) {
-                        Treerow row = (Treerow) comp;
-                        Checkbox checkbox = (Checkbox) row.getFirstChild().getFirstChild();
-                        DefaultTreeNode<Corpus> node = (DefaultTreeNode<Corpus>) parent.getValue();
-                        checkbox.setChecked(checked);
-                        toggleCorpusCheckbox(node.getData(), checkbox, row);
-                        updateParentItem(parent, checked);
-                        break;
-                    }
-                }
-
-            }
-        }
-    }
-
-    private void toggleCorpusCheckbox(Corpus corpus, Checkbox checkbox, Treerow row) {
-        if (checkbox.isChecked()) {
-            row.setZclass("z-treerow z-treerow-seld");
-            model.addToSelected(corpus);
-        } else {
-            row.setZclass("z-treerow");
-            model.removeFromSelected(corpus);
-        }
-    }
-
-    private void addCorpusDataIntoRow(Treerow dataRow, Corpus data) {
+    public void selectChildren(Treeitem openedTreeitem) {
+
+        for (Component comp : openedTreeitem.getChildren()) {
+            if (comp instanceof Treechildren) {
+                Treechildren item = (Treechildren) comp;
+                for (Treeitem childItem : item.getItems()) {
+                    updateItem(childItem, true);
+                }
+                break;
+            }
+        }
+    }
+
+    protected void addCorpusDataIntoRow(Treerow dataRow, Corpus data) {
         Treecell cell1 = createCellForSelectCorpus(data, dataRow);
         dataRow.appendChild(cell1);
@@ -194 +128 @@
         Treeitem eventItem = (Treeitem) dataRow.getParent();
         DefaultTreeNode<Corpus> eventNode = (DefaultTreeNode<Corpus>) eventItem.getValue();
-        if (model.isSelected(eventNode)) {
+        if (model.isCorpusSelected(eventNode)) {
             checkbox.setChecked(true);
             toggleCorpusCheckbox(eventNode.getData(), checkbox, dataRow);
@@ -226 +160 @@
         return cell;
     }
-    
+
     private Treecell createCellForCorpusNumberOfRecords(Corpus data) {
         Treecell cell = new Treecell();
@@ -245 +179 @@
             label.setParent(cell);
         } else {
-
-            A link = new A();
-            link.setTarget("_blank");
-            link.setHref(data.getLandingPage());
-            link.setImage("img/go-home.png");
+            A link = ZKComp.createCorpusHomeLink(data.getLandingPage());
             link.setParent(cell);
         }
@@ -278 +208 @@
 //        return cell;
 //    }
-    
-     private Treecell createCellForCorpusLanguage(Corpus data) {
+    private Treecell createCellForCorpusLanguage(Corpus data) {
         Treecell cell = new Treecell();
         if (data.getLanguages().isEmpty()) {
@@ -309 +238 @@
             label.setParent(langPanel);
             langPanel.setParent(cell);
-            
+
         }
         return cell;
@@ -319 +248 @@
             int okLength = 125;
             if (data.getDescription().length() > okLength) {
-                String descrStart = // data.getDescription().substring(0, 35) + " ...";
-                        getDisplayedText(data.getDescription(), okLength);
+                String descrStart = getDisplayedText(data.getDescription(), okLength);
                 Label label = new Label(descrStart);
                 label.setTooltiptext(data.getDescription());
                 label.setParent(cell);
             } else {
-            Label label = new Label(data.getDescription());
-            label.setMultiline(true);
-            label.setParent(cell);
-            
-            //ideally should contain html5 <details> element, but only
-            //chrome supports it as of now:
-            //<details>
-            //<summary>First sentence</summary>
-            //Rest sentences.
-            //</details>
-            
-            
+                Label label = new Label(data.getDescription());
+                label.setMultiline(true);
+                label.setParent(cell);
+
+                //ideally should contain html5 <details> element, but only
+                //chrome supports it as of now:
+                //<details>
+                //<summary>First sentence</summary>
+                //Rest sentences.
+                //</details>
+
+
             }
         } else {
@@ -353 +281 @@
         return description.substring(0, dotIndex) + "...";
     }
+
+    private void toggleCorpusCheckbox(Corpus corpus, Checkbox checkbox, Treerow row) {
+        if (checkbox.isChecked()) {
+            row.setZclass("z-treerow z-treerow-seld");
+            model.addToSelected(corpus);
+        } else {
+            row.setZclass("z-treerow");
+            model.removeFromSelected(corpus);
+        }
+    }
+
+    private void updateParentItem(Treeitem item, boolean checked) {
+
+        // if item becomes unselected, unselect it parent
+        if (!checked) {
+            Treeitem parent = item.getParentItem();
+            if (parent != null) {
+                for (Component comp : parent.getChildren()) {
+                    // update the item row
+                    if (comp instanceof Treerow) {
+                        Treerow row = (Treerow) comp;
+                        Checkbox checkbox = (Checkbox) row.getFirstChild().getFirstChild();
+                        DefaultTreeNode<Corpus> node = (DefaultTreeNode<Corpus>) parent.getValue();
+                        checkbox.setChecked(checked);
+                        toggleCorpusCheckbox(node.getData(), checkbox, row);
+                        updateParentItem(parent, checked);
+                        break;
+                    }
+                }
+
+            }
+        }
+    }
 }

SRUAggregator/trunk/src/main/java/eu/clarin/sru/fcs/aggregator/sopt/CorpusRendererCached.java

@@ -1 +1 @@
 package eu.clarin.sru.fcs.aggregator.sopt;
 
-import eu.clarin.sru.fcs.aggregator.app.WebAppListener;
-import org.zkoss.zk.ui.Component;
-import org.zkoss.zk.ui.Executions;
 import org.zkoss.zk.ui.event.Event;
 import org.zkoss.zk.ui.event.EventListener;
 import org.zkoss.zk.ui.event.Events;
-import org.zkoss.zul.A;
-import org.zkoss.zul.Checkbox;
 import org.zkoss.zul.DefaultTreeNode;
-import org.zkoss.zul.Groupbox;
-import org.zkoss.zul.Label;
-import org.zkoss.zul.Treecell;
-import org.zkoss.zul.Treechildren;
 import org.zkoss.zul.Treeitem;
 import org.zkoss.zul.TreeitemRenderer;
@@ -19 +10 @@
 
 /**
+ * Renders treeitem of corpora tree from corpora cache. I.e. corpora sub-resources
+ * are pre-loaded and the information about them is taken from CorpusModelCached,
+ * which accesses the cache. 
  *
  * @author Yana Panchenko
+ *
  */
-/**
- * The structure of tree
- *
- * <pre>
- * &lt;treeitem>
- *   &lt;treerow>
- *     &lt;treecell>...&lt;/treecell>
- *   &lt;/treerow>
- *   &lt;treechildren>
- *     &lt;treeitem>...&lt;/treeitem>
- *   &lt;/treechildren>
- * &lt;/treeitem>
- * </pre>
- */
-public class CorpusRendererCached implements TreeitemRenderer<DefaultTreeNode<Corpus>>, CorpusRendererI  {
-
-    private Languages languages;
-    private CorpusModelCached model;
+public class CorpusRendererCached extends CorpusRenderer
+        implements TreeitemRenderer<DefaultTreeNode<Corpus>> {
 
     public CorpusRendererCached(CorpusModelCached model) {
-        languages = (Languages) Executions.getCurrent().getDesktop().getWebApp().getAttribute(WebAppListener.LANGUAGES);
-        this.model = model;
+        super(model);
     }
 
@@ -59 +37 @@
             @Override
             public void onEvent(Event event) throws Exception {
-
                 Treeitem openedTreeitem = (Treeitem) event.getTarget();
-                //if (openedTreeitem.isOpen()) {
-                //    if (model.hasChildren(openedTreeitem)) {
-                //        model.loadChildren(openedTreeitem);
-                //        openedTreeitem.setOpen(model.hasChildren(openedTreeitem));
-                //    } else {
-                //        openedTreeitem.setOpen(false);
-                //    }
-                //}
-
                 if (model.isCorpusSelected((DefaultTreeNode<Corpus>) openedTreeitem.getValue())) {
                     selectChildren(openedTreeitem);
@@ -78 +46 @@
     }
 
-    //TODO check if it's working only on open children???
-    public void selectChildren(Treeitem openedTreeitem) {
-
-        for (Component comp : openedTreeitem.getChildren()) {
-            if (comp instanceof Treechildren) {
-                Treechildren item = (Treechildren) comp;
-                for (Treeitem childItem : item.getItems()) {
-                    updateItem(childItem, true);
-                }
-                break;
-            }
-        }
-
-    }
-
-    public void updateItem(Treeitem item, boolean select) {
-
-        for (Component comp : item.getChildren()) {
-            // update the item row
-            if (comp instanceof Treerow) {
-                Treerow row = (Treerow) comp;
-                Checkbox checkbox = (Checkbox) row.getFirstChild().getFirstChild();
-                DefaultTreeNode<Corpus> node = (DefaultTreeNode<Corpus>) item.getValue();
-                checkbox.setChecked(select);
-                toggleCorpusCheckbox(node.getData(), checkbox, row);
-                // update the item children
-            } else if (comp instanceof Treechildren) {
-                Treechildren children = (Treechildren) comp;
-                for (Treeitem childItem : children.getItems()) {
-                    updateItem(childItem, select);
-                }
-            }
-        }
-
-    }
-
-    private void updateParentItem(Treeitem item, boolean checked) {
-
-        // if item becomes unselected, unselect it parent
-        if (!checked) {
-            Treeitem parent = item.getParentItem();
-            if (parent != null) {
-                for (Component comp : parent.getChildren()) {
-                    // update the item row
-                    if (comp instanceof Treerow) {
-                        Treerow row = (Treerow) comp;
-                        Checkbox checkbox = (Checkbox) row.getFirstChild().getFirstChild();
-                        DefaultTreeNode<Corpus> node = (DefaultTreeNode<Corpus>) parent.getValue();
-                        checkbox.setChecked(checked);
-                        toggleCorpusCheckbox(node.getData(), checkbox, row);
-                        updateParentItem(parent, checked);
-                        break;
-                    }
-                }
-
-            }
-        }
-    }
-
-    private void toggleCorpusCheckbox(Corpus corpus, Checkbox checkbox, Treerow row) {
-        if (checkbox.isChecked()) {
-            row.setZclass("z-treerow z-treerow-seld");
-            model.addToSelected(corpus);
-        } else {
-            row.setZclass("z-treerow");
-            model.removeFromSelected(corpus);
-        }
-    }
-
-    private void addCorpusDataIntoRow(Treerow dataRow, Corpus data) {
-        Treecell cell1 = createCellForSelectCorpus(data, dataRow);
-        dataRow.appendChild(cell1);
-        Treecell cell4 = createCellForHome(data);
-        dataRow.appendChild(cell4);
-        Treecell cell2 = createCellForCorpusLanguage(data);
-        dataRow.appendChild(cell2);
-        Treecell cell6 = createCellForCorpusNumberOfRecords(data);
-        dataRow.appendChild(cell6);
-        Treecell cell5 = createCellForCorpusInstitution(data);
-        dataRow.appendChild(cell5);
-        Treecell cell3 = createCellForCorpusDescription(data);
-        dataRow.appendChild(cell3);
-    }
-
-    private Treecell createCellForSelectCorpus(Corpus data, Treerow dataRow) {
-        Treecell cell = new Treecell();
-        Checkbox corpusCheckbox = createCheckboxForSelectCorpus(dataRow);
-        corpusCheckbox.setParent(cell);
-        if (data.getDisplayName() == null) {
-            Label corpusLabel = new Label("");
-            corpusLabel.setParent(cell);
-        } else {
-            String name = data.getDisplayName();
-            // TODO: this is temp, before some endpoint
-            // fix long names for their resources, that look 
-            // more like a description
-            if (name.length() > 50) {
-                int firstWordEnd = name.indexOf(" ");
-                if (firstWordEnd > 0) {
-                    data.setDisplayName(name.substring(0, firstWordEnd));
-                    if (data.getDescription() == null) {
-                        data.setDescription(name);
-                    }
-                }
-            }
-            Label corpusLabel = new Label(data.getDisplayName());
-            corpusLabel.setParent(cell);
-        }
-        return cell;
-    }
-
-    private Checkbox createCheckboxForSelectCorpus(Treerow dataRow) {
-        Checkbox checkbox = new Checkbox();
-        checkbox.setValue(dataRow);
-        Treeitem eventItem = (Treeitem) dataRow.getParent();
-        DefaultTreeNode<Corpus> eventNode = (DefaultTreeNode<Corpus>) eventItem.getValue();
-        if (model.isSelected(eventNode)) {
-            checkbox.setChecked(true);
-            toggleCorpusCheckbox(eventNode.getData(), checkbox, dataRow);
-        }
-        checkbox.addEventListener("onCheck", new EventListener() {
-            @Override
-            public void onEvent(Event event) throws Exception {
-                Checkbox eventCheckbox = (Checkbox) event.getTarget();
-                Treerow eventRow = (Treerow) eventCheckbox.getValue();
-                Treeitem eventItem = (Treeitem) eventRow.getParent();
-                DefaultTreeNode<Corpus> eventNode = (DefaultTreeNode<Corpus>) eventItem.getValue();
-                toggleCorpusCheckbox(eventNode.getData(), eventCheckbox, eventRow);
-                updateParentItem(eventItem, eventCheckbox.isChecked());
-                for (Treeitem item : eventItem.getTreechildren().getItems()) {
-                    updateItem(item, eventCheckbox.isChecked());
-                }
-            }
-        });
-        return checkbox;
-    }
-
-    private Treecell createCellForCorpusInstitution(Corpus data) {
-        Treecell cell = new Treecell();
-        if (data.getInstitution() == null) {
-            Label label = new Label("");
-            label.setParent(cell);
-        } else {
-            Label label = new Label(data.getInstitution().getName());
-            label.setParent(cell);
-        }
-        return cell;
-    }
-    
-    private Treecell createCellForCorpusNumberOfRecords(Corpus data) {
-        Treecell cell = new Treecell();
-        if (data.getNumberOfRecords() == null) {
-            Label label = new Label("");
-            label.setParent(cell);
-        } else {
-            Label label = new Label(data.getNumberOfRecords().toString());
-            label.setParent(cell);
-        }
-        return cell;
-    }
-
-    private Treecell createCellForHome(Corpus data) {
-        Treecell cell = new Treecell();
-        if (data.getLandingPage() == null) {
-            Label label = new Label("");
-            label.setParent(cell);
-        } else {
-
-            A link = new A();
-            link.setTarget("_blank");
-            link.setHref(data.getLandingPage());
-            link.setImage("img/go-home.png");
-            link.setParent(cell);
-        }
-        return cell;
-    }
-
-//    private Treecell createCellForCorpusLanguage(Corpus data) {
-//        Treecell cell = new Treecell();
-//        if (!data.getLanguages().isEmpty()) {
-//            StringBuilder langs = new StringBuilder();
-//            for (String lang : data.getLanguages()) {
-//                String langName = languages.nameForCode(lang);
-//                if (langName != null) {
-//                    langs.append(langName);
-//                } else {
-//                    langs.append(lang);
-//                }
-//                langs.append("\n ");
-//
-//            }
-//            Label label = new Label(langs.toString());
-//            label.setMultiline(true);
-//            label.setParent(cell);
-//        } else {
-//            Label label = new Label("");
-//            label.setParent(cell);
-//        }
-//        return cell;
-//    }
-    
-     private Treecell createCellForCorpusLanguage(Corpus data) {
-        Treecell cell = new Treecell();
-        if (data.getLanguages().isEmpty()) {
-            Label label = new Label("");
-            label.setParent(cell);
-        } else if (data.getLanguages().size() == 1) {
-            Label label = new Label(languages.nameForCode(data.getLanguages().iterator().next()));
-            label.setParent(cell);
-        } else {
-            Groupbox langPanel = new Groupbox();
-            langPanel.setMold("3d");
-            langPanel.setContentStyle("border:0");
-            langPanel.setClosable(true);
-            langPanel.setOpen(false);
-            langPanel.setTitle("multiple");
-            StringBuilder langs = new StringBuilder();
-            for (String lang : data.getLanguages()) {
-                String langName = languages.nameForCode(lang);
-                if (langName != null) {
-                    langs.append(langName);
-                } else {
-                    langs.append(lang);
-                }
-                langs.append("\n ");
-
-            }
-            Label label = new Label(langs.toString());
-            label.setMultiline(true);
-            label.setParent(langPanel);
-            langPanel.setParent(cell);
-            
-        }
-        return cell;
-    }
-
-    private Treecell createCellForCorpusDescription(Corpus data) {
-        Treecell cell = new Treecell();
-        if (data.getDescription() != null) {
-            int okLength = 125;
-            if (data.getDescription().length() > okLength) {
-                String descrStart = // data.getDescription().substring(0, 35) + " ...";
-                        getDisplayedText(data.getDescription(), okLength);
-                Label label = new Label(descrStart);
-                label.setTooltiptext(data.getDescription());
-                label.setParent(cell);
-            } else {
-            Label label = new Label(data.getDescription());
-            label.setMultiline(true);
-            label.setParent(cell);
-            
-            //ideally should contain html5 <details> element, but only
-            //chrome supports it as of now:
-            //<details>
-            //<summary>First sentence</summary>
-            //Rest sentences.
-            //</details>
-            
-            
-            }
-        } else {
-            Label label = new Label("");
-            label.setParent(cell);
-        }
-
-        return cell;
-    }
-
-    private String getDisplayedText(String description, int okLength) {
-        int dotIndex = description.indexOf(" ", okLength);
-        if (dotIndex < 0 || dotIndex > okLength + 50) {
-            dotIndex = okLength;
-        }
-        return description.substring(0, dotIndex) + "...";
-    }
 }

SRUAggregator/trunk/src/main/java/eu/clarin/sru/fcs/aggregator/sopt/CorpusRendererLive.java

@@ -1 +1 @@
 package eu.clarin.sru.fcs.aggregator.sopt;
 
-import eu.clarin.sru.fcs.aggregator.app.WebAppListener;
-import org.zkoss.zk.ui.Component;
-import org.zkoss.zk.ui.Executions;
 import org.zkoss.zk.ui.event.Event;
 import org.zkoss.zk.ui.event.EventListener;
 import org.zkoss.zk.ui.event.Events;
-import org.zkoss.zul.A;
-import org.zkoss.zul.Checkbox;
 import org.zkoss.zul.DefaultTreeNode;
-import org.zkoss.zul.Label;
-import org.zkoss.zul.Treecell;
-import org.zkoss.zul.Treechildren;
 import org.zkoss.zul.Treeitem;
 import org.zkoss.zul.TreeitemRenderer;
@@ -18 +10 @@
 
 /**
+ * Renders treeitem of corpora tree in a 'live' mode. I.e. corpora sub-resources
+ * are loaded on demand. It is necessary that CorpusModelLive is used with this
+ * renderer.
+ *
+ * TODO: maybe remove the loaded sub-resources from the tree when they are
+ * closed by the user.
  *
  * @author Yana Panchenko
  */
-/**
- * The structure of tree
- *
- * <pre>
- * &lt;treeitem>
- *   &lt;treerow>
- *     &lt;treecell>...&lt;/treecell>
- *   &lt;/treerow>
- *   &lt;treechildren>
- *     &lt;treeitem>...&lt;/treeitem>
- *   &lt;/treechildren>
- * &lt;/treeitem>
- * </pre>
- */
-public class CorpusRendererLive implements TreeitemRenderer<DefaultTreeNode<Corpus>>, CorpusRendererI {
-
-    private Languages languages;
-    private CorpusModelLive model;
+public class CorpusRendererLive extends CorpusRenderer
+        implements TreeitemRenderer<DefaultTreeNode<Corpus>> {
 
     public CorpusRendererLive(CorpusModelLive model) {
-        languages = (Languages) Executions.getCurrent().getDesktop().getWebApp().getAttribute(WebAppListener.LANGUAGES);
-        this.model = model;
+        super(model);
     }
 
@@ -66 +47 @@
                 if (openedTreeitem.isOpen()) {
                     if (model.hasChildren(openedTreeitem)) {
-                        model.loadChildren(openedTreeitem);
+                        ((CorpusModelLive) model).loadChildren(openedTreeitem);
                         openedTreeitem.setOpen(model.hasChildren(openedTreeitem));
                     } else {
@@ -80 +61 @@
 
     }
-
-    //TODO check if it's working only on open children???
-    public void selectChildren(Treeitem openedTreeitem) {
-
-        for (Component comp : openedTreeitem.getChildren()) {
-            if (comp instanceof Treechildren) {
-                Treechildren item = (Treechildren) comp;
-                for (Treeitem childItem : item.getItems()) {
-                    updateItem(childItem, true);
-                }
-                break;
-            }
-        }
-
-    }
-
-    public void updateItem(Treeitem item, boolean select) {
-
-        for (Component comp : item.getChildren()) {
-            // update the item row
-            if (comp instanceof Treerow) {
-                Treerow row = (Treerow) comp;
-                Checkbox checkbox = (Checkbox) row.getFirstChild().getFirstChild();
-                DefaultTreeNode<Corpus> node = (DefaultTreeNode<Corpus>) item.getValue();
-                checkbox.setChecked(select);
-                toggleCorpusCheckbox(node.getData(), checkbox, row);
-                // update the item children
-            } else if (comp instanceof Treechildren) {
-                Treechildren children = (Treechildren) comp;
-                for (Treeitem childItem : children.getItems()) {
-                    updateItem(childItem, select);
-                }
-            }
-        }
-
-    }
-
-    private void updateParentItem(Treeitem item, boolean checked) {
-
-        // if item becomes unselected, unselect it parent
-        if (!checked) {
-            Treeitem parent = item.getParentItem();
-            if (parent != null) {
-                for (Component comp : parent.getChildren()) {
-                    // update the item row
-                    if (comp instanceof Treerow) {
-                        Treerow row = (Treerow) comp;
-                        Checkbox checkbox = (Checkbox) row.getFirstChild().getFirstChild();
-                        DefaultTreeNode<Corpus> node = (DefaultTreeNode<Corpus>) parent.getValue();
-                        checkbox.setChecked(checked);
-                        toggleCorpusCheckbox(node.getData(), checkbox, row);
-                        updateParentItem(parent, checked);
-                        break;
-                    }
-                }
-
-            }
-        }
-    }
-
-    private void toggleCorpusCheckbox(Corpus corpus, Checkbox checkbox, Treerow row) {
-        if (checkbox.isChecked()) {
-            row.setZclass("z-treerow z-treerow-seld");
-            model.addToSelected(corpus);
-        } else {
-            row.setZclass("z-treerow");
-            model.removeFromSelected(corpus);
-        }
-    }
-
-    private void addCorpusDataIntoRow(Treerow dataRow, Corpus data) {
-        Treecell cell1 = createCellForSelectCorpus(data, dataRow);
-        dataRow.appendChild(cell1);
-        Treecell cell4 = createCellForHome(data);
-        dataRow.appendChild(cell4);
-        Treecell cell2 = createCellForCorpusLanguage(data);
-        dataRow.appendChild(cell2);
-        Treecell cell5 = createCellForCorpusInstitution(data);
-        dataRow.appendChild(cell5);
-        Treecell cell3 = createCellForCorpusDescription(data);
-        dataRow.appendChild(cell3);
-    }
-
-    private Treecell createCellForSelectCorpus(Corpus data, Treerow dataRow) {
-        Treecell cell = new Treecell();
-        Checkbox corpusCheckbox = createCheckboxForSelectCorpus(dataRow);
-        corpusCheckbox.setParent(cell);
-        if (data.getDisplayName() == null) {
-            Label corpusLabel = new Label("");
-            corpusLabel.setParent(cell);
-        } else {
-            Label corpusLabel = new Label(data.getDisplayName());
-            corpusLabel.setParent(cell);
-        }
-        return cell;
-    }
-
-    private Checkbox createCheckboxForSelectCorpus(Treerow dataRow) {
-        Checkbox checkbox = new Checkbox();
-        checkbox.setValue(dataRow);
-        Treeitem eventItem = (Treeitem) dataRow.getParent();
-        DefaultTreeNode<Corpus> eventNode = (DefaultTreeNode<Corpus>) eventItem.getValue();
-        if (model.isSelected(eventNode)) {
-            checkbox.setChecked(true);
-            toggleCorpusCheckbox(eventNode.getData(), checkbox, dataRow);
-        }
-        checkbox.addEventListener("onCheck", new EventListener() {
-            @Override
-            public void onEvent(Event event) throws Exception {
-                Checkbox eventCheckbox = (Checkbox) event.getTarget();
-                Treerow eventRow = (Treerow) eventCheckbox.getValue();
-                Treeitem eventItem = (Treeitem) eventRow.getParent();
-                DefaultTreeNode<Corpus> eventNode = (DefaultTreeNode<Corpus>) eventItem.getValue();
-                toggleCorpusCheckbox(eventNode.getData(), eventCheckbox, eventRow);
-                updateParentItem(eventItem, eventCheckbox.isChecked());
-                for (Treeitem item : eventItem.getTreechildren().getItems()) {
-                    updateItem(item, eventCheckbox.isChecked());
-                }
-            }
-        });
-        return checkbox;
-    }
-
-    private Treecell createCellForCorpusInstitution(Corpus data) {
-        Treecell cell = new Treecell();
-        if (data.getInstitution() == null) {
-            Label label = new Label("");
-            label.setParent(cell);
-        } else {
-            Label label = new Label(data.getInstitution().getName());
-            label.setParent(cell);
-        }
-        return cell;
-    }
-
-    private Treecell createCellForHome(Corpus data) {
-        Treecell cell = new Treecell();
-        if (data.getLandingPage() == null) {
-            Label label = new Label("");
-            label.setParent(cell);
-        } else {
-
-            A link = new A();
-            link.setTarget("_blank");
-            link.setHref(data.getLandingPage());
-            link.setImage("img/go-home.png");
-            link.setParent(cell);
-        }
-        return cell;
-    }
-
-    private Treecell createCellForCorpusLanguage(Corpus data) {
-        Treecell cell = new Treecell();
-        if (!data.getLanguages().isEmpty()) {
-            StringBuilder langs = new StringBuilder();
-            for (String lang : data.getLanguages()) {
-                String langName = languages.nameForCode(lang);
-                if (langName != null) {
-                    langs.append(langName);
-                } else {
-                    langs.append(lang);
-                }
-                langs.append("\n ");
-
-            }
-            Label label = new Label(langs.toString());
-            label.setMultiline(true);
-            label.setParent(cell);
-        } else {
-            Label label = new Label("");
-            label.setParent(cell);
-        }
-        return cell;
-    }
-
-    private Treecell createCellForCorpusDescription(Corpus data) {
-        Treecell cell = new Treecell();
-        if (data.getDescription() != null) {
-            //if (data.getDescription().length() > 40) {
-            //    String descrStart = data.getDescription().substring(0, 35) + " ...";
-            //    Label label = new Label(descrStart);
-            //    label.setTooltiptext(data.getDescription());
-            //    label.setParent(cell);
-            //} else {
-            Label label = new Label(data.getDescription());
-            label.setMultiline(true);
-            label.setParent(cell);
-            //}
-        } else {
-            Label label = new Label("");
-            label.setParent(cell);
-        }
-
-        return cell;
-    }
 }

SRUAggregator/trunk/src/main/java/eu/clarin/sru/fcs/aggregator/sopt/Languages.java

@@ -17 +17 @@
 public class Languages {
     
-    private Map<String,String> code2Name = new HashMap<String,String>();
-    private Map<String,String> code22Code = new HashMap<String,String>();
+    
+    private Map<String,Language> code2Lang = new HashMap<String,Language>();
+    
     public static final String LANGUAGES_FILE_PATH = "/lang/ISO-639-2_utf-8.txt";
     public static final String LANGUAGES_FILE_ENCODING = "UTF-8";
@@ -27 +28 @@
     }
     
-    public String nameForCode(String code) {
-        return this.code2Name.get(code);
+    /**
+     * Gets language by its ISO 639 language code.
+     * 
+     * @param code ISO 639/1, 639/2T or 639/2B language code
+     * @return language
+     */
+    public Language langForCode(String code) {
+        return this.code2Lang.get(code);
     }
     
-    public String code2ForCode(String code) {
-        return this.code22Code.get(code);
+    /**
+     * Gets language name by the ISO 639 language code.
+     * 
+     * @param code ISO 639/1, 639/2T or 639/2B language code
+     * @return language name in English
+     */
+    public String nameForCode(String code) {
+        if (this.code2Lang.containsKey(code)) {
+            return this.code2Lang.get(code).getNameEn();
+        }
+        return code;
     }
-    
+
+    /**
+     * Gets all known to it ISO 639/1, 639/2T and 639/2B language codes.
+     * 
+     * @return language codes
+     */
     public Set<String> getCodes() {
-        return this.code2Name.keySet();
+        return this.code2Lang.keySet();
     }
 
@@ -53 +74 @@
                     String alpha2 = splitted[2];
                     String enName = splitted[3];
+                    Language lang = new Language(alpha2, alpha3t, alpha3b, enName);
                     if (!alpha3b.isEmpty()) {
-                        this.code2Name.put(alpha3b, enName);
-                        this.code22Code.put(alpha3b, alpha2);
+                        this.code2Lang.put(alpha3b, lang);
                     }
                     if (!alpha3t.isEmpty()) {
-                        this.code2Name.put(alpha3t, enName);
-                        this.code22Code.put(alpha3t, alpha2);
+                        this.code2Lang.put(alpha3t, lang);
                     }
                     if (!alpha2.isEmpty()) {
-                        this.code2Name.put(alpha2, enName);
-                        this.code22Code.put(alpha2, alpha2);
+                        this.code2Lang.put(alpha2, lang);
                     }
                 }

SRUAggregator/trunk/src/main/java/eu/clarin/sru/fcs/aggregator/sresult/SearchResult.java

@@ -21 +21 @@
     private Corpus corpus;
     private String searchString;
+    private int startRecord;
+    private int endRecord;
     private Future<SRUSearchRetrieveResponse> futureResponse;
     private SRUSearchRetrieveResponse response;
@@ -36 +38 @@
     }
     
-    public SearchResult(Corpus corpus, String searchString) {
+    public SearchResult(Corpus corpus, String searchString, int startRecord , int endRecord) {
         this.corpus = corpus;
         this.searchString = searchString;
+        this.startRecord = startRecord;
+        this.endRecord = endRecord;
+    }
+
+    public int getStartRecord() {
+        return startRecord;
+    }
+
+    public int getEndRecord() {
+        return endRecord;
     }
 

SRUAggregator/trunk/src/main/java/eu/clarin/sru/fcs/aggregator/sresult/SearchResultRecordRenderer.java

@@ -7 +7 @@
 import eu.clarin.sru.client.fcs.DataViewKWIC;
 import eu.clarin.sru.client.fcs.Resource;
-import eu.clarin.sru.fcs.aggregator.sresult.SearchResult;
-import java.util.ArrayList;
 import java.util.List;
 import java.util.logging.Level;
@@ -17 +15 @@
 import org.zkoss.zul.A;
 import org.zkoss.zul.Cell;
-import org.zkoss.zul.Checkbox;
-import org.zkoss.zul.DefaultTreeNode;
 import org.zkoss.zul.Hlayout;
 import org.zkoss.zul.Image;
 import org.zkoss.zul.Label;
-import org.zkoss.zul.Popup;
 import org.zkoss.zul.Row;
 import org.zkoss.zul.RowRenderer;
-import org.zkoss.zul.Treeitem;
-import org.zkoss.zul.Treerow;
 import org.zkoss.zul.Vlayout;
 import org.zkoss.zul.Window;
@@ -194 +187 @@
     }
     
+    
+    
 }

SRUAggregator/trunk/src/main/webapp/about-fcs.html

@@ -22 +22 @@
         <p style="text-align:left">
         Currently, 11 centers from Germany, Netherlands and Czech Republic make more than 40 resources available 
-        to the linguistic researches via the common interface, 
+        to the linguistic researchers via the common interface, 
         and this number is growing. 
         </p>

SRUAggregator/trunk/src/main/webapp/help-fcs.html

@@ -14 +14 @@
         <h3>Search Options - adjusting search criteria</h3>
         <p style="text-align:left">To select specific corpora based on their name or language and to specify number of search results 
-            per corpus per page, click on the 'Search options' link. Here, you can filter resources based on the language, 
-            select specific resources, set the number of hits per corpus per page.</p>
+            (hits) per corpus per page, click on the 'Search options' link. Here, you can filter resources based on the language, 
+            select specific resources, set the maximum number of hits.</p>
         <br/>
         <h3>Search Results - inspecting search results</h3>
@@ -23 +23 @@
             go to the menu and select either 'Export to Personal Workspace', 'Download' or 'Use WebLicht' menu item. 
             This menu appears only after all the results on the page have been loaded. 
-            To get the next N records from each corpus, click the 'next' arrow at the bottom of 'Search results' page.
+            To get the next hits from each corpus, click the 'next' arrow at the bottom of 'Search results' page.
         </p>
         <br/>
@@ -30 +30 @@
             <a target="_blank" href="http://weblicht.sfs.uni-tuebingen.de/weblichtwiki/index.php/FCS_Aggregator">
                 Aggegator wiki page</a>.
-            If you still cannot not find an answer to your question, or if want to send a feedback,
+            If you still cannot find an answer to your question, or if want to send a feedback,
             you can write to Clarin-D helpdesk:
         </p>

SRUAggregator/trunk/src/main/webapp/index.zul

@@ -100 +100 @@
                             <progressmeter visible="false" id="pMeter" value="0" width="300px"/>
                             <hlayout>
-                                <a id="prevButton" visible="false" image="img/arrow_left_blue.png"
+                                <a id="prevButton" visible="false" image="img/arrow_left_blue.png" tooltip="tooltipPrev"
                                     hoverImage="img/arrow_left_green.png" style="margin-right:20px;"/>
-                                <a id="nextButton" visible="false" image="img/arrow_right_blue.png"
+                                <a id="nextButton" visible="false" image="img/arrow_right_blue.png" tooltip="tooltipNext"
                                     hoverImage="img/arrow_right_green.png" dir="reverse"
                                     style="margin-left:20px;"/>
@@ -159 +159 @@
             </grid>
         </popup>
+        
+        <popup id="tooltipPrev"><label id="tooltipPrevText" value="previous N hits from each resource" style="font-weight:bold"/></popup>
+        <popup id="tooltipNext"><label id="tooltipNextText" value="next N hits from each resource" style="font-weight:bold"/></popup>
 
     </window>