Elastic Indexer#

The Elastic Indexer is a command line task runner that synchronizes the Colectica Repository items with an Elasticsearch index. The Colectica Repository web application only reads from Elasticsearch; the indexer is the only component that writes to the index, so it must be scheduled to run regularly.

Prerequisites#

The Elastic Indexer requires .NET 10 to be installed.

Download and install the .NET Runtime 10.0.x from https://dotnet.microsoft.com/download/dotnet/10.0

Deploy the Elastic Indexer#

First, make sure Colectica Repository and Colectica Portal are installed.
Populate Colectica Repository with metadata.
Download the Elastic Indexer package. This will be named ColecticaElasticIndexer-version-windows.zip or ColecticaElasticIndexer-version-linux.zip and should be included with your Colectica Repository delivery. A Dockerfile is included for container deployments.
On Windows, before extracting the Elastic Indexer package, make sure Windows does not have the file blocked. To check this:
1. In Windows Explorer, right click the Zip file and choose Properties.
2. Near the bottom of the Properties window, in the Security area, see if there is a checkbox labeled Unblock.
3. If there is an Unblock checkbox, check the box and click OK.
4. If there is no such checkbox, proceed to the next step.
Extract the contents of the Elastic Indexer package. In this documentation, the directory to which you extract the file will be referred to as ElasticIndexerDir\.

Configure the Elastic Indexer#

Navigate to ElasticIndexerDir\.
If there is not a file named appsettings.json, then copy the appsettings.json.dist file to appsettings.json.
In the appsettings.json file, update the following settings.

Data:DefaultConnection:ConnectionString
The full connection string of the database to use to store authentication tables, when using Colectica Portal’s built in user management.

Data:ColecticaRepository:ConnectionString
The full connection string of the Colectica Repository database.

Elasticsearch:Host
The hostname of the Elasticsearch service.

Elasticsearch:ApiId, Elasticsearch:ApiKey
Optional API key credentials for the Elasticsearch service.

ElasticsearchIndexes
A list of the indexes to maintain. Each entry has an IndexName (the prefix used by the portal’s Elasticsearch:IndexName setting), optional IncludeItemTypes and ExcludeItemTypes lists, an optional ExcludeItemsWithTag value, and optional TrackedSets and ReindexSets settings that limit the index to particular item sets and cause child items to be reindexed when a set’s root changes. If this setting is absent, the single index named by Elasticsearch:IndexName is maintained.

ElasticIndexer:EnableApiVersioningHeader
Set to true when using an Elasticsearch 8.x server.

ElasticIndexer:ExcludeItemTypes
Item types that are never indexed.

The ElasticIndexer section also contains batch size, timeout, and retry settings, whose defaults are suitable for most deployments.

Note

The ConnectionString and Elasticsearch settings should match the settings in the Colectica Portal configuration.

Run the Elastic Indexer#

Open a command shell and navigate to ElasticIndexerDir\.
Run the following command:
```
Colectica.Portal.ElasticIndexer.exe
```
On Linux, run ./Colectica.Portal.ElasticIndexer.

Note

Depending on the size of your data, the Elastic Indexer may take a long time to run during the initial ingest. Subsequent runs will only index new content.

The indexer uses a mutex so that only one copy runs at a time. If another copy is already running, it exits with code 100. To run separate indexers for different indexes at the same time, give each one a distinct --instanceName.

Reoccurring task for the Elastic Indexer#

The Elastic Indexer should be run periodically to index new documents using the Windows Task Scheduler or a cron job. An interval of 5 minutes is recommended for near real time indexing. Also set the Start In directory to the directory containing the elastic indexer executable.

Note

For more information about Windows Task Scheduler, see https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page

Reindex using the Elastic Indexer#

The Elastic Indexer can recreate the Elastic Index and reset the indexed state of the items in the Repository.

Open a command shell and navigate to ElasticIndexerDir\.

Run the following command:

Colectica.Portal.ElasticIndexer.exe --reindex

The Indexer will drop and create a new index, and reset the item indexed states in the Repository.

Use --indexName to reindex only one of the configured indexes.

A reindex can also be requested from the System Status page of the portal administration area. The index is rebuilt the next time the indexer runs.

Command Line Options#

--reindex: Recreate the index and reset the indexed state of all items.
--config <file>: Use an alternative configuration file instead of appsettings.json.
--indexName <name>: Apply the operation to a single configured index. If not specified, all indexes are processed.
--instanceName <name>: A name that distinguishes this indexer from others running on the same machine.
--version: Print the build revision.

The trackset and removeset commands add or remove a tracked item set for an index, using the --urn of the set’s root item, the --indexName, and the --trackLatest or --trackLatestTag options to follow the latest version or a tagged version of the set.

Running on alternative versions of Elasticsearch#

Colectica uses the Elasticsearch 7.x client library. Historically elasticsearch has not maintained compatibility across their major versions.

Starting with Elasticsearch 8.x, the Elasticsearch server provides some backwards compatibility using a version header in its REST api calls.

To use a newer version of Elasticsearch with Colectica, you can enable this version header by setting the ElasticIndexer:EnableApiVersioningHeader setting to true in the ElasticIndexer and the Elasticsearch:EnableApiVersioningHeader setting to true in the Colectica Repository.