7.3

7.3.9742

January 2nd, 2024

Colectica Repository and Portal

Added

  • Repository indexing for advanced item search on postgresql

  • Repository automatically manage all databases using EF core and migrations

  • Elastic indexer option to Disable Dynamic Mapping

  • Portal dataset explore view UI refresh

  • Portal show icon for confirmed email addresses in user list

  • New control to show the Name property

  • CSS classes for concordance header rows

  • Concordance view: appsettings configuration to determine whether Variables, Questions, Studies views are available

  • In concordance views, show variable type icons with tooltips for code list details. Indicate code list uniqueness across variables with a number.

  • an ID on the concept tree list items

  • Repository add DecimalPropertyFacet to advanced search request

  • Repository include versionless code list search for codevalues

  • Repository database layouts for advanced search

  • Repository add string interning and reading for advanced search

  • For static content blocks, allow including HTML from a file

  • Advanced search text indexing extractor

  • Configuration for whether to display VersionRationale and VersionResponsibility

  • Repository feature flag to store based on relationships

  • Repository respect relationship predicate in adjacency matrix computations

  • Option to determine whether first visit tours are displayed

  • Repository allow disabling XForwardedFor checking

  • Repository CLI reindex advanced search command

  • Repository CLI migrate database commands

  • Repository real time elastic indexing of deprecated and deleted items

  • Repository admin site information backup downloads

  • Repository Admin setting to disable public basket sharing

  • Repository allow deleting a user account from the Portal Admin user details page

  • Repository new portal setting to allow additional html tags on custom pages using AllowAdditionalTagsOnCustomPages

  • Repository Allow custom html sanitizer for html markdown rendering

  • Use GET or POST method when searching from the home page, according to the options.

  • Show Organization.Nickname in portal

  • Show Group.SeriesInformation on Portal

  • Portal: Show Software information under DataCollectionMethodology

  • Portal: Show CollectionEvent.DataSource information

  • Repository add searching by tag for SQL Server backend

  • Repository add next result offset search for SQL Server backend

  • Repository add identifier only search for SQL Server backend

  • Repository return row ids for items for SQL Server backend

  • Repository remove old sort orderings for SQL Server backend

  • Repository remove metadata rank for SQL Server backend

  • Repository prefix search on SQL Server

  • Portal update Repository search result positional summary

  • Portal show DDI format that the item is stored as in Repository

  • Portal: Include QuestionGrids in the explore/questions view.

  • Repository config setting to allow skipping smtp certificate validation and revocation check

  • Replace all versionable lists with versionable tables, showing icon, name, and label

  • Portal: Add view for ClassificationCorrespondenceTable items

  • Portal: additional French translations

  • Portal csv download of code comparison

Changed

  • For concordance tree, consistently show {item name} - {item label} for each node. Don’t show RepresentedVariables with no Variables pointing to them.

  • Extra logging for ForgotPassword, so it is knowable when and why emails aren’t sent

  • In multi-language popup, always show all languages, even if they are displayed on the page

  • Show breadcrumbs according to language settings

  • Don’t show download button for plotly box plots

  • Display two decimal places for summary statistics, instead of all of them

  • Disable option for descending concordance views

  • upgrade OpenTelemetry

  • On variable page, show the PhysicalInstance Title in the link to the PI page, not the file location

  • For multilingual custom fields that have only 1 language with content, show that value

  • When PhysicalInstance does not have Coverage information, use Coverage information from StudyUnit

  • In search results, always show Name and Label, or Title, as properties in the card body. Wrap in CSS classes to allow hiding these.

  • Repository optimized GetTags for Postgresql

  • Repository don’t use transactionscopes anymore

  • Repository new GetLatestVersionNumbersPostgresqlAsync using unnest, greatly improves large requests on postgresql

  • In explore-by-concept view, allow linking to a concept that will be selected on load

  • Don’t render markdown for variable labels in search results

  • Repository Upgrade to MailKit from smtpclient

  • Json use DateTimeZoneHandling.Utc everywhere we use json settings

  • Concordance displays: when a ConceptualVariable does not have a Label, show the ItemName instead

  • Repository Set the security stamp timeout to zero seconds

  • Repository Add configuration switch to use __Host prefix with the cookie name, set RepositorySettings:Http:SetHostCookiePrefix=true

  • Repository New setting to enable PDF downloads in the Admin->Site Information, a 404 is returned for the pdf if not enabled.

  • Repository If the feedback email address is empty in the Admin->Site Information, a 404 is returned for the feedback form.

  • Repository Disable formatting in basket text fields

  • Repository Option to show generic validation details instead of detailed json validation errors if incorrect json is sent to the API Set RepositorySettings:Features:DisableValidationDetailsReports=true

  • Repository Added a configuration option in appsettings to disable the swagger ui. Set RepositorySettings:Portal:DisableSwaggerUI=true.

  • Repository used a default language when a request comes from a non-browser agent without an Accept-Language header

  • Repository changed status code to 404 when an invalid route is accessed

  • Repository return a 501 not implemented status code for future APIs that are not yet available

  • Reports Removed version string from creator tag

  • Render markdown for custom Explore message

  • Cross-series concordance: when determining whether a ConVar already exists in a harmonization group, compare IDs instead of locator/name

  • Upgrade HtmlSanitizer and System.Security.Cryptography.Pkcs to latest versions

  • PhysicalInstance page: show HumanReadableLocation for public DataFileIdentifications

  • Show more DataRelationship information directly on PhysicalInstance page

  • Portal only look at instrument location scheme when an absolute uri

  • Portal fewer db queries on item page

  • Custom fields; display the category label when the value is from a controlled vocabulary. Show boolean values.

  • Log errors when DDI Set generation fails

  • Portal instrument flowchart: expand response options by default; configuration to make the collapsable.

  • Portal question grid display: by default, show dimensions and response options linearly; config option to show a table

  • Portal search results: don’t show question text or description in the result card body if it is already shown in the result card header.

  • Nicer style for OtherMaterial group headers

  • Only use metadata languages from config, not the system language

  • Repository advanced search allow searching by versionResponsibility

  • Repository advanced search include isPublished, versionDate, and versionResponsibility in search responses when include all fields is specified

  • Portal VersionableTables: handle questions and variables

  • Portal: VersionableTable output for Organization and Individual

  • Portal: make the search error message localizable

  • Portal: don’t highlight Explore in top nav for the explore-single-dataset page

  • Repository allow cluster HostId to be a string

  • Portal concordance view: use letters instead of numbers to represent code list distinctness

  • Portal: show more results on Explore/Index page

  • Portal - Instrument Summary: always show question name when it exists (previously it was not shown when a question label existed)

  • Portal - Instrument Summary: show content in multiple languages

  • Portal In search, Configure elastic to assign a class to highlights, instead of only using <em>.

  • Portal: Limit the Markdown extensions to use; don’t get rid of all text inside { braces }

  • Elasticsearch command line to add and remove tracked sets

Fixed

  • Fix links in Multilingual texts

  • Fix French translations of “DDI 3.3” to say “3.3” not “3.2”

  • Repository reuse connection for timestamp cache when in transaction

  • When using a language from the browser’s Accept-Languages setting, don’t also include it the hiddenLanguages

  • One-to-many: the one and the many should have the same background shading

  • Fix overlapping data type icon in concordance table

  • Portal frequencies display: locate category label correctly for nested codes

  • Repository when installing postgresql unaccent extension, continue migration even if unable to install. Make it a runtime error within advanced search if requested.

  • Repository fix recording last delete time upon delete completion

  • Explore page: make sure Reset button fully resets the dropdown

  • Explore page item list: set to Variables by default, to avoid an error when clicking Filter before selecting a concept

  • Explore page: after Filter button press, only call Search() once, via the PageChange event

  • Proper counting of option aggregates for batched Elasticsearch queries

  • Correctly show SubTitle instead of Subject

  • Respect the Smtp:EnableSsl setting

  • Portal fix creation of expired classification badge when only one side of the date range is present

  • Portal search page: don’t show “Add to list” when the item type is not allowed in the basket

  • Portal search results: Don’t show “Log in to add to basket” message when the is actually logged in, but the item isn’t allowed in the basket

  • Elastic don’t duplicate versionlessParentIds in elastic document when multiple versions of a parent exist

  • Portal: use correct model type for RecordLayout page

  • Portal search page: update search syntax examples to be consistent

  • Portal: Fix spelling error in aria labels

  • Localization: spelling fixes and use DdiLocalizer instead of PortalLocalizer where appropriate

  • Portal frequency display: Don’t show “% of Valid” for invalid categories

  • Repository perform EF upgrade after upgrading a pre-EF database to EF without requiring a restart

Colectica Designer

Added

  • QualityStandard command to add ComplianceDefinitions from a ConceptScheme

  • Designer PhysicalInstance editor: add ProcessingCheck

  • Designer Quality Statement view: method to remove quality statement items

  • KindsOfData control for Group view

  • Designer item picker: support searching for multiple item types

Changed

  • Designer: show text on editor tabs, in addition to the icons. Make the tab control styled like a normal tab control.

  • Designer check remote version numbers in the background when calling RefreshItemStates from the DesignerApi (Don’t wait for Repository status to speed up navigator)

  • In Stata import, mark categorical variables as NonAdditive

  • Designer SKOS import: run on background thread, show progress, and improve performance

  • Designer: For rich text boxes, show a button that opens a dialog with the markdown preview, instead of using right side pane

  • Designer when importing summary frequencies from spreadsheets, create managed missing values for blank code values not in the existing codelist

  • Designer: show the markdown editor buttons again

  • Do not compute quartiles for non-additive variables in SAS or other row-by-row formats

  • When populating QualityStatementItems, update ComplianceConcepts to latest versions when new versions are present in the source scheme or standard

  • When loading an Excel sheet, if only one sheet exists, use that instead of requiring a matching sheet name

  • Cross series concordance ingest: ignore whitespace around conceptual variable names

  • In Update from File command, when previously shared code lists now differ, create a new code list for the changed items

  • In PDFs, don’t show frequencies when there are more than 75 items

  • Metadata import sheets: create variables that don’t yet exist

  • For data file imports, make the top level item PhysicalInstance instead of ResourcePackage

  • Re-use newly created RepresentedVariables during a concordance import

  • Always populate variables’ code lists during a concordance import

  • Multilingual control for StatementText

  • Wider display for classification item values.

  • Designer: in CollectionEvent list, show the label or name for list items, and update the display immediately after the values are edited.

  • Designer: show useful list item text for DataCollectionMethodology lists.

  • Designer: for DateTime values, allow specifying the time.

  • Designer item population: Use SetPopulator instead of GraphPopulator for performance increase

  • Designer: use controlled vocabulary resolution for CustomFields with CVs.

  • DDI 2 When there are no `fileDscr`s, make a PhysicalInstance to use as the default one

  • DDI 2 Use varGrp/name instead of /id when the /id is not present

  • DDI 2 Map varGrp/defntn to something

  • DDI 2 Speed up varGrp child location

  • Variable Metadata Import: allow specifying multiple groupings for columns prefixed with “group:”

  • Designer Series editor: when selecting Subseries, allow searching both Series and legacy SubSeries item types

  • Designer: when saving changes to a legacy SubSeries item, save as a regular Series

  • Designer navigator: show legacy Subseries items

  • Designer: Specify supported UI language on startup so WpfLocalizeExtension doesn’t search 800+ cultures, slowly

  • Designer ConceptScheme view: efficient concept hierarchy creation to make it possible to open large concept sets

  • Designer ConceptScheme view: show DDI ID of selected concept

Fixed

  • Show all codes on paper forms

  • Designer upgrade/rewrite item table if local database was upgraded from v5->v6->v7

  • Designer PhysicalInstance view: ensure “More Storage Locations” editor is not hidden

  • Designer: fix editors for SoftwarePackage (CodeValue) and ReleaseDate.

Colectica Questionnaires

Added

  • Colectica Questionnaires enable building Blaise addin on net7

  • Questionnaires: Set agency identifier in File - Options

Colectica Workflow

Added

  • Workflow Published Items page: show the source and target repositories

  • Workflow: additional French translations

  • Workflow repository configuration: allow multiple languages for repository labels.

  • Workflow Admin state configuration: multiple languages for Family and State labels

  • Workflow: show version number in Admin - General

  • Workflow request list: More consistent layout - Workflow request list: show preview indicator for requests that have previews

  • Workflow Web: use Serilog for logging

Changed

  • Workflow improve default handling of supported ui languages setting

  • Workflow Upgrade to MailKit from smtpclient

  • Workflow: Show only “English” instead of “English (United States)” in language picker

  • Workflow: Create preview request and replication request at the same time; show preview pending state until it is ready

  • Workflow: Create preview request for state change requests that would trigger a replication when approved

  • Repository-to-Workflow requests: send repository server ID in the request header

  • Workflow: remove use off Workflow:ApiKey config value. Always use repository-specific API keys

  • Workflow API: When a pending replication request already exists, do not add a duplicate

  • Workflow Web: Use ErrorBoundary to catch top level exception and show nicer error messages

Fixed

  • Workflow: when denying a request, don’t require a message

  • Workflow admin state page: check ContainsKey before accessing content for a language

Colectica SDK

Added

  • Model embed postgresql unaccent rules list

  • Model utility to replace diacritical marks and some ligatures to Latin-ASCII text

  • Model unaccent separate surrogates collection to avoid most common char to int casts

  • IHasCoverage for class that have a Coverage property

  • Index CodeValue values for advanced search

  • Index DateSpecification and Date properties for advanced search

  • Index based on relationships

  • SDK use new 3.3 Label and Description fields for Question, QuestionGrid, and QuestionBlock

  • SDK add user attributes to identifiable objects in 3.3 including Access, Code, DataCollectionItem (ModeOfCollection, ActionToMinimizeLosses, CollectionSituation), Embargo, GrossFileStructure, GrossRecordStructure, LogicalRecord, MethodologyItem (DataCollectionMethodology, TimeMethod, SamplingProcedure, DeviationFromSampleDesign), Parameter, RecordRelationship, SpatialCoverage, TemporalCoverage, TopicalCoverage, AggregationVariables,

  • SDK add TypeOfAccess to Access when using DDI 3.3

  • SDK add AccessPermissions to Access

  • SDK add LifecycleEvents to Archive

  • SDK add new 3.3 versionable type WeightingMethodology

  • SDK move some child handling to identifiable base since it has custom fields now with relationships

  • SDK allow Individual references in ArchiveOrganizationReference and ContactOrganizationReference

  • Support PhysicalInstance on Group

  • SDK parse DDI urns from strings in addition to identifier triples

Changed

  • SetPopulator instead of GraphPopulator for instruments

  • SDK improve performance using set based population for children in getitems call

  • SDK allow multiple AnalysisUnits on Study and Group

  • Workflow show the user’s own count of pending items on the home page

Fixed

  • Serialization fixes for ResponseText. It doesn’t get wrapped in ResponseDomainInMixed.

  • SDK handle invalid months in DDI year month to datetime conversion

  • SDK try to deploy parquet native on 472 Data nuget