7.3

7.3.10002

April 26th, 2024

Colectica Repository and Portal

Added

  • Repository parallel and bulk advanced search indexing

  • Repository use unnest on postgresql for bulk register items

  • Repository use openjson on sqlserver for bulk register items

  • Repository use channels for register items

  • CLI show live bar charts during direct database DDI file imports

  • Repository SQL Server based streaming registration

  • Repository latest version bookkeeping for streaming registration

  • Repository streaming indexing for advanced search on postgresql

  • Repository retry text interning when conflicted/deadlocked under high load

  • Repository log invalid proxied responses received when requesting workflow get targets api

  • Repository show verbatim item tags on item info tab

  • Repository request option for FilteredJsonSet API to return an error instead of of a partially populated set

  • Portal: when a language is specified on a Keyword or Subject, show the language

  • Portal: Show InformationClassification Type and Level properties on Group and PhysicalInstance pages

  • Portal: series/study/dataset explorer. Hidden behind a feature flag

  • Portal item tables: Hyperlink the image, in case an item has no name or label

  • Portal: sort lists of related items alphabetically

  • Portal: Option to show a search targets dropdown, to limit search to all, name, label, or description

  • Portal: for variables, show new Start Position, End Position, and Width properties

Changed

  • Repository update System.IdentityModel.Tokens.Jwt due to CVE

  • Repository upgrade Microsoft.Data.SqlClient for CVE

  • Repository pin Microsoft.IdentityModel.JsonWebTokens for CVE

  • Repository upgrade System.Data.SqlClient CVE-2024-0056

  • Portal Don’t show Universe.Description on Variable page; the link via the Universe Label will suffice

  • Repository CLI aggregate progress when importing multiple files

  • Portal Process text for markdown even if it looks like likely xhtml

  • Portal Show category labels as multilingual strings

  • Portal Site setting for processing markdown in category labels within codelists

  • Dataset explore page: variable count and case count on the Dataset Info page

  • Repository set application name on postgresql db connections

  • Portal: Remove indirect jstree references where it is not used; reference it where it is used.

  • Repository return an error on advanced search when it is not enabled

  • Portal variable concordance view: Don’t include variables from deprecated PhysicalInstances

  • Repository switch default log level to Warning

  • Elastic indexer retry failed db calls during second pass item processing

  • Portal Elasticsearch results: only use content from Highlights when full content is not already present

  • Portal: When a Description is used for an item link, don’t hyperlink it

  • Portal: Show ProcessingEvent information directly on StudyUnit page

  • Portal: translations for AggregationMethod terms

  • Portal: show some PhysicalInstance properties on the dataset explorer page

  • Portal: Indicate “unweighted” for statistics that are unweighted

  • Portal Search page pager: don’t show jump to last button.

  • Portal Home page item lists: only show the top X items.

  • Portal: always show VersionDate in the browser’s timezone and locale

  • Portal: show comment time in the browser’s timezone and locale

  • Portal: When only one UI language variant is configured as supported, only show the language name

  • Portal upgrade jquery and jquery validation

  • Portal upgrade jsTree to 3.3.16

  • Portal remove tether js

  • Portal upgrade shepherd to 11.2

  • Portal upgrade plotly.js to 2.31.1

  • Portal update intro tours to new version of shepherd

  • Portal explore view: default to Questions view when it is the only one configured

  • Portal explore view: In Questions view, also show Question Grids and Question Blocks

  • Portal show the language for CodeValue properties that have a controlled vocabulary assigned

Fixed

  • Elasticsearch don’t pass an empty byte[] to UTF8.GetString, otherwise that gives an error and nothing useful is logged

  • Dataset Explore: correctly populate variable groups

  • Portal data extract: Use prefixed variable names when making SPSS datasets

  • Repository don’t create local JWT tokens when local JWT auth is disabled

  • Portal Explore view: make breadcrumb URLs correctly for variable and question lists

  • Portal: properly group OtherMaterials by their type, instead of one header per material

  • Portal when using a custom theme, use the default fav icon if a theme specific one is not available

  • Portal basket: immediately remove items after clicking the remove button.

Colectica Designer

Added

  • Designer: new properties on Variable to control whether frequencies are calculated for valid and invalid values

  • Designer: more efficient variable lookup during statistics calculation

  • Designer: use less memory for summary statistics generation

  • Designer add deleting prior versions and prior sets to Repository delete command

  • Designer: button to remove a tag when viewing an item under a repository

  • Designer: Provide default dropdown values for CodeValue controls that used to be dropdowns (NumericDomain.Type, DateTimeDomain.Type).

  • Designer: additional French(-CA) translations

  • Designer: French translations for dropdowns in variable editor

  • Designer: Log Off command in File-Logoff to clear active Repository logins

  • Designer: survey instrument editor: for conditionals and loops, add a button to Delete and leave contents

Changed

  • Designer concordance input sheets: look for datasets directly within Series

  • Designer: “weight” checkbox should not be present for RepresentedVariable, since the property is not in DDI.

  • Designer PhysicalInstance editor: save version dates even when no version number is specified. DDI requires the version number attribute, but it is allowed to be blank.

  • PhysicalInstance PDFs: indicate whether statistics are unweighted, or what the weight variable is.

  • Designer: use full width for Code List and Classification items

  • Designer frequency display: correct numeric sorting for code values

Fixed

  • Designer watch AnalysisUnits for dirty changes

  • Designer Classification editor: don’t show Updates field twice

  • Designer date editor: bind to correct property for DateOnly

  • Designer: Correctly watch for changes to Date properties in DateSpecification

  • Designer: values of custom fields of types MultipleRelationship and Relationship were not correctly displayed

  • Fix binding for RepresentedVariable.HasRepresentation

  • Designer publish dialog: show correct language of initial states, when state configuration is available from the repository

  • Designer Classification Index view: ensure ClassificationItems referenced by index entries are populated, so their title is displayed correctly.

  • Designer - Data Collection - Collection Event editor: add a horizontal scrollbar when the date range controls are wider than the window

  • Designer Blaise import: set category label correctly.

Colectica for Excel

Changed

  • Excel switch Word to Html report

  • Excel handle 97-2003 compatibility mode better where max supported columns (256) is less than copy partition size

  • Excel add assembly resolver hook

  • Excel reference new pdfsharp

Colectica Workflow

Added

  • Workflow favicon

  • Workflow repository list

  • Workflow website show admin’s the current Repositories

  • Workflow website log more possible configuration errors and error conditions

  • Show View in Portal button in the Requests view

  • Workflow: on the Requests page, only show Designate button for PublicationRequests

  • Workflow: on request page, show version date and rationale for the selected item

  • Workflow: Repository browser: show version number of item, and version number of item at each timeline event

  • Workflow: show metadata content using the selected language

  • Workflow: more French Canadian localizations

  • Workflow: Always show user-local times, and in the same format

  • Workflow admin: edit email template for decision notifications

  • Workflow: send notification emails to state change approvers; log when emails cannot be sent due to no email address

  • Workflow default filters for removing PII information enabled via configuration

Changed

  • Workflow Web Published Items page: show version from the Checkout

  • Show requester email address in request details

  • Workflow: Don’t use ServerPreRender

  • Workflow: tabular display for home page replication status counts

  • Workflow: always show “None” when no state has been assigned to an item

  • Workflow: show times in the user’s time zone

  • Workflow don’t show logout on Windows Auth

  • Validate state configuration form to ensure tokens are set for all non-None states

  • Workflow Processes page: show scheduled replication time when one is set

  • Workflow use new Delete API instead of workspacesync cleaning when removing published sets

  • Workflow remove all javascript libraries

  • Workflow Enable item deletion in workflow task runner for removal requests

Fixed

  • Workflow compare against username not name when loading valid approvers

  • Workflow Web Processes page: show status message correctly

  • Workflow Web Published page: show “Item not present in [repository]” message correctly

  • Workflow if the user is not authenticated, don’t display a blank shell of the application and redirect to login

  • Workflow: properly determine whether a state is the None state, and allow editing the token if it is not the None state

  • Workflow sanitize email templates

  • Workflow: Use the correct comment content for Approval email notifications

  • Workflow handle blank usernames during replication from the Repository

  • Workflow Web: proper whitespace for “no processing” message

Colectica SDK

Added

  • SDK Parallel deserialization of Fragment Instance

  • SDK don’t wait for deserialization when reading fragment instance from disk

  • SDK use read-in xml instead of reserializing when making repository items

  • SDK add NullItemCache class to turn off deserializer item caching

  • Add extension for typeOfInformationClassification and levelOfInformationClassification to Study Group Subgroup and PhysicalInstance

  • Add advanced search indexing of typeOfInformationClassification and levelOfInformationClassification extensions

  • Datasets Column reading for SPSS

  • Datasets streaming single columns SAS data

  • Datasets converting SPSS to parquet

  • Datasets determine column types for csv imports

  • Datasets Stata using memory mapped file for reading

  • Datasets reuse buffers when reading Stata

  • Datasets read single values when doing column reading

  • SDK add new DDI3.3 GeneralDataFormat to StudyUnit and Group

  • SDK use extension:GeneralDataFormat to add GeneralDataFormat to DDI 3.2 serialization

  • DDI 2-to-DDI 3 mapping updates

  • Support Language for InternationalCodeValue types (Keywords and Subjects)

  • SDK add TypeOfItem for Group, SubGroup, and Study, new in 3.3

  • Designer show list of unmatched variables on set source instrument command

Changed

  • SDK update markdig version in reporting nuget

  • SDK allow lifecycle events without a date

  • Reporting switch to official PdfSharp for all builds

  • Reporting log fonts that can’t be loaded

Fixed

  • Designer and SDK: when determining the number of rows in a CSV file, do not map records to objects. This prevents and error when multiple columns have the same name.

  • Reporting sanitize html codebook output

  • Fix error when exporting DDI 2 with a contributor without a role

7.3.9742

January 2nd, 2024

Colectica Repository and Portal

Added

  • Repository indexing for advanced item search on postgresql

  • Repository automatically manage all databases using EF core and migrations

  • Elastic indexer option to Disable Dynamic Mapping

  • Portal dataset explore view UI refresh

  • Portal show icon for confirmed email addresses in user list

  • New control to show the Name property

  • CSS classes for concordance header rows

  • Concordance view: appsettings configuration to determine whether Variables, Questions, Studies views are available

  • In concordance views, show variable type icons with tooltips for code list details. Indicate code list uniqueness across variables with a number.

  • an ID on the concept tree list items

  • Repository add DecimalPropertyFacet to advanced search request

  • Repository include versionless code list search for codevalues

  • Repository database layouts for advanced search

  • Repository add string interning and reading for advanced search

  • For static content blocks, allow including HTML from a file

  • Advanced search text indexing extractor

  • Configuration for whether to display VersionRationale and VersionResponsibility

  • Repository feature flag to store based on relationships

  • Repository respect relationship predicate in adjacency matrix computations

  • Option to determine whether first visit tours are displayed

  • Repository allow disabling XForwardedFor checking

  • Repository CLI reindex advanced search command

  • Repository CLI migrate database commands

  • Repository real time elastic indexing of deprecated and deleted items

  • Repository admin site information backup downloads

  • Repository Admin setting to disable public basket sharing

  • Repository allow deleting a user account from the Portal Admin user details page

  • Repository new portal setting to allow additional html tags on custom pages using AllowAdditionalTagsOnCustomPages

  • Repository Allow custom html sanitizer for html markdown rendering

  • Use GET or POST method when searching from the home page, according to the options.

  • Show Organization.Nickname in portal

  • Show Group.SeriesInformation on Portal

  • Portal: Show Software information under DataCollectionMethodology

  • Portal: Show CollectionEvent.DataSource information

  • Repository add searching by tag for SQL Server backend

  • Repository add next result offset search for SQL Server backend

  • Repository add identifier only search for SQL Server backend

  • Repository return row ids for items for SQL Server backend

  • Repository remove old sort orderings for SQL Server backend

  • Repository remove metadata rank for SQL Server backend

  • Repository prefix search on SQL Server

  • Portal update Repository search result positional summary

  • Portal show DDI format that the item is stored as in Repository

  • Portal: Include QuestionGrids in the explore/questions view.

  • Repository config setting to allow skipping smtp certificate validation and revocation check

  • Replace all versionable lists with versionable tables, showing icon, name, and label

  • Portal: Add view for ClassificationCorrespondenceTable items

  • Portal: additional French translations

  • Portal csv download of code comparison

Changed

  • For concordance tree, consistently show {item name} - {item label} for each node. Don’t show RepresentedVariables with no Variables pointing to them.

  • Extra logging for ForgotPassword, so it is knowable when and why emails aren’t sent

  • In multi-language popup, always show all languages, even if they are displayed on the page

  • Show breadcrumbs according to language settings

  • Don’t show download button for plotly box plots

  • Display two decimal places for summary statistics, instead of all of them

  • Disable option for descending concordance views

  • upgrade OpenTelemetry

  • On variable page, show the PhysicalInstance Title in the link to the PI page, not the file location

  • For multilingual custom fields that have only 1 language with content, show that value

  • When PhysicalInstance does not have Coverage information, use Coverage information from StudyUnit

  • In search results, always show Name and Label, or Title, as properties in the card body. Wrap in CSS classes to allow hiding these.

  • Repository optimized GetTags for Postgresql

  • Repository don’t use transactionscopes anymore

  • Repository new GetLatestVersionNumbersPostgresqlAsync using unnest, greatly improves large requests on postgresql

  • In explore-by-concept view, allow linking to a concept that will be selected on load

  • Don’t render markdown for variable labels in search results

  • Repository Upgrade to MailKit from smtpclient

  • Json use DateTimeZoneHandling.Utc everywhere we use json settings

  • Concordance displays: when a ConceptualVariable does not have a Label, show the ItemName instead

  • Repository Set the security stamp timeout to zero seconds

  • Repository Add configuration switch to use __Host prefix with the cookie name, set RepositorySettings:Http:SetHostCookiePrefix=true

  • Repository New setting to enable PDF downloads in the Admin->Site Information, a 404 is returned for the pdf if not enabled.

  • Repository If the feedback email address is empty in the Admin->Site Information, a 404 is returned for the feedback form.

  • Repository Disable formatting in basket text fields

  • Repository Option to show generic validation details instead of detailed json validation errors if incorrect json is sent to the API Set RepositorySettings:Features:DisableValidationDetailsReports=true

  • Repository Added a configuration option in appsettings to disable the swagger ui. Set RepositorySettings:Portal:DisableSwaggerUI=true.

  • Repository used a default language when a request comes from a non-browser agent without an Accept-Language header

  • Repository changed status code to 404 when an invalid route is accessed

  • Repository return a 501 not implemented status code for future APIs that are not yet available

  • Reports Removed version string from creator tag

  • Render markdown for custom Explore message

  • Cross-series concordance: when determining whether a ConVar already exists in a harmonization group, compare IDs instead of locator/name

  • Upgrade HtmlSanitizer and System.Security.Cryptography.Pkcs to latest versions

  • PhysicalInstance page: show HumanReadableLocation for public DataFileIdentifications

  • Show more DataRelationship information directly on PhysicalInstance page

  • Portal only look at instrument location scheme when an absolute uri

  • Portal fewer db queries on item page

  • Custom fields; display the category label when the value is from a controlled vocabulary. Show boolean values.

  • Log errors when DDI Set generation fails

  • Portal instrument flowchart: expand response options by default; configuration to make the collapsable.

  • Portal question grid display: by default, show dimensions and response options linearly; config option to show a table

  • Portal search results: don’t show question text or description in the result card body if it is already shown in the result card header.

  • Nicer style for OtherMaterial group headers

  • Only use metadata languages from config, not the system language

  • Repository advanced search allow searching by versionResponsibility

  • Repository advanced search include isPublished, versionDate, and versionResponsibility in search responses when include all fields is specified

  • Portal VersionableTables: handle questions and variables

  • Portal: VersionableTable output for Organization and Individual

  • Portal: make the search error message localizable

  • Portal: don’t highlight Explore in top nav for the explore-single-dataset page

  • Repository allow cluster HostId to be a string

  • Portal concordance view: use letters instead of numbers to represent code list distinctness

  • Portal: show more results on Explore/Index page

  • Portal - Instrument Summary: always show question name when it exists (previously it was not shown when a question label existed)

  • Portal - Instrument Summary: show content in multiple languages

  • Portal In search, Configure elastic to assign a class to highlights, instead of only using <em>.

  • Portal: Limit the Markdown extensions to use; don’t get rid of all text inside { braces }

  • Elasticsearch command line to add and remove tracked sets

Fixed

  • Fix links in Multilingual texts

  • Fix French translations of “DDI 3.3” to say “3.3” not “3.2”

  • Repository reuse connection for timestamp cache when in transaction

  • When using a language from the browser’s Accept-Languages setting, don’t also include it the hiddenLanguages

  • One-to-many: the one and the many should have the same background shading

  • Fix overlapping data type icon in concordance table

  • Portal frequencies display: locate category label correctly for nested codes

  • Repository when installing postgresql unaccent extension, continue migration even if unable to install. Make it a runtime error within advanced search if requested.

  • Repository fix recording last delete time upon delete completion

  • Explore page: make sure Reset button fully resets the dropdown

  • Explore page item list: set to Variables by default, to avoid an error when clicking Filter before selecting a concept

  • Explore page: after Filter button press, only call Search() once, via the PageChange event

  • Proper counting of option aggregates for batched Elasticsearch queries

  • Correctly show SubTitle instead of Subject

  • Respect the Smtp:EnableSsl setting

  • Portal fix creation of expired classification badge when only one side of the date range is present

  • Portal search page: don’t show “Add to list” when the item type is not allowed in the basket

  • Portal search results: Don’t show “Log in to add to basket” message when the is actually logged in, but the item isn’t allowed in the basket

  • Elastic don’t duplicate versionlessParentIds in elastic document when multiple versions of a parent exist

  • Portal: use correct model type for RecordLayout page

  • Portal search page: update search syntax examples to be consistent

  • Portal: Fix spelling error in aria labels

  • Localization: spelling fixes and use DdiLocalizer instead of PortalLocalizer where appropriate

  • Portal frequency display: Don’t show “% of Valid” for invalid categories

  • Repository perform EF upgrade after upgrading a pre-EF database to EF without requiring a restart

Colectica Designer

Added

  • QualityStandard command to add ComplianceDefinitions from a ConceptScheme

  • Designer PhysicalInstance editor: add ProcessingCheck

  • Designer Quality Statement view: method to remove quality statement items

  • KindsOfData control for Group view

  • Designer item picker: support searching for multiple item types

Changed

  • Designer: show text on editor tabs, in addition to the icons. Make the tab control styled like a normal tab control.

  • Designer check remote version numbers in the background when calling RefreshItemStates from the DesignerApi (Don’t wait for Repository status to speed up navigator)

  • In Stata import, mark categorical variables as NonAdditive

  • Designer SKOS import: run on background thread, show progress, and improve performance

  • Designer: For rich text boxes, show a button that opens a dialog with the markdown preview, instead of using right side pane

  • Designer when importing summary frequencies from spreadsheets, create managed missing values for blank code values not in the existing codelist

  • Designer: show the markdown editor buttons again

  • Do not compute quartiles for non-additive variables in SAS or other row-by-row formats

  • When populating QualityStatementItems, update ComplianceConcepts to latest versions when new versions are present in the source scheme or standard

  • When loading an Excel sheet, if only one sheet exists, use that instead of requiring a matching sheet name

  • Cross series concordance ingest: ignore whitespace around conceptual variable names

  • In Update from File command, when previously shared code lists now differ, create a new code list for the changed items

  • In PDFs, don’t show frequencies when there are more than 75 items

  • Metadata import sheets: create variables that don’t yet exist

  • For data file imports, make the top level item PhysicalInstance instead of ResourcePackage

  • Re-use newly created RepresentedVariables during a concordance import

  • Always populate variables’ code lists during a concordance import

  • Multilingual control for StatementText

  • Wider display for classification item values.

  • Designer: in CollectionEvent list, show the label or name for list items, and update the display immediately after the values are edited.

  • Designer: show useful list item text for DataCollectionMethodology lists.

  • Designer: for DateTime values, allow specifying the time.

  • Designer item population: Use SetPopulator instead of GraphPopulator for performance increase

  • Designer: use controlled vocabulary resolution for CustomFields with CVs.

  • DDI 2 When there are no fileDscr elements, make a PhysicalInstance to use as the default one

  • DDI 2 Use varGrp/name instead of /id when the /id is not present

  • DDI 2 Map varGrp/defntn to something

  • DDI 2 Speed up varGrp child location

  • Variable Metadata Import: allow specifying multiple groupings for columns prefixed with “group:”

  • Designer Series editor: when selecting Subseries, allow searching both Series and legacy SubSeries item types

  • Designer: when saving changes to a legacy SubSeries item, save as a regular Series

  • Designer navigator: show legacy Subseries items

  • Designer: Specify supported UI language on startup so WpfLocalizeExtension doesn’t search 800+ cultures, slowly

  • Designer ConceptScheme view: efficient concept hierarchy creation to make it possible to open large concept sets

  • Designer ConceptScheme view: show DDI ID of selected concept

Fixed

  • Show all codes on paper forms

  • Designer upgrade/rewrite item table if local database was upgraded from v5->v6->v7

  • Designer PhysicalInstance view: ensure “More Storage Locations” editor is not hidden

  • Designer: fix editors for SoftwarePackage (CodeValue) and ReleaseDate.

Colectica Questionnaires

Added

  • Colectica Questionnaires enable building Blaise addin on net7

  • Questionnaires: Set agency identifier in File - Options

Colectica Workflow

Added

  • Workflow Published Items page: show the source and target repositories

  • Workflow: additional French translations

  • Workflow repository configuration: allow multiple languages for repository labels.

  • Workflow Admin state configuration: multiple languages for Family and State labels

  • Workflow: show version number in Admin - General

  • Workflow request list: More consistent layout - Workflow request list: show preview indicator for requests that have previews

  • Workflow Web: use Serilog for logging

Changed

  • Workflow improve default handling of supported ui languages setting

  • Workflow Upgrade to MailKit from smtpclient

  • Workflow: Show only “English” instead of “English (United States)” in language picker

  • Workflow: Create preview request and replication request at the same time; show preview pending state until it is ready

  • Workflow: Create preview request for state change requests that would trigger a replication when approved

  • Repository-to-Workflow requests: send repository server ID in the request header

  • Workflow: remove use off Workflow:ApiKey config value. Always use repository-specific API keys

  • Workflow API: When a pending replication request already exists, do not add a duplicate

  • Workflow Web: Use ErrorBoundary to catch top level exception and show nicer error messages

Fixed

  • Workflow: when denying a request, don’t require a message

  • Workflow admin state page: check ContainsKey before accessing content for a language

Colectica SDK

Added

  • Model embed postgresql unaccent rules list

  • Model utility to replace diacritical marks and some ligatures to Latin-ASCII text

  • Model unaccent separate surrogates collection to avoid most common char to int casts

  • IHasCoverage for class that have a Coverage property

  • Index CodeValue values for advanced search

  • Index DateSpecification and Date properties for advanced search

  • Index based on relationships

  • SDK use new 3.3 Label and Description fields for Question, QuestionGrid, and QuestionBlock

  • SDK add user attributes to identifiable objects in 3.3 including Access, Code, DataCollectionItem (ModeOfCollection, ActionToMinimizeLosses, CollectionSituation), Embargo, GrossFileStructure, GrossRecordStructure, LogicalRecord, MethodologyItem (DataCollectionMethodology, TimeMethod, SamplingProcedure, DeviationFromSampleDesign), Parameter, RecordRelationship, SpatialCoverage, TemporalCoverage, TopicalCoverage, AggregationVariables,

  • SDK add TypeOfAccess to Access when using DDI 3.3

  • SDK add AccessPermissions to Access

  • SDK add LifecycleEvents to Archive

  • SDK add new 3.3 versionable type WeightingMethodology

  • SDK move some child handling to identifiable base since it has custom fields now with relationships

  • SDK allow Individual references in ArchiveOrganizationReference and ContactOrganizationReference

  • Support PhysicalInstance on Group

  • SDK parse DDI urns from strings in addition to identifier triples

Changed

  • SetPopulator instead of GraphPopulator for instruments

  • SDK improve performance using set based population for children in getitems call

  • SDK allow multiple AnalysisUnits on Study and Group

  • Workflow show the user’s own count of pending items on the home page

Fixed

  • Serialization fixes for ResponseText. It doesn’t get wrapped in ResponseDomainInMixed.

  • SDK handle invalid months in DDI year month to datetime conversion

  • SDK try to deploy parquet native on 472 Data nuget