7.3¶
7.3.10002¶
April 26th, 2024
Colectica Repository and Portal¶
Added¶
Repository parallel and bulk advanced search indexing
Repository use unnest on postgresql for bulk register items
Repository use openjson on sqlserver for bulk register items
Repository use channels for register items
CLI show live bar charts during direct database DDI file imports
Repository SQL Server based streaming registration
Repository latest version bookkeeping for streaming registration
Repository streaming indexing for advanced search on postgresql
Repository retry text interning when conflicted/deadlocked under high load
Repository log invalid proxied responses received when requesting workflow get targets api
Repository show verbatim item tags on item info tab
Repository request option for FilteredJsonSet API to return an error instead of of a partially populated set
Portal: when a language is specified on a Keyword or Subject, show the language
Portal: Show InformationClassification Type and Level properties on Group and PhysicalInstance pages
Portal: series/study/dataset explorer. Hidden behind a feature flag
Portal item tables: Hyperlink the image, in case an item has no name or label
Portal: sort lists of related items alphabetically
Portal: Option to show a search targets dropdown, to limit search to all, name, label, or description
Portal: for variables, show new Start Position, End Position, and Width properties
Changed¶
Repository update System.IdentityModel.Tokens.Jwt due to CVE
Repository upgrade Microsoft.Data.SqlClient for CVE
Repository pin Microsoft.IdentityModel.JsonWebTokens for CVE
Repository upgrade System.Data.SqlClient CVE-2024-0056
Portal Don’t show Universe.Description on Variable page; the link via the Universe Label will suffice
Repository CLI aggregate progress when importing multiple files
Portal Process text for markdown even if it looks like likely xhtml
Portal Show category labels as multilingual strings
Portal Site setting for processing markdown in category labels within codelists
Dataset explore page: variable count and case count on the Dataset Info page
Repository set application name on postgresql db connections
Portal: Remove indirect jstree references where it is not used; reference it where it is used.
Repository return an error on advanced search when it is not enabled
Portal variable concordance view: Don’t include variables from deprecated PhysicalInstances
Repository switch default log level to Warning
Elastic indexer retry failed db calls during second pass item processing
Portal Elasticsearch results: only use content from Highlights when full content is not already present
Portal: When a Description is used for an item link, don’t hyperlink it
Portal: Show ProcessingEvent information directly on StudyUnit page
Portal: translations for AggregationMethod terms
Portal: show some PhysicalInstance properties on the dataset explorer page
Portal: Indicate “unweighted” for statistics that are unweighted
Portal Search page pager: don’t show jump to last button.
Portal Home page item lists: only show the top X items.
Portal: always show VersionDate in the browser’s timezone and locale
Portal: show comment time in the browser’s timezone and locale
Portal: When only one UI language variant is configured as supported, only show the language name
Portal upgrade jquery and jquery validation
Portal upgrade jsTree to 3.3.16
Portal remove tether js
Portal upgrade shepherd to 11.2
Portal upgrade plotly.js to 2.31.1
Portal update intro tours to new version of shepherd
Portal explore view: default to Questions view when it is the only one configured
Portal explore view: In Questions view, also show Question Grids and Question Blocks
Portal show the language for CodeValue properties that have a controlled vocabulary assigned
Fixed¶
Elasticsearch don’t pass an empty byte[] to UTF8.GetString, otherwise that gives an error and nothing useful is logged
Dataset Explore: correctly populate variable groups
Portal data extract: Use prefixed variable names when making SPSS datasets
Repository don’t create local JWT tokens when local JWT auth is disabled
Portal Explore view: make breadcrumb URLs correctly for variable and question lists
Portal: properly group OtherMaterials by their type, instead of one header per material
Portal when using a custom theme, use the default fav icon if a theme specific one is not available
Portal basket: immediately remove items after clicking the remove button.
Colectica Designer¶
Added¶
Designer: new properties on Variable to control whether frequencies are calculated for valid and invalid values
Designer: more efficient variable lookup during statistics calculation
Designer: use less memory for summary statistics generation
Designer add deleting prior versions and prior sets to Repository delete command
Designer: button to remove a tag when viewing an item under a repository
Designer: Provide default dropdown values for CodeValue controls that used to be dropdowns (NumericDomain.Type, DateTimeDomain.Type).
Designer: additional French(-CA) translations
Designer: French translations for dropdowns in variable editor
Designer: Log Off command in File-Logoff to clear active Repository logins
Designer: survey instrument editor: for conditionals and loops, add a button to Delete and leave contents
Changed¶
Designer concordance input sheets: look for datasets directly within Series
Designer: “weight” checkbox should not be present for RepresentedVariable, since the property is not in DDI.
Designer PhysicalInstance editor: save version dates even when no version number is specified. DDI requires the version number attribute, but it is allowed to be blank.
PhysicalInstance PDFs: indicate whether statistics are unweighted, or what the weight variable is.
Designer: use full width for Code List and Classification items
Designer frequency display: correct numeric sorting for code values
Fixed¶
Designer watch AnalysisUnits for dirty changes
Designer Classification editor: don’t show Updates field twice
Designer date editor: bind to correct property for DateOnly
Designer: Correctly watch for changes to Date properties in DateSpecification
Designer: values of custom fields of types MultipleRelationship and Relationship were not correctly displayed
Fix binding for RepresentedVariable.HasRepresentation
Designer publish dialog: show correct language of initial states, when state configuration is available from the repository
Designer Classification Index view: ensure ClassificationItems referenced by index entries are populated, so their title is displayed correctly.
Designer - Data Collection - Collection Event editor: add a horizontal scrollbar when the date range controls are wider than the window
Designer Blaise import: set category label correctly.
Colectica for Excel¶
Changed¶
Excel switch Word to Html report
Excel handle 97-2003 compatibility mode better where max supported columns (256) is less than copy partition size
Excel add assembly resolver hook
Excel reference new pdfsharp
Colectica Workflow¶
Added¶
Workflow favicon
Workflow repository list
Workflow website show admin’s the current Repositories
Workflow website log more possible configuration errors and error conditions
Show View in Portal button in the Requests view
Workflow: on the Requests page, only show Designate button for PublicationRequests
Workflow: on request page, show version date and rationale for the selected item
Workflow: Repository browser: show version number of item, and version number of item at each timeline event
Workflow: show metadata content using the selected language
Workflow: more French Canadian localizations
Workflow: Always show user-local times, and in the same format
Workflow admin: edit email template for decision notifications
Workflow: send notification emails to state change approvers; log when emails cannot be sent due to no email address
Workflow default filters for removing PII information enabled via configuration
Changed¶
Workflow Web Published Items page: show version from the Checkout
Show requester email address in request details
Workflow: Don’t use ServerPreRender
Workflow: tabular display for home page replication status counts
Workflow: always show “None” when no state has been assigned to an item
Workflow: show times in the user’s time zone
Workflow don’t show logout on Windows Auth
Validate state configuration form to ensure tokens are set for all non-None states
Workflow Processes page: show scheduled replication time when one is set
Workflow use new Delete API instead of workspacesync cleaning when removing published sets
Workflow remove all javascript libraries
Workflow Enable item deletion in workflow task runner for removal requests
Fixed¶
Workflow compare against username not name when loading valid approvers
Workflow Web Processes page: show status message correctly
Workflow Web Published page: show “Item not present in [repository]” message correctly
Workflow if the user is not authenticated, don’t display a blank shell of the application and redirect to login
Workflow: properly determine whether a state is the None state, and allow editing the token if it is not the None state
Workflow sanitize email templates
Workflow: Use the correct comment content for Approval email notifications
Workflow handle blank usernames during replication from the Repository
Workflow Web: proper whitespace for “no processing” message
Colectica SDK¶
Added¶
SDK Parallel deserialization of Fragment Instance
SDK don’t wait for deserialization when reading fragment instance from disk
SDK use read-in xml instead of reserializing when making repository items
SDK add NullItemCache class to turn off deserializer item caching
Add extension for typeOfInformationClassification and levelOfInformationClassification to Study Group Subgroup and PhysicalInstance
Add advanced search indexing of typeOfInformationClassification and levelOfInformationClassification extensions
Datasets Column reading for SPSS
Datasets streaming single columns SAS data
Datasets converting SPSS to parquet
Datasets determine column types for csv imports
Datasets Stata using memory mapped file for reading
Datasets reuse buffers when reading Stata
Datasets read single values when doing column reading
SDK add new DDI3.3 GeneralDataFormat to StudyUnit and Group
SDK use extension:GeneralDataFormat to add GeneralDataFormat to DDI 3.2 serialization
DDI 2-to-DDI 3 mapping updates
Support Language for InternationalCodeValue types (Keywords and Subjects)
SDK add TypeOfItem for Group, SubGroup, and Study, new in 3.3
Designer show list of unmatched variables on set source instrument command
Changed¶
SDK update markdig version in reporting nuget
SDK allow lifecycle events without a date
Reporting switch to official PdfSharp for all builds
Reporting log fonts that can’t be loaded
Fixed¶
Designer and SDK: when determining the number of rows in a CSV file, do not map records to objects. This prevents and error when multiple columns have the same name.
Reporting sanitize html codebook output
Fix error when exporting DDI 2 with a contributor without a role
7.3.9742¶
January 2nd, 2024
Colectica Repository and Portal¶
Added¶
Repository indexing for advanced item search on postgresql
Repository automatically manage all databases using EF core and migrations
Elastic indexer option to Disable Dynamic Mapping
Portal dataset explore view UI refresh
Portal show icon for confirmed email addresses in user list
New control to show the Name property
CSS classes for concordance header rows
Concordance view: appsettings configuration to determine whether Variables, Questions, Studies views are available
In concordance views, show variable type icons with tooltips for code list details. Indicate code list uniqueness across variables with a number.
an ID on the concept tree list items
Repository add DecimalPropertyFacet to advanced search request
Repository include versionless code list search for codevalues
Repository database layouts for advanced search
Repository add string interning and reading for advanced search
For static content blocks, allow including HTML from a file
Advanced search text indexing extractor
Configuration for whether to display VersionRationale and VersionResponsibility
Repository feature flag to store based on relationships
Repository respect relationship predicate in adjacency matrix computations
Option to determine whether first visit tours are displayed
Repository allow disabling XForwardedFor checking
Repository CLI reindex advanced search command
Repository CLI migrate database commands
Repository real time elastic indexing of deprecated and deleted items
Repository admin site information backup downloads
Repository Admin setting to disable public basket sharing
Repository allow deleting a user account from the Portal Admin user details page
Repository new portal setting to allow additional html tags on custom pages using AllowAdditionalTagsOnCustomPages
Repository Allow custom html sanitizer for html markdown rendering
Use GET or POST method when searching from the home page, according to the options.
Show Organization.Nickname in portal
Show Group.SeriesInformation on Portal
Portal: Show Software information under DataCollectionMethodology
Portal: Show CollectionEvent.DataSource information
Repository add searching by tag for SQL Server backend
Repository add next result offset search for SQL Server backend
Repository add identifier only search for SQL Server backend
Repository return row ids for items for SQL Server backend
Repository remove old sort orderings for SQL Server backend
Repository remove metadata rank for SQL Server backend
Repository prefix search on SQL Server
Portal update Repository search result positional summary
Portal show DDI format that the item is stored as in Repository
Portal: Include QuestionGrids in the explore/questions view.
Repository config setting to allow skipping smtp certificate validation and revocation check
Replace all versionable lists with versionable tables, showing icon, name, and label
Portal: Add view for ClassificationCorrespondenceTable items
Portal: additional French translations
Portal csv download of code comparison
Changed¶
For concordance tree, consistently show {item name} - {item label} for each node. Don’t show RepresentedVariables with no Variables pointing to them.
Extra logging for ForgotPassword, so it is knowable when and why emails aren’t sent
In multi-language popup, always show all languages, even if they are displayed on the page
Show breadcrumbs according to language settings
Don’t show download button for plotly box plots
Display two decimal places for summary statistics, instead of all of them
Disable option for descending concordance views
upgrade OpenTelemetry
On variable page, show the PhysicalInstance Title in the link to the PI page, not the file location
For multilingual custom fields that have only 1 language with content, show that value
When PhysicalInstance does not have Coverage information, use Coverage information from StudyUnit
In search results, always show Name and Label, or Title, as properties in the card body. Wrap in CSS classes to allow hiding these.
Repository optimized GetTags for Postgresql
Repository don’t use transactionscopes anymore
Repository new GetLatestVersionNumbersPostgresqlAsync using unnest, greatly improves large requests on postgresql
In explore-by-concept view, allow linking to a concept that will be selected on load
Don’t render markdown for variable labels in search results
Repository Upgrade to MailKit from smtpclient
Json use DateTimeZoneHandling.Utc everywhere we use json settings
Concordance displays: when a ConceptualVariable does not have a Label, show the ItemName instead
Repository Set the security stamp timeout to zero seconds
Repository Add configuration switch to use __Host prefix with the cookie name, set RepositorySettings:Http:SetHostCookiePrefix=true
Repository New setting to enable PDF downloads in the Admin->Site Information, a 404 is returned for the pdf if not enabled.
Repository If the feedback email address is empty in the Admin->Site Information, a 404 is returned for the feedback form.
Repository Disable formatting in basket text fields
Repository Option to show generic validation details instead of detailed json validation errors if incorrect json is sent to the API Set RepositorySettings:Features:DisableValidationDetailsReports=true
Repository Added a configuration option in appsettings to disable the swagger ui. Set RepositorySettings:Portal:DisableSwaggerUI=true.
Repository used a default language when a request comes from a non-browser agent without an Accept-Language header
Repository changed status code to 404 when an invalid route is accessed
Repository return a 501 not implemented status code for future APIs that are not yet available
Reports Removed version string from creator tag
Render markdown for custom Explore message
Cross-series concordance: when determining whether a ConVar already exists in a harmonization group, compare IDs instead of locator/name
Upgrade HtmlSanitizer and System.Security.Cryptography.Pkcs to latest versions
PhysicalInstance page: show HumanReadableLocation for public DataFileIdentifications
Show more DataRelationship information directly on PhysicalInstance page
Portal only look at instrument location scheme when an absolute uri
Portal fewer db queries on item page
Custom fields; display the category label when the value is from a controlled vocabulary. Show boolean values.
Log errors when DDI Set generation fails
Portal instrument flowchart: expand response options by default; configuration to make the collapsable.
Portal question grid display: by default, show dimensions and response options linearly; config option to show a table
Portal search results: don’t show question text or description in the result card body if it is already shown in the result card header.
Nicer style for OtherMaterial group headers
Only use metadata languages from config, not the system language
Repository advanced search allow searching by versionResponsibility
Repository advanced search include isPublished, versionDate, and versionResponsibility in search responses when include all fields is specified
Portal VersionableTables: handle questions and variables
Portal: VersionableTable output for Organization and Individual
Portal: make the search error message localizable
Portal: don’t highlight Explore in top nav for the explore-single-dataset page
Repository allow cluster HostId to be a string
Portal concordance view: use letters instead of numbers to represent code list distinctness
Portal: show more results on Explore/Index page
Portal - Instrument Summary: always show question name when it exists (previously it was not shown when a question label existed)
Portal - Instrument Summary: show content in multiple languages
Portal In search, Configure elastic to assign a class to highlights, instead of only using <em>.
Portal: Limit the Markdown extensions to use; don’t get rid of all text inside { braces }
Elasticsearch command line to add and remove tracked sets
Fixed¶
Fix links in Multilingual texts
Fix French translations of “DDI 3.3” to say “3.3” not “3.2”
Repository reuse connection for timestamp cache when in transaction
When using a language from the browser’s Accept-Languages setting, don’t also include it the hiddenLanguages
One-to-many: the one and the many should have the same background shading
Fix overlapping data type icon in concordance table
Portal frequencies display: locate category label correctly for nested codes
Repository when installing postgresql unaccent extension, continue migration even if unable to install. Make it a runtime error within advanced search if requested.
Repository fix recording last delete time upon delete completion
Explore page: make sure Reset button fully resets the dropdown
Explore page item list: set to Variables by default, to avoid an error when clicking Filter before selecting a concept
Explore page: after Filter button press, only call Search() once, via the PageChange event
Proper counting of option aggregates for batched Elasticsearch queries
Correctly show SubTitle instead of Subject
Respect the Smtp:EnableSsl setting
Portal fix creation of expired classification badge when only one side of the date range is present
Portal search page: don’t show “Add to list” when the item type is not allowed in the basket
Portal search results: Don’t show “Log in to add to basket” message when the is actually logged in, but the item isn’t allowed in the basket
Elastic don’t duplicate versionlessParentIds in elastic document when multiple versions of a parent exist
Portal: use correct model type for RecordLayout page
Portal search page: update search syntax examples to be consistent
Portal: Fix spelling error in aria labels
Localization: spelling fixes and use DdiLocalizer instead of PortalLocalizer where appropriate
Portal frequency display: Don’t show “% of Valid” for invalid categories
Repository perform EF upgrade after upgrading a pre-EF database to EF without requiring a restart
Colectica Designer¶
Added¶
QualityStandard command to add ComplianceDefinitions from a ConceptScheme
Designer PhysicalInstance editor: add ProcessingCheck
Designer Quality Statement view: method to remove quality statement items
KindsOfData control for Group view
Designer item picker: support searching for multiple item types
Changed¶
Designer: show text on editor tabs, in addition to the icons. Make the tab control styled like a normal tab control.
Designer check remote version numbers in the background when calling RefreshItemStates from the DesignerApi (Don’t wait for Repository status to speed up navigator)
In Stata import, mark categorical variables as NonAdditive
Designer SKOS import: run on background thread, show progress, and improve performance
Designer: For rich text boxes, show a button that opens a dialog with the markdown preview, instead of using right side pane
Designer when importing summary frequencies from spreadsheets, create managed missing values for blank code values not in the existing codelist
Designer: show the markdown editor buttons again
Do not compute quartiles for non-additive variables in SAS or other row-by-row formats
When populating QualityStatementItems, update ComplianceConcepts to latest versions when new versions are present in the source scheme or standard
When loading an Excel sheet, if only one sheet exists, use that instead of requiring a matching sheet name
Cross series concordance ingest: ignore whitespace around conceptual variable names
In Update from File command, when previously shared code lists now differ, create a new code list for the changed items
In PDFs, don’t show frequencies when there are more than 75 items
Metadata import sheets: create variables that don’t yet exist
For data file imports, make the top level item PhysicalInstance instead of ResourcePackage
Re-use newly created RepresentedVariables during a concordance import
Always populate variables’ code lists during a concordance import
Multilingual control for StatementText
Wider display for classification item values.
Designer: in CollectionEvent list, show the label or name for list items, and update the display immediately after the values are edited.
Designer: show useful list item text for DataCollectionMethodology lists.
Designer: for DateTime values, allow specifying the time.
Designer item population: Use SetPopulator instead of GraphPopulator for performance increase
Designer: use controlled vocabulary resolution for CustomFields with CVs.
DDI 2 When there are no `fileDscr`s, make a PhysicalInstance to use as the default one
DDI 2 Use varGrp/name instead of /id when the /id is not present
DDI 2 Map varGrp/defntn to something
DDI 2 Speed up varGrp child location
Variable Metadata Import: allow specifying multiple groupings for columns prefixed with “group:”
Designer Series editor: when selecting Subseries, allow searching both Series and legacy SubSeries item types
Designer: when saving changes to a legacy SubSeries item, save as a regular Series
Designer navigator: show legacy Subseries items
Designer: Specify supported UI language on startup so WpfLocalizeExtension doesn’t search 800+ cultures, slowly
Designer ConceptScheme view: efficient concept hierarchy creation to make it possible to open large concept sets
Designer ConceptScheme view: show DDI ID of selected concept
Fixed¶
Show all codes on paper forms
Designer upgrade/rewrite item table if local database was upgraded from v5->v6->v7
Designer PhysicalInstance view: ensure “More Storage Locations” editor is not hidden
Designer: fix editors for SoftwarePackage (CodeValue) and ReleaseDate.
Colectica Questionnaires¶
Added¶
Colectica Questionnaires enable building Blaise addin on net7
Questionnaires: Set agency identifier in File - Options
Colectica Workflow¶
Added¶
Workflow Published Items page: show the source and target repositories
Workflow: additional French translations
Workflow repository configuration: allow multiple languages for repository labels.
Workflow Admin state configuration: multiple languages for Family and State labels
Workflow: show version number in Admin - General
Workflow request list: More consistent layout - Workflow request list: show preview indicator for requests that have previews
Workflow Web: use Serilog for logging
Changed¶
Workflow improve default handling of supported ui languages setting
Workflow Upgrade to MailKit from smtpclient
Workflow: Show only “English” instead of “English (United States)” in language picker
Workflow: Create preview request and replication request at the same time; show preview pending state until it is ready
Workflow: Create preview request for state change requests that would trigger a replication when approved
Repository-to-Workflow requests: send repository server ID in the request header
Workflow: remove use off Workflow:ApiKey config value. Always use repository-specific API keys
Workflow API: When a pending replication request already exists, do not add a duplicate
Workflow Web: Use ErrorBoundary to catch top level exception and show nicer error messages
Fixed¶
Workflow: when denying a request, don’t require a message
Workflow admin state page: check ContainsKey before accessing content for a language
Colectica SDK¶
Added¶
Model embed postgresql unaccent rules list
Model utility to replace diacritical marks and some ligatures to Latin-ASCII text
Model unaccent separate surrogates collection to avoid most common char to int casts
IHasCoverage for class that have a Coverage property
Index CodeValue values for advanced search
Index DateSpecification and Date properties for advanced search
Index based on relationships
SDK use new 3.3 Label and Description fields for Question, QuestionGrid, and QuestionBlock
SDK add user attributes to identifiable objects in 3.3 including Access, Code, DataCollectionItem (ModeOfCollection, ActionToMinimizeLosses, CollectionSituation), Embargo, GrossFileStructure, GrossRecordStructure, LogicalRecord, MethodologyItem (DataCollectionMethodology, TimeMethod, SamplingProcedure, DeviationFromSampleDesign), Parameter, RecordRelationship, SpatialCoverage, TemporalCoverage, TopicalCoverage, AggregationVariables,
SDK add TypeOfAccess to Access when using DDI 3.3
SDK add AccessPermissions to Access
SDK add LifecycleEvents to Archive
SDK add new 3.3 versionable type WeightingMethodology
SDK move some child handling to identifiable base since it has custom fields now with relationships
SDK allow Individual references in ArchiveOrganizationReference and ContactOrganizationReference
Support PhysicalInstance on Group
SDK parse DDI urns from strings in addition to identifier triples
Changed¶
SetPopulator instead of GraphPopulator for instruments
SDK improve performance using set based population for children in getitems call
SDK allow multiple AnalysisUnits on Study and Group
Workflow show the user’s own count of pending items on the home page
Fixed¶
Serialization fixes for ResponseText. It doesn’t get wrapped in ResponseDomainInMixed.
SDK handle invalid months in DDI year month to datetime conversion
SDK try to deploy parquet native on 472 Data nuget