Build 14.0 Plan
The Planetary Data System (PDS) is a long-term archive of digital data products returned from NASA’s planetary missions, and from other kinds of flight and ground-based data acquisitions, including laboratory experiments. But it is more than just a facility - the archive is actively managed by planetary scientists to help ensure its usefulness and usability by the world wide planetary science community.
The following page outlines the planned PDS software enhancements, bug fixes, and impacted requirements for the upcoming Build .
A more detailed breakdown of these tasks by date and priority can be found in our Zenhub Roadmap (login required).
Key Dates
Date |
Name |
Responsible Party |
Description |
|---|---|---|---|
08/14/2023 |
SCR Freeze Date |
DDWG |
Freeze date for identifying SCRs to be implemented in this release. |
08/15/2023 - 09/16/2023 |
Standards Documents Updates Due |
Document Authoring Team |
Updates to PDS4 standards documents due to EN for review. |
10/17/2023 - 11/07/2023 |
Beta Testing Period |
DNs, IPDA |
Changes to PDS4 IM and system components available for beta testing. |
10/17/2023 - 11/07/2023 |
dLDD Integration and Test |
dLDD Stewards |
Auto-generated dLDDs should be reviews by dLDD Stewards, and final updates to dLDDs should be completed during this time. |
09/28/2023 - 10/26/2023 |
Standards Documents Review Due |
Document Review Team |
Review updates to PDS4 standards documents. |
12/01/2023 |
System Release |
EN |
PDS4 system is operationally deployed and website is updated accordingly. |
Date |
Name |
Description |
|---|---|---|
09/12/2023 |
Delivery to I&T |
The PDS4 IM and all software updated during this build cycle are delivered to EN I&T. |
09/25/2023 |
Test Readiness Review (TRR) |
PDS EN internal review to verify readiness to enter I&T phase. |
11/20/2023 |
Delivery and Deployment Review (DRR) |
PDS internal review with PDS Software Working Group to verify readiness to deploy build system. |
12/01/2023 |
Operational Deployment |
PDS4 system is operationally deployed and website is updated accordingly. |
For a more detailed schedule, see https://pds-engineering.jpl.nasa.gov/content/schedules.
Planned PDS4 Information Model Changes
This section details the planned changes to the PDS4 Information Model for this build. These changes are deliverables by the PDS Data Design Working Group (DDWG) to the Engineering Node for implementation.
Planned Software Changes
This section details the planned software changes for this build.
Planned tasks for the next release cycle are grouped by software component, and sub-tasks are included where known at the time of this plan generation. Each section also includes additional information, such as:
Software description
Link to 🔍 User Guide
Link to 💻 Github Repo
Link to 🚑 Issue Tracking
Link to 📒 Backlog - future work not planned for this release cycle
Don’t see the tool you are looking for? This means there is no active development planned for this build cycle. Head to the PDS Tool Registry to see all possible tools available for PDS data.
Have a new feature or tool you think we should be working on? Create a ticket in the applicable Github repo or in our PDS EN Operations repo and we will see how that can fit into our plans moving forward.
validate
Validates PDS4 product labels, data and PDS3 Volumes
- validate#534 B14.0 Content Validation Improvements: Additional Table Types, Additional File Areas
validate#7 Update to support ComplexLSB8 data types and investigate Floating point exception
validate#190 Validation fails to catch real value in ASCII_NonNegative_Integer field (Table_Delimited)
validate#211 Update validate with additional data type support for image transformations
validate#217 As a user, I want to validate content for all possible PDS4 table types
validate#343 As a user I want to see the name of a table/array in errors, if one is specified
validate#431 warning.table.characters_between_fields missing for last record in table
validate#436 Improve error messages for overlapping objects in a label
validate#476 Check for duplicate LIDVIDs in collection inventory
validate#480 Validate does not calculate overlaps correctly when Header is not first object in file
validate#508 CRLF/LF checks do not seem to be working as expected for Table_Character
- validate#606 Support for Encoded Video and Encoded Audio
validate#604 As a user, I want to validate MP4/H.264 encoded video as observational data
validate#605 As a user, I want to validate MP4/H.264/AAC encoded audio as observational data
- validate#607 Support for Improved Datetime Checks
validate#520 Refactor regex handling to use Google’s RE2J library and compile
validate#608 Update datetime regex for content validation
- validate#610 B14.0 Enhancements to Initial Version of Registry Referential Integrity Validator
validate#619 Improve new referential integrity checker timeout tolerance
validate#620 verbosity flag does not appear to output INFO messages
validate#622 Update to use non-zero exit code if any ERROR in execution
validate#643 Document how to use validate-refs tool in Validate Operation User Guide
- validate#629 Add Check for Schematron/Schema Version Mismatch
validate#628 As a user, I want to throw a WARNING when a product’s schematron version does not match the schema version
pds4-information-model
The software tools and data necessary for generating the Information Model including PDS4 ontology, data, and information model.
- pds4-information-model#627 Support LBLX label file extension
pds4-information-model#554 As a user, I want to output a dictionary label with the .lblx extension.
- pds4-information-model#635 B14.0 Develop Planetary Systems Target Ontology
pds4-information-model#634 Iterate and improve planetary systems target ontology
- pds4-information-model#652 B14.0 LDDTool/IMTool Code Refactoring
pds4-information-model#651 Add TermMaps to default output JSON
pds-github-util
github utility functions to enforce the PDS engineering node software life cycle.
- pds-github-util#72 Refactor github-util Into Smaller Subpackages
pds-github-util#73 Define the pds-github-util subpackages
software-issues-repo
Issue tracking repository as a centralized entry point for general PDS software bugs and feature requests.
software-issues-repo#63 B14.0 Release Planning
pds-api
PDS web APIs specifications and user’s manual
- pds-api#237 Support Searching for Past Versions of Products with Differing LIDs
pds-api#148 As a user, I want to search for past versions of a product where the LID changed during the product history
pds-api#255 Enable Authorized Access to Restricted Data
operations
Tickets for the PDSEN Operations Team
- operations#379 Phase 2: Wordpress migration for pds-engineering website
operations#338 Request last re-sync of most recent files then verify
operations#339 Test redirects
operations#343 Install Feedback widget in WordPress PDS Engineering site
operations#362 update information file for Metrics’ Multi-Selection Visualization Chart
operations#391 Migrate and verify latest content for PDS Eng WP
operations#392 Incorporate JPL logo into header for PDS Eng WP
doi-ui
The web interface for the PDS DOI Service providing the ability management PDS archive DOIs. See the DOI Service for more details on the available capabilities. https://nasa-pds.github.io/doi-service/
doi-ui#184 Develop DOI Service User Guide and Training
cloud-tasks
PDS Cloud Migration documentation, issue, tracking and simple tools for assisting in the PDS hybrid cloud study and migration efforts.
- cloud-tasks#29 Migrate Web Analytics prototype to PDS Cloud
cloud-tasks#30 Work with Web Unification team to define system architecture for migration
web-analytics#2 As a data engineer, I want to sync logs from PDS website in an automated fashion
web-analytics#3 Migrate PDS Web Analytics dashboards in PDS-managed AWS environment
cloud-tasks#35 MCP Pilot: Deploy Registry
cloud-tasks#61 Design and Implement Initial File System Viewer
cloud-tasks#64 MCP Pilot: Deploy Nucleus
cloud-tasks#65 Initial NGAP-to-MCP Migration Research
planetarydata.org
Website and related services for the International Planetary Data Alliance, nominally run at https://planetarydata.org/
- planetarydata.org#7 Operational Deployment of new planetarydata.org WordPress site
planetarydata.org#8 Deploy WordPress in an EC2 instance in JPL Cloud
planetarydata.org#9 Migrate existing content from Plone to Wordpress
planetarydata.org#10 Design / select skin for the new website
planetarydata.org#11 Complete review of website with IPDA Website Update team
planetarydata.org#12 Complete URS request for review of website for DNS switch
planetarydata.org#13 Finalize website migration with DNS switch
planetarydata.org#15 Iterate on new planetarydata.org updates per feedback
planetarydata.org#16 planetarydata.org accessibility scan part 2
planetarydata.org#18 Enable public access to Wordpress site for external review
devops
Parent repo for PDS DevOps activities
- devops#37 Automatically deploy an up to date I&T platform where all software can be tested
devops#36 For registry components, add a github action which runs the integration test when something is pushed on a dev branch
registry-sweepers#6 Create the docker image and push it to docker hub as part of the CICD
devops#41 Push docker image on AWS ECR in github-actions, in addition to docker hub
devops#44 Ask new Application developer profile on JPL AWS
devops#45 Have a github action self-hosted runner on NGAP
devops#46 Fully terraform the registry application
nucleus
None
nucleus#17 Deploy Baseline Nucleus for SBN-PSI Catalina Sky Survey Pilot
nucleus#38 Develop Logging and Monitoring Strategy
nucleus#41 Determine DAG definition standard
registry
PDS Registry provides service and software application necessary for tracking, searching, auditing, locating, and maintaining artifacts within the system. These artifacts can range from data files and label files, schemas, dictionary definitions for objects and elements, services, etc.
registry#47 Support LBLX extension in Registry Loader tools
- registry#99 Populate EN Registry with PDS3 Data Sets
registry#62 As a user, I want to ingest legacy PDS3 data sets from EN legacy registry
registry#63 Deploy harvest / registry-mgr as cron to regularly upload PDS EN data to registry
operations#270 Deployment of Registry Loader Tools and Initial Ingestion of Engineering Node Registry on Latest AWS Deployment
registry#168 As a manager, I want to see the progress of data sets ingested into registry vs. legacy registry
- registry#178 Improved Fault Tolerance for Registry and Registry API
registry-common#26 [SECURITY] Fix code scanning alert - TrustManager that accepts all certificates
registry-api#292 Queries for data products with lots of metadata attributes crash in browser
registry-api#297 As a EN Operator, I want to the registry API to be fault tolerant
registry#176 As a EN Operator, I want to the registry to be fault tolerant
registry-api#300 Update Java params and Docker image to expand JVM memory beyond 50%
harvest#119 As a developer, I want to know what version of Harvest was used to load a product
- registry#181 Support Registration and Search of Products Replicated Across Archives
registry#93 As a user, I want to be able to ingest mirrored archive data
registry-api#306 If >1 products with the same LIDVID are registered by different nodes, we should respond with one of them
- registry#185 Implement Registry Multi-tenancy
registry-mgr#66 Update to utilize new multi-tenancy approach
registry#179 Implement authorization handshake with API Gateway and Lambda for multi-tenancy approach
harvest#118 Update to utilize new multi-tenancy approach
registry-api#304 Update to utilize new multi-tenancy approach
validate#621 Re-test validate-refs using populated multi-tenant registry
registry#188 Update Terraform scripts to support multi-tenancy
registry#186 Enhance Terraform Scripts for Continuous Deployment and MCP
registry-api
Web API service for the PDS Registry, providing the implementation of the PDS Search API (https://github.com/nasa-pds/pds-api) for the PDS Registry.
- registry-api#285 Implement Faceting
registry-api#283 As a client developer, I want to facet on 1 or more fields in the registry
registry-api#284 As an API user, I want to know the unique values for a specific API field.
registry-api#323 As a user, I want the /properties endpoint to display counts for products having each property
registry-api#324 As a user, I want the /properties endpoint to display range/enum type and values for each property
registry-api#325 As a user, I want the /properties endpoint to include a link to explanatory documentation (i.e. DD or similar)
data-upload-manager
Data Upload Manager (DUM) component for managing the interface for data uploads to the Planetary Data Cloud from Data Providers and PDS Nodes.
data-upload-manager#4 Develop Cost Model
- data-upload-manager#11 Refine Design and Prototype
data-upload-manager#3 Develop initial design doc
data-upload-manager#5 Develop Ingress Service Routing Logic
data-upload-manager#6 Develop Ingress Lambda Logging Conventions
data-upload-manager#8 Develop Ingress Client Interface
data-upload-manager#12 Develop IaC for Deployment
data-upload-manager#13 Test and Iterate with SBN and CSS
Other Information
Reference Documents
This section details the controlling and applicable documents referenced for this release. The controlling documents are as follows:
PDS Level 1, 2 and 3 Requirements, April 20, 2017.
PDS4 Project Plan, July 17, 2013.
PDS4 System Architecture Specification, Version 1.3, September 1, 2013.
PDS4 Operations Concept, Version 1.0, September 1, 2013.
PDS General System Software Requirements Document (SRD), Version 1.1, September 1, 2013.
PDS Harvest Tool Software Requirements and Design Document (SRD/SDD), Version 1.2, September 1, 2013.
PDS Preparation Tools Software Requirements and Design Document (SRD/SDD), Version 0.3, September 1, 2013.
PDS Registry Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.
PDS Report Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.
- PDS Search Service Software Requirements and Design Document (SRD/SDD), Version 1.0, September 1, 2013.
PDS Search Scenarios, Version 1.0, September 1, 2013.
PDS Search Protocol, Version 1.2, March 21, 2014.
PDAP Search Protocol, Version 1.0, March 21, 2014.
PDS Security Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.
PDS NSSDC Delivery Software Requirements and Design Document (SRD/SDD), Version 0.1, October 29, 2019.
System Requirements, Dependencies, and Assumptions
See individual applications for specific dependencies and system requirements.