Inside Scoop: Draft Guidance for MLMD Application Submissions

*This summary of Health Canada’s guidance document provides compliance details for manufacturers submitting a new or amended application. Under the regulations, this applies to Class II, III, and IV for MLMD. It does not cover the non-ML information required in a medical device license application.*

Artificial intelligence (AI) encompasses algorithms and models that perform tasks like learning and decision-making. Machine learning (ML) is a subset of AI, where algorithms learn from data to create models. When ML is applied to medical devices, they are called machine learning-enabled medical devices (MLMD), subject to regulations.

“Transparency” means clear information about a device’s impact on risks and patient outcomes, which is crucial for safety and informed decisions.

A predetermined change control plan (PCCP) allows Health Canada to manage planned changes to ML systems addressing known risks. Terms and conditions (T&Cs) may be added to medical device licenses to enhance safety.

Health Canada follows the International Medical Device Regulators Forum (IMDRF) terms. An “ML training algorithm” establishes model parameters from data, the “ML model” makes predictions, and the “ML system” is a medical device with ML capabilities, all regulated by Section 1 of the regulations.

Relevant guidance relating to medical devices, including the following:

• Guidance on supporting evidence to be provided for new and amended license applications for Class III and Class IV devices, not including in vitro diagnostic devices (IVDDs)
• Class 3, in vitro diagnostic devices (IVD), new and amendment applications
• Class 4, in vitro diagnostic devices (IVD), new and amendment applications
• Software as a medical device (SaMD): Definition and classification
• Pre-market requirements for medical device cybersecurity – Summary
• Guidance for the interpretation of significant changes in a medical device
• Guidance on clinical evidence requirements for medical devices

Policy Objective:

The policy objective of this guidance is to provide manufacturers with a framework for demonstrating the safety and effectiveness of Machine Learning-Enabled Medical Devices (MLMDs) in various contexts. This guidance applies when manufacturers seek Class II, III, or IV medical device licenses or amend such permits throughout the device lifecycle.

Policy Statements:

1. MLMD Definition: An MLMD can be standalone software meeting the medical device definition or a medical device incorporating software meeting the definition.
2. In Vitro Diagnostic Devices (IVDDs): MLMDs can be either IVDDs or non-IVDDs, with risk classifications ranging from Class I to Class IV.
3. Cover Letter Statements: Manufacturers must explicitly state in their cover letters for Class II, III, and IV MLMD applications that the device employs ML. For MLMDs with a Predetermined Change Control Plan (PCCP), manufacturers must mention the presence of a PCCP in the cover letter to prevent application delays.
4. Classification Justification: Manufacturers should justify the proposed medical device classification of the MLMD, referring to the classification rules in Schedule 1 of the regulations.
5. Regulatory Requirements: MLMDs must meet the applicable requirements outlined in sections 10 to 20 of the regulations. Manufacturers must ensure the availability of objective evidence supporting the device’s intended use, safety, effectiveness, and associated claims.
6. Safety and Effectiveness Assurance: Applications must demonstrate that the MLMD, including the PCCP, if applicable, adheres to safety and effectiveness requirements and maintains a favorable risk-benefit profile for patients.
7. Application Information: Class II, III, and IV applications must include the information in section 32 of the regulations. Health Canada may request additional information during the review process.
8. Evidence Consideration: Health Canada recognizes the diversity of information, methodologies, and evidence manufacturers use to establish MLMD safety and effectiveness. The guidance offers considerations rather than fixed requirements to accommodate different scenarios.
9. Representative Data: Data used by manufacturers should adequately represent the Canadian population and clinical practices, including factors like skin pigmentation, sex differences, and other identity-related variables.
10. Predetermined Change Control Plan (PCCP): Changes made under an authorized PCCP do not require a separate medical device license amendment application. Such changes remain subject to post-market regulatory oversight.
11. Amendments Outside PCCP: For modifications outside an authorized PCCP, manufacturers should consult the regulations and relevant guidance to determine if the change constitutes a significant change requiring a license amendment application.
12. PCCP Submission: Manufacturers may submit a PCCP with applications for new medical device licenses or amendments.
13. Policy Evolution: This guidance represents Health Canada’s current approach, subject to revision as technology advances and regulatory oversight matures. Health Canada remains adaptable to optimize regulatory practices.

Health Canada MLMD Product Lifecycle Components

Health Canada emphasizes the significance of product lifecycle information in establishing the safety and effectiveness of Machine Learning-Enabled Medical Devices (MLMDs). The MLMD lifecycle comprises several key components, including:

1. Design: The initial design phase is where the MLMD’s architecture, features, and functionalities are conceptualized and developed.
2. Risk Management: The assessment and management of risks associated with the MLMD throughout its lifecycle, ensuring patient safety.
3. Data Selection and Management: The careful selection and effective management of data used in MLMD development and operation is vital for accuracy and reliability.
4. Development and Training: The process of training ML algorithms, establishing parameters, and refining the ML model to enhance performance.
5. Testing and Evaluation: Rigorous testing and evaluation to verify the MLMD’s functionality, accuracy, and safety under various conditions.
6. Clinical Validation: Validation through clinical studies and real-world use to confirm that the MLMD meets its intended medical purpose.
7. Transparency: Ensuring clear and comprehensive information about the MLMD is available, contributing to informed decision-making.
8. Post-Market Performance Monitoring: Ongoing monitoring of the MLMD’s performance and safety once it is in the market, including detecting any adverse events or issues.

According to Health Canada’s perspective, these components collectively form the MLMD product lifecycle, which is critical in demonstrating its safety and effectiveness.

Good Machine Learning Practice (GMLP) for MLMDs

Implementing Good Machine Learning Practice (GMLP) is crucial in all Machine Learning-Enabled Medical Device (MLMD) development and maintenance phases. GMLP ensures the creation of safe, effective, and high-quality medical devices.

When applying for an MLMD license, manufacturers should provide evidence of adopting GMLP practices throughout their organization and the product’s lifecycle. This evidence should also include details about quality practices to support changes outlined in a Predetermined Change Control Plan (PCCP).

Predetermined Change Control Plan (PCCP): Concept

A PCCP is a documentation that characterizes the MLMD, its boundaries, intended ML system changes, change management protocols, and impacts. It is an integral part of the device’s design.

PCCPs should be risk-focused, evidence-supported, and provide transparency. They should consider the entire product lifecycle.

Changes listed in a PCCP must ensure the device remains within its intended use. Changes to medical conditions, purposes, or benefits require a separate medical device license amendment application before implementation.

Appropriate changes for inclusion in a PCCP require pre-authorization to address known risks while preserving patient benefits. For instance, changes may aim to maintain or enhance performance, countering the threat of ML performance decline due to environmental alterations or input data variations.

PCCPs facilitate proactive risk management while maintaining rigorous regulatory standards to ensure the safety and effectiveness of the device.

MLMD Risk Management Guidance

Manufacturers are crucial in managing Machine Learning-Enabled Medical Devices (MLMDs) risks. This involves providing descriptions of:

1. Identified Risks and Controls: Manufacturers should detail the risks linked to the MLMD and the measures to mitigate or eliminate these risks.
2. Risk Assessment Technique: The method employed for initial and ongoing risk assessments and the system for categorizing and evaluating risk levels.
3. Risk Assessment Outcomes: The results of the risk assessment process.

The risk analysis should also consider the following factors, where applicable:

• Erroneous Outputs include instances of false positives, false negatives, or incorrect information used for diagnosis or treatment.
• Bias: While SGBA Plus analysis may address some intolerance, manufacturers should consider and manage unwanted bias in MLMDs.
• Overfitting: This issue occurs when a model is overly tailored to specific training examples, making it less applicable to broader problems.
• Underfitting: When a model doesn’t capture all relevant properties of the training population, resulting in limited applicability.
• Performance Degradation: ML system performance can decline due to various factors like demographic shifts, changes in clinical practice, or alterations in input data.
• Automation Bias occurs when users overly rely on device outputs, potentially disregarding contradictory data or human decisions.
• Alarm Fatigue: Users become desensitized to alarms due to excessive exposure, which can lead to missed alarms.
• PCCP-Related Risks: Consider risks associated with Predetermined Change Control Plans (PCCPs) and their impacts on risk management.

For comprehensive risk management in MLMDs, manufacturers should refer to ISO 14971, which guides applying risk management to medical devices.

Our experts at Quality Smart Solutions are here to help and offer medical device-related regulatory advice and support on successfully securing your medical device license.  We can help you by responding to potential information requests, keeping your license updated, and reviewing your device labels (510k Medical Device Registration, Facility Registration & FURLS, IVD Device Registration, and SaMD Classification. 

For more information, click here to read the full guidance document: