Algorithm Selection for PQC

Why PQC algorithm selection is a systems decision

Post-quantum cryptography (PQC) is not one algorithm and it is not a drop-in replacement for every existing public-key operation. Traditional algorithms such as RSA and elliptic-curve cryptography can be used for both key-establishment and signature purposes, whereas the cited engineering guidance describes current PQC choices as function-specific. When upgrading a protocol, an existing key-agreement or key-transport use should be replaced with a PQC KEM, while an existing authentication or signing use should be replaced with a PQC signature method. KEMs also have a different interface from traditional key agreement and key transport, so protocol or application changes may be required.12

The practical starting point is therefore a requirements statement: what must the operation accomplish, who verifies or receives it, what objects cross the network or are stored, what security category is required, and which standards or approvals apply? The answer should be recorded before benchmarking. This avoids selecting an algorithm because it performs well in an isolated test while overlooking a protocol packet limit, a validation requirement, or a platform that cannot safely support the implementation.

Start with the approved function and standard

The principal NIST PQC standards released in August 2024 specify ML-KEM in FIPS 203, ML-DSA in FIPS 204, and SLH-DSA in FIPS 205. NIST states that these two digital-signature standards and the KEM standard are expected to provide the foundation for most deployments and that they can and should be put into use now. This makes them the appropriate baseline for a production selection exercise when the relevant function and deployment requirements are satisfied.2

ML-KEM, ML-DSA, and SLH-DSA should be compared only within their distinct functions. ML-KEM addresses key establishment. ML-DSA and SLH-DSA address digital signatures. A procurement or architecture decision should document this boundary explicitly, because replacing a KEM with a signature algorithm—or a signature algorithm with a KEM—does not preserve the original security function.345

The status of an algorithm matters. NIST reports that Falcon and HQC were selected for ongoing standardization and that additional digital-signature schemes are being considered as possible backups to ML-DSA or for unique use cases. That status is materially different from the finalized FIPS standards. An organization can monitor such candidates and design for agility, but it should not silently present an experimental, additional, or still-under-standardization candidate as the production default.2

Evaluate security category, parameter sets, and object sizes

After identifying the function and standard, select a security category or parameter set appropriate to the application’s policy and data classification. NIST’s FIPS 203 material identifies ML-KEM-512, ML-KEM-768, and ML-KEM-1024 as security categories 1, 3, and 5, respectively. The same material recommends ML-KEM-768 as a default because it offers a large security margin at a reasonable performance cost, while noting that another parameter set may be appropriate when performance makes it impractical or when higher security is required. That recommendation is not a universal mandate: the application’s requirements remain decisive.3

Security category is only one dimension. Larger keys, ciphertexts, signatures, or internal working objects can affect packet size, storage, certificate handling, memory, hardware interfaces, and latency. The cited migration guidance warns that PQC algorithms may require larger sizes and different interfaces, while the hybrid-deployment evidence notes that algorithm objects can range from hundreds of bytes to more than one hundred kilobyte. Measure the complete protocol message and certificate or key-management object, not merely the primitive’s benchmark output.678

For ML-DSA, FIPS 204 gives the following private-key, public-key, and signature sizes: ML-DSA-44 uses 2,560-byte private keys, 1,312-byte public keys, and 2,420-byte signatures; ML-DSA-65 uses 4,032-byte private keys, 1,952-byte public keys, and 3,293-byte signatures; and ML-DSA-87 uses 4,896-byte private keys, 2,592-byte public keys, and 4,595-byte signatures. These values illustrate why a security-category decision must be tested against certificates, signed artifacts, message formats, and storage constraints.4

For each candidate, record the acceptable security category, public-key and private-key sizes, ciphertext or signature size, maximum message or certificate size, and any limits on state, randomness, or key reuse. Then test those values in the real protocol. A choice that passes a cryptographic test but causes fragmentation, rejected certificates, storage exhaustion, or incompatible field sizes is not a deployment-ready choice.678

Assess performance, implementation assurance, and platform support

Performance is application-specific. The evidence states that PQC performance can vary significantly with the algorithm, platform, and implementation approach. Measure key generation, encapsulation and decapsulation, signing and verification, memory use, concurrency, energy, and tail latency on the actual software, firmware, hardware, and workload. Include startup, key rotation, certificate validation, and failure-handling paths where they affect service behavior.8

Implementation assurance is separate from mathematical security. FIPS 204 and FIPS 205 permit implementations in software, firmware, hardware, or combinations of these and describe validation programs for conformance. NIST SP 800-227 warns that implementation mistakes can lead to vulnerabilities or loss of usability, and requires attention to approved cryptographic elements, parameter validity, randomness, and applicable FIPS 140-3 guidance. Therefore, selection should include a credible implementation, secure randomness, side-channel considerations, independent review where appropriate, test vectors, conformance testing, and a path to the required validation.965

Platform support must be checked at every layer: cryptographic libraries, operating systems, hardware security modules, trusted platform modules, accelerators, certificate systems, protocol stacks, and application APIs. The migration evidence specifically identifies the need to update libraries and hardware for larger key sizes and different computational requirements, and to optimize implementations while protecting against side-channel attacks. A selected standard is not enough if the organization cannot deploy it safely on its supported platforms.6

Check protocol constraints and interoperability

Protocol fit can eliminate an otherwise attractive option. The cited ETSI evidence describes IKEv2 constraints in which large post-quantum public keys may not fit the available space without fragmentation. It also notes that hybrid exchanges can add messages or round trips, and that bandwidth and latency tolerance depends on the system and network. Test negotiation, packetization, fragmentation, retransmission, certificate exchange, middleboxes, and downgrade resistance with every relevant peer.8

Interoperability is not merely the ability to encode an algorithm identifier. Both endpoints must agree on the same standard, parameter set, object format, API behavior, and negotiation rules. Applications and services may need code refactoring, extensive testing, protocol updates, and interface redesign to accommodate PQC. Record which peers are standardized, which are legacy, and which combinations are actually tested rather than assuming that two implementations with the same algorithm name are interchangeable.6

A hybrid design may be justified when a deployment needs transitional interoperability, must retain a traditional algorithm for protocol or accreditation reasons, or has a defined defense-in-depth objective. NIST states that applications must decide whether they can afford the implementation cost, performance reduction, and engineering complexity of hybrid key establishment or dual signatures. Hybrid mechanisms also require correct secret combining or additional signature-verification logic; errors in either can weaken the design.108

Do not use “hybrid” as a substitute for an architectural rationale. The evidence describes different hybrid goals, including confidentiality, authentication, interoperability, and implementation security, and notes that one hybrid scheme cannot achieve every property. ETSI also cautions against deploying PQC algorithms that have not undergone standardization or sufficient analysis, even inside a hybrid. Define the property being sought, specify the combiner or verification construction, protect negotiation against downgrade, and test the full implementation.108

Include validation, regulation, and lifecycle planning

Regulatory and accreditation requirements can change the feasible choice. The cited material notes that FIPS 140-3 validation requires approved public-key algorithms in the relevant context, while a PQC algorithm may be included in a FIPS-compliant hybrid with an approved traditional algorithm. NIST’s FIPS standards also require approved cryptographic algorithms and techniques for implementations complying with those standards. Confirm the exact scope of the applicable requirement with the responsible validation and compliance teams; do not infer that a hybrid is automatically acceptable everywhere.8195

Plan for lifecycle rather than selecting once. Record the standard and document version, parameter set, implementation version, validation status, supported peers, replacement procedure, and cryptographic inventory location. NIST’s crypto-agility work describes operational mechanisms for changing cryptographic algorithms while preserving security and ongoing operations. In practice, agility means algorithm identifiers and policies can be changed without rewriting every application, and that systems can test, negotiate, monitor, and roll back changes safely.112

A lifecycle plan should also address long-lived data and the possibility of future cryptanalytic or implementation findings. The NIST project describes additional standards as backups or alternatives, while the evidence on hybrid deployment notes that new PQC implementations may contain bugs and that traditional components may remain necessary because of regulatory constraints. This supports a measured approach: deploy finalized standards, maintain inventory and migration capability, monitor standardization and security developments, and define transition triggers.211

A repeatable selection workflow

1. Define the operation: key establishment, digital signature generation, signature verification, or more than one distinct use.
2. Identify the approved standard and deployment status. Start with ML-KEM for KEM use and ML-DSA or SLH-DSA for signature use; document any reason to consider another candidate.
3. Set the required security category from policy, data classification, threat model, and lifecycle. Do not choose a level solely from a performance chart.
4. Model keys, ciphertexts, signatures, certificates, messages, storage, memory, bandwidth, and latency in the real protocol.
5. Benchmark the complete implementation on each target platform, including failure paths, concurrency, energy, and operational tasks.
6. Review implementation assurance: randomness, side-channel defenses, independent review, conformance tests, validation pathway, and secure key handling.
7. Test interoperability with every required peer, including negotiation, downgrade protection, packet limits, fragmentation, certificates, and legacy behavior.
8. Decide whether a hybrid is necessary for a documented security, interoperability, protocol, validation, or regulatory reason, and quantify its overhead.
9. Record lifecycle controls: inventory, algorithm agility, update and rollback procedures, monitoring, revalidation, and triggers for moving to another approved or future standardized option.

The result should be a decision record, not just an algorithm name. It should state the function, standard and version, parameter set, security rationale, measured size and performance limits, implementation and validation evidence, interoperability scope, regulatory assumptions, hybrid rationale if any, and lifecycle owner. This record makes later review possible when platforms, peers, standards, or threat assumptions change.112

Conclusion

The right PQC algorithm is the one that satisfies the required function and approved standard while fitting the application’s security, protocol, size, performance, implementation, interoperability, validation, lifecycle, and regulatory constraints. ML-KEM, ML-DSA, and SLH-DSA are complementary standardized choices, not competing universal winners. Begin with a function-specific baseline, test it in the complete system, use hybrids only for a defined transitional or assurance need, and keep the architecture agile enough to accommodate future approved alternatives.12