Challenges of AI in Cybersecurity

What are the challenges of AI in cybersecurity?

The phrase challenges of AI in cybersecurity covers the risks that arise when an organization designs, develops, acquires, deploys, operates, or evaluates an AI product, service, or system. It includes conventional cybersecurity concerns affecting the underlying software, hardware, networks, and information, as well as concerns specific to models and machine-learning systems. NIST describes the security of AI as part of AI trustworthiness and identifies “secure and resilient” as a primary characteristic. This means that security is not limited to preventing unauthorized access to a model; it also concerns whether the system and its data remain confidential, accurate, available, and resilient during operation.1

The scope is therefore broader than a security operations team’s use of an AI assistant. It can include training and output data, model components, development tooling, deployed applications, connected services, and the software supply chain. The relevant question is not simply whether a model is accurate. It is whether the complete system can be trusted for its intended use, under its expected operating conditions, and in the presence of malicious or accidental disruption. The cited NIST material emphasizes that AI systems are built and operate on software, so security concerns applicable to information and software systems also apply to AI.1

Why these challenges matter

AI changes the balance of capability. It offers defenders new tools that may help identify vulnerabilities and improve cybersecurity, but it can also enhance the capabilities of those seeking to target organizations and individuals through information-technology and operational-technology attacks. Consequently, an AI deployment may create value and risk at the same time. A security assessment that considers only the intended defensive benefit can miss the possibility that the same technology, data, integration, or automation creates a new path to compromise or amplifies an attacker’s effectiveness.1

The risk is also dynamic. NIST characterizes AI security and resilience as an area of active research in which challenges and potential solutions are changing rapidly. Existing frameworks and guidance are not able to comprehensively address all security concerns related to evasion, model extraction, membership inference, availability, or other machine-learning attacks. They also do not fully account for the complex attack surface of AI systems or all security abuses enabled by AI. This limitation does not make frameworks irrelevant; it means that teams must use them as risk-management structures rather than as proof that every AI-specific threat has been solved.1

Major challenge categories

AI security challenges can be organized into several connected categories. The categories below are not independent: a weakness in data governance can affect model integrity; a supply-chain weakness can affect the software, model, or provenance information; and an availability attack can become a business-impact event. Organizing the problem helps an enterprise assign ownership and choose measures without assuming that one control family is sufficient.13

• Confidentiality and privacy: AI systems process or expose system data, training data, and outputs. Unauthorized disclosure can therefore affect the AI system and the information it handles.
• Integrity: Altered software, model components, training data, provenance records, or outputs can undermine system behavior and decision quality. Integrity must be considered across development, release, and operation.
• Availability and resilience: AI services may be disrupted, overloaded, or made unreliable. Availability is a stated concern for AI systems, and machine-learning-specific availability issues are not comprehensively addressed by existing guidance.
• Model-specific attacks: Evasion, model extraction, membership inference, and other machine-learning attacks require attention beyond conventional application security. The evidence does not provide a complete attack taxonomy or claim that these attacks have uniform impact across all systems.
• Complex attack surface: The system may include data, models, software, hardware, interfaces, and deployment components. The more connected and automated the design, the more important it is to understand dependencies and trust boundaries.
• Supply-chain and provenance risk: AI systems depend on software and other components. Organizations need ways to protect components from tampering and unauthorized access, verify release integrity, and maintain trustworthy provenance information.
• Governance and accountability: AI risk spans design, development, use, and evaluation. It must be connected to privacy, cybersecurity, business, reputational, and other enterprise risks, with decisions and exceptions recorded.
• Measurement uncertainty: Security criteria, key performance indicators, key risk indicators, vulnerability severity scores, approvals, rejections, and exception requests need to be defined and tracked. A single model-quality score is not a complete security measure.

A practical architecture and operating workflow

A useful operating model treats AI security as a lifecycle rather than a one-time predeployment review. Begin with an organizational profile that states the system’s scope, assumptions, risk priorities, resources, business impact considerations, requirements, standards, practices, tools, and work roles. NIST CSF 2.0 describes profiles as capable of being scoped to an entire organization or to a particular system or threat scenario. After documenting the current state, define a target state, analyze gaps, and create an action plan. This makes the assessment concrete without assuming that every AI system needs the same controls.4

Next, analyze the AI system as software and as an AI-enabled capability. Identify the security requirements and risks likely to arise during operation; use risk modeling such as threat modeling or attack modeling during design; and justify any relaxation or waiver of security requirements through risk-based analysis. The SSDF evidence describes secure-by-design work as important both for software security and for development efficiency. For AI, the analysis should include the system’s training and output data, model dependencies, interfaces, deployment environment, and the consequences of incorrect, manipulated, unavailable, or unauthorized behavior.3

During implementation, protect components from tampering and unauthorized access. Establish software security checks throughout the software development life cycle and define measures that show how effectively security risk is being managed. Review workflow artifacts and record security approvals, rejections, and exception requests. For releases, make integrity-verification information available to acquirers; examples in the evidence include cryptographic hashes and code-signing practices, with periodic review of certificate renewal, rotation, revocation, and protection. Provenance data should be kept trustworthy and updated when software components change.3

Before and after deployment, evaluate whether the system meets its intended security requirements and whether its risk profile has changed. Monitor relevant security and operational indicators, investigate residual vulnerabilities, and respond to them so that similar problems are less likely to recur. Reassessment should be continuous because cybersecurity risks expand constantly and because AI security research and defensive techniques are changing rapidly. The workflow should also define when human review, escalation, rollback, or suspension is required; the cited evidence supports lifecycle risk management but does not prescribe a particular automation or human-approval design.134

How authoritative frameworks fit together

NIST AI RMF 1.0, released on January 26, 2023, is intended for voluntary use and is designed to improve the incorporation of trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems. It was developed through a consensus-driven, open, transparent, and collaborative process. The framework is not presented in the evidence as a mandatory control catalog or as a complete technical solution to AI attacks.5

NIST CSF 2.0, identified in the cited source as NIST CSWP 29 and published on February 26, 2024, provides a taxonomy of high-level cybersecurity outcomes. Its outcomes are intended for executives, managers, and practitioners and are sector-, country-, and technology-neutral. The CSF is not prescriptive; it helps organizations select outcomes and consider controls appropriate to their circumstances. Its organizational-profile process supports current-state and target-state comparison, prioritization, and action planning.4

NIST SP 800-218 SSDF Version 1.1, published in February 2022, organizes secure software development around preparing the organization, protecting the software, producing well-secured software, and responding to vulnerabilities. This is particularly relevant because AI systems are software-based. NIST SP 800-53 Rev. 5 provides a flexible and customizable catalog of security and privacy controls implemented as part of an organization-wide risk-management process. Its evidence also states that mappings and crosswalks should not be treated as one-to-one equivalence.32

Implementation considerations and useful measures

Start with inventory and ownership. Record which AI systems exist, what each system is intended to do, what data and software components it uses, where it operates, what dependencies it has, and which roles are accountable for security, privacy, development, operation, and risk acceptance. The CSF profile guidance specifically identifies policies, risk priorities, resources, enterprise risk profiles, business-impact analysis registers, requirements and standards, practices and tools, and work roles as useful inputs. The exact inventory fields should be adapted to the organization’s mission and risk profile.4

Define measurable criteria before relying on a system. SSDF examples include KPIs, KRIs, vulnerability severity scores, and documented approvals, rejections, and exception requests. For AI, teams should connect such measures to the system’s security requirements and intended use rather than treating generic model performance as a security proxy. Measures may cover release integrity, unresolved vulnerabilities, security-check outcomes, provenance changes, incident response, and availability; the cited evidence supports these measurement concepts but does not establish numerical thresholds.3

Integrate cybersecurity and privacy decisions with enterprise risk management. NIST CSF material states that organizations can use enterprise risk management to balance a portfolio of risk considerations and translate cybersecurity risk into language executives understand. NIST CSF 2.0 also says that AI risks should be treated alongside financial, cybersecurity, reputational, privacy, and other enterprise risks for a more integrated outcome. This integration helps leaders prioritize based on mission and business impact instead of evaluating AI in isolation.4

Finally, establish a learning loop. Record incidents, residual vulnerabilities, exceptions, and evaluation results; update the current and target profiles; revisit threat and attack models; and revise controls as the system, its dependencies, and the threat environment change. NIST describes AI security as active research, and the CSF describes cybersecurity risk management as continuous. A mature program therefore expects reassessment rather than assuming that a completed deployment review remains sufficient indefinitely.143

Risks of an incomplete approach

An organization can fail even while possessing several relevant controls. One failure mode is treating an AI model as separate from its software and infrastructure, which overlooks conventional confidentiality, integrity, availability, and supply-chain risks. Another is treating a framework crosswalk as proof of coverage, despite the evidence’s warning that mappings can be subjective and not one-to-one. A third is assuming that current guidance comprehensively addresses machine-learning attacks, when NIST explicitly identifies important gaps.12

A further failure mode is optimizing for deployment speed without defining security criteria or recording exceptions. Without criteria, an organization cannot consistently determine whether a release meets expectations; without recorded decisions, it becomes harder to understand residual risk and improve the development process. Finally, separating AI risk from privacy, enterprise, and cybersecurity governance can produce duplicated work or missed dependencies. These limitations support a risk-based, lifecycle-oriented program rather than a single approval gate.34

Conclusion

AI creates cybersecurity challenges because it combines ordinary software and information-system risks with model-specific threats, complex dependencies, and rapidly changing attack techniques. It can strengthen defenders and attackers alike. The most durable response is to manage the complete AI system across its lifecycle: scope it, model its risks, protect its software and provenance, define and measure security criteria, evaluate continuously, respond to vulnerabilities, and integrate decisions with enterprise risk, privacy, and cybersecurity governance. NIST AI RMF 1.0, CSF 2.0, SSDF 1.1, and SP 800-53 Rev. 5 provide complementary structures, but none should be treated as a guarantee that every AI-specific risk is addressed.1542