{"@ID": "1434", "@Name": "Insecure Setting of Generative AI/ML Model Inference Parameters", "@Abstraction": "Base", "@Structure": "Simple", "@Status": "Draft", "Description": "The product has a component that relies on a\n\t  generative AI/ML model configured with inference parameters that\n\t  produce an unacceptably high rate of erroneous or unexpected\n\t  outputs.", "Extended_Description": {"xhtml:p": "Generative AI/ML models, such as those used for text\n\t\tgeneration, image synthesis, and other creative tasks, rely on\n\t\tinference parameters that control model behavior, such as\n\t\ttemperature, Top P, and Top K.  These parameters affect the\n\t\tmodel's internal decision-making processes, learning rate, and\n\t\tprobability distributions. Incorrect settings can lead to\n\t\tunusual behavior such as text \"hallucinations,\" unrealistic\n\t\timages, or failure to converge during training. The impact of\n\t\tsuch misconfigurations can compromise the integrity of the\n\t\tapplication. If the results are used in security-critical\n\t\toperations or decisions, then this could violate the intended\n\t\tsecurity policy, i.e., introduce a vulnerability.", "#text_blocks": ["Generative AI/ML models, such as those used for text\n\t\tgeneration, image synthesis, and other creative tasks, rely on\n\t\tinference parameters that control model behavior, such as\n\t\ttemperature, Top P, and Top K.  These parameters affect the\n\t\tmodel's internal decision-making processes, learning rate, and\n\t\tprobability distributions. Incorrect settings can lead to\n\t\tunusual behavior such as text \"hallucinations,\" unrealistic\n\t\timages, or failure to converge during training. The impact of\n\t\tsuch misconfigurations can compromise the integrity of the\n\t\tapplication. If the results are used in security-critical\n\t\toperations or decisions, then this could violate the intended\n\t\tsecurity policy, i.e., introduce a vulnerability."]}, "Related_Weaknesses": {"Related_Weakness": [{"@Nature": "ChildOf", "@CWE_ID": "440", "@View_ID": "1000", "@Ordinal": "Primary"}, {"@Nature": "ChildOf", "@CWE_ID": "665", "@View_ID": "1000"}, {"@Nature": "CanPrecede", "@CWE_ID": "684", "@View_ID": "1000"}, {"@Nature": "PeerOf", "@CWE_ID": "691", "@View_ID": "1000"}]}, "Weakness_Ordinalities": {"Weakness_Ordinality": {"Ordinality": "Primary"}}, "Applicable_Platforms": {"Language": {"@Class": "Not Language-Specific", "@Prevalence": "Undetermined"}, "Architecture": {"@Class": "Not Architecture-Specific", "@Prevalence": "Undetermined"}, "Technology": [{"@Name": "AI/ML", "@Prevalence": "Undetermined"}, {"@Class": "Not Technology-Specific", "@Prevalence": "Undetermined"}]}, "Modes_Of_Introduction": {"Introduction": [{"Phase": "Build and Compilation", "Note": "During model training, hyperparameters may be set\n\t\t\twithout adequate validation or understanding of their\n\t\t\timpact."}, {"Phase": "Installation", "Note": "During deployment, model parameters may be adjusted\n\t\t\tto optimize performance without comprehensive\n\t\t\ttesting."}, {"Phase": "Patching and Maintenance", "Note": "Updates or modifications may be made to the model\n\t\t\tthat alter its behavior without thorough\n\t\t\tre-evaluation."}]}, "Common_Consequences": {"Consequence": [{"Scope": ["Integrity", "Other"], "Impact": ["Varies by Context", "Unexpected State"], "Note": "The product can generate inaccurate, misleading, or\n\t\t\tnonsensical information."}, {"Scope": "Other", "Impact": ["Alter Execution Logic", "Unexpected State", "Varies by Context"], "Note": "If outputs are used in critical decision-making\n\t\t\tprocesses, errors could be propagated to other systems or\n\t\t\tcomponents."}]}, "Detection_Methods": {"Detection_Method": [{"Method": "Automated Dynamic Analysis", "Description": "Manipulate inference parameters and perform\n            comparative evaluation to assess the impact of selected\n            values. Build a suite of systems using targeted tools that\n            detect problems such as prompt injection (CWE-1427) and\n            other problems.  Consider statistically measuring token\n            distribution to see if it is consistent with expected\n            results.", "Effectiveness": "Moderate", "Effectiveness_Notes": "Given the large variety of outcomes,\n            it can be difficult to design testing to be comprehensive\n            enough, and there is still a risk of unpredictable\n            behavior."}, {"Method": "Manual Dynamic Analysis", "Description": "Manipulate inference parameters and perform\n            comparative evaluation to assess the impact of selected\n            values. Build a suite of systems using targeted tools that\n            detect problems such as prompt injection (CWE-1427) and\n            other problems.  Consider statistically measuring token\n            distribution to see if it is consistent with expected\n            results.", "Effectiveness": "Moderate", "Effectiveness_Notes": "Given the large variety of outcomes,\n            it can be difficult to design testing to be comprehensive\n            enough, and there is still a risk of unpredictable\n            behavior."}]}, "Potential_Mitigations": {"Mitigation": [{"Phase": ["Implementation", "System Configuration", "Operation"], "Custom_Phase": {"@Domain": "AI", "#text": "Fine-tuning"}, "Description": "Develop and adhere to robust parameter tuning\n\t\t\tprocesses that include extensive testing and\n\t\t\tvalidation."}, {"Phase": ["Implementation", "System Configuration", "Operation"], "Description": "Implement feedback mechanisms to continuously\n\t\t\tassess and adjust model performance."}, {"Phase": "Documentation", "Description": "Provide comprehensive documentation and\n\t\t\tguidelines for parameter settings to ensure consistent and\n\t\t\taccurate model behavior."}]}, "Demonstrative_Examples": {"Demonstrative_Example": {"Intro_Text": {"xhtml:p": ["Assume the product offers an\n\t\t\tLLM-based AI coding assistant to help users to write code\n\t\t\tas part of an Integrated Development Environment (IDE).\n\t\t\tAssume the model has been trained on real-world code, and\n\t\t\tthe model behaves normally under its default settings.\n\t\t\tSuppose there is a default temperature of 1, with a range\n\t\t\tof temperature values from 0 (most deterministic) to\n\t\t\t2.", "Consider the following configuration."], "#text_blocks": ["Assume the product offers an\n\t\t\tLLM-based AI coding assistant to help users to write code\n\t\t\tas part of an Integrated Development Environment (IDE).\n\t\t\tAssume the model has been trained on real-world code, and\n\t\t\tthe model behaves normally under its default settings.\n\t\t\tSuppose there is a default temperature of 1, with a range\n\t\t\tof temperature values from 0 (most deterministic) to\n\t\t\t2.", "Consider the following configuration."]}, "Example_Code": [{"@Nature": "Bad", "@Language": "JSON", "xhtml:br": null, "xhtml:div": {"@style": "margin-left:1em;", "xhtml:br": [null, null, null, null, null], "#text": "\"model\": \"my-coding-model\",\n\t\t\t\t\"context_window\": 8192,\n\t\t\t\t\"max_output_tokens\": 4096,\n\t\t\t\t\"temperature\", 1.5,\n\t\t\t\t..."}, "#text": "{\n\t\t\t  \n\t\t\t  }", "#text_blocks": ["{", "\"model\": \"my-coding-model\",", "\"context_window\": 8192,", "\"max_output_tokens\": 4096,", "\"temperature\", 1.5,", "...", "}"]}, {"@Nature": "Good", "@Language": "JSON", "xhtml:br": [null, null], "xhtml:div": {"@style": "margin-left:1em;", "xhtml:br": [null, null, null], "#text": "...\n\t\t\t\t\"temperature\", 0.2,\n\t\t\t\t..."}, "#text": "{\n\t\t\t  \n\t\t\t  }", "#text_blocks": ["{", "...", "\"temperature\", 0.2,", "...", "}"]}], "Body_Text": [{"xhtml:p": ["The problem is that the configuration contains a temperature\n\t\t\thyperparameter that is higher than the default. This significantly\n\t\t\tincreases the likelihood that the LLM will suggest a package that did\n\t\t\tnot exist at training time, a behavior sometimes referred to as\n\t\t\t\"package hallucination.\"  Note that other possible behaviors could\n\t\t\tarise from higher temperature, not just package hallucination.", "An adversary could anticipate which package names could be generated\n\t\t\tand create a malicious package. For example, it has been observed that\n\t\t\tthe same LLM might hallucinate the same package regularly. Any code\n\t\t\tthat is generated by the LLM, when run by the user, would download and\n\t\t\texecute the malicious package. This is similar to typosquatting.", "The risk could be reduced by lowering the temperature so that it\n\t\t\treduces the unpredictable outputs and has a better chance of staying\n\t\t\tmore in line with the training data. If the temperature is set too\n\t\t\tlow, then some of the power of the model will be lost, and it may be\n\t\t\tless capable of producing solutions for rarely-encountered problems\n\t\t\tthat are not reflected in the training data. However, if the\n\t\t\ttemperature is not set low enough, the risk of hallucinating package\n\t\t\tnames may still be too high.  Unfortunately, the \"best\" temperature\n\t\t\tcannot be determined a priori, and sufficient empirical testing is\n\t\t\tneeded."], "#text_blocks": ["The problem is that the configuration contains a temperature\n\t\t\thyperparameter that is higher than the default. This significantly\n\t\t\tincreases the likelihood that the LLM will suggest a package that did\n\t\t\tnot exist at training time, a behavior sometimes referred to as\n\t\t\t\"package hallucination.\"  Note that other possible behaviors could\n\t\t\tarise from higher temperature, not just package hallucination.", "An adversary could anticipate which package names could be generated\n\t\t\tand create a malicious package. For example, it has been observed that\n\t\t\tthe same LLM might hallucinate the same package regularly. Any code\n\t\t\tthat is generated by the LLM, when run by the user, would download and\n\t\t\texecute the malicious package. This is similar to typosquatting.", "The risk could be reduced by lowering the temperature so that it\n\t\t\treduces the unpredictable outputs and has a better chance of staying\n\t\t\tmore in line with the training data. If the temperature is set too\n\t\t\tlow, then some of the power of the model will be lost, and it may be\n\t\t\tless capable of producing solutions for rarely-encountered problems\n\t\t\tthat are not reflected in the training data. However, if the\n\t\t\ttemperature is not set low enough, the risk of hallucinating package\n\t\t\tnames may still be too high.  Unfortunately, the \"best\" temperature\n\t\t\tcannot be determined a priori, and sufficient empirical testing is\n\t\t\tneeded."]}, {"xhtml:p": ["In addition to more restrictive temperature settings, consider adding\n\t\t\t  guardrails that test that independently verify any referenced package\n\t\t\t  to ensure that it exists, is not obsolete, and comes from a trusted\n\t\t\t  party.", "Note that reducing temperature does not entirely eliminate the risk of\n\t\t\t  package hallucination.  Even with very low temperatures or other\n\t\t\t  settings, there is still a small chance that a non-existent package\n\t\t\t  name will be generated."], "#text_blocks": ["In addition to more restrictive temperature settings, consider adding\n\t\t\t  guardrails that test that independently verify any referenced package\n\t\t\t  to ensure that it exists, is not obsolete, and comes from a trusted\n\t\t\t  party.", "Note that reducing temperature does not entirely eliminate the risk of\n\t\t\t  package hallucination.  Even with very low temperatures or other\n\t\t\t  settings, there is still a small chance that a non-existent package\n\t\t\t  name will be generated."]}]}}, "References": {"Reference": {"@External_Reference_ID": "REF-1487"}}, "Mapping_Notes": {"Usage": "Allowed", "Rationale": "This CWE entry is at the Base level of abstraction, which is a preferred level of abstraction for mapping to the root causes of vulnerabilities.", "Comments": "Carefully read both the name and description to ensure that this mapping is an appropriate fit. Do not try to 'force' a mapping to a lower-level Base/Variant simply to comply with this preferred level of abstraction.", "Reasons": {"Reason": {"@Type": "Acceptable-Use"}}}, "Notes": {"Note": {"@Type": "Research Gap", "#text": "This weakness might be under-reported as of CWE 4.18,\n\t\t\tsince there are no clear observed examples in\n\t\t\tCVE. However, inference parameters may be the root cause\n\t\t\tfor various vulnerabilities - or important factors - but\n\t\t\tthe vulnerability reports may concentrate more on the\n\t\t\tnegative impact (e.g. code execution) or the weaknesses\n\t\t\tthat the insecure settings contribute to. Alternately,\n\t\t\tdynamic techniques might not reveal the root cause if the\n\t\t\tresearcher does not have access to the underlying source\n\t\t\tcode and environment."}}, "Content_History": {"Submission": {"Submission_Name": "Lily Wong", "Submission_Organization": "MITRE", "Submission_Date": "2024-06-28", "Submission_Version": "4.18", "Submission_ReleaseDate": "2025-09-09"}, "Modification": {"Modification_Name": "CWE Content Team", "Modification_Organization": "MITRE", "Modification_Date": "2026-04-30", "Modification_Version": "4.20", "Modification_ReleaseDate": "2026-04-30", "Modification_Comment": "updated Relationships"}, "Contribution": {"@Type": "Content", "Contribution_Name": "AI WG \"New Entry\" subgroup", "Contribution_Date": "2025-02-28", "Contribution_Version": "4.18", "Contribution_ReleaseDate": "2025-09-09", "Contribution_Comment": "Participated in regular meetings from February to August 2025 to develop and refine most elements of this entry."}}, "entry_type": "Weakness"}