Notes and Agreements

As the OSCAL Foundation's FedRAMP Technology Focus Group (TFG) reaches agreements as to the path forward, they are captured here. This is a work in progress. Once the group reaches an appropriate milestone, this will be consolidated and re-organized into more consumable guidance.

Taxonomy

We defined the following DRAFT KSI Taxonomy to align with CR26 terminology and Security Decision Record (SDR) schema fields. We expanded on this where necessary, but avoided deviations.

The taxonomy of a KSI includes:

• Requirement: The required KSI as defined by the PMO in CR26.
• Design: Eplanation of measures, objectives and cycles demonstrating KSIs as required by Consistent with SDR-CSX-KSI, this content is provided by the CSP in the SDR. The design includes:
  • Subject: The capability the KSI's claim is about.
  • Attribute: The specific observable property of that subject being examined.
  • Cycle: The timing pattern governing when data is collected. Expressed either as a frequency or as a trigger or both.
  • Data Source: Where evidence of that attribute is found.
  • Rationale: Why the measurement supports the KSI.
  • Data Interpretation: The correlation logic, thresholds, or other rubric applied to telemetry to produce a finding.
  • Data Types: The format and units of collected data.
• Collection Mechanisms: A description of the automated mechanisms performing the data acquisition, including the method of acquiring the evidence, such as an API call or logging query.
• Evidence:
  • Telemetry: The actual data acquired by the collection mechanisms from its data sources, per the defined Cycles.
  • Finding: The conclusion reached by applying the Design's Data Interpretation rubric to the actual Telemetry. The is the KSI-aligned determination of the subject's perfomance.

Guiding Principles

We defined the following DRAFT guiding principles.

• Subjects: Each KSI must identify at least one assessment ojbect, and may define multiple subjects.
• Data Interpretation: Each KSI's data interpretation must clearly deifne the intended result of data collection, either:
  • a pass/fail determination; or
  • a consistently quantifiable degree of health.
• Data Source Relevance: Each KSI data source must align to the KSI's intended result.
• Collection Mechanism: Each defined KSI data stream must describe its acquisition mechanism.
• DATA QUALITY: Each defined KSI data stream must distinguish base measures, derived measures, and probabalistic measures.
• CYCLES: Each KSI definition must identify collection cycles.

INTENDED RESULT

All KSI telemetry is gathered in support of a clear and specific result.

Pass/Fail: Most compliance frameworks seek to define clear pass/fail criteria against specific controls or requirements, While pass/fail is still a preferred target for FedRAMP KSIs, it is not mandatory.

Degree of Health: FedRAMP KSIs open the door to other quantifiable metrics that can be interpreted to indicate a system's overall cybersecurity "health".

Telemetry reflecting "degree of halth" should include an unambiguous interpretation mechanism whenever practical, such as thresholds.

In some instances, measurements may require organization-specific or system-specific "baselining" before thresholds can be defined. This should be conducted within a defined period of time and adjusted periodically as necessary.

• SUBJECTS: Each KSI must identify at least one primary assessment ojbect, and may define additional primary subjects as well as supporting subjects.

SUBJECT

A Primary subject is the assessment object that is the focus of the KSI. A Secondary subject provides supporting data that aids in assessing a primary subject. There may be more than one Primary and Secondary subjects.

For example, an identity management workflow is a primary subject. Secondary subjects include the account privliges assigned in the identity management system and the log data generated by the workflow. Correlating these two secondary subjects is necessary to determine if the identity management workflow is an effective control.

TIMING AND TRIGGERS

The intervals and/or triggers for KSI telemetry must be included in the KSI definition. KSI collection and delivery mechanisms must be aligned with this definition.

The FedRAMP PMO intends for CSPs to collect KSI telemetry via automation. Defined timing and triggers enable monitoring tools to know when to perform the collection and emit the telemetry. It also allows ingesting tools to know when to expect telemetry.

While defined intervals (frequency) is more predictable and preferred, some collection or telemetry cannot be anticipated and must be handled based on an event (trigger).

DATA QUALITY

ISO/IEC 15939 (software measurement process), which defines:

• Base measure — directly observed, single source, no computation (your "direct read")
• Derived measure — function of two or more base measures (your "calculation")
• Indicator — a derived measure used to support a decision (all KSI drived measures in used to support a satisfaction or health decision)

Agentic AI

• PMO has stated a desire to lean into Agentic AI
• There is a great deal of variance among Agencies on the acceptability of security telemetry, that has been generated or analyzed by Agentic AI.
• There must be a mechanism for clarifying Agentic AI derived telemetry from more deterministic measures.

KSI Approach

• KSI Statement Analysis

  • Identify KSI goal(s)

• KSI Data Requirements:

  • Intent: A simple, unambiguous status. Typically Pass/Fail
  • Decision Point: Sampling or full coverage?
  • Identify evidence to collect in support of KSI goal(s)
  • Define evidence interpretation
    • Evidence type (count, true/false, setting)
    • Collection frequency
    • Evidence fidelity
    • Correlation
    • Thresholds
      • Could be more granular than just pass/fail
      • Example: Satisfactory, Degraded, Critical

• Organizational Considerations:

  • Action Triggers: Define the triggers for that system/org
    • Define required action(s) when triggered

• KSI Techical Collection Approach

  • Identify all evidence source/component
    • Identify the source format(s)
  • Define centralized evidence collection target
  • Define the automation required to acquire evidence from each source/component and deliver to centralied collection target
    • Decision Point: Evidence delivered raw or normalized?

• KSI Technical Interpretation Approach

  • Apply data requirements/thresholds to produce Findings
    • Findings are continuously updating as new data is received and analyzed
  • Raise action triggers when appropriate