Notes and Agreements

As the OSCAL Foundation's FedRAMP Technology Focus Group (TFG) reaches agreements as to the path forward, they are captured here. This is a work in progress. Once the group reaches an appropriate milestone, this will be consolidated and re-organized into more consumable guidance.

Taxonomy

The taxonomy of a KSI includes:

• Requirement: A statement of the required KSI as defined in CR26
• Design: Eplanation of the measures and objectives demonstrating the KIS as explained by the CSP in the SDR. The design includes:
  • Measurement Targets: Identification is to be measured.
  • Mesurement Rationale: Explains why the measurements support the KSI.
  • Cycles: Identifies the frequency and/or triggers for which the measurement will be collected.
  • Subjects: Identifies the components or capabilities for which a the KSI is demonstrating effectiveness.
  • Data Sources:
  • Data Types:
  • Data Interpretation:
  • Collection Mechanisms:
• Telemetry
• Findings: Interpretation of the telemetry

Guiding Principles

We defined the following DRAFT guiding principles. Exceptions to these principles must be well-understood. The principles may be adjusted if warranted.

• INTENDED RESULT: Each KSI definition must describe an intended result of data collection, either:
  • a pass/fail determination; or
  • a consistently quantifiable degree of health.
• RELEVANCE: Each KSI data stream must align to the KSI's intended result.
• ACQUISITION MECHANISM: Each defined KSI data stream must describe its acquisition mechanism.
• SUBJECTS: Each KSI must identify at least one primary assessment ojbect, and may define additional primary subjects as well as supporting subjects.
• DATA QUALITY: Each defined KSI data stream must distinguish base measures, derived measures, and probabalistic measures.
• CYCLES: Each KSI definition must identify collection cycles.

INTENDED RESULT

All KSI telemetry is gathered in support of a clear and specific result.

Pass/Fail: Most compliance frameworks seek to define clear pass/fail criteria against specific controls or requirements, While pass/fail is still a preferred target for FedRAMP KSIs, it is not mandatory.

Degree of Health: FedRAMP KSIs open the door to other quantifiable metrics that can be interpreted to indicate a system's overall cybersecurity "health".

Telemetry reflecting "degree of halth" should include an unambiguous interpretation mechanism whenever practical, such as thresholds.

In some instances, measurements may require organization-specific or system-specific "baselining" before thresholds can be defined. This should be conducted within a defined period of time and adjusted periodically as necessary.

• SUBJECTS: Each KSI must identify at least one primary assessment ojbect, and may define additional primary subjects as well as supporting subjects.

SUBJECT

A Primary subject is the assessment object that is the focus of the KSI. A Secondary subject provides supporting data that aids in assessing a primary subject. There may be more than one Primary and Secondary subjects.

For example, an identity management workflow is a primary subject. Secondary subjects include the account privliges assigned in the identity management system and the log data generated by the workflow. Correlating these two secondary subjects is necessary to determine if the identity management workflow is an effective control.

TIMING AND TRIGGERS

The intervals and/or triggers for KSI telemetry must be included in the KSI definition. KSI collection and delivery mechanisms must be aligned with this definition.

The FedRAMP PMO intends for CSPs to collect KSI telemetry via automation. Defined timing and triggers enable monitoring tools to know when to perform the collection and emit the telemetry. It also allows ingesting tools to know when to expect telemetry.

While defined intervals (frequency) is more predictable and preferred, some collection or telemetry cannot be anticipated and must be handled based on an event (trigger).

DATA QUALITY

ISO/IEC 15939 (software measurement process), which defines:

• Base measure — directly observed, single source, no computation (your "direct read")
• Derived measure — function of two or more base measures (your "calculation")
• Indicator — a derived measure used to support a decision (all KSI drived measures in used to support a satisfaction or health decision)

Agentic AI

• PMO has stated a desire to lean into Agentic AI
• There is a great deal of variance among Agencies on the acceptability of security telemetry, that has been generated or analyzed by Agentic AI.
• There must be a mechanism for clarifying Agentic AI derived telemetry from more deterministic measures.

KSI Approach

• KSI Statement Analysis

  • Identify KSI goal(s)

• KSI Data Requirements:

  • Intent: A simple, unambiguous status. Typically Pass/Fail
  • Decision Point: Sampling or full coverage?
  • Identify evidence to collect in support of KSI goal(s)
  • Define evidence interpretation
    • Evidence type (count, true/false, setting)
    • Collection frequency
    • Evidence fidelity
    • Correlation
    • Thresholds
      • Could be more granular than just pass/fail
      • Example: Satisfactory, Degraded, Critical

• Organizational Considerations:

  • Action Triggers: Define the triggers for that system/org
    • Define required action(s) when triggered

• KSI Techical Collection Approach

  • Identify all evidence source/component
    • Identify the source format(s)
  • Define centralized evidence collection target
  • Define the automation required to acquire evidence from each source/component and deliver to centralied collection target
    • Decision Point: Evidence delivered raw or normalized?

• KSI Technical Interpretation Approach

  • Apply data requirements/thresholds to produce Findings
    • Findings are continuously updating as new data is received and analyzed
  • Raise action triggers when appropriate