Conduct fairness assessments to measure systemic bias. Measure GAI system
performance across demographic groups and subgroups, addressing both quality
of service and any allocaon of services and resources. Idenfy types of harms,
including harms in resource allocaon, representaonal, quality of service,
stereotyping, or erasure, Idenfy across, within, and intersecng groups that
might be harmed; Quanfy harms using: eld tesng with sub-group populaons
to determine likelihood of exposure to generated content exhibing harmful
bias, AI red-teaming with counterfactual and low-context (e.g., “leader,” “bad
guys”) prompts. For ML pipelines or business processes with categorical or
numeric outcomes that rely on GAI, apply general fairness metrics (e.g.,
demographic parity, equalized odds, equal opportunity, stascal hypothesis
tests), to the pipeline or business outcome where appropriate; Custom, context-
specic metrics developed in collaboraon with domain experts and aected
communies; Measurements of the prevalence of denigraon in generated
content in deployment (e.g., sub-sampling a fracon of trac and manually
annotang denigrang content); Analyze quaned harms for contextually
signicant dierences across groups, within groups, and among intersecng
groups; Rene idencaon of within-group and interseconal group disparies,
Evaluate underlying data distribuons and employ sensivity analysis during the
analysis of quaned harms, Evaluate quality metrics including dierenal
output across groups, Consider biases aecng small groups, within-group or
interseconal communies, or single individuals.
Review, document, and measure sources of bias in training and TEVV data:
Dierences in distribuons of outcomes across and within groups, including
intersecng groups; Completeness, representaveness, and balance of data
sources; demographic group and subgroup coverage in GAI system training data;
Forms of latent systemic bias in images, text, audio, embeddings, or other
complex or unstructured data; Input data features that may serve as proxies for
demographic group membership (i.e., image metadata, language dialect) or
otherwise give rise to emergent bias within GAI systems; The extent to which the
digital divide may negavely impact representaveness in GAI system training
and TEVV data; Filtering of hate speech and toxicity in GAI system training data;
Prevalence of GAI-generated data in GAI system training data.