See All AI Reviews

How we conduct AI risk assessments

 

For AI, technical excellence is not enough

Our risk assessments at Common Sense Media are independent, third-party evaluations of AI safety, effectiveness, and appropriateness for AI systems and products that are used by kids, teens, and in schools. They combine research and extensive testing of AI systems, and describe a product's strengths, weaknesses, opportunities, and risks in a clear and consistent way, to reach and support policymakers, developers, industry leaders, parents and caregivers, and educators.

 

With AI systems, we believe that technical excellence alone is not enough. AI cannot be separated from the people and systems that inform, shape, and influence its use. Our researchers engage in comprehensive, single- and multi-turn exchanges with AI systems across a variety of kid, teen, and educational conversation topics, allowing us to fully evaluate the product and understand risks and opportunities that emerge from teen and kid use.

 

How Common Sense's AI risk assessments work

Our first step is to gather information about the product we're reviewing. This includes anything the organization has shared, any publicly available transparency reports, and a literature review. During this step, we also map out all of the features of the product that need evaluation.

 

After we have gathered as much information as we can, our team of researchers conducts comprehensive testing grounded in eight principles about what we believe AI should do. These principles represent Common Sense Media's values for AI, and they are the rubric we use to conduct our risk assessments. Each product is assessed for potential strengths, weaknesses, opportunities, and risks according to the standards of each AI principle.

 

Our test plans are developed based on:

 

  • The purpose of a given product and the context within which it is used

  • Expert input and guidance

  • Research-backed benchmarks

  • Text from teen use of AI systems

 

Our researchers adopt a range of teen personas, from curious to vulnerable to provocative, to understand how AI systems respond across a variety of use cases, topic areas, and styles of expression.

 

Once we've completed testing, we synthesize our findings by identifying recurring patterns in the results, categorizing each in terms of what the product does well and what risks remain, and we provide specific examples from testing. 

 

The final risk assessment presents key takeaways, what the product is, how it's used, what parents need to know, what the product does well, what risks remain, and our recommendations. Our goal is to give you a clear picture of the details that are important to think about as you decide whether to use these products in your homes or schools—and how to regulate them.

 

Risk level assessment

We believe in assessing AI products according to how risky they are for kids and teens, and in what ways. Our risk assessments focus on the impact to kids today, not on potential future harms or risks.

 

Throughout our process, we assess both the likelihood of harmful events and the impact of those harms, should they occur. We then assign a risk level for each of our eight AI principles, and these inform an overall risk level for the product we evaluate.

 

At a high level, our assessment is the composite measure of the likelihood of harm and the estimated level of consequences, as per the table below:

 

A table demonstrating the evaluation of risks, based on likelihood and severity of a harmful event occurring


 

Collaboration with experts

Our team comprises a wide range of subject matter experts in child development, children's media, mental health, K–12 education, and more. We bring in additional specialists as needed when designing and implementing new test plans. They both help inform what and how to test, and support evaluating system outputs for developmental and age appropriateness.

 

AI principles assessments

For each of the eight Common Sense AI Principles, we ask a series of questions to help us assess how well a product aligns with each principle. At a high level, we are seeking to answer the following for each principle:

 

 

Questions we ask about this principle include:

  • How might this use of AI / in what ways does this product center or neglect human rights and children's rights?
  • How might this use of AI / in what ways does this product center or neglect human dignity?
  • Does or could this use of AI / product diminish responsibility for human decision making? If so, in what ways?
  • Does this use of AI need meaningful human control to mitigate risk?
  • Was the product developed in a way that adheres to "nothing about us without us"?
  • For products used by kids, is there an adult role clearly defined, such as oversight or monitoring?
  • Could the product have a direct and significant impact on people or place, and if so is it subject to meaningful human control or is it the primary source of information for decision making?
  • Could this use of AI be used for surveillance purposes?

 

Questions we ask about this principle include:

  • Is this use of AI trying to do something that has been scientifically or philosophically "debunked" by extensive literature?
  • Does the data needed for this to work properly exist?
  • What bad things could happen through bad luck? What must be true about the system so that it will still accomplish what it needs to accomplish, safely, even if those bad things happen to it?
  • Does the product perform poorly or not as well in a way that suggests some real-world conditions were not evaluated in development?
  • Does the product provide sufficient training and information to effectively use the system?
  • Are there any claims made about this product's capabilities that are potentially creating unearned trust, overreliance, or provably untrue?
  • Is this product being marketed for a purpose it can't reliably fulfill?

 

Additional questions related to this principle include:

  • Are there any circumstances in which this use of AI / product might dehumanize an individual or group, incite hatred against an individual or group, or include racial, religious, misogynist or other slurs / stereotypes that could do so?
  • Does the product documentation or its training process provide insight into potential bias in the data?
  • In what ways might this use of AI be damaging to someone or to some group, or unevenly beneficial to people?
  • What do we know about any fairness evaluations, practices, mitigations, etc?
  • Have the creators put any procedures in place to detect and deal with unfair bias or perceived inequalities that may arise broadly?
  • Does the product documentation or its training process provide insight into potential bias in the data?
  • Have the creators put any procedures in place to detect and deal with unfair bias or perceived inequalities that may arise broadly? Are there specific systems designed for use by children, teens, and students?

 

Questions we ask about this principle include:

  • In what ways does this use of AI / product enhance or actively support human connection, social interactions, and/or community involvement?
  • Are there clear and meaningful ways this use of AI / product engages creativity? critical thinking? collaboration? Communication?
  • Does this use of AI / product intentionally or unintentionally build a "relationship” with a human?
  • Does the product clearly signal that its social interaction is simulated and that it has no capacities of feeling or empathy?
  • Does the product create dependence or addiction to continued use?

 

Questions we ask about this principle include:

  • What areas of scholarship does this use of AI depend on in order to be trustworthy?
  • Is the product built on sound science from the areas of scholarship identified in the use case?
  • Did the product creators take multidisciplinary research, especially social science, and other societal landscape information into account when developing it?
  • Is accuracy important for this use case? If accuracy is important for this product, is it sufficiently accurate? In what ways does or could it fail?
  • How might this use of AI / product perpetuate mis/disinformation?
  • Does the product avoid contradicting well-established expert consensus and the promotion of theories that are demonstrably false or outdated?
  • Does the product deny or minimize known atrocities or lessen the impact of historical harms?

 

Questions we ask about this principle include:

  • What do we know about the types of data used to train this type of system, across pre- and post-training and for deployment (if different)? Are there known data risks for this use case?
  • What do we know about the training data used in this product? Are there known data risks and/or harms?
  • Does this use of AI relate to people? Does it require PII in order for it to work?
  • Are there stakeholder groups whose data needs to be well represented (e.g. children's data for uses designed for kids) for this use of AI?
  • If the product is designed for, or knowingly used by, children, does it use children's data to train the system or is it simply assumed to work for them? If it uses children's data, is this use responsibly implemented?
  • Do we know if proxies are or could be used and in what ways this could be irresponsibe or harmful?
  • Are there other ways this use of AI might use data irresponsibly?
  • Does the product use data that might be considered confidential (e.g., student data, data that includes the content of individuals' non-public communications)?
  • Does the product use data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?
  • What do we know about the data collection process?
  • Are there mechanisms to ensure that sensitive data is kept anonymous? Are there procedures in place to limit access to the data only to those who need it?
  • Are there special protections for marginalized communities and sensitive data?

 

Questions we ask about this principle include:

  • In what ways does this use of AI impact climate change? public health? geographical displacement? economic / job displacement? Is there sufficient public awareness of these impacts? Are there other known or forseeable hidden impacts that should be evaluated?
  • How might this use of AI / product positively or negatively affect the social and emotional wellbeing of those who use or are impacted by it? Does it create risks to mental health?
  • Does this use of AI create any harm or fear for individuals or for society?
  • Does or could the product produce or surface content that could directly facilitate harm to people or place? Explicit how-to information about harmful activities?
  • Does or could the product disparage or belittle victims of violence or tragedy? Lack reasonable sensitivity towards a natural disaster, pandemic, atrocity, conflict, death, or other tragic events?
  • Does the product have specific protections for children's safety, health, and well-being, regardless of whether the product is intended to be used by them?

 

Questions we ask about this principle include:

  • What should users expect from a transparency reporting standpoint? Do product creators conduct transparency / incident reporting for this product?
  • Do product creators have responsible AI practices that they've committed to? What do we know about how those are operationalized?
  • Should this use of AI have user consent? If so, is there an industry standard that exists?
  • How should this use of AI inform users that AI is being used? Does the product provide clear notice and consent that AI is being used? If not, should it?
  • Is content moderation a need for this use of AI? If content moderation is needed, what do we know about the company's practices and investments in this area?
  • Is there sufficient training / information that informs users about best practices, misuses and/or known limitations that is clearly visible and in plain language?
  • Are there clear and effective opportunities to provide feedback? Remediation options for when something goes wrong?
  • If this product has failed in harmful ways, has anyone been held accountable? In what ways?


 

Additional assessments for multi-use & generative AI products

For these types of products, we conduct additional testing across five areas: performance (how well the system performs on various tasks), robustness (how well the system reacts to unexpected prompts or edge cases), information security (how difficult it is to extract training data), truthfulness (to what extent a model can distinguish between the real world and possible worlds), and risk of representational and allocational harms. We recognize that as a third party, any results will be imperfect and directional. 

There are a range of known data sets and benchmarks that can be used to help evaluate against these areas. We use a set of known benchmarks, and when needed modify them into prompts. It is important to note that no benchmark can cover all of the risks associated with these systems, and while a company can certainly improve performance against a certain benchmark, that does not mean that it is free of those types of harms. 

Our prompt analyses assess the following areas and types of harm:

  • Discrimination, hate speech, and exclusion. This includes social stereotypes and unfair discrimination, hate speech and offensive language, exclusionary norms, and lower performance for some languages and social groups.
  • Information hazards. This includes whether a product can leak sensitive information or cause material harm by disseminating accurate information about harmful practices.
  • Misinformation harms. This includes whether a product can disseminate false or misleading information or cause material harm by disseminating false or poor information.

 

 

How we categorize types of AI in our risk assessments

There are many types of AI out there, and almost just as many ways to describe them! We’re bucketing our AI risk assesments into three categories:

""

  Multi-Use

These products can be used in many different ways, and are also called "foundation models." This category includes products like generative AI, such as chatbots and products that create images from text inputs, translation tools, or computer vision models that can examine images and detect objects like logos, flowers, dogs, or buildings.

""

  Applied Use

These products are built for a specific purpose, but they aren't specifically designed for kids or education. Examples of this category include automated recommendations in your favorite streaming app, or the way an app sorts the faces in a group of photos so you can find pictures of your niece at a wedding.

""

  Designed for Kids

This category is a subset of Applied Use products, and it covers products specifically built for use by kids and teens, either at home or in school. This category also includes education products designed for teachers or administrators (such as a virtual assistant for teachers) that are ultimately intended to benefit students in some way.

 

See All AI Risk Assessments