Generated text reflects the underlying dataset's biases, which can include historical and systemic bias; often lacks representation from diverse perspectives; and may not reflect the full reality of possible legal outcomes.

Examples:

• GenAI may recommend a harsher sentence for a Black defendant compared to a White defendant with similar circumstances
• Reliance on a dataset of one judge's opinions may disproportionately reflect that judge's legal philosophy in a response
• A question about outcomes based on a case law dataset does not take into consideration outcomes from the large number of matters that settle out of court

It is usually unclear how GenAI tools determine which cases or other sources to reference, summarize, or otherwise bring into a generated response—this is the "black box" of AI.

Examples:

• The system may be biased in favour of certain types of cases or legal arguments over others
• The system may be biased in favour of older cases that are heavily cited versus more recent cases

These tools also typically do not account for contextual information, which means that their output reflects bias due to the narrow information it is drawing on. 

Example:

• A genAI tool that summarizes individual cases may not be able to identify whether that case has been overturned by a higher court because it lacks context

Many legal research tasks do not have one right answer; a series of possible answers or angles to the problem may exist. Even a relatively simple task like summarizing a case reflects a certain bias—both in terms of what aspects of the case to prioritize in the summary, and in the language used to summarize the facts.

Examples:

• One genAI system might summarize a case based on its outcome, while another might focus on the legal test used
• A genAI tool might use language to summarize the actions of an individual in a way that heightens or lessens the severity of those actions

The analysis layer may also be biased based on the language model itself, which may have learned certain assumptions about words and phrases based on its dataset that do not reflect their legal interpretation.

Example:

• A system may predict that the word "disability" is used more frequently in relation to physical disabilities than mental disabilities, and its answer may therefore be skewed to reflect that interpretation of language

A researcher's personal bias (worldview, implicit bias, personal experiences, etc.) will be reflected in the prompt they provide, which will in turn affect what the system provides as output.

Example:

• A researcher might frame their prompt in a way that confirms a preexisting belief that a client is guilty

Automation bias: A researcher may be more likely to trust the output of a system because it comes from a machine. 

Example:

• A researcher might be more likely to believe hallucinated content that matches what they expect to find and neglect to conduct further research