BI Blog
AI Bias Audit for Hiring and Candidate Scoring Tools
Will this enterprise deal stall on the bias question?
Are we on the hook under NYC Local Law 144?
Independent bias audits for AI hiring tools
Independent AI bias audit for hiring and candidate scoring tools
We test your tool’s outcomes for adverse impact by sex, race and ethnicity, and explain the results in plain English.
Did our last model update change outcomes?
Has our last bias audit passed the one-year mark?
- Independent: we do not build or sell hiring tools
- Black-box: no model or code access needed
- Fixed price after a short scoping call
Did you know?
-
of 391 employers hiring in New York City had posted a bias audit report
-
potential Local Law 144 compliance issues the NY State Comptroller found across 32 companies
City regulator’s own review: 1
-
the four-fifths rule: a group selected at under 80% of the top group’s rate is generally treated as evidence of adverse impact
Sources: Wright et al., Null Compliance: NYC Local Law 144 (arXiv, June 2024); NY State Comptroller, audit report 2024-N-6 (2 December 2025); EEOC Uniform Guidelines, 29 CFR 1607.4(D).
Few employers can show an audit. Regulators are now checking how the law is enforced.
Sound familiar?
-
Procurement sent an AI risk questionnaire.
What the audit delivers
Independent evidence to attach.
-
Your vendor’s audit pools many employers’ data.
What the audit delivers
Results on your own candidates, alongside it.
-
Half your applicants skipped the race question.
What the audit delivers
Clear handling of unknown data.
-
Legal asked if scores differ by group.
What the audit delivers
Clear impact ratios, plain English.
Independent bias audit vs an internal self-check
Testing your own tool is useful. NYC Local Law 144 asks for an independent auditor.
Internal self-check
- Who runs it
- Your own data or product team
- NYC Local Law 144
- Does not count: the law requires an independent auditor
- Model access
- Full access to code and weights
- What buyers see
- Your own claim about your tool
Independent bias audit
- Who runs it
- An auditor with no stake in your tool
- NYC Local Law 144
- Aligned with NYC Local Law 144 metrics
- Model access
- Not needed: we test score outputs
- What buyers see
- A third party report with method shown
What an AEDT bias audit covers
Your company logo
Independent audit by BI Company
Selection rates for pass or fail tools, and scoring rates for tools that output scores, by sex and by race and ethnicity.
Each group’s rate divided by the top group’s rate. A ratio under 0.8 fails the four-fifths rule of thumb.
Chi-square tests show whether a gap is statistically meaningful or just noise in a small sample.
Sex by race and ethnicity combinations, as the NYC rules require. Very small groups are flagged, not hidden.
No self-reported data? We can infer race and ethnicity with BISG and gender from first names, and state the uncertainty. NYC Local Law 144 tables do not accept inferred data, so there we count unknowns separately.
Illustrative. Each audit includes a written report with statistical findings, fairness metrics and recommendations.
How an AI hiring bias audit works
-
Short call
Scope the tool
Which tool, which roles, which data and which laws matter to you.
-
Minimal fields
Share the data
Score or selection outputs, plus demographic fields or candidate names for inference. Location data helps but is optional.
-
Typically 2 to 3 weeks
We run the tests
Rates, impact ratios, chi-square tests and the four-fifths rule.
-
Plain English
Get the report
Statistical findings, fairness metrics and clear next steps.
Who needs an AI bias audit
-
Will this enterprise deal stall on bias?
What do we attach to the questionnaire?
-
Did the last model update shift scores?
Can we show customers fair outcomes?
-
Is our vendor’s audit enough for us?
What must we publish before we use the tool?
-
Does our screening treat every group fairly?
Could we explain a rejection pattern?
-
Has our data been examined for possible bias?
What evidence will EU buyers ask for?
AI hiring laws at a glance
Of the rules below, only New York City requires a bias audit. Other rules still expect evidence that a tool treats groups fairly. Status as of October 2026.
- NYC Local Law 144
- A bias audit by an independent auditor no more than one year before use. Publish a summary of results and give candidates notice at least 10 business days before use. Penalties: up to $500 for a first violation, then $500 to $1,500 for each later one, and each day of use counts separately.
- Illinois
- Since 1 January 2026, employers may not use AI that has a discriminatory effect in employment decisions, or zip codes as a proxy, and must give notice. AI analysis of video interviews needs notice and consent.
- Colorado
- The 2024 AI Act was replaced in 2026. From 1 January 2027, notice, an explanation after an adverse outcome and human review on request apply to employment decisions made with automated tools.
- EU AI Act
- AI used to filter applications or evaluate candidates is high-risk, and providers must examine data for possible biases. These rules apply from 2 December 2027. Fines can reach EUR 15 million or 3% of worldwide turnover.
- United Kingdom
- No annual audit rule. The ICO expects employers using automated hiring decisions to monitor for bias regularly, be transparent and offer human review.
General information, not legal advice. Laws change often, so check your obligations with qualified counsel.
Sources: NYC DCWP, Illinois HB 3773, Illinois 820 ILCS 42, Colorado SB 26-189, EU AI Act Annex III, EU Regulation 2026/1744, ICO.
From a real audit: 15,265 candidate scores
The client is not named. The illustrative chart shows how impact ratios are read against the four-fifths line.
Illustrative, not client data
- Impact ratio by group
- Four-fifths line (0.8)
- Scope: 15,265 candidate scores from an AI hiring tool
- Access: black-box, score outputs only
- Demographics: BISG for race and ethnicity, name-based inference for gender
- Tests: disparate impact ratios, chi-square tests, four-fifths rule
- Result: no adverse impact found
Inside a real audit report: 81,165 candidate scores
Pages from an August 2025 audit of an AI candidate scoring system. The client name and logo are removed. Click a page to see it full size.
- 81,165historical candidate scores, black-box
- 4bias checks: sex, race and ethnicity, White versus Non-White, intersectional
- 0.802lowest impact ratio, still above the 0.80 line
- Clearno adverse impact found in any check
Trusted by leaders who need answers they can act on
What clients say about working with BI Company analysts.
-
Great experience working with BI. Alina outperformed the project and helped me to take the right decision moving forward.
WasimOnline recruitment platform startup, Canada
-
BI analyzed data that we had collected in a very professional way.
RebeccaThe Lean Marketer LTD, Israel
-
The quality of work is excellent and comparable to top consulting firms.
Wesley TriceDirector of BAYADA, USA
Reviews of BI Company research projects. Read more on our homepage.
600+ clients worldwide, including
AI bias audit pricing
Fixed price
Quoted after a short scoping call
- Price agreed before work starts
- No model or code access needed
- Anonymized sample report on request
- Findings in plain English
The price depends on the tool, the roles covered and the data you hold.
Ask for the sample report in the same form
AI bias audit FAQ
An independent statistical test of whether a hiring or scoring tool’s outcomes differ by sex, race and ethnicity. We compare selection or scoring rates, calculate impact ratios and test whether any gap is statistically significant.
Yes. Employers and employment agencies in New York City may not use an automated employment decision tool unless it had a bias audit by an independent auditor no more than one year before use. They must publish a summary of the results and notify candidates at least 10 business days before use.
Up to $500 for a first violation and for each additional violation on the same day, then $500 to $1,500 for each later violation. Each day a tool is used in violation counts as a separate violation.
No. We test the tool as a black box, using its score or selection outputs plus demographic fields, or candidate names where we infer demographics. Location data helps but is optional.
We can infer race and ethnicity with BISG and gender from first names, and we report the uncertainty. NYC Local Law 144 audit tables do not accept inferred data, so for those we report unknowns as a separate count.
Not by that name. Since 1 January 2026, Illinois bars AI with a discriminatory effect in employment decisions. Colorado adds notice and human review duties from 1 January 2027. Under the EU AI Act, high-risk duties for hiring AI apply from 2 December 2027. An audit gives you evidence for each. This is general information, not legal advice.
It is a fixed price, quoted after a short scoping call. The price depends on the tool, the roles covered and the data. You can ask for an anonymized sample report in the same message.
Need audit evidence before your next deal?
Related services
-
Competitor analysis
Benchmark rival HR tech tools.
-
Market intelligence
Size and track your HR tech market.
-
White label research for consulting firms
For advisers with HR tech clients.
-
White label research for agencies
For agencies serving HR tech brands.
Book a scoping call
Tell us the tool, the roles and the data you hold. Ask for an anonymized sample report in the same message.
