BI Blog

In a 2024 study by researchers at Cornell University, the Data and Society Research Institute, and Consumer Reports, 155 student investigators checked the websites, LinkedIn pages, and job listings of 391 New York City employers for two documents that NYC Local Law 144 has required since July 2023: a published bias audit report and a candidate notice. Of the 267 employers in that sample with open NYC job listings, the researchers found a bias audit report on just 5 percent of them, 14 employers, and a notice on 4 percent (Wright et al., “Null Compliance: NYC Local Law 144 and the Challenges of Algorithm Accountability,” FAccT 2024).

The researchers were careful to note that a missing audit does not prove a violation. A company might not use a covered automated employment decision tool, or might simply be hard to find online. But for a law that has required an independent, annual bias audit since 2023, a 5 percent visible compliance rate is a signal worth taking seriously, whether you are the employer who could face a fine, or the hiring technology vendor whose tool is the one quietly going unaudited.

This page covers what NYC Local Law 144 actually requires, who needs an AI bias audit, what BI Company’s audit methodology looks like, and a real, anonymized example of an audit BI Company completed on an AI-powered hiring platform.

What NYC Local Law 144 actually requires

Local Law 144 applies to employers and employment agencies that use an Automated Employment Decision Tool, or AEDT, to screen or score candidates for employment or promotion within New York City. An AEDT covered by the law must be the subject of an independent bias audit conducted within the year before it is used, and the employer or employment agency must post a public summary of the audit results, typically on its own website. The New York City Department of Consumer and Worker Protection began enforcing the law on July 5, 2023.

The law carries civil penalties for non-compliance, assessed per violation. Beyond the compliance exposure, the research above suggests a second, quieter cost: a missing or unpublished audit is itself a visible gap, one a candidate, a journalist, or a competitor can notice as easily as a researcher can.

Who needs a bias audit

Two groups ask BI Company for this work.

Employers and employment agencies that use an AEDT, resume screener, candidate-scoring system, or interview-analysis tool for New York City hiring or promotion decisions, and need an independent audit to meet Local Law 144’s annual requirement.

AEDT vendors that build and sell hiring or workforce-scoring technology to enterprise customers. A growing number of vendors in this space are starting to treat an independently audited bias result as a trust signal in their own marketing and trust-center pages, the same way a SOC 2 report or a security audit has become expected proof rather than a nice-to-have. An independent audit gives a vendor something concrete to point to when a prospective enterprise customer’s own compliance or legal team asks the question directly.

Beyond New York City, employment AI is drawing regulatory attention more broadly: the EU AI Act classifies employment-related AI systems as high risk, subject to its own conformity and documentation obligations, and a small but growing number of US states and cities are considering or have passed their own AEDT bias-audit requirements. The specifics vary by jurisdiction, so this is not a substitute for legal advice on which rules apply to a specific tool and market, but the direction is consistent: independent testing for bias in employment AI is becoming an expectation, not an edge case.

BI Company’s bias audit methodology

BI Company runs a black-box audit: we test the system’s real, historical outputs rather than its source code, and we do not intervene in or modify the system itself. Audits are run on the client’s own historical data, provided as de-identified scoring records and system configuration data, and analyzed using standard statistical testing software.

Every audit applies the same four tests, each measured against the EEOC’s four-fifths rule (an Impact Ratio below 0.80 signals potential adverse impact) and graded on a three-tier scale: Clear (Impact Ratio 0.80 or above), Consider (0.60 to 0.79), or Concern (below 0.60).

BI Company’s four-part bias audit methodology, applied in every engagement.

Where a system does not directly collect demographic data such as race, ethnicity, or gender, which is common and often deliberate, we infer it using established, disclosed methods: name-to-gender inference for sex, and Bayesian Improved Surname Geocoding, which combines surname and geography, for race and ethnicity. These are standard, published inference techniques used across algorithmic fairness research, not a proprietary black box.

A real audit, anonymized: what a passing result actually looks like

BI Company recently completed an independent annual bias audit of an AI-powered hiring and candidate-scoring platform, performed to support the platform operator’s compliance with NYC Local Law 144. The client and its product are not named here. The engagement is anonymized at the client’s request, consistent with how some of BI Company’s other case studies are published.

The audit covered 81,165 historical candidate-scoring records from a one-week audit window in the summer of 2025, analyzed for sex bias, race and ethnicity bias, the White versus Non-White binary, and every intersectional combination of sex and race or ethnicity. Every test came back Clear. Male and female applicants were selected at statistically equivalent rates (41.0 percent and 41.4 percent, an Impact Ratio of 0.9913). All five race and ethnicity groups tested, Arabic, Asian, Black, Hispanic or Latino, and White, scored above the four-fifths threshold. The White versus Non-White comparison showed a 41.3 percent versus 41.0 percent selection rate, an Impact Ratio of 0.992. Every intersectional subgroup, each combination of sex and race or ethnicity, also scored Clear, with impact ratios roughly in the 0.88 to 1.00 range.

As with any bias audit, a Clear result across every tested dimension means no statistically significant adverse impact was found within the scope of this specific analysis and testing framework. It is evidence the system performed equitably on the data and dimensions tested, not an absolute, permanent guarantee that bias is impossible under every future condition. Read the full, anonymized case study for the complete methodology and results.

Why a passing audit is becoming a credibility asset, not just a compliance box to check

The research cited above shows most employers covered by Local Law 144 have nothing publicly visible to show for it yet. That gap is exactly what makes a completed, independent audit valuable beyond the legal requirement itself. A hiring technology vendor that can point to an audit like the one above, even anonymized, is answering a question enterprise buyers and their compliance teams are increasingly asking before signing a contract. An employer that can point to a public summary is ahead of roughly 95 percent of the companies the 2024 study looked at.

What’s included in a BI Company bias audit

• A black-box statistical audit of your AEDT’s real, historical outputs, no source code access or system changes required
• Testing across sex, race and ethnicity, the White versus Non-White binary, and every relevant intersectional subgroup
• Results scored against the EEOC four-fifths rule and a clear Clear, Consider, or Concern grading
• A decision-ready report formatted for public posting under Local Law 144, or for internal and enterprise-customer use
• A documented methodology your legal team, a prospective customer, or a regulator can review line by line

Frequently asked questions

What counts as an AEDT under Local Law 144?

Broadly, any system that uses algorithms, including machine learning or AI, to substantially assist or replace a human decision in screening, scoring, or ranking candidates for hiring or promotion in New York City. Resume screeners, candidate-scoring platforms, and automated interview-analysis tools are common examples. Whether a specific tool is covered depends on how it is used, which is worth confirming with your legal counsel before scoping an audit.

How often does an audit need to happen?

Local Law 144 requires a bias audit within the year before a covered AEDT is used, which in practice means most employers and vendors treat this as an annual requirement, re-running the audit on updated data each year the tool stays in use.

We are a vendor, not a New York City employer. Do we still need this?

Not under Local Law 144 directly, since the legal obligation sits with the employer or employment agency using the tool. In practice, many vendors commission their own audit anyway, both to support customers who are themselves covered by the law, and because an independent audit result is becoming a trust signal enterprise buyers ask for before they sign.

Does a Clear result mean the tool is permanently free of bias?

No. A Clear result means no statistically significant adverse impact was found within the scope of that specific audit’s data, time period, and testing framework. It is strong evidence, not a permanent guarantee, which is why the law requires the audit to be repeated annually rather than done once.

Do you sign an NDA and keep the audit confidential?

Yes. The audit itself and the underlying data stay confidential. What gets published, whether as your own Local Law 144 summary or in case-study form, is agreed with you in advance.

Talk to us about a bias audit for your AEDT

Talk to Us About a Bias Audit

See the full AI bias audit case study for the complete methodology and results from a real engagement.