NY City Council Rams Through Once-Promising but Deeply Flawed Bill on AI Hiring Tools

New York City became the first place in the U.S. this week to pass significant legislation aimed at mitigating the risks of discrimination associated with the use of automated employment decision tools. Unfortunately, Introduction 1894 (I.1894), the once-promising bill the City Council adopted, is deeply flawed. Many of its most harmful provisions were introduced and rammed through in a rushed process that excluded workers, civil rights groups, and other stakeholders from providing any input.

The Council’s decision to pass this weakened bill without consulting impacted stakeholders should worry workers, advocates, and the broader public in New York City and the rest of the country alike.

In recent years, employers have increasingly used automated tools when making crucial employment decisions. This trend has generated a wide-ranging conversation about how these technologies can discriminate against candidates, especially people with disabilities, capturing the interest of employers and workers, civil society organizations, and recently prompting new initiatives from White House and U.S. Equal Employment Opportunity Commission. The New York City bill could have been a model for jurisdictions around the country to follow. Instead, it is a missed opportunity that fails to hold companies accountable, and leaves important forms of discrimination unaddressed.

Below, we delve into the concerns about the New York City legislation with the hope that the Council will amend the bill–or, at the very least, that other policymakers avoid the bill’s harmful flaws.

Background on I.1894: How a Bill Should Not Become a Law

The original version of I.1894 was introduced in February of 2020. The Council’s Committee on Technology held a hearing on the bill last November, where a number of civil rights and public interest organizations raised a variety of concerns, particularly with respect to the vagueness of the bill’s audit requirements, the inadequacy of its notice provisions, and the absence of strong enforcement mechanisms. Our earlier analysis of the original bill noted the bill’s failure to address the high potential for automated tools to worsen discrimination against disabled workers. CDT and a coalition of disability advocacy and other civil society organizations laid out for the Council the ways in which the draft bill failed to protect disabled workers.

Given those concerns, we were initially hopeful when we saw that a revised version of the bill had landed on the Council’s website on Tuesday. Perhaps the Council had heeded our recommendations to strengthen the bill’s auditing provisions, give candidates a meaningful opportunity to raise concerns about the bill, and shore up the bill’s regulatory and enforcement provisions? To our dismay, the revised bill was, if anything, more flawed than the original, with a narrower scope and weaker audit requirements and enforcement provisions, as we discuss below. But CDT and other interested parties never had an opportunity to raise the alarm about these weakening changes because of how rapidly the revised bill was adopted and passed.

On the Council’s website, the revised bill (download) is dated November 9th, accompanied by Mayor Bill de Blasio’s “message of necessity” (download) (also dated November 9th) certifying the need for the Council to pass the draft ordinance “immediately.” The Committee held a “hearing” on the revised bill the very next morning. The scare quotes around “hearing” are necessary – the Committee Report (download) does not indicate that the Committee actually solicited or heard any testimony on the revised bill at this hearing, instead referring readers back to the materials from the November 2020 hearing on the original bill. As far as we can tell, there was no public notice of yesterday’s Committee hearing. The lack of advance notice effectively precluded interested parties from reviewing and responding to the revised bill.

The Committee approved the bill 4-1 immediately following the hearing, sending it to the full Council. Less than two hours later, the Council passed the bill, sending it to the Mayor’s desk for signature. The time from when the Council formally announced the revised bill (which was dated November 9th) to when the Council approved it (November 10th) was approximately 24 hours, if not less. Given the message of necessity that the Mayor had issued the previous day, it seems likely that he will sign it, though we hold out hope that strong pressure from public interest organizations may dissuade him from doing so.

Where the Revised Bill Takes Steps Backwards

There are a number of areas in which the revised bill, which is now set to become law, is not merely deficient, but weaker than the original bill.

Narrowing of scope and application

At the highest level, the revised bill applies to a narrower class of employment decisions than the original. Where the original bill defined an “employment decision” as including all decisions on hiring, compensation, and all other “terms, conditions or privileges of employment,” the revised bill applies only to hiring and promotion. This leaves many substantial employment decisions that dramatically impact workers’ lives – including those relating to compensation, scheduling, and working conditions – outside the bill’s scope.

The revised bill also applies only to workers who are residents of New York City, rather than to all employees of NYC-based employers. Given the sheer volume of non-NYC residents employed by NYC employers – particularly in the dawning post-COVID age – this represents a significant narrowing of the bill’s applicability.

Weaker bias audit requirements

I.1894 requires employers to conduct a “bias audit” on an automated tool before using it to evaluate candidates. The original bill required bias audits to assess a tool’s “predicted compliance” with New York City’s employment discrimination laws “and any other applicable law relating to discrimination in employment.” This definition of bias audit, while somewhat vague, implied that employers must assess a tool’s compliance with all anti-discrimination laws. This meant that the bill would cover “a wide range of protected traits that are too often neglected when examining bias,” such as age and disability.

The revised bill replaced this broad definition of bias audits with language that, in essence, merely reaffirms employers’ existing requirements under federal regulations to check whether the tool would have a disparate impact based on race, ethnicity, or sex. This narrow requirement is accompanied by language that gives the illusion of greater inclusivity, saying that a bias audit “shall include but not be limited to” checking for disparate impact on race, ethnicity, and sex. But in practice, the revised bill gives employers and vendors no reason to do more than this. After all, why would an employer choose to conduct analyses that might reveal (and thus put them on notice of) violations of law that might otherwise remain hidden?

The main effect of the revisions is, therefore, to relieve employers of any incentive to check for other forms of discrimination, such as:

• discrimination against disabled, older, or LGBTQ+ workers; 
• intersectional discrimination where a multiply marginalized candidate is disadvantaged based on a combination of protected characteristics; and 
• forms of disparate treatment that can arise when automated systems “learn” from biased training data to recognize (and discriminate against) protected characteristics without being explicitly programmed to do so.

The bill also requires employers to post a “summary” of the bias audit on their website, but it says nothing about what this summary must contain. This leaves open the possibility that employers will omit information about potential sources of discrimination even with respect to the limited analysis that the bias audit requires.

Inadequate notice requirements

The revised bill appears, at first blush, to improve on the original in its notice and disclosure requirements. The original bill required candidates to be notified that an automated tool had assessed them, as well as the “job qualifications or characteristics” that were the subject of the assessment. But where the original bill did not require such notice until after the evaluation occurred, the revised bill requires pre-assessment notice. The revised bill also requires the notice to “allow a candidate to request an alternative selection process or accommodation.” Lastly, upon a written request from a worker, the employer must provide additional information regarding the data that the automated tool uses, if the information is not already on their website.

A closer examination reveals these notices still fail to give candidates who may otherwise face discrimination a meaningful chance to voice concerns or request accommodation. The notice does not have to include any information on how candidates’ qualifications and characteristics will be assessed. Workers, particularly disabled workers, need this information to know whether they might require accommodations or an alternative selection procedure to ensure they get a fair assessment.

Moreover, “job qualifications and characteristics” are never defined, and employers could easily interpret them in a narrow way that further limits the information candidates receive. This too is particularly problematic for disabled workers, since the bill has been stripped of any other provisions that require employers to consider impacts on disabled workers.

Because nothing else in the bill requires employers or tool designers to audit for disability discrimination or consider potential accommodations when designing or deploying a tool, this provision is likely to have little practical effect. Consequently, employers might only provide accommodations that are ill-suited for candidates’ needs, if they even offer accommodations at all.

Lastly, the disclosure regarding the data the tool uses need only be provided if a candidate requests it in writing. Even if such a request is made, the bill is once again vague on exactly what information about “data collection” and “sources” must be provided, giving employers leeway to provide only cursory information on the data undergirding a tool.

Removal of rulemaking authority, and weakened enforcement provisions

The revised bill makes it harder to mitigate the bill’s flaws by stripping the New York City Commission on Human Rights of the authority to clarify and expand on the bill’s provisions and by transferring enforcement authority away from the Commission. The original bill gave “[t]he department [of Consumer and Worker Protection], the commission on human rights and any other agency designated by the mayor” authority to “promulgate such rules as it deems necessary to implement and enforce the provisions of this subchapter.” This broad authority would have allowed the Commission, which has specific and deep expertise in civil rights laws, to flesh out the bill’s audit and notice requirements. By deleting those provisions, the revised bill creates ambiguity as to which entity, if any, has that authority. Certainly, it is clear that the Commission, which is the only City agency with the necessary expertise, will not have that authority.

The revisions also transferred enforcement authority from the Commission to New York City’s Corporation Counsel, a generalist office tasked with a wide range of legal responsibilities. As a result, the odds of aggressive enforcement of the law’s already weakened provisions are greatly reduced.

The Committee Report Accompanying the Revised Bill Misleads

The Committee Report further demonstrates how ill-advised it was for the Council to pass the revised bill without seeking additional public input. The report is highly selective in its characterization of data-driven human resources (HR) tools, focusing on certain types of problematic automated tools and assessments while overlooking others.

The report speaks positively of technology that is “used to quickly evaluate resumes,” not acknowledging the serious potential for discrimination that resume screening technologies have demonstrated. It refers only fleetingly to risks for “ethnic or minority workers,” and only with respect to “automated searches” that exclude these workers due to housing segregation when looking for workers who live near worksites.

The report does allude to potential issues with “questionable assessments of observable physical factors,” but makes no mention of the serious concerns that also arise when tools assess candidates based on personality traits or other characteristics associated with mental health or cognitive disabilities. Moreover, while the report mentions the risk that some tools that measure physical factors “might systematically disadvantage applicants with disabilities,” it neglects to mention that the revised bill actually stripped the language that would have required employers to check for such disability discrimination.

Overall, the report paints a picture of automated employment decision tools that fails to truly recognize how such tools can harm marginalized workers; in fact, the main risk that the report cites is simply that automated tools may “potentially recommend[] an unqualified candidate.” The report thus suggests that the Council acted on far-from-complete information about the risks HR technologies pose.

Conclusion

I.1894 was already the most significant proposed legislation on algorithmic, data-driven, or AI-powered HR tools in the U.S. to date. Now that it is about to become law, other jurisdictions might view it as a potential model for regulating HR technologies. That would be a serious mistake.

While we appreciate the Council’s willingness to tackle these issues, we fear that the ordinance will create a situation worse than the status quo ante. The intent of the ordinance, and indeed its ultimate effect, seems to be to make it easy for vendors and employers to comply. By marketing their compliance with such weak protections, vendors and employers would create the false impression that their tools are valid and nondiscriminatory. That could well create additional pressure for employers to adopt such tools, creating a boon for vendors but threatening further harm to vulnerable workers.

Civil society organizations must be vocal in pointing out the many ways in which this ordinance falls short, and fight hard to ensure that future legislation does not suffer from the same serious flaws.

Updated on November 15, 2021.