AI Frameworks,
Guidelines,
Toolkits
ACLU, Algorithmic Equity Toolkit
ACT-IAC Emerging Technology COI, White Paper: Ethical Application of AI Framework, (2020)
ACT-IAC Emerging Technology COI, AI Playbook for the US Federal Government, (2020)
Ada Lovelace Institute, AI Now Institute and Open Government Partnership. Algorithmic Accountability for the Public Sector, (2021)
AI4People (Atomium - European Institute for Science, Media and Democracy), Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations
AI4People (Atomium - European Institute for Science, Media and Democracy), On Good AI Governance 14 Priority Actions, a S.M.A.R.T. Model of Governance, and a Regulatory Toolbox
AI4People (Atomium - European Institute for Science, Media and Democracy), 7 AI Global Frameworks
AI Global: Responsible Design Assistant
AI Now Institute, Algorithmic Impact Assessments: A Practical Framework for Public Agency Accountability
AI Now Institute, Algorithmic Accountability Policy Toolkit
AISP, A Toolkit for Centering Racial Equity Throughout Data Integration
Alan F. Winfield ; Katina Michael ; Jeremy Pitt ; Vanessa Evers: Machine Ethics: The Design and Governance of Ethical AI and Autonomous Systems, Published in: Proceedings of the IEEE ( Volume: 107 , Issue: 3)
Alan Turing Institute and the Information Commissioner’s Office (ICO), Project ExplAIn
Algofairness, BlackBox Auditing Tool
AlgoritmWatch, Ethics and algorithmic processes for decision making and decision support
Allied Media: People’s Guide to AI
All Tech is Human: Guide to Responsible Tech: How to Get Involved & Build a Better Tech Future
Amazon - AWS Conditional Demographic Disparity (CDD)
Amazon - AWS Responsible AI training modules
Amazon - AWS AI Service Cards
Amazon - AWS SageMaker Model Cards
Ambacher, B. u. a. , Trustworthy Repositories Audit & Certification: Criteria and Checklist (TRAC), CRL Center for Research Libraries, Chicago, (2007)
Arogyaswamy, B. Big tech and societal sustainability: an ethical framework. AI & Soc (2020)
Australian Human Rights Commission: Using artificial intelligenceto make decisions:Addressing the problemof algorithmic bias
ATARC: Artificial Intelligence and Data Analytics (AIDA) Guidebook
B. d’Alessandro, C. O’Neil, T. LaGatta, Conscientious classification: A data scientist’s guide todiscrimination-aware classification, Big data 5 (2) (2017)
Barcelona City Council Open Digitisation Plan
Barocas, Solon and Hood, Sophie and Ziewitz, Malte, Governing Algorithms: A Provocation Piece, (2013)
BASIC - A Toolkit and Ethical guidelines for Applying Behavioural Insights in Public Policy, OECD, (2018)
Ben Shneiderman (2020) Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy, International Journal of Human–Computer Interaction, 36:6
Bethge Lab: AI Foolbox
Berkley Haas: Mitigating Bias in Artificial Intelligence Playbook
Brown University, A Framework for Making Ethical Decisions
BSA: Confronting Bias: BSA’s Framework to Build Trust in AI
Brundage, Miles et al. “Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims.” ArXiv abs/2004.07213 (2020)
Canada Algorithmic Impact Assessment
Centre for Applied Data Ethics, Ethical Self-assessment Tool
Center for Democracy & Technology (CDT), DD Tool
Christian Sandvig, Kevin Hamilton, Karrie Karahalios and Cedric Langbort. When the Algorithm Itself Is a Racist: Diagnosing Ethical Harm in the Basic Components of Software, Int’l. J. Comm. 10: (2016)
Christian Sandvig, Kevin Hamilton, Karrie Karahalios and Cedric Langbort. “Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms.” In Data and Discrimination: Converting Critical Concerns into Productive: A preconference at the 64th Annual Meeting of the International Communication Association. Seattle, WA, (2014)
CIGREF, Digital Ethics
Council of Europe: Human Rights, Democracy, and the Rule of Law Assurance Framework for AI Systems
Dafoe, Allan. "AI Governance: A Research Agenda", 2018
Databaseline, ML Cards for D/MLOps Governance
DataEthics, White Paper on Data Ethics in Public Procurement of AI-based Services and Solutions, 2020
DataEthics, Impact Assessment, 2021
Data for Children Collaborative with UNICEF, Ethical Assessment
Data Society, Algorithmic Impact Assessment for the Public Interest
David Freeman Engstrom, Stanford University Daniel E. Ho, Stanford University, Catherine M. Sharkey, New York University, Mariano-Florentino Cuéllar. "Government by Algorithm: Artificial Intelligence in Federal Administrative Agencies"
Dawson D and Schleiger E*, Horton J, McLaughlin J, Robinson C∞, Quezada G, Scowcroft J,and Hajkowicz S†(2019) Artificial Intelligence: Australia’s Ethics Framework. Data61 CSIRO, Australia.
Deepcheck, Test Suites for Validating ML Models & Data
Dent, Kyle and Dumond, Richelle and Kuniavsky, Mike, A Framework for Systematically Applying Humanistic Ethics when Using AI as a Design Material (July 1, 2019). Temes de Disseny 35
Diakopoulos, N., “Algorithmic Accountability Reporting: On the Investigation of Black Boxes”, (2014)
Digital Catapult: Ethics Framework
doteveryone, Consequence Scanning Kit
D. Peters, K. Vold, D. Robinson and R. A. Calvo, "Responsible AI—Two Frameworks for Ethical Design Practice," in IEEE Transactions on Technology and Society, vol. 1, no. 1, pp. 34-47, March 2020
DQ Institute: DQ Framework Global Standards on Digital Literacy and Skills
Driven Data, Deon ethics checklist
Expert Group Data Ethics, Ethical Codex for Data- Based Value Creation, (2019)
European Commission High Level Expert Group on AI, Ethics Guidelines for Trustworthy AI, 2019
European Commission High-Level Expert Group on AI, Trustworthy AI Assessment List
European Commission High-Level Expert Group on AI, Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self-assessment
European Commission High-Level Expert Group on AI, Towards a European strategy on business-to-governmentdata sharing for the public interest
European Commission, White Paper on Artificial Intelligence -A European approach to excellence and trust
European Commission & ADAPT, EnTIRE (Mapping Normative Frameworks for EThics and Integrity of REsearch)
European Council Committee of Ministers, Recommendation CM/Rec(2020)1 of the Committee of Ministers to member States
on the human rights impacts of algorithmic systems, 2020
European Parliament, A governance framework for algorithmic accountability and transparency, (2019)
Facets, Analyzing machine learning datasets by visualization
FactSheets: Increasing Trust in AI Services through Supplier’s Declarations of Conformity
FairLearn, Open source Python toolkit
FairLens, Open source Python toolkit
FairSight, Fair DecisionMaking Tool
FBPML Wiki, Best Practices in ML
F. Kamiran, T. Calders, Data preprocessing techniques for classification without discrimination, Knowledge and Information Systems 33 (1) (2012)
Floridi, L. (2020). Ethical Foresight Analysis: What it is and Why it is Needed. Minds and Machines
Floridi, L. (2020). How to Design AI for Social Good: Seven Essential Factors
Fujitsu, AI Ethics Impact Assessment & Toolkit
GAO (Government Acountability Office) AI Accountability Framework for Federal Agencies and Other Entities
GitHub, CodeSearchNet
Global AI Standards Repository
Google, Playing with AI Fairness: What-if Tool
Google, Perspectives on Issues in AI Governance
Google, ML-fairness-gym: A Tool for Exploring Long-Term Impacts of Machine Learning Systems
GovEx, the City and County of San Francisco, Harvard DataSmart, and Data Community DC, Ethics and Algorithms Toolkit
GovLab, Re-imagining Governance in PracticeBenchmarking British Columbia’s Citizen Engagement Efforts, 2013
GovLab, The Open Policy Making Playbook
GovLab, Mapping and Comparing Responsible Data Approaches, 2016
Government of the Netherlands, Fundamental Rights and Algorithms Impact Assessment (FRAIA)
Hallensleben, Sebastian and Hustedt, Carla. From Principles to Practice: An interdisciplinary framework to operationalise AI ethics, Bertelsmann Stiftung 2020
Harvard University - Berkman Klein Center for Internet and Society, Ethics and Governance of AI
Humanitarian Data Science and Ethics Group (DSEG), A Framework for the Ethical Use of Advanced Data Science Methods in the Humanitarian Sector
Information Accountability Foundation (IAF), Comprehensive Publications About Information Accountability
IAPP, Building Ethics into Privacy Frameworks for Big Data and AI
IBM, AI Fairness 360 Open Source Toolkit
IBM, AI Explainability 360 Toolkit
IBM, AI FactSheets 360 Toolkit
IBM, IBM Watson OpenScale
IBM, Everyday Ethics for Artificial Intelligence
IBM, Advancing AI ethics beyond complianceFrom principles to practice
IBM, Adversarial Robustness Toolbox
ICAEW: New technologies, ethics and accountability
ICO, Guidance on the AI auditing framework
ICO, Guidance on AI and data protection
IDEO, AI Ethics Cards
IEEE, Ethically Aligned Design, Version 1 (EADv1)
IEEE, Global Initiative on Ethics of Autonomous and Intelligent Systems. Ethically Aligned Design: A Vision for Prioritizing Human
Well-being with Autonomous and Intelligent Systems, Version 2 - Request for Input. IEEE, 2017.
IEEE, CertifAIEdTM certification program
IFTF & Omidyar Network, Ethical OS Framework
Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell,Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. In Conference on Fairness, Accountability, and Transparency (FAT* ’20), January 27–30, 2020, Barcelona,Spain .ACM, New York
Institute for Ethical AI & ML, AI-RFX Procurement Framework
Institute and Faculty of Actuaries (IFoA) and the Royal Statistical Society (RSS), A Guide for Ethical Data Science
Institute for the Future of Work, Policy Briefing on algorithmic impact assessments
ISO/IEC JTC 1/SC Standardization on Artificial Intelligence
Japanese Society for AI (JSAI), Ethical Guidelines
Jessica Morley, Luciano Floridi, Libby Kinsey, Anat Elhalal. From What to How: An Overview of AI Ethics Tools, Methods and Research to Translate Principles into Practices and Supporting Material
Jobin, A., Ienca, M. & Vayena, E. ,The Global Landscape of AI Ethics Guidelines, Nat Mach Intell 1, (2019)
Judy Goldsmith, Emanuelle Burton, Why Teaching Ethics to AI Practitioners Is Important, AAAI Conference on Artificial IntelligenceThirty-First AAAI Conference on Artificial Intelligence, 2017
Kaminski, Margot E. and Malgieri, Gianclaudio, Algorithmic Impact Assessments under the GDPR: Producing Multi-layered Explanations. U of Colorado Law Legal Studies Research Paper No. 19-28. (2019)
Kat Zhou, Design Ethically Toolkit
Kathy Baxter (Salesforce Research), How to Build Ethics into AI — Part I Research-based recommendations to keep humanity in AI
Kathy Baxter (Salesforce Research): Building Ethics into AI: Lessons Learned from Pioneers in the Trenches
Kiritchenko, Svetlana and Mohammad, Saif M., Equity Evaluation Corpus (EEC)
Koshiyama, Adriano and Kazim, Emre and Treleaven, Philip and Rai, Pete and Szpruch, Lukasz and Pavey, Giles and Ahamat, Ghazi and Leutner, Franziska and Goebel, Randy and Knight, Andrew and Adams, Janet and Hitrova, Christina and Barnett, Jeremy and Nachev, Parashkev and Barber, David and Chamorro-Premuzic, Tomas and Klemmer, Konstantin and Gregorovic, Miro and Khan, Shakeel and Lomas, Elizabeth, Towards Algorithm Auditing: A Survey on Managing Legal, Ethical and Technological Risks of AI, ML and Associated Algorithms (February 4, 2021)
Kriebitz, A., & Lutge, C. Artificial Intelligence and Human Rights: A Business Ethical Assessment. Business and Human Rights Journal, 5(1), 84–104. Cambridge University Press (2020)
Leidner, Jochen L. and Vassilis Plachouras. Ethical by Design: Ethics Best Practices for Natural Language Processing
Leighton Andrews, Bilel Benbouzid, Jeremy Brice, Lee A. Bygrave, David Demortain, Alex Griffiths, Martin Lodge, Andrea Mennicken, Karen Yeung. Algorithmic Regulation. The London School of Economics and Political Science, (2017)
Lepri, Bruno et al. “Fair, Transparent, and Accountable Algorithmic Decision-Making Processes.” Philosophy & Technology 31, 4(December 2018): 611–627 © 2017 Springer Science+BusinessMedia
Leslie, D. Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of AI systems in the public sector, The Alan Turing Institute, (2019)
LinkedIn Fairness Toolkit (LiFT)
Linux Foundation, Apache NiFi ‹› AI Fairness 360 (AIF360) Integration – Trusted AI Architecture Development Report 1
Linux Foundation, AI Explainability 360
Lütge, Christoph. AI Ethics and Governance “Building a Connected, Intelligent and Ethical World”, 2020
Madaio, Michael A. and Jennifer Wortman Vaughan. “Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI.” (2020).
Markkula Center for Applied Ethics, A Framework for Ethical Decision Making
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* ’19). Association for Computing Machinery, New York
MetaAI, Fairness Flow
Microsoft, AI Fairness Checklist
Microsoft, Judgment Call
Microsoft, Error Analysis
Microsoft, Responsible AI Toolbox
Microsoft, Human-AI eXperience (HAX) Toolkit
Ministry of Economy, Trade and Industry (METI) of Japaan: Contract Guidelines on Utilization of AI and Data
Moral Framework for Understanding of Fair ML through Economic Models of Equality of Opportunity
NetHope Solutions Center, Artificial Intelligence (AI) Suitability Toolkit for Nonprofits
NIST, Artificial Intelligence and User Trust (NISTIR 8332)
NIST, A Plan for Federal Engagement in Developing Technical Standards and Related Tools
NIST, AI Risk Management Framework (AI RMF)
NIST, AI Risk Management Playbook - draft
NIST, Generative AI Profile - draft
OECD, Framework for the Classification of AI Systems
Office of the Privacy Commissioner for Personal Data, Ethical Accountability Framework for Hong Kong
Open Data Institute, The Data Ethics Canvas
Open Robo Ethics, AI Ethics Assessment Toolkit
Oracle, Skater
Oxford Handbook of Ethics of AI: Online Supplement
PARC, AI Ethics Review
Partnership on AI, About ML
Platform for the Information Society, Artificial Intelligence Impact Assessment
PrepareCenter.org: Data Playbook
Princeton, REVISE: REvealing VIsual biaSEs
ProPublica, Data Store
pymetrics, audit AI
R. Benjamins, A. Barbado, D. Sierra, Responsible AI by design
Responsible Research and Innovation, Self-Reflection Tool
Responsible Innovation Compass, Self-Check
Rolls Royce, AIetheia Framework
Ruf, B., Boutharouite, C., & Detyniecki, M. Getting Fairness Right: Towards a Toolbox for Practitioners.
Sabou, Marta & Bontcheva, Kalina & Derczynski, Leon & Scharl, Arno. (2014). Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines. Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC'14).
Salesforce, AI Economist, Simulation Card Foundation Economic Simulation Framework
Santa Clara University, Markkula Center, An Ethical Toolkit for Engineering/Design Practice
Sasha Costanza-Chock, Maya Wagoner, Berhan Taye, Caroline Rivas, Chris Schweidler, Georgia Bullen, & the T4SJ Project, 2018. #MoreThanCode: Practitioners reimagine the landscape of technology for justice and equity. Research Action Design & Open Technology Institute.
SciPy 2021 Tutorial Fairness in AI systems: From social context to practice using Fairlearn
Sendak, M.P., Gao, M., Brajer, N. et al. Presenting machine learning model information to clinical end users with model facts labels. npj Digit. Med. 3, 41 (2020).
SHAP (SHapley Additive exPlanations)
SHERPA (Shaping the ethical dimensions of smart information systems–a European perspective): Guidelines for the Ethical Development of AI and Big Data Systems: An Ethics by Design approach
Singapore, Infocomm Media Development Authority & Personal Data Protection Commission: Model Artificial Intelligence Governance Framework, 2nd Edition
Singapore, Infocomm Media Development Authority & Personal Data Protection Commission: Implementation and Self-Assessment Guide for Organizations
Singapore, Infocomm Media Development Authority & Personal Data Protection Commission: AIverify - AI Governance Testing Framework & Toolkit
Smart Dubai, AI System Ethics Self-Assessment Tool
Stanford Center on Philanthropy and Civil Society, Digital Impact Toolkit
Stark, Jeannette and Nicholas Diakopoulos. “Algorithm Tips : A Resource for Algorithmic Accountability in Government.” (2017)
TensorFlow, Fairness Indicators
TensorFlow, Model Analyzer
TensorFlow, Responsible AI - Model Remediation
Thilo Hagendorff, The Ethics of AI Ethics -- An Evaluation of Guidelines
Tilburg University, Handbook on Non-discrimination by Design
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford, "Datasheets for Datasets," (2018)
Timnit Gebru, Emily Denton, FATE/CV Tutorial (Fairness, Accountability, Transparency, Ethics / Computer Vision)
UC Berkeley Center for Long-Term Cybersecurity (CLTC), Decision Points AI Governance
UK Government: UK Data Ethics Framework
UK Government: Ethics, Transparency and Accountability Framework for Automated Decision-Making
UK Government: Artificial Intelligence and Public Standards Report
UK Government: Algorithms: How they can reduce competition and harm consumers
UK Government: Algorithmic Transparency Standard
UK Government: Al Assurance Guide & Roadmap
UNDP, UN Global Pulse, ‘A Guide to Data Innovation for Development: From Idea to Proof of Concept,’ 2016
University of Chicago Center for Data Science and Public Policy, Aequitas Bias & Fairness Audit
University of Washington, LIME
Urban Institute: Spatial Equity Data Tool
US Health & Human Services - Trustworthy AI Playbook
Utrecht University, Data Ethics Decision Aid (DEDA)
Value Sensitive Design and Information Systems
Vidgen, R., Hindle, G., and Randolph, I., (2020). Exploring the ethical implications of business analytics with a business ethics canvas. European Journal of Operational Research, 281(3)
Vollmer, Sebastian & Mateen, Bilal & Bohner, Gergo & Király, Franz & Ghani, Rayid & Jonsson, Pall & Cumbers, Sarah & Jonas, Adrian & McAllister, Katherine & Myles, Puja & Granger, David & Birse, Mark & Branson, Richard & Moons, Karel & Collins, Gary & Ioannidis, John & Holmes, Chris & Hemingway, Harry. (2018). Machine learning and AI research for Patient Benefit: 20 Critical Questions on Transparency, Replicability, Ethics and Effectiveness.
World Economic Forum (WEF) with Centre for the Fourth Industrial Revolution Fellows from Accenture, BBVA, IBM, Suntory Holdings, Australian Institute of Company Directors, Best Practice AI, Latham & Watkins, and Splunk, with contributions from AI4All, AI Board Toolkit
World Economic Forum (WEF) Unlocking Public Sector AI Toolkit: AI Procurement in a Box
World Economic Forum (WEF) AI Procurement in a Box: AI Government Procurement Guidelines
World Economic Forum (WEF) AI Procurement in a Box: AI Procurement in a Box: Workbook
World Economic Forum (WEF) Responsible Use of Technology
World Economic Forum (WEF) AI Governance - A Holistic Approach to Implement Ethics into AI
World Economic Forum (WEF) C-Suite Toolkit
Zuiderwijk, Anneke and Chen, Yu-Che and Salem, Fadi (2021), Implications of the Use of Artificial Intelligence in Public Governance: A Systematic Literature Review and a Research Agenda, Government Information Quarterly
10 Simple Rules for Responsible Big Data Research
510, F.A.C.T Score for Responsible AI