Introducing PIBBSS Horizon Scanning Initiative

TLDR: PIBBSS is prototyping a new initiative aimed at finding novel and promising research directions, as well as established academics which could act as research leads to these directions.

Horizon Scanning

Our aim is to scan the research landscape for novel ideas that could serve as the core tenets of high-impact research programmes within AI safety, as well as scouting for established academics who could lead these programmes. We currently place high value on research agendas which:

1. Could provide theoretical basis for principled safety guarantees for deep learning-based systems
2. Are amicable to empirical iteration
3. Are well positioned to develop safety-relevant applications within 2-5 years of vigorously supported “normal science”.

Why Do This?

We believe that research programmes which fit the above desiderata to be important, and field building efforts supporting this problem to be both relatively neglected and tractable.

Importance

• Lack of Foundations: We don’t currently possess the foundational scientific understanding required to establish high assurance safety guarantees of advanced AI systems.

• Urgency: The likelihood of capabilities outpacing safety inteventions which are not grounded in rigorous foundational science in the next 2-5 is high, given the current pace of AI development.

• Uncertainty Over the Current Portfolio: While many AI safety research programmes have preliminary empirical corroboration with current SOTA systems, uncertainty regarding their robustness, scalability, and domain of applicability remains (when, for example, are features expected to be linear?)

Neglect   

• Out of Scope For Safety Labs: Rather than cultivating a large portfolio of research bets, research labs have the responsibility to work on advancing whatever scientific paradigm is at the intersection of their expertise and interest.

• Out of Scope For Funders: Grantmakers often lack the in-house domain expertise to proactively identify promising research directions which tend to be neglected by the field, and largely rely on reacting to applications that they receive.

• Difficulties & Hurdles in Talent Building Pipeline: It has been noted before that developing mentorship pipelines for technical talent aimed at advancing theoretical research takes, compared to other kinds of talent, significantly more time and exposure. We believe this can be mitigated by targeting researchers who already have research expertise in domains which are natural to understanding artificial intelligence. Unfortunately, the field currently lacks programs which are specialized at integrating mid-career professionals into AI Safety.

Tractability

• Nothing is new under the sun: The ocean of knowledge is deep, wide and vastly interconnected. Progress in analogous problem domains exists, forming a hidden interdisciplinary framework that, if uncovered, could drive progress. This has been observed multiple times in AI Safety, for example: Learning theoretic agenda and imprecise probabilities, natural abstractions and sufficient statistics, and superposition hypothesis and compressed sensing, to name a few.

• Interest in engaging in AI is widespread: Talk of (and resources for) AI Safety is becoming more common within academic circles, with cross-disciplinary groups aiming to explain intelligence in their own terms. In our experience, most academic researchers are reachable and excited to speak to people who understand their specific research interests.

• Incubation of this talent is possible: Spawning from one of the successes of our first affiliateship cohort, we have a demonstrated track record of incubating research agendas into a technology readiness level such that their research can be both legible and demonstrable to funders. We do this by surveying and relationship building with the relevant expertise communities, organizing events and providing them with the appropriate mentorship.

Our Team

Our team specifically has a wide range of experience in theoretical domains in academia, mentoring and teaching, ML research engineering as well as object-level AI Safety research experience. The horizon team is led by PIBBSS’ research director, Lucas Teixeira, an AI safety researcher with a background in ML and philosophy. Technical staff members include Lauren Greenspan, a physicist turned AI safety researcher with a passion for STS, and Dmitry Vaintrob, a mathematician currently focused on AI interpretability. We are fortunate to have operations support from Dušan Nešić, AI safety researcher and co-founder of EA Serbia.