Alignment in artificial intelligence or AI alignment is a subfield of artificial intelligence security that further falls under one of the goals of artificial intelligence as a field. It centers on steering AI systems toward achieving desired human-centric outcomes. The concept can be traced as far back as 1950 but the gradual and more recent rapid developments in the field have brought forth concerns about various alignment problems while also advocating for scholars, researchers, and organizations to tackle AI development in consideration of the importance of AI alignment.
What is AI Alignment: Understanding the Importance of Aligning Artificial Intelligence, the Main Alignment Problems, and Notable AI Alignment Approaches
Importance of AI Alignment
English mathematician and computer scientist Alan Turing introduced the Turing Test in 1950 as a method for assessing whether a machine can exhibit intelligent behavior indistinguishable from humans. It was also around this time when he considered that one of the implications of artificial intelligence is the arrival of machines that can pursue their goals unhinged. Turing is considered one of the founding fathers of artificial intelligence.
Norbert Wiener, an American mathematician and computer scientist who pioneered cybernetics, also raised concerns over the development of machines with mechanical agency. This was part of his several predictions about the possible moral and technical consequences of automation. He warned in a paper published in 1960 that there should be an assurance that the purpose put into the machine is the same purpose that humans desire.
Several experts have weighed in. John McCarthy introduced in 1983 the idea of artificial moral agents tasked to promote standards in AI systems. Computer scientist Vernor Vinge coined the term singularity to describe a point in time when AI surpasses human intelligence and becomes unpredictable and philosopher Nick Bostrom published a 2020 paper that regarded AI as one of the biggest existential risks unless aligned with human values.
The insights above have contributed to the modern conception of AI alignment. Nevertheless, according to its definition, the purpose of aligning AI systems is to bridge the gap between their actual behavior and the intentions of AI developers. A well-aligned AI system acts in accordance with human preferences, ethical considerations, and the overall well-being of society while a misaligned system produces undesirable and harmful outcomes.
Notable AI Alignment Problems
Research in AI alignment seeks to align three main goals. These are the intended goals of the human developers and operators, the specified goal of a particular AI system as defined in its data set, and the emergent goals that are advanced or sought by the same AI system.
A misalignment transpires when one or more of these three main goals do not coincide. There are two broad categories of misalignment. The more specific inner misalignment transpires from an unaligned specific goal and emergent goal while the outer misalignment happens when there is a mismatch between the intended goal and specific goal.
These mismatch or misalignment instances can transpire due to unresolved issues or the emergence of unforeseen issues. These issues are called AI alignment problems. They represent the main challenges in aligning an AI system. Below are examples of AI alignment problems:
1. Specification Gamification or Reward Hacking
An AI system can produce the outcome it is designed to do without going through the processes specified by its developers. This means that it can accomplish a task in a more efficient manner while overriding the processes it is intended to go through. This AI alignment problem is called specification gamification or reward hacking and it happens when either the human developer or an operation fails to provide complete and specific instructions. A more capable AI system or a larger AI model tends to be more effective at gaming its specifications.
2. Power-Seeking Tendencies and Strategies
The current systems and models of artificial intelligence still lack advanced capabilities such as long-term planning and situational awareness. However, some future iterations are expected to develop unwelcome tendencies and strategies to seek power. This is also more possible to transpire in general artificial intelligence systems. Hypothesized examples include unsupervised proliferations or the acquisition of computational resources. This is an AI alignment problem because it has the potential to render a particular AI system uncontrollable.
3. Misinterpretation and Conflict of Objectives
Another common AI alignment problem is the misinterpretation of human-specified objectives and the emergence of situations or specific use cases that can result in a conflict of objectives. A particular system that misinterprets its intended purpose can produce actions that are still aligned with the literal interpretation but not the intended meaning. Furthermore, when it comes to facing conflict of objectives, a system can have a problem producing a desired outcome because of the mismatch between its training data and its deployment or actual operations.
4. Hallucination Tendencies and Emergent Goals
The widespread applications of generative artificial intelligence have demonstrated a critical problem with large AI models. Intelligent chatbots and agents that use a large language model or computer vision models used in image recognition can still hallucinate or produce outcomes that appear plausible but are nonsensical. Some large models can also develop emergent goals or the potential to demonstrate unanticipated goal-directed behavior.
5. Developing Advanced Artificial Intelligence
The development of advanced AI systems and AI models is an AI alignment problem in itself. A particular system that becomes more powerful and autonomous also becomes more difficult to align. It is interesting to note that large language models used in natural language processing applications have achieved a level of complexity that even their developers are having a hard time understanding and explaining how they work. This creates a black-box problem that can make it difficult to evaluate, predict, or correct their behaviors.
6. Pressure From Commercialized AI Systems
Companies such as Google, Nvidia Corporation, and OpenAI have deployed systems and models for commercial consumption. Other startup companies have emerged. Established companies are also developing and deploying their artificial intelligence strategies. The problem with an AI strategy is that it can pressure companies or their developers to deploy an unsafe or untested AI system in an attempt to gain a competitive advantage as fast as possible or acquire the incentives or benefits from having an operational artificial intelligence system.
7. Effective Altruism vs Effective Accelerationism
The debate between effective altruism and effective accelerationism is related to AI alignment problems stemming from developing advanced artificial intelligence or commercializing artificial intelligence systems. The former is a concept that approaches development through the use of evidence and reason to maximize net benefits while the latter is an approach focusing on maximizing progress as fast as possible. Effective altruism is seen to slow down development while effective accelerationism hastens and prioritizes development.
8. Artificial Intelligence as Existential Risk
Remember that artificial intelligence has been considered one of the most serious existential risks to humans. This AI alignment problem will emerge when a particular AI system demonstrates other issues such as reward hacking or power-seeking strategies. Observers have noted that an advanced AI system or even a large AI model has the potential to cause catastrophic outcomes once used to control critical infrastructures such as the national power grid, military assets such as nuclear weapons, and the connected computer systems of the world.
Key Approaches to AI Alignment
It is important to reiterate the fact that the purpose and importance of aligning artificial intelligence systems center on the need to promote the greater welfare of humankind while also benefitting from the practical and proper use of modern technological developments.
Aligning an AI system involves several approaches. There are technical approaches that revolve around the development or embedding of technologies to align a particular system. Normative approaches are concerned with integrated moral and ethical standards in a system or following these standards during the development and deployment of this system.
Companies such as OpenAI and Google have their own strategies for AI alignment. Nonprofit organizations and government agencies have been tasked to come up with standards or serve as watchdogs. The following are the notable approaches to AI alignment:
1. Value and Preference Learning
The most common approach to AI alignment is to develop methods for AI systems to understand and demonstrate human values and preferences. This can be done through demonstrations during training, continuous feedback during inference, reinforcement learning or reward systems, and instilling of machine ethics based on moral values and ethical standards.
2. Scalable Oversight Solutions
A system that becomes more powerful and autonomous also becomes more difficult to align through human supervision. Scalable oversight explores solutions for reducing the time and effort needed for supervision and assisting human supervisors. Possible approaches include active learning, semi-supervised reward learning, and reward model.
3. Oversight Automation Techniques
Researchers have also worked on another scalable oversight solution centered on automation. This involves the use of technology to assist humans in overseeing artificial intelligence systems. A more specific example is the use of an automated AI agent that can perform oversight functions. This is seen as applicable in the superalignment of advanced AI systems.
4. Honest Artificial Intelligence
Generative AI models such as large language models have been shown to repeat falsehoods from their training data and even confabulate new falsehoods. Another approach to AI alignment is to ensure that these models or entire systems are honest and truthful. This is done through using curated datasets, citation of sources, and better reasoning capabilities.
5. Distillation and Amplification
A particular large and complex artificial intelligence model can be simplified and made small. This is called distillation. The distilled AI model can then be embedded into another larger AI model. This is called amplification. The entire process is repeated several times until a final and more improved or better-aligned artificial intelligence model is produced.
FURTHER READINGS AND REFERENCES
- Ji, J., Qiu, T, Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y., He, Z., Zhou, J., Zhang, Z., Zeng, F., Ng, K. Y., Dai, J., Pan, X., O’Gara, A., Lei, Y., Xu, H., Tse, B., Fu, J., McAleer, S., Yang, Y., Wang, Y., Zhu, S-C., Guo, Y., and Gao, W. 2023. “AI Alignment: A Comprehensive Survey.” arXiv. DOI: 48550/arXiv.2310.19852
- Korinek, A. and Balwit, A. 2023. “Aligned with Whom?” The Oxford Handbook of AI Governance. C4.S1-C4.N19. DOI: 1093/oxfordhb/9780197579329.013.4
- Norbert, W. 1960. “Some Moral and Technical Consequences of Automation: As Machines Learn They May Develop Unforeseen Strategies at Rates that Baffle Their Programmers.” Science. 131(3410): 1355-1358. DOI: 1126/science.131.3410.1355
- Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J. 2022. “Self-Critiquing Models for Assisting Human Evaluators (Version 2). arXiv. DOI: 48550/ARXIV.2206.05802
- Zhang, B., Anderljung, M., Kahn, L., Dreksler, N., Horowitz, M. C., and Dafoe, A. 2021. “Ethics and Governance of Artificial Intelligence: Evidence From a Survey of Machine Learning Researchers.” Journal of Artificial Intelligence Research. 71. DOI: 1613/jair.1.12895
- Zhang, X., Chan, F. T. S., Yan, C., and Bose, I. 2022. “Towards Risk-Aware Artificial Intelligence and Machine Learning Systems: An Overview.” Decision Support Systems. 159: 113800. DOI: 1016/j.dss.2022.113800
