Autonomous weapons systems: how to work towards a total ban?

1. Introduction

The potential uses of artificial intelligence are extensive, ranging from finance, health care, justice, education to military uses. While the application of AI has been widely acclaimed in non-military technologies, its use by the military has been the subject of intense debate. In particular, it is the ethical and legal implications of Autonomous Weapon Systems, also referred to as Lethal Autonomous Weapon Systems, which has caused great controversy.

Approaches to the development and use of regulation concerning AWS vary depending on the state.  While some states believe new legislation is required, others prefer less stringent political measures and guidelines.¹ The debate ranges from a complete ban on their development, production and use (because of their inability to apply human judgment and compliance with the law), to their benefits of being more precise and their potential to reduce casualties during conflicts.

It should be highlighted that there is no internationally agreed formal definition for AWS or LAWS. For the purposes of this paper, the term AWS will be used and is defined as:

“Any weapon system with autonomy in its critical functions — that is, a weapon system that can select (search for, detect, identify, track or select) and attack (use force against, neutralize, damage or destroy) targets without human intervention.”²

We provide an overview of the main issues debated with regards to AWS and international law, and note ways forward for regulating AWS.

2. AWS and its legal and ethical implications

2.1 From artificial intelligence to lethal autonomous weapons

AI’s definition shifts depending on the entity providing it and the goals they are trying to achieve with an AI system.³ AI makes everything scalable, where a single system can automate millions of devices. The majority of the AI fielded today is concerned with what is called “narrow AI”, which involves creating programs that demonstrate intelligence in one or another specialized area, such as chess-playing, medical diagnosis or automobile driving.

Artificial general intelligence (AGI) is the construction of a software program that can solve a variety of complex problems in a variety of different domains, and that controls itself autonomously, with its own thoughts, worries, feelings, strengths, weaknesses and predispositions.  Like all major scientific discoveries and technical breakthroughs, AGI has the potential to revolutionize our life and even the fate of the human species, either in a desired way or an undesired way.⁴  AGI is expected to successfully perform any intellectual task that a human being can, including reasoning, solving problems, making judgments when uncertain, and even being innovative and creative.

AI is leading towards a new algorithmic warfare battlefield that has no boundaries or borders, may or may not have humans involved, and will be impossible to understand and perhaps control across the human ecosystem in cyberspace, geospace (near earth outer space) and space.⁵ This is why it is time to align these new developments with existing laws and regulations that currently guide what is considered to be morally acceptable warfare.

2.2 AWS and autonomy

The lack of a universally accepted definition regarding what constitutes AWS is often cited as a reason for not proceeding towards any kind of international governance over autonomous weapons.⁶  The main reason for this is the difficulty to select the threshold for a weapon to be considered autonomous. Their various capabilities, applications and levels of human interactions create a wide spectrum of autonomy. Generally, the definitions of autonomous weapon systems can be classified into three groups.⁷

The first ones are based on the kind of human involvement, with varying degrees of autonomy : 1) human-in-the-loop control: where semi-autonomous weapon systems require human operator selection and authorization to engage specific targets; 2) human-on-the-loop: human-supervised autonomous weapon systems, which allow human intervention and, if needed, termination of the engagement, with the exception of time-critical attacks on platforms or installations; and 3) human-out-of-the-loop weapons: autonomous weapon systems, which upon activation can select and engage targets without human intervention.⁸ Within each of these categories, there are many intermediate levels in the way human and machine decision-making may be designed.

The second category of definitions are based on capability parameters, such as understanding intent and direction.⁹

The third category are structured along legal lines and lay emphasis on the nature of tasks that the systems perform autonomously.¹⁰ This category includes the definition by the International Committee of Red Cross (ICRC) of those systems able to independently select and attack targets with autonomy in the “critical functions” of acquiring, tracking, selecting and attacking targets without human intervention.^11]

The term autonomy is important for understanding debates about AWS. Autonomy results from the delegation of a decision to an authorized entity to take action within specific boundaries. An important distinction is that systems governed by prescriptive rules that permit no deviations are automated, but they are not autonomous. To be autonomous, a system must have the capability to independently compose and select among different courses of action to accomplish goals based on its knowledge and understanding of the world, itself, and the situation.¹²

2.3 Assessment of the legality of AWS

Although, the use of AI may reduce the collateral damage and harm of war by allowing better targeted decision-making, fully autonomous weapons would also create new moral, technical, and strategic dilemmas, which is why some scientists, activists and world governments call for its preemptive ban.¹³  

AI raises a number of ethical concerns, including issues of bias, unfairness, safety, lack of transparency¹⁴, and poor accountability. Moreover, automated weapons systems may not be predictable.¹⁵ With millions of lines of code in each application, it is difficult to know what values are inculcated in software and how algorithms actually reach decisions.¹⁶ The very same algorithm can serve a variety of purposes, which makes the ethics of decision-making very difficult. For example, automation algorithms can be found in their geo-location and driving systems, in the control of their sensors, actuators and weapons, in their health management, but also in targeting, deciding and during attacks.

It is one thing to support general goals, such as fairness and accountability, but another to apply those concepts in particular domains and under specific political conditions. One cannot isolate ethics discussions from the broader political climate in which technology is being deployed.

In the foreseeable future, a growing number of combat operations are expected to be carried out by AWS. This raises questions in terms of compatibility with international humanitarian law (IHL), such as the principles of distinction, proportionality and responsibility. For instance, AWS will find it very hard to determine who is a civilian and who is a combatant.¹⁷

Current versions of unmanned systems operate with direct human input and human operators make the majority of tactical decisions. However, future systems will not only follow pre-determined routes or hit a pre-programmed target, but will also operate in a manner that allows the systems to select and acquire a target, choose a route to reach the target area, decide whether to deploy weapons and, if so, decide which weapon system to deploy.¹⁸

Although new technologies of warfare are not specifically regulated by IHL treaties, their development and employment in armed conflict does not occur in a legal vacuum.¹⁹ These challenges may raise the question of whether existing international rules are sufficiently clear or whether there is a need to clarify them or develop new ones.

3. AWS and international law

AWS are not specifically regulated by international law. However, it is undisputed that international customary principles and rules apply to their use.

3.1 Jus ad bellum and the right to self-defence

Primarily, it is important to recall jus ad bellum which governs the use of force between states. Any resort to armed force on the territory of a foreign state, without its express consent violates Article 2(4) of the UN Charter (Charter).²⁰

The jus ad bellum principle is counterbalanced by the inherent right of individual or collective self-defence if an armed attack occurs against a Member of the United Nations, recognized under Article 51 of the Charter.²¹

Moreover, under customary international law, any resort to force in self-defence must comply with the conditions of necessity and proportionality. Necessity requires that force should only be used when other non-forcible measures are not effective, feasible or have been exhausted. The principle of proportionality requires that the state using force should respond in a manner that is proportional to the need to repel the threat.²²

3.2 International humanitarian law

Furthermore, if AWS are deployed in armed conflicts they would need to be able to evaluate and make judgements that comply with IHL.²³

One of the fundamental rules of IHL requires parties to an armed conflict to direct their operations only against combatants and military objectives.²⁴

Additionally, its rules prohibit attacks on persons who are hors de combat and therefore vulnerable.²⁵ Under customary international law being hors de combat is understood as being:

• in the power of an adverse party
• defenceless because of unconsciousness, shipwreck, wounds or sickness; or
• being anyone who clearly expresses an intention to surrender.²⁶

3.3 International law of law enforcement

Finally, a number of criminal justice instruments may also apply to the use of AWS in law enforcement,²⁷ including the 1979 Code of Conduct for Law Enforcement Officials,²⁸ and the 1990 Basic Principles on the Use of Force and Firearms by Law Enforcement Officials.²⁹

The international law of law enforcement defines when use of force by a state’s law enforcement officials is lawful and its conditions are as follows:

• in self-defence;
• to prevent crime;
• to effect or assist in the lawful arrest of offenders or suspected offenders;
• to prevent the escape of offenders or suspected offenders; and
• to maintain public order and security.³⁰

3.4 Convention on Conventional Weapons

The Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects, commonly referred to as Convention on Conventional Weapons (CCW)³¹ prohibits use of specific types of weapons that are considered to cause unnecessary or unjustifiable suffering to combatants or to affect civilians indiscriminately. While the CCW does not expressly mention AI and automated systems, its general principles still apply to AWS.

The CCW was adopted in 1980 and Canada was among the first group of countries to sign it in 1981. The CCW is important not only because it may apply to AWS, but also because it provides technical, legal and political experts a forum to exchange information, agree on definitions and to have a dialogue on new issues of concern with conventional weapons use and development, including AWS.³²

3.5 Group of Governmental Experts and Canada’s perspective

In 2016, at the occasion of the Fifth Review Conference to the CCW, Canada and other High Contracting Parties established an open-ended Group of Governmental Experts (GGE) on AWS.³³

In March 2016, 14 countries called for a ban on the development of AWS, however, Canada was not among them.³⁴ In its opening statement, Canada reiterated that while it continues to believe that IHL is sufficiently robust to regulate emerging technologies, it also recognizes that AWS may raise unique challenges with regards to the weapons review process, such as those related to testing and evaluation and in general of ensuring the lawful use of AWS.³⁵

In November 2017, over 200 leaders in AI from across Canada signed an open letter to Prime Minister Justin Trudeau urging the government to “take a strong and leading position against AWS on the international stage.”³⁶ Subsequently, at the 2017 GGE meeting, Canada stated that it has “committed to maintaining appropriate human involvement in use of military capabilities that can exert lethal force.”³⁷ In March 2019, the GGE met to discuss developments and strategies of AWS. Canada did not issue any further statements regarding AWS.

Canada’s current position is that work to restrict or ban weapon systems prone to indiscriminate effects or which are excessively injurious is critical.³⁸ The belief that AWS should be banned is not shared by the Canadian government, as is not interested in banning AWS technology. For example, in Canada’s recent Defence Policy Document “Strong, Secure, Engaged”, the Government of Canada has committed to “maintain appropriate human involvement in the use of military capabilities that can exert lethal force”³⁹, while pointing out that technological developments is the future of defence, which is “expected to be vastly different than today, with a greater emphasis on information technologies, data analytics, deep learning, autonomous systems”.⁴⁰ Alarmingly, nowhere is “appropriate human involvement” determined or defined in the document, nor does it openly support the idea that meaningful human control is critical for AWS.

The second part of the 2019 GGE session will take place from 20 to 21 August 2019. It still remains to be seen if Canada will take any formal positions regarding AWS. As the future of AWS is still being decided, it is incumbent on all of us to ensure that the choices made and the actions taken are done so in the most morally acceptable way possible.

4. Pathways to possible solutions

While the military use of AI development seems unavoidable and a complete ban on the use of AI does not seem to be the answer, with the rapid advancements of autonomous technologies and more importantly the positions held by a number of states, some clear strategies for regulating AWS in the context of armed conflicts need to be developed (including those for a complete ban on AWS).

4.1 State action for a ban

The history of negotiation and ratification of the Convention on the Prohibition of the Use, Stockpiling, Production and Transfer of Anti-Personnel Mines and on their Destruction, known informally as the Ottawa Treaty⁴¹ could be an example on how an initiative to ban a weapon ban with little international support may be fast-tracked to become a widely recognized international treaty.

The First Review Conference of the 1980 Convention on Conventional Weapons closed in Geneva on 3 May 1996 and introduced a number of changes that were widely welcomed, but fell far short of totally prohibiting these weapons. Canada’s delegation announced that Canada would host a meeting of pro-ban States to develop a strategy to move the international community towards a global ban on anti-personnel mines. The conference, held in October 1996 in Ottawa, set the scene for what would become known as the “Ottawa process” — a fast-track negotiation of a convention banning antipersonnel mines. Only 14 months later, representatives of 121 governments signed the Ottawa Treaty. At the end of April 1998, there were a total of 124 signatories and 11 States had already ratified the Convention that entered into force six months after 40 States had formally adhered to it.⁴² Today, there are 133 signatories and 164 Parties to the Ottawa Treaty.

Similarly, the process towards a complete ban of AWS could start with legal reviews at the national level, then an international declaration introducing a non-binding code of best practices and then preparing the ground for a fast negotiation and conclusion of a convention banning AWS.

4.1.1 National code of best practices: AWS legal review

Therefore, the first step for states would be to develop sets of guiding principles for the safe, ethical and responsible use of AWS at a national level.

There are several propositions on how this solution could be approached. Probably the most comprehensive is the one that Michael W. Meier discussed in the form of five questions that could help states considering AWS development in the context of compliance with the principles and rules of IHL:

• Whether there is a specific rule, such as a treaty obligation or a customary rule, prohibiting or restricting the use of the weapon;
• Whether, in its normal or intended circumstances of use, the weapon is of a nature to cause superfluous injury or unnecessary suffering;
• Whether the weapon is inherently indiscriminate, i.e., whether the weapon is capable of being used in compliance with the rule of discrimination (or distinction);
• Whether the weapon is intended, or may be expected, to cause widespread, long-term and severe damage to the natural environment; and
• Whether there are any likely future developments in the law of armed conflict that may be expected to affect the weapon subject to review. ⁴³

Based on these five questions, policy makers would need to coordinate with their respective governments to ensure that any such systems are consistent with national policies and to consider developments at the CCW.⁴⁴

This solution would have a considerable advantage because it would be based on existing principles of international and humanitarian laws and would leave the flexibility of defining best practices to states, thereby, allowing some states to move ahead of those harbouring doubts about regulating AWS and create a norm that is eventually adopted by the majority of states.

4.1.2 Political declaration

The next step would be the adoption of a political declaration, such as a UN resolution, affirming the principle of human control over weapons of war, accompanied by a non-binding code of conduct. Such a measure would require engagement and restraint of all declaring parties at all times to ensure compliance with international law.⁴⁵

4.1.3 International ban of AWS

The final solution would be the adoption of a legally binding international ban on the development, deployment and use of AWS. Such a ban could come in the form of a new CCW protocol, a tool used to address weapon types not envisioned in the original treaty⁴⁶ or in a form of a separate convention.

4. 2 Corporate responsibility and code of ethics

While it is true that corporate social responsibility is no substitute for government regulation, it can be an important element in a larger regulatory system.⁴⁷ Companies can make sure their products are sold by licensed and reputable dealers that conduct background checks or they could commit to selling military and law enforcement equipment only to government clients.⁴⁸ Otherwise, should they be held responsible for the foreseeable consequences that flow from use of their products? This responsibility could include civil liability and, in cases involving war crimes and violations of human rights, responsibility under international human rights standards.

Other initiatives, primarily driven by employee protests at tech companies such as Amazon⁴⁹ and Google might contribute in this direction. For example in 2018, over 3,000 Google employees signed an open letter⁵⁰ in protest against the company’s involvement with a U.S. Department of Defense AI Maven project that studies imagery and could eventually be used to improve drone strikes in the battlefield. Google employees expressed concern that the U.S. military could weaponize AI and apply the technology towards refining drone strikes and other kinds of lethal attacks. Google reaction was to stop its involvement in the project beyond its original contract. In addition, in its code of ethics, Google wrote that it will not design or deploy AI in weapons or other technologies designed to cause or directly facilitate injury to people; in technologies that gather or use information for surveillance violating internationally accepted norms; or technologies for any purpose that contravene widely accepted principles of international law and human rights.⁵¹

5. Conclusion

Ethical codes and guidelines are rarely backed by enforcement, oversight, or consequences for deviation. Ethical codes can only help close the AI accountability gap if they are truly built into the processes of AI development and are backed by enforceable mechanisms of responsibility that are accountable to the public interest.⁵² Thus, should embedding existing and future ethical and legal frameworks into AWS be part of the solution? It remains to be seen whether the IHL principles of distinction and proportionality can be encoded into digital formats.⁵³

Some advocates of a ban on autonomous weapons systems seek to ban not merely production and deployment but also research, development, and testing of these machines. As history shows, a complete ban of AWS is simply impossible. Therefore, states must develop an understanding of and rules on which uses of autonomy are appropriate and which go too far and surrender human judgment during times of conflict.

These rules must preserve what we value about human decision-making, while attempting to improve on the many human failings in war.⁵⁴ The term autonomy in the context of AWS should be understood and used in the restricted sense of the delegation of decision-making capabilities to a machine. Since different functions within AWS may be delegated to varying extents, and the consequences of such delegation depends on the ability of human controlling them. A human should still be responsible for programming the behavior of the autonomous system and its actions must be consistent with the laws and strategies provided by humans.

It remains to be seen if the second part of the 2019 GGE session will become an occasion for Canada and like-minded countries, accompanied by corporate actors, to engage in a process similar to the one leading to the Ottawa Treaty and resulting in an international ban of AWS use in military conflicts.

Christiane Saad is Lawyer and Managing Director of the Programme de pratique du droit at the University of Ottawa. Ewa Gosal is Avocate chargée de projet at Association des juristes d'expression française de l'Ontario.