Dear Evan,

Thank you for writing about ICRAC and I a pleased that you said that you cannot judge on the basis of a news report. The committee has been set up to try to get international discussion going about controlling the use of robotics technology in warfare. It is just beginning and is meeting to discuss some of the very concerns that you express hear about definitions -among many other issues.

I am sure that you don’t want to suggest that because you can call a cruise missile a robot that we should not have international discussions about the progress of robotics technology in warfare; that we should not try to get governments to think about the impact of this technology on the fundamental laws of just war and the potential impact on civilians.

Your own admission of confusion about what a robot weapon is – autonomous, semi-autonomous or tele-operated and questions like “is a cruise missile a robot” or even a mine are the very points that need to be addressed and discussed. As you well know there has been insufficient evaluation.

The committee has been organised in response to reading many thousands of pages of military plans, discussions with the international military community and the propagation of robotics technology in the militaryin the air, on the ground and at sea. There is no sign in your article of any substantial reading on the topic either in terms of the US military road maps or the many articles in the defense journals (I can send you a ton of references if you like).

The statement that most concerns me in your article is, “there is absolutely no reason why you couldn’t program a system to do just that, arguably more consistently and reliably than a human could, and robotics experts not only agree, they’re actually working on it already.”

This is called “the technological solution on the horizon” argument and has been the mainstay of new weapons development since World War II (and probably long before). “We know this is very dangerous for civilians now, but let us go ahead with it because there is a technological solution round the corner.”

As a robotics expert, I am well aware of the work being carried out in this direction but it is an extremely difficult problem that even the most optimistic roboticists say will take between 10 and 25 years. I would dearly love to hear in detail exactly what method are proposing. If you can demonstrate a solution, I would be extremely happy.

This is not just a matter of the considerable problem for vision and sensor systems it also requires very large scale development in common sense reasoning for real world applications in very difficult dynamically changing conditions and threats. In insurgent warfare, humans are not 100% at it by any means, but robots or AI systems cannot make the discrimination at all yet. This is fundamental to the laws of war (as in the Geneva and other conventions and treaties).

The other cornerstone of the laws of war is the principle of proportionality: any killing of civilians or others who are morally immune from attack must be proportional to the military advantage. How is a computer system going to calculate that. There are not objective criteria.

In robotics and AI we are notorious for getting our predictions out by many years and this does not matter with all the great beneficial robotics applications that we dream up that eventually come to pass – and there are very many. But when it comes to the lives of civilians, then the responsible attitude is to look at what we have NOW and not what might be. Fortunately most people in the robotics community grasp this point and I hope that you will too.

I do understand your new jerk reaction here Evan, and I in no way wish to criticize you for it. When people first come upon these ideas the think that it is hype or trying to scare people. But when you read the documents and see what is happening you will realise that there is a very serious issue here that is not being discussed at an international level. Scary hype would be talking about terminator style robots turning on us. What we are talking about is current robot technology and its proliferation (over 50 countries with programmes now).

The committee does not have the answers to the questions it is raising and that is why it is we need as much discussion as possible both within the technology community and with the military and policy makers. That is why I welcome your article as part of this discussion.

+++++++++++++++++++++++++++++++++++++++++

PS The concern about nuclear bomb carrying robots arose because of articles such as Adam Lowther’s in the “Armed Forces Journal” suggesting that that Unammanned Air Systems should be used to carry a nuclear deterrent around the world on a 24 hour basis.

Hi Noel, thanks for stopping by.

First, I’d like to apologize if my article came off rather harsh. I’m sure you’re familiar with the negative publicity that military robots have been getting lately, and I’ve become fairly (some might say overly) sensitive to it.

I’m glad we agree that “robot” is a very poorly defined term. Indeed, many of the readers of BotJunkie take issue with some of the content that I post since it’s not (by their definition) “robot-y” enough. For the purposes of this discussion, I was defining a robot as an entity with the ability to make, and act on, a decision (based, of course, on how its programming instructs it to react to data).

Discussion is great, and I think that this is an important issue. However, I think that so far, ICRAC seems to be looking at the future of combat robotics in a negative and pessimistic way, which in my opinion is not only not constructive, but in many ways, detrimental to the entire industry. As I said before, it suggests that autonomous systems are inherently dangerous and that they can’t (and shouldn’t) be trusted. As you point out, weapons development is an inevitability, and especially with the kind of conflicts that armed forces are currently involved in, moving towards unmanned systems is the next logical step. Realistically, whatever the ethical implications may be, we are going to have active unmanned combat systems in the very near future, for the simple reason that it puts fewer “friendly” human lives at risk. It seems to me that rather than demanding that robots be banned from one thing or another, or warning of threats that are unlikely to exist for some time, the ICRAC would be much more effective if it were to optimistically work to develop solutions instead of attempting to resist change. The ICRAC acts as though military and industry are somehow underestimating or unconcerned by the potential for harm from combat robots, but from what I’ve read, that’s far from the case. It’s a very serious issue, and the people designing, building, and programming the robots are aware of this.

To address a few of your specific points, I freely admit that no, I’m not very familiar with military robotics roadmaps. Well, that’s not true, I’ve seen a few. I might suggest that “roadmaps” is somewhat of a misnomer. From what I’ve seen (and feel free to correct me on this) such reports are full of unspecific projections suggesting that by such and such a date, some type of combat force will be replaced by some unmanned systems. There may even be specifics, but as a military projection, my guess is that the focus is on the asset and not the implications. I can see how that would worry you. Again, I’m not very familiar with that kind of thing… It would be helpful, at some point, if the ICRAC could provide specific examples to help educate people on what ICRAC is concerned about. What I do see are reports on active systems like TALON SWORDS: armed robots that are actually being tested in active deployment. I read about how cautious the developers are, and how even the tiniest problem can set their systems back years. And it seems clear that the people who are actually building these robots are taking things very, very, very seriously when it comes to armed robots with any sort of potential for autonomous, semi-autonomous, or even simply remote operation… For ethical reasons, certainly, but also for legal and financial ones. And as I pointed out, there are experienced researchers who are working solely on the issue of robot ethics.

You suggest that there is a need for “very large scale development in common sense reasoning for real world applications in very difficult dynamically changing conditions and threats.” I agree that this is true, if the necessary goal is implied to be that robots are as capable of making decisions as human soldiers are. I don’t think that this is at all the case. We already have many fully autonomous (albeit defensive) robotics systems programmed with instructions no more complicated than, effectively, “if someone shoots something at you, shoot back.” Why not have defensively deployable autonomous combat robots with similar logic? While I’m not qualified to discuss sensor or hardware specifics, I have little doubt that a robotic system under sustained fire could locate the source of that fire with more accuracy than a human. Even such a limited system would lives. And the larger point is, of course, that it’s the robotic system that’s under fire. As General Lynch points out in the other article I posted, we’re risking (and losing) human lives in situations where such losses are completely unnecessary.

My reaction to ICRAC isn’t knee-jerk, mostly because this issue keeps getting raised over and over in the media. You’re experienced enough to understand that when someone issues a report like the ICRAC is planning on issuing (or has issued?), only negative headlines are going to result from it. Yes, it’s going to make people think, but primarily because they’re scared, and it’s that fear of robots that’s so terrible for the entire world of robotics. This is why I think ICRAC is going about attempting to resolve these issues in entirely the wrong way. Imagine, for example, if ICRAC were to tackle the issue from the other side, and instead release a report entitled, “Preliminary Suggestions On The Deployment Of Safe Armed Robots.” Such a report would get people thinking about the same issues, but in a completely different way. A positive way. Really, it’s not so much what ICRAC is trying to do… I understand that, it’s a good idea, and I support it. It’s how it’s being done that bothers me.

That said, I welcome discussion on this topic as much as you do, and I hope that some of my comments may be taken into consideration. I look forward to more detailed information from ICRAC, and also to the development of safe and ethical robots, however we may end up getting there.

Thanks Evan – I do find you comments and discussion helpful even if we are on different sides of the fence. I think that you will find that very much of the negative media surrounding armed robots emanates from me.

Having said that you will also find that much of the media about the positive aspects of robotics emanates from me as well. I also want to be inclusive and am not hard set on my opinions.

Discussions about the armed robot issues are not simple – you point out one case of a defensive robot sentry for example. That is fine for some cases – for example the fully autonomous CWIS Phalanx has been deployed on US Navy Ships for about 15 years to allow sailors to sleep well at night.

But we must be every wary of words like “defensive” and keep an eye on what can be meant by that. It is one of my tasks to seek clarity about words such as “defensive”.

Anyway, this was only an initial announcement of ICRAC and we are working on a report that will probably take a year. It has to be very careful as our job is not to interfere with the military but to get discussion going for all our benefit. We will keep you posted.

As an aside, it would be great to run a challenge to see if we can get someone to define robot precisely. A good definition would only include robots and not items like alarm clocks and printers.

best,
noel

Good points Mr. Sharkley. This has to be a definition that will communicate with both the general public and the specialists. In day to day conversation I’ve found it interesting that people “know” what a robot is but that term doesn’t have a definition in their minds similarly to how the problem manifests itself in the research environment.

When K.Chapek coined the term he was referring to a machine that did repetitive, drudging work in harsh conditions. His robots were biological, humanoid and autonomous but that didn’t stop everyone from adopting it for a number of systems. An industrial arm is still defined as a robot. At the same time a lot of systems that execute motion in an industrial
environments are not considered robots (like a blender).

From what I’ve noticed a common person brands something as a robot when:
- the machine has motion capability
- it is somewhat independent of the large machine it’s operating in (autonomous systems are automatically branded robots but robotic arms in industrial areas are also considered robots because people perceive them as “working on the assembly line” not part of the assembly line).
- it is not a unitasking machine. Even if it it doing a single thing over and over again the concept that it can be quickly reprogrammed to do something else breaks some sort of psychological barrier between “robot” and “device”

If the above are true there’s one more criteria that people use to tell themselves if something is a robot and that is if it can be anthropomorphised somehow. That’s a shallow reason but think about it. On discovery channel the promos are with a couble of industrial arms talking to each other. When an assembly line accident happens with a robot people say “the robot hurt/killed Jack”. Packbots become the buddies of the people operating it in the squad. Anthropomorphism has a powerful role in the robot concept.

Anthropomorphism is part of why a cruise missle isn’t considered a robot even if it meets some of the above criteria. Put a feedback system on it like sending back text messages with status updates like “I’ve almost reached the target.” and “Weather within expected parameters” gathered from it’s own sensors and that point will become fuzzy.

The only criteria it doesn’t meet is that it’s a unitasking machine. You can’t reprogram a cruise missle to do recon. It’s a missle. You send it – it explodes. It’s a unitasker and the people operating it know that. That’s why they don’t call it a robot.

Just to begin, all there committees and organizations aren’t much more than plain politics. They are not seeking human safety, but a way to acquire power, therefore money.

If they were really concerned about people and life, they should move against war. But war is so profitable, and they are so enslaved by money that no real effort against it is made.

Now about autonomy and semi-autonomy. That’s quite gross to say that autonomous systems have just “many more” “if” conditions.

An AI system may have very complex data structures. For example. suppose that Asimo is able to walk in any stairs, learning it on the go. In other words, those rules are not hard-written in a source code to trace and debug it. The more complex a system is, the harder it is to determine where are the problems.

But about the whole thing, my opinion is that nuclear/biological/chemical/etc weapons in autonomous systems are not much dangerous than in human hands. They would not condemn robots using them, but it’s own existence.