Keywords
robots - health information systems - operating theater - team–machine interaction
- hybrid action - health care
Introduction
How conscious and “free” are decisions in a pervasively digitized work environment
like a robotic operating theater? How will self-perception and self-conception change
when decisions and physical actions of health care professionals heavily rely on robotic
assistants? Is it empirically feasible and normatively justifiable to refer to a particular
physician as “having the last word” in such a team–machine interaction? What ethical
and legal consequences for self-conception, responsibility, and liability arise? Within
our current age of digitization, which is bringing enormous changes to health care,[1]
[2]
[3]
[4] such questions have to be raised also in the context of how this will impact and
be impacted by the design of health information systems (HISs).
HISs have seen incredibly rapid development during the past decades. For comparison
we can refer the reader to the seminal paper of Peter Reichertz[5] on hospital information systems, their architectures, functionalities, and challenges
in the 1980s, to Haux[6] for developments up to the first decade of the 21st century, and to Arnrich et al[7] Friedman et al[8] and Winter et al[9] for more current observations with a focus on pervasiveness, strategic management,
and learning. In Haux's work,[6] a seventh important line of development for HIS is noted as “the steady increase
of new technologies to be included, now starting to include ubiquitous computing environments
and sensor-based technologies,” using terms like “active environments” and “sensor-based
ICT.”[6]
Outstanding examples in this trend of intensively including new technologies are operating
theaters, where robotic systems are increasingly supporting surgical teams, way beyond
providing simple tools. The more these robotic systems act autonomously and intelligently
(meaning, among other things, providing the basis for human decisions), the more questions
need to be raised about the implications for clinical workflow and for revising team
work that includes machines as well as humans.
In this article, we want to address one of the key questions that needs to be answered
in this context: Does this new hybrid action between humans and machines imply a sharing
of autonomous decisions between (teams of) humans on the one hand and machines on
the other hand and if so, what consequences can be foreseen from this?
Underlying questions with important consequences include: which work processes will
change through the use of robots in operating theaters? What do patients expect and
will they benefit? Will there be new requirements for surgeons' and how they deploy
their skills to work successfully with robots? What additional technical know-how
will be needed for them and for nurses in surgical teams? How do roles, responsibilities,
and communication styles change? Who is authorized to include the robots in surgery?
With additional data, coming from robots, are new criteria for quality management
needed, and if so, set by whom and to whose advantage will that be? That is, who determines
what is “good surgery” and who are “good surgeons,” and what norms and deviations
from norms are professionally, legally, and ethically tolerable?[10]
Objective and Structure
Discussing “highly original and relevant research fields for biomedical and health
informatics”[11] and stimulating “multidisciplinary communication on research that is devoted to
high-quality, efficient health care”[12] is one of the major goals of Methods of Information in Medicine.[6]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20] New forms of team–machine interaction in health care with robotic systems in operating
theaters are an example of highly original and relevant human–machine interaction.
Our objective is to discuss from several viewpoints the challenges of such new team–machine
interactions and their consequences for HISs, in particular on how to adequately take
into account hybrid action. We will present our thoughts on this topic by using robotic
systems in operating theaters, and, as far as we know, this is the first time that
this topic is addressed in a multidisciplinary way.
The mentioned challenges are presented from surgical, robotics, information systems,
legal, and ethical points of view. The present contribution elaborates on the authors'
presentations at a recent related symposium.[21]
On the authors' contributions to this manuscript, J.S. and D.F. respectively initiated
and catalyzed our discussions on the topic. R.H. is responsible for “the information
systems viewpoint” section and for the overall organization of the manuscript. D.F.
is responsible for “the surgical viewpoint” section, J.S. for “the robotics viewpoint”
section, S.B. for “the legal viewpoint” section and A.M. for “the ethical viewpoint”
section. For all other sections, the authors are jointly responsible.
The Surgical Viewpoint
Team interaction is highly important in operating theaters. Every team member has
his or her own work assignments, but communication is required for a good interacting
team[22] to obtain the best possible surgical results and outcomes for the patient.[23]
[24] When a robotic system is introduced in this established situation, significant changes
in professional roles, interactions, and communication can be expected,[24]
[25] furthermore the former team interaction is radically changed into a team–machine
interaction.[22]
[26] Possibly, there will also be changes in surgical procedures, due to the requirements
of robotic systems.[27] Because of this team–machine interaction, all coworkers will have to learn new skills,
as well as to integrate machines as team members.[22]
[24]
Physicians, especially surgeons, believe that good doctors need to accumulate a lot
of experience to become a high-level practitioner.[28] They perceive their profession as an art, a challenging craft that requires a lot
of creativity, courage, and determination. When analyzing complicated situations within
a surgery procedure, including cooperation requirements in operating theaters, much
beyond precision alone is required. There are situations that need nothing less than
creativity. At the moment, we solely attribute creativity to humans.[29] Thus, if machines are able to substitute surgical work, if only in part, this fundamentally
questions the self-conception of surgeons. It will likely deeply influence the professional
behaviors, self-conceptions, self-perceptions,[30] self-efficacy, and the teamwork structures in health care.[31]
The medical profession and its fields of action will change dramatically, but so will
the work profiles of other professions, associated with physicians, especially in
operating theaters.[22] These professions seemed largely irreplaceable until now, because of their specialization.
But general conditions in health care systems and the requirements in operating theaters
in particular will likely change considerably and continuously in the near future.
New professions may be forming and well-known ones may disappear. In part, they may
be replaced by intelligent technology,[32] i.e., algorithms and machines that have control functions and take over responsibilities
for reporting and evaluation. Such machines may be more precise in measuring and acting
than humans are[33] and they can perform in a more reproducible manner. Furthermore, by using machine
learning or artificial intelligence (AI), they may resolve problems more efficiently
than humans can, and they should be designed so as to not harm patients (precautionary
principle).[34]
[35] But, can we really count on this kind of seamless embedding of machines in the surgery
team work or is this only a simplified projection of our technological wishes? Much
data must still be collected and new educational environments and scenarios will have
to be developed to fulfill these visions as “next decade medical solution.”[36]
Furthermore, it is necessary that the use of machines is based on good evidence,[22]
[24] even more than with any other new surgical technique.[37] Therefore, new techniques including robotic or other digital devices need to prove
superiority over, or at least equivalence to, established conventional processes and
procedures. Claimed benefits of robotic surgery include not only miniaturization,
but also the “virtual reality” ability of giving the surgeon the feeling of being
within the body of the patient, e.g., the abdomen or chest etc., to provide navigation
in small spaces with the best possible view, and also improved overall control from
consoles and through communication techniques.[27] For now, however, a first and most basic necessity is to conduct studies on when
and how machines should be used in operating theaters.[22]
[26]
As of today, most people do not want to be treated by fully automated systems, nor
are surgeons that confident in delegating their existing responsibilities to autonomous
machines. Patients mostly require the contact with a “real-life” physician carrying
out the decision-making processes in their individual case.[38] To date, the majority of people in European countries, where health care systems
in general work well, are not willing to be treated for major surgery just by robots
instead of human surgeons.[38] Nevertheless, the use of information and robotic assistants is increasing and more
widespread and, in the future, one might speculate that it might even be rated as
the better practitioner. But it is important to define what we mean by a robot: is
it a fully autonomous system or an integrated robotic system, interacting with humans
and with other systems (electronic patient files, electronic hospital information
system, complication registry, searchbots for integrating best publications, data
resources on surgical/medical experience, etc.) to optimize the surgical result, through
interaction processes, as a kind of smart assistant?[38]
Thus, introducing robotics and smart information processing in operating theaters
rises at least the following basic questions:
-
How will responsibilities change with the use of robots in operating theaters?
-
How and when might it be safer for patients, if robots do the work?
-
Is it necessary that robots be supervised by surgeons or even new health care professionals?
-
Who will do surgical trouble-shooting for the many possible complications of surgery,
if surgeons will only care for machines? Do we need specialized teams, trained in
open surgery to intervene in case of problems?[39] A dramatic situation that has to be managed is, if machines make fatal errors, e.g.,
system breakdowns,[35]
[40] operating crews could be completely blocked, and patients could be in great danger.
In this case, all staff should be able to react by de-docking of robots and continue
and finish the operation using conventional operating techniques,[23] e.g., traditional microinvasive surgery or open surgery. How can surgery teams be
trained to prepare for such situations, especially when decreasing expertise in conventional
surgery might follow from increasing robotic surgery?
-
What about other professions surrounding the surgeon: anesthetists—should not robots
also do anesthesia? Nursing in operating theaters—should not robotic systems, for
example, change instruments automatically? And furthermore: intensive care unit staff—could
not they be replaced by robotic systems in performing surveillance?
-
Who will work in future in operating theaters? Operators, technicians, assistants,
supervisors—will they need to have medical as well as computing or technical experience?[22]
-
Is it promising to work with robotic systems as training partners or as educational
tools[36] in a more controlled context? This will only happen if robotics is developed for
conventional surgical techniques, rather than only microinvasive surgery[27] as is predominant today.[38] But this would broaden all the previously discussed problems over all disciplines
of surgery.
-
On the level of data processing, the digital nature of the robot's system control
functions generates detailed data about and within the procedure. It creates highly
sensitive electronic traces of the patient on the one hand and of the surgical team
on the other hand ([Table 1]). This raises many open questions such as: who will be the owner of what parts of
the data, what will be done with them, and will they be used with beneficial or detrimental
intentions, and for whom?[22] The discussion of these questions cannot only consider economic evaluations[41] which, in the worst case, could lead to limitations in the quality of patient care
and security.
Table 1
Electronic data traces from robotic surgical procedures: patients and surgical teams.
Topics that should to be discussed on the background of introducing robotic systems
in operating theaters
|
Patient
|
Surgical team
|
|
Controllability
|
|
Intraoperative actions
|
Efficiency of surgical procedure, time management, etc.
|
|
Quality of work (e.g., suture, etc.)
|
Selection of approved surgeons
|
|
Treatment improvement in conjunction with outcome parameters
|
Improvement of training of skills, speed of surgical performance
|
|
Accessibility
|
|
Quality of surgical outcome: organ functioning, mobilization, social/familial reintegration,
etc.
|
Quality of surgical outcomes and complications: wound healing, pain, bleeding, etc.
|
|
Evaluation of the result: activity of daily living, function, aesthetics, satisfaction,
etc.
|
Evaluation of the result: organ functioning, functional limitations
|
|
Recovery/convalescence
|
Time management efficiency
|
|
Resource consumption
|
|
Who will be treated and who will benefit from the procedure?
|
Material, time, staff
|
To summarize, robots may be beneficial, if studies prove their advantages in operating
theaters.[42] These studies ought to yield tools that evaluate if the technology improves patient
care and how to avert harm from surgical teams.[22] A new structure of professions with new skills is most likely to arise. But for
the case of dangerous situations for patients, teams should still be trained[43] in conventional surgical techniques.[23] Economic aspects are important.[28] Nevertheless, they are not the most important ones considering the introduction
of new team–machine robotic systems in operating theaters.[29] Medical, technical, social, political, ethical, and legal aspects as well as patients'
views and wishes are very important and need to be discussed and analyzed. But a major
overarching question in this discussion is, who will define quality criteria and for
whom?
The Robotics Viewpoint
The operating theater is a particularly complex, interesting, and relevant domain
to investigate the interplay of robotics and advanced information technology and their
interaction with the different types of users, which range from the surgeon to the
hospital's management level. Evidently, the digitalized operating theater is a highly
sensitive environment and relevant in terms of the implications of novel technologies.[44] It has a history of using robots in the spirit of full automation[45]
[46] and robotic surgery is a very active field of technical research.[47] Thus there is a richer literature on the impact of technology in the operating theater
than for other fields of medicine. There are increasingly powerful assistive systems
for all levels of physical action, information processing, and decision-making, both
in the surgical suite and beyond. It has, for instance, already been argued that a
high degree of pervasion with robotics and AI makes the assignment of responsibility
in the operating theater difficult,[48] when decisions are prepared, supported, or taken through data processing systems.
The increasing emphasis on using machine learning and other AI methods which are difficult
to understand strongly contributes to this challenge. It has also been argued that
there is the trend away from fully automatic robot operation to continuous guidance
and assistance,[46] which we believe will change the nature of the human–machine interaction significantly.
In physical assistance, continuous interaction takes place, where the robot suggests,
guides, or corrects the human's action in an ongoing manner over longer time spans.
In decision-making, continuous assistance occurs through the real-time processing
of the relevant data for conducting the procedures. The division of labor between
machines and humans is even less clear when work is performed in a team like in the
operation theater where cooperation and communication is mediated by information displays
and where roles, hierarchies, and the value of specialized expertise change. By providing
the technical communication framework and real-time information, the technology then
also strongly contributes to a common ground for teamwork. Overall, this can strongly
blur the boundaries between the human and the machine and we denote this type of continuous
team–machine interaction as hybrid action.
We here focus on the use of the DaVinci system, although similar considerations hold
for other advanced robotics as well. The computer-based application system, displays,
and the robot arms of the DaVinci system provide assistance to the surgeon and the
team, and de facto also strongly influence the structure and organization of all surgery
teamwork and the procedure itself. Nevertheless, the vendor claims that the surgeon
is solely and at all times in full control of the robot, which thus is regarded as
an advanced but teleoperated tool, which also provides the basis for its certification.
If one follows this account, there is no problem in assigning responsibility and roles
between the robot and the operating human. This standard treatment, however, falls
short of accounting for the complexity of hybrid action in the operating theater.
We here argue that in a broader context including the teamwork in the operating room,
the picture is not that simple. Hybrid action as we see it rather blurs the border
between humans and machines and is in the vein of earlier notions of hybrid systems[49] and hybrid agency.[50] It does not fit in the scales of the standard and much discussed notion of robot
autonomy,[51]
[52] which assumes a clear distinction between the machine (i.e., automation system,
robot, software agent, decision assistant) and the human, with both sides carrying
out their tasks mostly independently. Typically, the human is required to set the
overall goals and often is supposed to be ultimately in charge of the final decision,
i.e., to have the so-called last word. Notions of mixed initiative control,[53] shared autonomy,[54] or supervised autonomy[55] have been introduced to denote the increasing amount of cooperation between artificial
agents and humans about how and which parts of a task shall be executed and by whom,
but they still assume a clear and strict distinction between the robot and the human.
Shared control and shared autonomy are also standard concepts in telerobotics,[56] mostly denoting the superposition of control signals from human and the robot, which
also is the case for the DaVinci system. The latter is regarded noncritical as long
as the human still sets the goals. In this context, the robot in the operating theater
falls in the class of assisted teleoperation, which implies a very low degree of autonomy.
This ignores, however, the increasing pervasion of the operation theater with the
respective information systems,[44] where information systems are used in a broad sense to comprise both dedicated expert
systems as well as the even more pervasive background hospital information and data
processing systems, including those accompanying an assistive robot. Note that this
comprehensive embedding of the physical technical system, here the robot, in the networked
information systems and its enhancement with powerful tools from AI is in general
seen as necessary and desired to realize a specific “intelligent function” of the
robot. And we expect that this will hold all the more for future more advanced AI
systems that include even more decision-making and will routinely touch all parts
of the “sense–plan–act” cycle. It is therefore not an option to limit ourselves to
simpler systems without such a function. Thus, it stands to reason that it is necessary
to consider what it means to receive continuous assistance for the human and what
the respective hybrid action implies in particular for competences and capacities.[57]
The interplay between robotics and advanced data processing enables also a meta-level
for introducing machine learning and other advanced data analytics. It can, for example,
safely be assumed that through a larger number of robotic installations scaling effects
appear that allow for aggregation of data across HISs, robots, and local data infrastructure.
Some trends include, for instance, the collection of robotic and haptic data from
different robots and surgeons to investigate automatic skill evaluation[10] by means of data-driven learning systems.[58] The latter is an example of the trend to develop novel quantitative measures of
manual skills through assistive robotics. This may have far-reaching consequences,
as it may become feasible to provide online evaluation and case-by-case feedback in
an unprecedented manner. It can be assumed that there will be much interest from stakeholders
to use automatic skill evaluation to reduce costs and possibly to introduce quality-based
payment schemes. In another vein, too much assistance may lead to skill deprivation
and loss of competences.[57] Automatic and assistive systems implicitly set norms and it will be difficult to
ensure that the inevitable mistakes that novices and apprentices make will be tolerated
in a highly assisted environment.
In summary, we find that the notion of autonomy is not appropriate to fully embrace
the cooperative, shared, and implicit nature of the human–technology relationship
which we denote by hybrid action. From a robotics and data-processing point of view,
continuous assistance in physical action and decision-making like in the surgery room
creates a complex mutual dependency between the single human, the team, and the robotic
system which should be considered more when introducing such systems. Networking and
possible scaling effects for big-data applications may introduce novel means to quantitatively
assess performance in the operation theater, implying far reaching consequences and
creating novel risks like skill deprivation, which is well known in automation.
The Information Systems Viewpoint
The Information Systems Viewpoint
From an information systems viewpoint, the temptation might be to regard robotic systems
in operating theaters just as a new but mainly similar type of modality within hospitals
as, e.g., X-ray machines in radiology departments. This would imply that there are
no real new consequences for information system architectures and for respective information
management strategies, as dealing with such modalities is well known. In our opinion,
this view is much too simplistic and completely inadequate. To explain this, we need
first to explain how information systems personnel (e.g., chief information officers
of hospitals or medical informatics researchers focusing on HISs) view information
systems and their management. For this view it is better to consider hospitals as
a whole, with operating theaters constituting parts of hospitals.
Let us start with a common definition on hospital information systems (HOISs), which
themselves are important instances of HISs: “A hospital information system is the
socio-technical subsystem of a hospital, which comprises all information processing
… . Typical components of hospital information systems are enterprise functions, business
processes, application components, and physical data processing systems.”[59] Literature on HOISs, its architectures, and its information management strategies
spans many decades.[5]
[6]
[60]
[61]
[62] In larger hospitals, HOISs consist today of thousands of connected computers (including
modalities with embedded computers), with about 100 functionally comprehensive and
connected computer-based application systems implemented on these computers. These
application systems support users (health care professionals, i.e., physicians, nurses,
etc.) in a variety of functions for patient care, hospital management, and biomedical
research. Important services are to appropriately store data, in particular patient
data, to have these data available when needed, and to support decisions. How HOISs
can at best provide adequate services, in particular for health care professionals,
is an ongoing challenge.[63]
One of the pioneers of medical informatics, François Grémy, summarized the situation
as follows: “Any technology sets a relationship between human beings and their environment,
... This is especially relevant when dealing with large automatic information systems,
developed to contribute to the management and integration of large organizations,
such as hospitals. In such a context, the environment is mainly made up of humans.”[64] In other words, HISs are setting relationships between human beings, here mainly
between health care professionals, who themselves are caring for patients. For health
care professionals it is most important to know from their HOISs about a patient's
diagnoses, her or his medications, and about the existence of problems such as allergies.
Why should robotic systems and X-ray machines not be regarded as mainly similar types
of modalities? Modalities like today's X-ray machines act passively. They produce
images with additional information, which are then viewed and which are also stored
in picture archiving and communication systems for, maybe, further use and further
analyses. The functionality of these machines is limited compared with the functionality
of robotic systems. In contrast, robotic systems in operating theaters are increasingly
active participants. This is why team–machine interaction and hybrid action of humans
and machines have to be considered, as discussed in the sections above. Through increased
digitization of health care, with sensors and actors, denoted, e.g., as assistive
systems for pervasive health[65] or as health-enabling technologies,[66] these robotic systems are not the only machines, or “nonliving entities,” respectively,
which can be viewed in this way. We nowadays also have “intelligent” beds or “intelligent”
rooms (e.g., in wards) which behave similarly and can know a lot about patients, their
behavior, and their health. Similar developments can be expected for future cars (patient
logistics, etc.) or for future implants like, e.g., pace makers. It is important that
these nonliving entities can adequately communicate with other entities, in particular
with “living entities” like health care professionals or patients and their relatives.
This might be denoted as “seamless interactivity with automated data capture and storage
for patient care, and beyond (from perception to high-level semantic concepts, related
to human–human, machine–machine, as well as human–machine interaction; “beyond” in
the meaning of not being restricted to certain disease episodes).”[67]
Similarly, as for living entities it might be also helpful for nonliving entities
to have access to patients' data, and may be even to get support in making decisions.
For robotic systems in operating theaters as well as for intelligent beds in wards,
etc. information on their patients' diagnoses, their medication, and the presence
of allergies may also be of importance to appropriately doing their services. From
this point of view, robotic systems in operating theaters as well as related nonliving
entities in hospitals may no more be regarded simply as modalities of HOISs. In analogy
to health care professionals, they should be viewed as HOISs users.
HIS architectures, with HOISs as important instances, as well as information management
strategies have to consider this dramatic change, caused by a significantly increased
hybrid action of humans and machines in hospitals. Putting it into a nutshell with
reference to the quoted text of François Grémy: HOISs should set a relationship between
both human beings (living entities) and “intelligent” machines (nonliving entities)
as well as with their environment.
The Legal Viewpoint
Law sets a societally constraining framework for technological developments. Testing
and using robotic systems in medical contexts is only possible if the persons involved—producer,
programmer, user, etc.—know about legal risks and how to avoid them. If the danger
of being (individually) responsible for damages is too high, one might restrain oneself
from developing the technology further. Thus, it is important to understand the legal
circumstances that might pertain to new technologies such as robotic systems in operating
theaters.
First, one has to analyze what is special and new about these systems compared with
the machines we are already using[52]: the machines lead to distance between doctors and patients, create new potentials
for violating the patient or mistreatment (such as malfunctioning machines or wrongful
usage by the doctors). Traditional education is not sufficient to know how to deal
with these machines and knowledge of traditional operations might get lost with the
use of these systems which is known as skill deprivation in automation. The systems
will collect huge amounts of data and will be connected to platforms and other machines.
And, most importantly, some of these machines might act (partly) autonomously, preparing
or even making decisions and taking part in actions.
All these new characteristics will produce new challenges for the legal system in
different areas.
One relevant legal area is administrative law, as these machines have to be licensed,
tested in hospitals, approved by ethics committees, etc.[68]
[69]
[70] In this complex legal field, there are many laws applicable to our robotic systems,
such as, for example, the Directive 93/42/EWG about Medical Devices, the Medical Devices
Act, or on a nonstate level, the DIN EN ISO 14155 or the Declaration of Helsinki (about
research on humans). These laws are meant to ensure the safety of the patients and
therefore set the framework of acceptable actions and treatments. One has to keep
in mind, though, that these laws are not designed for autonomous systems or AI. Testing
products, balancing advantages and risks, involving ethic committees, and explaining
the device to the patient all these become more complex and difficult in the case
of a device developing over time, being connected to other machines, being trained
by different people, etc. Therefore, it is necessary to adapt the administrative laws
to the technological developments and balance the interests—and the ethical as well
as legal responsibilities defined for the people involved.
In civil law, one can find regulations on the relations between a doctor (clinic)
and the patient, as well as between a producer and a consumer (clinic/doctor). The
contract between doctors and patients in Germany is regulated in §§ 630a et seqq.
BGB.[71] The doctor is obligated to act according to his professional standards, to explain
all circumstances to the patient so that she or he can give fully informed consent.[72] As robotic systems in operating theaters are partly new and not fully tested systems,
the doctor also has to inform the patient that he is not using traditional methods
but new systems. Especially if we are confronted with AI (but not only then), the
risks for all persons involved, especially for the patient, are not fully known. The
system might develop in unpredictable ways and not be fully understandable and truly
controllable. This has to be included in the explanation of the operation.[72] One also has to be aware that, until now, there are no specific standards for the
contractual obligations of all people involved—for the doctor, the clinic, but also
for the producer: the novelty of these machines means that one does, so far, not fully
know what to expect of the product, of the producer, as well as of the user. Therefore,
the obligations—including the contractual obligations of the producer—are not clear
and should be clarified by the interpreter of the law as well as the lawgiver.
These missing standards could be one of many problems for the liability and responsibility
of the people involved: the programmer, the producer, the hospital management, the
doctor, and the nursing staff could all, in general, be liable for damages resulting
from using the machines.[73]
[74] Generally, one only is liable in the case of negligent behavior, except in some
cases of strict liability. The producer, for example, could be liable for defects
of the products even if he did not act negligently. The liability of the doctor, for
example, does depend on his acting wrongfully, meaning to transgress the generally
accepted professional standards. In this context, the doctor might not only face liability
in terms of having to pay damages, but even individual criminal responsibility. The
clinic could be responsible for either installing machines too dangerous for the patients
or, at some point, not installing machines although they might be provably more efficient
than traditional machines and/or human beings. In the case of new robotic systems,
we will be faced with, as mentioned, missing standards and other changes of the existing
liability/responsibility regime.[72]
[75] Not only is it difficult to prove what went wrong in the case of such new machines,
it also is problematic to talk about misconduct or negligence if there are no clear
and specific standards about how to act correctly. One will have to distribute the
liability between the people involved—not only because of the difficulties to prove
where something went wrong, but because the end result, the treatment, is in fact
the result of the hard-to-untangle collaboration of different participants. For the
distribution of responsibilities, again, so far there are no established rules. Especially
for the patient, it would be preferable if he would not have to prove who made a mistake,
who acted negligently, but to be able to address the collective directly. For this
undertaking, it is discussed to introduce the so-called “electronic person.”[76] This construct allows, analogous to the already existing legal person, to address
the machine as a symbol of the people involved. The machine would have its own monetary
account, which could then be used to pay legally determined economic damages. These
accounts would have to be negotiated, legally justified, and value-assigned by the
humans involved. The machine could be registered and addressed from the moment of
registration onwards. This construct does carry its own problems which are too complex
to be discussed in detail here—just as an example: the construct only addresses a
specific machine, while, in reality, most of these machines will be connected and
act on information from these connections, leading to implausibility of one of these
machines being liable for damages stemming from these networks. But there will be
more problems, and one will have to consider if the advantages of such a construct
outweigh its disadvantages. Regardless of how this question is answered, one would
also need to develop other mechanisms, such as mandatory insurances, social liability,
etc. The most important aspect should be that the patient, if (s)he becomes a victim,
or bearer of the harms or damages, should not face the difficulties of the new technology
alone but should be insured so that the damages should be paid by the ones who draw
the largest profits from the new technologies.
One of the biggest problems in the context of responsibility, though, will be the
responsibility of the “human in the loop.” In many contexts of rising AI, it is demanded
that there should be a human in the decision loop to ensure the inclusion of human
values and human capacities in the decisions.[77] It might even be questionable if this does, in all cases, increase the quality of
the decision. But even if it did, this does not answer the question if this demand
actually is excessive compared with what the human being is able to achieve: if the
car is driving, the human driver will not pay enough attention to overtake another
car at any given second of time, but leave it to the automatic system. If the computer
suggests a specific diagnosis, the human doctor might be strongly influenced by this,
and thus forget other potential diagnoses he might have thought of by himself. This
is unavoidable and should, therefore, not be imposed on the human individual who often
did not even choose to use the machine. Altogether, it becomes clear that we have
to find new solutions for liability and responsibility in the context of robotics
and AI.
Finally, our focus is directed toward a specific problem of the machines we are analyzing
here: the protection of data. On the one hand, the further development of AI depends
on the possibility of using huge amounts of data (Big Data), on the other hand, the
data in our context are very sensitive and have to protected with special care.[68]
[69] The current data protection regime is focusing strongly on the protection of the
individual; especially if one regards anonymization or pseudonymization as almost
impossible nowadays, it becomes difficult to feed AI programs with enough data to
develop them further. This is especially the case for data protected even by criminal
law such as the data of patients. The strong protection is not necessarily problematic,
but one should be aware of the problems it might cause for the technology discussed
here. It might be necessary to find new balances between the different interests in
the future.
To conclude, it cannot be doubted that robotics and AI pose enormous challenges for
the traditional legal regime. This will also be the case for the context of the operating
theaters. The interpreter of the existing laws and especially the law giver are challenged
to adapt the legal situation to the reality we are facing—not in a few years, but
now.
The Ethical Viewpoint
Dealing with ethical implications of new technologies like robotics in operating theaters
means, first of all, to explain what an ethical point of view is. To put it in a nutshell:
ethics is a way of reflecting methodologically on questions with which human beings
decide upon the way they are living together and what it means to be a human being
who behaves with integrity or ethical standards on his or her own and in relation
to all others. Furthermore, questions like what are we to do? and what should we refrain
from? are part of ethical deliberation. However, before coming to a point of arriving
at answers and concrete recommendations, we should set out some more general considerations
about technology and human–machine interaction.
It is more than a laconic stance, that it is difficult or impossible to precisely
foresee the future. In terms of technology there is a very specific problem that we,
human beings, try to avoid unwanted side effects of any technology deployed as long
as we cannot estimate these side effects being acceptable when compared with the expected
benefits. But, how to assess this ratio with respect to a technology that is not yet
established? On the other hand, we do know quite well about unwanted side effects
once a technology has been deployed, but then, it is very hard to retract this technology
bearing more harm than benefit. This “dilemma of control” has been named after David
Collingridge, a chemist and philosopher of technology,[78] and still is challenging our thinking when establishing a new technology.
So is the case with robots and expert systems in the operating theater. We can imagine
that this technology will impact the number and structure of employment in respective
teams and companies, as new technology did in any industrial sector in the past centuries.
We may suppose that it will change the professional roles and hierarchies. Will it
be for good or for harm, or indifferent? Will the up-skill and down-skill effects
on employees which are already observable in other sectors working with robots have
a balanced score and will it tend to have negative[79] or positive[80] effects on employment? There is no clear answer yet. Whom should we trust: the prophets
of “creative destruction”[81] telling us that the rate of employment and innovation will far outweigh the loss
of former jobs? Or should we listen to those warning against a new caste of “redundant
people”[82] caused by digitization and robotics in every branch profitable for automation? Obviously,
these are not technical questions, nor are they issues, which can be addressed in
economic utility terms alone, but it is a vital socio-political question and an ethical
one in the sense that we decide upon how we will live and work together in the near
future.
Another open question to be considered is how interaction of human beings and machines
can be conceptualized and the direction it is supposed to take in surgical and clinical
practice. There is no doubt that robots equipped with AI and high-precision handling
features already have left behind the state of being a mere instrument. In the literature,
they are called “cooperative partners,” “companions,” or “co-workers,” raising several
questions concerning the technical, juridical, societal, and ethical aspects of cooperation.
Referring to the ethical ones, it is still unclear whether any or which moral status
these machines should be assigned to. Even if European politicians think about an
“electronic-personhood”[83] as a status of responsibility and liability for most sophisticated autonomous robots,
ethical concerns are not covered by this proposal. Ethical reflections in this field
have to address the social and moral status of these machines and the consequences
derived from it. One crucial issue involves the distribution of roles between humans
and machines. Is the human still in control of the machine, or do we have to acknowledge
a superiority of the machine being smarter, more efficient, and more precise? Are
surgeon's teams allowed to deviate from the expert system's recommendation on how
to treat a patient, when the expert system has retrieved data from all relevant databases
to determine the “best” treatment options for this particular patient—in a period
of time never achievable for humans? If these teams were not allowed to do so due
to reasons of evidence, standards, and potential claims for compensation, the question
is whether the human would increasingly be driven into dependency on the machines?
Furthermore, how would this impact the self-conception of human beings as moral subjects,
i.e. as beings who decide upon their actions with free will and a rationale combining
ends and means with their own personal ethical viewpoint on what are acceptable tradeoffs
in responsibilities and actions. It looks like we may be asked to revise strong convictions
about a specific superiority of mankind in terms of decision-making efficiency and
purported rationality—now based in AI on “limited” constraints that are, in practice,
almost always based on economic values from utility theory. Of course, one may ask
whether the fact of being dependent on a faster and better calculating system will
contest human agency and responsibility at all. This is going to be not only an empirical
question, but an ethical—in the sense of analytical—question as well. How do we, human
beings, perceive the way we form an opinion and shape our actions accordingly? What
does it mean to be autonomous in a pragmatic and a moral sense, and how does this
autonomy is compatible with the autonomy of the robot and the expert system? How can
we assert the supremacy of human autonomy theoretically and pragmatically? No doubt,
this is a pressing issue as the committee on rules for robots of the European Parliament
stated: “The Commission is called upon to establish criteria for the classification
of robots that would need to be registered. They also considered it essential, in
the development of robotics and AI, to guarantee that humans have control over intelligent machines at all times and that special attention should be paid to the possible development of an emotional
connection between humans and robots—particularly in vulnerable groups (children,
the elderly, and people with disabilities),”[84] emphasized by the author. What does it mean to have the control over machines at
all times? And how shall we achieve it? The report suggested that advanced robots
should be equipped with a “black box” which records data on every transaction carried
out by the machine, including the logic that contributed to its decisions[84]. Would that help to (re-)gain control over the machine, taking into account that
no human being would be able to retrace the logical combinations executed by self-learning
algorithms in massive amounts of data? Norbert Wiener made a hint towards the systematic
problem behind this demand[85] in the early 1960s already. Philosopher of technology, Christoph Hubig, pleads for
a parallel communication on three layers informing the human beings about what is
happening at the intersection of human and machine.[86]
In any case, the cooperation of humans and machines in general challenges our social
capabilities as well as our mindset to establish new practices complemented by new
theoretical insights into traditional concepts of autonomy, agency, responsibility,
and decision-making. Working on these questions will mean on the one hand investigating
or co-evolving research on empirical questions (e.g., how does the team work changes
in operating theaters when deploying a robotic and AI system?) and theoretical questions
(e.g., what is cooperation between humans and machines in terms of agency, outcome,
and modus operandi? what is the difference between human and artificial autonomy,
intelligence, or agency?). On the other hand, ethical deliberation can never only
refer to empirical data. Ethics first and foremost deals with the question of what should be considered as ethical and moral norms, so that benefits and demands against others
to make the world a better place can be considered. This leads to more specific ethical
questions: what are the criteria for good ethical guidance? The above-mentioned paper
of the European Parliament referred to bioethical principles, i.e., beneficence, nonmaleficence,
autonomy, and justice. Furthermore, researchers and designers were asked to act responsibly
and bear in mind the need to respect, dignity, privacy, and human safety.[87] What does this mean in effect for concrete interaction in the operating theater,
and for the new staff composition?[57] This is the empirical part of research, that ethical deliberation cannot afford
to cast aside. On the other hand, ethical thinking has to stimulate the process of
designing and constructing socio-technical arrangements that meet the criteria mentioned
above. In doing so, ethics has to go into details, exploring and explicating the meaning
of, e.g., privacy with respect to the patient, the surgeon, or any other people involved.
Will the monitoring of every movement of the team in the operating theater lead to
a benchmarking with other teams (or other days) and will this lead to standardization
and a demand for continuous improvement on the basis of objective data? How should
technical artifacts be constructed, how should social practices be established in
order to avoid unwanted consequences and constellations provoking rejection from an
ethical point of view?
Finally, the debate on this issue may not be done by experts only, but should be opened
to the public as a whole, because what is at stake is a question of public concern,
and thus a very pressing one as the European Parliament states: “Humankind stands
on the threshold of an era when ever more sophisticated robots, bots, androids, and
other manifestations of AI seem to be poised to unleash a new industrial revolution,
which is likely to leave no stratum of society untouched. The development of robotics
and AI raises legal and ethical issues that require a prompt intervention at EU level.”[87]
Consequences
Although coming from different scientific disciplines we all agree that the questions,
raised in the Introduction section, are timely and highly relevant and that fundamentally
new approaches are needed to construct a reasonable concept of hybrid action and of
an according ascription of responsibility.
We are convinced that new hybrid actions between humans and machines imply at least
to a certain extent the mentioned shared decision-making between (teams of) humans
on the one hand and machines on the other hand. Collaboration of surgical teams in
operating theaters as well as roles, competencies, and responsibilities of humans
(health care professionals) and machines (robotic systems) needs to be reconsidered.
Hospital information systems will in the future not only have humans as users, but
also intelligent machines. Both will need access to patient data, ask for decision
support and for appropriate communication facilities. These tremendous changes have
to be mapped into legal as well as professional, societal, and economic negotiated
frameworks among others for appropriately considering responsibilities. For all these
matters, ethical considerations have to be the basis.
Although team–machine interaction, and hybrid actions of humans and machines, has
been discussed here in the context of robotic systems in operating theaters, it became
evident that these issues are relevant for health care in general and even beyond.
We are convinced that the expected significant changes in the relationship of humans
and machines can only be appropriately dealt in inter- and multidisciplinary collaboration.
We fully agree with Jan van Bemmel's statement that “Interdisciplinary research is
not a category of research but a consequence of addressing a complex problem in society,
involving the collaboration between and methods drawn from multiple disciplines. Because
research is people, personal interactions are critical for interdisciplinary research.”[88]
