Building Interactive Communication Tools to Support Interdisciplinary Responsible Innovation

Recent science policy encourages the installation of Responsible Research & Innovation (RRI) practices, which should help solve grand societal challenges and be more readily adopted by society. RRI may be implemented by setting up interdisciplinary innovation development teams, bringing together technical and non-technical experts from various disciplines and backgrounds, enabling engineers to let their work become inspired by – or even partly coshaped by – societal insights and viewpoints, while societal actors get acquainted with techno-scientific context. We developed a Decision Support Tool to support interdisciplinary innovation teams, that visualizes innovation project performance and success chances. It supports communication and collaboration in interdisciplinary teams by proposing practical improvement areas, based on shared expertise, including socio-ethical, societal, economic and management related aspects. Still, further investigation is needed to learn how such a tool can be used to systematically integrate RRI in practice, to harness its full innovative potential.


Context and background: Responsible Research & Innovation in practice
Much is expected of technological innovators in terms of addressing current and future societal challenges.Current academic technological research grant applications even have dedicated sections in which applicants are asked to highlight possible future technology implementations in light of resolving societal issues, regarding e.g.environmental sustainability and healthcare relevance.At the same time commercial and industrial research institutes are more and more expected to take corporate social responsibility not merely as a guideline, but as a starting principle for their innovation practices.So, both industrial and academic institutes are stimulated to deploy 'responsible research and innovation' (RRI) practices, either via market demand or via public policies.
In the downstream, decisions are made on how to implement new research and development ideas.To convey the functionalities of these ideas, communication is essential.Still, while decisions are made here on how to install innovations in society, and communication is deployed to support that process, also in the downstream no actual innovations are shaped.
We therefore wish to focus on the midstream, the phase where research and development actions are carried out, and actual innovations are shaped.More specifically, we focus on responsiveness of all involved actors towards socio-ethical and socio-economic aspects, which is essential for RRI to take shape.'Responsiveness' on the midstream concerns 'doing things right'.From an RRI perspective, this includes more than taking anticipation, reflexivity, and inclusion into consideration.'Consideration' in fact could mean that things are debated on, but not actually used to develop new innovative ideas -hence the contrast between consideration and responsiveness.Exactly at this last point, being responsive on the midstream, remains difficult for innovators on the laboratory floor, i.e. the actual scientists and engineers developing new innovations during their daily practices.Possibly they are preoccupied with their technological work (Brunner & Asher, 1992).Or, from a historical perspective, they may see ethics as a break on progress, where ethics indicates what scientists and engineers should not to do (cf.Van der Burg, 2009;Shelley Egan, 2010).Still, some researchers may be aware of the social and ethical aspects of engineering practices, but possibly fail to think about the repercussions of their own work (Patra 2011).More worrisome, some engineers are even explicitly asked to focus on technological development, and asked to ignore 'distracting' social aspects (cf.Fisher & Miller, 2009).
In our experience, scientists and engineers are neither unwilling nor unable to take socio-ethical and socio-economic considerations into account (Flipse et al., 2013a).
Still, these elements are initially 'blind spots' for them, not by definition high on their priority list.Even though such aspects are important to consider, to increase innovation project success.In order to be responsive to socio-ethical and socioeconomic considerations, these considerations need to be made explicit.Apart from institutional support and voluntary participation, interaction with critical 'outsiders' is essential for that to happen.The potential positive role of such critical non-technical / non-scientific experts (Collins & Evans, 2002) to broaden research considerations has been identified and acknowledged earlier ( Van de Poel, 2000;Wilsdon, 2005), but social scientific research to prove these effects have only recently appeared.The way in which we researched this effect in our earlier research, was through the installment of a 'collaborative space' in which scientists and engineers collaborated with an 'embedded humanist' who helped them reflect on social and ethical considerations (Flipse et al., 2014a).'Midstream Modulation' was used as the method to facilitate such interactions.In this method decisions of innovators are 'modulated' into the various elements that they are made up of (i.e.opportunities, considerations, evaluation of alternatives and projected outcomes, see e.g.Fisher 2007).The results of Midstream Modulation research are positive in the sense that the participating researchers appreciated 'opening up' their labs to external viewpoints, and letting their work become inspired not only by their own technological considerations, but also by social and ethical aspects.Similar studies show the ability and willingness of scientists and engineers (see e.g.Conley, 2011;Schuurbiers, 2011).
Still, these studies also show similar results in terms of their limitations.All studies are quite extensive (i.e. 12 weeks or longer), relying on sustained interaction between researchers and outsiders that are allowed to have a critical opinion on on-going innovation practices.Still, extensive collaboration places quite a burden on both the embedded humanist and the involved researchers and engineers.Possibly, 'responsiveness' effects in innovation practice could emerge more quickly if there was a way for embedded humanists or other critical outsiders to know sooner what social, ethical and economic contexts are relevant for researchers to include.Also, such effects could more readily emerge if researchers can more easily relate to such aspects in light of the quality of their own research.An interactive communication support tool could facilitate relevant and quality related interaction between the In our earlier work we aimed to support interaction between multidisciplinary stakeholders in order to functionally shape RRI practices on the laboratory floor through Midstream Modulation.For that we first researched the possibility and utility of interactive communication between researchers and a critical outsider (called the 'embedded humanist'), without compromising the quality and speed of on-going innovation work.The results of this preliminary study (Flipse et al., 2013a) showed that interaction is not only valuable for a more thorough and creative technological development process, but also appreciated by the involved researchers.They claim to explore more research trajectories (exploring more scenarios) than they would have done otherwise, and that they can prioritize their activities better when reflecting better on the societal implications of their research decisions.In any case, the research stresses the importance of communication between involved actors.
The observed effects could be considered important for RRI installment, however there was no 'tool' yet to support RRI, integrated in daily innovation practices.In the meantime, various approaches have been published that present approaches that allow for the 'mapping' of social responsibility (Glerup & Horst, 2014) and even quality criteria and indicators for RRI (Wickson & Carew, 2014).These are valuable starting points to make RRI more concrete on the innovation working floor.However, the direct link with innovation project success needs to be further evaluated.We consider such a link imperative for innovators in order to allow them to seriously consider RRI related aspects.
In a follow-up study we therefore developed a way for external outsiders to learn sooner what is the relation between 'external' broadening aspects and research quality, and for researchers to learn what are the relevant 'external' broadening (RRIrelevant) aspects to take into consideration to further their research.The idea behind this method is that both the outsider and the researcher mutually learn about one another's considerations.Such mutual learning is important for establishing a relationship in which critical viewpoints are not only tolerated, but also valued and actively taken into account (i.e.responsiveness towards socio-ethical and socioeconomic aspects).
We shaped these broadening aspects into innovation Key Performance Indicators (KPIs), which form the backbone of the tool we are presenting later in this paper.The assessment method of KPIs is based on the Wageningen Innovation Assessment Toolkit (WIAT), developed by Fortuin & Omta (2007).WIAT was developed to help organizations in innovation project selection and execution, by providing relevant management information.Based on a statistical analysis of finished projects' features and success rates1 , we distilled how success chances depend on numerous project characteristics.The eight KPIs identified in one case study at a Dutch research organization (Flipse et al., 2013b) relate to external social aspects, including RRIrelevant aspects such as sustainability and health (1), available financial, material and people-based resources (2), communication and cooperation quality (3), technical skills available (4), the technological project superiority in relation to other available technologies (5), the culture of the (internal) customer's R&D culture ( 6), the clearness of (internal) customer's wishes and demands ( 7), and the strategic value to the customer (8).Of course, KPIs can differ per context, yet in any case KPIs are an important units of analysis when studying socio-technical innovation systems, especially when the aim is to find relevant improvements related to communication and decision making.

Methodological considerations for communication tool development
We designed a plan to develop a functional tool that supports interaction between stakeholders in innovation projects, eventually leading to more RRI-relevant decisionmaking.Below we describe first the requirements we had for making the tool that we present later, followed by a set of building blocks that the tool is made up of.How the tool looks is presented thereafter in the 'Results' section.

Requirements for tool
This tool should meet several requirements.First, the tool should help give insight into what could make innovation projects 'socially responsible'.As such, the 'soft' elements of innovation practices that could help guide projects in this direction should be includable.Yet, these elements can only be functionally included if they are assessed in relation to on-going project management.This means that the soft elements that could influence project performance first need to be assessed in relation to the organization in which the tool should be used.Such aspects may range from environmental impact to social impact of innovations, to worker safety and working environments, which may be different in every organization.So, the tool should have a dynamic character and be adaptable to different organizations.
Second, the tool should be considered functional in industrial innovation practices.This means that researchers and engineers should recognize its functionality.This implies that we should develop this tool in collaboration with industrial partners, with continuous user input to safeguard usability in practice.In addition, this means that the interaction with the tool should be such, that critical scientists and engineers are open to the tool's input and visualizations.This means that a certain degree of 'measurability' of quality performance should be incorporated into the tool, which should be visualized in a way that scientists and engineers are used to, such as graphs and relative scores.
Third, the tool should provide critical outsiders with relevant input on other project performance related elements that are considered important by the scientists and engineers who they work with within their joint collaborative space.This means that a certain degree of technical and economic project performance indicators should also be included, to help these outsiders make estimations on the extent to which projects can be influenced on these content-based elements.This should allow for faster knowledge and experience exchange between e.g. the researchers and the embedded humanist, allowing for responsible innovation elements to emerge more readily than they would without such a tool.As such, the second and third point combined, safeguard that all those involved in a multidisciplinary innovation project, have their 'blind spots' covered through some representation in the tool.
Fourth, the tool should also be considered relevant in terms of project management.So, the tool should visualize the effect of using the tool in time in terms of project performance, and therefore should provide insights into e.g.'resources saved' (both on personnel and financial resources levels) and performance changes in time.
Evidence on these levels should help organizations decide more readily that there is an institutional need for such a tool.

Building blocks for online project evaluation tool to stimulate communication
The tool should provide experts with visual output on which KPIs are scoring good, and which can be improved upon.To arrive there, innovation project team members should provide the input to deliver such visual output.Yet such input needs to be compared to a database of earlier projects, in order to be meaningful for that organization.Based on the presented requirements, we envisioned a digital communication tool to support RRI to work as follows.
First, the organization's innovation projects' KPIs are determined using an online survey system.Using a questionnaire with approximately 50-60 potential project success related elements, employees are asked to score a successful and a less successful finished innovation project on a 9-point Likert-type scale.What it means to be successful as a project depends on the organization2 .Relevant success related criteria can then be identified partly based on innovation management literature relevant for the context in which it is used, and can be supplemented with organization-specific elements and social responsibility (e.g.environmental sustainability, and worker/producer safety) related elements, based on both experience and literature (see e.g.Wickson & Carew, 2014).In any case, this first step results in a list of questions and the involved innovators' answers for two kinds of projects: successful ones and less successful ones.
Second, based on the scores of these finished projects, the items are clustered into organization-relevant KPIs through statistical analysis based on exploratory factor analysis.The data could show that for only e.g.30-40 of the total (50-60) items a statistically significant relation can be shown in relation to project success.Only these relevant items are included in KPI determination.Using logistic regression analysis, e.g. in SPSS, the identified KPIs and their interrelations and relation to project success are determined.The result of this step is an overview of KPIs and the elements (questions) of which they consist, each accompanied with an average value for successful and less successful projects.Third, this model is transformed into a visual benchmark that basically contains four 'lines' per KPI that can be mathematically calculated: the lowest possible score for a KPI (on a 9-point scale) is 1, then we have the average of all less successful projects, then a line for the average of all successful projects, and last a line for the maximum score for that KPI, i.e. 9.These lines can be normalized mathematically, so that the resulting graphs have three 'scoring areas' in which projects can be scored: from a minimum value to averagely less successful, from less successful to successful, and from successful to a maximum value.
Fourth, in a subsequent step, this benchmark model based on recently finished and evaluated projects is used to compare running projects to.This works as follows.
Researchers working on innovation projects score their current projects in an online tool, the same way as in the first phase (except with fewer elements, since only the significant items are used now).Based on the project scores, the tool automatically makes a visualization of performance in relation to the developed benchmark.Based on the scores on different KPIs, the researchers and others involved in the project (e.g.outsiders, but also managers, team members, colleagues, etc.) get an idea of what is currently going well, and what can be improved on.In collaboration, the researchers can determine which elements to take decisions and action on.Frequently, this indicates that actions are required on their 'blind spots', things they have not been aware of (just yet).
Fifth, the various inputs of different users could be compared to one another.This way, differences in insights in project quality and performance can be highlighted, discussed, and potential issues can be solved.Sixth and last, the KPI average scores are transformed into a model that (to some extent) can predict innovation project success chances.Using a Structural Equation Modeling (SEM) approach, e.g. using AMOS3 , the KPIs and their relation to innovation project success (as described above) are analyzed and mapped.This results in a model that links the KPIs to one another and to success.Using an agent-based modeling approach in combination with the model that results out of the SEM, the scores of current projects can be transformed into scenarios.Namely, based on the insights of successful and less successful projects in the benchmark, the model can estimate what happens to the success chance if one KPI of a currently running project is increased or lowered.The lower score on that one KPI can have an effect on another KPI, and eventually on project success.This way, users can use their scores in combination with the model to think about possible scenarios of things that could happen to a project, e.g. when they know that a certain KPI will drastically change in the following period (e.g.due to budget cuts, retirements, etc.).

Outcomes
Based on the requirements presented above, the building blocks of the tool presented above, and our preliminary case studies, we developed a tool that aims to support communication about KPI based project performance in multidisciplinary innovation project teams in an industrial context, with an additional focus on enabling RRI through interaction.Here we elaborate on how we used the requirements for usability mentioned above in the design of a functional online 'dashboard' that project team members (both 'insiders' and 'outsiders') and their managers may use evaluate project performance.For visual representation of the first three building blocks, as presented in Section 2.2, we refer to our earlier work (Flipse et al., 2014b).These primarily concern lists of KPIs and their values, as gathered and calculated using computer software like IBM's SPSS Statistics.The other elements are presented below.
The fourth building block can compare running projects to the database of earlier projects.This is depicted in Figure 1, which shows the performance of a dummy project (as scored by an imaginative innovator or his/her team).The benchmark element (building block 3) is depicted as the three areas in the graph, between the bottom line, less successful project line, successful project line, and above.This particular project apparently has many features in common with averagely less successful projects, as earlier defined by this dummy-organization.Communication about these aspects with team members or external advisors can be the strating point for improvement of these aspects.The fifth building block, the comparative analysis element, is depicted in Figure 2. It features the ability to compare different projects to one another, or to compare different moments in time for one project, or even to compare different input of a project by different team members at any given point in time.In terms of communication, the latter part is especially useful if two team members disagree on one particular KPI, so they can more easily resolve differences, possibly complementing one another's viewpoints.Using the sixth building block, users can build scenarios that help them predict what happens to their projects' success odds if they change the value of a certain KPI.This is an additional tool functionality that allows for further exploration of potential improvements.The effect of hypothetical project changes is calculated based on the SEM outcomes.This model visually presents the hypothetical KPIs and their interrelations (Figure 3), may differ per context or per organization, and are based on the earlier project benchmark data (building block 2). Figure 4 subsequently presents a possible function of this scenario prediction tool.E.g., increasing the score of one KPI score could mean that other KPI scores decrease, depending on the KPI interdependencies that are determined in the earlier statistical analyses.Based on the project team members' estimations, they may together determine courses of action on how to improve on certain KPIs.Also, they can visualize what would happen if e.g.budget would decrease suddenly, if team compositions change, or if (internal or external) customers change their attitudes towards the project (compare e.g. the left and right image in Figure 4).Together, the team members may then devise counteractions in order to prevent project quality decreases.By playing with these sliders, based on their predictions on how the project will change in time (e.g.due to staff and resource changes) or what they plan on doing (e.g.acquire more resources or improve communication with the customer), the users can estimate what the effect of their actions could be on entire project performance.Fig. 4. Tool that helps users to explore which KPIs to improve on, with the largest effect on project success.(L) When users enter the tool, all scores are as they were entered during the last project evaluation.Since the tool is based on an evaluation of the project portfolio of one organization, the significance of the elements will differ per company, meaning that each organization has its own version of the SEM-model in Figure 3. (R) When users play with the slider bars, the other values are also affected.E.g. in this case, the societal aspects KPI is lowered (e.g.hypothetically due to the fact that the innovation is apparently less eco-friendly as was initially anticipated).According to the SEM-model (Figure 3), hereby also other KPIs are affected, lowering their scores as well, along with the entire success chance of the project.This shows it is apparently important (in this dummy-model) to safeguard the value of societal aspects.
The different building blocks are combined into a single 'dashboard' (Figure 5).).This dashboard contains three elements, apart from the header with a title and short project summary.These three elements include 'input' area where users can determine what they wish to visualize (top left), a display of performance based on various KPIs (right), and a display that visualizes performance of the project in time (bottom).Through the use of filter settings for the visualizations, the users can select which information they wish to visualize in the dashboard.They can filter per project on date, on individual KPIs, and on different user inputs of the different team members who work on the same project.Using the comparison function, they can compare scores of different projects.Using the 'tool' button, they enter the scenariobuilding tool which was described above.1).The bottom part displays overall project performance in time, an additional feature that allows for project quality monitoring.The top left part displays selectable project parameter display filters.From top to bottom, these are: project display selection, project evaluation moment, user selection, and specific KPI scores selection.Also, this part has buttons.One checkbox launches a dashboard in which projects can be compared to one another (Figure 2).The other launches a tool that estimates performance change when changing one KPI (see Figure 4).

Tool requirements
As stated above, we had four basic requirements in making this tool.The project performance in time should be monitored, it should explicitly include elements of Responsible Innovation, it should be useable by outsiders as well as insiders, and it should be functional in terms of suggesting possible improvement points.The bottom part of the dashboard display (Figure 5) visualizes project performance in time, so users have an immediate idea of how their current score compares to previous scores.The aspect of proposing concrete improvement points is addressed in two ways.First, low KPI scores, or at least lower than the benchmark of less successful projects, indicates that there are possible improvement points on that KPI.The individual KPIs are clickable, and when clicked the different elements' scores that make up a KPI appear.By hovering on these elements, the different element descriptions appear.When project team members discuss why these values may be low, they together explore possibilities for improvement.Second, potential scenarios can be developed that can help innovators predict what happens to their project based on anticipated changes.These scenarios can then be the starting point of discussions aimed at preventing decrease in quality, or even help teams come up with concrete improvement points.

Prospects
Now that the tool has been developed, based on continuous insights and reflections with potential users, we plan on further implementing it in multiple professional innovation environments.In an earlier preliminary study, without this tool but with visualizations of performance, we tested the use of KPIs in the form of a project scoring benchmark as a means to start discussion with researchers on what they could improve in their currently running projects.In this study (Flipse et al., 2014b) we asked researchers to evaluate their current projects on the same characteristics as those used to identify KPIs in the previous study, in collaboration with a critical outsider (the embedded humanist).We then compared their running projects' scores to the database of finished projects and visualized their projects' performance in comparison to the benchmark KPI scores.The results of that study show that researchers appreciate discussions based on such visualizations, since it makes 'soft' project characteristics such as communication and customer relations 'harder' through the use of visualized performance graphs.
In contrast, those researchers who were not involved in communication with a critical outsider, scored their projects significantly lower in performance after 12 weeks -i.e.without interaction with a critical outsider.We therefore concluded that interactions help identify potential project pitfalls (both on technical content level and on 'softer' elements regarding communication) sooner than would be the case without such interactions.However, the use of these data still required intensive preparation by the embedded humanist, since no automated visualization tool had been developed just yet.An interactive decision support tool could further speed up this process, allowing researchers to see even more readily what they can do to improve their work, and allowing external team members to more readily assess what they can contribute to the project.
In future research, we plan on testing the tool's functionality in terms of userfriendliness, but also in terms of stimulating responsible research and innovation decisions and actions.This means that an implementation testing phase would be accompanied by a qualitative assessment of its use, probably through the use of an embedded humanist who will be interacting with innovators while acting as a critical outsider.We could also test the tool's functionality with outsiders without any explicit affinity with the project, such as randomly selected consumers, members of the public or, also interesting, public policy makers or RRI advocates.Through the installment of interactive collaborative innovation spaces, where tools such as ours may be used, we hope to further the tool as well as RRI practices and their outcomes.Additionally, we hope to also encourage others to use our methods and critically reflect on our proposed ideas, in order to be able to harness its full innovative potential.

Fig. 1 .
Fig. 1. (L) Scores of a dummy-project on 8 possible KPIs, as described earlier.The weighted average is depicted on the right of the figure.(R)When zooming in on one specific KPI, the dashboard displays the scores of various elements out of which the KPI is constructed.E.g. the KPI 'Societal Aspects' consists of 3 elements in this dummy-model.This way, the user gets information on which KPI-specific elements of this KPI are good, and which aspects can be improve on.

Fig. 2 .
Fig. 2. Building block for comparison of projects, comparison of performance in time, or comparison of different team-members' input in a project at a certain point in time.In this example, the combined score of all users in week 6 of the project is compared to a random other project score.

Fig. 3 .
Fig. 3. Result of Structural Equation Modeling, depicting the relation between the KPIs and their relation to innovation project success.

Fig. 5 .
Fig.5.Basic project overview dashboard with three distinct elements.The right part displays performance on various KPIs (green lines) and the overall project (blue line) in relation to successful and less successful previous projects (also see Figure1).The bottom part displays overall project performance in time, an additional feature that allows for project quality monitoring.The top left part displays selectable project parameter display filters.From top to bottom, these are: project display selection, project evaluation moment, user selection, and specific KPI scores selection.Also, this part has buttons.One checkbox launches a dashboard in which projects can be compared to one another (Figure2).The other launches a tool that estimates performance change when changing one KPI (see Figure4).
The different 'entry dates' are clickable, so the users can click on an earlier date in But more importantly, this means that elements of responsible innovation, such as issues relating to environmental health or societal relevance, are continuously 'on the radar' of innovators through the tool, and not just during moments of explicit thinking about such elements during e.g.dedicated sessions.Such sustained attention to these elements should allow innovators to be more responsive to societal aspects, which is a staring point for RRI.In addition, if no significant improvements are monitored, this could lead to quicker decision-making on project go / no-go decisions, potentially saving resources for the organization.This way the tool provides innovators with a little more grip on uncertainty when it comes to their decision making in the dynamic and complex environment of innovation practice.The elements of responsible innovation that are important to consider, can be part of the KPI composition.Additionally, if desired, even elements can be explicitly included that are outside of the statistical analysis.E.g., if an extra 'KPI' is proposed, it can be included in the model separately, if this would further support responsible innovation design thinking.The usability by both insiders and outsiders is visualized trough the appearance of KPIs that the insiders and outsiders can relate to.E.g., technical content experts may relate more to the technical and skill related elements, but customer insights might be blind spots to them.In contrast, social elements might be more operationalizable by critical outsiders such as critical outsiders, who might initially have less knowledge of and experience with the technical content.Through communication, based on the tool scorings, these different stakeholders can interact more functionally, both not forgetting the final aim of the project, i.e. increasing project performance.