Decision-making system for recommending and evaluating competences networks based on interaction data

Abstract

Companies face increased product complexity that involves reviewing and optimizing product development business processes.This leads to an increasingly multidisciplinary approach. Research units and multi-located organizations need also to set up multidisciplinary projects. Finding the right skills when building teams is then very crucial.A data analysis-based system that identifies and recommends skilled people on user request can help companies meet this challenge. In this paper,we present a decision-making tool, based on activity traces analysis, that helps its users in effectively spotting the right skilled people when needed, as well as providing indicators to assess recommendations in terms of accuracy or relevance.

Introduction

Traditionally, the collaboration between different departments of a company throughout the product development cycle is already an issue. Companies nowadays are facing a significant growth in product complexity that involves an increasingly multidisciplinary approach (Bardoscia et al. 2017). Let’s take Mechatronics as an example. A mechatronic product is actually a combination of standard mechanics, automation (instrumentation, actuators), and embedded computing. This growing-complexity requires therefore a greater collaboration level because, in some cases, several competences are needed for a single development process. This trend is observable not only for products, but also for production processes of these products. This evolution is part of the concepts called Industry 4.0 in Germany or the Industry of the Future in France.

Today’s organizations are constantly deploying new tools, such as Product Life-cycle Management (PLM) or Customer Relationship Management (CRM) applications in order to carry out their activity. The PLM concept consists of an information system that processes data collected from several sources (including Computer-Aided Design, Computer-Aided Manufacturing, numerical simulation or Product Data Management). Therefore, a successful PLM (or CRM) tool strongly depends on the relevance of the collected data. Collecting and analyzing this data is a difficult task. Some companies can also search for competences from external partners, which brings about the concept of extended organizations or networked organizations. In order to adapt to the needs of projects and maintain good competitiveness, such organizations are in a continuous need of reconfiguration. Responding to a project call, for example, requires the identification, within a limited time, of potential partners from a large network. These partners can be selected for their common experience, competences, or acquaintances.

These evolving characteristics of today’s organizations induce a rapid expansion of digital communication between persons. It results in a large use of digital systems on the Internet, that generates an increasing dynamism among organizations. Networks arising from bringing together many individuals, who participate in a collaborative professional activity through digital systems, can help in building recommendation systems. Data stemming from users’ interactions within digital systems during these collaborative activities, as well as mutual interactions between these users, is called interaction data. Recommendation systems can also rely on other types of data, such as transcribed data (videos) or manually entered information.

This evolution requires big changes in the organization. The global objectives for the Human Resource Management officer is to reconfigure dynamically human resources affectation and identify the needs for evolution of human resources’ competences. This study works towards evaluating recommending tools based on traces left by users in collaborative softwares. This paper describes the proposed concepts and tools that provide competence identification of qualified people on whom organizations could rely on to develop new products or lead/participate in projects. This work also aspires to assess the contribution of competences recommendation for a company (time-saving, better human resources management). We mainly based these tools on interaction data analysis in order to provide the competence identification and recommendations, as well as the evaluation process.

In this work, we are mainly interested the following scientific aspects expressed through these two research questions:

How to design an automated system, based on interaction data, for competences recommendation and evaluation? The automated task here relates to collecting and processing data. This task is out of scope of this work. Our study is based on the prior existence of both competences repository and user information database. The proposed system is rather a decision-making tool that assists the user by helping them to cope with either the scarcity or the overload of information. We aimed to conceive and develop tools in order to provide competences recommendations, and to assess their relevance by means of interaction data such as user feedback. We did also focused on the representation format of the recommended competences, in order to provide the user with intuitive visualization of the computed recommendations.

How can a company benefit from this decision-making tool? We propose an evaluation model which can help in measuring the added value of an interaction data-based competences recommendation system for a company. This added value can be measured in terms of number of cooperations, and time saved, and be qualitatively assessed through a better overview of both internal and external company networks. In perspective, this evaluation can help Human Resource departments in revising its competences management strategies, which would contribute to a better projects management.

In the upcoming section, we discuss related works in the context of our approach. In the following section, we present an overview of our designed platform consisting of an interaction data-based decision-making tool for competences recommendation and evaluation. The designed model and the corresponding experimental platform are also described. The next section details the experimental context and how the used dataset was collected. Results are reported and discussed. Later, we introduce the evaluation process through a brief literature review, while outlining the evaluation requirements. The last section presents conclusions and future work.

Related works

Definition of competence and related models

There are various definitions of “competence” in literature. In (Belkadi 2006), a detailed summary of those definitions is provided. For our research needs, we choose the definition stated by Le Boterf, which defines a competent person as: “a person who knows how to act appropriately in a particular context, by choosing and mobilizing a double equipment: personal resources (knowledge, qualities, culture, emotions...) and network resources (databases, documents, expertise networks,...) ”(Le Boterf 2000).

Bonjour et al. (2002) examined the competence concept and proposed a cognitive model for defining competence. This model aims to simplify its representation and to clarify the link between knowledge and competence. Later, (Belkadi et al. 2007) proposed a method based on fuzzy logic for competence characterization related to a work situation. It consists of analyzing the characteristics of a given situation, after tracing the corresponding activity. This method can not be used in our research case, because we do not aim to model activities in order to characterize competences.

Several competence models had been proposed in the literature (Paquette 2004; Belkadi 2006; Hlaoittinun et al. 2009; Mota 2009; Coulet 2011). There is also the European e-Competence Framework (e-CF)¹ which provides a model of 40 competences related to the Information and Communication Technologies (ICT) field. E-CF framework is a common standard throughout Europe. In 2016, it became a European norm and was officially published as EN 16234-1. This framework is although specific to ICT field and does not apply to other fields. There are also business competence models such as the Compmetrica Competence Model². Furthermore, many organizations have their own competence model, such as CERN³. Other examples of competence models could be found in the APEC⁴ and Pôle Emploi⁵ “job descriptions”.

Interaction data and social networks

In order to automatically identify user competences, one can process the data produced during working sessions such as collaborative activities or human-machine interactions. A large and growing community of researchers, mainly in Human Sciences field, has investigated data analysis including interaction data. Implementing routines to produce such data is usually a costly time-consuming task. Platforms generating interaction data sets with their context is therefore a considerable asset. For sharing traces corpus contextualized in specific formats, platforms have been developed so far, such as PSLC Datashop (Koedinger et al. 2010) or MULCE (Chanier et al. 2010). The BEATCORP platform (Courtin and Tomasena 2016) proposes tools for sharing interaction traces’ corpus of any format, and also provides analysis tools for those corpora. The platform is based on the PROXYMA (PROXY for Multiple Analyses) approach (Chebil et al. 2015). In perspective, the designed approach aspires to provide, via Web data analysis, the recommendation of resources, such as documents or skills, and the appraisal of such recommendations through evaluation methods.

Several studies have tackled the issue of using interaction data for social network analysis. In (Wu et al. 2008), a research was made on face-to-face communication network. The authors used sociometrics to measure offline interactions at the workplace in order to assess work performance and therefore improve group structuring. Aral et al. (2007) presented a research study based on email data analysis aiming to understand how social networks impact information worker productivity.

Competence recommendation system

Recommendation systems help users getting straight to the essential and benefit from “suggestions (...) which they would not have spontaneously paid attention to” (Béchet 2011). The search time can significantly be reduced by such systems. Nonetheless, these systems can achieve such a performance level only when the computed recommendations are relevant and judged useful by the user. Consequently, downstream evaluation processes must be developed. Most competence recommendation systems rely primarily on Web data. For example, the Aardvark project (Horowitz and Kamvar 2010) (now abandoned) implemented a platform for posting questions to people having the appropriate profile to respond. Those people need to belong to the same on-line social network as the question asker. The DemonD project (Delalonde and Soulier 2007) used a method similar to Aardvark project in extracting profiles, and relied on the traces of discussions within the system in order to recommend various resources: documents, discussions, people or articles. The SmallBlue project (Lin et al. 2012) extracted employees’ profiles from their personal data such as e-mails. The recommendations were proposed as a graphical representation of the user’s social connectivity with the recommended persons.

So far, among the competence recommendation systems listed above, the ones offering a graphical representation of recommended competences did not consider the role of spatial distribution of the graph nodes for providing more meaningful recommendations (the distance between the nodes, node dispersion, graph expansion, communities detection). There are spatial layout algorithms which can allow a trimmer distribution of network nodes, which would make recommendation networks more significant for the user as he/she can visualize and interpret them better. For example, the Gephi graph editor and analysis tool provides a list of these algorithms. It⁶ is a free open source interactive software for networks visualization based on graphical elements such as nodes, edges or hubs. This exploratory data analysis software is used in visual analytic research field. Gephi expects CSV files as input, where data represent network’s matrix structure (list of edges with their weight). It would therefore be propitious to use spatial layout algorithms along with recommendation algorithms and to study the incidence of this combination on the quality of the competence recommendations provided to the user.

The reviewed systems above share a common feature: endogenous data analysis. Otherwise speaking, systems taking into account relationships between users rely on their intra-system participation (except for e-mail). But if we consider that users are already active on digital systems to communicate, coordinate and produce contents (on e-mailer, Enterprise Resource Planning tools, Social Networks), we can barely assume that they would be inclined to join a new on-line social network in order to benefit from recommendations. It would then be more suitable to observe their activity in already-used systems, and offer them the most immersive recommendation service. The analysis of interaction data produced within digital systems can automatically enrich users’ profiles with operational competencies.

Evaluation methods

Summarized in (Shani and Gunawardana 2011), there are three different strategies for recommendation systems’ evaluation. The first, called off-line, is based on pre-collected data relating to behaviors of a group of users such as ranking or choosing items. Such data is used to predict their future behaviors. Called on-line, the second strategy is applied to a randomly-selected users during a real-time interaction with the system, observing and comparing their behavior to other users. User study is the third strategy, which involves some voluntary users answering qualitative or quantitative questions when using the recommendation system.

Off-line and user study are the most suitable strategies fitting to our experimental context. Off-line experiment would enable, as a first step, a number of algorithms to be executed and compared, so the most accurate algorithm can be selected thereafter. In order to collect quantitative measurements (the time taken to perform the task, recommendations’ relevance), we can observe and record users’ behavior. For this purpose, some voluntary users would perform various tasks while interacting with the system. By means of a questionnaire, qualitative data such as the ease of use of the user interface, can also be collected.

There are also metrics for recommendation system assessment in addition to strategies, such as the accuracy metric which refers to the quality of recommendations, the recall metric which calculates the rate of relevant recommended items, or Mean Absolute Error (MAE) which computes the deviation between predicted recommendations and actual users’ choices, as well as some other metrics proposed and discussed in the literature such as coverage, learning rate, novelty and serendipity or mean similarity (Miller et al. 2004; Ziegler 2005). In (Said et al. 2012), a 3D model for evaluating a recommendation system is proposed. It assesses system quality according to three axes: user requirements, technical constraints and expected business model’s objective. Said and Bellogín (2014) present, in a recent study, Rival which is a free java tool for evaluating recommendation systems that ensures comparison and reproducibility of results in any experimental context.

Our evaluation model includes some of the above metrics such as precision, recall, Mean Absolute Error, and coverage. We have completed the model with other measures presented and explained in the last section.

Proposed methods

Most of competence recommendation systems mainly rely on text analysis (summary documents, databases), web browsing history analysis, or e-mail. Other types of activity traces can also be considered to identify experts within an organization such as activity logs (a tool/system use frequency indicator), or user interactions with professional social networks. These types of traces would allow user profile to be dynamically updated and associated competences identification to be improved.

A survey was carried out among a list of companies to identify the tools usually used by DHR or managers to find competent people for leading projects or carrying out collaborative activities. The survey also aimed to find the gaps in the used tools and determine the needs of recruiters/ managers. The results of this survey were considered while designing the proposed decision-making tool.

In this research, we describe and deploy concepts of the interaction data-based decision-making tool for competence recommendation and evaluation. The followed methodology is described and the designed model is presented. A case study was carried out to illustrate the process of identifying the appropriate competence profiles corresponding to user needs, based on interaction data analysis. Processing the big amount of available data is a major challenge, but the lack of relevant data can also complicate the competence identification process.

The competence model used in this work is a simplified model, which relates to the listed models in related works. Competences, in this model, are classified by type (hard skills, soft skills) and by level (3 levels ranging from “not competent” to “very competent”). The competences repository is a database which differs from one company to another. This database is an input to our decision-making tool proposed for the automatic competence identification and their recommendation.

When designing the platform for competences recommendation and evaluation, we chose a graphical representation for system recommendations. In fact, recommended competences are displayed as networks, which are “human-centered”. All the nodes represent people, and the connections represent the characterized relationships (trust degree, collaboration degree), with a focus on one node called “ego”. As a perspective, other representations could be considered depending on the intended utilization. For example, a “product-centric” representation could incorporate other elements into the graph which is centered on one or more products, such as the corresponding manufacturing processes, that are associated with competences profiles, themselves linked to people.

Interaction data-based decision-making tool for competence recommendation and evaluation

General description

In order to set up the decision-making tool for competence recommendation and evaluation, we based our concept on interaction data analysis for both competence identification and system evaluation processes. A considerable amount of useful data within existing digital systems such as collaborative platforms, professional social networks, or mailers can be created. This data have to be collected and analyzed in order to extract appropriate information and provide true interpretations⁷. This can represent a major challenge.

The analysis of the interaction data collected has three objectives. The first goal is to update users’ profiles with new competences or endorse the existing ones. For example, analyzing user log files can help identifying the user’s ability in achieving some tasks. It will therefore update the competences repository used by the system. The second objective is to identify social links between users in order to generate community networks. For example, people interacting a lot with each other are likely to share a strong relationship. The third objective is to evaluate the use of the proposed tool, through user interactions, regarding its interest and relevance.

The proposed tool is user-centered as shown in Fig. 1, and has mainly two goals: providing the user with competence profile recommendations on specific request, and assessing the relevance of those recommendations. The usefulness or the added-value of such system is appraised through an evaluation process. The user-profile database consists of a set of information about each user such as user name, contact information, and list of skills.

Fig. 1

Decision-making system overview. The system analyzes Web interaction data and user feedbacks to provide the user with profile’s recommendations. 1: data 2: process 3: database 4: data flow

Figure 2 shows the main use case: a user with a need for a given profile sends a request to the decision-making system. The system extracts potential candidates via the user profile database. Former users’ interactions and requesters’ feedbacks are used to build this database.

Fig. 2

Main use case

Our system is centered on a user-profile database. We retrieve information from the Web in order to feed and update this database. In order to estimate the use frequency of social networks among companies and identify the most used networks to retrieve information from them, we carried out field investigations⁸ for this purpose. According to the results, the use rate of social networks is around 83% and LinkedIn is the most popular one.

We classified extracted competences through web scraping process into two categories: user-declared and system-computed. User-declared category refers to competences provided by a given network member on the Web such as a skills list in his/her social profile pages or personal Web pages. Competences deduced from Web data analysis belong to system-computed category, such as experts’ identification in Web forums. To ensure more truthful profiles, deduced competences can be certified by each network member as a next step.

User interactions with the decision-making tool can be either requests or feedbacks. Requests are processed by the system in order to provide recommendations, which are displayed as a network of recommended profiles. Afterwards, for evaluation purpose, the system analyzes feedbacks on these recommendations.

Model and experimental platform

We have designed a platform which consists of different components (Fig. 3): a human-machine interface for the communication between users and the system, a web scraping tool, a competence repository and the graphical analysis tool Gephi.

Fig. 3

System architecture & experimental platform with two main user interfaces: Gephi for network visualization, and a developed HCI for I/O data processing. 1: user interface 2: process 3: database 4: data flow 5: text file 6: python script file

Human-computer interface

By the means of an input/output window, the user interacts with the system through either a request or a feedback. The recommendation process, implemented in python, has a read-only access to the users’ profiles database (SQL DB server) and produces two output text files. The first one, named output filter file, is used to filter the community network in order to let the user visualize system recommendations instead of the whole network. The second file named output nodes file, enables the spatial apportionment of the recommended network nodes based on their social connectivity with the requester. A current study is focusing on developing the evaluation process, along with the recommendation process. This study would focus on user feedbacks analysis and metrics measurement by means of a set of tools. The user should visualize interaction data-based indicators obtained from this process via the same input/output interface.

Web scraping of social networks

Further information retrieved from the web such as social networks can help in updating the users’ profiles database. LinkedIn have been targeted for this study. A python parser was developed to this purpose, in order to get users’ LinkedIn profiles and collect useful information like declared skills or achieved projects. Figure 4 shows an example of the list of data obtained from scraping a random user’s LinkedIn profile.

Fig. 4

An example of an output generated with the developed LinkedIn python parser

The scraping of LinkedIn has been designed to be punctual (low access frequency). The developed parser accesses only public profiles. The information contained in this type of profiles -and retrieved by our system- is therefore publicly available. It has been made available by the user himself, who has chosen to make his profile and personal data visible in the Web. Considering this fact, we therefore did not consider necessary applying for legal authorization to access such public data.

Competences repository

The proposed system was designed to be generic. To generate the associated social network with skills in the studied field, the system should be able to process any data set describing users and field-related competences. For a first use of the system, such a data set should be stored in a database beforehand.

Gephi analysis tool

For a better spatial apportionment of the network nodes, we applied spatial layout algorithms. In order to offer to the user a more harmonious display in a Flat Design mode, but also the ability of interacting with the system and looking over the network, the graphical visualization of computed recommendations delivered by the system was therefore exported in html version. The two files output filter and output nodes are used as input data for Gephi. They are processed respectively by filter tools and by a Jython console.

Experimental setup

Dataset and experimental context

We have suggested using interaction data analysis to identify competences and to combine it with social network data to generate profile’s recommendations. We carried out two different experiments aiming to illustrate and test this proposal with real data. In (Yahiaoui et al. 2018), we presented and described an experiment implemented with students of the Internet and multimedia department at the university. In this paper, we describe and discuss the results obtained with an experiment carried out at the SYMME⁹ laboratory.

The main idea was to automatically collect all the scientific publications of each member of the laboratory from Google Scholar. The aim of this experiment was to analyze publications in order to characterize the relationships between researchers and to create the laboratory’s network. This social network was built with collaboration degrees between members. The collaboration degree is the number of common publications between two members. An algorithm has been developed to automatically calculate this degree from the publications database. An extract of this database is shown in Fig. 5.

Fig. 5

An extract of the publications database: the listed publications were automatically retrieved from Google Scholar

Figure 6 exposes a graphical visualization of the studied social network in Gephi analysis tool.

Fig. 6

Full Community Network generated with Gephi

Results and discussion

In this paper, we present the first recommendations obtained with our suggested system. We used some of the available spatial layout algorithms in Gephi (node dispersion, graph expansion, communities detection) to detect collaboration groups among the whole network. We used Force Atlas 2 for community detection, which is a force-directed algorithm implemented in Gephi, and developed by combining existing techniques (Barnes Hut simulation, degree-dependent repulsive force)(Jacomy et al. 2011). It’s a user-friendly algorithm that corresponded to our computation needs. The initial graph was then reorganized due to obtained results. The new graph allows the user to easily identify people by groups. Figure 7 shows the obtained recommendation graph, visualized in HTML mode. Each collaboration group was highlighted by a color. The distance between two nodes in a group is proportional to the interaction (collaboration) degree between those persons. People who collaborate more are closer to each other.

Fig. 7

Community Network processed with spatial layout algorithms (HTML visualization)

These first tests let us highlight the link between collaboration degree and people’s competences. Researchers publishing a lot together are likely sharing a number of complementary competences, and are contributing to the same applied field. In this academic context, a high collaboration degree between two or more researchers can therefore help in computing competences recommendations. The above assumption may vary from field to field and further analysis is required in order to tell if collaboration groups can also be considered as competence groups, i.e. group of people having common competences.

Using spatial layout algorithms along with recommendation algorithms can be advantageous in making easier interpretation of the recommendation graphs. In fact, the spatial layout enhances the recommendations computed by the system by ensuring a better network visualization. The user can adjust the importance of the search criteria, such as preferring soft skills over hard skills and vice versa, or social proximity over skills, resulting in new spatial distribution of the recommendation graphs.

This experiment demonstrated that the analysis of interaction data can play a role in the automatic identification of interpersonal relationships (collaborations) between network members, which can help in initiating competence recommendations. The challenge is to have the appropriate tools for collecting and analyzing these data in order to extract the most accurate information. Therefore, we need to set an evaluation model to assess the use of interaction data and the authenticity of the extracted information (competences, performance indicators). Moreover, this experiment provided a graphic visualization that can be used to analyze how the laboratory’s internal policy may impact its members’ interactions. In fact, there are several competence fields in the laboratory (mechanics, electronics, optics, imaging, computer science, and others) whose general theme is mechatronics. The main goal consists in having all these teams work together to innovate on mechatronic projects. The graph obtained in this experiment shows that there are more collaborations within the same competence field compared to collaborations between different fields.

Evaluation

Users had to assess, during the recommendation process, in terms of relevance, the recommended profiles by means of a rating grid. These feedbacks are used as indicators for the evaluation of the recommendation system. We present quickly in this section the known evaluation strategies and the known metrics of recommendation systems. Along with the developed requirements model, we present next the selected strategy and explains how it best fits our system evaluation requirements.

Evaluation model

We propose a two-phase evaluation model for the proposed competence recommendation system. Figure 8 shows the proposed evaluation model, developed in SysML. Each evaluation phase has its own criteria with the corresponding metrics or indicators.

Fig. 8

Evaluation model

Phase 1: short-term evaluation

also called “Operational evaluation”: assessing the efficiency and performance of competence recommendation through a number of appropriate indicators.

Authenticity of Web extracted competences

This criterion is measured by the truthfulness rate, which is proportional to the appearance frequency of a competence in the retrieved Web data. This metric can be adjusted by network members feedback (manual competence validation by members).

Relevance of system recommendations

We propose two indicators to measure this criterion: user selection and the type of algorithm used in recommendation. User selection is defined by the Mean Absolute Error (MAE) which computes the gap between predicted recommendations and actual users’ choices. The type of recommendation algorithm can be valued by comparing the recommendation results of different algorithms: collaborative filtering or hybrid filtering.

System performance

There are four metrics to calculate system performance:

• Precision: probability of relevant profiles being recommended = number of relevant recommendations/total number of recommendations

• Recall: probability of recommended profiles being relevant = number of relevant recommendations proposed by the system/total number of possible relevant recommendations

• Coverage: use rate of available resources by the recommendation system

• User satisfaction: a group of volunteers answer a questionnaire about system use (quality of recommendations, display mode, ease of use)

Phase 2: long-term evaluation

also called “Impact evaluation”: allowing to appraise the competence recommendation’s contribution for a company by estimating the system use impact and its potential financial effect.

Cooperation Rate

This criterion aims to estimate the cooperation between company members (new collaborations after using the system).

Turnover

The rate of replacement of staff in a company, measured over one calendar year.

Bore-out

A rate indicating boredom at work that could lead to resignations (low bore-out = no quits).

Conclusions and future work

This paper discusses interaction data’s ability for competence identification as well as for competences recommendation and the evaluation process of these recommendations. The proposed decision-making tool based on interaction-data analysis should provide the necessary tools to efficiently spot the right skilled people from a large network, and assess the recommendation process both quantitatively and qualitatively. A data set of scientific publications from researchers at the SYMME laboratory enabled experiments to be carried out. The results comforts the ability of an interaction-data based approach in the identification and recommendation of competences. A special attention was paid to the representation format of recommended competences. The user visualizes through a graphical interface the competence recommendation in the form of networks. This type of display can enhance user interactivity and ease decision-making.

More apportionment options, such as hard skills levels option which would allow the best skilled users to be displayed close to the applicant, can be chosen by the requester in future experimentation. Consequently, different recommendation scripts can be compared during the evaluation process.

In order to develop the evaluation process and measurement techniques, further research is necessary. It would conclude to the contribution of the designed platform in optimizing the identification of competences in any activity area. Other possible contributions of the decision-making tool can also be investigated in perspective of this work, through the evaluation process, such as reinforcing the management reactivity within the organization, or helping organizations (especially Human Resource Department) to implement competences management strategies resulting in a more optimal distribution of tasks.