Description of work

T1.1 – Stakeholder analysis and educational scenarios (EA) [M1-M14]

Define the stakeholders involved in a learning situation involving virtual, online learning environments such as virtual labs, focusing on the teachers and learners but also considering other relevant groups in the design/development and learning process. Analyze the requirements of both groups of stakeholders in terms of a) behavioral analytics and b) online authoring environments. The stakeholder analysis will be based on interviews, direct observation and reviews of best practices with the plethora of such learning environments that EA is already employing and their participation in international projects on virtual learning spaces. Based on the stakeholder analysis provide the educational scenarios that will be implemented as the prototype demonstrators. These scenarios will feed into WP5 by specifying an equal number of virtual labs to be designed and developed using the authoring environment (WP4) and will provide the test bed for evaluating the effectiveness of the developed technologies in WP2-4 to address the stakeholder requirements.

T1.2 – Data structure requirements for learning analytics (EA, GIO) [M1-M14]

Based on the learning environments in use at EA (notably the Open Discovery Space, Inspiring Science Education and Go-Lab), as well as current projects internationally, the current behaviors and metrics that are captured through telemetry will be investigated and defined to provide an overview of the kinds of behaviors that can be captured through user behavior (learners and teachers). Behaviors that can be captured across online virtual labs, as well as behaviors that are more specific to certain kinds of labs, will be considered. In addition, based on experiences from game analytics, a structure for event-based tracking, where the individual events can be completely customized, will be defined. Jointly, these provide the baseline for the learning analytics package (WP2) and specifically for Task 2.1 which focuses on translating the tracking points into the GIO’s back end. Guided by best practices in game analytics, it will be investigated which of the identified behaviors, and potentially additional behaviors, are useful to capture in order to address the four key game analytics technologies to be transferred to learning analytics: Profiling, prediction, A/B testing and retention, thus offering the ability to test the design of online learning situation in a virtual lab.

In parallel to the specification of the data structure requirements, as part of the activities of this task a set of behavioral telemetry data from current virtual labs will be collected, pre-processed and become available to the project so as to allow the immediate initiation of the analytics-related activities in WP2 and WP3.

T1.3 – Functional requirements in virtual learning spaces (scenario authoring environment) (EA, CERTH) [M1-M14]

This task will investigate current authoring tools in games and education to develop the requirements and specifications for the authoring tool itself across functionality, design and interface. The requirements should enable the production of the type of educational scenarios defined in T1.1, and form the initial basis for the development and implementation of the authoring tool in WP4. Moreover, this task will investigate current ways that visualization is used in learning analytics and game analytics to define best practices for how to present insights to the different stakeholders (learning analytics package). The results of this early work will be continued and implemented in T2.3.

Partners’ role:

EA will lead the activities of this work package by providing the stakeholder analysis along with the several educational scenarios. Moreover, EA will be responsible for providing ENVISAGE with behavioral data that have already been logged from existing virtual labs at the very first day of the project.

GIO and CERTH will be responsible to investigate the requirements of virtual labs in terms of data structures and functionalities involved in the process of improving virtual labs.

Description of work

T2.1 – Tracking infrastructure for user data collection and aggregation (GIO) [M3-M21]

This task is dedicated to:

• Compilation of a list of tracking points in virtual labs
• Integration of tracking components on the client side
• Extending the existing infrastructure to handle learning analysis data

Initially, the current practices in behavior modeling of virtual lab users have to be analyzed in order to identify suitable tracking points. These tracking points typically depend on the particular lab, however, best practices can be worked out to speed up the integration process. Once these tracking points have been defined, GIO’s service needs to be integrated into different learning apps or virtual labs so that a data-stream to the backend is enabled. On the backend, GIO‘s existing infrastructure has to be prepared to collect the user data from learning apps. Typically, a raw stream of event data is stored and in a pre-processing step aggregated on a user level. After aggregation this data is typically stored in an appropriate database to allow simple, as well as complex, queries. Depending on the tracking points, the data processing pipeline will be adapted to meet the requirements of learning analytics within ENVISAGE.

T2.2 – Learning analytics from usage data (AAU, GIO) [M4-M16]

The task is dedicated to:

• Access the collected data
• Enrich the data with shallow statistics

Utilizing data points that can be easily collected in virtual lab software, T2.2 will identify methods of deriving learning-based analytics from the data established in T2.1. The processes will mostly follow a quantitative process, deriving descriptive statistics of the learning outcomes and performance. Here, one starts with adapting known metrics from the field of game analytics. For example, session counts or time spent in the virtual lab. Such simple count statistics can also be computed for groups of users, e.g., by taking the mean. Such shallow statistics already give an overview which users are more or less engaged in a learning task. More sophisticated analytics in games also look at heatmaps for example. If it is possible to obtain such data in virtual labs, a spatial analysis of user data in virtual labs may also yield interesting insights which can be used to influence the design of virtual labs in the future.

T2.3 – Data visualization for course progress reports (UoM, GIO) [M4-M16]

The purposes of the shallow metrics collected under T2.1 and analyzed as descriptive statistics under T2.2 are threefold: (a) for the management of an educational facility to gauge the successes and failures of past virtual labs and take strategic steps towards improving the virtual learning environment and its use, (b) for the educator to monitor in-progress courses delivered through virtual labs and identify problems in past virtual lab use through historical data, (c) for the learners to track their own learning progress and compare it with that of the class or their friends. All of these purposes require that data is presented in a concise, easy to understand way which however also prompts reflection and recognition of problematic patterns (e.g. in at-risk learners). Task 2.3 will tackle the issue of presenting the raw data and learning analytics to these different stakeholders (management, educators, learners).

In Task 2.3, current trends in visualizing data for feedback to the development teams (e.g. reports of A/B tests) as well as for feedback to the end-users (e.g. through public websites which aggregate gameplay data and show current trends) will be transferred to the field of education and particularly to the visualization of learning analytics. The task will primarily identify methods used in the game industry which would be most appropriate for the stakeholders of educational facilities, especially educators. By integrating specifications from T1.3 regarding the learning outcomes and educators’ methodologies, the creation of reports which are intuitive, responsive (e.g. for real-time monitoring), and show relevant information will be devised to assist the educator’s task during the course of a virtual lab and for designing new virtual labs. Moreover, visualizations of learners’ progress will be designed, developed and tested as a feedback mechanism and intrinsic motivation for learners. Finally, concise reports (with aggregated statistics and visualizations) will be created for a high-level overview of the state in virtual labs, for the purposes of management of educational facilities.

Partner’s role

GIO leads the activities in this work package and is responsible for collecting and aggregating the data as well as identifying the appropriate learning-biased metrics in order to extent current analytics infrastructure towards the learning domain.

AAU will perform shallow analytics on the obtained data, which will allow for profiling and modeling the behavior of the learners.

UoM will be responsible for the effective visualization of the shallow analytics results.

Description of work

T3.1 – Learners’ future behavior prediction (GIO, UoM, AAU) [M4-M16]

The task is dedicated to:

• Building datasets for the machine learning models
• Learn machine learning models for different purposes and optimize their parameters
• Make predictions accessible

Using appropriate metrics and features from past user behavior, models can be constructed which predict the future learning performance in unseen tasks or the performance of other students on the same task. For example, having observed several students completing an entire virtual lab, one can make prediction if a student will successfully complete a virtual lab early on. In case the algorithms doubts their success, an intelligent learning system should adapt the difficulty level. Additionally, other methods for engagement can be incorporated to increase the motivation of students to finish an exercise. For example, providing hints for solving the solution. The constructed models can be personal, i.e., one model per learner, can be aggregated across clusters of students with similar user behaviors, or can be grouped in terms of general class performance. The learning outcomes and performance metrics from T2.1 can be used as possible values which can be predicted in unseen data.

T3.2 – Computational adaptation of course material (UoM) [M6-M16]

The predictive power of the models developed under T3.1 can be used in the automatic adaptation of the course material. Task 3.2 will transfer the mature knowledge of the game industry regarding procedural content generation (PCG) in order to automatically (or semi-automatically) generate a personalized course structure of the course material in order to accommodate the predicted learning performance of the user. Following the specifications from T1.3 regarding the learning outcomes and educators’ methodologies, the appropriate generative algorithms will be devised in order to (a) adapt the content in order to better assist the learning of at-risk (or over-competent) learners, while (b) still allowing the educator, when needed, to control the type of content presented and its order. The final algorithms are expected to obey educator-specific constraints on the structure of the virtual lab, while having the freedom to provide learners with content which is predicted as most likely to increase their engagement and learning potential. The predictive model will be updated based on the responses of learners to such personalized course material, leading to a real-time adaptive system which adapts itself and the course material to its users. In order to select appropriate content to show to learners, the algorithms developed under T3.2 will observe usage patterns of past learners with a similar usage behavior (as developed under T3.1) and how they engaged with course material; the material which resulted in the best learning outcomes (as specified by the learning analytics developed under T2.2) will be selected for current learners. As current learners progress in the courses, they become past learners and appropriately update the model of appropriate course material. This positive feedback loop allows the system to continuously learn and improve the delivery of the course material to better fit the learning processes of current learners.

Partner’s role

UoM will lead the activities of this WP towards delivering learning content adapted to the specific needs of individual learners by consulting the predictions of machine learning algorithms.

GIO and AAU based on the shallow analytics results from WP2 will develop appropriate machine learning methods in order to predict the future behavior of the students.

Description of work

Τ4.1 – Architecture and interface design (CERTH, UoM) [M3-M5]

Relying on the functional requirements gathered in T1.3, define the architecture of the “Virtual labs authoring tool” that will be capable of integrating the functionalities developed in WP2 and WP3 into a web interface component able to communicate with a game engine remotely in order to produce a lab. As regards the integration with WP2 and WP3, the adopted architecture should make provision for effectively injecting the necessary metrics in the virtual lab project to assess the user behavior during the experience of the virtual lab. The selection of the basic technologies (e.g. the utilized 3D engine such as Unity [1], Unreal [2], Torgue3D [3] or Godot engines [4]) will be made in the context of T4.1. Another responsibility of this task is to analyze the functional requirements provided by WP1 (T1.3) in order to deliver an accurate interface design of the authoring tool suitable for the virtual labs. Mockups will be used to simulate the workflow between the supported functionalities and also act as a running exercise between the interface designers and the author-users in defining the look and feel of the authoring tool.

T4.2 – Implementation of the virtual labs authoring tool with generalizable templates (CERTH, UoM) [M4-M18]

The “Virtual labs authoring tool” that will be implemented will allow teachers to design experiments with an easy to manipulate graphic user interface. The teachers will be able with drag-n-drop functionalities to design the experiment in 3D space and allow the user to interact as an avatar with the virtual items. The purpose of this task is the core of the project, namely, implement an authoring tool in a web browser (web page) that it is able to generate virtual lab projects in a certain game engine format, e.g. Unity3D format. These projects could be compiled for each platform that the game engine (Unity3D) supports, i.e. desktop, mobile, console, and web. The compiling of the project will be done remotely in a server which has the game engine installed. Summarizing, the pipeline of the authoring tool consists of the following components that will be developed independently: a) web interface (the Editor), b) Unity3D game format saving mechanism (the Translator), c) compiling request mechanism (the Uploader), d) compiling server machine setup (the Game-Engine-Server), and e) the transfer of the result to the author (the Downloader). The goal of this task is also to offer the appropriate templates that will be used to design virtual labs for different use cases according to the specification of the scenarios in T1.1. These templates will have the necessary metrics measurement mechanisms embedded that will monitor users’ behavior and will communicate this metrics to the game and visual analytics components as defined in WP2 and WP3.

T4.3 – Technology assembly, integration and validation for the implementation of the authoring tool (CERTH, All) [M8-M18]

The goal of this task is the technical verification of the products of the project with respect to their expected functionality before reaching the prototype level. This task will define a group of internal beta-testers (teachers from EA partner) for the “Virtual labs authoring tool” ensuring that the enabling technology has reached the necessary quality level for supporting the pilots developed in WP5. In order to ensure that the developed solution meets the expected requirements, validation sessions will be put in place. These validation sessions will rely on an Assembly, Integration and Validation (AIV) plan that will be written in parallel with the development tasks. The AIV plan will specify the integration protocol such as fixing the integration interfaces and practices between the individual technology components in the authoring tool. The goal of this protocol is to allow for individual components to work independently on algorithmic refinements and optimizations, while ensuring their smooth integration with the architecture of the authoring tool. Finally, the validation tests will be put in action by the EA partner with teachers in various sciences. In this way, we will verify that ENVISAGE technologies function to the expected performance criteria in an environment close to that of the author-user. This will ensure that the enabling technology has reached the necessary quality level for supporting the prototype demonstrators developed in WP5.

Partner’s role

CERTH will lead this work package by driving the development effort in defining the architecture, designing the interface and implementing the “Virtual labs authoring tool”.

UoM will contribute in defining the architecture and designing the interface of the “Virtual labs authoring tool” given its long established experience in gaming technologies. UoM will also provide necessary components related to game analytics metrics that should be injected inside the templates by the information from UoM, AAU and GIO in WP2 and WP3.

EA will be the internal beta-tester of the “Virtual labs authoring tool” that should be refined internally with respect to its easiness in perception and manipulation. EA is a good candidate for this task because it is a non-technical partner that simulates the final customer.

[1] http://unity3d.com

[2] https://www.unrealengine.com

[3] http://garagegames.com

[4] http://www.gotodengine.org

This work package will run under an agile framework and cover two cycles of iterative pilot execution, evaluation and revision before the final version is cleared for dissemination and exploitation in WP6. Each pilot will integrate the results of WP1-WP4 and will consist of the following steps: a) Train students to use virtual labs and run a number of experiments as these have been identified in WP1; b) Monitor students’ activities while using labs and log behavioral data streams through the analytics back-end installed in WP2; c) Employ the authoring tool (WP4) in order to enhance virtual labs taking into consideration the deep analytics results presented to the teacher through the front-end (WP3); and d) Evaluate conditions 1-3 as outlined in the Objectives, above.

Evaluation will take place at EA with support from the other partners, notably AAU on the analytics component. Particularly, the process of using the authoring tool as a means for building virtual labs will be evaluated by teachers and e-learning experts. The support offered by the analytics tools in the process of improving virtual labs will be evaluated by teachers and game analytics experts. The delivered virtual labs and the learning content will be evaluated by both teachers and students as well as user interface experts. The primary evaluation tools will be observation, user testing and user feedback collected online, combined with behavioral telemetry, which provides detailed behavioral data for all these three evaluation conditions. The evaluation will follow best practices in evaluating learning software as well as digital games. The evaluation protocol for each virtual lab will be defined in multiple levels so as to cover measures of effectiveness, efficiency, usability, functionality as well as user experience and benefit for the educational organizations in terms of the extent to which the developed technologies have achieved to motivate the engagement of students with virtual labs. To maximize the benefit of the evaluation process we plan to adopt (and extend when appropriate) the evaluation and experimental protocols that are currently in place at EA for evaluating their current solutions. In this way, we can leverage their experience of what exactly do we need to measure, how we can measure it and how we can analyze the test results for deriving conclusions about technology improvements. The above will be accomplished by undertaking the following tasks.

T5.1 – Building virtual labs: Small-scale pilots addressing the educational scenarios (EA, All) [M9-M21]

This task will focus on developing a set of virtual labs with the ENVISAGE assets, based on the educational scenarios identified in T1.1. Using the data inserted in the project by T1.2 and the combination of the modules that were implemented in WP2-4, the specific educational scenarios will be executed towards building and delivering a set of virtual labs that aim to meet the goals and needs of the use case partner (EA). Focus will be on producing and evaluating different versions of virtual labs, using an A/B testing approach towards improving the design and functionality of the virtual labs that have been initially built by the authoring tool. For this purpose, at least two pilot iterations are needed to effectively perform the A/B testing. In particular, two pilot iterations will be executed during the lifetime of the project so as to enable the refinement of the developed modules, while more iterations can be executed by the use case partner after the end of the project when the final software package will be delivered, towards further optimizing their virtual labs.

T5.2 System test and evaluation (AAU, All) [M9-M21]

 The objective of this task is to evaluate the ENVISAGE solution by evaluating each of the three conditions outlined above, which are described hereunder as subtasks:

1. Authoring tool evaluation: Following the internal evaluation of the authoring tool in T4.3, this subtask will focus on evaluating the authoring tool as a means to easily and effectively build virtual labs. More specifically, a number of external evaluators (i.e., teachers) will utilize the authoring tool and evaluate several aspects of the tool like for instance its usability, its effectiveness, the friendliness of the interface, the ease of use, etc. The evaluation outcomes will be very important for the second cycle of the project, where an updated and improved version of the authoring tool will be developed and used in the second evaluation phase.
2. Analytics tools evaluation: The focus of this subtask will be to demonstrate the effect of the analytics tools in the process of optimizing the design and the learning process of virtual labs. AAU will take the lead in evaluating the analytical results from the labs/scenarios and liaise with EA to measure the benefit and support provided by the analytical techniques (both shallow and deep) in terms of their applicability in the practical life of a designer and teacher.
3. Virtual labs evaluation: This subtask will focus on evaluating the usefulness and effectiveness of the delivered virtual labs and learning content towards fostering the engagement of the students with the lab and performing retention. The evaluation of the virtual labs will be accomplished by both teachers and students in order to be ensured that the process of improving the virtual labs has achieved to fulfill the expectations of both.

Feedback from the first evaluation phase will be used as the basis for revising the ENVISAGE solution by the relevant partners (GIO, UoM, CERTH, AAU).

Partners’ role:

AAU will work with EA on planning and executing the evaluation of the different components of the ENVISAGE system, forming the basis for the iterative analysis of the behavioral telemetry collected. In order to evaluate the impact of the analytics tools in the ENVISAGE system, close collaboration between AAU and EA will be necessary to cross-check the actionability or impact of these tools. AAU will also take the lead on mapping and recommending changes to the analytical front end, working closely with UoM, GIO and CERTH on ensuring a smooth iterative evaluation process on the analytics support.

EA will bring the perspective of the users (teachers and learners) to the process of evaluating and assessing the authoring tool and the virtual labs that can be constructed using this tool, as well as the underlying analytics support during both of these steps. EA will develop the virtual labs based on the educational scenarios developed in WP1, and form the basis for the planning and execution of the system tests. EA will host the physical evaluation of the tool and labs, and define the evaluation protocol in collaboration with AAU. Fianlly, EA will also measure the impact of the tool in the process of designing, running and improving virtual labs, working with AAU on the analytics aspects of the evaluation.

CERTH will ensure the applicability of the authoring tool during the pilot execution, and will integrate the evaluation results into the authoring tool between and following test iterations.

GIO will iteratively integrate the changes needed on the analytical back end with respect to data collection.

UoM will iteratively integrate the results of evaluation in between and following iterations as they pertain to the analytics front end.

Description of work

T6.1 – Dissemination activities (CERTH, UoM, AAU) [M1-M24]

The goal of this task is to establish the dissemination plan by specifying the dissemination objectives, target groups, directions, instruments and impact indicators. This plan will be initially established in the beginning of the project and will be refined after evaluating the results of the first pilot studies. An initial assessment for the plan has been already done in Sections 2.2.1 and 2.2.2. After defining this information the strategy that we plan to follow is to provide every partner with a template suitable for documenting the important characteristics of the intended dissemination activity, such as the utilized instrument, the date and place, its target audience, its purpose, its measurable impact indicator and its link to the project’s exploitable assets (see Section 2.2.1). The filled-in templates will be subsequently summarized into tables, composing a list of concrete dissemination actions that will allow us to assess, follow up and eventually correct the planned actions in terms of the reached measured impacts.

T6.2 – Exploitation activities (EA, GIO) [M1-M24]

The goal of this task is to establish the project’s exploitation plan by specifying the project’s exploitable assets, the strengths-weaknesses-opportunities-threats for these assets, the market dynamics, the restrictions imposed by the existing open source licences and the applicable business models. A preliminary version of this plan is established in Section 2.2.3, while Section 2.2.4 discusses in detail our strategy for knowledge management and intellectual property. The exploitation plan will act as the basis for the joint and individual exploitation activities of all ENVISAGE partners. Particular attention will be placed on taking into account the IPR document developed in T7.3, as well as on establishing clear links with the activities foreseen in the dissemination plan.

T6.3 – Fostering innovation (GIO) [M13-M24]

The goal of this task will be to establish some processes outside the strict technical work plan of the project that will encourage all partners to think more as entrepreneurs than researchers or retailers. In order to stimulate this environment, all partners will be encouraged to hold small, task related meetings, where they can brainstorm ideas collectively, and to have demo sessions at consortium meetings so that new results can be rapidly shared.

Partners’ role: All partners of the project are involved in dissemination and exploitation activities. Activities concerning the use case partner EA and the project SME GIO include the direct interactions with virtual lab designers, developers, etc, and the direct exploitation of ENVISAGE assets (i.e., the back-end data log infrastructure, the front-end analytics widgets, the authoring tool, etc). The focus of research (CERTH) and academic (UoM, AAU) partners will be mostly on academic dissemination through research publications and on exploitation via their spin-off companies.

Description of work

T7.1 – Administrative and financial management (CERTH) [M1-M24]

With respect to administrative management the responsibility of this task is to establish a stable management basis and ensure a firm and accepted foundation to the project prior to commencement of the work. This task includes building upon the terms and conditions specified in the Consortium Agreement, define and set up the practical mechanisms which implement the management functions. These include communication between the Steering Board and ENVISAGE partners, EC officials and external organizations and individuals. Moreover, the administrative management entails also the organization of the consortium meetings, meetings with the advisory board and technical meetings. A set of documents will be generated as described in Section 3.2 supporting management activities (Management Charter, Management Handbook, Communication Plan and Quality Plan). The financial management entails the establishment of a plan ensuring that the expected income, expenditure, and disbursements are clear and verifiable. It also includes reviewing the financial status monthly, report on progress to the Steering Board at quarterly meetings, verifying that audits are carried out for all partners and that audit certificates are received.

T7.2 – Technical and quality management (CERTH, All) [M1-M24]

The technical management is a repeatable process that aims at monitoring and establishing the specific form and content of each deliverable, resolving what activities are necessary for their creation and delivery and at determining specific dependencies between deliverables. The technical management includes also the monitoring and control activities of the Project Management Board (PMB) and the Project Technical Manager (PTM) in ensuring that a WP stays on course and reacts to unexpected events. The process forms the core of the PTM’s effort on the project since it is the process that handles day-to-day project issues. Throughout a WP there will be a cycle consisting of authorizing work to be done, gathering progress information about that work, watching for changes, reviewing the situation, reporting and taking any necessary corrective action. The role of technical management is to cover these activities, together with the on-going work of risk analysis and contingency planning, change control and bi-monthly reporting to the PMB. To efficiently handle the large volume of project documents (including reports), all project members will use a specialized web-based Document Management Tool supported by the Project Technical Manager and Project Assistant.

The quality management of the project will be implemented through a Quality Assurance Plan (QAP) that will be established with the objective of ensuring high quality outputs of the project activities, thorough review of the project deliverables and regular assessment of the project progress and achievements. The QAP will specify all quality control procedures, including responsibilities of project participants (Project Technical Manager, WP Leaders, Task Leaders and other partners), contract management, documentation control, documentation formats and exchange rules, deliverable assessment, etc. The QAP will ensure that work conforms to the requirements identified in each WP, that the work is done and that completed modules/products/deliverables meet the quality standards set. The QAP will request internal progress reports to be provided by each partner on a six-month basis summarizing progress, achievements and usage of resources. This task also includes strategic monitoring and possible re-planning of ENVISAGE activities to ensure all project objectives are achieved on time.

Finally, this task will also undertake the activities related to the management of data that will be generated during the pilots (i.e. data logged during the engagement of students with the virtual labs). More specifically, we will specify the strategy adopted in ENVISAGE for making our research data open to the public and maximize accessibility and reuse. Our preliminary thoughts are already described as part of our exploitation plan (cf. Section 2.2.2)

T7.3 – Management of knowledge and Intellectual Property Rights (CERTH) [M1-M24]

This task deals with the detection and assessment of the knowledge generated by the consortium. This knowledge will be initially identified at the beginning of the project and will be constantly monitored throughout the project’s lifetime based on an IRP plan. Since it is possible that key IPR could be delivered at any time in the project two versions are scheduled. This plan, as it is described in more detail in Section 2.2.3, shows how all WPs will be evaluated for their expected patent applications and software copyrights. This task will also make sure that the necessary steps are taken for protection of this intellectual property, e.g., by ensuring confidentiality of information as necessary, and by the timely filing of patent applications by the appropriate partner.

Partners’ role: CERTH will lead this activity as the project coordinator. All other partners will make sure that their interests are consistently reflected in the Financial and IPR plans and contribute to the quality control of the technical outcomes.