"Get Your Back Straight" backstraight: Evaluation of the app created
Data-Analysis mmla: Is there a difference in the data?
AI in Assessment
Systemarchitectures and -design
Definition and general challenges with assessments
High-stakes assessment
Provide reliable testing assessment
Guarantees for display and answers
Curated editing process elicitation
MYSKILLs
Project at DIPF
Initiated with Bundesagentur für Arbeit
Experimentation with Bildungsträger
Online assessment with 100-200 parallel exams
Refugees and retraining
Assessment
Systemarchitectures and -design
Old architecture
Monolithic application based on TAO 2.5 (old!)
Failover required prior duplication of tests (and much more)
Each server instance could handle max 50 exams
MYSKILLs NEXT architecture
Follow-Up project (Internal Funding: 2 FTE)
Distributed system with load-balancing only on critical loops
Decoupling from monitoring and execution
Event-driven system
Public APIs and token based authorization system (custom)
Offline and adaptive testing
Assessment
Systemarchitectures and -design
MYSKILLs NEXT architecture
The MYSKILLs NEXT Architecture: Assessment as a Service
Assessment
Systemarchitectures and -design
DeCoS - Delivery Content Store
Main challenge:
There must be explicit lists of validated and versioned documents, i.e. snapshots of tests
git-based JSON files
GraphQL interface
Partial schema(less)
NodeJS server
Alternatives Considered:
Redis
MongoDB
Both would require a custom index to be maintained!
Learning and Assessment
ComPleTT
Common Platform for electronic Teacher Training
Working environment
Used in all 16 states of Germany!
Collection and distribution of courses and media
Impact and Outreach
Started 2020 with 3 states (RLP, HE and BY)
Currently over 800 persons registered
Creating and sharing materials for teacher education
Permanent funding via KMK approved in 2022
Strong connection to lernen:digital (BMBF Project, 2 FTE)
APIs for OpenScience
Definition
OpenScience requires to follow several principles
We use FAIR
Allows for free exchange of knowledge
Challenges
Recursive, especially if using FAIR digital objects (FDO)
Abort criterion in A2 is "weak"
Datasets tend to degrade or disappear over time because there is a cost to maintaining
an online presence for data resources. When this happens, links become invalid and users
waste time hunting for data that might no longer be there. Storing the metadata
generally is much easier and cheaper. Hence, principle A2 states that metadata should
persist even when the data are no longer sustained. A2 is related to the registration
and indexing issues described in F4.
Assumption that metadata stays "fixed" is unfeasible
Access might / must be still restricted
Uniform access might enable more sharing
APIs in OpenScience
NFDI
Konsort SWD
Consortium on social, behavioral, educational and economic sciences → Konsort SWD
Development of APIs for data exchange (TA5-M3, 3.2 FTE)
EduCS furthermore has
Connections via RDCs
Relation to NFDIxCS
Strong partnership with other Leibniz institutes regarding metadata
APIs in OpenScience
Examples
KonsortSWD: FAIR API konsortswd
OpenData in EdTec delfi_opendata,iticse_opendata
What is a good API? api
Enhancement of tools rdmo
Shared and safe data space (Presented at GO-INTER)
FDO model used as a building block konsortswd
What are important criteria for an API api
Computer Science Education
Definition
Teaching can be supported using computer technology
Especially computer science education lacks good tools
ChatGPT / Co-Pilot have a huge impact
Challenges
Data is not "selectable" → What sources are used
Data is not "clean" → Good / bad examples
Features of current ai systems
DevUX
Comparable with assessment
What is good feedback?
How to adapt to student's performance
Plan of EduCS
Use infrastructure to run and evaluate
Reach out to "try" it in the field
DALL-E prompt: Generate a picture of a young computer-science student programming with AI
Computer Science Education
Examples
Impact on computer science assessments llm-edu
How to use chatbots finding a course in IWWB maze
How to design a system nlp
How good can ChatGPT solve programming tasks llm-edu
Visualization of available offers in the region [IWWB-PLUS]
Chatbots for lifelong learning maze
Other research topics at EduCS
Selection of topics
Data visualization (OpenLAIR, IWWB-PLUS dashboard)
connecting,indicators
Data protection and security (FDB/NFDI)
Data distribution and information / access sharing (lernen:digital, ComPleTT)
complett,vigor
Open educational resources and quality (OERinfo)
Knowledge Graphs
Projects
Overview
Name
Funder
Own Funding
Role
lernen:digital (2023-2026)
BMBF
525k€
Co-PI (with Prof. Rittberger)
NEPS TBT (2023-2027)
LIfBi
205k€
Co-PI (with Prof. Goldhammer)
OERinfo (2023-2028)
BMBF
370k€
Co-PI (with Prof. Rittberger)
ComPleTT (2022-)
KMK
115k€ / year
PI
TAEPS (2022)
DIE
16k€
PI
IWWB-PLUS (2021-2024)
BMBF
637k€
Co-PI (with Prof. Rittberger)
SchuMaS (2021-2026)
BMBF
385k€
Co-PI (with Prof. Goldhammer, Prof. Rittberger, Prof. Maaz)
KonsortSWD (2020-2025)
BMBF / DFG
465k€
Co-PI (with Prof. Wolf)
ComPleTT (Feasability Project, 2020)
Digitalpakt
35k€
PI
Adult Basic Literacy Skills Denmark (2020)
VIVE
28k€
PI
QualiBi (2020-2023)
DFG
163k€
Co-PI
PIAAC Cycle 2 (2019-2023)
OECD
84k€
PI
The Next Level (2019-2022)
BMBF
338k€
Co-PI (with Prof. Horz)
CATS (2016-2019)
BMBF
210k€
Co-PI (with Prof. Krömker)
Future Work
Increase involvement in Working Groups
Leibniz AG for software development
AK GI OpenScience
Projects
ComPleTT + lernen:digital
IWWB-PLUS → Scientific Use-Files
AI and Infrastructures → Support manual collection of metadata
MMLA → Seamless integration into learning environments
APIs and data visualizations
Literature
@article{mmla,
title={Novices make more noise! The D\&K effect 2.0? (accepted)},
author={Schneider, Jan and Asyraaf Mat Sanusi, Khaleel and Limbu, Bibeg and Schiffner, Daniel and
Schmitz, Marcel},journal={CROSSMMLA LAK 2023 Workshop: Leveraging Multimodal Data for Generating
Meaningful Feedback},
year={2023}
}
,
@inproceedings{iticse_opendata,
doi = {10.1145/3587102.3588860},
url = {https://doi.org/10.1145%2F3587102.3588860},
year = 2023,
month = {jun},
publisher = {{ACM}},
author = {Natalie Kiesler and Daniel Schiffner},
title = {Why We Need Open Data in Computer Science Education Research},
booktitle = {Proceedings of the 2023 Conference on Innovation and Technology in Computer Science
Education V. 1}
},
@inproceedings{complett,title={ComPleTT - Common Plattform for electronic Teacher Training. Ein
Blick auf die Entstehung der gemeinsamen Arbeitsplattform der Länder zur Lehrkräfteausbildung},
author={Kiesler, Natalie and Schiffner, Daniel},year={2022},booktitle={DeLFI 2022 - Workshop
Schule}}
,
@article{delfi_opendata, doi = {10.18420/DELFI2022-034}, url =
{http://dl.gi.de/handle/20.500.12116/38835}, author = {Kiesler, Natalie and Schiffner, Daniel},
keywords = {Open Science, FAIR principles, epistemology, technology-based research, software,
research data management}, language = {en}, title = {On the Lack of Recognition of Software
Artifacts and IT Infrastructure in Educational Technology Research}, publisher = {Gesellschaft für
Informatik e.V.}, year = {2022}},
@inproceedings{indicators, doi = {10.5220/0010995800003182}, url =
{https://doi.org/10.5220%2F0010995800003182}, year = 2022, publisher = {{SCITEPRESS} - Science and
Technology Publications}, author = {Atezaz Ahmad and Jan Schneider and Joshua Weidlich and Daniele
Di Mitri and Jane Yau and Daniel Schiffner and Hendrik Drachsler}, title = {What Indicators Can I
Serve You with? An Evaluation of a Research-Driven Learning Analytics Indicator Repository},
booktitle = {Proceedings of the 14th International Conference on Computer Supported Education}},
@article{connecting, doi = {10.1111/jcal.12716}, url = {https://doi.org/10.1111%2Fjcal.12716}, year
= 2022, month = {jul}, publisher = {Wiley}, author = {Atezaz Ahmad and Jan Schneider and Dai
Griffiths and Daniel Biedermann and Daniel Schiffner and Wolfgang Greller and Hendrik Drachsler},
title = {Connecting the dots {\textendash} A literature review on learning analytics indicators from
a learning design perspective}, journal = {Journal of Computer Assisted Learning}}
@misc{llm-edu,
title={Large Language Models in Introductory Programming Education: ChatGPT's Performance and
Implications for Assessments},
author={Natalie Kiesler and Daniel Schiffner},
year={2023},
eprint={2308.08572},
archivePrefix={arXiv},
primaryClass={cs.SE}
}
@inproceedings{api,
title={What is a Good API? A Survey on the Use and Design of Application Programming Interfaces},
author={Kiesler, Natalie and Schiffner, Daniel},
year={2023},
booktitle={2nd EAI International Conference on the Internet of Everything}
}
@article{maze,
title={Caught in the lifelong learning maze. Helping people with learning analytics and chatbots to
find personal career paths},
author={Ahmad, Atezaz and Kiesler, Natalie and Schiffner, Daniel and Schneider, Jan and Wollny,
Sebastian},
journal={International journal of information and education technology},
volume={13},
number={3},
pages={423--429},
year={2023}
}
@incollection{doh,
doi = {10.1007/978-3-030-86436-1_11},
url = {https://doi.org/10.1007%2F978-3-030-86436-1_11},
year = 2021,
publisher = {Springer International Publishing},
pages = {137--148},
author = {Tetiana Buraha and Jan Schneider and Daniele Di Mitri and Daniel Schiffner},
title = {Analysis of the {\textquotedblleft}D'oh!{\textquotedblright} Moments. Physiological Markers
of Performance in Cognitive Switching Tasks},
booktitle = {Technology-Enhanced Learning for a Free, Safe, and Sustainable World}
},
@InProceedings{backstraight,title={Get your back straight! Learn Pilates with the Pilates Correction
Game},author={Meik, Anna and Schneider, Jan and Schiffner, Daniel},booktitle={DELFI},year={2021}}
,
@InProceedings{vigor,author={Gattinger, Thorsten and Sacher, Patrick and Wei{\ss}, David and
Schiffner, Daniel},title={Developing a Cooperative Platform For Researching The Use Of Video-Based
Learning Units In University-Based Teacher Education},series={12th International Conference on
Education and New Learning Technologies},booktitle={EDULEARN20
Proceedings},isbn={978-84-09-17979-4},issn={2340-1117},doi={10.21125/edulearn.2020.1324},url={http://dx.doi.org/10.21125/edulearn.2020.1324},publisher={IATED},location={Online
Conference},year={2020},pages={5080-5089}}
,
@InProceedings{assessment,author={Frenken, Lena and Libbrecht, Paul and Greefrath, Gilbert and
Schiffner, Daniel and Schnitzler, Carola},title={Evaluating Educational Standards using Assessment
``with'' and ``through'' Technology},booktitle={Proceedings of the Tenth ERME Topic Conference (ETC
10) on Mathematics Education in the Digital Age
(MEDA)},year={2020},pages={361--368},url={https://hal.archives-ouvertes.fr/hal-02932218}}
,
@InProceedings{nlp,author={Libbrecht, Paul and Declerck, Thierry and Schlippe, Tim and Mandl, Thomas
and Schiffner, Daniel},year={2020},title={NLP for Student and Teacher: Concept for an AI based
Information Literacy Tutoring System},editor={Conrad, Stefan AND Tiddi,
Ilaria},booktitle={Proceeding of the CIKM 2020 Workshops},url={http://ceur-ws.org/Vol-2699/}}
,
@misc{konsortswd,
author={Limani, Fidan, and Johannes, Roland and Zhang, Yudong and Schiffner, Daniel},
title={KonsortSWD Task Area 5 Measure 3: Milestone 1 & 2 Report},
doi={https://zenodo.org/doi/10.5281/zenodo.6497189},
year={2022}
}
@inproceedings{rdmo,
author = {Natalie Kiesler and
Daniel Schiffner and
Axel Nieder{-}Vahrenholz},
editor = {Ren{\'{e}} R{\"{o}}pke and
Ulrik Schroeder},
title = {Adapting {RDMO} for the Efficient Management of Educational Research
Data},
booktitle = {{DELFI} 2023, Die 21. Fachtagung Bildungstechnologien der Gesellschaft
f{\"{u}}r Informatik e.V., 11.-13. September 2023, Aachen},
series = {{LNI}},
volume = {{P-338}},
pages = {51},
publisher = {Gesellschaft f{\"{u}}r Informatik e.V.},
year = {2023},
url = {https://doi.org/10.18420/delfi2023-51},
doi = {10.18420/delfi2023-51},
timestamp = {Wed, 18 Oct 2023 15:31:18 +0200},
biburl = {https://dblp.org/rec/conf/delfi/KieslerSN23.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@InProceedings{elicitation,
author={Chituc, Claudia-Melania
and Herrmann, Marisa
and Schiffner, Daniel
and Rittberger, Marc},editor={Herzog, Michael A.
and Kubincov{\'a}, Zuzana
and Han, Peng
and Temperini, Marco},title={Towards the Design and Deployment of an Item Bank: An Analysis of the
Requirements Elicited},booktitle={Advances in Web-Based Learning -- ICWL
2019},year={2019},publisher={Springer International
Publishing},address={Cham},pages={155--162},abstract={Assessments are an important phase in the
learning process. Information and communication technologies advancements determined the development
of e-learning software tools which support e-learning activities, including e-assessment. The
increasing usage of summative and formative e-assessments led to the challenge of managing items.
The concept of an item bank is meant to support teachers and students alike, to provide an overview
when taking assessments or creating exams. This article presents an on-going R{\&}D project towards
the design and deployment of an item bank for computer-based tests, and discusses its role within a
service-oriented system architecture which enables the execution of activities related to
e-assessment, ranging from item design and test creation, to the analysis of event logs generated by
test-takers. The research methodology followed for the requirements elicitation and main findings
are presented, and directions for future work are discussed.},isbn={978-3-030-35758-0}}
@inproceedings{presentationtrainer,
author = {Schneider, Jan and B\"{o}rner, Dirk and van Rosmalen, Peter and Specht, Marcus},
title = {Presentation Trainer, Your Public Speaking Multimodal Coach},
year = {2015},
isbn = {9781450339124},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/2818346.2830603},
doi = {10.1145/2818346.2830603},
abstract = {The Presentation Trainer is a multimodal tool designed to support the practice of public
speaking skills, by giving the user real-time feedback about different aspects of her nonverbal
communication. It tracks the user's voice and body to interpret her current performance. Based on
this performance the Presentation Trainer selects the type of intervention that will be presented as
feedback to the user. This feedback mechanism has been designed taking in consideration the results
from previous studies that show how difficult it is for learners to perceive and correctly interpret
real-time feedback while practicing their speeches. In this paper we present the user experience
evaluation of participants who used the Presentation Trainer to practice for an elevator pitch,
showing that the feedback provided by the Presentation Trainer has a significant influence on
learning.},
booktitle = {Proceedings of the 2015 ACM on International Conference on Multimodal Interaction},
pages = {539–546},
numpages = {8},
keywords = {multimodal interfaces, sensors, presentation training., immediate feedback, public
speaking},
location = {Seattle, Washington, USA},
series = {ICMI '15}
}
Thank you for your attention
Teaching
Lectures and exercises
Connection to research
Small projects for students
Basis for theses (just hit 130 persons recently)
Positive feedback
Teaching evaluation
Profillinie FB12: Evaluation result for HCI (N=28, WS18/19)
Profillinie FB12: Evaluation result for "Plattformen und Systeme fürs eLearning" (N=10, WS22/23)
Profillinie FB12: Evaluation result for "Plattformen und Systeme fürs eLearning" (N=8, WS23/24)
DALL-E prompt: Generate a picture of a young computer-science student programming with AI
