Tuesday, 22 February 2011

Exposing VLE Activity Data - The Project Plan

CARET (University of Cambridge), in conjunction with Hull University and the University of Oxford will be working on a JISC funded project to bring together activity and attention data for our collective institutional VLE environments.

We've already set up the Google code site for the project here. Over the next few posts, we expect to explore our project plan, and a few of the early experiences in collecting and processing data we've already got stored.

Aims, Objectives and Final Outputs of the project -

From this project we aim to analyse the logging data to help us produce behavioural activity reports and statistical data. It will also highlight the ways in which the VLE platforms are being used and for what purpose.

A small amount of background: Cambridge's VLE (Virtual Learning Environment) is called CamTools, and is based on the Sakai software used by universities and colleges worldwide.

Our objectives are to find out more about:
  • how people are using our VLE. This will allow us to look at potential areas for growth in VLE use in our institution
  • how well support requests reflect usage patterns, so we can improve our support services
  • what information is already available to us in our event logs, and how we can present this to management

Final outputs will include:
  • this blog, which will contain detailed methods and reflections on the tools, data, and our experiences. This can then be used by the community; hopefully Hull and Oxford Universities will already be doing so by the end of the project
  • the hypotheses about VLE and support service improvements, which we will have tested, informing people about the value of activity data to improve institutional services.
  • activity information datasets, released so that other people can conduct research into this area.

Risk Analysis and Success Plan -

Staffing risk: low, as we already have staff in post, and the project team has worked together before. Alternative staff are available for all roles if substitution is necessary (we have already demonstrated this, as our project manager who created the initial bid for this project has left, and we've been able to replace her with Tony).

Risk that we will not be able to release datasets: Senior stakeholders in the VLE may not approve the release of datasets, even if we anonymise those datasets. This is a moderate risk, but we can alleviate its impact by documenting those concerns for the benefit of the wider community

Risk of disengagement of senior stakeholders: Our senior stakeholders may not be interested in or engaged by the results we derive from activity data. Thus there is a moderate risk that the problems we wish to solve using this data may remain unsolved, if we cannot persuade management to take action on the basis of the data. However, we'll still have gained useful tools for our production team, and for the Sakai community as a whole.

Risk: over-ambitious hypotheses. We may find that we're over-ambitious in the hypotheses we wish to test. However, we can continue the project to some extent using institutional funds, once the project infrastructure has been created. So while this is a moderate risk, we can make sure that its impact is low.

Technology risk: low, as we've already conducted a brief feasibility study, and there are plenty of data visualisation tools are available.

Project management risks: low, as we will use a lightweight management methodology to track risks as we go. We are proactive about identifying our targets, and have an aggressive timescale for meeting these, so we can reduce the risk of going over time or budget constraints.

Our criteria for success are encapsulated in our expected final outputs, as mentioned above.


All software outputs will be released under an Apache2 licence (as mentioned above, it's all going in our Google code site), and all documents under a Creative Commons "BY" (Attribution) licence. This means that people can reuse our outputs, even commercially, which can support the creation of business models for more sustainable systems, including collaborative development across both nonprofit and commercial organisations.

Project Team Relationships and End User Engagement -

Project Manager : Tony Stevenson - Tony has led and delivered many projects throughout his career, varying in size and methodologies. Whilst this is hist first time managing a JISC project he has the experience to lead the project.

Script Developer : Raad Al-Rawi - Raad is not only the lead developer for the Cambridge institutional VLE, he is also a respected Sakai community member. Raad will work with Tony and the other members of this project team to help identify, access and use the VLE activity data from within Cambridge.

Technical Support : Daniel Parry - Daniel is a member of the operational team within CARET and will be able to help the team with technical issues arising in the obtaining of data and it's analysis.

Researchers : Verity Allan & Katy Cherry. The researchers will be our primary point of contact alongside Tony with the Cambridge user base. We expect the researchers will offer invaluable insights into the way that the institutional VLE is being used, by whom and for what purpose. Katy is an experienced research assistant, practised at communicating with academics, and at producing communication materials. Verity is also an experienced researcher, with extensive expertise in supporting academics using CamTools, the Cambridge VLE platform.

End user engagement will likely take many forms; it is not entirely clear what methods will work best initially. So we will use this blog to report on the methods we used, and which worked best.

Projected Timeline, Workplan & Overall Project Methodology-

Workplan 1: Project Management. This will continue throughout the project.
Workplan 2: Data Harvesting Phase 1: this will involve collating our existing logs, and starting work on finding appropriate visualisation tools
Workplan 3: Data Analysis Phase 1: this will use the data from the previous phase to create powerful visualisations of log data
Workplan 4: Data Harvesting Phase 2: this will include hand examination of the CamTools sites, to categorise them. This is the phase that will also include user questionnaires and interviews
Workplan 5: Data Analysis Phase 2: analysis of the data collected in Workplan 4
Workplan 6: Evaluation and Write-up: We will present the results to senior management, and evaluate the results of our data.
Workplan 7: Dissemination and Engagement. Throughout the period we will be writing our blog, engaging with users and management, and the JISC and Sakai communities.

Our project methodology combines statistical analysis of large datasets, and collecting individual information. Thus we will analyse our existing datasets of events, and convert them into useful information. We will be looking at ways to anonymise our datasets. We will also be hand-inspecting all sites in our VLE to classify them as teaching, research, social, administrative or other sites. We will consulte senior stakeholders to find out what reports on activity data they would most value, and will be working with them to try to secure release of anonymised data sets for further research. Once we've analysed our datasets, we will create visualisations of them to produce activity information which is meaningful to humans. And we'll be sharing our methodologies and (hopefully) our data with the community.

Budget -