More about our data

A lot of our time so far has gone into marshalling our data (as explained in our Data blog post), and getting it into a workable form, properly indexed and so forth. This means that it should now be comparatively quick to process the data into chart form - no setting it going and leaving it overnight now!

The Sakai event table, which is most of our data, has given us 240million rows of data for the last 5 years (a little over, actually). Only 70m of those turned out to be useful, as the rest were mostly things like the 'presence' event. We turned that event off in 2007 because it was causing unacceptable server load. Basically, our VLE is made up of a series of sites, and the presence event was a row written to the database every 5 seconds for every logged in user, saying which site they were currently looking at, so that an up-to-date list of who was in the site was displayed. (Facebook's indication of who's around in Chat at the moment probably does something similar.) So you can guess that this presence event generated an awful lot of data until we turned it off.

We also have 500m rows of Apache web data logs, telling us who visited which pages using which web browser and operating system. This is currently not something we're looking at so much (beyond a quick analysis of which web browsers we need to do our most thorough testing against), but it will be most useful when we're looking at which of our help resources have been most visited.

For our sets of data, we've been breaking it up by academic year (well, academic financial year - the year end is 31st July), and by week, so that we can see the fluctuation in usage. (We're starting to break it up by day as well, but this takes a long time to index.)

The story so far...

Sorry about the quietness here over the past couple of weeks: you must be wondering what we were up to.

• We've been extracting the data from Sakai, which was more difficult than it sounds. Sakai stores its events in a massive SQL table, one after the other, so that it's tens of millions of rows long before very long at all. Merging tables, fixing corrupt old data, that kind of thing. Anyway, all done now.
• We're investigating tools to help us analyse the data. Pentaho looks very promising.

But all this is just detail (albeit time-consuming, irritating detail) around the core issue of what data have we got and what can we do with it. To that end we've had a few internal workshops, sent out a few emails, bent some ears, and so on.

Though none of this should be treated as doctrine, and we're still definitely open to ideas, we thought it was time to do some initial data investigations, now that we have it. The key structuring concept for me is:

Who will be interested in our data, and what would they like to know?

An easy to imagine, but not entirely encompassing imaginary situations are these.

• If someone else were running the VLE, what would we want to know about it?
• If we could get secret, spy-style access to our deadliest rival institution (identity an exercise for the reader) what would we want to find out to make our VLE more awe-inspiring than theirs?
• If a charismatic leader were to rouse academics or students to come to our door bearing pitchforks and burning torches, demanding VLE data, what would be the rhetoric -- what would they be demanding?

If we bear these (and similar) questions in mind when we are steering, we shouldn't go far wrong. Let's not get caught producing a series of odd, disconnected charts, they need to inspire thought and change. We need charts, data and stats that connect with the machinery of change.

In terms of the data, what we have is:

who does what

So to do a meaningful analysis we have two axes: Who and What. While we'll give away as much raw data as is possible, we need to provide supporting mappings. Who is dps10? What is site 85? We also need to make sure, when we anonymise that we don't lose those aspects that enable external people to ask questions.

We're working out how we should take a first stab at Who and What, and are looking at finding sources. I imagine that when we've done this first round of analysis we'll discover the world doesn't divide up how we imagine. That seems to be the near universal experience of user experience analysis, certainly we learnt in our JISC Academic Networking project that the world of networking isn't divided up in quite the way we imagined. As we discover this from the activity data, we will iterate around, trying again and again.

It might even be worth applying Bayesian Clustering or Entropy-Based Tree Building to see how a machine would cluster behaviour. All very exciting (to me, anyway!). See pages 15-21 of this powerpoint by Allan Neymark at SJSU to see all this simply explained in terms of Simpsons characters.

Exciting times. At the same time, extremely tedious for the guys doing the database extraction and normalisation. Personally, I seem to have escaped that bit for this project. Phew!

Data visualisation

We're looking at a couple of tools here: BIRT and Pentalho, both of which have free business visualisation software packages. We're hoping that they can offer us more than you can get from Excel pivot tables, and be easier to set up than a bespoke solution involving some PHP and graphing software.

This isn't as straightforward as you might imagine. Raad's been working on setting up a Pentalho instance for most of the last week, and hasn't yet managed to get a significant improvement on what Excel provides, though it's taken considerable effort to get this far. Pentalho requires various modules to be installed, but its documentation is rather incomplete, especially the documentation for creating aggregate tables. Aggregate tables are essential when dealing with large volumes of data - we have over 10m rows of Sakai event data, so without aggregate tables, every time we try to look at a large section of the dataset, we run out of resources. So thus far, our suggestion would be that if you want business information software, you may be better off paying for a commercial product.

The Data

We’ve just started work on our JISC project on Exposing VLE Activity Data. First, we’ve had to get our data (first, catch your rabbit..), from when we started using CamTools (our current institutional VLE) to December 31 2010. This involved retrieving archived data, which didn’t go as smoothly as we’d hoped. We had to do some restoration of corrupted data, and we’re missing about two weeks of data as well. This just illustrates the problems of dealing with data that’s collected but not looked at very often.


The kinds of data we’ve collected are all the events from the Sakai event table. Sakai is the underlying software that powers our VLE (Virtual Learning Environment). Its event tables contains details of software ‘events’ - something that’s happened. Typical events are things like ‘content.read’ (someone’s read some content), ‘content.update’ (someone’s updated some content) or ‘search’ (this is probably easy to work out!). We’ve also collected data about who’s visited which web pages inside Sakai, when they did it, and which web browser they were using at the time - more typical access log data for web pages..

Now that we’ve got all this data from our logs, we need to make sure it’s in a format where we can process it, to find the answers to some of our questions about how the VLE is used. However, we may also want to collect other, ‘softer’ data, such as what each area of the VLE is used for (teaching, research, admin, or something else), and why it’s used. This will require more human input, whether by examining individual sub-sites of the VLE, questionnaires or interviews.
General Observations on what the limitations of the data are

As mentioned above, we mostly can’t determine what a site is used for, other than by human inspection. The exception to this is sites designed to support lecture or degree courses, for which we maintain a list. So while we may be able to track usage patterns for an individual site, we can’t easily do so for a set of related sites, unless we define the relation manually.

We’ve observed sites being used for: teaching, research, administration, social activities and testing (using sites as a sandbox to try things out before updating a site that’s already being used by students or researchers). More specifically, we’ve seen sites used for teaching lecture courses, whole degree programs, small-group teaching, language learning. We’ve seen sites used to organise research projects, from PhD theses up to large international collaborations. CamTools has been used to administer part of the college applications process, and for university and college societies, and to organise conferences. But unless a human looks at a site, we’ve got no way of deducing this from the data (we don’t capture extensive metadata on site creation).

So, how do we categorise a site?
Currently, sites which are associated with a specific lecture course or a degree course are tagged with metadata on creation. This is a relatively new procedure, so only sites active from October 2010 are tagged. However, signalling that a site is no longer in active use for teaching (because a new site has been created for the new academic year, for example), is harder. The case I just mentioned can be done by editing the metadata, because we will have information that there is a new site; but if a lecture course has been discontinued, we can’t currently update that.
For other sites, we have to rely on manual inspection. What is the site called? How many people are in it? What documents are stored there? Which tools does it use? From this information, we can usually deduce what the site is used for.

Does a site’s purpose change?
There are two aspects to this question: does a site, for example a small-group teaching site, turn into something else - perhaps a research site, or a site for that teacher’s lecture course? Or, does someone set up a site expecting it to be used in one way (putting in certain tools), and it turns out to be used in another?
The former is difficult to determine. All we can do is examine a site and say that it was being used in a particular way at a particular time, unless we can find out particular ‘signatures’ which denote the type of a site (at the moment, we don’t know whether sites would have distinctive signatures). The latter may be more amenable to analysis, in two ways. One, we can look at tool usage: tool X was added in 2008, but was never used, tool Y was added in May 2010, and has some hits. Two, we can conduct interviews with site owners, and ask them what they thought they were going to do, and what they actually found. (This does have the problem that people’s memories may be unreliable, but we can check what they say against the data we hold about their site.)

These kinds of approaches allow us to augment the automatically collected data from the past four years of running the VLE.

Hypotheses

We have four hypotheses we want to test:

1) Senior stakeholders in our VLE would like richer information about VLE/VRE usate, so that we can show growth potential, whether across the campus or in specific faculties or departments. We will test this by presenting the visualisations of our activity information to our Centre's management committee (or equivalent decision-making committees) and gathering their responses to the information, as well as obtaining their opinion on whether a case is made for a change in investment level.

2) We aim to identify 'usage signatures' which indicate either skilled use of the VLE, or users who may be struggling, but who do not ask for help. In the former case, we'd like to share what they're doing; in the latter case, we'll look at the relation between access to our help documentation and our helpdesk tickets. We will test this by correlating a usage signature with the reported experiences of academics, gained via phone or email interview.

3) We believe we can change our academics' attitudes towards the institutional VLE, by providing clear presentations including visualisations of activity information. We plan to test this by experimenting with different presentations of elements of our activity information to establish what the most effective and engaging presentational formats are. We will survey academics at the start and end of the project to measure their attitudes towards the VLE, which should allow us to measure our results.

4) We think a comparison of VLE usage information across the universities of Cambridge, Oxford and Hull should prove valuable to the sector, as we may be able to identify similarities and differences in VLE usage which may inform future consideration of the transferability of VLE project results and concepts across institutions.

Exposing VLE Activity Data - The Project Plan

CARET (University of Cambridge), in conjunction with Hull University and the University of Oxford will be working on a JISC funded project to bring together activity and attention data for our collective institutional VLE environments.

We've already set up the Google code site for the project here. Over the next few posts, we expect to explore our project plan, and a few of the early experiences in collecting and processing data we've already got stored.

Aims, Objectives and Final Outputs of the project -

From this project we aim to analyse the logging data to help us produce behavioural activity reports and statistical data. It will also highlight the ways in which the VLE platforms are being used and for what purpose.

A small amount of background: Cambridge's VLE (Virtual Learning Environment) is called CamTools, and is based on the Sakai software used by universities and colleges worldwide.

Our objectives are to find out more about:

• how people are using our VLE. This will allow us to look at potential areas for growth in VLE use in our institution
• how well support requests reflect usage patterns, so we can improve our support services
• what information is already available to us in our event logs, and how we can present this to management

Final outputs will include:

• this blog, which will contain detailed methods and reflections on the tools, data, and our experiences. This can then be used by the community; hopefully Hull and Oxford Universities will already be doing so by the end of the project
• the hypotheses about VLE and support service improvements, which we will have tested, informing people about the value of activity data to improve institutional services.
• activity information datasets, released so that other people can conduct research into this area.

Risk Analysis and Success Plan -

Staffing risk: low, as we already have staff in post, and the project team has worked together before. Alternative staff are available for all roles if substitution is necessary (we have already demonstrated this, as our project manager who created the initial bid for this project has left, and we've been able to replace her with Tony).

Risk that we will not be able to release datasets: Senior stakeholders in the VLE may not approve the release of datasets, even if we anonymise those datasets. This is a moderate risk, but we can alleviate its impact by documenting those concerns for the benefit of the wider community

Risk of disengagement of senior stakeholders: Our senior stakeholders may not be interested in or engaged by the results we derive from activity data. Thus there is a moderate risk that the problems we wish to solve using this data may remain unsolved, if we cannot persuade management to take action on the basis of the data. However, we'll still have gained useful tools for our production team, and for the Sakai community as a whole.

Risk: over-ambitious hypotheses. We may find that we're over-ambitious in the hypotheses we wish to test. However, we can continue the project to some extent using institutional funds, once the project infrastructure has been created. So while this is a moderate risk, we can make sure that its impact is low.

Technology risk: low, as we've already conducted a brief feasibility study, and there are plenty of data visualisation tools are available.

Project management risks: low, as we will use a lightweight management methodology to track risks as we go. We are proactive about identifying our targets, and have an aggressive timescale for meeting these, so we can reduce the risk of going over time or budget constraints.

Our criteria for success are encapsulated in our expected final outputs, as mentioned above.

IPR -

All software outputs will be released under an Apache2 licence (as mentioned above, it's all going in our Google code site), and all documents under a Creative Commons "BY" (Attribution) licence. This means that people can reuse our outputs, even commercially, which can support the creation of business models for more sustainable systems, including collaborative development across both nonprofit and commercial organisations.

Project Team Relationships and End User Engagement -

Project Manager : Tony Stevenson - Tony has led and delivered many projects throughout his career, varying in size and methodologies. Whilst this is hist first time managing a JISC project he has the experience to lead the project.

Script Developer : Raad Al-Rawi - Raad is not only the lead developer for the Cambridge institutional VLE, he is also a respected Sakai community member. Raad will work with Tony and the other members of this project team to help identify, access and use the VLE activity data from within Cambridge.

Technical Support : Daniel Parry - Daniel is a member of the operational team within CARET and will be able to help the team with technical issues arising in the obtaining of data and it's analysis.

Researchers : Verity Allan & Katy Cherry. The researchers will be our primary point of contact alongside Tony with the Cambridge user base. We expect the researchers will offer invaluable insights into the way that the institutional VLE is being used, by whom and for what purpose. Katy is an experienced research assistant, practised at communicating with academics, and at producing communication materials. Verity is also an experienced researcher, with extensive expertise in supporting academics using CamTools, the Cambridge VLE platform.

End user engagement will likely take many forms; it is not entirely clear what methods will work best initially. So we will use this blog to report on the methods we used, and which worked best.

Projected Timeline, Workplan & Overall Project Methodology-

Workplan 1: Project Management. This will continue throughout the project.

Workplan 2: Data Harvesting Phase 1: this will involve collating our existing logs, and starting work on finding appropriate visualisation tools

Workplan 3: Data Analysis Phase 1: this will use the data from the previous phase to create powerful visualisations of log data

Workplan 4: Data Harvesting Phase 2: this will include hand examination of the CamTools sites, to categorise them. This is the phase that will also include user questionnaires and interviews

Workplan 5: Data Analysis Phase 2: analysis of the data collected in Workplan 4

Workplan 6: Evaluation and Write-up: We will present the results to senior management, and evaluate the results of our data.

Workplan 7: Dissemination and Engagement. Throughout the period we will be writing our blog, engaging with users and management, and the JISC and Sakai communities.

Our project methodology combines statistical analysis of large datasets, and collecting individual information. Thus we will analyse our existing datasets of events, and convert them into useful information. We will be looking at ways to anonymise our datasets. We will also be hand-inspecting all sites in our VLE to classify them as teaching, research, social, administrative or other sites. We will consulte senior stakeholders to find out what reports on activity data they would most value, and will be working with them to try to secure release of anonymised data sets for further research. Once we've analysed our datasets, we will create visualisations of them to produce activity information which is meaningful to humans. And we'll be sharing our methodologies and (hopefully) our data with the community.

Budget -