Using Gephi to Visualise our Sites and Users

Discussion

We took as our starting point the talk by Tony Hirst on visualisation http://blog.activitydata.org/2011/07/online-exchange-4-event-recording-21.html, in particular the slides produced using Gephi.

The problem with this approach was that all the examples using Gephi were using it to map a network of the same items – eg a network of Twitter users. We wanted to visualise a network of sites and their users – a bipartite network. I found a reference to another tool that had some capabilities for the display and analysis of bipartite networks: ORA (developed at Carnegie Mellon, part of the CASOS (sic) package: http://www.casos.cs.cmu.edu/projects/ora/ but the comment also said that 'the visualizations are simply not on par with those of Gephi'. A quick look at this did not make me feel that it would be useful for our needs.

Instead I decided to try using the number of common users to link individual sites and to produce a Gephi visualisation where the sites were the nodes with their size representing the number of active users of the site. The thickness of the connecting lines between the sites would be related to the number of common users they had.

To do this, I developed a Perl script to determine the number of shared users between each of a list of sites and produce the nodes and edges data files that could be imported into Gephi. The detailed steps taken to produce network visualisations using this data will be reported in a separate blog post.

The visualisation above shows the Sites used within one of our colleges and links them by the number of shared users. The size of the circle increases with the number of users of that site. So we see the biggest site is the College main site with the next site being Computing with many shared users.

The Modularity Algorithm within Gephi was used to partition the sites into clusters where the sites are tightly associated with each other and more loosely linked to other sites. We can see that the green coloured sites could be categorised as 'college services' (IT, catering etc), the blue sites are mostly admissions related and the red sites include many subject related sites.

A similar process was applied to one of our larger departments (Illustration 1). This showed clustering and shared usage around their first year sites as well as heavy usage of their library/resource site. We will be able to use this visualisation as a resource to help us quickly identify important sites within this department when we upgrade their VLE.

This procedure can be followed for either a list of users or a list of sites to produce similar visualisations which we believe will be helpful in understanding more about how our sites are related and linked in the future.

Analysis of when peak VLE activity has occurred

Purpose

To look at the total counts from events logging and determine when peak activity occurs.

Background

Caret has developed a tool that can be run to extract total event counts by month or week from the VLE logging data. This document will explore some of the trends we have found from this data.

Warnings

The event logging data we are working with was collected until February 2011 so does not include full data for the last academic year (2010-11).

The week numbers were added to the underlying events file by using a function within MySQL. This is not ideal in that it leaves part weeks at the beginning and end of year and these points needed some manual fixing within the spreadsheet. It may also have helped to number weeks starting from the beginning of the Academic Year.

On initial analysis, a very large proportion of event activity was found to be from an event (pres.begin and pres.end) for which logging was later switched off. This has been disregarded from our counts.

Method

A script was run that produced monthly event counts from the VLE activity data over the last 5 years – this ran until Feb 2011.

The count for each event type was compared with the total count to produce a percentage. These results were then filtered to produce charts according to the scale of usage.

Results

The top 2 events were found to be reading content (green - 44% of events logged) and user logins (dark green - 22% of events logged). The chart below shows monthly counts for these events.

Whilst the content read shows a regular peak in Oct/Nov with a secondary peak around May, the logins do not show such a clear pattern. Both show a clear gradual underlying increase over the years. More investigation would be needed to find out exactly what is causing the peaks in logins. This is especially the case since it bears little relation to the number of user logouts recorded (see later graph).

The following shows a weekly count for content read giving more granularity of when exactly the peaks are. The main peaks are repeated in late October with secondary peaks in Feb and mid-June.

This considers weekly counts over the last year. (Feb 2010-2011).

Term dates were:

12^th January - 12^th March 2010. Easter (5th April 2010) shows a slight dip.

20^th April - 11^th June 2010. A gradual increase with a peak at the end of May.

This coincides with the examination period. There is a long dip over the summer vacation period.

5^th October-3^rd December 2010 First peak is early October with the highest peak at the end of October. This peak at the end of October might not be expected and it would be interesting to find out more about what is going on here. There is a large dip over the Christmas period

18^th January - 18^th March 2011. There is a wide peak during January and early February.

Although content read does have a peak at the start of the Academic Year in October, there is a higher peak in November. It would be interesting to investigate if there is an explanation for this peak half way through the term – such as a large number of students starting assignments which rely on Camtools.

This shows the next active events in terms of counts. Note the scale is roughly 10% of that for viewing content and logging in. However the overall peaks for site visits, search and user logout very much mirror those for viewing connect and logins showing consistent periods of peak activity corresponding to term dates.

As one might expect, lots of new content (blue line) has been added each October, at the start of the academic year, but there is a more recent and much higher peak in January of this year.

The following series of charts show analysis of the next highest peaks broken down by event type. Note the scale is several times smaller that that for reading content.

The calendar event is logged when users create (red line) or revise (yellow line) site calendars which may be used to display lecture times, assignment submission dates and so on. There is a clear October peak for both creating and revising calenders.

This tool has been added in the last few years and is used to for student surveys. The green update peaks in February would correspond to lecturers preparing surveys at this point which are returned by students roughly a month later (blue line).

The counts for Wiki activity show a peak in new material just prior to the October start of year with high revision activity at the start of October.

Site Maintenance demonstrates a clear October peak with a secondary peak in Jan/Feb. There is clearly a gradual increase in the number of sites being updated (red line) whereas the number of new sites (blue line) has leveled off.

Permissions changes reflect people being added or updated on sites. Again peak activity is in October.

Forums again show a peak in October but their usage looks like it has tailed off.

The chat tool also shows peaks in October and for the last year in November too.

Again the roster tool, which is used to view other members of a site shows peak usage in October.

One area that I was particularly asked to look at was the Tests and Quizzes (T&Q) tool. The scale for this is again of a much smaller order of magnitude but again shows October peaks:



This final chart shows the overall totals for each month (blue) and the monthly totals excluding logins and content read (orange). This again shows a peak in October and other peaks corresponding to the terms. There is not much growth, in terms of event counts, over recent years in these other activities.

Conclusions

Many activities show a clear usage peak corresponding with the start of the Academic year in October. However reading content, which accounts for over 44% of the counts, has a higher peak in November than in October. It would be interesting to learn exactly what is causing this November peak.

It also seems likely, given the different pattern in logouts, that the high spikes in login activity seen in the first chart are caused by spurious data and further investigation is needed here.

Releasing anonymised data - Stage 1

We are now in a position to make an initial release of our VLE logging data. This blog details the process we went through to get to this stage and the decisions that we made along the way.

We were helped in this process by David Evans from Cambridge Computer Lab (http://www.cl.cam.ac.uk/~de239/) who is an expert in data privacy issues. We also considered the Privacy Policy for the site and consulted with our Legal Services team on releasing anonymous logging data under an Open Licence.

There are 3 files that we will be releasing:
Sakai Event File
– Individual events such as a site visit which occur within a Session

(held as a separate file for each Academic Year)

Sakai Session File
– Session details which include browser information and session times

Sakai Site File
– Details of the VLE Site being accessed

A tarball file can be obtained from here - however this is a 4GB file that will expand to over 12GB when uncompressed.


Our first step was to provide David Evans with the database schema for these files so he could consider, based on discussions with us, which fields might contain sensitive data. We then discussed the implications of making these fields public and what information they might expose. This was of course a balancing act between releasing data from which others could draw value and not giving away details of what a known individual might have used the VLE system for.

We decided on a first data release which would use a cautious approach to data privacy. Should other institutions find this data of interest, we can work with them to reveal more information in the area of interest to them in a manner that does not compromise individuals.

This cautious approach meant hiding any data that would identify an individual user, site names and anything that might link a session back to a particular user. We settled on a hashing algorithm to use to obscure any such items of data yielding a string that can be determined uniquely from the value; we also used a salt to prevent inversion of the hash through exhaustive search of short inputs.

At this stage, we also looked at some sample data to reinforce our decisions.
The decision on what to hash was straightforward in many cases such as concealing any field with Site, User Name, URL or Content in it. Some things were less clear cut. For instance, the skin around a site could be used to identify a Department. The Session Ids that we looked at appeared to be numeric and we decided there was little risk in leaving this in its raw state. However, later testing revealed that, in some instances and points of time, this has included a user identifier so we agreed to hash this. It is worth remembering that the hashing algorithm is consistent so even though the value of the Session Id has been changed, it can still be used to link the Event and Session tables.

The Session Server holds both the Host Name and software module Id. We decided to hash the Host Name, in case this might reveal something about the network's internal structure, but leave the numeric part (software module id) alone as it reveals nothing other than which particular instance of software processed that session. We discovered that the format of this field had changed over time so we needed a mildly complex function to detect and extract from both formats of the Session Server.

The Session User Agent may provide the Browser Name and Version and the Operating System and Version of the computer used for the session. However this is a free-form field which may be changed by the user. There was a danger that this could identify a given user. A visual inspection showed at least one College Name, some company names and some school names within this field which could present a risk. Ideally we would extract and expose data such as the Browser Name but as this is a free-form field this in non-trivial. We therefore took the decision to hash this field.

As a final sanity check, we revisited some sample data from each of the tables once they had been hashed to satisfy ourselves that there was no raw data left that might possibly contain a user identifier.

Tooling

As part of this project we have learnt that not all the OOTB tools will meat our criteria, or represent VLE data in much of a meaningful manner.   We tried commercial tools, OSS tools, and even some rudimentary spreadsheet representations.  None of which really came up to scratch. 

Part of the problem, is that it is really hard to get a product to show you meaningful output if you: 

1. Dont asking reasonable questions of it, and 
2. If you dont know what your data is actually telling you. 

As a result of this one of our developers has written some custom perl to extract useful statistics from the data.  We needed to use something that was quite capable of munging large volumes of data as our SQL activity data contained over 200 million rows over the period that we have retained our data. 

This perl script can be found here held in the CARET public SVN repo space -  http://source.caret.cam.ac.uk/svn/projects/vledata/Sakai-Stats/  - Please note that while this is a public repository you will not be able to commit changes directly to it.  We would however love to hear from you if you take this code and use it, or modify it for your own needs. 

HowTo Guide - Using Pivot Tables in OpenOffice

We have been using pivot tables within OpenOffice to help us drill into the large data sets more easily.  Anne has produced a short HowTo guide in how to use these effectively, and this can be read here

More about our data

A lot of our time so far has gone into marshalling our data (as explained in our Data blog post), and getting it into a workable form, properly indexed and so forth. This means that it should now be comparatively quick to process the data into chart form - no setting it going and leaving it overnight now!

The Sakai event table, which is most of our data, has given us 240million rows of data for the last 5 years (a little over, actually). Only 70m of those turned out to be useful, as the rest were mostly things like the 'presence' event. We turned that event off in 2007 because it was causing unacceptable server load. Basically, our VLE is made up of a series of sites, and the presence event was a row written to the database every 5 seconds for every logged in user, saying which site they were currently looking at, so that an up-to-date list of who was in the site was displayed. (Facebook's indication of who's around in Chat at the moment probably does something similar.) So you can guess that this presence event generated an awful lot of data until we turned it off.

We also have 500m rows of Apache web data logs, telling us who visited which pages using which web browser and operating system. This is currently not something we're looking at so much (beyond a quick analysis of which web browsers we need to do our most thorough testing against), but it will be most useful when we're looking at which of our help resources have been most visited.

For our sets of data, we've been breaking it up by academic year (well, academic financial year - the year end is 31st July), and by week, so that we can see the fluctuation in usage. (We're starting to break it up by day as well, but this takes a long time to index.)

The story so far...

Sorry about the quietness here over the past couple of weeks: you must be wondering what we were up to.

• We've been extracting the data from Sakai, which was more difficult than it sounds. Sakai stores its events in a massive SQL table, one after the other, so that it's tens of millions of rows long before very long at all. Merging tables, fixing corrupt old data, that kind of thing. Anyway, all done now.
• We're investigating tools to help us analyse the data. Pentaho looks very promising.

But all this is just detail (albeit time-consuming, irritating detail) around the core issue of what data have we got and what can we do with it. To that end we've had a few internal workshops, sent out a few emails, bent some ears, and so on.

Though none of this should be treated as doctrine, and we're still definitely open to ideas, we thought it was time to do some initial data investigations, now that we have it. The key structuring concept for me is:

Who will be interested in our data, and what would they like to know?

An easy to imagine, but not entirely encompassing imaginary situations are these.

• If someone else were running the VLE, what would we want to know about it?
• If we could get secret, spy-style access to our deadliest rival institution (identity an exercise for the reader) what would we want to find out to make our VLE more awe-inspiring than theirs?
• If a charismatic leader were to rouse academics or students to come to our door bearing pitchforks and burning torches, demanding VLE data, what would be the rhetoric -- what would they be demanding?

If we bear these (and similar) questions in mind when we are steering, we shouldn't go far wrong. Let's not get caught producing a series of odd, disconnected charts, they need to inspire thought and change. We need charts, data and stats that connect with the machinery of change.

In terms of the data, what we have is:

who does what

So to do a meaningful analysis we have two axes: Who and What. While we'll give away as much raw data as is possible, we need to provide supporting mappings. Who is dps10? What is site 85? We also need to make sure, when we anonymise that we don't lose those aspects that enable external people to ask questions.

We're working out how we should take a first stab at Who and What, and are looking at finding sources. I imagine that when we've done this first round of analysis we'll discover the world doesn't divide up how we imagine. That seems to be the near universal experience of user experience analysis, certainly we learnt in our JISC Academic Networking project that the world of networking isn't divided up in quite the way we imagined. As we discover this from the activity data, we will iterate around, trying again and again.

It might even be worth applying Bayesian Clustering or Entropy-Based Tree Building to see how a machine would cluster behaviour. All very exciting (to me, anyway!). See pages 15-21 of this powerpoint by Allan Neymark at SJSU to see all this simply explained in terms of Simpsons characters.

Exciting times. At the same time, extremely tedious for the guys doing the database extraction and normalisation. Personally, I seem to have escaped that bit for this project. Phew!