The value of analytics in an educational and learning context

I had some time to reflect on the presentations and conversations at the 1st International Learning Analytics and Knowledge Conference in Banff. There were many thought provoking presentations that gave an inkling as to the direction people interested in education, or interested in educational technology, might take analytics in the coming years. One of the figures I found most interesting, came from Abelardo Pardo, who worked with Carlos Delgado Kloos on a 'virtual machine' that they used in their research. They asked their students to install the file on their regular computer and use it for their course work, while Abelardo and Carlos could track student activity. When they looked at the browsing behavior, only 28.51% of students accessed LMS-related pages for their learning activities, while all other pages accessed were outside this institutionally controlled environment. Now of course it is quite likely that there are contextual factors that influenced this behavior, but still, it clearly points towards a finding that students only make a limited use of the institutional LMS for their learning and that if analytics are to be meaningful, they will have to include student learning activities outside the LMS. Quite some analytics presented at the conference were related to the institutional LMS. Of course this begs the question: if students only use the LMS for such a limited amount of their learning, and data on the other learning is not collected, what will be the relevance and value of carrying out analytics on this LMS environment?

Some presentations showed that analytics might be used to enhance the effectiveness and streamlining of the processes taking place in educational institutions in four ways:
1. To support the administration
2. To adapt the learning support services to make up for deficiencies in student performance.
3. To show learners their analytics in order for them to reflect on their performance and perhaps adapt their learning and learning behavior in certain ways.
4. To adapt teaching to analytics findings about student learning and learning behavior

What do I think of these four?
1. Analytics to support the bureaucracy must always be a bad thing, as analysis of data always means inputting of data, from which follows that learners and educators will have to engage in this added burden. There is enough evidence to support that the bureaucratization of university is a negative, rather than a positive development (Foucault, Reading, Delanty).
2.I like the idea that analytics might make it possible for student support services to be better matched to student needs, but coming from a background in adult education and widening access to Higher Education, I have seen my fair share of problems with using the deficiency model to support learners. I feel more comfortable with
3. the analytics model promoted by Erik Duval who runs analytics on student activities and shows the students the results. This seems more empowering to learners as it involves a need for reflection on their learning.
4. Analytics can also be run as a research tool, so teaching staff might get a better understanding of the learner experience and the problems learners might come across in order to better match their teaching. Caroline Haythornthwaite showed us some of her visualizations of communication and group forming, which highlighted insights that analytics might provide in the ties between learners in learning settings.

If the analytics are solely run on the LMS related activity and the 28.51% figure is in any way generalizable to other institutions, of course all these analytics will only tell roughly a quarter of the story. It means that people will have to start using analytics outside the institution, on the network, as Helene Fournier and I have done here at NRC in Moncton. Of course carrying out analytics on networks is not easy as people access services in a distributed environment and the analytics would be most meaningful if these could somehow be linked, perhaps by using the same identification for all of them. It would be cool, though, and could enhance the learning experience of self-directed learners, if they would be able to quickly check if they would meet their learning goals through visualizations of their activity. As that is one thing I have learned engaging in analytics: visualisation does clarify activity pretty well.

Some other developments related to linked data are the research and design of recommender systems for learning. Currently there are problems with the testing of these as large data-sets are required to ensure reliability and consistency of results as Katrien Verbert highlighted in her talk. Some other analytics-related systems are currently under development at the Open University in the UK, such as Cohere, a discourse argumentation tool that aims for depth in discussion, and iSpot, related to BBC nature programmes, that uses a novel ranking system.

A theme running throughout the conference in several of the presentations, was the ethical dimension. analytics is about human behavior and of course there are some important ethical considerations to the collection of human data. This will be another post before long.

Learning Analytics is clearly a developing field and there is still a lot to learn for all involved! Thanks again George for bringing us all together in such a wonderful location.

Learning about learning analytics

I have been at the 1st International Learning Analytics conference in Banff over the past two days and it has been an eye opener to the power of analytics. Not just to enhance efficiency of educational institutions, but also to help learners in their learning and researchers and educators in getting an understanding in the dynamics in class rooms and on networks. I will write more about the most interesting points later.

For now I have included the link to the presentation at the conference by Helene Fournier and me that tells a bit more about our use of analytics in our MOOC research. If you prefer a different format, Doug Clow has been tapping away on his live-blog and given a near verbatim account of the event.You can find the link here.

Learning on MOOCs

Back to writing my blog after some time writing different types of papers and having been 'up to my elbows' in data, yes MOOC data, to make sense of the learning processes on PLENK2010. Helene Fournier and I have been quite busy over the past months analysing data and writing papers and preparing presentations, which we will post here as they are being published.

The post by Martin Weller on the responsibility of learners in a MOOC, and the comments to it, has made that it seems time to write some of the findings of our research here. The issues I found most interesting in the PLENK data relate to Martin's point on responsibility of the learner, the nature of active participation on MOOCs, the factors affecting active participation and how people on PLENK experienced their participation.

From the data (that we collected using qualitative and quantitative methods) it was clear that there are a number of issues that stand out.

1.  Power relations on the MOOC
2. Confidence levels of novice MOOCers
3. The level of presence of participants and facilitators
4. The willingness to help by all involved.

Participants felt they had a new responsibility to actively participate  in new ways: eg. by aggregating resources, by RSS or Twitter, to do something with these resources, but not necessarily in the way that the facilitators would have hoped for, by actively producing something. The majority of participants (1580 were not really big producers, there were only around 60 people who did this) shared resources with other participants, but also with people outside the MOOC. The level of creative production very much depended on the level of confidence of the participant and the the amount of time the participant had to invest on learning in the MOOC.

I have included here a link to the slidecast of a presentation I gave of a paper on New Dimensions to Selfdirected Learning on Open Networked Learning Environments, that Helene Fournier and I  prepared for the 25th International Self-directed Learning Symposium that was held in Cocoa Beach, Florida earlier in February.

Who is to say what is active participation?

The online discussion that George started by his post on lurking on #PLENK2010 made that Helene and me at NRC wanted to know a little more on how people perceived themselves to be: 'lurkers' or 'active producers', what determined their choice and how important lurking or actively producing has been for their learning as this really goes to the heart of learning on MOOCs. That’s why we produced the surveys.

People who have read my blog posts and earlier comments on George's and Jenny's blogs know that I researched 'lurking' in the past and already know that especially self-directed learners will fall into the lurking category. Of course we should not forget that the majority of PLENK participants are mature adults with a high level of education and a predisposition to self-directed learning through their natural development as human beings, so it is not surprising that only 40-60 people chose to be active producers on the course and all others (around 1550) chose not to during the 10 weeks that PLENK ran.   I also found the paper by Nielsen (reference by Eduardo Peirano as one of the comments on George's blog post) on lurking on the Web illuminating as it confirmed that the numbers of 'lurkers' in PLENK are no different from the numbers in other online networked engagement. 

The basis of MOOCs has always been four activities: 1. Actively aggregating, 2. Actively relating these aggregated resources to earlier experiences and knowledge, what Stephen Downes calls remixing,  3. Actively repurposing; producing a digital artifact with this mix of thoughts, and 4. An actively sharing stage. 

If it is true that people don't require the producing activity for their learning, we might have to reconsider if it is necessarily to promote it as strongly as it has been done at the start of MOOCsin the past. Of course as some people mentioned in the discussion, if nobody is an active producer, there is not much to base the remixing stage on. It also takes away a lot of the creativity that people showed through the artifacts they produced and that we all discussed, admired and used to develop our own ideas and thoughts.

Ctscho (commenting on Jenny's blog), I agree with you that George was wrong to add negative connotations to the activities of most of the PLENK participants, as of course the ‘lurkers’  have been actively engaged in the course through the other three activities: aggregating, remixing and sharing. And our research so far shows that people were actively engaged in these activities, although the sharing took mostly place outside the PLENK course structure and sometimes after the course had finished because people needed time to think about the high level of resources and information they had to digest during the course. It would not surprise me if George did this to provoke the discussion ;-)

If the majority of PLENKers think that their active participation, without the producing stage, is legitimate (and the empirical evidence that we collected so far clearly points in that direction) it might be that George has to eat his hat ;-), and reconsider his ideas, beliefs and feelings regarding what type of activity is required for learning on a course of this nature.  Current theories of learning show that activity is conductive to learning, but luckily they do not prescribe what type of activity this would have to be!

 Another issue that needs to be considered is if the format of the course, which has not changed much since the initial MOOC CCK08, is really conductive to people actively engaging and producing.  It came up throughout the course that the active producing and engagement in the course might be stimulated by a higher level of ‘strong’ rather than ‘weak’ ties amongst participants. Of course this will always be challenging in a course with this high number of participants (1614). People suggested for instance a ‘buddy’ system, where old-timers would be encouraged by facilitators at the start of the course to form some groups to support new-comers. It was also highlighted that the latter were least likely to have the confidence to produce, and the closer ties might create an atmosphere of trust that would make people feel more at ease than on the wide open open course.  The Second Life group also turned out to be perceived as a place to make people feel comfortable. Of course it cannot be denied that in a course like this there were some strong characters and power-plays between participants (and facilitators) as in any other place where humans congregate that might have influenced engagement and participation.

For the development of our PLE, it is important to get as much feedback from you as possible on this as we have to decide what features we can or should build into our PLE to create the best possible environment to support learning. The surveys are still open and so far 68 people have filled out the lurking one and 28 the active producer one. Comments on other issues are also welcome, in the surveys or on any of the blogs and discussion forums with the #PLENK2010 tag.

Active participation - lurking on PLENK

It is a while that we had the last PLENK2010 session, but quite an interesting debate has flared up about the importance (or not) of active participation in Massive Open Online courses such as #PLENK2010. You might have seen George Siemens' blog post and comments, and the ones that followed by Jenny Mackness and John Mak .To learn more from PLENK participants on this issue, Helene Fournier and I hoped participants would not be quite 'surveyed out' yet and would be willing to fill out one of the two surveys below.

NRC researchers would like to invite Active Participants in PLENK2010 to fill out a survey on their experiences in this Massive Open Online Course. Active participants include learners who actively contributed to discussion forums in the course Moodle, blogs, twitter, social networking sites, and in the sharing and production of artifacts. The Active Participant Survey can be found here http://www.surveymonkey.com/s/JDC58ZM.

NRC researchers would like to invite PLENK2010 Lurkers to fill out a survey on their experiences in this Massive Open Online Course. Lurking is defined in this context as passive attention, silent participation, and/or self-directed learning. There has been a lot of interesting discussion on the topic of lurking on the BlogSphere so we would like to explore the issues a bit more through this survey. The PLENK2010 Lurker survey can be found here http://www.surveymonkey.com/s/JJ8ZXMC.

The research team thanks you in advance for your invaluable contribution to the research! Contact Helene Fournier (helene.fournier@nrc-cnrc.gc.ca) for any further inquiries about the surveys or the research.

Modelling PLE based learning

InWednesday's  #PLENK2010 session Sebastian Fiedler gave us his thoughts and ideas on Personal Learning Environments. He moved the discussion from the technology to the concept and made us think about the personal learning, rather than the learning environment in PLE. His model to analyse personal learning looked like this:

During the discussion following his presentation quite a few questions were asked about this framework: would people move from 1 through a continuum to 5? How would this work?  Howard Johnson in his blog pointed out that PLE based learning is very much embedded in the context in which it takes place. Learners are in a constant flux, working here, playing there, interacting on Twitter, drinking coffee in the local community centre. That is also my problem with this model: it is all about the self, but this self doesn't operate in a vacuum. I would prefer to see a model of learning that encompasses the context and interactions that people engage in. I produced a model of PLE based learning a while back that incorporates the learning context. It has Kolb's learning cycle at its heart, but also shows the process of aggregation of information, relation of the materials to earlier experiences and knowledge, creation of digital artefacts and sharing of these with the wider world and communicate about them with others.

Of course while learners are going about their lives and are involved in activities that make that they learn, their personal development continues and I have found the Perry stages of development, as described in an earlier post, helpful in understanding how this might work.

Feeling uncomfortable with Personal Knowledge Management

Harold Jarche illuminated participants on #PLENK2010 on his ideas of Personal Knowledge Management on Friday. The way he sees it seems to boil down to doing something with information in an effective way: share it and/or action it.  Vlimaka on Cryselina linked some of the readers of the week in an interesting way to what Harold had to say and made the connection between PKM and Knowledge Management in organizations and how the two might compare.

I have some problems with Knowledge Management and even more so with the term Personal Knowledge Management.

1. My first problem is that it gives me the jitters to see that another business term has crept into the social sciences. There have been widespread protests at the use of the word 'capital', to describe the value of our social, human and cultural interactions as human beings in terms such as 'social capital', 'cultural capital' and 'human capital' as it was suggested that in these more is at stake than monetary gain. Do we really have to call the way we organize our personal interactions with others and our creative actions which help us to transform information into knowledge management? Is there not a more inspirational term available? I would say 'enrichment', 'organization' or 'development' would already be better terms, or perhaps 'creation' or even 'control' but perhaps people in PLENK will be able to come up with a better term.

2. To me there also seems to be a tension between the words 'knowledge' and 'management'. In my idea of knowledge there is not much of it that you can manage. It is related to the problems Stephen Downes already saw in 2003 with knowledge management . Knowledge is not the same as information or data, which you can capture and store in systems.

Knowledge in my view is related to the context in which it emerges. Depending on your view of knowledge, it can be constructed, which makes that it is related to the earlier experiences and or knowledge of individuals or others in his or her surroundings. This makes that it would be hard to manage as it would be related to the context in which it is produced over which you don't have control.

It might emerge through a process of immersion, sharing on networks and actions related to information processing, eg. through blogging and receiving from and tweeting information and resources and artifacts on to other people in a connectivist view of knowledge, this makes the management of it even more problematic as knowledge would be embedded in the context in which it emerges, and span a person's mind, the people involved in the interactions and the tools used to carry out the actions and interactions. The only management you would be able to do is to ensure that the conditions to achieve knowledge development are being met, for instance by installing tools on your machine that allow you to  communicate, create artifacts, and push out the artifacts produced onto the Web, but to me these are the least significant in the knowledge creation process.

The only view of knowledge that I can see that would allow for management is the traditional view, where institutions such as universities who produce a high form of knowledge can chop it into bite size chunks, make these into course packages that instructors can pick of the shelf for the students to digest. And even there one would argue in a quality teaching situation, more will happen than the transfer of this managed knowledge; students would be expected to make connections with what they already know and information they find outside the institution, or with information and resources provided by the instructor. And there would be communication to share what is known. Learning outcomes might not quite be what the manager would have intended.

 What then about knowledge management in the workplace? What defines the current knowledge workers job? Here as example a description of 'portfolio' adult educators by Fenwick, that I think is also applicable to other workers today (2003, p.175):

'They are increasingly expected to adapt themselves to conditions of flexible jobs, flexible knowledge and skills and flexible work loctions . . .Within these flexibilised employment arrangements, a career has become an individual's responsibility, a lifelong human resource project crafted through the process of continuous reflexive self-assessment, continuous learning and adaptation and of course self-marketing.'

To me people's career and flexible knowledge once again mean that knowledge in the workplace is again related to the context in which it is generated; the workplace, the knowledge worker him/her self, the professional network. The only instance I can see related to the management of knowledge has to do with the optimization of work by employers, businesses and corporations. What are they to do when a valued worker leaves? How to capture what the worker has learned? Well, I would say they need to allow sufficient time between someone leaving and a new person to be appointed to ensure that work-related knowledge is not lost. There needs to be a process of enculturation and immersion of the new employee in the proximity of the leaving worker, but it seems in the work place of today time is not allowed for this.

So, what do I think of (personal) knowledge management?  No thanks, not for me!