#LAK12 Gamifying SNAPP

In a recent blog post over at David Jennings' Quickened with the Firewater blog David asks: "What would happen if we put learners in charge of analysing their own data about their performance?"

Here's how it could happen with SNAPP, which is a software tool that helps online instructors analyze the communication patterns among students who use the LMS' communication tool:

• the instructor would allow communication to happen using the LMS, perhaps providing some task guidance or parameters to inspire the online discussion.



• The network visualization (like the one at the right, from the SNAPP group) could be shared with the students, who would use the visualization to help them think critically about how communication unfolded and what they individually added to the discussion.



• Learners could then be challenged to change the visualization by changing their own communication patterns as a group, thereby gamifying the system.

Learners would gain insight into their group communication processes and play with different communication roles. This might be especially useful in a management or communication course, but such insights would be valuable to anyone no matter their course of study. But what learners might also gain, in addition to learning about communication principles, is insight into how such systems can be manipulated, and how the systems might also be used to manipulate learners -- both worthy learning goals.

P.S. thank you Shane Dawson, for a fantastic presentation yesterday about SNAPP for the learning analytics massively open online course!

#LAK12 Word Cloud of Data Privacy & Ethics Chat

We had a lively online chat during Erik Duval's presentation on privacy and ethics in learning analytics.
Here are some details that you can see on the word cloud:
--"brother" as in Big...some of us brought up the "Big Brother" theme when the question was asked, "What do you worry about with sharing your data?"
--Lies vs. truth: is it considered lying when you present yourself as other-than-you? Is it your responsibility to present the truth about yourself online?
--transparency is a concern: who can see the data? who can see the models?
--power: who has it in learning analytics? who doesn't? Do teachers, learners, or administrators have power in the system?
--We discussed Google and Target as two corporations who are mining our data for marketing insights.
--medical: How are learning analytics issues similar to issues encountered with personally identifiable medical information?

#LAK12 Do Computer Scientists Do Science?

Dragan Gasevic's presentation about evidence-based semantic web shows that the software engineering field is beginning to adopt the paradigm of evidence-based practice (EBP) which has already been increasingly adopted in medicine, nursing, education, social work, and other human services fields.

In the evidence-based paradigm (and it indeed is a paradigm shift, especially for the medical field that birthed it), the randomized controlled trial is considered to be the study type offering the strongest possible evidence to support a hypothesis. (Note, however, that not every research question lends itself to an RCT. In software engineering, there may be other study designs which are more appropriate.) Systematic reviews and meta-analyses of many randomized controlled trials represent even stronger evidence. As its name suggests, a systematic review requires a systematic and methodical search of the literature in order to present an overall synthesis of results from the highest-quality studies that can be located. A meta-analysis goes several steps farther in that you would take the results of several related studies (all of them RCTs, or cohort studies, or case control studies, etc.) and pool the data, with the effect of creating one large study which can then be analyzed. Both systematic reviews and meta-analyses are important contributions to a field. Although we might not think of them as empirical scientific studies in themselves, they synthesize the entire body of empirical work that has been done on a topic to date. This synthesis is more than the sum of the results of the component studies.

The state of software engineering, as Gasevic and others seem to point out, is that most so-called evidence in the field consists of case studies or even simply expert opinion -- both at the very bottom of the EBP evidence hierarchy. The higher-level empirical studies that have been performed are often with small n, thereby decreasing the power of the studies to detect a statistically significant intervention effect. This is similar to the situation in other fields that have begun to adopt EBP. Gasevic shows that the study design methods, sampling methods, and data collection methods of published papers in software engineering are lacking in quality. If science is the application of rigorous methods to hypothesis testing, then is this a situation wherein computer scientists & engineers aren't practicing much science at all?

#LAK12 Semantic Web: Glimpses of Understanding

What I understand about the semantic web is that it:

-- relies on metadata that codes the metaphysical identity of a piece of data and how it relates to other things or concepts. Is it an image? a person? an idea? An author? This reminds me of bibliographic cataloging in libraries. ("The Semantic Web: An Introduction")

--exists on a very small scale currently ( Tim Berners-Lee's TED talk).

--would allow us to find and visualize relationships between any two bodies of information, whether the information is a person or an image or a body of raw data.

--would allow the implementation of learning analytics on a much more far-reaching scale.

This is what I understand and it isn't much, but what I don't understand is a lot. I really did not understand the "semantic web: an introduction" paper, nor the specifics of Hilary Mason's talk, but I found it fascinating none the less, and was particularly pleased that she used a disease-related example to illustrate Bayesian statistics. I made a connection from that to the concept of evidence-based medicine which I've also been learning about in the past couple of years.

Still need to watch Dragan Gasevic's presentation from last week. Perhaps it will make all things clear.

#LAK12 Knewton Love

Thoughts from my viewing of the Knewton video:

Flashback to 1992: I'm teaching at a private high school in Memphis, Tennessee. It's my first year of teaching at this school AND my first year of teaching high school. With 5 sections of students in two subjects, finding time to simply deal with the paperwork, classroom discipline, and preparing a new lesson plan for each day is a challenge -- let alone individualizing instruction for my students. A parent contacts me about her son. Without being overly accusatory, she tells me that one of the reasons she placed him in this school is that she hoped he would get some specialized instruction, but that's not happening. He's a gifted student, and now he is bored. I feel frustrated because just devising a single lesson plan to reach the average student is challenging enough; there's no way (I felt at the time) I can meet his needs too.

Knewton would have helped. It represents a way to use LA to do what all good teachers should be doing, but many don't have enough time to do:

--group students by learning preference/style, rather than by ability. This allows faster learners to learn more by teaching their peers.

--identify "study buddies" for students based on specific concepts plus learning preferences. Again, this allows students who have a mastery of a concept to learn more by sharing their mastery, plus students who still need to learn a concept are surrounded by more potential teachers. This would allow teachers to implement peer teaching in their classrooms.

I did notice at least one red flag: towards the end of the video, the narrator mentions that one of the benefits of Knewton for publishers is the establishment of a "lifetime relationship with students" allowing them to develop rich data on a student that could not be "shared, mined, or pirated." There's the catch! And raises the question, who really owns this data? The student or the system designer?

#LAK12 Community colleges: the perfect candidates for LA

As illustrated by Vernon Smith's presentation on Rio Salado's implementation of LA, community colleges would serve as the perfect testbed/incubator/adopter of learning analytics. Here's why:

1) Community colleges are bursting at the seams. Taken as a group, they represent a very large population = large "n" for doing analytics.

2) Community college faculty, for the most part, are there to teach, not to do research. There is a pragmatic focus on student success. Therefore, there would probably be more buy-in at the faculty level for LA.

3) You have a larger "n" of "at risk" students. Percentage-wise, the"at-risk" students represent a larger chunk of the community college population since the barriers to entry (such as price and high school performance) are lower than those at a four-year college/university. Therefore, you would probably get more return on investment with implementation of LA due to increased retention of these students.

4) In terms of human development/capacity, the ROI would probably be much greater at the community college level too, as many CC students are first-generation college students. Retaining a greater number of these students would have huge positive implications for society at large. (especially if education helps them see the need for changing the system status quo)

In contrast, I work at a medical school where we have a huge barrier to entry and a population of only about 145 students per cohort. This is a small "n" and I don't believe retention is an issue at all. "Completion" is not our concern. Improved learning is our concern, but in terms of return on investment for American society, I think money spent on LA at the community college level would yield a greater payoff. (unless, that is, the federal government and medical schools start aggressively pursuing a different student population in hopes of solving the huge problem of lack of access to health care of rural and urban Americans...)

#LAK12 UMBC MyActivity vs. Purdue Signals: BEATDOWN

How do the UMBC and Purdue systems compare? I watched the video demo of University of Maryland Baltimore County's CheckMyActivity analytics tool and compared it to John Campbell's presentation on Purdue's Signals, as well as Kimberly Arnold's EDUCAUSE piece on Signals.
Intuitiveness: SIGNALS. The use of color instantly symbolizes student grade status. The traffic light symbol is easy to understand and a powerful message and is really the genius of this system. The UMBC system, on the other hand, leverages the functionality of the course management system but, in so doing, also retains the somewhat IT-centric wording ("hits", "sessions," "generate report" and "get gradebook items") and the non-intuitive structure of the reports. It was difficult for me to quickly ascertain which column on the CheckMyActivity reports were the most important to focus in on for student performance (average hits per user? sessions?) ; so I'm sure that students probably have the same difficulty. Plus, you still must interpret what the numbers mean. You don't have to do that in Signals.

Visual Appeal: SIGNALS. The red/green/yellow colors are attractive and the interface looked more streamlined than UMBC's system . Plus, they have an attractive logo, and that's always a plus.

In-Your-Face quotient: SIGNALS by a landslide. Students have to take action to access the UMBC tool; with Signals, no additional action is necessary past logging in to the CMS, which they will do anyway. This is a limitation of UMBC's system. In fact, 16% of students in one course who responded to a survey said that they had never used UMBC CheckMyActivity at all. The fact that the red, yellow, and green lights are staring students in the face every time they log on to the CMS makes Signals impossible to ignore and therefore, students are probably more likely to follow up on the instructor's remediation suggestions.



Model-Building Quality: UNKNOWN. I would really like to see what data models lie behind each system, but I haven't found that information yet. UPDATE 2/9/2012: I suppose CheckMyActivity does not actually have any models upon which it is built. The students are only seeing the raw data of their log-ins to the course compared to the log-ins of people with grades of A, B, C, etc. Therefore UMBC's CheckMyActivity tool would be an example of "transaction only" level of sophistication rather than Purdue Signals' "predictive modeling" level of sophistication.

#LAK12 What teaching/learning and technical skillsets does LA require?

In today's LAK12 MOOC presentation, John Campbell of Purdue's Signals Project stated that there are two important skills domains in learning analytics: technical AND teaching/learning. In fact, both must be present in order to effectively design tools in learning analytics. He stated that both of these sets are very difficult to find in one person, and at Purdue, they have had the most luck in looking for people with the teaching & learning skillset and THEN training them for the technical side.

So here's the question: For success in learning analytics, what teaching & learning skills would you look for? What technical skills?

#LAK12 Recap of "Purdue Signals" presentation by John Campbell

Great presentation by John Campbell of Purdue's Signals Project! If you didn't participate live, you'll want to watch the recording. Highlights for me were:

1) Hearing from a creator about the nuts and bolts of design, implementation, and ongoing maintenance and improvement of an analytics tool that is currently used by students in 80 introductory courses, with the goal of use by 20,000 unique student users.

2) Students have lodged no complaints about privacy issues and are positive about its use; participating faculty are so positive that it works that they have begun refusing to participate in controlled trials of Signals.

3) Privacy considerations include: faculty can only view data for their own students. They have begun IRB approval was obtained for both testing (easy IRB process) and implementation (more time-consuming IRB process) phases. They don't let students "opt out..." I guess the principle is that

4) Customization of messages to students and the platform on which they're delivered is important...email doesn't work; slight customization of messages does work for grabbing student attention. (I've found that Facebook works for getting instantaneous responses from students, especially during class time when they are ALL monitoring it!!)

5) Excellent "practical suggestions" which he ended with...such as, "think about the theoretical basis of your analytics project." The Purdue Signals project is based on Tinto's input/environment/output model (actually I think it's Astin's model as the link shows...)

6) AND last but not least...that it is very difficult to find people with the teaching/learning AND technical skillsets that are required. They are finding that in-house training works for the technical part.

#LAK12 Is LA strictly behaviorist?

I thank Bianka Hadju for calling my attention to this statement about behaviorism in Siemens' and Long's Penetrating the Fog piece:

Since we risk a return to behaviorism as a learning theory if we confine analytics to behavioral data, how can we account for more than behavioral data?

Behaviorism is a psychological and educational theory which, as alluded to in the statement, is no longer in favor in most educational circles. The primary criticism of behaviorism in plain terms is that this educational theory would simply explain differences between learners by observing and measuring their outward behaviors. Behaviorism does not account at all for inner mental states, thoughts, feelings, cognition, meanings ascribed to events by learners, etc. If there is no difference in behavior, then there is no difference between the learners. A fascinating explanation of the theory and it's criticisms is available at the online Stanford Encyclopedia of Philosophy.

This ties back to Ryan S.J.d. Baker's presentation last week about model building in educational data mining. He described an online reading tutor system which was tested with U.S. students and students in the Phillipines. Baker used the system to build and test his analytic model of "gaming the system." He could tell who was gaming the system by observing behaviors such as the click patterns, order of clicks, wait time, etc. What was even more interesting to me, though, was his finding that although both groups of students gamed the system, the meanings behind this observable action --their feelings about the activity, their attitudes toward the use of the system -- were vastly different in each cultural group. The students' feelings and attitudes, which are hugely important to the educational process, cannot be measured by clicks at all. The two groups displayed the same behavior, but their reasons for doing so were very different.

Will we focus on measurable behaviors without considering the other important changes we wish to inculcate as educators, such as attitudinal and affective changes?

Looking forward to hearing about Purdue's "Signals" project today!