Global Anthological Journal of Action Learning

Volume 3. Issue 1. 1999

4th ICIT: Hong Kong April 7/9 1999

QUALITY METRICS FOR LEARNING PROCESSES

Mollie Ainslie and Gordon Prestoungrange

Sales Development / Quality Manager & International President

International Management Centres with Oxford Brookes University, UK

The implementation of ISO 9002 protocols within an Internet resourced global action learning Business School led immediately to a demand that a similar approach be applied to the learning processes. Key quality drivers were identified as the programme structure and assignments required; the functioning of the learning Sets; the supportiveness of the workplace; and the subject matter contributions of Faculty and library services. In all of these areas preliminary triggers and measures have been isolated and improvements both within Sets and for generalisable knowledge are reported.

This paper is a sequel to Designing a Quality Action Learning Process for Managers by the authors presented at the 2^nd ICIT at Luton Hoo, England (Wills and Ainslie, 1997). We analysed there the reception given to the implementation of ISO 9002 protocols in the Registry systems and how they had provided reassurance as much as quality assurance for our academic partners. The reassurance was that our espoused tutorial model, action learning, was susceptible to monitoring without necessarily undermining the learning dynamics of the Set of managers involved. They were able to continue determining which way they could learn best from one another, from workplace colleagues and from tutors rather than having a normative structure imposed upon them.

Faculty members, whilst applauding the initiative as constructive, were however immediately resentful that so much energy and resources had been focused on the administrative systems rather than the quality of the learning processes within the Set, and as between Set members called Associates and their colleagues at work. Rather than taking such comments as criticism the authors resolved to take them as an invitation to seek to understand some of the metrics that could be employed to trigger continuous improvement in the learning processes.

Assignments and Programme Structures

Clearly, the structures of the programmes offered e.g. frequency of meeting/style of Set behaviours/role plays by tutors as prescribed in the original design of, say, an MBA or DBA programme, have a critical impact on the learning processes. They demand certain activities that can then be managed for better or for worse. The sheer volume of words required to be penned for assignments, and the number and focus of assignments again greatly condition what can be accomplished. These requirements can if not carefully managed dominate the Set interactions not just as a finale as assignments go in and as feedback is provided, but as a fascination or dread throughout the exploration of any topic.

As we examined the structures and assignments which had held constant for over a decade on MBA and DBA programmes it became apparent that the originators understanding and rationale for their introduction was lost on most contemporary Faculty and most Associates. The sharing of these elements with every new cadre of Associates seemed to be a valuable way of increasing learning in particular through an understanding of the allocation of marks within assignments and the way in which Associates are perceived to have performed well or less well on each one. Indeed, the evaluation at this stage has already triggered a study of comparative marking around the globe and the reallocation of marks to reflect, in particular, the implementation of action plans. We rediscovered the decision made a decade earlier that because the implementation of projects within the time-scale of the programme of studies was seldom feasible, Associates were discussing how they might rather than how they did implement. Little or no learning was being captured and generalised about the challenges or the real politik of implementation of change, although considerable research had been accomplished on the cost benefit of the changes that did make their way through the enterprises concerned (Wills and Oliver 1996). Attempts to capture ‘what happened next’ using a device known as A+, where Graduates were asked one year after completion to share the outcomes were not a success and it proved difficult to motivate a representative sample to create such evaluations spontaneously. The two obvious ways forward in this respect seemed to be either (i) to introduce action plans based on research earlier in the programmes so that implementation could take place and be evaluated prior to the completion of the programme; or (ii) to undertake fieldwork research in depth with a representative sample of Graduates.

Learning Processes per se

The concern with learning about and from implementation outwith the formal structure, and how the design of the programme could be used to bring it within, were surprising aspects of the search for useful metrics of the processes we used. The more predictable had been clearly articulated in the late 1980s as the composition of the Set and its consequent interactions; the workplace context from which the managers came and how it responded to them as they went about their workplace learning assignments; and the way Faculty members and library resources complemented and empowered Associates. The questionnaires administered for nearly 20 years at the Graduation Congregations have consistently reported that Associates meeting together from around the world, who have followed the imposed common structures and assignments, are literally amazed at how much of a common language they have gained and a common perspective on the processes of management. But more than that, they have all united behind the view that the interaction with fellow Set Associates was the most powerful learning trigger - both encouraging and sharing experiences in broadly similarly challenged workplaces.

A very considerable majority expressed disappointment with their workplace colleagues as supporters of their learning. Whether their expectations, because of how their fellow Set members who were similarly challenged had responded, were frankly unrealistic we do not yet know. But we have come to the realisation that a measure of how well the workplace does support an Associate is an urgent need. Once we can identify the elements of perceived supportiveness we can empower the Associates to manage them more effectively. Strategies have already begun to emerge on an ad hoc but not on a well researched basis.

Faculty members and library resources were seemingly taken for granted by Associates within the Sets. Provided they did not seek to dominate with traditional teaching models, expectations and demands were not high. They were treated more as equals in the process and weaknesses were tolerated and compensated for rather than derided. Whilst this clearly gives Faculty members an enjoyable and relaxed learning situation in the Set, it implied that metrics needed to be identified to bring out superior performances. And these are of course not meant to be about end of programme popularity assessments, but ongoing feedback which gives a fillip to superior contributions.

Interactions amongst the Learning Set

It goes without saying, and notwithstanding the opportunities for Web based discussion forums which we use, that attendance at Set meetings is a prerequisite for much desirable interaction. Yet we have eschewed attendance registers and been understanding and tolerant as have the fellow Set members of absences on business or because of family exigencies. And again, because we are seldom overwhelmed with applicants for any given Set, we have infrequently been in a position to attempt to select and catalyse a Set based on any prior model of how they can be expected to interact powerfully. We have and do attempt this for sub-Sets within a programme, getting the Set themselves to understand how and why, as a critical element of learning for their own workplace. To achieve this we use team working preferences (Margerison and McCann, 1985) and most specifically shared learning styles (Honey and Mumford, 1986/Mumford, 1996) as developed by Faculty members.

The role of the Faculty member in knowing how to facilitate good interactions within the Set, and to share with the managers themselves how s/he is achieving that goal should and frequently does outstrip the ‘natural’ interactions that the make up of the Set will engender. And we believe that is as it should be. Management in the workplace is seldom about selecting and working with a perfect team; it is most frequently about making the most of the resources to hand. So too is it within a learning Set.

To assist the Associates come to a fuller understanding of how they learn from and with one an other, in a Set context enriched and facilitated by the process skills, is accordingly a valuable managerial learning outcome. It is one of the fields where not inconsiderable ad hoc research has been conducted (Thomas, 1993) and where assignments and Set meetings have been engineered to highlight the issues. The assignments gain significant credit points towards the qualifications awarded. Following Thomas’ thesis however, that it must be a continuous processes or there is a danger a shortfall will only be fully revealed at the end of the programme, the structured course materials and Faculty guidelines called routinely for self evaluation by the Set of how it is functioning. What was still missing was any serious attempt to measure then capture and compare the outcomes of that self evaluation so that generalisations could be offered.

Cyril Atkinson (1998), IMC’s Chair of the Global School of Quality Management, then proposed and implemented a quantified pattern of measures of Session Effectiveness (derived from Schein’s work on Process Consulting) and of Set Health Checks (after Taylor’s work at Hoechst Marion Roussell on Facilitation of Self Supervised Work Teams). The latter specifically enables the Set to create and evaluate its own Morale Chart over its life together, to understand what is falling short and to resolve to act to improve and measure again later. It is too early to say how powerful such an approach will eventually be but it has gained immediate acceptance amongst Faculty as both valid and valuable, and constitutes a real breakthrough.

We began this section by commenting on how valuable face to face interaction is. The arrival of Internet discussion forums within IMC since 1994 has however led to heated debate about the relative cost benefits of face to face and virtual discourse. Not surprisingly the emergent solution is a mixture of the two, but our perception of what can be accomplished via Internet discussion forums is still growing daily - even though it is obscured frequently by Associate and Faculty difficulties with the technologies. Working with Marriotts International Hotel Group we have established so called Golden Rules for effective/quality action learning within the CyberSets which have members from all five continents working on team projects. And we have crafted assignments such as the Evaluative Assessment of Virtual Learning which requires Marriotts’ Associates consciously to assess how the new medium is working and its relativity with face to face workshop sessions which have also been held.

At the completion of the first tranche of Marriott programmes we shall be creating through the Director: Published Learning (see below) their distilled outcomes for the future balanced use of CyberSets and face to face high cost meetings.

Learning Support from the Workplace

We have identified already that the reactions and support of workplace colleagues to the action learning and action research work Associates were undertaking has been consistently disappointing in their eyes. Research showed that that was seldom the perception of the superiors or colleagues concerned however. They saw what had happened, and what the Associate had achieved as well worth while and normally meritorious. As such we have long concluded that there could never be enough support from colleagues at work for an Associate undertaking the programmes which were always done in post rather than full time to ensure connectivities with real work are not lost.

We resolved recently with British Airports Authority (BAA plc) in the UK, which placed 50 Associates on three joint IMC/University of Surrey postgraduate programmes simultaneously, that we should take a different perspective on workplace support. We had exhaustively briefed colleagues and involved then via Steering Groups including project selection, and as Counterpart Tutors to the IMC Faculty to no apparent incremental improved effect. The different perspective was to take the outputs that were created by Associates, officially distil their crucial /best elements and use the Faculty team as advocates and disseminators of such distilled outcomes around the enterprise.

The endeavour is called Project Harvest and a Director: Published Learning, as she is called, has been appointed full time to ensure it is well carried forward. Its current manifestation gives high prominence to the Associates’ outcomes and assertively requires attention from workplace colleagues. The outcomes are also ‘published’ so considerable pride can be taken in them. This does not of course replace the solo efforts of individual Associates to publicise and win support for their own propositions, but it adds status to them.

It must be said that it did not begin thus. Initially Project Harvest was conceived by our senior colleague Richard Dealtry as a database of all the assignments produced, that was key-worded and searchable. But in successive debates at IMC’s bi-annual face to face global meetings it was enhanced into a proactive use of the outcomes. The data base is still there but the emphasis is on the creation of distilled crucial/ best issues for determined communication. Faculty working with in-company Sets are required not simply to mark assignments but to draft the initial distillation, present it to the whole Set on the Internet as part of the feedback, and thereby invite all Set members to polish it prior to its ‘publication’ around the enterprise.

The Director: Published Learning’s role is to monitor all this activity and as may be possible to ensure that ‘publication’ does whenever possible also reach open dissemination in refereed journals. IMC also maintains a suite of three of its own house journals under the imprint of Internet Free Press that carries all the outputs including literature reviews as first instance publications for other members of IMC globally, with the refereeing process seen as the responsibility of the Faculty members concerned.

This lateral thinking looks well placed to trigger measures of effective enterprise dissemination and of refereed quality determination for the generalised body of knowledge. In this way, we believe ongoing workplace supportiveness will be enhanced because the dissemination is continuous throughout the programme.

To conclude on this element of quality in the learning process it is important not to underestimate just how much support has always and continues to come from colleagues at work, even though it seems many Associate’s appetites can never be sated. Since each assignment must necessarily be about the workplace and since the assignments necessarily take the Associate into other realms of activity s/he does not normally engage with, they cannot be accomplished without collaboration. Optimising that quite clearly requires good interpersonal skills as well as the insight to play a non zero sum game strategy. Quantitative measures such as those introduced by Atkinson on Set Health Checks are clearly worthy of early extension to this external collaborative activity and are now being incorporated into the assignment guidelines and marks allocated for the way in which workplace colleagues have been motivated to contribute and assist.

Finally, it should be noted that the major impact of the above comments has been for in-company Sets rather than Open Sets. Associates on the latter do also express disappointment at the lack of contribution by their own workplace colleagues, but seem to be more understanding thereof. As with in-company Associates their response in line with Rosenthal’s findings on teams (1997) is to strengthen the within Set solidarity and perceived supportiveness.

Faculty and Library Resources

We have considered the significance of Faculty members in facilitating self learning Sets already. Here we shall look at the way in which Faculty are looked upon as repositories of subject area wisdom and the same for the ‘library’ function broadly defined as books/articles/grey literature/coursewares and guidance notes. But initially it will be useful also to comment on the authority role of the Faculty member in adult education.

In action learning, where the Associate is deliberately encouraged to define the curriculum to be studied, the Faculty member clearly exercises a reduced but ultimately quasi absolute pattern of authority. Each Associate will submit assignments to be marked, and the Faculty member is responsible for allocating the marks. Amongst adult learners this we have discerned can only be achieved effectively if the Associate accepts the judgements implied and that the quality of feedback given is timely and actionable to improve subsequent performance. To ensure this CyberSets are now inviting Doctoral Associates (one of the most adult and educationally experienced groups one could expect to work with outside ‘own’ staff development groups), who have received feedback, to then give their own feedback to Faculty on the extent to which it was timely and actionable Ö and indeed perceived as a fair judgement. To everyone’s delight and amazement a very high level of debate took place from all sides, and the CyberSet evolved its own protocols for the process including an assertive attitude towards service levels guaranteed and met.

The use of Faculty members as ‘subject area experts’ is most strongly felt in doctoral work, where the Associate is surely seeking to advance knowledge rather than find a helpful way around what is already known in whole or in part. We have developed no coherent measure for this element of quality on our programmes since it is seldom possible to separate it out from the learning approach that suggests Associates should find out for themselves by searching the literature. Faculty members as subject area experts are expected rather to know how to help others find the specifics of a particular industry rather than have ex cathedra knowledge to pass on. In this respect our embedded key-word searches from ANBAR Management Intelligence and the Virtual Library support provided from EMERALD and the British Library are incomparable (Sandelands, 1998).

Ironically however these seemingly satisfactory outcomes spurred further action and development as Faculty development Sets were initiated in the wake of IMC’s joint ventures first with the University of Surrey and then with Oxford Brookes University.

IMC has since its inception insisted on Faculty induction workshops to ensure the specifics of action learning are understood and the Associates not left stranded. However, in a serious endeavour to share the nature of the university alliances that were being forged, and how they paradoxically had the potential to enrich but also the acute danger to dilute action learning, we resolved to convene Faculty development workshops on a continuous basis. Effectively Faculty were formed into ‘permanent’ self learning Sets not driven by programmes they were formally delivering as a Faculty team but for the sake of learning together. Not surprisingly the provision for them to take IMC qualifications as a focus was also followed at Masters, doctoral and Distinguished Professorial levels. Earlier attempts to focus simply on ‘delivery effectiveness’ had not been sustainable.

The Faculty Sets met extensively face to face and explored issues such as improved coursewares and learning resources generally, and most recently using a CyberSet process globally to enhance research supervisory skills amongst senior Faculty members. This latter proposition has been most especially vital for the growing demands at doctoral level in the Orient, and will surely be shortly mirrored in the need for Virtual Tutors and Set Advisers for CyberSets - most especially via our colleagues at the Canadian School of Management which provides global SYSOPS for CyberSets. It grew from Ainslie’s extension (1998) of her pioneering ISO 9002 activities on just such a Faculty development Set.

No matter how well any given Set or CyberSet relates with its given tutorial team - and we have empowered Sets to change tutors if they find them unsatisfactory after deliberate mutual attempts at improvement - there will always be a feeling that perhaps out there somewhere is a Faculty member who could be better matched. To enable this the Mk IV Registry and Faculty Database maintained by global EDI have been made partly searchable via Internet access. Associates and Faculty can search to see what previous dissertations/ theses have been written, what Faculty members CVs contain, what current research activities are in process, and via the Director: Published Learning what grey literature is available on outcomes. These all use the same thesaurus as ANBAR Management Intelligence for ease of access. Finally, IMC sponsors via MCB University Press, ANBAR’s publishers, and thereafter provides global access to, numerous subject area and regional management Forums and to the Best Practice Club of enterprises willing to share and compare.

There is little room to doubt that the quantum leap in availability of knowledge, including grey literature, via the Internet from pioneering investors such as MCB University Press, has transformed even democratised the processes of learning. The quality of this resourcing is so clearly way beyond most of our previous experience that any cogent critique is almost non-existent. Whilst this will not continue for too long, it does enable attention to be focussed more on the other elements of the learning dynamics discussed earlier.

Scope of course remains for continuous improvement, but what is most encouraging is the manner in which the search for and deployment of metrics within the learning processes has been welcomed by Faculty and Associates alike and applauded by sponsoring employers.

References

1. Ainslie, M., (1998) "Dynamic Quality Assurance in International Management Centres", Quality Management Program UK7, IMC Buckingham UK

2. Atkinson, C., (1998) "Set Dynamics: The Creation of a Learning Environment", 13^th Annual Professional Congress of IMC, Cape Town, April, IMC Buckingham UK

3. Honey, P. and Mumford, A., (1986) "Using Your Learning Styles", Honey, Maidenhead; also Mumford, A., (1996) "Effective Learners in Action Learning Sets", Employee Counselling Today, Vol 8 No 6

4. Margersion, C. and McCann, D., (1985) "How to Lead a Winning Team", MCB University Press

5. Rosenthal, E., (1997) "Social networks and the team performance", Team Performance Management, Vol 3 No 4

6. Thomas, J., (1991) "How IMC Helps its Associates Learn How To Learn", MPhil thesis, IMC Buckingham UK

7. Sandelands, E., (1998) "Creating an online library to support a virtual learning community", Internet Research, Vol 8 No 1; reprinted as Chapter 16 in "The Virtual University", ed. Teare, R. et al, (1998) Cassell

8. Wills, G. and Ainslie, M., (1997) "Designing a Quality Action Learning Process for Managers", 2^nd ICIT @ Luton Hoo, England; reprinted in Journal of Workplace Learning, Vol 9 No 3; further reprinted as Chapter 10 in "The Knowledge Game", ed. Wills, G., (1998) Cassell

9. Wills, G. and Oliver, C. (1996) "ROI in Management Development", Management Development Review, Vol 9 No 1; reprinted as Chapter 10 in The Knowledge Game op cit; further reprinted as Chapter 30 in "Action Learning at Work", ed. Mumford, A., (1997) Gower Press