Change Management
|
Document filename: |
TTI Documentation-Change Management.doc |
||
|
Version: |
000.01 |
Issue date: |
21-Mar-25 |
|
Document custodian: |
Phone: |
04 05 790 444 |
|
|
Custodian e-mail: |
|
||
The material in this publication has been prepared by, and for the guidance and use of employees of the Teseract Technologies International only. These contents are not to be made public without prior authorisation.
Document control
Copies of this document and any subsequent revisions shall be distributed electronically.
With the exception of a single, signed hard copy maintained by the Document Custodian, hard copies of this document are deemed uncontrolled.
Approval
|
|
|
|
|
|
|
1 |
Managing Director Marc Narder |
|
Date |
|
Table of Contents
1.1.1 Understanding Change at an Individual Level
1.1.2 Using ADKAR with Traditional Change Management Activities
1.2.1 The Business Dimension of Change
1.2.2 The People Dimension of Change
1.3 THE ADKAR MODEL IN A PERSONAL CHANGE
1.4 THE ADKAR MODEL IN AN ORGANIZATION
1.4.7 Applying the ADKAR Assessment Results
1 Change Management
The Prosci® ADKAR® Model is a goal-oriented change management model to guide individual and organizational change.
ADKAR is an acronym that represents the five milestones an individual must achieve for change to be successful: awareness, desire, knowledge, ability, reinforcement
When applied to organizational change, this model allows leaders and change management teams to focus their activities on what will drive individual change and therefore achieve organizational results. ADKAR provides clear goals and outcomes for change management activities. It also provides a simple, easy-to-use framework for everyone in the organization to think about change. Employees, managers and senior leaders alike can all use ADKAR to describe and discuss change together.
1.1 WHY USE THE ADKAR MODEL
Change is often a complex and difficult process, and - what's more - it is inevitable. Managing change on the personal and organizational level requires new thinking, new models for change and new frameworks and tools to enable the smooth implementation of the desired change. ADKAR can be applied to a wide variety of changes to drive change success.
1.1.1 Understanding Change at an Individual Level
Change happens at the individual level; in order for a group or organization to change, all the individuals within that group must change. The best project management, vision or solution will not result in successful change. The secret to successful change is rooted in something much simpler: how to facilitate change with one person.
To affect change in our organizations, businesses and communities, we must first understand how to affect individual change. Oftentimes helping an individual change can be ambiguous, and ADKAR provides direction and structure.
1.1.2 Using ADKAR with Traditional Change Management Activities
ADKAR outlines the individual’s successful journey through change. Each step of the model also naturally fits into the typical activities associated with change management.
For example:
Awareness of the business reasons for change. Awareness is the goal/outcome of early communications related to an organizational change
Desire to engage and participate in the change. Desire is the goal/outcome of sponsorship and resistance management
Knowledge about how to change. Knowledge is the goal/outcome of training and coaching
Ability to realize or implement the change at the required performance level. Ability is the goal/outcome of additional coaching, practice and time
Reinforcement to ensure change sticks. Reinforcement is the goal/outcome of adoption measurement, corrective action and recognition of successful change
The goals and outcomes defined by ADKAR are sequential and cumulative, they must be achieved in order for effective and sustainable change to take place.
1.2 CHANGE DIMENSIONS
To use the ADKAR model effectively, it is important to understand all of the factors at play during a change initiative and their effect on change success. Change happens on two dimensions: there is the business side of change and the people side of change. Successful change is a result of both dimensions of change maturing simultaneously (see below).
1.2.1 The Business Dimension of Change
Listed below are the standard business elements of a typical change project. Most managers will feel comfortable managing these phases:
· Identify a business need or opportunity
· Define the project (scope and objectives)
· Design the business solution (new processes, systems and organizational structure)
· Develop the new processes and systems
· Implement the solution into the organization
These are the tangible, concrete aspects of projects and are usually the default steps when implementing a new solution. Much less frequently, however, are managers comfortable with the other side of the change: the people side.
1.2.2 The People Dimension of Change
The most commonly cited reason for project failure is problems with the people side of change.
"Effective change management with employees" is consistently listed as one of the top five project success factors in Prosci's benchmarking studies. To effectively manage the people side of change, a plan must be developed that includes similarly tangible and concrete steps to achieve the five key goals of ADKAR in the people who are required to change as a result of the project. This cannot be left to chance or assumed it will happen naturally.
1.3 THE ADKAR MODEL IN A PERSONAL CHANGE
To help build a clearer understanding of the model and how to apply it, think about a change you want to make in your personal life. A good example is adding a regular exercise regimen; a change many people attempt but struggle to sustain over time. Now let’s apply the ADKAR model:
Awareness: are you aware of the need to exercise? Articles or TV reports that describe the health benefits of regular exercise may build awareness.
Desire: do you have the personal motivation to start exercising? Awareness will not be enough to make the change. You will need to make a personal decision to engage in this change based on your own unique motivations.
Knowledge: do you know how to effectively and safely exercise? To gain knowledge, you might hire a personal trainer, attend an exercise class or order a workout video. In order to effectively change, you need to know how.
Ability: can you put your knowledge into practice? Knowing how to do something and being able to it are very different. You may need to rearrange other commitments to make time for new behaviours, and you might consider working one-on-one with a coach or personal trainer to develop your new skills.
Reinforcement: do you have reinforcements in place to prevent you from reverting to your old habits? Perhaps you have a reward system for yourself when you hit certain exercise milestones. Or you might have a workout buddy who holds you accountable for showing up to the gym.
Using exercise as our example, it is easy to see how change occurs on a personal level. Now let’s consider how this framework applies to employees during a change process.
1.4 THE ADKAR MODEL IN AN ORGANIZATION
The ADKAR model helps us understand an individual’s needs during a change at work and directs what kind of support we can provide to help them successfully transition. Let’s apply ADKAR to the implementation of a new software tool:
1.4.1 1. Awareness
Is your employee aware of the need for change? If the change is implemented and the employee is not aware that any changes are needed, their reaction might be: “This is a waste of time, it was fine before.” Awareness of the business or organizational need for the change is critical. Awareness may include explaining to the employee that the old software will no longer be supported by the vendor, and that new software is necessary to meet customer needs and improve efficiency. Organizational awareness messages are most effective when delivered from the most senior leaders in the organization. Based on this awareness, the reaction will likely be very different: “How soon will this happen and how will this impact me?”
1.4.2 2. Desire
Does your employee have the desire to participate in the change? If an employee has no desire to change, you may hear: “I’m not interested in changing. What’s in it for me?” In this case, the resistant employee’s direct manager or supervisor is in the best position to help. They are closest to employees and understand their day-to-day work best. Through one-on-one conversations, managers can uncover their employee’s personal reasons for resisting and can remove any barriers to the employee buying in to the change. The manager can also help to create desire by translating the change into meaningful terms and helping to answer “What’s in it for me?” While the manager plays a key role here, ultimately the employee must make a personal decision to participate in this change based on their own unique motivations.
1.4.3 3. Knowledge
Does your employee have the knowledge to make the change? In order to effectively change, you need to know how. Knowledge-building should only be provided after the milestones of awareness and desire have been achieved. If training is provided before this, employees will not connect the training to the change and will not engage in knowledge-building. To make the most of a training investment, also ensure that training is specific to the employee’s role in the change.
1.4.4 4. Ability
Can your employee put their knowledge into practice? Knowing how to perform in the future changed state and having the ability to actually perform in the future changed state are very different. If an employee has knowledge but not ability, you might hear: "I’m not getting these new steps right" or "I get there, but it takes me twice as long." To bridge the knowledge to ability gap, employees benefit from hands-on coaching and practice in an environment where they can make mistakes and ask questions. To realize a change, employees also need time. When ability is achieved, the change takes place, and you will see the new demonstrated behaviours.
1.4.5 5. Reinforcement
Do you have reinforcements in place to prevent your employee from reverting to old habits? When reinforcement is not in place, employees may use work-arounds or rely on their old spreadsheets instead of the new system. You may hear things like: "The new way takes too long; I’m going to keep doing it my way" or "I keep forgetting to include the new department." The human brain is wired for habit, and physiologically we are programed to revert to old habits. We must have reinforcements in place to sustain the change. Monitor whether the change is being sustained or not, and where the change has taken hold, celebrate and recognize it. Positive recognition is a great way to reward employees for making the change and to demonstrate that participation is important. If some employees are reverting to old processes or habits, check to see if they need more training or coaching and reinforce that they are expected to continue working in the new way.
1.4.6 ADKAR Exercise
This exercise will help to separate and clarify the key elements of the ADKAR model, in a real life or work setting.
Identify a friend, family member, work associate or employee, who despite your best efforts to support them through a change, is not having success.
Answer the questions below with this person in mind, assigning a score for each question. For a printed version of the table below please download the ADKAR eBook.
1.4.7 Applying the ADKAR Assessment Results
Identify the first area that scored 3 or below. This is your “barrier point” and what needs to be addressed first. By addressing the first area with a low score, you will positively impact all the goals that follow.
1.4.8 Actionable Steps
If awareness is needed: discuss and explore the reasons and benefits for this change. Discuss the risks of not changing and why the change needs to happen now.
If more desire is needed: to move this person forward you must understand and address their inherent desire to change (which may stem from negative or positive consequences). These motivating factors have to be great enough to overcome the individual’s personal threshold to resisting the change.
If more knowledge is needed: avoid dwelling on reasons for change and motivating factors, as this is unnecessary and could be discouraging. Focus now on education and training for the skills and behaviours necessary to move forward.
If more ability is needed: first, time is needed to develop new abilities and behaviours, and this person simply may need more time to develop new skills with proficiency. Second, ongoing coaching and support could be required - consider outside intervention, continued support and mentoring.
If more reinforcement is needed: investigate if the necessary elements are present to keep the person from reverting to old behaviours. Address the incentives or consequences for not continuing to act in the new way and re-visit the knowledge and ability milestones. It may be that more training and education is needed as processes develop and evolve.
1.5 TO CONCLUDE
The ADKAR Model is an essential tool for leaders and change professionals. It is effective, easy to grasp and can be applied in a wide variety of organizational settings.
Using the ADKAR model will help you to plan effectively for a new change and also help diagnose where a current change is failing, so that you can take corrective action. Each step of the model outlines the individual’s successful journey through change and provides an outcome orientation for your change management activities. For example, you will no longer develop a communications plan for the sake of having a communications plan; now you will develop a communications plan to specifically drive awareness of the need for the change. If you are ready to change, either personally or professionally, this results-oriented approach will increase your change success.
For a more in-depth study of the ADKAR Model, we highly recommend the ADKAR eBook below, where all examples, applications and explanations are explored in more detail. The ADKAR exercise and table are also included.
104 Laws of Project Management
- "A badly planned project will take three times longer than expected - a well planned project only twice as long as expected."
- "A change freeze is like the abominable snowman: it is a myth and would anyway melt when heat is applied."
- "A little risk management saves a lot of fan cleaning."
- "A minute saved at the start is just as effective as one saved at the end."
- "A problem shared is a buck passed."
- "A project ain't over until the fat cheque is cashed."
- "A project gets a year late one day at a time."
- "A project is one small step for the project sponsor, one giant leap for the project manager."
- "A two year project will take three years, a three year project will never finish "
- "A user is somebody who tells you what they want the day you give them what they asked for."
- "A user will tell you anything you ask about, but nothing more."
- "A verbal contract isn't worth the paper it's written on."
- "Activity is not achievement."
- "All project managers face problems on Monday mornings - good project managers are working on next Monday's problems."
- "Any project can be estimated accurately (once it's completed)."
- "Anything that can be changed will be changed until there is no time left to change anything."
- "At the heart of every large project is a small project trying to get out."
- "Estimators do it in groups - bottom up and top down."
- "Everyone asks for a strong project manager - when they get him they don't want him."
- "Fast - cheap - good: you can have any two."
- "Feather and down are padding - changes and contingencies will be real events."
- "Finely chopped cabbage in mayonnaise - Coleslaw."
- "For a project manager overruns are as certain as death and taxes."
- "Furious activity does not necessarily equate to progress and is no substitute for understanding."
- "Good control reveals problems early - which only means you'll have longer to worry about them."
- "Good estimators aren't modest: if it's huge they say so."
- "Good project management is not so much knowing what to do and when, as knowing what excuses to give and when."
- "Good project managers admit mistakes: that's why you so rarely meet a good project manager."
- "Good project managers know when not to manage a project."
- "I know that you believe that you understand what you think I said but I am not sure you realise that what you heard is not what I meant."
- "If an IT project works the first time, it is wrong."
- "If at first you don't succeed, remove all evidence you ever tried."
- "If everything is going exactly to plan, something somewhere is going massively wrong."
- "If it can't possibly go wrong, it will - O'Malley's corollary to Murphy's law."
- "If it can go wrong it will - Murphy's law."
- "If it happens once it's ignorance, if it happens twice it's neglect, if it happens three times it's policy."
- "If it looks like a duck, walks like a duck and quacks like a duck, it probably is a duck."
- "If it wasn't for the 'last minute', nothing would get done."
- "If project content is allowed to change freely the rate of change will exceed the rate of progress."
- "If there is a 50% chance of something going wrong then 9 times out of 10 it will."
- "If there is anything to do, do it! "
- "If there were no problem people there'd be no need for people who solve problems."
- "If you're 6 months late on a milestone due next week but really believe you can make it, you're a project manager."
- "If you can interpret project status data in several different ways, only the most painful interpretation will be correct."
- "If you can keep your head while all about you are losing theirs, you haven't understood the plan."
- "If you don't attack the risks, the risks will attack you."
- "If you don't know how to do a task, start it, then ten people who know less than you will tell you how to do it."
- "If you don't plan, it doesn't work. If you do plan, it doesn't work either. Why plan!"
- "If you don't stand for something, you'll fall for anything."
- "If you don’t know where you’re going, any road will take you there."
- "If you fail to plan you are planning to fail."
- "If you have time to do it over again, you'll never get away with doing it right the first time."
- "It's not the hours that count, it's what you do in those hours."
- "It takes one woman nine months to have a baby. It cannot be done in one month by impregnating nine women (although it is more fun trying)."
- "It will go wrong in the worst possible way - Sod's law."
- "Managing IT people is like herding cats."
- "Metrics are learned men's excuses."
- "Murphy, O'Malley, Sod and Parkinson are alive and well - and working on your project."
- "Never underestimate the ability of senior management to buy a bad idea and fail to buy a good idea."
- "No plan ever survived contact with the enemy."
- "No project has ever finished on time, within budget, to requirement - yours won't be the first to."
- "Nothing is impossible for the person who doesn't have to do it."
- "Of several possible interpretations of a communication, the least convenient is the correct one."
- "Overtime is a figment of the naïve project manager's imagination."
- "People under pressure do not think faster."
- "Planning is an unnatural process, doing something is much more fun."
- "Planning without action is futile, action without planning is fatal."
- "Powerful project managers don't solve problems, they get rid of them."
- "Projects happen in two ways: a) Planned and then executed or b) Executed, stopped, planned and then executed."
- "Quantitative project management is for predicting cost and schedule overruns well in advance."
- "Some projects finish on time in spite of project management best practices."
- "Some things that don't count are counted, many things that count aren't counted."
- "The bitterness of poor quality last long after the sweetness of making a date is forgotten."
- "The conditions attached to a promise are forgotten, only the promise is remembered."
- "The first 90% of a project takes 90% of the time the last 10% takes the other 90%."
- "The first myth of management is that it exists."
- "The more desperate the situation the more optimistic the situatee."
- "The more ridiculous the deadline the more money will be wasted trying to meet it."
- "The more you plan the luckier you get. "
- "The most successful project managers have perfected the skill of being comfortable being uncomfortable."
- "The most valuable and least used phrase in a project manager's vocabulary is "I don't know"."
- "The most valuable and least used word in a project manager's vocabulary is "NO"."
- "The nice thing about not planning is that failure comes as a complete surprise rather than being preceded by a period of worry and depression."
- "The person who says it will take the longest and cost the most is the only one with a clue how to do the job."
- "The project would not have been started if the truth had been told about the cost and timescale."
- "The same work under the same conditions will be estimated differently by ten different estimators or by one estimator at ten different times."
- "The sooner you begin coding the later you finish."
- "The sooner you get behind schedule, the more time you have to make it up."
- "There's never enough time to do it right first time but there's always enough time to go back and do it again."
- "There are no good project managers - only lucky ones."
- "There is no such thing as scope creep, only scope gallop."
- "There is such a thing as an unrealistic timescale."
- "To estimate a project, work out how long it would take one person to do it then multiply that by the number of people on the project."
- "Too few people on a project can't solve the problems - too many create more problems than they solve."
- "Users get the systems they deserve."
- "Warning: dates in the calendar are closer than you think."
- "What is not on paper has not been said."
- "What you don't know hurts you."
- "When all's said and done a lot more is said than done."
- "When the weight of the project paperwork equals the weight of the project itself, the project can be considered complete."
- "Work expands to fill the time available for its completion - Parkinson's law."
- "You can build a reputation on what you're going to do."
- "You can con a sucker into committing to an impossible deadline, but you cannot con him into meeting it."
- Project management with lot of documents is not project management
1 Introduction
1.1 Purpose
The purpose of this document is to define the Information Quality Metrics for the Information Quality Management Framework. This is a sub component of the overall Information Quality Management Framework being implemented through the Performance Management Program .
This document provides the detail of each of the specific components that need to be defined developed and implemented to support the initial adoption of better Information Quality Management in and that are essential to the Program’s Project Delivery Stage. In most cases specific documentation will be developed for each component.
Articulation of the overall vision, strategy, program scope and critical success factors as well as the overall program governance is covered in the Program Management Plan.
1.2 Background
An organisation’s approach to Information Management is critical to its success. Information Management provides the framework that the business needs in order to maintain, verify, and take control of the flow of data from source to destination, and to plan it’s archival or deprecation.
Implementing Information Quality Management provides a measurable and confidence in reporting and information analysis as the final output; the Information in the report can be trusted and tracked. If necessary, a data lineage can be reported on to show the information’s source and any transformations that have occurred during its journey to the final user.
The Information Management Methodology describes the framework and approach to improving how information is managed, measured and improved upon across . The methodology consists of components such as principles, policies, practices, tools, processes and procedures, and governance, that work together to manage the information throughout its lifecycle, including maintenance, validation and archival or disposal.
The methodology also involves the application of management techniques to collect information, communicate it within and outside the organisation, and process it to enable managers to make quicker and better decisions.
The Information Management Methodology is the vehicle for addressing Information Quality issues and establishing the associated information governance structures and processes. Quality issues are measures against the metric described in this document.
1.3 Objectives
The Information Quality Management Framework document will provide the subcomponent of the Information Quality Management Framework. It will describe the definitions of metrics. This document is a self contained subcomponent of the overall Quality framework.
2 Information Quality Metrics
Data Governance Metrics provide the information quality objectives that the organisation plans to achieve. Just like other business performance measures, metrics should be managed and tracked at the executive level. Metrics are created by either executives or data stewards with input from data analysts. Each KDE is measured against the defined metric category through the appropriate measurement technique.
There are different types of Qualitative Metric categories that can be measured in varying fashions: Accuracy, Integrity, Consistency, Completeness, Validity, Accessibility, and Timeliness. Data Governance Metrics
This task defines each of these categories and the measurement techniques and processes that will be used for measuring each KDE. Sometimes, less tangible metrics are also assessed. A rating scale should be defined for metrics at an aggregate level that is supported by the detail of the assessment.
Output / Artefacts:
· Definition of Metric Categories and Measurement Techniques
· Current-State Metrics on KDEs
· Target Metrics on KDEs
Typically, organisations do not only measure along quantitative dimensions, but also include softer/intangible dimensions to justify their investment and measure success. Defining data quality KPIs requires interplay of organisational support, governance and accountability, processes, policies and standards, as well as an overall support (either automatically or by analysis from members of the data governance team) by a set of tools. The following diagram outlines an approach to measuring data quality:
2.1 Detailed Information Quality Metrics
|
How are Data Governance Metrics Measured? |
||
|
Metric Category |
Description |
How is Metric Measured? |
|
Accuracy |
Does the data accurately represent reality or a verifiable source? |
Audit |
|
Integrity |
Do broken links exist between data that should be related? |
Profiling / Business Rules |
|
Consistency |
Is there a single representation of data? |
Profiling / Business Rules |
|
Completeness |
Is any key information missing? |
Profiling / Business Rules |
|
Validity |
Is the data stored in acceptable format and contain valid values? |
Profiling / Business Rules |
|
Accessibility |
Is the data easily accessible, understandable, and used consistently? |
Survey |
|
Timeliness |
Is information recorded and made available to systems as rapidly as is required? |
Survey |
2.1.1 Accuracy/Correctness
Description:
· The degree of agreement between a data value (or set of values) and a source assumed to be correct. The source may be a reference set obtained by comparison to “real world” data, or by reference to a data set on another system or file that is deemed “correct”.
Typical issues uncovered during assessment:
· It is expected that some names and addresses will be spelt incorrectly, or not an accurate representation of the real party, or address. Other free form text, including customer description information may not be correctly representing the party.
· Effectiveness of profiling to assess this area:
· Accuracy is very difficult to assess without verifying customer details directly with the customer, or using a “correct” 3rd party data source. Where a reference data set is available, e.g. a list of postcodes or addresses, this may be assessed; otherwise the source system data is assumed to be accurate.
2.1.2 Completeness
Description:
· The degree that the full values are present in the attributes that require them, and the degree that the attributes cover the user data requirements.
Typical issues uncovered during assessment:
· Free text fields not completed (E.g. Name, address, phone numbers, types, descriptions, and null values). Target mandatory fields not populated, or present in the source system extract file. The source systems may not have all the attributes required to load the target systems.
· Effectiveness of profiling to assess this area:
· The completeness of the field values is tested as part of the specific TBA tests. The completeness of the attributes to satisfy the user requirements are assessed in the source to target mapping documents that highlight where a target attribute cannot be populated because a source field not available
2.1.3 Timeliness / Currency
Description:
· Currency measures how up-to-date the data is, and whether the data required can be provided by the required time.
Typical issues uncovered during assessment:
· The source extracts may not be provided by the required time. Product and type codes that change over time may cause products, or types assigned before a particular date for one product, or type, and those assigned after the cut-over date are a slightly different product with the same code or now have a different type.
· Effectiveness of profiling to assess this area:
· The source system product codes are expected to be unchanged over the product life. If these assumptions do not hold the change in product code is handled during the transformation of source data in the target data acquisition.
· Changes to customer type, product types, and other codes will be treated the same way as product codes. Delays in providing the source system extracts by the specified time are identified as part of the general project reporting.
2.1.4 Consistency / Uniqueness (No Duplicates, Integrity)
Description:
· Consistency is the extent that there is a single representation of data. Consistency also includes the extent that data is duplicated within a system, e.g. customer duplicates.
· The ability to establish the uniqueness of a data record (and data key values).
Typical issues uncovered during assessment:
· Duplicate customer records
· Effectiveness of profiling to assess this area
· Determining whether there are customer duplicates in large volume data sets require specialist tools.
· De-duplication also requires a dedicated data quality improvement project that includes the appropriate governance, and sponsorship as well as appropriate tools to establish and implement the de-duplication.
2.1.5 Validity
Description:
· The data is stored in an acceptable format, and is within a reasonable range of possible values.
· Target enforced formats, lengths, and data types not implemented in the source systems. (E.g. Date YYYY-MM-DD is stored as a character in most host systems and if these fields contain other characters they cannot be transformed into the target date format)
Typical issues uncovered during assessment:
· Target enforced formats, lengths, and data types not implemented in the source systems. (E.g. Date YYYY-MM-DD is stored as a character in most host systems and if these fields contain other characters they cannot be transformed into the target date format).
· Target codes, and types may not map 1-to-1 to the source system codes, and types. Values for some attributes are not within an acceptable range.
· Effectiveness of profiling to assess this area
· It is generally not practical to check all the values in every attribute loaded are the correct type, length, and format. The most important data types to assess are date, and numeric fields.
· Check, and convert where possible source system date to the YYY-MM-DD standard target format.
· Check the format of identification numbers.
· Range of important balance, rate fields. This will only be an atomic value checks, it will not net accounts, and check the range.
· Compound attribute tests: Check the value of one attribute that is dependant on another attribute.
2.1.6 Accessibility
Description:
· Ability for users to extract the existing data they require. Users must not have different interpretations of the same data. Ability for users to extract the existing data they require. Users must not have different interpretations of the same data.
Typical issues uncovered during assessment:
· The meaning of each file attribute, and the interrelationship between attributes, and files is difficult to determine. This requires specific product system, and business knowledge.
· This may cause errors in the mapping of source fields, to target fields.
· The layout, and extract process documentation may not match the actual source system files delivered.
· Effectiveness of profiling to assess this area:
· This is a manual check of source system data, and metadata availability.
· If the meta-data does not match the actual file an issue is raised.
The relevant subject matter experts will be consulted to confirm, or explain the definition of ambiguous or conflicting attributes.
3 Data Profiling
Possible outcomes for information quality enhancement:
1. Do nothing (measure the Quality problem
2. Correct the information in transit (ETL data correction)
3. Root Cause Analysis of Data Governance Issues
Preventing Data quality issues involves analysing those process activities or application automation that prevents Data quality issues from occurring in the first place. Root Cause Analysis of Data Governance Issues is concerned with correcting root cause issues as opposed to addressing the symptoms.
Outputs:
· Data Governance Issues related to Source System Edits, Business Process, and Technology Architecture.
· Recommended Changes for Improved Data Governance
3.1 Objectives of Data Profiling
· Identify data quality issues - measurements are taken against a number of dimensions, to help identify issues at the individual attribute level, at the level table and between tables.
· Capture metadata as a by-product of the process – Metadata is the key to the success of subsequent phases of the data load including the ongoing use and maintenance of the data. Useful metadata is identified as part of the data profiling exercise and the tool-based approach will ensure it is captured in a reusable format.
· Identify business rules – The next step is to perform the data mapping. Data profiling will assist in gaining an understanding of the data held in the system and in identifying business rules for handling the data. This will feed into the data mapping exercise.
· To assess the ‘fitness for purpose’ of the source system data to satisfy the business requirements. This assessment is often done in the context of a Data Warehouse, CRM, or ERP system initiative. Therefore, the focus is on gaining a very detailed understanding of the source data that will feed these target systems, to ensure that the quality level is sufficient to meet the requirements of the target system.
This section is meant to provide guidelines as to the tests that are commonly used and the approach taken to measure data quality. Each project should consider these to be a starting point only and should ensure that the specific tests that will uncover greatest value for a client are uncovered as part of the discovery process for this client.
Key Output / Artefacts:
· Data Quality Report (per Source Systems)
· Definition of Mapping Rules and Business Rules
3.2 Different Types of Profiling
Data sources are profiled in multiple dimensions: down columns (Column Profiling); across rows (Table Profiling); and across tables (Multi-Table Profiling). The output from Data Profiling can then be used in a number of fashions, including data cleansing and mapping to a target environment.
3.2.1 Column Profiling
Column profiling typically examines the data found in a single column/field in either a table or a flat file. This analysis can either (1) examine the actual values found in the column (e.g. produce a complete frequency report listing each and every value found), (2) look for the various “data patterns” for the values in the column or (3) discover the underlying “parsed patterns” that might exist within more complex, free-form columns. Assessments are typically conducted to verify the validity/uniqueness, accuracy and completeness; column profiling can help answer the following questions:
· What do the fields mean?
· Which ones are of interest to the project?
· What is the quality of data in each field?
· And, given this, is the data of sufficient quality to load to into the target information environment?
Analysis on complex fields such as names, addresses, and contact information can be done to determine the patterns and consistency of the data. This provides information on the consistency of the values in a standard format and if the format is one that has been used already across similar fields on other tables within the database. From a vendor perspective, separate tools may be used to profile simple fields (ones that do not require parsing) as opposed to complex fields.
3.2.2 Table Profiling
Table Analysis is used to determine the key information and also relationship and dependency patterns between the data in each field within the table. It is done across every field within each table in the source system. Table Analysis is important as there are some situations where it is absolutely essential to examine values in two or more columns. A blank value in Column A might be acceptable if there is a valid value in Column B, but unacceptable if Column B is also blank. Similarly, a valid value in Column A (e.g., product code “HL”) and a valid value in Column B (e.g., sub-product code “521”) might be an invalid code combination. In such situations you must look at value combinations across columns rather than just examine values within a single column. A tools-based approach is undoubtedly the best choice for examining these cross-column or “table based” dependencies.
3.2.3 Multi-Table Profiling
Multi-Table Analysis looks for relationships among columns existing in different tables, rather than just within a single table. Multi-Table profiling shows the relationships between tables using key fields as links to bridge the multiple tables. The aim is to analyse and determine the existence of referential integrity issues between each table analysed, e.g. orphans and clients without address and contact information. If the profiling analysis reveals two columns in different tables share the same set of values and one is not a foreign key and the other a primary key, than redundant data exists between the two tables. For example, an employee table and an employee demographics table might both contain a “home phone number” column and the same values within these columns. Thus, an employee’s home phone number might be stored redundantly in two separate tables.
3.3 The Iterative Profiling Process
It is important to note that there is an iterative approach to profiling within each of the analysis steps below. Each step involves:
· Running the analysis within the appropriate Vendor Tool
· Analysing the results of each analysis
· Verify the results with the Source System SME
· Documenting the results (both in deliverables and within the profiling tools)
· Plan further analysis based on results
The data investigation process should verify what the source system owners say about the data against the actual data; it should also verify what the data says against what the source system owners say.
This process is iterative – for example, a source system owner may say that all customer names must have a full first name and full surname. However, when this rule is checked against the data, this shows that 10% of the records have only a first initial. In this case, this must be discussed with the source system owner. This type of anomaly may be explained by a business rule that was applied to new data that was not applied to historical data. Further analysis is performed in this case to verify that all anomalous records were created before the expected data. Data Re-Engineering also follows and iterative process for standardize, correct, match and enrich data.
3.4 Typical Investigations Performed
3.4.1 Column Profiling (Simple Fields)
Column Profiling of Simple Fields is most typically done to check for Completeness, Uniqueness and Validity. An example would be the Account Number field within the Client_Accounts table. The Account Number column is analysed for:
|
Column Profiling (Simple Fields) |
||
|
Field |
Test |
Description |
|
Account Number |
Completeness |
All records contain a non-zero value |
|
|
Uniqueness |
All records contain a different account number |
|
|
Validity |
Ensure all account numbers are numeric or follow the source systems account number structure |
3.4.2 Column Profiling (Complex Fields)
Column Profiling of Complex Fields is most typically conducted to verify the validity/uniqueness, format accuracy and completeness. An example would be looking at Customer Records:
|
Column Profiling (Complex Fields) |
||
|
Field |
Test |
Description |
|
Multiple Fields |
Completeness |
As per Client Name. All the name fields should contain a title, firstname, and lastname. Not all person names contain middle names and suffixes such as Jnr etc. |
|
|
Uniqueness |
Presence of Title at start of name Has a first name, and Has a last name |
|
|
Validity |
No characters, i.e.: #,@% present in all the name fields. |
3.4.3 Table Profiling
Table Profiling is most typically conducted to verify the validity/uniqueness, format accuracy and completeness. An example would be looking at Customer Records:
|
Table Profiling |
||
|
Field |
Test |
Description |
|
Customer ID |
Completeness |
Is a unique value populated within this field to uniquely identify each customer record? If so is there a high population ratio on this field, i.e.: 100% populated. |
|
|
Uniqueness |
Presence of a Primary Key or Foreign Key in relational databases. Are these keys unique to each record? |
|
|
Validity |
Are these key fields populated with valid value, i.e.: unique values? |
|
|
Format |
Are these values in a consistent format, i.e.: numeric or does it contain leading alphas etc. |
3.4.4 Multi-Table Profiling
Multi-Table Profiling is conducted to verify the validity/uniqueness and accuracy of the key fields to ensure referential integrity. An example would be to assess the Customer Table vs. a Customer Loan:
|
Customer Table |
||
|
Field |
Test |
Description |
|
Customer ID |
Completeness |
Is a unique value populated within this field to uniquely identify each customer record? If so is there a high population ratio on this field, i.e.: 100% populated. |
|
|
Uniqueness |
Presence of a Primary Key or Foreign Key in relational databases. Are these keys unique to each record? |
|
|
Validity |
Are these key fields populated with valid value, i.e.: unique values? |
|
|
Format |
Are these values in a consistent format, i.e.: numeric or does it contain leading alphas etc. |
|
Multi-Table Profiling Customer Table: |
||
|
Field |
Test |
Description |
|
Customer ID |
Completeness |
Is a unique value populated within this field to uniquely identify each customer record within the Customer table? If so is there a high population ratio on this field, i.e.: 100% populated. |
|
|
Uniqueness |
Presence of a Primary Key or Foreign Key in relational databases. Are these keys unique to each record? |
|
|
Validity |
Are these key fields populated with valid value, i.e.: unique values? |
|
|
Format |
Are these values in a consistent format, i.e.: numeric or does it contain leading alphas etc. |
Information Management Principles
|
Prepared by: |
Marc Narder |
Copy number: |
|
|
Document Status: Dop: |
Draft |
Date: |
|
|
Version: |
Version 0.1 |
Deliverable No: |
|
|
Document Name |
Information Management Principles |
|
Document Location |
|
|
Document Owner |
Marc Narder, Manager, Information Management |
1 Introduction
1.1 Purpose
The purpose of this document is to define the Information Quality Principles for the Information Quality Framework. This is a sub component of the overall Information Quality Management Framework., which is itself a subcomponent of the overall Information Quality Management Framework being implemented in the Performance Management Program.
This document provides the detail of each of the specific components that need to be defined developed and implemented to support the initial adoption of better Information Quality Management in and are essential to the program’s Project Delivery Stage. In most cases specific documentation will be delivered for each component.
Articulation of the overall vision, Strategy, program scope and critical success factors as well as the overall program governance is covered in the Program Management Plan.
1.2 Background
An organisation’s approach to Information Management is critical to its success. Information Management provides the framework that the business needs in order to maintain, verify and take control of the flow of data fro source to destination, and to plan it’s archival or deprecation.
Implementing Information Quality Management provides a measurable confidence in reporting and information analysis as the final output; the information in the report can be trusted and tracked. If necessary, a data lineage can be reported on to show the information’s source and any transformations that have occurred during its journey to the final user.
The information Quality Framework describes the framework and approach to improving how information is managed, measured and improved upon across . This framework consists of components such as principles, policies, practices, tools, processes, procedures and governance, that work together to manage the information throughout its lifecycle, including maintenance, validation and archival or disposal.
The methodology also involves the application of management techniques to collect information, communicate it within and outside the organisation, and process it to enable managers to make quicker better decisions.
The Information Management Framework is the vehicle for addressing Information Quality Issues and establishing the associated information governance structures, principles and processes.
1.3 Objectives
The Information Quality Framework document will provide the subcomponent of the Information Management Framework. It will describe the quality principals used to support the overall quality Framework.
2 Information Quality Principles
The Information Quality Principles are derived from and support the Information Management Principles. They define the specific information quality premises that serve as the foundation for the courses of actions for information / data quality within .
The six (6) Information Quality Principles are:
2.1.1 IQ Principle 1: Fact-based decision making
Both strategic and operational decisions are based on facts that can be sourced back to data / information that is held by .
2.1.2 IQ Principle 2: Integrated data with consistent definitions
One of ’s major assets is information. Each Division or unit is part of the whole and can leverage enterprise information assets in an integrated and synergistic way (that is, the whole is greater than the sum of parts).
2.1.3 IQ Principle 3: Appropriate retention of detailed data
Information is retained whenever physically possible within the constraints of government legislation, corporate ethics and privacy commitments.
2.1.4 IQ Principle 4: Quality of data will be measured
Decision makers not only need access to information, but more importantly they also need to understand the timing, reconciliation, completeness and accuracy of that data.
2.1.5 IQ Principle 5: Appropriate enterprise access
Every member of staff is trusted to handle information appropriately and sensitively. The default position is that a staff member can access information unless there is a specific commercial, legal or ethical reason as to why not.
2.1.6 IQ Principle 6: Every data item has one person or role as ultimate custodian
Every item of data requires unique and ultimate custodianship by a single role and person to ensure that issues or conflicts always have an ultimate point of escalation.