Operational Change Control

Best Practices

Author:Byron Love, MBA, PMP, CEC, IT Project+, MCDBA
Internosis, Inc

Executive Summary

Effective change management is vital to providing high-quality IT services in modern computing environments. Prior to Windows Active Directory, Windows was more of a departmental operating system. An organization could effectively manage its Windows infrastructure on a department by department basis. Now with Active Directory, Windows has become an enterprise operating system and cannot be managed effectively using a departmental model. Enterprise-wide IT governance is now essential. Changes at the departmental level can have an enterprise-wide impact and must be managed in a systematic manner.

Communication

The overriding theme of effective change control is communication. One’s IT governance structure, change and configuration management processes, and supporting tools referenced in the best practices serve to ensure the integrity of the network infrastructure baseline, and the systematic communication of changes to that baseline. Integration of change control with problem management processes fosters communication of network service disruption information to those responsible for the design and configuration of the network baseline.

Without this communication, the architects of the baseline will not know about design or configuration flaws that cause services disruptions and will not know about requirements to reconfigure the infrastructure to make improvements or to resolve problems. They will not have the opportunity to update the baseline to correct flaws or to proactively prevent service disruptions.

Unauthorized changes – those not sanctioned by engineers responsible for the architecture – corrupt the baseline and the integrity of operations. They create a disparity between the configuration the architects designed and believe to be deployed and the actual operational baseline.

Operational Change Control Defined

Operational change control consists of the change management process and configuration management processes within an organization. Gartner has determined that operational change management is a prerequisite to providing high IT service quality. It is not optional. IT not only supports functional business processes, but it is an enabler that functional organizations rely on, and expect to be available, and expect to provide high service quality. These expectations cannot be met without change management processes.

Change Management

Change management includes

  • Project change management
  • Managing the design, development, testing and implementation of change
  • Operational change management
  • Managing the approval, scheduling and coordination of change.

Improving change management processes is one of the best investments enterprises can make.

Integrated Change Control

The Project Management Institute defines integrated change control at the project level as processes concerned with

  • Influencing the factors that create changes to ensure that changes are agreed upon
  • Detecting changes and,
  • Managing changes in real time.

The original defined project scope and the integrated performance baseline must be maintained by continuously managing changes to the baseline, either by rejecting new changes or by approving changes and incorporating them into a revised configuration baseline.

Integrated change control requires:

  • Maintaining the integrity of performance measurement baselines (configuration control libraries).
  • Coordinating changes across knowledge areas. For example, a proposed schedule change will often affect cost, risk, quality, and staffing.

All organizations require standardized methods and procedures to manage change and guide day-to-day operations. IT systems in production are more stable when formal governance policies guide efficient and effective change management.

Awareness of the Impact of Change

Another key goal of the change management process is to ensure that all parties affected by a given change are aware of and understand the impact of the impending change. Effective communications is a principle of effective IT operations.

Since networks, applications, and operating systems are heavily integrated, changes in any component can have a profound affect on other components. Change management attempts to identify all affected systems and processes before the change is deployed in order to mitigate or eliminate any adverse effects. Typically, the target or managed environment is the production environment, but it should also include key integration, testing, and staging environments.

The categories of assets that should be placed under change control are broad and include, but are not be limited to, hardware, communications equipment and software, system software, applications software, processes, procedures, roles, responsibilities, and documentation that are relevant to the running, support, and maintenance of systems in the managed environment. In other words, any asset that exists in the environment and is necessary for meeting the service level requirements of the organization should be placed under change control.

Elements of Operational Change Control

There are several elements that compose an effective change and configuration control system in best-in-class IT environments. The change control system is a set of documented procedures for monitoring and evaluating performance. It documents the activities for changing systems documentation and the processes and tools necessary to authorize changes. The change control system must include provisions for emergency and routine changes, which may not require prior review. However, even those changes must be documented in order to maintain the integrity of the baseline. The following are elements of a change control system:

Change Control Policy

The foundation of effective change control within an enterprise is the change control policy. This policy must be developed specifically for the operating enterprise and requires input from all groups that are stakeholders in IT operations. These stakeholders include functional business units, end users, developers and engineers, systems and network administrators, and IT management. Senior management’s endorsement and enforcement of this policy is mandatory to successful implementation.

Attributes of an effective change control policy should include:

  • Change Policy – Providing a set of guidelines for the change management process.
  • Workflow – Mapping workflow to enable complex activities to be broken down into their elements, creating a clear, visual picture of roles, responsibilities and areas of interaction. The results can be translated into a list of criteria for organizational structure, staffing needs, outsourcing possibilities and peer-support integration.
  • Approvals and Notifications – Establishing policies for approvals and notifications based on the size of any risk, the category and the type of change.
  • Interfaces to Other Processes – Documenting change processes so they are linked to other process areas, such as problem, configuration and asset management.
  • Dependencies – Coordinating change projects and tasks to provide adequate lead time and sufficient prior notification to supporting or affected groups.
  • Change Categories, Types and Risks – Determining the scope of the effects of change, as well as categories, types and risks so they are well-defined and documented. Documentation tools include:
  • Requests for change (RFC): Change requests are formalized with descriptions of the change, components affected, business need, cost estimates, risk assessment, resource requirements, and approval status. o Emergency Change Requests;

Change management must provide a mechanism for performing emergency changes when necessary. This mechanism should be used minimally.

  • Metrics and Reporting – Defining metrics to provide performance analysis data for management reporting and organizational benchmarking.
  • Documentation samples – Gathering samples from each category for training and guidance.
  • Roles and Responsibilities – Mobilizing an adaptive work environment to help ensure successful change management. Established IT roles must add newly identified and defined responsibilities in areas of the submission, building, testing and implementing of any changes. The documentation of these roles should fit within the standardized tasks decided on in the change workflow guidelines. Common roles and responsibilities can include:

    o Change Requester – The stakeholder responsible for modifying the system or environment. The requestor submits the request for change.
    o Change Specialist – Responsible for assisting the requestor in analysing scope effects.
    o Change Coordinator (or Change Manager) – Responsible for directing change control and conflict management.
    o Change Owner – The stakeholder proposing the change.
    o Change Approver – Responsible for analysing the change and approving or rejecting the request.
    o Change Implementer – The stakeholder that enacts the change.

  • Board Charter Requirements – The Change Control Board charter should include the authority, which explains the policy basis upon which the Change Control Board is established, roles and responsibilities, and definition of Change Control Board Participants, such as Co-Chairs, Executive Secretary and permanent members.
  • Board Procedure Requirements – The change control policy should provide guiding principles, which provide a basic approach of ensuring issues have been fully coordinated prior to being submitted to the board for disposition. An appeals process for rejected changes may be a procedural requirement. Procedures for modifying change control procedures should be provided.

Operational Change Management

Operational change management activities include coordinating and scheduling enterprise infrastructure change requests in order to reduce downtime and risk. The Change Manager controls this process and normally reports to IT operations.

The Manager:

  • Processes change requests
  • Coordinates business and technical risks with stakeholders
  • Coordinates with organizational management and change boards to approve or reject changes
  • Coordinates the scheduling of changes so that business risks are minimized
  • Captures lessons learned from change activities
  • Provides change control reports to senior management

Figure 1: IT Change Management Processes (Source: Gartner Research)

Configuration Management

Configuration management is any documented procedure used to apply technical and administrative direction and surveillance to:

  • Identify and document the functional and physical characteristics of an item or system
  • Control any changes to such characteristics
  • Record and report the change and its implementation status
  • Audit the items and system to verify conformance to requirements.

Key IT components are referred to as Configuration Items (CIs). Configuration management processes identify, record, and track CIs, including the description, version, components, relationships to other CIs, and status. The items should include hardware, system software, application software, and the policies, operations procedures, and other documentation that support the CI. The basic activities of configuration management are:

  • Configuration management planning. Planning and defining the scope, objectives, policies, procedures, organizational, and technical context for configuration management.
  • Configuration identification. Identifying CIs and their owner, interrelationships, documentation structure, and versioning.
  • Configuration control. Ensures CIs are managed throughout their life cycle according to approved change control procedures.
  • Configuration status accounting. Status and historical reporting of events associated with the CI throughout its lifecycle.
  • Configuration verification and audit. Periodic physical inspection of CIs to verify adherence to change control and configuration management procedures.

Configuration management has proven to be very difficult to implement because of the manual effort needed, the constant changes to the IT environment and the lack of tools.

Change Control Board

A Change Manager and project level configuration control systems escalate enterprise-wide and critical changes to an operational Change Control Board. The CCB is a cross-functional group set up to evaluate change requests for business need, priority, cost/benefit, and potential impacts to other systems or processes. This group is responsible for approving or rejecting proposed changes in an effort to maintain a stable, available, and reliable IT infrastructure capable of meeting stakeholder functional requirements and performance expectations.

Typically the CCB will make decisions for deployment, further analysis, deferment, or cancellation of changes. The CCB is authorized by the enterprise’s change control policy and is established and governed by a charter. Not all organizations define their change management committee as a CCB. Other common occurrences are the Engineering Review Board (ERB), Technical Review Board (TRB), Technical Assessment Board, and a variety of others.

Changes should be presented to the CCB in a prioritised manner such that the change category, risk, and expected results can be easily evaluated. The CCB should be presented with a summary of the low-risk changes, such as maintenance activities, and should severely scrutinize high-risk changes. A mechanism must be in place to handle emergency changes, which should be agenda items for the CCB meeting following the change. Proposed changes not on this agenda should not be implemented.

Change Management Tools

The use of tools is mandatory for effective communication across a large user base. Standardized change management tools provide a common repository to track the registration, status and history of change activity. These tools enforce configuration control, facilitate status accounting, and are used to perform verification and audit functions. A paper-based or manual system would severely inhibit any large enterprise’s operational change integration process.

The following are needed:

  • Project management tools that show the need for planning (and cross-collaboration) by identifying who is doing what and when
  • Service desk software that provides problem and change management applications enabling ticket and case entry and tracking
  • A Web-based calendar to notify users of planned changes – globally, as well as by location
  • Trend analysis of the problem resolution database and incident reports to enable trends to be noted early enough to have an effect on planned and unplanned downtime
  • Integration of change and incident-problem databases to enable a better understanding of the problems caused by changes, which enables better future planning by reducing unplanned outages.

Change Management Repository

(Baseline) An important automated tool is the change management repository. This repository is a central storage and retrieval system for configuration management and change control data. Current configuration information on all configuration items (CI) in the enterprise, including application software, systems software, databases, servers, and desktops, should be stored in this repository.

The CI data repository is referred to as the configuration management database (CMDB). Hardware specifications, assembly instructions, and network configurations are also stored in the CMDB. The contents of this repository are referred to as the ‘configuration baseline’. The tools and procedures for accessing this repository are referred to as the ‘configuration management library system’. Whenever possible, this database should be self-maintaining, with automated updates to CIs.

Release Management

The focus of release management is to facilitate the introduction of software and hardware releases into managed IT environments. Typically this includes the production environment and the managed pre-production environments. Release management is the coordination point between the release development/project team and the operations groups responsible for deploying the release in production.

The Enterprise Configuration Control Board has the responsibility to develop and maintain a release schedule which orchestrates system deployments and upgrades. Managing the IT environment according to a release schedule facilitates effective communication and sets stakeholder expectations. Release management activities include acquisition, testing, integration, packaging, and documentation of IT components, obtaining deployment approval, deployment, and post deployment review.

Examples of routine maintenance activities that require release management included desktop image management, network device operating system maintenance, and deployment of security patches to desktops and servers.

The relationship between change management, configuration management, and release management is depicted in Figure 2.

Figure 2: Relationship between Change, Configuration, and Release Management (Source: Microsoft Operations Framework)

Operational Change Control Relationships to Other Processes

Event Management Event management is concerned with the identification, recording, tracking, analysis, and resolution of events that affect the IT infrastructure of an enterprise. An event is defined as an occurrence that, if not resolved, will eventually cause disruptions to IT services and/or dissatisfied IT stakeholders.

There are two types of events:

  • Service requests
  • Incidents

Service requests come from IT infrastructure users, such as requests for additional storage or new services. Incidents are disruptions to IT services, either caused by defects in the infrastructure design or configuration, hardware failures, software failures, or security breaches. Incidents always involve one or more IT configuration items and are resolved by action items. Incident management is also known as problem management.

Change control, configuration management, and problem management should all rely on the same configuration management database. The processes for these management areas are integrated as well as they are designed to reach the same goals . In most organizations, the service desk owns the incidents and their contribution to the change management and incident management processes reduce downtime and facilitate effective communication with end users.

The following diagram depicts this interrelationship.

Figure 3: The Relationship between Service Desk Incident Management and Change Control (Source: Microsoft Operations Framework)

Quality Assurance

The quality assurance function provides management visibility into the processes being used to deliver IT services. This function is responsible for reviewing and auditing the working procedures, standards, and server delivery activities, including the operational change control activities, to ensure they comply with applicable standards and procedures.

The quality assurance function is independent of the of the IT operations group and may be outside the enterprise altogether. Senior management is responsible for acting on the results of the quality assurance activities.

Asset Management

Configuration management is often confused with asset management. Microsoft defines asset management as an accountancy process that includes depreciation and cost accounting. Asset management systems typically maintain information on assets above a certain value, their business unit, purchase date, supplier, and location. The relationship to other assets is not usually recorded and the information is primarily used to track the whereabouts of expensive equipment. Many organizations start with asset management and then move on to configuration management.

Benefits of Operational Change Management

There are many benefits to implementing configuration control. According to Gartner, organizations have experienced the following benefits from instituting operational change policies:

  • Reduced unplanned downtime, by as much as 25 percent to 35 percent, because changes are better planned and tested. In addition, back-out plans are typically required so that quick restoration of the previous configuration can be done in the event of a failure or unforeseen problem during change implementation.
  • Reduced planned downtime, by as much as 25 percent or more, because more changes are done in a single downtime period and also because of more appropriate task sequencing, which minimizes recurring tasks such as reboots.
  • Improved quality of service (from reduced downtime), which reduces overall costs (IT support and business downtime costs) and corporate risk.
  • Repeatable processes that enable faster, more-agile response to continual changes in business requirements and technology shifts, and provide an essential ingredient in evolving toward a real-time infrastructure.
  • Increased communication with users, ensuring that downtime occurs during off-peak business cycles, which reduces corporate risk.
  • Reduced volume and effects of change-related incidents and requests for technical support by end users.
  • Improved client satisfaction and higher levels of customer service through better communication with users and the service desk. Publishing planned-downtime schedules helps to set expectations and reduce user frustration.
  • Reduced support costs are enabled through standard desktop configurations and control of change for enterprises that include desktop management in their change management process,. This is also true for standard server configuration environments.

Industry Models

Control Objectives for Information and Technology (COBIT)

COBIT is a framework of best practices developed to help bridge the gap between business risks, control needs, and technical issues. These practices represent a consensus of experts in IT management, providing assistance in optimising information technology investments and providing a standard to measure against when things go wrong. COBIT provides maturity models for control over IT processes to provide a measurement scale for organizational compliance with best practices.

IT Services Capability Maturity Model (CMM)

The IT Services CMM applies the Software CMM generic structure, developed by the Software Engineering Institute, to IT Services. The model was designed to give IT service providers and customers of IT services a method to determine the maturity level of the service provider. Customers can use this maturity level as a gauge to distinguish the quality of competing service providers, and service providers can use it to measure and improve the quality of their services.

For a full list of references see the .PDF file which is available for download or printing

About the Author

Byron Love received a Bachelors of Arts in Computer Science from Thomas Edison State College in 1991 and a Masters in Business Administration from Averett University in 1998. He is a Project Management Institute (PMI�) certified Project Management Professional (PMP�), he is CompTIA IT Project+ certified, he is an Institute of Certified E-Commerce Consultants (ICECC) Certified E-Commerce Consultant (CEC), and he is a Microsoft Certified Database Administrator (MCDBA). Byron is a Major in the US Air Force Reserves assigned to Headquarters Air Force, Directorate of Communications Operations and has 20 years of military service. He was Reserve Officer of the Year for this directorate in 2003. Byron is a member of the Project Management Institute, the IEEE Computer Society, and the Black Data Processing Association.

He is currently a Client Principal at Internosis, Inc. (www.internosis.com) where he leads knowledge management and infrastructure transformation engagements. He is currently leading knowledge management projects for the US Marine Corps and for the US Joint Forces Command Joint Battle Center. Committed to community service, Byron serves as Chairman of the Unity Economic Development Corporation, a non-profit, 501(c)3 organization chartered to promote community-business relationships and financial empowerment. Byron resides in Odenton, MD, USA and can be reached at IT-PM@byronlove.com.

Copyright Project Perfect