President's Perspective

The Many Facets of Assessment

An assessment of an organization’s IT infrastructure that brings true value to the business has many facets. It should assess a comprehensive set of IT assets; understand the interrelations between assets; evaluate resource utilization, processes, and skills; be as non-intrusive as possible; and produce a detailed report with clearly defined actions and benefits of those actions.

Assessments may involve servers, storage, and/or software, and tools are usually employed to gather data, perform analysis, and produce resultant reports. A good assessment tool will offer comprehensive inventorying

of IT systems including detailed configurations, installed software/middleware, and resource utilization information… all in a manner that is non-disruptive to the business and conforms to the organization’s security policies and practices.

Where the value of an assessment should really crystallize is in the analysis of the information, the review of elements not typically assessed by an automated tool, and the findings report (the key deliverable). The information found in assessments may paint a picture of logical consolidation targets, areas ripe for virtualization, and other key recommended actions, but a world-class assessment also includes reviews of a variety of processes and skills.

Processes related to purchasing IT assets, change control, architectural review, solution development, routine support, technology refreshes, application of corporate standards, and periodic review of and modification to corporate standards... all have an impact on solution path recommendations.

Furthermore, a review of the skills of architects, system administrators, and developer personnel has a significant impact on the conclusions and recommendations of the assessment. Such recommendations should include skill development and education path suggestions, where needed, to ensure that the organization’s staff is optimally poised to manage the IT environment and the business applications they support.

Growing and maintaining consolidated and virtualized solutions requires specific and tailored education programs in most cases, and the payoffs must be articulated and realized. Thus, it is clear that a truly valuable assessment must cover a variety of dimensions – assets, processes, skills, and more – to enable the organization to implement needed changes and associated solutions that the business can grasp and execute to… and provide no doubt as to the economic value in terms of lower costs, increased revenue, improved productivity, efficiency gains, compliance, and the like.

An example of an assessment offering which exhibits these characteristics is that offered by Datatrend Technologies. Datatrend uses a variety of best-of-breed tools in conjunction with its assessment capabilities. The tools are non-pervasive, agent-less products that, when fully explained, pass security reviews from organizations contemplating an assessment of installed IT assets. They also provide unparalleled inventorying of IT assets including comprehensive reporting unmatched in the industry today. Datatrend's Optimization Pro server assessment tool, for example, has exceptional resource utilization monitoring capability. This feature is key in that it provides a true picture of utilization and should not be substituted for client assumptive data.

The theme of this edition of TrendSetter is somewhat related to our previous edition, "Virtualize To Optimize". While it is true that some assessment recommendations may not include a call to virtualize, such a paradigm is becoming increasingly rare. Being able to build and maintain consolidated and virtualized solutions has become imperative, in order to take cost reduction, improved manageability, and proactive technology leadership to the next level. IT organizations that get with the program in these areas will have a direct impact on the ability of the lines of business to develop competitive offerings in the marketplace that aid in gaining share and improving client loyalty.

Simplifying IT solutions and enabling responsiveness to line of business needs directly affects total organization success in many ways. And an effective, value-based assessment may be the critical first step in achieving that success.

Technology assessments — undeniable value

The return on technology assessments can be large and rapid. After the hectic activity of the “IT sprawl decade”, most organizations know that optimizing their IT assets could yield benefits in many areas. But why should this be an area where you reach for outside help? Surely no one knows your current set-up better than your own people? Surely this sort of “tidying up” exercise could be done in-house?

It turns out there are very good reasons for getting the help of specialists, including:

• The use of automated tools can produce a huge increase in the productivity and effectiveness of collecting and analyzing data about the current set-up.

• People who have done dozens of previous assessments not only know what to look for but also know the solutions or action plans that work.

Put simply, this type of assessment is something you only need to do once every few years; therefore, by definition, your own staff can never have the experience or the tools to do an efficient and thorough execution. Not to mention that, removing them from their current value-producing tasks may incur more cost than reaching for specialized help.

The following articles and case studies explain in more detail why we strongly recommend periodic technology assessments to most of our customers – and why that recommendation has become particularly strong in the last few years. You will see why our customers are saying things like:

“We decided to undertake a consolidation effort based on the results of an assessment we outsourced. Our total number of servers went from 220 to 86 — a $1.5M labor cost savings for our company.”

"An overdue assessment told us we had far more servers with the Windows operating system than we needed. We were able to renegotiate our license agreement and save $250K/year — we only wish we had done it sooner!"

Assessment = a comprehensive review

The road to assessments is paved with audits

Two decades ago, the word “audit” was generally used in reference to business practices — operational or financial. With the coming of the computer age, the definition of audit began evolving from one of “a process for testing the accuracy and completeness of an organization’s financial statements” to a method of testing the security of a company’s computer systems.

In the mid to late 90s, as firms expanded their systems infrastructures and brought business on-line, the security audit became a critical piece of on-going systems strategy. Ten years later, the audit evolved into the “health check” as many companies realized their rapid addition of servers and software to accommodate this ever-changing business landscape was, in some cases, adversely affecting the overall health of the infrastructure.

Fast forward to 2006 when the term “technology assessment” has evolved again beyond the security audit and the health check to a comprehensive review of an organization’s system infrastructure including servers, storage, and security.

Assessing Servers

The breadth of server-related items to be analyzed is rather large and beyond the scope of this overview. Here we will briefly look at four key areas: processors, operating system licensing, back-up restore, and systems management software.

Processors vs memory

As we have discussed in previous issues of this newsletter, server proliferation was a rapid response to needs for change during the 90s. And as we have also said, “server sprawl” is now ironically the inhibitor to providing a rapid response to the need for change. If proliferation is the problem, optimization is the answer, and technology assessments are the roadmap.

Any assessment worth its salt will begin by evaluating processor utilization, reflected in processor speed and response times. Here are several eye-opening statistics that have been identified from performing assessments on thousands of servers:

• 95% of Windows servers have been over-configured such that they are exhibiting processor utilization of less than six percent.

• 70% of Windows servers have been inadequately configured with memory, resulting in paging rates that limit performance and affect response times.

• Customers generally have more servers than they even know about.

Given the price-performance differences between processors and memory, it is easy to see how customers can reap huge benefits from consolidating underutilized servers (once they find out about all of them!) and reconfiguring the reduced number of servers with appropriate memory.

Software licensing

Case Study: Software Licensing An energy company in Texas recently undertook a broad-base assessment which included software licensing. In the end, they derived great value from three recommendations: Renegotiate licensing requirements. Of the 450  servers with the Windows operating system installed, it turned out that 410 didn't need it at all. Microsoft was very receptive to the request for renegotiating their license agreement. Annual savings = $250K Reduce the number of Enterprise editions. Of the 450 servers, the remaining 40 were installed with the more expensive version of the operating system. The company was able to cut this number in half — only 20 servers actually needed to be running Windows Enterprise Edition. Annual savings = $135K Remove unneeded software from the standard image. Microsoft Office had been included as part of the standard build on all 450 servers. Office was removed completely from 410 servers (those without the operating system). Annual savings = $75K Total annual savings = $450K

Once a clear picture of the hardware is obtained, an assessment inventories the number/type of operating system licenses with an eye towards reducing the total number while remaining in license compliance.

• Almost two-thirds of customers running Windows have more Advanced or Enterprise editions than is necessary.

Deploying these more expensive versions only when the machine configurations demand it can result in significant cost savings. Additionally, converting to the Standard Windows operating system where possible can reduce annual maintenance and support costs.

Back-up/Restore

In some cases, the primary benefit of completing an assessment isn’t financial — or directly financial. Implementing single, consistent back-up/restore procedures, for instance, minimizes effort for operations and support personnel. And, of course, not having back-up data if and when you need it carries a very large price tag.

• In over 90% of inventories infrastructures, at least 15% of the Windows servers have no identifiable method of back-up/restore.

• Of the servers with back-up/restore software installed, over 40% have old/mixed versions or several completely different programs altogether.

Systems management software

About half of the companies participating in these assessments have some form of systems management software installed (e.g. HP OpenView, IBM Tivoli, CA Unicenter, or BMC Patrol).

• When systems management products are in use on an enterprise level, it is common to find 40% of the Windows servers without agents installed.

• Of the servers with the software installed, as many as 10% have agents that are installed but not currently running.

This is an example of a tool which is only as good as its use — systems management software relies on data collected from agents to be effective.

Assessing Storage

Case Study: Storage Consolidation Imagine you are the CIO of a large company where violations of corporate data policy are taken very seriously. Now imagine you have hired an outside firm to do a storage assessment in which 10 TB of data were analyzed to identify: 1. data violations/redundancies. 2. underutilized devices. With the assessment tool running, a query is done real-time on your data stores, revealing that 1.5TB is devoted to MP3 files (needless to say, a policy violation) — 15% of all your stored data! Precisely the situation a CIO of a pharmaceutical company on the East Coast recently found himself in. This same CIO got another big surprise when the assessment results repeatedly emphasized the company's over utilized storage devices, as opposed to the underutilized he was initially focused on. Over utilized devices are at least as important, if not more so, since running critical processes or applications against them could fail without warning. While the CIO could not put a price tag on these two important assessment discoveries, their value to his company is well, undeniable!

Windows

As with servers, many companies in recent years have added storage devices which they are now finding to be greatly underutilized. Of the companies with Windows infrastructures included in these results, over 80% store their data on internal or captive external disk subsystems. Use of storage area networks (SANs) and network attached storage (NAS) still appear to be the exception.

• With predominately internal storage, Windows storage utilization averages 25-35% — in some cases going as low as 14%.

Enter storage consolidation. In cases where Windows infrastructures have been converted to rely primarily on SANs, enterprise storage and NAS, utilization typically improves to 60-75%.

UNIX/Linux

In the case of UNIX/Linux infrastructures, the numbers look quite different since approximately 70% of these companies already store their data on enterprise storage systems.

• UNIX/Linux storage utilization averages 50-60% overall.

Though not quite as dramatic as in Windows environments, there are still benefits to be realized from increased reliance on enterprise SANs and NAS.

Assessing Security

Windows servers

In a Windows world, three recurring security-related issues have been identified with this assessment control group:

• Use of Windows Workgroups (less secure, more costly) rather than domains (more secure, less costly).

• More domains than are necessary (creates complexity and additional work for security administrators)

• Remote servers whose domain administrator account have been removed or disabled (to prevent enterprise/central IT management).

In most cases, one Windows domain with appropriate use of, for example, Group Policy Objects (GPOs) will suffice. Servers in Windows workgroups should be migrated to domains with security settings that conform to policies covering virus protection, systems and network management, backup-restore, software licensing and Windows services. Lastly, the domain environment should enable system administrators to gain central access to all servers for more efficient management.

UNIX/Linux servers

There is surprisingly little or no consistency within most UNIX security environments: administrators are typically organized by geographic location and/or operating system (Solaris, HP-UX, AIX, etc.) and for the most part establish their own security “policies” and practices such as unique user IDs and passwords for each server. If this type of security environment is difficult to assess, it is virtually impossible to adequately manage.

While the Windows environment provides some security structure with the domain concept, UNIX/Linux environments require a security architecture to be defined externally and, ideally, implemented in a consistent way. UNIX does, however, provide superior built-in facilities such as trusted host support and encrypted keys.

A comprehensive review

As we stated at the beginning of this article, an assessment is a comprehensive review of an organization’s system infrastructure. In this article, we highlighted just a few of the areas that are assessed and some of the typical findings.

In the chart below, we show a full list of the areas that would be studied during an assessment of a Windows Server environment. (Those highlighted in blue have been discussed in some detail above.)

As you can see in this chart, there any many areas of concern where the primary benefit is improved reliability and efficiency. Ultimately, those too will result in cost savings.

An apple a day (or an assessment every few years) keeps the doctor away

Given the breadth and depth of technology assessments and the benefits to be gained, you probably now see why we strongly recommend periodic technology assessments to most of our customers. A good technology assessment is more than just a health check - it is key to developing sustainable IT strategies and to improving efficiency and performance, as well as reducing costs. And the results will reassure you that you won't need to call in the IT "doctor" anytime soon.

... on how to take the first steps in assessing your IT systems

An assessment in action

These days, there is no shortage of technology consultants offering infrastructure assessments. So how do you know which one is worth your time, energy, and of course money? And which one delivers real results (providing you with recommendations you can use right away), instead of merely advising you to do another study?

We asked ourselves those very questions when developing our assessment services. As you know, we do not usually recommend specific solutions in our newsletter, but we have worked very hard to develop the best offering in this area for our customers. A solution that really catches attention is our "soup to nuts" asset inventory, which performs discovery, inventory, utilization data collection, and consolidation design of any size IT Infrastructure - quickly and accurately with minimal impact.  We call this the Datatrend Optimization Pro.

Perhaps you have already written off both the assessment in general and this tool in particular as “something only big companies do”. If so, keep in mind that companies of all sizes will at some point face the need to do:

• Strategic technology planning and migration

• Data center relocation, consolidation analysis, planning and design

• Capacity planning and budgeting

• Server and storage consolidation and centralization

• Business continuation, disaster recovery planning and design

• Software license compliance review.

No installation required

Datatrend Optimization Pro is an “agent-less” tool which requires no installation on the target devices (host servers or workstations) — it runs from a single PC with network access. It talks to the target host via Application Program Interfaces (APIs), Remote Procedure Calls (RPCs), or read-only display commands depending on the remote host’s operating system. Datatrend Optimization Pro communicates directly with the target devices to collect detailed information for an inventory captured in a fully relational database.

The architecture

The program is divided into four modules, each of which supports a basic question:
 

Discovery

This module uses two different approaches to finding infrastructure objects: Fingerprinting and SNMP Discovery. Fingerprinting provides the capability to scan IP address ranges, determine which IP addresses are in use by active devices, and provide one or more “educated guesses” as to what the device is.

As an alternative, Simple Network Management Protocol (SNMP) can be used to interrogate and gather information about IP devices (including hosts) attached to IP networks. (SNMP is the Internet-wide standard for data collection between two machines.)

Credentialed Inventory

Once the object with a specific IP address has been discovered, it must be verified by presenting authorized credentials such as user name and password. Credentialed inventory is complete after successfully logging on to an administrative account on the host and executing APIs or RPCs to retrieve data about the server. Physical assets of both the host and storage objects are included in the inventory.

Utilization

This module monitors performance to collect utilization data that is stored in the Datatrend Optimization Pro database. All the ports in an infrastructure are monitored for traffic every 15 minutes over a six-week period (a period of time which follows most business cycles).

Utilization polls the hosts and returns data including which processes are running, disk I/O, and counters (user-defined categories). The results of this performance monitoring, executed on the most recently closed Run, is stored in the Datatrend Optimization Pro database for use in the Consolidation module.

Consolidation

This newest module, known as the SCON (Server Consolidation) Wizard, automates much of the effort and eliminates most of the uncertainty of designing a new server configuration. It does so by allowing "what if" consolidation scenarios that include any combination of:

• physical or "bare metal" servers (each boots off an operating system)

• logically-partitioned servers (LPARs, VMware)

• Blade or rack-based servers.

The SCON Wizard provides the capability to create one or more Consolidation Plans.   A plan consists of a set of selected host computers and their utilization data, and one or more Consolidation Scenarios. A scenario consists of a target server configuration, a set of existing host configurations and planned workloads for the target, as well as  a forecast of the target server’s resulting utilization. The SCON Wizard supports the full range of platforms for the targets including Windows, UNIX, Linux, VMware ESX, HP Superdome, IBM System p, and Sun E-Series.

Here are the main steps the Consolidation module guides a user through (Step 6 is shown and explained in more detail with the screen capture below):

1. Create a new consolidation plan

2. Select existing host computers for consolidation analysis

3. Load resource utilization data

4. Create a consolidation scenario

5. Select target host or configuration

6. Select consolidation candidates

7. Optimize consolidation scenario

8. Review and analyze projected target utilization

9. Print consolidation plan.

10. Save plan and scenario/s for later use.

Step 6: Candidates are selected for consolidation by dragging them from the staging area grid to the (green) consolidation candidate pane of the scenario window.

As the candidate servers and their workloads are consolidated onto the target server, their icons are displayed in the consolidation candidate pane and the target resource utilization meters are updated with the calculated values.

Each host computer row in the staging area will be flagged as selected with a red row label.

... on assessing your technology infrastructure

TechTip: Are your tape based mksysb backups viable?

By Mark Neuman, Technical Services Manager, Datatrend Technologies

Installing Optional Programs

During the install of AIX, you have the option (option 3, More Options) to modify many of the characteristics of the installed system. You will be able to have the install process add different optional packages of additional software during the BOS (Base Operating System) install, such as the Mozilla browser. Having these optional programs installed via this vector can save you problems down the line. For example, if you were to install the Mozilla browser at a later time, you would need to install all of its prerequisites first - some of which carry prerequisites of their own. For this reason, during this phase of the BOS install, it is much easier to have the system manage this aspect of the install.

"Enable System Backups to Install on any System"

One item to notice during the BOS install is selection #6, on the first page of selections, "Enable system backups to install on any system." With AIX 4.X, the default install would only install the device driver file sets for the hardware that was discovered on the system at install time. With AIX 5.X, that behavior has been changed to default install all device driver file sets. The result of this is that with AIX 4.X, you had to go through some extra gyrations to clone a system using the mksysb image from a different system if it contained any "new" devices.

This was a special problem if the primary storage device was different and the appropriate device filesets were not on the mksysb. With the default behavior of AIX 5.X, this problem should not exist, except for any devices released after the original install. Datatrend recommends changing this option to "No" in order to prevent issues that may come up with tape based mksysb described below.

The drawback to the default behavior is that it takes somewhat more disk to hold these additional filesets; but more important, it will cause a tape based mksysb to fail during the install. You will see a short warning during the creation of the mksysb tape, if you look for it. It will create the tape and complete the mksysb with a successful return code, but the boot block will be larger than the 12MB limit for tape based boot module. This is due to a firmware limitation on the restore side and, while there are some firmware hacks that will allow you to get around this, it is much easier to avoid this in the first place.

Alternatively, you can boot the system with the appropriate level of AIX from a CD and then complete the restore from the tape - the easiest way of recovering from this problem. You can check if this is an issue for your system by executing the following procedure - again, this is for tape based mksysb backups only.

• Using a file system with at least 50 Mb of free space execute the following command "bosboot -ad /dev/rmt0 -b /path_to/tape.boot.image -M normal"

• The system will return something like this response "bosboot: Boot image is 19438 512 byte blocks." and may even return the following warning:

***** ATTENTION *****
The boot image you just created might fail to boot because
the size exceeds the system limit. For information about
fixes or workarounds, see /usr/lpp/bos.sysmgt/README.
***** ATTENTION *****

• If the boot image size is larger than 24000 blocks and/or you get the warning then the mksysb backups made to a tape device may not directly boot from tape.

• If you get the responses as shown above, you can go through your list of device packages and remove the devices that do not exist on your system, which will reduce the size of the boot block.

• Once you are below the 24000 block limit on the boot block, the tape based mksysb should behave normally during a restore.

Please be very careful that the device packages being removed are from truly non-existent adapters so as to not disrupt the operation of your system.

Other than recommending that the Mozilla browser be changed to "Yes" and having the BOS install process manage the install of that product, Datatrend suggests that the rest of the selections be left in their default state. Note that the install process will ask for the "AIX Linux Toolbox for Linux" CD and the "Mozilla" CD at the end of the BOS install, so be sure to have them handy.

Contact us | Visit Datatrend website

All trademarks, registered trademarks and service marks are the property of their respective owners.
IBM, the IBM logo and other referenced IBM products and services are trademarks
or registered trademarks of the International Business Machines Corporation in the United States,
other countries, or both. All rights reserved. Windows is property of Microsoft Corporation.

UNIX is a registered trademark of The Open Group. Linux is a registered trademark of Linus Torvalds.

Brought to you by Datatrend Technologies Inc.

6815 Meadowridge Court, Alpharetta, GA 30005