The Elusive High Availability in the Digital Age

Well, the summer is over, even if we have had great weather into September. My apologies for the delay in a new post, and I know I have several topic requests to fulfill ūüôā Given our own journey at Danske Bank on availability, I thought it was best to re-touch this topic and then come back around to other requests in my next posts. Enjoy and look forward to your comments!

It has been a tough few months for some US airlines with their IT systems availability. Hopefully, you were not caught up in the major delays and frustrations. Both Southwest and Delta suffered major outages in August and September. Add in power outages affecting equipment¬†and multiple airlines recently in Newark, and you have many customers fuming over delays and cancelled flights. And the cost to the airlines was huge — Delta’s outage alone is estimated at $100M to $150M and that doesn’t include¬†the reputation impact. And such outages are not limited to the US airlines, with British Airways also suffering a major outage in September.¬†Delta and Southwest are not unique in their problems, both United and American suffered major failures and widespread impacts in 2015. Even with large IT budgets, and hundreds of millions invested in upgrades over the past few years, airlines are struggling to maintain service in the digital age. The reasons are straightforward:

  • At their core, services¬†are based on antiquated systems that have been partially refitted and upgraded over decades (the core reservation system is from the 1960s)
  • Airlines have struggled earlier this decade to make a profit due to oil prices, and minimally invested in the IT systems to attack the technical debt. This was further complicated by multiple integrations that had to be executed due to mergers.
  • As they have digitalized their customer interfaces and flight checkout procedures, the previous manual procedures are now backup steps that are infrequently¬†exercised and woefully undermanned when IT systems do fail, resulting in massive service outages.

With digitalization reaching even further into the customer interfaces and operations, airlines, like many other industries, must invest in stabilizing their systems, address their technical debt, and get serious about availability. Some should start with¬†the best practices in the previous post on¬†Improving Availability, Where to Start.¬†Others, like many IT shops, have decent availability but still have much to do to get to first quartile availability. If you have made good progress but realize that three 9’s or preferably four 9’s of availability on your key channels is critical for you to win in the digital age this post covers what you should do.

Let’s start with the foundation. If you can deliver consistently¬†good availability, then your team should¬†already understand:

  • Availability is about quality. Poor availability is a quality issue. You must have a quality culture that emphasizes quality as a desired outcome and doing things right if you wish to achieve high availability.
  • Most defects — which then cause outages — are injected by change. Thus, strong change management processes that identify and eliminate defects are critical to further reduce outages.
  • Monitor and manage to minimize impact. A capable command center with proper monitoring feeds and strong incident management practices may not prevent the defect from occurring but it can greatly reduce the time to restore and the overall customer impact. This directly translates into higher availability.
  • You must learn and improve from the issues. Your¬†incident management process must be coupled with a disciplined root cause analysis that ensures teams identify and correct underlying causes that will avoid future issues. This continuous learning and improvement is key to reaching high performance.

With this base understanding, and presumably with only smoldering areas of problems for IT shop left, there are excellent extensions that will enable your team to move to first quartile availability with moderate but persistent effort. For many enterprises, this is now a highly desirable business goal. Reliable systems translate to reliable customer interfaces as¬†customers access the heart of most companies systems now through internet and mobile applications, typically on a 7×24 basis. Your production performance becomes very evident, very fast to your customers. And if you are down, they cannot transact, you cannot service them, your company loses real revenue, and more importantly, damages it’s reputation, often badly. It is far better to address these problems and gain a key edge in the market by consistently meeting or exceeding costumer availability expectations.

First, if you have moved up from regularly fighting fires, then just because outages are not everyday, does not mean that IT leadership no longer needs to emphasize quality. Delivering high quality must be core to your culture and your engineering values. As IT leaders, you must continue to reiterate the importance of quality and demonstrate your commitment to these values by your actions. When there is enormous time pressure to deliver a release, but it is not ready, you delay it until the quality is appropriate. Or you release a lower quality pilot version, with properly set customer and business expectations, that is followed in a timely manner by a quality release. You ensure adequate investment in foundational quality by funding system upgrades and lifecycle efforts so technical debt does not increase. You reward teams for high quality engineering, and not for fire-fighting. You advocate inspections, or agile methods, that enable defects to be removed earlier in the lifecycle at lower cost. You invest in automated testing and verification that enables work to be assured of higher quality at much lower cost. You address redundancy and ensure resiliency in core infrastructure and systems. Single power cord servers still in your data center? Really?? Take care of these long-neglected issues. And if you are not sure, go look for these typical failure points (another being SPOF network connections). We used to call these ‘easter eggs’, as in the easter eggs that no one found in a preceding year’s easter egg hunt and then you find the old, and quite rotten, easter egg on your watch. It’s no fun, but it is far better to find them before they cause an outage.

Remember that quality is not achieved by not¬†making mistakes — a zero defect goal is not the target — instead, quality is achieved by a continuous improvement approach where defects are analyzed and causes eliminated, where your team learns and applies best practices. Your target goal should be 1st quartile quality for your industry, that will provide competitive advantage. ¬†When you update the goals, also¬†revisit and ensure you have aligned the rewards of your organization to match these quality goals.

Second, you should build on your robust change management process. To get to median capability, you should have already established clear change review teams, proper change windows and moved to deliveries through releases. Now, use the data to identify which groups are late in their preparation for changes, or where change defects are clustered around and why. These understandings can improve and streamline the change processes (yes, some of the late changes could be due to too many approvals required for example). Further clusters of issues may be due to specific steps being poorly performed or inadequate tools. For example, often verification is done as cursory task and thus seldom catches critical change defects. The result is that the defect is then only discovered in production, hours later, when your entire customer base is trying but cannot use the system. Of course, it is likely such an outage was entirely avoidable with adequate verification because you would have known at the time of the change that it had failed and could have take action then to back out the change. The failed change data is your gold mine of information to understand which groups need to improve and where they should improve. Importantly, be transparent with the data, publish the results by team and by root cause clusters. Transparency improves accountability. As an IT leader, you must then make the necessary investments and align efforts to correct the identified deficiencies and avoid future outages.

Further, you can extend the change process by introducing production ready.  Production ready is when a system or major update can be introduced into production because it is ready on all the key performance aspects: security, recoverability, reliability, maintainability, usability, and operability. In our typical rush to deliver key features or products, the sustainability of the system is often neglected or omitted. By establishing the Operations team as the final approval gate for a major change to go into production, and leveraging the production ready criteria, organizations can ensure that these often neglected areas are attended to and properly delivered as part of the normal development process. These steps then enable a much higher performing system in production and avoid customer impacts. For a detailed definition of the production ready process, please see the reference page.

Third, ensure you have consolidated your monitoring and¬†all significant¬†customer impacting problems are routed through an enterprise command center via an effective¬†incident management process. An Enterprise Command Center (ECC) is basically an enterprise version of a Network Operations Center or NOC, where all of your systems and infrastructure are monitored (not just networks). This modern¬†ECC also has capability to facilitate and coordinate triage and resolution efforts for production issues. An effective ECC can bring together the right resources from across the enterprise and supporting vendors to diagnose and fix production issues while providing communication and updates to the rest of the enterprise. Delivering highly available systems requires an investment into an ECC and the supporting diagnostic and monitoring systems. Many companies have partially constructed the diagnostics or have siloed war rooms for some applications or infrastructure components. To fully and properly handle production issues these capabilities must be consolidated and integrated. Once you have an integrated ECC, you can extend it by moving from component monitoring to¬†full channel monitoring. Full channel monitoring is where the entire stack for a critical customer channel (e.g. online banking for financial services or customer shopping for a retailer) has been instrumented so that a comprehensive view can be continuously monitored within the ECC. The instrumentation is such that not only are all the infrastructure components fully monitored but the databases, middleware, and software components are instrumented as well. Further, proxy transactions can and are run at a periodic basis to understand performance and if there are any issues. This level of instrumentation requires considerable investment — and thus is normally done only for the most critical channels. It also requires sophisticated toolsets such as AppDynamics. But full channel monitoring enables immediate detection of issues or service failures, and most importantly, enables very rapid correlation of where the fault lies. This rapid correlation can take incident impact from hours to minutes or even seconds. Automated recovery routines can be built to accelerate recovery from given scenarios and reduce impact to seconds. If your company’s revenue or service is highly dependent on such a channel, I would highly recommend the investment. A single severe outage that is avoided or greatly reduced can often pay for the entire instrumentation cost.

Fourth, you cannot be complacent about learning and improving. Whether from failed changes, incident pattern analysis, or industry trends and practices, you and your team should always be seeking to identify improvements. High performance or here, high quality, is never reached in one step, but instead in a series of many steps and adjustments. And given our IT systems themselves are dynamic and changing over time, we must be alert to new trends, new issues, and adjust.

Often, where we execute strong root cause and followup, we end up focused only at the individual issue or incident level. This can be all well and good for correcting the one issue, but if we miss broader patterns we can substantially undershoot optimal performance. As IT leaders, we must always consider the trees and the forest. It is important to not just get focused on fixing the individual incident and getting to root cause for that one incident but to also look for the overall trends and patterns of your issues. Do they cluster with one application or infrastructure component? Does a supplier contribute far too many issues? Is inadequate testing a common thread among incidents? Do you have some teams that create far more defects than the norm? Are your designs too complex? Are you using the products in a mainstream or unique manner Рespecially if you are seeing many OS or product defects? Use these patterns and analysis to identify the systemic issues your organization must fix. They may be process issues (e.g. poor testing), application or infrastructure issues (e.g., obsolete hardware), or other issues (e.g., lack of documentation, incompetent staff). Discuss these issues and analysis with your management team and engineering leads. Tackle fixing them as a team, with your quality goals prioritizing the efforts. By correcting things both individually and systemically you can achieve far greater progress. Again, the transparency of the discussions will increase accountability and open up your teams so everyone can focus on the real goals as opposed to hiding problems.

These four extensions to your initial efforts will set your team on a course to achieve top quartile availability. Of course,¬†you must couple these efforts with diligent engagement by senior management, adequate investment, and disciplined execution.¬†Unfortunately, even with all the right measures,¬†providing robust availability for your customers is rarely a straight-line improvement. It is a complex endeavor that requires persistence and adjustment along the way. But by implementing these steps, you can enable sustainable and substantial progress and achieve top quartile performance to provide¬†business advantage in today’s 7×24 digital world.

If your shop is struggling with high availability or major outages, look to apply these practices (or send your CIO the link to this page ūüôā ).

Best, Jim Ditmore

Posted in Best Practices, Infrastructure Engineering, Uncategorized, Vision and Leadership, World Class Production Availability | Tagged , , | 6 Comments

Infrastructure Engineering – Leveraging a Technology Plan

Our recent post discussed using the Infrastructure Engineering Lifecycle (IELC) to enable organizations to build a modern, efficient and robust technology infrastructure. One of the key expressions that both leverages and IELC approach and helps an infrastructure team properly plan and navigate the cycles is the Technology Plan. Normally, the technology plan is constructed for each major infrastructure ‘component’ (e.g. network, servers, client environment, etc). A well-constructed technology plan creates both the pull – outlining how the platform will meet the key business requirements and technology objectives and the push – reinforcing proper upkeep and use of the IELC practice.

Digitalization continues to sweep almost every industry, and the ability of firms to actually deliver the digital interfaces and services requires a robust, modern and efficient infrastructure. To deliver an optimal technology infrastructure, one must utilize¬†an ‚Äėevergreen‚Äô approach and maintain an appropriate technology pace matching the industry. Similar to a dolphin riding the bow wave of a ship, a company can optimize both the feature and capability of its infrastructure and minimize its cost and risk by staying consistently just off the leading pace of the industry. Often companies make the mistake of either surging ahead and expending large resources to get fully leading technology or eking out and extending the life of technology assets to avoid investment and resource requirements. Neither strategy actually saves money ‚Äėthrough the cycle‚Äô and both strategies add significant risk for little additional benefit.

For those companies that choose to minimize their infrastructure investments and reduce costs by overextending asset lives, they typically incur greater additional costs through higher maintenance, greater fix resources required, and lower system performance (and staff productivity). Obviously, extending your desktop PC refresh cycle from 2 years to 4 years is workable and reasonable, but extending the cycle much beyond this and you quickly run into:

  • Integration issues ‚Äď both internal and external compatibility as your clients and partners have newer versions of office tools that are incompatible with yours
  • potentially higher maintenance costs as much hardware has no maintenance cost for the first 2 or 3 years, and increasing costs in subsequent years
  • greater environmentals costs as power and cooling savings from newer generation equipment is not realized
  • longer security patch cycles for older software (though some benefit as it is also more stable)
  • greater complexity and resulting cost within your environment as you must integrate 3 or 4 generations of equipment and software versus 2 or 3 versions
  • longer incident times as the usual first vendor response to an issue is ‚Äėyou need to upgrade to the latest version of the software before we can really fix this defect‚Äô

And if you press the envelope further and extend infrastructure life to the end of the vendor’s life cycle or beyond, expect significantly higher failure rates, unsupported or expensively support software, and much higher repair costs. In my experience, where multiple times we modernized an overextended infrastructure, we were able to reduce total costs by 20 or 30%, and this included the costs of the modernization. In essence you can run 4 servers from 3 generations ago on 1 current server, and having modern PCs and laptops means far less service issues, fewer service desk calls, far less breakage (people take care of newer stuff) and more productive staff.

For those companies that have surged to the leading edge on infrastructure, they are typically paying a premium for nominal benefit. For the privilege of being first, frontrunners encounter an array of issues including:

  • Experiencing more defects – trying out the latest server or cloud product or engineered appliance means you will find far more defects.
  • Paying a premium – being first with new technology means typically you will pay a premium because it is well before the volumes and competition can kick in to drive better pricing.
  • Integration issues ‚Äď having the latest software version often means third party utilities or extensions have not yet released their version that will properly work with the latest software
  • Higher security flaws ‚Äď all the backdoors and gaps have been uncovered yet as there are not enough users. Thus, hackers have a greater opportunity to find ‚Äėzero day‚Äô flaws and exploit them to attack you

Typically, those groups that I have inherited that were on the leading edge, were doing so because they had either an excess of resources or were solely focused on technology product(and not business needs). There was inadequate dialogue with the business to ensure the focus was on business priorities versus technology priorities. Thus, the company was often expending 10 to 30% more for little tangible business benefit other than to be able to state they were ‚Äėleading edge‚Äô. In today‚Äôs software world, seldom does the latest infrastructure provide compelling business benefit over above that of a well-run modern utility infrastructure. Nearly all of the time the business benefit is derived by compelling services and features enabled by the application¬†software running on the utility. Thus, typically the shops that are tinkering with leading edge hardware or are always on the latest version first are shops that are doing hobbies disconnected from the business imperatives. Only where organizations are operating at massive scale or actually providing infrastructure services as a business does leading edge positioning make business sense.

So, given our objective is to be in the sweet spot riding the industry bow wave, then a good practice to ensure proper consistent pace and connection to the business is a technology plan for each of the major infrastructure components that incorporates the infrastructure engineering lifecycle. A technology plan includes the infrastructure vision and strategy for a component area, defines key services provided in business terms, and maps out an appropriate trajectory and performance for a 2 or 3 year cycle. The technology plan then becomes the roadmap for that particular component and enables management to both plan and track performance against key metrics as well as ensuring evolution of the component with the industry and business needs.

The key components of the technology plan are:

  1. Mission, Vision for that component area
  2. Key requirements/strategy
  3. Services (described in business terms)
  4. Key metrics (definition, explanation)
  5. Current starting point – explanation (SWOT) ‚Äď as needed by service
  6. Current starting point ‚Äď Configuration ‚Äď as needed by service
  7. Target ‚Äď explanation (of approach) and configuration — also defined by service
  8. Metrics trajectory and target (2 to 3 years)
  9. Gantt chart showing key initiatives, platform refresh or releases, milestones (can be by service)
  10. Configuration snapshots at 6 months (for 2 to 3 years, can be by service)
  11. Investment and resource description
  12. Summary
  13. Appendices
    1. Platform Schedule (2 -3 years as projected)
    2. Platform release schedule (next 1 -2 years, as projected)
    3. Patch cycle (next 6 ‚Äď 12 months, as planned)

The mission and vision should be derived and cascaded from the overall technology vision and corporate strategy. It should emphasis key tenets of the corporate vision and their implication for the component area. For example if the corporate strategy is to be ‘easy to do business with’ then the network and server components must support a highly reliable, secure and accessible internet interface. Such reliability and security aspirations then have direct implications on component requirements, objectives and plans.

The services portion of the plan should translate the overall component into the key services provided to the business. For example, network would be translated into data services, general voice services, call center services, internet and data connection services, local branch and office connectivity, wireless and mobility connectivity, and core network and data center connectivity. The service area should be described in business terms with key requirements specified. Further, each service area should then be able to describe the key metrics to be used to gauge its performance and effectiveness. The metrics could be quality, cost, performance, usability, productivity or other metrics.

For each service area of a component, the plan is then constructed. If we take the call center service as the example, the current technology configuration and specific services available would define the current starting point. A SWOT analysis should accompany the current configuration explaining both strengths and where the services falls short of business needs. The the target is constructed where both the overall architecture and approach are described as well as the target configuration (high to medium level of definition) is provided (e.g. where will the technology configuration for that area be in 2 or 3 years).

Then, given the target, the key metrics are mapped from their current to their future levels and a trajectory established that will be the goals for the service over time. This is subsequently filled out with a more detailed plan (Gantt chart) that shows the key initiatives and changes that must be implemented to achieve the target. Snapshots, typically at 6 month intervals, of the service configuration are added to demonstrate detailed understanding of how the transformation is accomplished and enable effective planning and migration. Then the investment and resource needs and adjustments are described to accompany the technology plans.

If well done, the technology plan then provides an effective roadmap for the entire technology component team to both understand how what they do delivers to the business, where they need to be, and how they will get there. It can be an enormous assist for productivity and practicality.

I will post some good examples of technology plans in the coming months.

Have you leveraged plans like this previously? If so, did they help? Would love to to hear from you.

All the best, Jim Ditmore


Posted in Best Practices, Infrastructure Engineering, Metrics and Process Improvement, Platforms, Vision and Leadership, World Class Production Availability | Tagged , , | 6 Comments

Digitalization and Its Broader Impacts

We have discussed several times in the past few years the impacts of digitalization on the corporate landscape with a particular focus on the technology industry. We have also explored the key constraints of organizations to keep pace and to innovate. But the recent pace of digitalization is causing rapid change in corporations and has implications for broader and more radical change at a societal level. These are certainly important topics for the IT leader, and there are more significant implications for the broader society.

It is perhaps relatively easy to consider¬†the latest technology innovations as additional¬†steps or an extension of what we have seen since the internet era began in the mid-90s. And it was nearly 20 years ago when Deep Blue beat Gary Kasparov in chess, so the advances in machine intelligence could be viewed as incremental and measured in decades. But the cumulative effect of technology advances in multiple realms over the past 40 years has now turned into not just a quantative¬†acceleration but a qualitative leap as well. This leap¬†is well explained¬†in The Second Machine Age by¬†Erik Brynjolfsson where he discusses the cumulative effects of exponential growth over time even from a small base. This is the essence of the power of digitalization. But where Erik portrays ‘a brilliant’ future with man and advanced machine working together, Martin Ford in his book,¬†The Rise of Robots,¬†sees a very troubled¬†world with massive unemployment and inequality.

As an IT leader in a corporation, you must ensure the competitiveness of your firm by leveraging the new technologies available at a pace that keeps you abreast or better, ahead of your competitors. It is a relentless pace across industries, driven not only by traditional competition but also by new digital competitors that did not even exist a few years prior. And every old line firm is driven by the fear of becoming another Kodak, while firms on top of their industry worry they will be another Nokia. For those firms able to keep pace and leverage the technologies, they are seeing substantially reduced costs of production, with significant qualitative advances. These advantages will occur across¬†industries as¬†digitalization is revolutionizing even ‘physical’ industries like logging and sawmills. But where does this leave communities and society in general? Is it a brilliant or troubled future?

Let’s explore some digitalization scenarios in different industries to shed light on the longer term. Shipping and logistics is an excellent example where near term there continue to be¬†significant improvements due to digitalization in shipment tracking, route management, navigation, and optimization of loads. Leveraging sensors and interconnected planning, scheduling and communication software can result in greatly improved shipment fill rates while increasing shipment visibility. In the next 5 years, the most advanced shipping firms will completely eliminate paper from the shipment chain and have integrated end-to-end digital processes. These fully digital processes will enable more timely shipment and¬†distribution while reducing errors and enabling greater knowledge and thus flexibility to meet emerging demands. They will also reduce the manual labor of administration – and with embedded sensors, reduce the need for intermediate checkpoints. The introduction of robotics in distribution warehouses (such as Amazon’s) currently greatly extends the productivity of the workers by having the robots run the floor and pick the product, bringing it back to the worker. The current generation of robots provide a 30% productivity gain. The next one – within 5 years, could expect perhaps a further 30% or even 50%? Amazon certainly made the investment by buying it’s own robotics company (Kiva) not just for its warehouses, but perhaps to build a relentlessly productive distribution chain able to deliver for everything (and not very dissimilar to their cloud foray). While the distribution center is being automated with robot assistants, within 15 years we will see commercial trucking move to highly autonomous trucks. Not unlike how commercial pilots today work with the autopilot. This could be good news as in the US alone, trucks are involved in over 300,000 accidents and cause more than 4500 deaths each year. It would be a tremendous benefit to society to dramatically reduce such negative impacts through autonomous or mostly autonomous trucks. Robots do not fall asleep at the wheel and do not drive aggressively in rush hour traffic. Drivers will become mostly escorts and guardians for their shipments while robots will handle nearly all of the monotonous driving chores. Convoying and 24 hour driving will become possible, all the while enabling greater efficiency and safety. And within 10 years, expect the shipment from the warehouse to the customer for small package goods to change dramatically as well. Amazon unveiled it’s latest delivery drone¬†and while it will take another 2 generations of work to make it fully viable (and of course FAA approval), when ready it will make a huge impact on how goods are delivered to customers and enable Amazon to compete fully with retail stores, fulfilling¬†same day if not same hour delivery.¬†In the US, the trucking industry overall employs about 8 million, with 3.5 million of those being truck drivers. So whether a scheduler, distribution clerk or truck driver, it is likely these positions will be both greatly changed and fewer in 10 to 12 years. Just these impacts alone would likely reduce labor requirements by 20 or 30% in 10 years and possibly 50% in 15 years. But there is the increasing volume¬†effect where digitalization is causing more rapid and smaller shipments as customers order goods online that are then rapidly delivered to their home, thus potential increasing demand (and required labor) over the longer term. Yet these effects will not overcome the reductions — expect a reduction of 20% of shipping and logistics¬†labor where humans partner with robot assistants and autonomous vehicles as the normal operating mode. And¬†increased demand in direct to customer shipments will come at a cost to the retail industry. Already online sales have begun to exceed in-store sales. This trend will continue resulting¬†much lower¬†retail employment as more and more commerce moves online and stores that do not offer an ‘experience’ lose out. It is reasonable to expect retail employment to peak around¬†5.3M (from the current 4.8M) ¬†in the next 5 years and then slowly decline over the following 10 years.

Manufacturing, which has leveraged the use of robotics for four decades or more, is seeing ever greater investments in machines, even in lower wage countries like China and India. Once only the domain of large companies and precisely designed assembly lines, the relentless reduction of the cost of robotics with each generation, and their increasing ease of use, is making it economical for smaller firms to leverage such technology in more flexible ways. The pace of progress in robotics has become remarkable. In another two generations of robotics, it will be unthinkable to be a manufacturer and NOT leverage robotics. And if you combine¬†robotics with the capabilities¬†of 3D printing, the changes become even greater. So the familiar patterns of moving plants to where there are lower wages will no longer occur. Indeed, this pattern which has repeated¬†since the first industrial revolution started in Britain is already being broken. Factories in China and India are being automated, not expanded or moved to lower cost countries or regions. And some manufacturing is being greatly automated and moved¬†back to high cost regions to be closer to demand to enable greater flexibility, better time to market, and control. The low cost manufacturing ladder, which has lifted so much of society out of poverty in the past two centuries is being pulled away, with great implications for those countries either not yet on the developing curve, or just starting. This ‘premature de-industrialization’ may forever remove the manufacturing ladder to prosperity for much of India and for¬†many African and Asian countries still in great poverty. ¬†And while these trends will require drive more designers and better creative services, the overall¬†manufacturing employment will continue its long term decline. Perhaps it will be partly offset with an explosion of small, creative firms able to compete against larger, traditional firms. But this will occur only the most developed regions of the globe. For the 12 million manufacturing jobs in the US, expect to see a very slight uptick even as factories are brought back¬†due to re-shoring in an automated format and the growth of smaller, highly automated factories leveraging 3D printing. But globally, one should expect to see a major decline in manufacturing jobs as robots take over the factories of the developing world.

And whither the restaurant business? McDonald’s is experimenting with self-service kiosks and robots making hamburgers, and new chains from Paris to San Francisco are re-inventing the automat – staple of the mid-1900s. While per store labor has declined by 30 to 50% in the past 50 years, there is potential for acceleration given the new skills of robots and the increasing demand for higher wages for low level employees. These moves, combined with easy-to-use mobile apps to order your food ahead of time likely means fewer jobs in 10 years, even with more meals and sales.¬†One should expect the return of the 1950s ‘automat’ in the next 5 or 10 years as restaurants leverage a new generation of robots that are far more capable than their 1950s predecessors.

Just a quick review of a handful of¬†major employment industries shows at best a mixed forecast of jobs and possibly a stronger negative picture. For developed nations, it will be more mixed, with new jobs also appearing in robotics and technology as well as the return of some manufacturing work and perhaps volume increases in other areas. But globally, one can expect a significant downward trend over the next 10 to 15 years. And the spread between the nations that have industrialized and those that haven’t will surely widen.

What jobs will increase? Obviously, technology-related jobs will continue to increase but these are a very small portion of the total pool. More significantly, any profession that produces content, from football players to musicians to filmmakers will see continued demand for their products as digitalization drives greater consumption through ever more channels. But we have also seen for content producers that this is a ‘winner take all’ world, where only the very best reap most of the rewards and the rest have very low wages.

Certainly as IT leaders, we must leverage technology wherever possible to enable our firms to compete effectively in this digital race. As leaders, we are all familiar with the challenges of rapid change. Especially at this pace, change is hard — for individuals and for organizations. We will also need to be advocates for smarter change, by helping our communities understand the coming impacts, enabling our staff to upscale to better compete and achieve a better livelihood, and¬†advising for better government and legislature. If all taxes and social structures make the employee far more expensive than the robot, than shouldn’t we logically expect the use of robots to accelerate? Increasing the costs of labor (e.g., the ‘living wage’ movement in the US) is actually more likely to hasten the demise of jobs! ¬†Perhaps it would be far better to tax the robots. Or even better, in twenty years, every¬†citizen will get their own robot – or perhaps two: one to send in to do work and one to help out at home. The future is coming quickly, let’s strive to adjust fast enough for it.

What is your view of the trends in robotics? What do you see as the challenges ahead for you? for your company? for your community?

Best, Jim Ditmore


Posted in Efficiency and Cost Reduction, Looking Ahead, Vision and Leadership | Tagged , , , | 7 Comments

The Infrastructure Engineering Lifecycle – How to Build and Sustain a Top Quartile Infrastructure

There are over 5,000 books on applications development methods on Amazon. There are¬†dozens of industry and government standards that map out the methodologies for application development. And for IT operations and IT production processes like problem and change management, IT Service Management and ITIL standards provide excellent guidance and structure. Yet for the infrastructure systems on which the applications rely on fully, there is scarcely a publication which outlines the approaches organizations¬†should use to build and sustain a robust infrastructure. ITIL ventures slightly in this area but really just re-defines a waterfall application project cycle in infrastructure terms. During many years of building, re-building, and sustaining top quartile infrastructure I have developed a life cycle methodology for infrastructure or ‘Infrastructure Engineering Life Cycle’ (IELC).

The importance of infrastructure should not be overlooked in our digital age. Not only have customer expectations increased for services where they expect ‘always on’ web sites and transaction capabilities, but they also require quick response and seamless integration across offerings. Certainly the software is critical to provide the functionality, but none of these services can be reliably and securely provided without a well-built¬†infrastructure underpinning all of the applications. A top quartile infrastructure delivers¬†outstanding reliability (on the order of 99.9% or better availability), zippy performance, excellent unit costs, all with robust security and resiliency.

Often enterprises make the mistake of addressing infrastructure only when things break, and they only fix or invest enough to get things back running instead of re-building correctly a modern plant. It is unfortunate because not only will they likely experience further outages and service impacts but also their full infrastructure costs are likely to be higher for their dated, dysfunctional plant than for an updated, modern plant. Unlike most assets, I have found that a modern, well-designed IT infrastructure is cheaper to run than a poorly maintained plant that has various obsolete or poorly configured elements. Remember that every new generation of equipment can basically do twice as much s the previous so  you have fewer components, less maintenance, less administration, less things that can go wrong. In addition, a modern plant also boosts time to market for application projects and reduces significantly the portion of time spent on fixing things by both infrastructure and application engineers.

So, given the critical nature of well-run technology infrastructure in the world of digitalization, how do enterprises and CIOs build and maintain a modern plant with outstanding fit and finish? It is not just about buying lots of new equipment, or counting on a single vendor or cloud provider to take care of all the integration or services. Nearly all major enterprise have a legacy of systems that result in complexity and complicate the ability to deliver reliable services or keep pace with new capabilities. These complexities can rarely be handled by a systems integrator or single service provider. Further, a complete re-build of the infrastructure often requires major capital investment and and can put the availability even further at risk. The best course is usually then is not to go ‘all-in’ where you launch a complete re-build or hand over the keys to a sole outsourcer, but instead to take a ‘spiral optimization’ approach which addresses fundamentals and burning issues first, and then uses the newly acquired capabilities to advance and address more complex or less pressing remaining issues.

A repeated, closed cycle¬†approach (‘spiral optimization’) is our management approach. This management approach is coupled with an Infrastructure Engineering Lifecycle (IELC) methodology to build top quartile infrastructure. For the first cycle of the infrastructure rebuild, it is important to address the biggest issues. Front and center, ¬†the entire infrastructure team must focus on quality. Poorly designed or built infrastructure becomes a blackhole of engineering time as rework demands grow with each failure or application built upon a teetering platform. And while it must also be understood that a everything cannot be fixed at once, those things that are undertaken, must be done with quality. This includes documenting the systems and getting them correctly into the asset management database. And it includes coming up with a standard design or service offering if none exists. Having 5000 servers must be viewed as a large¬†expense requiring great care and feeding — and the only thing worse is having 5000 custom servers because your IT team did not take the time to define the standard, keep it up to date, and maintain and patch it consistently. 5000 custom servers are¬†a massive expense that likely cannot be effectively and efficiently maintained or secured by any team. There is no cheaper time than the present moment to begin standardizing and fixing the mess by requiring that the next server built or significantly updated be done such that it becomes the new standard. Don’t start this effort though until you have the engineering capacity to do it. A standard design done by lousy engineers is not worth the investment. So, as an IT leader, while you are insisting on quality, ensure you have adequate talent to engineer your new standards. If you do not have it on board, leverage top practitioners in the industry to help your team create the new designs.

In addition to quality and starting to do things right, there are several fundamental practices that must be implemented. Your infrastructure engineering work should be guided by the infrastructure engineering lifecycle Рwhich is a methodology and set of practices that ensure high quality platforms that are effective, efficient, and sustainable.

The IELC covers all phases of infrastructure platforms Рfrom an emerging platform to a standard to declining and obsolete platforms. Importantly, the IELC is comprised of three cycles of activity that recognize that infrastructure requires constant grooming and patching where inputs come typically from external parties, and, all the while, technology advances regularly occur such that over 3 to 10 years nearly every hardware platform becomes obsolete and should and must be replaced. The three cycles of activity are:

  • Platform – This is the foundational lifecycle activity where hardware and utility software is defined, designed and integrated into a platform to perform a particular service. Generally, for medium and large companies, this is ¬†a 3 to 5 year lifecycle. A few examples could be a server platform, storage platform or an email platform.
  • Release – Once a platform is initial designed and implemented, then organizations should expect to refresh the platform on a regular basis to incorporate major underlying product or technology enhancements, address significant design flaws or gaps, and improve operational performance and reliability. Release should be planned for 3 to 12 month intervals over the life of the platform (which is usually 3 to 5 years).
  • Patch – A patch should also be employed where on a regular and routine basis, minor upgrades (both fixes and enhancements) are applied. The patch cycle should synchronize with both the underlying patch cycle of the OEM (Original Equipment Manufacturer) for the product and with the security and production requirements of the organization. Usually, patch cycles are used to incorporate security fixes and significant production defect fixes issued by the OEM. Typical patch cycles can be weekly to every 6 months.

Below is a diagram that represents the three infrastructure engineering life cycles and the general parameters of the cycles.

Infrastructure Engineering Cycles

Infrastructure Engineering Cycles

In subsequent posts, I will further detail key steps and practices within the cycles as well as provide templates that I have found to be effective for infrastructure teams.  As a preview, here is the diagram of the cycles with their activities and attributes.

IELC Preview

IELC Preview

What key practices or techniques have you used for your infrastructure teams to enable them to achieve success? I look forward to you thoughts and comments.

Best, Jim Ditmore


Posted in Best Practices, Cloud, Data Centers, Infrastructure Engineering, Platforms, World Class Production Availability | Tagged , , , | 2 Comments

The Recent Quarter Results Confirm Tech Industry Trends

Some surprising and not so surprising results for the tech industry this past quarter (2Q15) confirm both longer term industry trends and also high volatility for mismatches in expectations and performance.

First, Apple delivered strong growth in revenue and profits again (38% growth in profits to $10.8B), and yet, because it was slightly below expectations, lost $60 Billion in value. While Apple sold a record 47.5 million iPhones and saw Mac sales of 4.8 million units (up 9%), investors were apparently disappointed in both the number of iPhones sold and the lack of clear information on the iWatch. Even though it appears the iWatch is more successful at this point in the sales cycle than the iPad or iPhone, investors were apparently expecting a leadoff home run and sent the stock down 7% on the results.

And the reverse occurred for both Google and Amazon. Google delivered solid growth with 11% increase in revenue to $17.7B with net income of $3.9B which sent shares up 12%. Investors were surprised with the breadth of growth, particularly in mobile, and that managers showed some cost control. Amazon actually delivered some profit,¬†$214M on revenue of $29.33B, and showed continued robust¬†growth of 15%. Investors sent Amazon’s stock up on the profit results, a rarity given Amazon’s typical long term vision focus and ¬†willingness to spend for reach and scale even in areas well beyond its core.

What the quarterly results also reveal¬†is that¬†the tech platform companies (Amazon, Apple, Google) are continuing to be viewed as dominant but investors are uneasy about the long term stability of their platforms and thus have a quick trigger finger if they see any cracks in their future dominance. So, with Apple’s potential over-reliance on the iPhone, when there are fewer shipments than expected, or there is not clear evidence of a new platform extension¬†(e.g. iWatch) then investors react sharply. On the reverse, when Google appears to be overcoming the mobile threat to its core search platform, it is well-rewarded by investors.

What do the quarter’s results say about the tech product companies? Unless they have strong portfolio of winning products, it appears they will continue to struggle to regain form. IBM, AMD, HP and others all posted disappointing results as they grapple¬†with the technology revolutions in each of their industry sectors. AMD saw a 35% loss in revenue, dipping below $1B in quarterly revenue for the first time in years to $942M with a loss of $181M. Of course, the slow to declining PC sales worldwide is the primary cause and only console sales were healthy for AMD. Expect further difficult quarters as AMD adapts to the changing component industry (driven¬†by¬†impacts from the platform companies). HP continues a listless journey, its 2nd quarter reflecting a 7% slide in revenue from $27.3B to $25.5B, a 50% drop in operating margin, and a 10% drop in PC market shipments. While HP will split into two entities in November later this year which has some analysts upbeat, prospects look tough across all product segments with slow or declining growth except possibly enterprise software and 3D printing. IBM had mixed results, with better than expected profit on $20.81B in sales, yet saw continued revenue decline, which left investors nervous, sending the stock down. IBM did see strong growth in cloud services and analytics, but lackluster products and results in other core segments (e.g., hardware) which make up the vast bulk of IBM revenue¬†yielded¬†disappointing¬†revenue and profit showings. IBM recently sold off its low end server business as it views that sector becoming increasingly commoditized. Yet, IBM will continue to find that selling services when you have limited ‘winning’ products is a tough lower margin business. And cloud services are far lower margin business than its traditional hardware business – and one where Amazon and Google are first-comers with volume edges. IBM can certainly leverage its enterprise relationships and experience, but that is far easier to do when you have products that provide real advantage to the customer. Other than analytics (Watson) and some software areas, IBM lacks these winning products, having neglected their product pipeline (instead focusing on services) for many years. While the alliance with Apple provides some possibility of developing modern, vertical industry applications that will be compelling, there is far more IBM must do to get back on track and part of that innovation¬†must be in hardware.

EMC and Oracle are the exceptional large technology product companies that have been able to navigate the turbulent waters of their industry the past few years. Oracle did have weaker results this quarter, primarily due to currency fluctuations but also slowing software sales. Only EMC beat expectations and had new products overcome slowing demand for core areas.  Winning products for EMC like VMware and Pivotal as well as high demand for services and products in its information security division (RSA) and analytics more than overcame issues in the core storage division (which showed some recovery from 1Q). One could argue that with the VMWare franchise and leading engineered systems, EMC has established the strongest cloud platform, thus it has a more assured place with growth and margin in this rapidly changing sector.

The bottom line? Product companies will continue to struggle with revenue growth and margin pressure as technology advances undercut volumes and platform companies offer lower cost alternatives (e.g. public cloud options instead of proprietary server hardware and services, or smartphones instead of PCs) Unless technology product companies stay on the leading edge of innovating (or acquiring) compelling products, generating additional high margin revenues through services or software will be tough sledding. As we have mentioned here before, digitalization and the emergence of platform companies will result in more casualties in product companies – both in the tech space and outside it.

And of course, there is Microsoft. Microsoft¬†is in a unique spot where it still has a strong productivity platform (e.g. Office, Exchange) but a diminishing OS platform. And with only low margin business that are growing rapidly (e.g. cloud), the road back to dominance looks very tough. Further, their forays into other tech sectors have been middling at best and disastrous at worst. The second quarter results included an $8B write down of the Nokia acquisition, which was made two years¬†ago. The ‘devices and services’ strategy has shown to be¬†a ‘phenomenal error’¬†by some accounts. PC sales continue to decline, and Microsoft was unable to effectively crack the smartphone market. The past quarter revealed¬†declining revenue volume for phones even with 10% more volume as the only market segment MS gained traction was phone models at lower cost points. And it is hard to see that Samsung or other handset makers will add Windows OS to their product mix. Further, traditional Windows OS revenue (from OEMs) dropped 22%. The bright spots for MS were gaming (Xbox) and of course enterprise software and cloud services. There remain major concerns for the enterprise area where the rapidly growing¬†cloud services has far lower margin than their traditional software business. Microsoft should continue to worry that increasing import of dominance in the consumer space often translates later into winning business space – thus, ¬†the Google and Apple productivity platforms could be the long term trojan horses that blow up the enterprise cash cow for Microsoft. Microsoft may lose the war by trying to maintain its OS platform by limiting the reach of its¬†productivity platform¬†to consumers on their device of choice. Already, Google and Apple have changed the game by offering such software on the platforms for free, with free upgrades. Some assessments already show Microsoft lagging in feature without even considering its far higher cost. Windows 10 should be a solid hit for Microsoft, reversing some of the ground lost with Windows 8, but it will not dent the momentum of the Apple and Android platforms – especially when Microsoft introduces such new ways to monetize as the formerly free Solitaire’s lengthy advertisements or $9.99 annual subscription fee. They continue to misread the consumer market. Despite these continual missteps, or as recently called out in a New York Times article, their ‘feet of clay’, Microsoft has a strong enterprise business, a well-positioned productivity platform, and plenty of money. Can they figure out how to win in the consumer world while growing their productivity ecosystem with compelling extensions?

There remain multiple gaps that Microsoft, IBM, HP or even Oracle could exploit to win the next platform or obtain strong enterprise market share. While Apple and Android are pursuing the future car and the home platforms, the internet of things is still an open race. And there is opportunity given¬†that most of the gazillion apps in the Android and Apple space are games or other rudimentary (1st generation) apps oriented for consumers. But there could be tremendous demand¬†for myriad¬†vertical industry applications that can easily link to a company’s legacy systems. IBM has started down this road with Apple, but plenty of opportunity remains for enterprise software players to truly leverage the dominant platforms for their own gain. Let’s hope the tech product companies can rekindle their growth by bringing out great products again.

Best, Jim Ditmore

Posted in Looking Ahead, Mobile, Platforms | Tagged , , | 6 Comments

The Key Growth Constraints on Your Business

We have seen the accelerating impact of technology on a wide variety of industries in the past decade. We have witnessed the impact of internet shopping on retail industries, and seen¬†the impact of digital content and downloads on the full spectrum of traditional media companies across books and newspapers as well as movies and games. Traditional enterprises¬†are struggling to anticipate and stay abreast of the advances and changes. Even in those industries far away from the digital world, where they¬†seem¬†very physical, it is critical to leverage ‘smart’ technologies to improve output and products.

Let’s take logging and sawmills¬†as an example. Certainly there have been physical advances from improved hydraulic systems to better synthetic ropes, but playing an increasingly prominent role is the use of digital technology to assist operators to drive complex integrated machines to optimize the entire logging and sawing process. The latest ¬†purpose-built forestry machines operate at the roadside or nearby off-road cutting logs from whole trees combining steps and eliminating manual labor. These integrated machines are guided on-board computers and electronic controls. This enables the operator to optimize log products which are machine cut by¬†skillfully delimbing and ‚Äúbucking‚ÄĚ the whole trees into the best¬†log lengths and loading them onto trucks. Subsequently, the logs are take to¬†modern sawmills, where new scanning technologies and computers analyze each log and determine how to optimize the dimensional lumber cut from each log. Not only does this dramatically reduce manual labor and waste, but it improves safety and increases log product value by 20 or 30% from previous methods. And it is not just digital machinery leveraging computers to analyze and cut, but it is also mobile apps with mapping and image analysis so better decisions are made when and where to log in the field. When digitalization is revolutionizing even ‘physical’ industries like logging and sawmills, it is clear that the pace and potential to disrupt industries by applying information technology has increased dramatically. Below is a chart that represents the pace of disruption or ‘gain’ possible by digitalization over the mid-term horizon (7 to 10 years).


It is clear that digitalization has dramatically changed the travel and media industries already. Digital disruption has been moving down into other industries as either their components move from physical to digital (e.g., cameras, film) or industry leaders apply digital techniques to take advantage (e.g., Amazon, Ameritrade, Uber). Unfortunately, many companies do not have in place the key components necessary to apply and leverage technology to digitalize in rapid or disruptive ways. The two most important ingredients to successfully digitalize are software development capacity and business process engineering skill. Even for large companies with sizable IT budgets there are typically major constraints on both software development and business process engineering. And ample quantities of both are required for significant and rapid progress in digitalization.

Starting with software development, common constraints on this capability are:

  • a large proportion of legacy systems that consume an oversize portion of resources to maintain them
  • inadequate development toolsets and test environments
  • overworked teams with a focus on schedule delivery
  • problematic architectures that limit digital interfaces and delivery speed
  • software projects that are heavily oriented to incremental product improvement versus disruptive customer-focused efforts

And even if there are adequate resources, there must be a persistent corporate focus on the discipline, productivity and speed needed for breakout efforts.

Perhaps even more lacking are the necessary¬†business process engineering in many companies. ¬†Here the issue is often not capacity or productivity but inadequate skill and improper focus. Most corporate investment agendas are controlled by ‘product’ teams whose primary focus is on incrementally improving their product’s features and capabilities rather than end to end service or process views that truly impact the customer. Further, process engineering skills are not a hallmark of service industry product teams.¬†Most senior product leaders ‘grew up’ in a product focused environment, and unless they have a manufacturing background, usually do not have process improvement experience or knowledge. Typically,¬†product team¬†expertise lies primarily in the current product and its previous generations and not in the end-to-end process supporting the actual product.¬†Too often the focus is on a next quarter product release with incremental features as opposed to fully reworking the customer interface from the customer’s point of view and reworking end-to-end the supporting business process to take full advantage of digitalization and new customer interfaces. There is far too much product tinkering versus¬†customer experience design¬†and business process engineering. Yet, the end-to-end process is actually what drives the digital customer experience versus the product features. Service firms that excel at the customer experience utilize¬†the end-to-end process design from the customer viewpoint while taking full advantage of the digital opportunities. This yields a far better customer experience that is relatively seamless and easy. Further, the design normally incorporates a comprehensive interface approach that¬†empowers each of the customer interaction points with the required knowledge about the customer and their next step. The end result is a compelling digital platform that enables them to win in the market.

As an IT leader certainly you must identify and sponsor the key digitalization projects for your company, but you must also build and develop the two critical capabilities to sustain digitalization. It is paramount that you build a software development factory that leverages modern methods on top of discipline and maturity so you have predictable and high quality software deliverables. And ensure you are building on an architecture that is both flexible and scalable so precious effort is not wasted on arcane internal interfaces or siloed features that must be replicated across your estate.

Work with your business partners to establish continuous process improvement and process engineering as desired and highly valued skills in both IT and the business team. Establish customer experience and user experience design as important competencies for product managers. Then take the most critical processes serving customers and revisit them from an end-to-end process view and a customer view. Use the data and analysis to drive the better holistic process and customer experience decisions, and you will develop far more powerful digital products and services.

Where is your team or your company on the digital journey? Do you have an abundance of software development or business process engineering skills and resources? Please share your perspective and experience in these key areas in our digital age.

Best, Jim Ditmore


Posted in Best Practices, Looking Ahead, Vision and Leadership | Tagged , , | 2 Comments

IT Resolutions for 2015

While there is still one bowl¬†game left to be played and confetti to clean up, 2014¬†is now¬†done and leaves a mixed IT legacy.¬†¬†After 2013’s issues with the NSA leaks, the mishaps, and¬†the 40 million credit identities stolen from Target, 2014 did not turn out much better on security and availability. Home Depot, eBay, JPMC all had major incidents in the ‘year of the hacks‘. Add to that the celebrity photo leaks from the Apple hacks. Add to that of course the Sony uber-hack¬†and their playstation service failure at Christmas. All in all, 2014 was quite a dismal year for IT security. On the positive side, we saw continued advances in smart technology, from phones to cars. Robots and drones are seeing major reductions in price while leapfrogging in usability and capability. So, technology’s potential seems brighter than ever, yet we still underachieve in our ability to prevent its mis-use. Now 2015 is upon us and¬†I have compiled some IT resolutions that should contribute to greater success for IT shops in the coming year!

The first IT resolution is ‚Ķ. security, security, security. While corporate IT security has improved in the past several years, we are still well behind the hackers. The many breaches of 2014 demonstrate these shortcomings. Security is one of the fastest growing portions of IT (the¬†number 2 focus item¬†behind data analytics), but much more needs to be done though most of the crucial work is just basic, diligent execution of proper security practices.¬†Many of the breaches took advantage of¬†well-known vulnerabilities either at the company breached or one of its suppliers. For example, lack of current server patching was a frequent primary root cause on hacks in 2014. ¬†And given the major economic hits of the Sony and¬†Target breaches, these events are no longer speed bumps but instead threaten a company’s reputation and viability. Make the case now to your senior business management to double down your information security investment and not show up on the 2015 list of hacks. Not sure where to start? ¬†Here’s a good checklist on security best practices¬†that is still current and if fully applied would have prevented the majority¬†of the public breaches in 2014.

Next is to explore and begin to leverage real-time decisioning. It’s more than big data — it is where you use all the information about the customer and trends to make the best decision for them (and your company) while they are transacting. It is taking the logic for ‘recommendations of what other people bought’ and applying data analytics to many kinds of business rules and choices. For example, use all the data and hidden patterns to better and more easily qualify a customer for a home loan — rather than asking them for a surfeit of documents and proofs. And offer them optimal pricing on the loan most suited for them — again determined by the data analytics. In the end, business policies will move from being almost static where changes occurs slowly, to where business policies are determined in real-time, by the data patterns. It is critical in almost every industry to understand and begin mastery of this technology.

Be on the front edge of the flash revolution in the enterprise. 2015 will be the year of flash. Already many IT shops are using hybrid flash disk technologies. With the many offerings on the market and 2nd generation releases by mainstream storage vendors like EMC, IT shops should look to leverage flash for their most performance-bound workloads. The performance improvements with flash can be remarkable. And the 90% savings on environmentals in your data center is icing on the cake. Flash, factoring in de-duplication, is comparable in cost to disk storage today. By late 2015, it could be significantly less.

If you haven’t already, go mobile, from the ground up. Mobile is the primary way most consumers interface with companies today. And with better phones and data networks, this will only increase. But don’t rely on a ‘mobilized’ version of your internet site. Make sure you tuning your customer interface for their mode of interaction. Nothing is more cumbersome to a consumer than trying to enter data from a phone into an internet form designed for PC. Yes, its doable, but nowhere near the experience you can deliver with a native app. Go mobile, go native.

Bring new talent into the IT labor force. By 2020, the Bureau of Labor Statistics estimates there will be another 1.4 million IT jobs in the US — and not nearly enough computer science graduates to fill them. Companies big and small should be looking to hire both new graduates in the field AND encourage more to look to computers for their career. In the 1970s and 1980s, before there were formal computer science programs at universities, many outstanding computer scientists received their degrees in music, biology, languages, or teaching. We need another wave of converts for us to have the skilled teams required for the demands of the next decade. As IT leaders, let’s make sure we contribute to our field and help bring along the next generation.

What are your 2015 IT resolutions? Let us know what should be on the list!

Best, and have a great New Year!



Posted in Building High Performance Teams, Information Security, Looking Ahead, Vision and Leadership | Tagged , | 5 Comments

Improving Vendor Performance

As we discussed in our previous post on the inefficient technology marketplace, the typical IT shop spends 60% or more of its budget on external vendors – buying hardware, software, and services. Often, once the contract has been negotiated, signed, and initial deliveries commence, attentions drift elsewhere. There are, of course, plenty of other fires to put out. But maintaining an ongoing, fact-based focus on your key vendors can result in significant service improvement and corresponding value to your firm. This ongoing fact-based focus is proper vendor management.

Proper vendor management is the right complement to a robust, competitive technology acquisition process. For most IT shops, your top 20 or 30 vendors account for about 80% of your spend. And once you have achieved outstanding pricing and terms through a robust procurement process, you should ensure you have effective vendor management practices in place that result in sustained strong performance and value by your vendors.

Perhaps the best vendor management programs are those run by manufacturing firms. Firms such as GE, Ford, and Honda have large dedicated supplier teams that work closely with their suppliers on a continual basis on all aspects of service delivery. Not only do the supplier teams routinely review delivery timing,  quality, and price, but they also work closely with their suppliers to help them improve their processes and capabilities as well as identify issues within their own firm that impact supplier price, quality and delivery. The work is data-driven and leverages heavily process improvement methodologies like LEAN. For the average IT shop in services or retail, a full blown manufacturing program may be overkill, but by implementing a modest but effective vendor management program you can spur 5 to 15% improvements in performance and value which accumulate to considerable benefits over time.

The first step to implementing a vendor management program is to segment your vendor portfolio. You should focus on your most important suppliers (by spend or critical service). Focus on the top 10 to 30 suppliers and segment them into the appropriate categories. It is important to group like vendors together (e.g, telecommunications suppliers or server suppliers). Then, if not already in place, assign executive sponsors from your company’s management team to each vendor. They will be the key contact for the vendor (not the sole contact but instead the escalation and coordination point for all spend with this vendor) and will pair up with the procurement team’s category lead to ensure appropriate and optimal spend and performance for this vendor. Ensure both sides (your management and the vendor know the expectations for suppliers (and what they should expect of your firm). Now you are ready to implement a vendor management program for each of these vendors.

So what are the key elements of an effective vendor management program? First and foremost, there should be three levels of vendor management:

  • regular operational service management meetings
  • quarterly technical management sessions, and
  • executive sessions every six or twelve months.

The regular operational service management meetings Рwhich occur at the line management level Рensure that regular service or product deliveries are occurring smoothly, issues are noted, and teams conduct joint working discussions and efforts to improve performance. At the quarterly management sessions, performance against contractual SLAs is reviewed as well as progress against outstanding and jointly agreed actions. The actions should address issues that are noted at the operational level to improve performance. At the nest level, the executive sessions will include a comprehensive performance review for the past 6 or 12 months as well as a survey completed by and for each firm.  (The survey data to be collected will vary of course by the product or service being delivered.) Generally, you should measure along the following categories:

  • product or service delivery (on time, on quality)
  • service performance (on quality, identified issues)
  • support (time to resolve issues, effectiveness of support)
  • billing (accuracy, clarity of invoice, etc)
  • contractual (flexibility, rating of terms and conditions, ease of updates, extensions or modifications)
  • risk (access management, proper handling of data, etc)
  • partnership (willingness to identify and resolve issues, willingness to go above and beyond, how well the vendor understand your business and your goals)
  • innovation (track record of bringing ideas and opportunities for cost improvement or new revenues or product features )

Some of the data (e.g. service performance) will be  summarized from operational data collected weekly or monthly as part of the ongoing operational service management activities. The operational data is supplemented by additional data and assessments captured from participants and stakeholders from both firms. It is important that the data collected be as objective as possible Рso ratings that are high or low should be backed up with specific examples or issues. The data is then collated and filtered for presentation to a joint session of senior management representing their firms. The focus of the executive session is straightforward: to review how both teams are performing and to identify the actions that can enable the relationship to be more successful for both parties. The usual effect of a well-prepared assessment with data-driven findings is strong commitment and a re-doubling of effort to ensure improved performance.

Vendors rarely get clear, objective feedback from customers, and if your firm provides such valuable information, you will often be the first to reap the rewards. And by investing your time and effort into a constructive report, you will often gain an executive partner at your vendor willing to go the extra mile for your firm when needed. Lastly, the open dialogue will also identify areas and issues within your team and processes, such as poor specifications or cumbersome ordering processes that can easily be improved and yield efficiencies for both sides.

It is also worthwhile to use this supplier scorecard to rate the vendor against other similar suppliers. For example, you can show there total score in all categories against other vendors in an an anonymized fashion (e.g., Vendor A, Vendor B, etc) where they can see their score but can also see other vendors doing better and worse. Such a position often brings out the competitive nature of any group, also resulting in improved performance in the future.

Given the investment of time and energy by your team, the vendor management program should be focused on your top suppliers. Generally, this is the top 10 to 30 vendors depending on your IT spend. The next tier of vendors (31 through 50 or 75) should get an annual or biannual review and risk assessment but not the regular operational meetings or assessments and management assessment unless the performance is below par. Remediation of such a vendor’s performance can often be turned around by applying such a program.

Another valuable practice, once your program is established and is yielding constructive results, is to establish a vendor awards program. With the objective and thoughtful perspective of your vendors, you can then establish awards for your top vendors – vendor of the year, vendor partner of the year, most improved vendor, most innovative, etc. Perhaps invite the senior management of the vendor’s receiving awards to attends and awards dinner, along with your firm’s senior management to give the awards, will further spur both those who win the awards as well as those who don’t. Those who win will pay attention to your every request, those who don’t will have their senior management focused on winning the award for next year. The end result, from the weekly operational meetings, to the regular management sessions, and the annual gala, is that vendor management positively impacts your significant vendor relationships and enables you to drive greater value from your spend.

Of course, the vendor management process outlined here is a subset of the procurement lifecycle applied to technology. It complements the technology acquisition process and enables you to repairs or improve and sustain vendor performance and quality levels for a significant and valuable gain for your company.

It would be great to hear from your experience with leveraging vendor management.

Best, Jim Ditmore


Posted in Best Practices, Efficiency and Cost Reduction, Metrics and Process Improvement, Procurement | Tagged , , | 4 Comments

Expect More Casualties

Smart phones, tablets, and their digital ecosystems have had a stunning impact on a range of industries in just a few short years. Those platforms changed how we work, how we shop, and how we interact with each other. And their disruption of traditional product companies has only just begun.

The first casualties were the entrenched smart phone vendors themselves, as IOS and Android devices and their platforms rose to prominence. It is remarkable that BlackBerry, which owned half of the US smart phone industry at the start of 2009, saw its share collapse to barely 10% by the end of 2010 and to less than 1% in 2014, even as it responded with comparable devices. It’s proving nearly impossible for BlackBerry to re-establish its foothold in a market where your ‘platform’, including your OS software and its features, the number of apps in your store, the additional cloud services, and the momentum in your user or social community are as important as the device.

A bit further afield is the movie rental business. Unable to compete with electronic delivery to a range of consumer devices, Blockbuster filed for bankruptcy protection in September 2010 just 6 years after its market peak. Over in another content business, Borders, the slowest of the big bookstore chains, filed for bankruptcy shortly after, while the other big bookstore chain, Barnes & Noble, has hung on with its Nook tablet and better store execution — a “partial” platform play. But the likes of Apple, Google, and Amazon have already won this race, with their vibrant communities, rich content channels , value-added transactions (Geniuses and automated recommendations), and constantly evolving software and devices. Liberty Mutual recently voted on the likely outcome of this industry with its disinvestment from Barnes & Noble.

What’s common to these early casualties? They failed to anticipate and respond to fundamental business model shifts brought on by advances in mobile, cloud computing, application portfolios and social communities. Combined, these technologies have evolved to lethal platforms that can completely overwhelm established markets and industries. ¬†They failed to recognize that their new competitors were operating on a far more comprehensive level than their traditional product competitors. Competing on product versus platform is like a catapult going up against a precision-guided missile.

Sony provides another excellent example of a superior product company (remember the Walkman?) getting mauled by platform companies. Or consider the camera industry: IDC predicts that shipments of what it calls “interchangeable-lens cameras” or high end digital cameras peaked in 2012 and will decline 9.1% this year compared with last year ¬†as the likes of Apple, HTC, Microsoft, and Samsung build high-quality cameras into their mobile devices. By some estimates, the high-end camera market in 2017 will be half what it was in 2012 as those product companies try to compete against the platform juggernauts.

The casualties will spread throughout other industries, from environmental controls to security systems to appliances. Market leadership will go to those players using Android or iOS as the primary control platform.

Over in the gaming world, while the producers of content (Call of Duty, Assassin’s Creed, Madden NFLC, etc.) are doing well, the console makers are having a tougher time. The market has already forever split ¬†into games on mobile devices and those for specialized consoles, making the market much more turbulent for the console makers. Wii console maker Nintendo, for example, is expected to report a loss this fiscal year. If not for some dedicated content (e.g., Mario), the game might already be over for the company. In contrast, however, Sony’s PS4 and Microsoft’s Xbox One had successful launches in late 2013, with improved sales and community growth bolstering both “partial” platforms for the long term.

In fact, the retail marketplace for all manner of goods and services is changing to where almost all transactions start with the mobile device, leaving little room for traditional stores that can’t compete on price. Those stores must either add physical value (touch and feel, in-person expertise), experience (malls with ice skating rinks, climbing walls, aquariums), or exclusivity/service (Nordstrom’s) to thrive.

It is difficult for successful product companies to move in the platform direction, even as they start to see the platforms eating their lunch. Even for technology companies, this recognition is difficult. Only earlier this year did Microsoft drop the license fee for its ‚Äėsmall screen‚Äô¬† operating systems. After several years, Microsoft finally realized that it can’t win against a mobile platform behemoths that give away their OS while it charges steep licensing fees for its mobile platform.

It will be interesting to see if Microsoft’s hugely successful Office product suite can compete over the long term with a slew of competing ecosystem plays. By extending Office to the iPad, Microsoft may be able to graft onto that platform and maintain its strong performance. While it’s still early to predict who will ultimately win that battle, I can only reference the battle of consumer iPhone and Android versus corporate BlackBerry — and we all know who won that one.

Over the next few years, expect more battles and casualties in a range of industries, as players leveraging Android, iOS, and other cloud/community platforms take on entrenched companies. Even icons such as Sony and Microsoft are at risk, should they cling to traditional product strategies.

Meantime, the likes of Google, Apple, Amazon, and Facebook are investing in future¬†platforms — for homes, smart cars, robotics and drones, etc. As the ongoing impacts from the smart phone platforms continue, new platforms will add further impacts, so expect more casualties among traditional product companies, even seemingly in unrelated industries.¬†

This post first appeared in InformationWeek in February. It has been updated. Let me know your thoughts about platform futures. Best, Jim Ditmore.

Posted in Cloud, Looking Ahead, Vision and Leadership | Tagged , , | Comments Off on Expect More Casualties

Overcoming the Inefficient Technology Marketplace

The typical IT shop spends 60% or more of its budget on external vendors – buying hardware, software, and services. Globally, the $2 trillion dollar IT marketplace (2013 estimate by Forrester) is quite inefficient where prices and discounts vary widely between purchasers and often not for reasons of volume or relationship. As a result, many IT organizations fail to effectively optimize their spend, often overpaying by 10%, 20%, or even much more.

Considering that IT budgets continue to be very tight, overspending your external vendor budget by 20% (or a total budget overrun of 12%) means that you must reduce the remaining 40% budget spend (which is primarily for staff) by almost 1/3 ! What better way to get more productivity and results from your IT team than to spend only what is needed for external vendors and plow these savings back into IT staff and investments or to the corporate bottom line?

IT expenditures are easily one of the most inefficient areas of corporate spending due to opaque product prices and uneven vendor discounts. The inefficiency occurs across the entire spectrum of technology purchases Рnot just highly complex software purchases or service procurements. I learned from my experience in several large IT shops  that there is rarely a clear rationale for the pricing achieved by different firms other than they received what they competitively arranged and negotiated. To overcome this inefficient marketplace, the key prerequisite is to set up strong competitive playing fields for your purchases. With competitive tension, your negotiations will be much stronger, and your vendors will work to provide the best value. In several instances, when comparing prices and discounts between firms where I have worked that subsequently merged, it became clear that many IT vendors had no consistent pricing structures, and in too many cases, the firm that had greater volume had a worse discount rate than the smaller volume firm. The primary difference? The firm that robustly, competitively arranged and negotiated always had the better discount. The firms that based their purchases on relationships or that had embedded technologies limiting their choices typically ended up with technology pricing that was well over optimum market rates.

As an IT leader, to recapture the 6 to 12% of your total budget due to vendor overspend, you need to address inadequate technology acquisition knowledge and processes in your firm — particularly with your senior managers and engineers who are participating or making the purchase decisions. To achieve best practice in this area, the basics of a strong technology acquisition approach are covered here, and I will post on the reference pages the relevant templates that IT leaders can use to seed their own best practice acquisition processes. The acquisition processes will only work if you are committed to creating and maintaining competitive playing fields and not making decisions based on relationships. As a leader, you will need to set the tone with a value culture and focus on your company’s return on value and objectives – not the vendors’.

Of course, the technology acquisition process outlined here is a subset of the procurement lifecycle applied to technology. The technology acquisition process provides additional details on how to apply the lifecycle to technology purchases, leveraging the teams, and accommodating the complexities of the technology world. As outlined in the lifecycle, technology acquisition should then be complemented by a vendor management approach that repairs or sustains vendor performance and quality levels – this I will cover in a later post.

Before we dive into the steps of the technology acquisition process, what are the fundamentals that must be in place for it to work well? First, a robust ‘value’ culture must be in place. A ‘value’ culture is where IT management (at all levels) is committed to optimizing its company’s spending in order to make sure that the company gets the most for its money. It should be part of the core values of the group (and even better — a derivative of corporate values). The IT management and senior engineers should understand that delivering strong value requires constructing competitive playing fields for their primary areas of spending. If IT leadership instead allows relationships to drive acquisitions, then this quickly robs the organization of negotiating leverage, and cost increases will quickly seep into acquisitions. ¬†IT vendors will rapidly adapt to how the IT team select purchases — if it is relationship oriented, they will have lots of marketing events, and they will try to monopolize the decision makers’ time. If they must be competitive and deliver outstanding results, they will instead focus on getting things done, and they will try to demonstrate value. For your company, one barometer on how you are conduct your purchases is the type of treatment you receive from your vendors. Commit to break out of the mold of most IT shops by changing the cycle of relationship purchases and locked-in technologies with a ‘value’ culture and competitive playing fields.

Second, your procurement team should have thoughtful category strategies for each key area of IT spending (e.g. storage, networking equipment, telecommunications services). Generally, your best acquisition strategy for a category should be to establish 2 or 3 strong competitors in a supply sector such as storage hardware. Because you will have leveled most of the technical hurdles that prevent substitution, then your next significant acquisition could easily go to any of vendors . In such a situation, you can drive all vendors to compete strongly to lower their pricing to win. Of course, such a strong negotiating position is not always possible due to your legacy systems, new investments, or limited actual competitors. For these situations, the procurement team should seek to understand what the best pricing is on the market, what are the critical factors the vendor seeks (e.g., market share, long term commitment, marketing publicity, end of quarter revenue?) and then the team should use these to trade for more value for their company (e.g., price reductions, better service, long term lower cost, etc). This work should be done upfront and well before a transaction initiates so that the conditions favoring the customer in negotiations are in place.

Third, your technology decision makers and your procurement team should be on the same page with a technology acquisition process (TAP). Your technology leads who are making purchase decisions should be work arm in arm with the procurement team in each step of the TAP.  Below is a diagram outlining the steps of the technology acquisition process (TAP). A team can do very well simply by executing each of the steps as outlined. Even better results are achieved by understanding the nuances of negotiations, maintaining competitive tension, and driving value.


Here are further details on each TAP step:

A. Identify Need – Your source for new purchasing can come from the business or from IT. Generally, you would start at this step only if it is a new product or significant upgrade or if you are looking to introduce a new vendor (or vendors) to a demand area. The need should be well documented in business terms and you should avoid specifying the need in terms of a product — otherwise, you have just directed the purchase to a specific product and vendor and you will very likely overpay.

B. Define Requirements – Specify your needs and ensure they mesh within the overall technology roadmap that the architects have defined. Look to bundle or gather up needs so that you can attain greater volumes in one acquisition to possibly gain better better pricing. Avoid specifying requirements in terms of products to prevent ‘directing’ the purchase to a particular vendor. Try to gather requirements in a rapid process (some ideas here) and avoid stretching this task out. If necessary, subsequent steps (including an RFI) can be used to refine requirements.

C. Analyze Options – Utilize industry research and high level alternatives analysis to down-select to the appropriate vendor/product pool. Ensure you maintain a strong competitive field. At the same time, do not waste time or resources for options that are unlikely.

D, E, F, G. Execute these four steps in concurrence. First, ensure the options will all meet critical governance requirements (risk, legal, security, architectural) and then drive the procurement selection process as appropriate based on the category strategy. As you narrow or extend options, conduct appropriate financial analysis. If you do wish to leverage proofs of concept or other trials, ensure you have pricing well-established before the trial. Otherwise, you will have far less leverage in vendor negotiations after it has been successful.

H. Create the contract – Leverage robust terms and conditions via well-thought out contract templates to minimize the work and ensure higher quality contracts. At the same time, don’t forgo the business objectives of price and quality and capability and trade these away for some unlikely liability term. The contract should be robust and fair with highly competitive pricing.

I. Acquire the Product – This is the final step of the procurement transaction and it should be as accurate and automated as possible. Ensure proper receivables and sign off as well as prompt payment. Often a further 1% discount can be achieved with prompt payment.

J & K. The steps move into lifecycle work to maintain good vendor performance and manage the assets. Vendor management will be covered in a subsequent post and it is an important activity that corrects or sustains vendor performance to high levels.

By following this process and ensuring your key decision makers set a competitive landscape and hold your vendors to high standards, you should be able to achieve better quality, better services, and significant cost savings. You can then plow these savings back into either strategic investment including more staff or reduce IT cost for your company. And at these levels, that can make a big difference.

What are some of your experiences with technology acquisition and suppliers? How have you tackled or optimized the IT marketplace to get the best deals?

I look forward to hearing your views. Best, Jim Ditmore

Posted in Best Practices, Efficiency and Cost Reduction, Procurement | Tagged , , , | 4 Comments