Above the clouds: Taming multicloud chaos

To simplify multicloud management, enterprises are beginning to turn to a layer of abstraction and automation that offers a single pane of control.

The early days of cloud computing felt boundless. Freed from the limitations of on-premises servers, developers could build exciting new products and services tethered only by their imagination. Unconstrained by bureaucratic processes governing resource utilization, software engineers enjoyed an infinitely scalable palette that could seemingly manifest exactly what they needed at the push of a button. And as vendors began adding more and more capabilities providing advanced functionality like machine learning to their platforms, the cloud quickly became a one-stop-shop for all needs, everywhere.

But enterprises are awakening to the gray lining of the cloud. As developers’ enthusiasm for the latest cloud services and tools grew, so too did the number of platforms businesses were supporting. This has created a tangled web of cloud tools that are sometimes interconnected but just as often redundant.

Enterprise adoption of multicloud strategies—using a mix of cloud environments and providers—continues to grow. But while a multicloud strategy can at least in theory provide specialized capabilities and optimized pricing, applications and workloads can be challenging to design and operate due to the complexity of working with a heterogeneous mix of proprietary platforms, services, and interfaces. And that means many companies struggle to fully realize all the benefits of their cloud investments, which, when done right, can include on-demand self-servicing, broad network access, rapid elasticity, resource pooling, and measured service. To simplify this management, some enterprises are beginning to turn to a layer of abstraction and automation that sits above the burgeoning multicloud. Alternately known as metacloud, supercloud, or sky computing, the concept of putting a compatibility layer on top of multiple clouds is gaining steam, even though it still carries some important caveats that businesses should consider.

Now: Multicloud is a tangled web

Today, the vast majority of enterprises are living with multiple platform-as-a-service tools, whether they want to or not. As much as 85% of businesses are using two or more cloud platforms, and 25% are using at least five. This situation is unlikely to change anytime soon. Solution teams want to use

what they perceive to be the best tool for the job, regardless of what cloud it’s in.¹ They do not want to be subject to the availability of tools within a single vendor’s walled garden. Also, they’re using vendors’ competitors as leverage to obtain better terms for services.² Consolidating operations within a single cloud vendor is unlikely to be a tenable solution for most businesses, and multicloud will likely be a dominant thread for the foreseeable future.

However, many companies that are now in a multicloud environment find they got there inadvertently. They brought in new services ad hoc without a higher-level strategy for dealing with things like redundancy and security.³ Complexities in multicloud environments come from maintaining multiple security configurations and data repositories. Technology leaders would like to eliminate these complexities because their organizations are unlikely to realize the cost savings or operational efficiency gains that can come with cloud adoption.⁴ Their experience with multicloud complexity has led to problems such as paying for redundant services, holes in security, and difficulty finding workers to tame all the mess.

New: Simplicity as a service

Savvy business leaders aren’t simply living with the convolution created by multicloud and the growing technology footprint. Instead, they’re looking for ways to harness the operational gains that come from managing multiple cloud instances while slaying the dragon of multicloud complexity.

The approach known as metacloud involves building a compatibility layer that provides access to common services such as storage and compute, AI, data, security, operations, governance, and application development and deployment. This compatibility layer logically sits above a business’ various cloud platforms and leverages their native technical standards through APIs—with the result that applications still enjoy the strong security of the cloud provider, but in a consistent manner with centralized control. Metacloud does this through a common interface, giving administrators centralized control over their multiple cloud instances. In a paper presented at the Association of Computing Machines’ HotOS conference, computing luminaries Ion Stoica and Scott Shenker explain that the compatibility layer in the metacloud can be centered around APIs (figure 1). This is how the compatibility layer sends instructions to each of the separate cloud interfaces.

They compare the cloud compatibility layer with a computer’s operating system, managing a computer’s resources and exposing APIs to applications.

Benefits of metacloud

With an extra layer of abstraction and automation between the various cloud platforms, organizations don’t need as much specialization in their workforce. Instead of specializing in specific cloud platforms, cloud developers can build more general skills.

Metacloud may also elevate security. Each cloud platform generally has good security standards and, in a silo, performs well. Problems arise when enterprises start mixing and matching. With multiple platforms to manage, the task of configuring necessary security settings becomes more daunting. “Hackers can leverage multiple clouds against each other,” says David Linthicum, chief cloud strategy officer at Deloitte. “They’re not breaching technology; they’re breaching humans.”⁶ Metacloud can eliminate this problem by allowing developers to set one security configuration from the compatibility layer that is executed across each cloud platform through its native interface.

By eliminating unnecessary cloud services, enterprises can reduce their security exposure, enhance user privacy, lower costs, and do more with less. Teams may become less specialized in their abilities, and thus more capable of tackling whatever higher-level, generalized problem comes along.⁷

Challenges of metacloud

From a technical perspective, Stoica and Shenker believe metacloud makes perfect sense. From a business perspective, things get more complicated.

“We think achieving a widely usable compatibility layer is, on purely technical grounds, easily achievable,” they write. “The problem is whether the market will support such an effort because, while the compatibility layer has clear benefits for users, it naturally leads to the commoditization of the cloud providers, which may not be in their interests.”⁸

The other potential pitfall of this approach is that enterprises are on the hook for building it themselves. Right now, there are few vendors offering metacloud as a service. Instead, development teams will need to take the lead building each of the connections and the ultimate interface themselves. It’s a complex solution to dealing with complexity, but the ultimate outcome should be greater simplicity.⁹

Next: Consolidation and centralization

History suggests, however, that metacloud may only be an interim solution. Past efforts to reign in sprawling data centers, databases, and operating systems have ultimately resulted in consolidation, centralization, standardization, and rationalization—not via middleware or orchestration engines, but with refactoring and simplicity. Improving license utilization has always been a focus of consolidation, and most enterprises have policies in place that control utilization of cloud resources.

Further, cloud services are often sold to various teams within IT. This could mean that as soon as IT centralizes cloud services under a metacloud, it may discover additional platforms that need to be reined in. At this point it becomes a game of whack-a-mole.

What could end up taking the place of metacloud is a more tactical approach, one that borrows the centralization and control of metacloud but leaves in place the freedom developers currently have to choose the right tool for the job. This tactical metacloud could govern provisioning of cloud credentials and allocate resources only to users that have a valid business case and the technical knowhow to make use of cloud resources without creating complexities.

Self-service has been the ultimate game-changer in IT. Any approach to centralizing cloud resources will have to respect the end users’ desire for agility. In the past, centralization generally meant bureaucracy. Line-of-business users went around IT to get out from under burdensome processes that slowed their ability to solve business problems. But automation tools are making it easier for IT to deliver functionality without slowing down business initiatives and could play an important role in any effort to rein in multicloud complexity.¹¹

Multicloud may feel messy, but it’s the world we’re living in, and likely will be for the foreseeable future. Smart business and technology leaders should look for areas to reduce complexity wherever possible—potentially through approaches like metacloud—and eliminate security and redundancy problems created by maintaining multiple cloud instances.

This article is part of Deloitte's Tech Trends 2023 report. Deloitte’s 14th annual Tech Trends report explores the impact of emerging technologies on groundbreaking innovations and foundational business areas. Read the Prologue: A brief history of the future.