IDC

Region Focus: Worldwide

Network Intelligence and Insights: Driving Performance, Protection and Productivity in Observability

Christopher Kissel

Christopher Kissel

Research Vice President, Security & Trust Products

Mark Leary

Mark Leary

Research Director, Network Analytics & Automation

Product Type:
IDC: White Paper
This Excerpt Features: Gigamon
Watch Video

Executive Summary

As the digital business model accelerates across the world’s regions and industries, organizations of all sizes are working to address onrushing digital business requirements as well as external forces such as inflation, supply chain disruptions, and worker shortages. Serving as the foundation for the digital business model is a maximally resilient and responsive digital infrastructure. This infrastructure includes all of its many on-premises systems and cloud services, core and edge components, end-user and smart-endpoint clients, vital applications and data, and cybersecurity functions. For CIOs, CISOs, and their IT organizations chartered with the delivery of a high-quality, well-protected, fast-moving, and cost-effective digital infrastructure, detailed visibility into and precise control over infrastructure conditions and components are vital keys to success. This is where observability comes in.

Observability delivers comprehensive intelligence and insights focused on current and trending operating conditions across the digital infrastructure — from networking to computing and to cloud, security, applications, and end-user experience. Today, observability solutions are used rather sparingly, in only select areas of IT management (e.g., IT operations, security operations). Going forward, the greatest value from observability will be realized when management intelligence and insights are highly leveraged within and across all IT technology domains. Observability solutions must work more completely and in concert at all layers of the digital infrastructure. Here, the comprehensive visibility and control provided by detailed observations of such important items as business workloads, technology components, cloud services, secure exchanges, and system anomalies assure that the digital infrastructure and the IT organization deliver on their full promise in service to business demands and desired outcomes.

There are many significant use cases for observability; at the top of practitioners’ preferences is “Strengthen cybersecurity posture and practices.” This is not surprising. Detection and response capabilities are often highlighted, but the best cybersecurity events are the ones that do not happen. Basic cybersecurity posture assessment such as determining which assets should not be exposed to the internet, how microsegments are established, and what data access conditions are permissible within the network all work in concert to enforce safe cybersecurity before an incident occurs.

To better understand how observability solutions are being evaluated, applied, and judged now and into the future, IDC conducted a worldwide survey focused on observability as it relates to the measurement, monitoring, mitigation, and management of the digital infrastructure. Over 900 respondents qualified and contributed to IDC’s Deep Observability Survey, which covered three regions, 11 countries, and a mix of major industries (including financial, manufacturing, retail/wholesale, healthcare, transport/utilities, education, government, and professional services). All respondents represented organizations with 1,000 employees or more, with 46% representing organizations with more than 5,000 employees. All respondents held roles of manager or above, with two thirds holding director, vice president, or C-level positions within their respective IT organizations. And all respondents had managerial and decision-making responsibilities for observability functions and solutions that span across IT operational domains, including networking, security, and cloud.

While the following sections of this IDC white paper provide more detail on the survey results, select summary findings include:

  • The top benefits of observability include gains in security, staff productivity, digital/user experience, IT governance, and digital innovation.
  • Data collection and correlation within and across IT domains is a big challenge — data is locked up, limited, or left unused. Observability helps bolster data collection and processing, define parameters for access and sharing, and free up otherwise siloed monitoring and management approaches.
  • Observability enriches management via a virtuous cycle of intelligence, inspection, insights, and integration, leading to innovation.
  • With observability initiatives, several issues may arise: shortfalls in IT staff expertise and teamwork, problematic integration requirements and procedures, incomplete or overly complex cost justification, the presence of cloud or security blind spots, and solution/ vendor lock-in.
  • Observability enables a hierarchical platform-based approach in which detailed data and artificial intelligence (AI)/machine learning (ML)–driven analysis can produce a single source of truth, converge data and tools, and enable talent to deploy, operate, repair, and enhance the digital infrastructure.
  • Networking and cybersecurity are intertwined with observability. An unstable network or a network operating at capacity compromises a company’s cybersecurity posture. Faulty cybersecurity exposes the network and its associated resources and exchanges to the adversary for exploitation. Here, network-derived intelligence and insights provide an immutable source of truth that contributes strong value to security posture and practices when used in conjunction with log-based tools.

As a follow-on to IDC’s 2022WorldwideDeepObservabilitySurvey, a series of interviews were conducted with large organizations across five major industries: healthcare, financial services, technology, utilities, and ecommerce. Interview candidates were selected based on their advanced use of observability solutions and integrated approach to operations across networking, security, and cloud management. All subjects held senior management roles within

their respective IT organization (e.g., CIO, CISO) and could offer solid tactical and strategic insights into their journey to and results from their movement to a more conjoined and collaborative approach to IT observability and operations.

While each organization faced certain unique challenges within their industry and organization, and each had advanced to different stages of evolution of integrated operations, IDC was able to identify the following keys to success common across all of these studied IT organizations:

  • Prioritizing executive commitment and integration “champions” over organizational alignment
  • Driving shared evaluation, selection, integration, and use of management tools and intelligence
  • Promoting the development and advancement of staff members with cross-IT domain skills
  • Requiring multifunction teamwork for design, maintenance, troubleshooting, and change efforts
  • Moving to a unified proactive management approach: Prescribe. Predict. Prepare. Prevent.
  • Leveraging supplier solution and service capabilities fully; tying supplier success to the organization’s success

As they looked forward, certain planned focal points stood out for each organization. The following focal points all aimed at further improving management technology influence, staff and team productivity, and business outcomes and impact:

  • Consolidation: Streamline technologies, vendors, and staff requirements. Beware of going too far!
  • Resiliency: Continually build out and stress test the environment. Do not get comfortable!
  • Automation: Leverage for both repetitive tasks and dynamic responses. Validate constantly!
  • Governance: Bolster regulatory compliance, services oversight, cost containment, and IT key performance metrics (KPIs).
  • Forensics: Evaluate results and causation. Eliminate repeat failures and prolonged impact.

The above represents an overall summary of takeaways from the interviews conducted in support of this survey effort and white paper. Select and telling quotes from these interviews appear across this white paper.

Digital Acceleration and Observability

When evaluating top business priorities, it is easy to draw the line between a resilient and responsive digital infrastructure and the delivery of high-quality exchanges, rich services, strong security, and digital innovation. Whether the focus is on worker productivity, operational efficiency, customer satisfaction, or digital innovation, observability solutions deliver the visibility and control necessary to assure that the infrastructure provides the best possible support to the business and all critical resources and workflows (see Figure 1).

FIGURE 1: Observability and Automation Directly Support Top Business Priorities

Q: What are your organization’s top 3 business priorities?


n = 796, Source: IDC’s Future Enterprise Resiliency & Spending Survey — Wave 2, March 2022

The value presented by the detailed intelligence and in-depth analysis offered by observability solutions is also highlighted when examining the barriers to success when building out a digital infrastructure. Insufficient analytics and automation is a top 3 barrier. In addition, examining other top barriers reveals close ties to observability capabilities. Whether addressing digital priorities, workload performance and security concerns, staff challenges, utilization of cloud services, or edge and data management, more consistent, detailed, and intelligent analytics and automation help break down the top barriers to digital success (see Figure 2).

“The network architecture is something that is, for me, absolutely fundamental for protecting the core part of my business.”

Manager, cybersecurity and networks, national utility company

FIGURE 2: Intelligence and Insights Key to Digital Success Across Multiple Fronts

Q: What are the three greatest barriers to achieving your organization’s digital infrastructure resiliency goals over the next two years?

n = 796, Source: IDC’s Future Enterprise Resiliency & Spending Survey — Wave 2, March 2022

CURRENT STATE OF OBSERVABILITY:

The Reality of Basic Monitoring and Management

There is widespread agreement that observability is a vital strategic management capability that serves both the overarching needs of the end-to-end digital infrastructure (e.g., digital experience management) and the specific needs of IT domains (e.g., CloudOps, NetOps, SecOps, and DevOps). There is also strong agreement that sharing intelligence and insights and combining staff efforts across IT domains is critical to infrastructure and IT organization success. When reporting on their level of sharing across IT domains, most respondents stated that they are either actively sharing (55%) or fully synchronized (22%).

Of note is the increasing alignment of efforts, toolsets, and practices between NetOps and SecOps. Over 60% of organizations reported that they are making good or strong progress in leveraging network-derived intelligence and insights in their security management efforts, while another 18% rated themselves as fully mature in this practice. In evaluating their ability to combine NetOps and SecOps to perform such actions as gathering telemetry, performing triage, and initiating remediation, on average, 60% rated themselves “good” or “excellent.”

Looking into the critical area of cloud service management, 73% of organizations reported having real-time telemetry for both their cloud computing and networking environments. And most organizations agreed or strongly agreed that they leverage cloud service intelligence and insights to optimize costs (72%), secure information (72%), resolve service problems (69%), and track client activity across multicloud environments (68%).

Are these positive reports of extensive sharing, complete visibility, and effective teamwork to be believed? More detailed probing into observability use, challenges, and expectations reveals that high-level perceptions may not reflect the detailed reality of complex and critical digital infrastructure management. Most IT organizations need to raise the bar in applying observability intelligence and insights across their infrastructure.

For example, running counter to respondents’ indicating the ease of sharing data between NetOps and SecOps and gathering telemetry from cloud services, only 14.6% of respondents answered that collecting data from observability tools was “not difficult.” While observability intelligence and insights offer much promise across the digital infrastructure, organizations are challenged to break siloed approaches, find a common workflow with other tools through integration, and train their teams to use the enriched data observability solutions present.

“I would say that having the NOC [network operations center] and SOC [security operations center] separately provides challenges. More so on the SOC side because they are reacting to an alert that says there is a problem. Because they don’t manage the network and they don’t sit with those that do, SOC staff don’t necessarily understand the environment to the level that is useful.”

Manager, cybersecurity and networks, national utility company

Many Tools for Multiple Groups

While respondents indicated that sharing data and tools is widespread and that further progress is being made, indications are that observability tools and practices remain disjointed and dispersed. Just as the digital infrastructure itself must work in concert to serve the business, observability tools must work in concert to serve IT. For too many organizations, their observability toolsets, practices, and spending reflect more collection than concerted effort, as they are focused on specialized and standalone management functions. The sheer number of observability solutions in use reflects this complex collection approach. Almost 70% of organizations use six or more tools, and almost half of this group uses 11 or more tools. Amazingly, 7% use more than 20 observability tools to operate and optimize their digital infrastructure.

And where are all of these observability tools being used? Organizations report the use of observability solutions across every major IT management domain (see Figure 3). However, use is limited across all organizations. Just over half of organizations said that observability solutions are used only for IT operations; that means nearly half are not applying observability to IT operations. And it gets worse from there when examining other IT management domains (e.g., CloudOps, SecOps, NetOps). All fall below — and some well below — 50% utilization within responding organizations.

“I think the tools and the teams in the IT space have to come together.”

CIO, national healthcare provider

FIGURE 3: Observability Capabilities Vary Widely Across IT Domains

n = 912, Source: IDC’s Gigamon Deep Observability Study, June 2022

The Challenges with Existing Observability Solutions

As pointed out, there are too many observability solutions being used by too few IT management domains. This creates multiple challenges for organizations looking to advance digital infrastructure visibility and control (see Figure 4). And these challenges exacerbate other pressing and more strategic IT challenges. For example, staff requirements multiply as the observability toolset grows, adding to the problem of staff shortages. Total cost of ownership increases, further pressuring IT budgets. Information overload, siloed intelligence, and integration burden delay management actions, undercut service quality, and hinder innovation, slowing or restricting digital business efforts and success.

“It is very difficult to say there is any one tool anymore that has everything because it is too hard to get everything in one place and there is too much data.”

CISO, global financial firm

Figure 4: Complex Array of Observability Tools Presents Many Serious Challenges

n = 912, Source: IDC’s Gigamon Deep Observability Study, June 2022

There are numerous other indications from the survey that highlight concerning issues with current observability solutions. Many revolve around restrictions relating to observability data and the detailed visibility it provides. Over 40% of respondents indicated that it is difficult or even extremely difficult to derive actionable insights from the data collected. Over 60% agreed or strongly agreed with the statement that observability solutions serve narrow requirements and fail to offer a complete view into current operating conditions. Almost 70% advocated for a single source of truth to be established for use by all mission-critical IT management tools. And zooming in on security specifically, nearly 75% cited observability as being critical to forming a strong security posture and mitigating threats quickly and effectively.

ADVANCEMENTS IN OBSERVABILITY:

Heightened Intelligence, Insights, and Impact

Knowing the detailed status of digital infrastructure conditions and components at any moment in time has become a vital requirement for digital success. The movement of traffic, operational status of infrastructure services and components, contribution of cloud services, oncoming threats, and the end-user experience are all priorities as organizations accelerate and heighten their digital business initiatives. Beyond enabling complete visibility into the current state of the infrastructure, the detailed intelligence and insights provided by observability solutions also serve to direct precise actions, protect against threats, predict future outcomes, and improve IT staff productivity.

“I think eventually, you are going to be in a position where machine learning will be able to dictate and drive the scaling of your network and/ or your Kubernetes clusters based on behavioral patterns seen in your traffic.”

CTO, global ecommerce firm

Expected Areas of Improvement with Advanced Observability

To fulfill all of this promise, observability solutions must continue to advance across many fronts. Ties to IT automation boost the accuracy and timeliness of automated actions. Ease of usability and integration promotes tool usage and teamwork across
IT domains. End-to-end digital experience management assures consistent service quality to end users (e.g., workers, partners, and customers) and smart endpoints
(e.g., sensors, robots). Anomaly detection bolsters threat detection and mitigation efforts. Cloud observability brings the cloud infrastructure into full view, a critical requirement given the core role cloud services (and multicloud environments) play within the digital infrastructure of almost every organization. As evidenced by the survey results, the list of expected advancements — and the areas of impact from these advancements — is long and, even more importantly, reflects a rather equally strong importance for all (see Figure 5, next page). The future success of observability is based upon advancements along many fronts, not just one or even a few.

FIGURE 5: Expected Impact of Observability Investments\

n = 912 (Top 2 Box Summary), Source: IDC’s Gigamon Deep Observability Study, June 2022

Ranking the Benefits of Advanced Observability

Many respondents indicated that their use of network-derived intelligence and insights in support of their security management efforts is fully effective and mature, yet strengthening security postures and practices is viewed as the number 1 benefit of applying observability across the digital infrastructure (see Figure 6, next page). Obviously, there remains much concern for securing the digital infrastructure — and much hope for heightened security-driving observability intelligence and insights.

On the issue of IT staffing, everyone agrees that the shortage is real, and staff development, satisfaction, retention, and teamwork are all troubling challenges. Again, we see much hope for observability tools and data to bolster staff efforts and expertise, focused not only on their primary area of responsibility (e.g., network operator, security analyst) but also on their contribution to adjacent domains and overarching IT efforts (e.g., automation, digital experience management).

Beyond security and staffing improvements, observability is seen to deliver a mix of both tactical (e.g., resolution, continuity, tracking) and strategic (e.g., experience, governance, innovation) benefits. Observability offers potential impact along many fronts. When identifying requirements, evaluating solutions, and justifying purchases, it is key to understand all of the possible areas of impact.

“There is a lot of efficiency in automation. We automate about 78% of our alerts right now. And we are working to automate more. Our goal is to automate as much as possible, and then the team is really adding value as opposed to looking at alerts. It is not fun work. No one likes doing it. The more we can get into an automated playbook the better.

We don’t want our team to burn out and we want them working on higher-value things.”

CISO, global financial firm

Figure 6: The Benefits of Observability: From Security to Staff to Service to Savings

n = 912, Source: IDC’s Gigamon Deep Observability Study, June 2022

ADVANCED OBSERVABILITY:

Strengthening Security Postures and Practices

Bolstering Protection Along Critical Fronts

Cybersecurity can be thought of in three movements. The first and last movements
are easily understood. A “shift-left” approach starts with the DevOps environment; we can call this, largely, prevention. Increasingly, the success of businesses is that they are becoming developers; security starts with secure containers and the most updated and vulnerability free code (outfitting old Java scripts is not cutting it). Identity access management, audio/visual (AV) and firewall policies, micro-segmentation, and zero trust are part of a shift-left approach.

A “shift-right” strategy is also advisable. Businesses must react to a high-security alert with the installation of the proper playbook, initiate ephemeral response such as having end users access the network through multifactor authentication (MFA), and then start a workflow. The next set of circumstances that a business should be prepared for is data backup and disaster-recovery procedures.

The middle is what we call “shift-through,” and it is probabilistic; the idea is to consider not just vulnerabilities but also the higher concept of risk. If expressed as a formula, risk = probable outcomes multiplied by potential damages multiplied by indemnities (compliance, loss of reputation, etc.) divided by (prevention + security controls + mitigation + recovery).

This is all germane to how a network is monitored. If a NOC/SOC goes into alerts based solely upon vulnerabilities, an organization will forever be in whack-a-mole mode. Differentiating asset criticality matters, too — any threat to a web/email server is a big problem. Shift-through is a risk-based approach that allows companies to investigate incidents in such a way that a problem is properly remediated and a company’s security posture improves over time as a native part of the investigative process (i.e., if a team finds an internet-facing exposed S3 bucket, it will find others in the same fell swoop).

With advanced observability, NetOps and SecOps teams can gain predictive
insights and proper remediation strategies for shift-left, shift-right, and shift-through (see Figure 7, next page). Deriving the best insights from packet data does take some expertise, but the path toward fortifying cybersecurity resides in the integration of log and network metadata. Only when these are used in concert can NetOps and SecOps teams gain the ability to proactively detect and remediate threats at all layers of the digital infrastructure.

“Having network engineers and security engineers involved in vendor and product evaluations brings a holistic approach to how we pick tools as well as how we select vendors.”

Manager, IT infrastructure, global technology manufacturer

Figure 7: Strengthening Security with Observability: Top Expectations

n = 912, Source: IDC’s Gigamon Deep Observability Study, June 2022

Among the survey respondents, 78% indicated that observability helps with threat detection and mitigation efforts. However, the very first move after triage is the most important: The security team must determine if an incident is indeed a security event, and then if the attack is against a type of persona (C-level executives), an application type, an exploit of opportunity, or a pawning of an identity to set up a command-and-control (C2C) exfiltration beachhead.

Logs and packets together provide immutable truths about what is happening within the application and network layers of the digital infrastructure. Metadata collection is helpful for forensic investigations and event correlation, but remember that the adversary understands this as well. The adversary can disguise IP addresses and sometimes manipulate logs. However, observability with deep packet inspection creates an unalterable insight.

The following are use cases that highlight the role of an advanced observability solution, consisting of a deep observability pipeline and at least one observability tool, in identifying security issues that may not be detected by endpoint detection and response (EDR) or security information and event management (SIEM) tools:

  • Visibility in an encrypted packet session: Most perimeter security tools will look to see if there is an Secure Sockets Layer/Transport Layer Security (SSL/TLS) session providing egress from the network to the internet. Assuming that the adversary is in the network and can access the NetStat/S registry, they can initiate a “port spoofing” session that, to many security tools, appears to be encrypted but is in fact an obfuscated secure shell (SSH) session. Firewall, network monitoring, and intrusion detection systems/intrusion prevention systems (IDS/IPS) tools bypass inspection of the session, as it looks legitimate. With the additional detail provided by an advanced (deep) observability solution, these spoofing sessions can be correctly identified. A deep observability solution can also provide a decrypted form of the SSL/TLS sessions that can then be analyzed more thoroughly for behavioral and content aspects.
  • Unmanaged devices: Network administration and cybersecurity teams rightly worry about unmanaged devices, especially given the ease with which these devices can lose connection with the network due to everyday events such as reconfigurations, software updates, and even power surges. An advanced or deep observability solution can help rediscover these devices by identifying communications and network behavior associated with the profile of these devices. Rogue or undesirable devices can also appear in the network, which can be discovered and assessed for their behavior and threat level.
  • Kubernetes: As Kubernetes becomes an increasingly common way to deploy applications in cloud environments, CloudOps teams are becoming increasingly aware of some of the security challenges it presents, especially as it scales
    out. As with every other network environment, it is important to have visibility into Kubernetes containers to be able to track sessions and detect when a vulnerability is being exploited by an adversary. An advanced or deep observability solution provides this level of visibility.

When survey participants were asked what their top expectations are for the vendor/provider of an observability platform designed for cybersecurity, the top 2 replies were “Able to express indicators of compromise within the MITRE ATT&CK framework” and “Eliminate dwell time that an adversary can live off the land in the network.” The MITRE ATT&CK for Enterprise is a global framework deployed within security tools for monitoring, predicting, and remediating adversarial behavior on a network. MITRE ATT&CK recognizes two collection techniques and 12 specific techniques that can be used by the adversary in an attempted breach. There are 218 sub-techniques cataloged in the framework. If deployed properly, observability can find 90% of these techniques. The old colloquialism “Sunlight is the best antiseptic” definitely applies to cybersecurity tools.

SecOps and NetOps Convergence

In networks and in cybersecurity, the concept of observability is straightforward.
The network is the host where data, applications, and properly credentialled users meet. Vigilance of the network requires that the network be stable to enable access, and observability extends to preemptive observation of indicators of compromise (IOC) in hopes of preempting an attack, as well as the tools needed to investigate an incident if there is something more than an IOC.

Toward weaving network-derived intelligence and insights into security management, 32% of respondents said they are making fairly strong progress, and another 22% believed they had made excellent progress and are fully mature (see Figure 8, next page). When rating their ability to combine efforts, toolsets, and practices of NetOps and SecOps teams to identify compromised machines and personas, almost two thirds of the respondents rated themselves “good” or “excellent.” Maybe.

Why maybe? It is estimated that if organizations adhered to the top 5 of the SANS 20 Critical Security Controls, they would eliminate 98% of possible attack vectors. As easy as that sounds, organizations need to create inventories of authorized and unauthorized devices and software, secure configurations, continuous vulnerability assessment and remediation, and controlled use of administrative privileges. Observability solutions can absolutely facilitate these initiatives, but this requires both the proper tooling and telemetry and the organizational discipline to use them.

“Is our security posture improving? I think our posture is improving in that we now have a broader team who understands the tools we use to manage our security.”

CIO, national healthcare provider

Figure 8: Blending Network Intelligence and Security Insights: Progress or Perception?

n = 912, Source: IDC’s Gigamon Deep Observability Study, June 2022

Evaluating Observability Solutions

Observability solutions have much to contribute to the management, protection, and evolution of digital infrastructure, and this contribution occurs in two ways. First, as a primary observability solution focused on a specific IT management domain or even select functions within a domain, a solution should provide for leading-edge capabilities (e.g., data management, actionable insights, ease of use, trend analysis). Second, no matter how specialized its primary focus, an observability solution must contribute to adjacent and overarching observability efforts and tools. The survey results certainly highlight the effective use of network-derived intelligence and insights in security management. Sharing network data and analysis and promoting the shared use of network observability tools with SecOps, DevOps, and AIOps presents many tactical and strategic benefits to the IT organization and the business.

“The problem with machine learning, you need a massive amount of data to train the model — and that massive amount of data is network traffic.”

CTO, global ecommerce firm

Passing Judgment: Ranking Solution Attributes

Survey respondents indicated the importance for observability solutions to contribute to both domain-specific management and broader integration efforts. In support of specific domains (e.g., networking, cloud, security, etc.), capabilities such as cloud visibility, deep packet inspection of encrypted traffic, AI/ML-driven analysis, predictive analytics, complex correlations, anomaly detection, root cause analysis, and forensic histories are prioritized. In support of integration efforts, enabling systemwide observability, leveraging industry standards, linking to automation efforts, and unifying data collection gain strong consideration (see Figure 9, next page).

Figure 9: Key Capabilities of an Observability Solution

n = 912, Source: IDC’s Gigamon Deep Observability Study, June 2022

Rising Concerns with Observability

In this always-on, hyper-connected, and ever-threatened digital business environment, the heightened visibility and control offered by observability solutions certainly raise concerns for organizations, most prominently security vulnerability (blind spots). The staff, operational, and budget demands of observability solutions add further concerns as organizations build out the data sets, tools, talent, and practices designed to take full advantage of a concerted observability effort (see Figure 10, next page). Observability is intended to simplify and solidify digital infrastructure management. However, it does this over time, and it requires organizations to take the right steps along the path to completion.

“A lot of our security threats ride in through the network. So how our threats have happened, it starts with the network. We needed a holistic approach to security and network management.”

Manager, IT infrastructure, global technology manufacturer

Figure 10: As Observability Efforts Extend Across the Infrastructure, Concerns Are Rising

n = 912, Source: IDC’s Gigamon Deep Observability Study, June 2022

The Outlook For Observability

Owing to the many potential benefits associated with specialized and systemwide observability efforts, survey respondents indicate increased investment, particularly across key focal points for digital infrastructure buildout.

We did not do a reorganization around cloud, but we did a rewiring of people’s brains. You have to retool and retrain people for a cloud world.”

CTO, global ecommerce firm

Over the next two years, spending on observability will increase for 68% of the organizations that responded to IDC’s Deep Observability Survey. And close to half of these organizations indicated they will be increasing their observability budget by more than 10%.

Where is this increasing investment in observability aimed? The top 3 areas of investment match the top responses relating to observability solution attributes (cloud/multicloud), current use (IT operations and service management), and expected benefits (security), respectively. Reflecting the changing nature of network infrastructure and connected users and devices in this post-pandemic and accelerating digital business environment, network observability investment is distributed across major network subdomains, with wireless ranked highest, at number 4. To show how far and how fast the observability movement has progressed over the last two years, DevOps and application performance management — the longtime singular focal point for observability solutions and efforts — are ranked a distant number 5 in the top investment areas for observability (see Figure 11, next page).

Figure 11: Priority Investment Areas for Observability in 2023–2024

n = 912, Source: IDC’s Gigamon Deep Observability Study, June 2022

Keys to Success in Observability

As we see from the survey results, organizations cite many targets, capabilities, benefits, and concerns with their observability efforts.

The following are key steps to success in observability:

  • Organize for success: Develop an IT organizational structure, culture, practice, and skill
    set that is fast-acting, forward-looking, and team-oriented. Multi-skilled staff, shared data, common tools, and cross-IT collaboration are hallmarks of an IT organization well positioned to take full advantage of observability within specific management domains and across overarching IT management efforts.
  • Streamline the toolset: With observability, less is more: Less to learn. Less to operate. Less to integrate. Less to afford. Yet fewer tools should not translate to more management gaps or blind spots. Solutions that support multiple IT domains (e.g., cloud, security, networking) and readily complement other solutions (e.g., interoperate, integrate) provide more value and less complexity.
  • Promote expansive use of observability solutions: Ease of use for the individual
    staffer promotes fast analysis and action. It also allows all staffers, from junior operator
    to senior engineer, to be more efficient and effective. Role-based dashboards promote use of intelligence and insights across IT (e.g., SecOps using network-derived intelligence and insights to detect and remediate a threat). Standardized data formats, open source software, and application programming interfaces (APIs) heighten sharing between tools and teams.
  • Assure data integrity and integration: Use multiple sources and methods to ensure complete and timely collection, distribution, and processing of observability intelligence. For example, network telemetry data provides detailed and real-time views into traffic movement across the digital infrastructure. More and better data translates directly to more in-depth and accurate analysis. And remember, data shared across tools and teams multiplies its value by bolstering adjacent management efforts and filling visibility gaps within and between management domains.
  • Drive automation with analytics: The digital business model requires a resilient and dynamic infrastructure. IDC research indicates that using observability intelligence and insights to direct automated actions enhances the resiliency and dynamics of the digital infrastructure. Emerging problems can be avoided, changing workloads can be readily accommodated, and threats can be quickly detected and mitigated.

Download

Network Intelligence and Insights: Driving Performance, Protection, and Productivity in Observability

Appendix

The appendix includes select survey data for the following countries: Australia, France, Korea, Malaysia, Philippines, Singapore, and United Kingdom, as well as the financial services and manufacturing sectors.

EMEA (France, United Kingdom)

  • The survey suggests that in the United Kingdom there is a desire to unify IT, security, and networking. This seems to be in flight, as they are making good progress in marking east-west telemetry (requiring multiple departments). There are some concerns about vulnerabilities and workflow.
  • The French IT market seems to work from a network performance paradigm: good alerts, but problematic adaptability to multicloud environments. They believe in observability but either they lack training or the lack of automation in tools limits its effectiveness. There is some slight cynicism regarding the ability of observability to stitch various IT functions.

APAC (Australia, Korea, Malaysia, Philippines, Singapore)

  • The survey consistently shows that Australia has a more mature SOC and observability process than the global mean. Respondents agreed that observability is important in IT initiatives. Proportionally, Australian SOCs seem to be more confident about things they can control (such as configurations, data collection, and east-west telemetry) than about observability pertaining to the perimeter.
  • Korea has led the way in digital innovation and a sustained digital culture. Koreans have
    an important identity expressed in broadband media such as social media, gaming, and personas built into digital identities. Roughly 84% of Korean citizens and businesses have a fiber connection to the internet, and this per-capita participation leads the world.
  • In Malaysia, 48% of businesses plan to spend 11–25% more on observability platforms in the next two years. While there seems to be a willingness to spend on observability, there is some trepidation in the confidence of how to use the tools.
  • Organizations in the Philippines are planning to invest in observability for wireless LAN and WAN (e.g., Wi-Fi 6, private 5G, LTE, public 5G), security systems and services, and DevOps and application performance management/testing. There is a prevailing belief about the excellence of its SOCs and how processes are coming along.
  • Singapore is very affluent, and it also has very stringent breach disclosure laws. That said, its observability platform spending was only in line with global contemporaries.

Select Verticals: Financial Services and Manufacturing

  • In terms of spending, the financial services industry is a true believer in observability. It also seems that financial services is implementing observability across all IT platforms (including networking). That said, financial services does not seem wholly satisfied with the telemetry and configuration information coming from cloud environments.
  • The conventional wisdom has been that manufacturers could live with some security risk versus shutting down production lines. However, IDC sees new spending planned for security. Manufacturers are not ahead of nor behind other industries in the implementation of observability platforms.

Table1: What are your biggest concerns with observability solutions?

Australia Korea Malaysia Philippines Singapore United Kingdom France Financial Services Manufacturing
Security vulnerabilities (blind spots) 26.3 32.5 28.0 28.0 30.0 32.5 19.5 23.5 24.2
Limited visibility into cloud services use, performance, and costs 26.3 29.9 20.0 34.0 20.0 10.4 29.5 24.1 22.5
Integration burden limits solution access, exchanges, interactions, and impact 21.1 29.9 28.0 30.0 32.0 26.0 14.3 19.9 31.7
Lack of in-house expertise constrains solution utilization and benefit realization 28.9 14.3 40.0 22.0 24.0 24.7 32.5 25.3 25.8
Vendor lock-in, limiting use and impact of observability functions across the digital infrastructure 18.4 22.1 28.0 24.0 20.0 24.7 19.5 24.7 20.8
High costs and difficult to calculate and prove ROI 23.7 31.2 28.0 42.0 28.0 20.8 16.9 21.1 28.3
IT culture refuses to accept a shared single source of data and insights 25.0 31.2 18.0 26.0 28.0 27.3 19.5 24.1 15.8
Lack of focus on customer success and life-cycle management by the solution vendor 32.9 22.1 26.0 14.0 16.0 18.2 23.4 19.9 23.3
Slow and, possibly, mismatched technology road map development 27.6 16.9 12.0 18.0 32.0 22.1 23.4 19.3 20.8
Limited support for open source and industry standards relating to observability 22.4 18.2 24.0 20.0 14.0 20.8 19.5 20.5 19.2
Difficulty in tying observability intelligence and insights to automation efforts 18.4 19.5 30.0 28.0 30.0 19.5 23.4 21.7 24.2
Other 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
Unweighted Valid n 76 77 50 50 50 77 77 166 120
Australia (n = 76), Korea (n = 77), Malaysia (n = 50), Philippines (n = 50), Singapore (n = 50), United Kingdom (n = 77), France (n = 77), Financial Services (n = 166), Manufacturing (n = 120)
Source: IDC’s Gigamon Deep Observability Study, June 2022 
Note: Use caution when interpreting small sample sizes.

Table 2: Top 2 Box Summary Table: What are your top expectations for the vendor/provider of an observability platform designed for cybersecurity?

Australia Korea Malaysia Philippines Singapore United Kingdom France Financial Services Manufacturing
Eliminate dwell time that an adversary can live off the land in the network 35.5 27.3 34.0 26.0 36.0 33.8 40.3 33.1 36.7
Can quickly tell if an alert is a network, security, or application problem 52.6 51.9 48.0 46.0 42.0 51.9 48.1 42.2 52.5
Able to express indicators of compromise within the MITRE ATT&CK framework 32.9 28.6 34.0 30.0 30.0 35.1 29.9 39.8 31.7
Have data enriched in such a way as to correlate with current threats in the wild 31.6 49.4 26.0 42.0 42.0 29.9 44.2 38.0 32.5
Our observability platforms must be deeply integrated with our other tools such as firewall, SIEM, EPP/EDR, etc. 47.4 42.9 58.0 56.0 50.0 49.4 37.7 47.0 46.7
Unweighted Valid n 76 77 50 50 50 77 77 166 120
Australia (n = 76), Korea (n = 77), Malaysia (n = 50), Philippines (n = 50), Singapore (n = 50), United Kingdom (n = 77), France (n = 77), Financial Services (n = 166), Manufacturing (n = 120) Source: IDC’s Gigamon Deep Observability Study, June 2022.
Scale: 1 = most important, 5 = least important
Note: Use caution when interpreting small sample sizes.

Table 3: Do you believe observability tools add or will add in-depth intelligence and insights to your threat detection and mitigation efforts?

Australia Korea Malaysia Philippines Singapore United Kingdom France Financial Services Manufacturing
Yes 77.6 90.9 92.0 86.0 86.0 84.4 87.0 73.5 79.2
No 22.4 9.1 8.0 14.0 14.0 15.6 13.0 26.5 20.8
Unweighted Valid n 76 77 50 50 50 77 77 166 120
Australia (n = 76), Korea (n = 77), Malaysia (n = 50), Philippines (n = 50), Singapore (n = 50), United Kingdom (n = 77), France (n = 77), Financial Services (n = 166), Manufacturing (n = 120) Source: IDC’s Gigamon Deep Observability Study, June 2022.

Table 4: Top 2 Box Summary Table: In examining IT management tools in use within your organization, to what extent do you agree with each of the following statements?

Australia Korea Malaysia Philippines Singapore United Kingdom France Financial Services Manufacturing
Most observability tools serve narrow requirements and fail to enable a complete view into current operating conditions 65.8 49.4 62.0 50.0 64.0 63.6 59.7 60.2 65.0
My organization plans to converge previously siloed IT teams (e.g., NetOps and SecOps) and tools (e.g., NPM, APM, SIEM) to improve our overall approach to observability 73.7 67.5 72.0 68.0 66.0 71.4 57.1 61.4 72.5
Observability is critical to delivering the best possible digital experiences for customers and employees 85.5 74.0 88.0 76.0 78.0 80.5 61.0 68.1 80.0
Observability is critical to forming a strong security posture and mitigating threats fast and effectively 69.7 64.9 84.0 74.0 86.0 74.0 72.7 67.5 78.3
Detailed cloud service measurements are critical to the success of observability tools and efforts 81.6 55.8 80.0 68.0 82.0 80.5 71.4 72.3 77.5
My organization uses or plans to use observability solutions to support our IT automation efforts 73.7 76.6 82.0 72.0 82.0 76.6 61.0 75.9 80.0
AI/ML technologies must be incorporated within observability solutions for them to fully deliver on their promise 67.1 72.7 82.0 82.0 78.0 79.2 63.6 69.3 70.8
A single source of truth must be established for use by all mission-critical IT management tools 73.7 61.0 80.0 72.0 70.0 64.9 64.9 66.3 70.8
When IT teams use the same observability tools across technology domains, it fosters teamwork and operational success 78.9 74.0 80.0 70.0 84.0 71.4 70.1 69.9 82.5
No rated in top 2 box 1.3 1.3 0.0 0.0 0.0 0.0 3.9 2.4 0.8
Unweighted Valid n 76 77 50 50 50 77 77 166 120
Australia (n = 76), Korea (n = 77), Malaysia (n = 50), Philippines (n = 50), Singapore (n = 50), United Kingdom (n = 77), France (n = 77), Financial Services (n = 166), Manufacturing (n = 120) Source: IDC’s Gigamon Deep Observability Study, June 2022.
Scale: 1 = strongly disagree, 5 = strongly agree
Note: Use caution when interpreting small sample sizes.

Table 5: What are the biggest benefits from applying observability across the entire digital infrastructure?

Australia Korea Malaysia Philippines Singapore United Kingdom France Financial Services Manufacturing
Shift from reactive to proactive operations 19.7 16.9 16.0 18.0 18.0 23.4 18.2 16.9 15.0
Drive faster mean time to resolution (MTTR) 14.5 22.1 30.0 18.0 24.0 20.8 23.4 24.7 24.2
Improve digital experience for customers and employees 25.0 19.5 32.0 36.0 28.0 31.2 26.0 21.7 24.2
Accelerate digital innovation 22.4 22.1 30.0 28.0 14.0 22.1 19.5 21.7 25.0
Ability to track individual users and machines across multiple environments 25.0 29.9 28.0 34.0 34.0 18.2 16.9 22.9 25.8
Improve IT staff productivity and collaboration 25.0 35.1 40.0 38.0 22.0 31.2 24.7 30.7 32.5
Helps provide governance over multicloud/heterogeneous environments 27.6 27.3 28.0 34.0 34.0 24.7 24.7 22.9 23.3
Assure business resiliency and continuity 26.3 28.6 22.0 18.0 26.0 24.7 16.9 16.3 25.8
Strengthen cybersecurity posture and practices 44.7 32.5 34.0 38.0 40.0 35.1 33.8 31.3 31.7
Reduce/contain operational costs 21.1 28.6 22.0 12.0 20.0 13.0 22.1 23.5 21.7
Proves complaint practices to regulators 25.0 11.7 8.0 20.0 20.0 18.2 16.9 20.5 20.0
Unweighted Valid n 76 77 50 50 50 77 77 166 120
Australia (n = 76), Korea (n = 77), Malaysia (n = 50), Philippines (n = 50), Singapore (n = 50), United Kingdom (n = 77), France (n = 77), Financial Services (n = 166), Manufacturing (n = 120) Source: IDC’s Gigamon Deep Observability Study, June 2022.
Notes: Multiple dichotomous table — total will not sum to 100%. Use caution when interpreting small sample sizes.

Table 6: What are the most important attributes of an observability solution?

Australia Korea Malaysia Philippines Singapore United Kingdom France Financial Services Manufacturing
Unifies data collection and correlation 26.3 28.6 22.0 32.0 26.0 24.7 26.0 18.7 29.2
Captures actionable intelligence from a full spectrum of sources 23.7 23.4 20.0 26.0 26.0 23.4 29.9 31.9 21.7
Uses synthetic transactions to measure the end-user experience 19.7 16.9 18.0 12.0 14.0 10.4 20.8 16.3 16.7
Leverages AI/ML to identify root causes, anomalies, and threats 22.4 18.2 18.0 20.0 24.0 26.0 16.9 19.3 22.5
Couples analytics and automation to optimize deployments and operations 25.0 39.0 28.0 14.0 26.0 23.4 14.3 22.9 26.7
Ability to provide deep inspection and insights into encrypted traffic 23.7 23.4 28.0 32.0 26.0 29.9 29.9 24.1 23.3
Supports data and insights for cloud services and multicloud environments 23.7 32.5 28.0 38.0 38.0 35.1 22.1 25.9 30.0
Supports systemwide observability across the entire digital infrastructure 35.5 22.1 40.0 32.0 26.0 20.8 23.4 28.9 27.5
Support for industry standards in data formats, exchange mechanisms, and API-enabled access 23.7 20.8 28.0 34.0 28.0 27.3 23.4 24.7 23.3
Maintains historical records for later forensic analysis 22.4 22.1 20.0 30.0 16.0 16.9 22.1 18.1 19.2
Supports predictive analysis to aid forecasting and problem avoidance 27.6 24.7 36.0 28.0 30.0 27.3 15.6 22.9 27.5
Unweighted Valid n 76 77 50 50 50 77 77 166 120
Australia (n = 76), Korea (n = 77), Malaysia (n = 50), Philippines (n = 50), Singapore (n = 50), United Kingdom (n = 77), France (n = 77), Financial Services (n = 166), Manufacturing (n = 120) Source: IDC’s Gigamon Deep Observability Study, June 2022.
Notes: Multiple dichotomous table — total will not sum to 100%. Use caution when interpreting small sample sizes.

Table 7: How difficult is it to collect all of the data necessary to provide IT staff with a detailed and comprehensive view into network conditions, infrastructure components, application performance, end-user experience, and security threats?

Australia Korea Malaysia Philippines Singapore United Kingdom France Financial Services Manufacturing
Not difficult at all 10.5 14.3 8.0 10.0 8.0 11.7 10.4 15.7 10.0
Sometimes difficult 55.3 27.3 40.0 32.0 32.0 42.9 27.3 42.2 45.8
Very difficult 25.0 49.4 48.0 52.0 46.0 31.2 37.7 27.1 35.8
Extremely difficult 9.2 6.5 4.0 6.0 14.0 13.0 19.5 14.5 7.5
Not sure 0.0 2.6 0.0 0.0 0.0 1.3 5.2 0.6 0.8
Top 2 Box 34.2 55.8 52.0 58.0 60.0 44.2 57.1 41.6 43.3
Bottom 2 Box 65.8 41.6 48.0 42.0 40.0 54.5 37.7 57.8 55.8
Unweighted Valid n 76 77 50 50 50 77 77 166 120
Australia (n = 76), Korea (n = 77), Malaysia (n = 50), Philippines (n = 50), Singapore (n = 50), United Kingdom (n = 77), France (n = 77), Financial Services (n = 166), Manufacturing (n = 120) Source: IDC’s Gigamon Deep Observability Study, June 2022.
Scale: 1 = not difficult at all, 4 = extremely difficult
Note: Use caution when interpreting small sample sizes.

Message from the Sponsor

Today, over 90% of enterprises operate in a hybrid and multicloud world. In this environment, teams need to collaborate and break down silos by establishing cross-functional teams that leverage common toolsets and telemetry to get the most out of their cloud workloads and applications.

This research clearly shows that organizations are looking for observability to strengthen cybersecurity posture and practices, and yet the number 1 concern of observability solutions today is security vulnerabilities (blind spots). To strengthen cybersecurity and reduce risk, traditional observability tools need to evolve beyond metrics, events, logs, and traces (MELT) to inject real-time network intelligence derived from packets, flows, and application metadata to drive actionable insights. MELT combined with network context delivers true defense in depth and stronger performance management across your hybrid and multicloud infrastructure.

This combination enables you to see the immutable truth about what is happening within the application and network layers of your digital infrastructure. This is the power of deep observability from Gigamon.

Find out more about Gigamon Hawk Deep Observability Pipeline: