Democratize your IT Ops data with OpStream

By Al Wasserberger

Over the past 10-years, we’ve had the good fortune of working as systems integrators focused on IT Operations Management and Application Performance Management for leading enterprises in the global marketplace. Our focus has been on helping these enterprises improve their operationalization of technologies and journeying alongside them towards IT monitoring maturation. It’s been a rewarding journey.

For the past 5 or 6-years, the primary focus for many of our clients—and therefore for J9 as their systems integrator—has been the pursuit of a “single pane of glass.” We’ve essentially been focused on ensuring that our clients were able to tie all of their data together so that event management tools could do their jobs as effectively as possible. And over that same timeframe, the monolithic solutions—the exclusively HP, IBM, BMC and CA systems—were replaced by a loosely assembled set of best of breed tools—like Dynatrace, like AppDynamics, like Aternity, like Extrahop and Solarwinds and NetIQ and Microsoft SCOM—as our customers within the enterprise started to pay more attention to the best ways to manage each of their individual domains. Hence the widespread adoption of tools like Splunk.

So all this time, we’ve been pursuing the “single pane of glass” amidst a marketplace that has been shifting, and J9 has been integral in helping our clients achieve this pursuit. Whether via manual integrations or enablers like our Accelerator technology, many of our clients have managed to tie all (or most) of their data into a Manager of Managers and are able to effectively combine their event, metric and topology data into that single pane of glass more successfully than they otherwise would have been able to do. Our clients are happy. We’re happy. But through this process, we believe that the paradigm has changed. The goal posts have moved.

No rest for the weary.

As we’ve adopted and embraced—and maybe even perfected—the art of choosing best of breed tools, what we’ve found is that the MOM approach, the “single pane of glass,” doesn’t actually optimally equip the enterprise for success. Our experience has taught us that it’s more effective to have the right data being communicated between the right tools, including things like Analytics and Visualization (extending even into BI tools), to empower a “multi-pane of glass” paradigm. Getting the right data into the right tools delivers greater value to the enterprise than getting all of the data into a single tool. And so now we have a world where we demand not only best-of-breed domain managers but best of breed Analysis and Visualization as well.

The new paradigm is a multiple pane of glass. What we need now is many-to-many multidirectional connectivity instead of a many-to-one data flow.

From a business value standpoint, this is simply a fact. But IT Ops teams who have spent the past five years integrating tools to pass data into a MOM, who are maintaining possibly dozens of integrations to ensure the stability of their ecosystem, should deservedly buck at the thought of exponentially increasing the number of integrations within their ecosystem. You’ve worked so hard to be able to get everything into your HP BSM or into your IBM Netcool or into ServiceNow or Moog. The idea of also sending into separate Analytics, Visualization and Automation tools and integrating log analytics into the day-to-day operations and making all of this stuff work together becomes a task that’s not really possible for humans to achieve. Literally.

In the new paradigm, tools need to communicate with each other directly, as opposed to through a MOM-based protocol. The focus therefore becomes on the connections between the tools, so that the right data is being pulled from and sent to the right tools at the right time. Data needs to stream directly between tools in real-time and be transformed on-the-fly.

The integration matrix becomes huge. In most ecosystems, this would require dozens—or even hundreds—of individual integrations, using different protocols, built separately in different languages, by different people, all of which require ongoing care and maintenance to keep up with the changing tools landscape. It very quickly becomes unwieldy—and literally impossible to achieve with 100% reliability—if managed by humans.

Simply put, a best-of-breed IT Ops infrastructure cannot be achieved without automation. Some of you are going to try it, but we don’t think that you need to, because today we’re announcing OpStream.

OpStream is the natural evolution of our Accelerator product. Instead of being a point-to-point product that brings multiple domain managers into a MOM, achieving the “single pane of glass” pursuit, we now enable easy and intuitive connectivity between all of the tools that you use. OpStream is an enterprise-class solution that streams data between systems and tools that you choose.

We’ve built a product that allows our customers to send events, metrics, topology information … and more … all the types of data that an enterprise could reasonably want to share between tools. And we’ve embraced the emergence of stream-based architectures like Kafka, Spark and Storm … which allow—within an ecosystem—a single collection and mass dispersion of data between tools, eschewing the need for separate integrations between each platform within the ecosystem. No more batch processing.

We’ve also made it a priority to ensure that OpStream is scalable and resilient. We’re able to stream data in near real-time to single or multiple locations, from multiple collection points—whether within a cloud or a data center—with a posture that is congruent with the multi-tenant, clusterable, highly-available enterprise products that our customers depend on. We guarantee delivery of data, we have the ability to throttle data, and we’re secure.

In an ecosystem comprised of only 20 different domain tools and five managers, we’re talking about up to 200 integrations, each with different protocols. With 40 domain managers and 10 manager, visualization systems and analytic tools the matrix grows to approach 1000 separate integrations. The sheer volume is challenging – if even possible – to maintain for any IT Ops team. Our goal with OpStream is to remove the pressure for IT Ops teams that might otherwise attempt to manually build and maintain the integrations. When dealing with operational data and events that—if not caught or transmitted—could result in massive problems, maintaining hundreds of unique integrations, all based on different protocols, represents a massive risk.

Over the next several years, we expect that a “single pane of glass” will be viewed as an outdated and suboptimal archetype. The enterprises that win market share will be those who successfully achieve a multi-pane-of-glass ecosystem that allows for best-of-breed Visualization and Analytics tools to operate optimally within their domains.

And with the shift towards this new many-to-many paradigm, the volume of required integrations between best-of-breed tools will increase exponentially. J9’s OpStream is the first enterprise-class monitoring integration platform. Let us help unlock the true potential of your organization’s IT Ops data.

I’d like to take a minute to congratulate the development, consulting and support teams and to thank our clients and partners – all of whom have come together to create this wonderful new system. There have been a few major paradigm shifts in our industry – and I believe you have all contributed to what will be the next big trend.

Democratize your IT Ops Data, and let your tools get to work.


Topics: Opstream

Al Wasserberger

Written by Al Wasserberger