• February 2024 – AT&T’s cellular network goes down
  • Attorneys General of several states have announced investigations into the outage
  • Purportedly caused by improper process during a network expansion
  • AT&T is refunding $5 per user which will cost them millions for their avoidable actions
  • First responder FirstNet was supposedly part of the outage
  • We may never know the cost of SLA penalties or reputational damage
• January 2024 – Microsoft Teams suffered a mass Teams outage which they blame on a networking issue
• February 2024 – Red Sea undersea fiber optic cables are cut by factions in a civil war interrupt global service

Cloud exists in a datacenter, and network on physical infrastructure somewhere. How many of us ask to see the cable routes, network designs, resiliency plans, or demand testing and results for such services? I was correctly tortured as a young engineer at NYNEX by some of the smartest technology minds in the financial industry for such evidence. Yet today we incorrectly believe that such diversity and resilience is automatically there because its IP, right?

Your vulnerability is no longer the bored teenager, criminal hacker enterprise, or nation sponsored bad actors. On public shared infrastructure, centralized universal changes have universal consequences. Our quest for buzzword philosophies like ‘lean’ and ‘agile’ have encouraged and rewarded decreased supervision, expertise, process, and documentation which can lead to carelessness. Failing your customers with preventable service interruptions of any type is not a laudable corporate virtue.

The Solution

I use Skype daily to speak to colleagues and friend around the globe. Its free, simple to use, generally stable, and has not changed much over time. But I would never use it for highly confidential communication. Recently, German defense officials on a Webex shared public platform call were discussing sensitive Ukrainian war plans. The Russians eavesdropped on the call even though the German military had a secure internal network that they should have used. A 38-minute audio recording was released publicly embarrassing the German government. They had vast technology resources at their disposal. Could your business withstand a 38-minute audio leak of your most private discussions?

Collaboration tools such as Teams, Zoom, and Webex, are rapidly changing the voice communications landscape. Despite their issues they provide a good service for a competitive price. Like cell phones, they are not perfect, but they get the job done and provide additional features and functionality that make them worthwhile. However, an enterprise must consider if they want the critical content of their collaborations to use the same shared public infrastructure as Grandma speaking to her grandchildren.

Traders use turrets and private wires for speed, confidentiality, and guaranteed performance when they need it most. Turrets have a privacy/barge in buttons for a reason. Regulators and Financial institutions spend a fortune on compliance to prevent violations from alternative, unmonitored, unrecorded communications platforms such as social media communications services. Some financial firms have large global private networks larger than many service providers. Yet they and others use the public platforms of these collaboration providers. Ask yourself or better yet ask them:

• Do they provide a recording interface to your private system?
• How are you securely partitioned?
• Do they record the video, shared screens, shared files, and chat?
• Do they offer private infrastructure or hybrid cloud/data center deployments?
• Do they offer any access method from analog to WebRTC?
• How do they prevent phishing and social engineering attacks aimed at their own users?
• How do they respond to compromised accounts?
• How do they prevent data leaks?
• How do their API’s prevent risk from malicious 3^rd party apps?

This is the equivalent of PBX versus Centrex telephony. Public services simply cannot address it all and be all things to all people, the same way your business cannot. You need a private service that inherently locks out the public and provides a secure platform that is built for secure, compliant business. One such service is XOP Networks ConferWeb collaboration platform which ticks each of these boxes. The ConferWeb platform is designed for deployment behind the secure firewalls of an enterprise, thereby ensuring that the security of end user’s collaboration experience is not impacted by the vagaries of the service provider networks. Additionally, their Universal Services Node (USN) provides the any to any protocol conversion that can securely bridge all parties from analog to WebRTC and integrates your existing IAM tools such as LDAP. These mature technologies are out there waiting to be exploited and have been in use in the financial and command and control markets supporting critical infrastructure and services for over 20 years.

Conclusion

In the introduction, my contempt is not for Lean or Agile methodologies in software development or even in faster, leaner more general projects. It is for poor leaders who try to rationalize expediency and frugality with the philosophy du Jour which they do not fully understand and use as air cover for bad practices. Do you think the CEO of Wendy’s now fully comprehends the meaning of surge pricing? Buzzwords and viral trends are not a substitute for experience, knowledge, leadership, management, oversight, governance, and plain hard work. The concept of a minimum viable product is solid, depending upon how minimum viability is defined. And for more mature products, we see that poor process and supervision, most likely due to cost cutting, are still possible even for high tech giants.

While time to market is important, so is quality and service. Embracing error as a positive learning experience for your self-managed team simply does not work for critical infrastructure. You the customer pay the price for simplistic LinkedIn video sound bite management style. Critical communications simply cannot be allowed to fail! If your team does not understand that, then you may need a new team or a new team leader. Infrastructure must be highly available, fault-tolerant, tested prior to implementation of any expansion, patch, or upgrade, not have single points of failure, and be fully and instantly recoverable. How many of the 14,000 impacted Teams users do you believe got a live agent to help them on Saturday January 26^th? Financial leadership means so much more than reducing expenses to meet an artificial target. There is an adage that you can it have good, fast, or cheap: pick any two. If you select fast and cheap, ask yourself if you are ready for your darkest secrets to be on CNBC tomorrow or for your customers to be down in the middle of a business day?

Bill Wagner is a financial industry technology consultant with over 30 years’ experience as an industry executive in hardware, software, engineering, operations, R&D, product development and introduction, and strategic development.