Bias in Technology
As an entrepreneur I am aware of the risks of Confirmation Bias, surrounding yourself with opinions that match your own and using that to justify a position. Present Bias, where a decision is made that has significantly higher future costs based upon immediate benefit is another common bias in technology. However, perhaps the most significant bias in technology is Complexity Bias, choosing a complex solution when a simpler one could achieve the same result.
I was walking into Kubecon 2022 with a talented Engineering colleague, he commented “Kubernetes is so complicated but the people here are so deep in it, working on it every day, they have forgotten how complex it is”. No developer would agree they have Complexity Bias, in fact the predominate marketing position at Kubecon is that their products make Kubernetes and infrastructure simpler. If that’s the case, how come Kubernetes keeps getting more complicated? Take Confirmation Bias from the Kubernetes community that is convinced that their platform is the future of infrastructure, add developers deep in the technology such that they are normalized to the complexity and add Present Bias in the belief is the best way to solve the problem is by adding on to Kubernetes. Voila, a recipe for increasing and perhaps unsustainable complexity.
There is an even better example of Technology Bias, WiFi and Mobile (2G, 3G, 4G, LTE, 5G). Ubiquitous connectivity, everywhere without having to think about it should be the ultimate differentiation for Mobile, but we still have WiFi. WiFi is the poster child of simple while “the G’s” the polar opposite in complexity. Mobile has an alphabet soup of acronyms, each version building on the last adding more complexity, increasing the difficulty for new people and organizations to be involved, therefore reducing the number of skilled people available to continue to work on Mobile Wireless. Conversely, and while not a fair comparison, anyone can install a WiFi AP!
The three Biases are the center of the success of the hyperscale cloud providers. They have gathered up the skilled resources, packaged the complex components like Kubernetes and offered them as a service. It’s no surprise that most production Kubernetes clusters operate at the hyperscale cloud providers. In recent years, they have added Mobile expertise hiring people and acquiring companies to address the eventual reality that the Wireless Operators and their current vendors will not be able to sustain the increased complexity of operating their technology and will turn to them for help, just as so many other organizations have already done. But does the hyperscale cloud really solve the problem. By combining all of this complexity from so many other systems into one platform, with so many users, platform complexity increases exponentially. Cloud providers have to solve the new scaling challenges that no one has solved before every day. I don’t believe that the teams at the hyperscalers are any different from the Kubernetes or Wireless experts, they suffer from the same Complexity, Confirmation and Present bias that everyone else does.
I don’t pretend to know the answer, perhaps its AWS “2 pizza” team rule, but AWS has turned into a sprawl of services. But it’s been long documented that large, highly interlocked development teams don’t get much done.
What I do know is that we are conditioned to pay for complexity and the hyperscale providers have been significant beneficiaries. GitHub, GitLab and other repositories are filled with simple solutions to technology problems that have been abandoned because they didn’t generate enough income for a single or a small number of developers. In fact, those who develop software know that many of these abandonware packages are at the foundation of large scale production systems in use in business today.
Before you write a comment to illustrate how wrong I am, take a minute to consider your biases and permit me to tell a story. In the mid 90’s I lived in Cambridge, MA and was an early user of Cable Broadband, I had one of those LAN City “toast holders”, no dial-up for me. My father had just retired and came to visit. Computer illiterate but released from slow dial-up he became fascinated with the Internet and its ability to find information. One evening we were sharing a drink and he asked me, “Son, how does this Internet thing work?” A question that was not easy to answer……
Before proposing your next solution, perhaps consider if you can explain how your solution would work to someone far away from your Confirmation Bias circle. Perhaps it would result in starting to value the smaller contributions, grow the number of people who can contribute and more rapidly leverage technology to improve our world.
