Keep trying to connect the dots, and you'll remain blind to the future

Originally appeared in Inside Knowledge Magazine 10 Sep 2008, Vol 12, Issue 1.

On 12 September, 2001, I received an e-mail from the CEO of my company (a federal contracting firm located just outside Washington DC). As F-16s continued their combat air patrols over my neighbourhood, I read, paraphrasing: ‘John, yesterday [9-11] was a failure of knowledge management. In the years to come, this will be the critical area for improvement’.

We soon heard about failures to ‘connect the dots’ regarding behaviours among flight school students, an arrest in the Midwest not shared across the FBI, and so on.

Seven years forward and US national security is changing. ‘Need to share’ is the buzzword, hoping to replace ‘need to know’. The director of National Intelligence releases a vision calling for sharing intelligence with law enforcement. The Department of Defense releases its first Information Sharing Strategy. The implication, never explicit: if only we get the right knowledge to the right person at the right time, we can know the future and learn which dots pose a threat.

When then-National Security Adviser Condoleeza Rice stated “I don’t think anybody could have predicted… that they would try to use an airplane as a missile”, she was wrong. Someone in government had actually considered that scenario. There are thousands of scenarios considered daily across the national security system – some will always be seen in hindsight as predictive. While technically incorrect, Dr. Rice pointed up an underlying truth. There are thousands of scenarios considered daily and we do not know which scenario, which threat, which dot deserves our attention before the fact. And if we keep assuming there is a golden thread that, if pulled, will unravel the future – we never will.

Systems scientists, organisational theorists and business leaders are beginning to work in a world where control can be an illusion and adaptation preferred. We are starting to focus on nurturing networks and relationships;a recognition that certain systems are, by their very nature, non-linear, and they change their behaviors based on their starting points and the random events that might ensue, leading to emergent new behaviors that cannot be predicted.

Anticipation replaces prediction. While linear models are generally developed to predict the future; complexity helps us anticipate developing patterns of behavior.

In reforming the US national security system, it is vital that we question assumptions regarding the predictability of our world and instead understand that we connect not to find the haystack needle, but in order to better understand and discern patterns in the noise. The subtitle of the interim report from the Project on National Security Reform is a good beginning: ‘Ensuring Security in an Unpredictable World’. How we apply KM and complexity principles to national security reform will shape our ability to secure the nation’s future.

Words mean things.  One of the more obnoxious statements of the obvious, and yet I find myself saying it more often these days.  The more I delve into understanding complexity theory, network science, and struggle to understand cognition and neuroscience, the more frustrated I get when people use terms in ways appear at odds with the literature.

As I was preparing this blog to address the use of ‘complex’ versus ‘complicated,’ I found that I am certainly not alone in trying to retain some clarity of language.  Paul Jansen, in particular, has a great blog post on exactly this topic.  Nevertheless, I owe the nice people who followed this exchange on Twitter this week a brief explanation of my frumiosity.

This week, caught up in the holiday mood – I found myself engaging this week in an exchange with a gentleman, Roger Sessions, who has developed a method for IT architecture designed to ‘reduce complexity.’  His paper features references to “attacking complexity” and includes a method for measuring it.  He introduces the “standard complexity unit,” based on something he refers to as “Glass’s Law,” which posits that for every 25% increase of complexity in a problem space, there is a 100% increase in the complexity of the solution space. This reflects work from a 1979 paper by Scott Woodfield, who first posed this idea.  The idea is that increasing the complexity of problems tackled by software engineers does not increase the complexity of the solution in a linear sense, but on an exponential scale.  It is this problem that Sessions seeks to take on with his approach.

Now the notion of reduced complexity is attractive, if you understand complexity as a system that has developed so many connections as to become unmanageable. This is a common usage for ‘complex,’ which seems to translate to “something too hard to understand or manage or control or cost.”  The notion of ‘wicked problems‘ applies here as well.  The greater the connections you find among things, the greater are your odds of decision paralysis and “failure.”  Solution?  Easy, make things simple.  The danger, for me, comes in simplifying management behaviors in ways that deny the nature of the systems we are attempting to manage.  If you believe complex is nothing more than the ‘opposite of simple,’ you are missing some of the most promising areas of applied research in a half century.

When I engaged the gentleman on his use of the term complexity, I received what I believed was an odd response.  For someone who uses the word in titling his books and lectures, he did not appear terribly connected to the word itself.  He even invited me to suggest a different term for what he was trying to achieve. The closest I could come to his definition for complexity (admittedly, without buying his book) is an ‘exponential growth in system states with regards to information technology systems.’  To me, he is trying to help people with an architectural approach that makes overly-complicated IT systems more manageable.

For his part, Roger was comfortable with my discomfort, because in his world “complex” merely means the opposite of “simple.”  Several of us during this Twitter-fuffle suggested the use of “complicated,” which suggests a system that has known but prolific connections.  Cause and effect in complicated systems are related and knowable, but analysis by an expert will likely be needed to connect them when something goes wrong.  My example here is the ‘check engine light’ on my car – while I am at a loss to understand the cause, an expert with tools can ascertain it quickly.  Modern car engines are extremely complicated.

They are not, however, complex.  My car engine is unlikely to evolve new features anytime soon.  There is a reason medical doctors have different training regimes than auto mechanics.  The latter deal with complicated systems, the former with complex ones.

Complexity is a specific term.  Complexity, as described in the literature, is a science that seeks to explain how emergent order (often called ‘hidden order’ or ’self-organization’) is observed in systems or (most) networks. For what it’s worth, I believe those seeking to develop IT architectures could benefit from a deep understanding of complexity, as their users are sloppy humans in messy and evolving sub- and extra-organizational work networks.  Methods for complex systems management show some promise in ‘attacking’ the unmanageable IT systems that Mr. Sessions is tackling here.  It may be that observing and nourishing self-organization among human-based networks, rather than embedding and enforcing an existing or desired organization within them, will help architects develop more manageable and relevant IT systems.

As a blog post, however, this has gone on long enough.  I just wanted to explain my bristling at a usage of the term ‘complex’ in a way that conflicts with the literature. At one point, Roger reminded me that he is trying to tackle an extremely serious problem.  I respect that, of course, and was doing the same.  Given the great work that is ongoing around complexity and complex adaptive systems, we owe some respect to giants upon whose shoulders we seek to stand.