Table of contents
Flowchart
Box-and-arrow diagrams.
People sometimes use three common symbols:
- Rectangle = Process
- Oval = Start or end point
- Diamond = Decision point (i.e., a conditional, with usually two branches)
Also see business process mapping.
Swimlane diagram
Similar to a flowchart, but with steps separated by role (or person, or team, or system) into "swimlanes".
https://en.wikipedia.org/wiki/Swimlane
Causal diagrams
Causal diagrams include causal loop diagrams, directed acyclic graphs, and Ishikawa diagrams.
Cause-and-effect diagram (a.k.a. fishbone diagram or Ishikawa diagram)
Common uses of the Ishikawa diagram are product design and quality defect prevention to identify potential factors causing an overall effect.
Causes are usually grouped into major categories to identify and classify these sources of variation.
Five whys looks like a good companion tool to this.
Causal loop diagrams
Loops
Balancing loop:
Increasing "Adoption rate"
decreases "Potential adopters"
which decreases "Adoption rate" .
Reinforcing loop:
Increasing "Adopters"
increases "Adoption rate"
which increases "Adopters" .
CLDs help you see causality across multiple nodes — that is, 2nd-order effects, 3rd-order effects, etc.
Such causal loops are feedback loops. The output of a node eventually comes back in to that same node. A loop can be either reinforcing or balancing. Reinforcing loops are superlinear: they increase or decrease the value of something increasingly fast.
Finding feedback loops like this feels like a good way to learn potential control points in a system, but you surely have to be very careful about the links of causality that you create. Each link should have as much supporting evidence as possible.
Cynefin (Clear, Complicated, Complex, Chaotic)
Cynefin (Welsh for "habitat") helps you understand what kind of an environment you're in, and what type of response works best (or is possible to begin with).
In Clear and Complicated you can analyze and come up with a tight solution.
In Complex and Chaotic you need a lot of trial-and-error.
https://en.wikipedia.org/wiki/Cynefin_framework
Software development mostly takes place in the Cynefin Chaotic domain, and rarely in the Complex domain.
Agile is about steering. It is not about “controlling” work, it is about choosing work in small slices, doing the work, and seeing what happens.
Software implementation is problematic, but not because of lack of theory, but because it is inherently a process of learning, discovery, and complex interactions between people.
Value-stream mapping
This is a good video featuring John Shook: https://www.youtube.com/watch?v=4YCi57K2jac
Notes about the video
VSM is a system design tool, not only a waste finding tool.
Learning to see and measuring the degree of variability are the two most important goals of VSM. "If you don't know what else to do, first make it visual". Find some way to make what's happening visual, so that you can see why things happen in the process — so that you can see the flow, and the lack of flow (stagnation).
When you first step into the "factory" (or any other kind of place where a process happens), you have no idea what the hell is going on. Therefore you must first learn to see what the hell is going on.
Here you have to first admit that you have no idea. Don't pretend you know what's going on. Unknown unknowns -> Known unknowns -> Known knowns.
Sidenote: This makes me think of reading someone else's code. To be able to make changes to a program, you have to first understand how it works. Seeing the program's callstack as text or as an image helps you see what the hell is going on. Of course you first need to know how to read code. If you're in someone else's domain, you might have no expertise to understand how the details work, and you'll have to trust the experts who work daily in that domain.
At first John Shook was reluctant to use VSM as the primary way of designing systems, and he only thought of VSM as one tool among many. But eventually he realized "VSM helps you see operations differently — instead of always getting sucked down in the weeds and not seeing how things related, [VSM] could actually help people see [how things relate]."
Btw, Henry Ford originally called his assembly line "flow production", not "mass production".
VSM is also called MIFA (Material and Information Flow Analysis), but it could have just as well been called "Goods and information flow mapping". Material, goods, whatever.
Apparently VSM is even more powerful in types of processes that are difficult to see by default. It helps us see. This makes me think of Cynefin-complicated or Cynefin-complex systems. In the video, John Shook also comments briefly on Cynefin.
These books contain chapters or sections about value-stream mapping:
- Lean software development
- Data management at scale
- Contains a surprising amount of stuff about value streams, related to business capabilities
- The personal MBA
Here is another, older video about VSM: https://www.youtube.com/watch?v=AtZPQ-b4jyQ
Graphical methods
https://en.wikipedia.org/wiki/List_of_graphical_methods
The idea here is to collect data (or have experts come up with estimates) and just present the data graphically. Just seeing a graph and extrapolating it to the future graphically can be enough to make decisions.
Related: https://en.wikipedia.org/wiki/Exploratory_data_analysis
Monte Carlo methods
https://en.wikipedia.org/wiki/Monte_Carlo_method
The point is to come up with min-max estimates for the various variables in a system (with a 90% confidence interval and specific probability distributions for those variables) and then probabilistically simulate what happens in the system across time based on those estimates.
This allows you to do quantitative analysis even though you don't have exact numerical data.
How to measure anything (Hubbard) explains how to do Monte Carlo simulation in Excel. Hubbard has templates on his site (see chapter 6 downloads).
Agent-based modeling
https://en.wikipedia.org/wiki/Agent-based_model
It's fun to create agent-based models with a programming language. Each part of the system is a separate entity, and you can visualize material/information flows between those entities.
Final thoughts
Observation and seeing is the first step
https://youtu.be/l4-vpegxYPg?si=ASn0lc3MliLU1Lc5&t=1242
You need to first see. Humans are very fickle. We usually only see what we want to see. We suffer from confirmation bias. Therefore the first step is already very hard and important: see all of the system.
There's no point in creating a detailed mathematical model if you don't have enough knowledge to even know what the variables are, let alone what the important variables are.
Knowledge and learning comes first. In my experience, the best modeling tools during learning are simple flow charts and bullet lists. I might also start using causal loop diagrams, since they help with finding variables and relationships between those variables. Exploratory data analysis should be useful as well, if data is readily available or easy to collect/derive.
Related: https://en.wikipedia.org/wiki/Bloom's_taxonomy
Readings:
- Thinking and Deciding (Baron). Chapter 9 explains "actively open-minded thinking".
- See John Shook's comments about learning to see
- An Introduction to General Systems Thinking (Weinberg) also has a lot to say about observation and humans as observers, although the text is quite hard to follow
Assumptions
All models contain assumptions. Those assumptions may be wrong. See the famous quote.
Assumptions help keep a model simple. A model is always an approximation.
A brain contains a model of the real world (or multiple models, if you like). That model contains assumptions (heuristics). When those assumptions are systemically wrong in specific situations in the real world, we call them cognitive biases.
Types of models
There are many ways to categorize models.
https://sebokwiki.org/wiki/Types_of_Models
- Informal vs formal
- Informal: a simple picture; a text description. Has the problem of ambiguity: the "language" is not well-defined, so it's difficult for people to agree what the informal model describes.
- Formal: uses a well-defined modeling language, like algebra, calculus, graph theory, or a programming language
- https://www.sciencedirect.com/science/article/abs/pii/S0950584917304834
- Physical vs abstract
- Physical: concrete representation. Engine, processor, RAM, tire, wheel.
- Abstract:
- Descriptive (i.e., logical). Logical relationships between parts of the system. "Has-a", "is-a", "calls", "extends", "polls".
- Analytical (i.e., mathematical). Mathematical equations that describe the system and its parts.
- Dynamic (e.g., system dynamics, or an agent-based model). A model that takes into account how time affects the system state.
- Static. Doesn't take time into account in its computations.
The book Schaum's Outline of Operations Management contains this list:
- Complete certainty (all information)
- Algebra:
- Break-even
- Benefit/cost
- Calculus
- Mathematical programming:
- Linear
- Nonlinear
- Integer
- Dynamic
- Goal
- Algebra:
- Risk (some information)
- Statistical analysis:
- Objective and subjective probabilities
- Estimation and test of hypothesis
- Bayesian statistics, decision theory
- Correlation and regression
- Analysis of variance (ANOVA)
- Nonparametric methods
- Queuing theory
- Simulation
- Heuristic methods
- Network analysis techniques:
- Decision trees
- PERT and CPM
- Statistical analysis:
- Extreme uncertainty (no information)
- Game theory
- Flip a coin
Validating a computer simulation model
If you create a model and don't validate its accuracy against the real world, you can't confidently make predictions with the model.
Not validating the model makes it difficult to test how changing the parameters of the model would affect the real world, since you're not at all sure that the real world responds to the change in the same way as the model does.
- Step 1. Build a model that has high face validity
- A model that has face validity appears to be a reasonable imitation of a real-world system to people who are knowledgeable of the real world system
- Face validity is tested by having users and people knowledgeable with the system examine model output for reasonableness and in the process identify deficiencies
- Sensitivity to model inputs can also be used to judge face validity. For example, if a simulation of a fast food restaurant drive through was run twice with customer arrival rates of 20 per hour and 40 per hour then model outputs such as average wait time or maximum number of customers waiting would be expected to increase with the arrival rate.
- [Test that the outputs make any sense when you change the inputs, even for extreme values of the inputs]
- Step 2. Validate model assumptions
- Structural assumptions about how system works
- Assumptions made about how the system operates and how it is physically arranged are structural assumptions
- [Physical arrangement, parallel or sequential operation, types of relationships between parts, etc.]
- Data assumptions
- Lack of appropriate data is often the reason attempts to validate a model fail. Data should be verified to come from a reliable source.
- A typical error is assuming an inappropriate statistical distribution for the data. The assumed statistical model should be tested using goodness of fit tests and other techniques. Examples of goodness of fit tests are the Kolmogorov–Smirnov test and the chi-square test. Any outliers in the data should be checked.
- Step 3. Compare the model input-output transformations to corresponding input-output transformations for the real system
- The model is viewed as an input-output transformation for these tests
- Data recorded while observing the system must be available in order to perform this test
- The validation test consists of comparing outputs from the system under consideration to model outputs for the same set of input conditions
- Hypothesis testing
- Statistical hypothesis testing using [Student's t-test] can be used as a basis to accept the model as valid or reject it as invalid
- [Read more about the t-test in "How to measure anything" and "Statistics in a nutshell"]
- Confidence intervals can be used to evaluate if a model is "close enough" to a system for some variable of interest
- The difference between the known model value, , and the system value, , is checked to see if it is less than a value small enough that the model is valid with respect that variable of interest
- [...] statistically independent runs of the model are conducted and a mean or expected value [...] with a standard deviation [...] is produced
- Graphical comparisons: If statistical assumptions cannot be satisfied or there is insufficient data for the system, a graphical comparisons of model outputs to system outputs can be used to make a subjective decisions, however, other objective tests are preferable
https://en.wikipedia.org/wiki/Verification_and_validation_of_computer_simulation_models
Validation is an iterative process through those three steps, not a single event. Especially step 3 might take a long time.
Sidenote: All of the above is for computer simulation models. I wonder what we do with qualitative models. Do we only do steps 1 and 2 (face validity, and a subjective validation of the assumptions)?
Making changes to a system
Constantly making changes to a system is a core principle in Total Quality Management (TQM) and continuous improvement (e.g., Kaizen). Deming's plan-do-check-act (PDCA) cycle is an iterative process that makes this concrete.
- Plan: Identify a problem for improvement in the real-world system. Develop a hypothesis or action plan for improving it.
- Do: Implement a small change in the system
- Check: Measure the results of the change
- Act: If the change was successful, standardize it and integrate it into the system
— ChatGPT 4o
I think we need to be aware of this distinction:
- Building a model of the current system
- Changing the current system, which gives you a new system
After the "Do" stage, you would need a new model in the next "Plan" iteration.
If the model is accurate, you should be able to use it to simulate and predict how the system will react to a change: make a small change in the model, check if the model's output improves, and if it does, make the same change in the real-world system.
Value-stream mapping has a lot to do with continuous improvement: https://www.youtube.com/watch?v=4YCi57K2jac
(And continuous improvement has a lot to do with lean thinking and agile.)
After you have built a value-stream map of the current system, you can additionally build a map of the desired system. Then implement the desired system, check the result, and act on the results (i.e., backtrack or keep the change).
Continuous improvement emphasizes incremental change. This reminds me of the scientific process (change one thing at a time) and the agile software engineering process.
Continuous (incremental) improvement sounds like a process that would not necessarily even require building a complex model in the "Plan" stage.
Like in A/B testing, you can make a small change, check if it made things better or worse, and react. Taken to the extreme, you can make a change fully randomly, without any planning (i.e., without a model). That's similar to random genetic mutations. That kind of change would be very inefficient.
I suppose this goes back to Cynefin. In a Clear or Complicated system the work needed to create a quantitative/computational/mathematical model isn't too high. The problem is that most of software engineering and business resides in the Complex and Chaotic domains, which makes analysis unrealistic.
No matter how small a change you make to a system, it might be dangerous in Complex or Chaotic systems. A small change can have large and long-lasting results, since prediction is difficult or impossible in a complex or chaotic system. Therefore it makes sense to limit incremental improvements to a small subset of the users of the system. Incremental in every sense of the system: its structure (e.g., only change one module at a time), its users (e.g., only make the change for a couple of users at first), etc.
Flow charts and other simple modeling tools contain many qualitative aspects, but people still use them to help with making changes to a system.
I think even a very qualitative plan (e.g., a flow chart or a textual representation) can be enough as a basis for change: "This part of the system is too complex, and should be replaced with that existing library." In that case I think you really need strong forms of evidence (e.g., expert judgement). Idea for further reading: evaluating evidence.
Different levels of a system might need different types of modeling. You can improve a system at many levels:
For each desired change, make the change easy (warning: this may be hard), then make the easy change. - Kent Beck
I like that. Perhaps its correct to say this (using calculus terminology):
- First derivative: change the system to optimize the output of the system
- A mathematical model could work here (e.g., calculating the runtime of the functions in a module)
- Second derivative: change the system to optimize the process of making changes to the system
- This level might be too large to model realistically with a mathematical model, and you might use a simpler model instead (e.g., a causal loop diagram showing the long-term effects of a bad architecture, or a flow chart showing module relationships and team boundaries)