Marginal Thinking

One of the most most powerful ideas I have taken from economics is thinking at the margin. To quote one of Greg Mankiw's principles of economics, "Rational People Think at the Margin". I think like Mankiw occasionally tends toward, this quote is ironically mischievous, and to some extent the mischievousness undermines the profundity of the message. Mankiw is not backing a horse in the old debate of whether "people are rational", or "markets are rational" - the "rational thinking" applies to the analyst, not the actors. So what he is basically saying is "In analyzing a complex system of actors or events in light of possible changes, a rational analyst thinks in terms of how actors/events at the margins are affected by the changes, rather than trying to reason about the affect on all of them, or even an average one".

A good example is if you ask someone "Do mandatory seat belts cause there to be less fatal road accidents?". The immediate common response is, "of course, since fatal accidents can become merely injurious". This is true, but not complete, the followup question is "Do you suppose, by engendering a feeling of safety, that seat belts could cause more dangerous driving, and so more accidents". Here people think about their own driving, and whether it is influenced by their wearing of the seat belt, and conclude it isn't and so answer "No". But it is here that their thinking is less than rational. Yes, most people are good, safe, average drivers in their normal range of experience, and a fairly small accident rate across the population reflects this. But it is the margin where things matter; is there marginal drivers who might feel a little safer with their seat belt on, and so might drive a little closer to the edge? It seems the answer must be yes. So on balance then are mandatory seatbelts good or bad? Well from analysis of actual before/after measurement when laws have been enacted, results are mixed.

This is a fairly unintuitive result, and economists like Steven Landsburg And Steven Levitt have managed to fill books by applying similar reasoning to a range of similar systems, to get similar unintuitive outcomes. The important point is they follow their reasonings with actual experiments to test the reasoning, and it is this that carries the weight of the argument, and allows it to win over the default, naive, intuitive analysis.

Which brings me back to the "ironically mischievous" aspect of Mankiw's principle; that since results from thinking at the margin seldom match with peoples intuition, they are extremely difficult to convince people of over "average case" type thinking. Or in other words, most people are just not rational.

Lessons of Waterfall

No thinking person did Waterfall the way it was described in the literature written by the "Big Process" gatekeepers. Rather, they did what was appropriate to the problem at hand and then did the minimal amount of Waterfall waggle dancing needed to keep the gatekeepers off their back. Unfortunately this had the side affect of making both the gatekeepers and the management hierarchy think that their process was working, and the whole Big Process industry managed to perpetuate itself for well over 20 years. Until Extreme Programming came along, made a lot of noise about being different, and then through their "one right way" dogma managed to continue the charade, just with a different set of gatekeepers cashing in.

TDD as Future Self Paternalism

I have a lot of trouble stopping myself snacking when I am concentrating on a problem. In the moment when my brain is distracted I lack the mental awareness to control my behavior. I can be back and forth to the cupboard and halfway through a pack of chips before it even hits me what I'm doing. For a while, I struggled with this and struggled with my weight, but eventually I hit upon an ingenuously simple solution: just stop buying snacks. Then in the moment of thought, they won't be around to be eaten.

I am not a Test Driven Development (TDD) dogmatist, my coverage only averages about 60%, and I rarely write tests before code. However, I do force myself to write code in units, and write the test straight after each unit. The thing I noticed is that if I delay writing unit tests until after all the units are working together end to end, then because after all the system "already works" my subconscious enthusiasm for writing unit tests falls markedly, and their quality and coverage fall likewise. Whereas if I write the unit tests just after each unit, it's part of "getting everything to work", and so I am willing to put the effort into doing it well.

Ruby Line Noise

Ruby is a nice language but it shares many of the failings of Perl. One of these is with all the symbols it often reads like line noise to someone not fully familiar with the language. This is bad because for a script in the enterprise, pretty much every reader bar the author isn't going to be that familiar with the language. There's just too many choices of language and so the knowledge base is very fragmented.

The absolute worse decision along these lines was the symbolic operator overloading done in the container classes. Now, symbolic operating overloading may be fine as a general language feature to allow for well targeted use cases. You dont really want to be doing "new BigDecimal(5,6).plus(new BigDecimal(1,1))". But if you put shit like "|=" and "<<" in your basic Array class, it only encourages the dimwitted masses who follow to do similar. And because they are dimwitted, they won't pay enough attention to getting the semantics consistent with expectations, they probably won't even write a comment or a unit test, and you end up in a unreadable, unmaintainable mess. But hey, at least its a concise mess, right?

The simple rule here is: favor general readability over conciseness understood only by experts.

Pro Static Typing Argument 1

When you are writing code you are making what seems like one hundred decisions simultaneously on a whole variety of levels, from what algorithm to use to what data structures to use all the way down to what to name variables to make them consistent with the rest of the code base. I think even the best engineers can only keep a handful of these in the front of their brain at any time as "rational decisions", the rest are just taken care of by intuition and habit. Most of the time this works fine, but every so often the habitual or short circuited intuition decision is a bad one. Thus an important criteria of which tools and languages you use are ones that have very slow cost in exposing these errors. I think static typing is one of these tools.

Performance Reviews are not about Reviewing Performance

Annual performance reviews are not about reviewing, rewarding or appreciating every employees relative performance. This is impossible as basically every employee thinks they have performed relatively above average. Rather, they are about shaking out the marginal employees who would quit in the next year if they don't get some status recognition, deciding if the company really wants to keep those employees, and if so giving them a plum. Everyone else gets turnips.

Databases and Refactoring

Perfectly normalized database schemas are an anathema to having well factored code in the long term. The problem is even a simple refactoring like "Rename Column" is a tremendous amount of operational risk, and simultaneous work in all dependent services. The more I have worked in cases like this, the more I favor storing any field which doesn't need to queried on as part of a JSON blob. These are structured, human readable, easily read from almost any type of client, and much easy to put in policies to do with versioning and deprecation without risking an outage.

Dysfunctional Communication

It is often overlooked that dysfunctional team communication can go two ways. Sure, you can communicate too little. But you can also communicate too much and get stuck in a quagmire trying to rank competing intuitions rather than getting things done. You need a "gray space of trust" where individuals can get their part done without needing to fully justify their intuitions.

How Code Is Produced

Naive programmers believe that when they program they are using their "logical proof" part of their brain - the code is a "proof" that can be verified by inspection. In reality they are using their "intuitive pattern solving" part - just building up an imperative map from input to expected output. This is why unit tests are so important. Code without tests is just a hypothesis with no experimental evidence.

Functional Programming

People generally think best in terms of recipes, the imperative. Machines reason best about declarations. Functional programming is based upon the notion that if you limit humans to express themselves to terms favored by machines, then better software will result. This seems misguided.

Fragments Are The Only Form I Trust

The contents here are going to be short and pithy. No stories or anecdotes for comfortable consumption. Because they are short they are going to generalizations, and as such they are going to be something short of the truth. The hope is they contain some truth. They are certainly not nothing.