Another big problem with the way chains of reasoning decay into mush is that most people know their conclusion before they start. If you have this luxury, then you know in advance that if you're sufficiently patient, you can reach that conclusion using steps that individually have a high chance of being right. Again, this approach rewards longer arguments. How much reasoning is unintentionally done in this way?
It’s a cliché that a chain is only as strong as its weakest link, and that’s true, but in this context the expected damage of a weak link near your starting point is larger than one near the end. The error I mentioned in Andrew Wiles proof was a long way in and he was able fix it in a couple of years. The error didn’t destroy his life’s work, but it could have if it came earlier.
Each field of enquiry has its own assumptions and axioms on which everything is built. When doing research in that field it’s ok to be speculative (high chance of being wrong) in trying to discover new stuff, but it isn’t ok if those starting assumptions are wrong.
All this is a roundabout way of endorsing Occam’s razor. Keep everything as simple as possible. Minimise the number of assumptions that you use and scrutinise them extremely carefully. Beware long, intricate chains of reasoning and always be aware of their probabilistic nature.
I think Bayesian rationality is the response to all this handwringing. Now if only I could plough through this badly written primer :-(