The Vedas of programming
In the old books, published in the previous century, just before 2000, the old-timers, who as a rule had a deep mathematical or software engineering background (like Barbara Liskov or Joe Armstrong) it has been stated that applying of just a few fundamental principles will solve the most problems, which arise in programming of complex systems.
The Abstraction Principle of Bruce MacLennan, and the Data Abstraction Principle of Barbara Liskov are the most fundamental ones and underline the whole Modular Programming paradigm, which is the main methodology for dealing with inherent complexity.
Abstracting out a recurring (re-emerging) pattern and to give it a name (which is a reference for it to be reused) is how our minds evolved to “work”. This is how we invented our mathematics.
Composing these abstracted out notions is what functional programming is about. Functional composition and interfaces are the very core of any programming. It reflects something deeper that just conventions - it is about how complex systems of nature has been built.
Most of the books of the previous century teach principles, not particulars. The universal data structures (lists or sequences, trees, and lookup tables) and their time and space complexities has been studied in details and declared to be good-enough for everything.
Consistent Hashing algorithms and resulting hash-tables has been considered the last major innovation. Table-based languages, like Clojure or Lua emerged as a result, and these were the last good things that happened.
Decomposition
The best metaphor I know is how a competent and experienced mechanics disassemble a rusted engine, carefully examine and fix (or replace) every single part and then assemble the engine back just like a new. You can get lots of such videos on Facebook.
Of course, programming is very different kind of activity, but the principles are the same. No one gives you an engine, you will have some problem domain instead.
Nevertheless, the first thing to do is to disentangle, decompose and take apart the problem (into subproblems, subsystems, core concepts, rules and communication protocols, which are also sets of rules).
One have to “disassemble” everything down to sets, formal logic and particular algorithms (which has to be derived from the rules of “business logic”).
Only after everything “part” (or detali) has been “placed on the table” (being formally specified) one can begin to assemble an “engine” (this is actually a good metaphor!).
One begin to prototype by enumerating the core algebraic data types, with corresponding modules that exports public interfaces. Then one tries partially implement these interfaces and to compose actual functions. This is a bottom-up process.
Partial implementation is possible because both product- and sum-types can be partially specified and still be correct (just a few slots of a record, just a couple of data-constructors of a sum-type). Functions defined by pattern-matching will be correct by definition.
The whole thing is a spiral-shaped process, where previous results act as foundation of the next iteration or a “sprint”.
The key point here is that both actual representations and conceptual understanding are mutually refined (yes, just like a mutual recursion) and everything, including formal specifications and validation tests gets simplified and reduced to the fundamental data structures and corresponding standard idioms. This is how one does assembling - by function composition.
Use the fundamentals
In developing, and especially in prototyping of a system the most crucial part is to restrict oneself to the fundamental, well-understood universal building blocks - well-typed function and data abstractions.
It is difficult to sell this in the modern world, but this is the way to program.o
Abstract Data Types, Algebraic Data Types and functions defined by pattern-matching (in small, specialized, loose-coupled modules) is all you need.
This was the major result and the peak achievement of the Golden Age. All You Need Is Lambda, was the golden mantra of the time. This is still true. Lambda and a lot of syntactic sugar and advanced typing, which is what Haskell is.
In short, if one does not want to fail with a programming project one has to do it in the now “ancient” way, because not just this is the only way, but it is already the Vedas of programming.
The Upanishads of programming would explain Whys and Hows and this is what I am trying to write time and again.
Remember that things like Erlang or Scala or Ocaml has been actual experimental results to confirm our principles and methodologies. You probably know that WhatApp is Erlang and Twitter is Scala.
Why these things work is exactly because they compose well-typed and well-understood fundamental abstractions as stable intermediate building blocks. You probably should do it too.
How do we program?
Well, we just do what Erlang and Clojure authors advocated We write small pure functions (turn functions into pure ones by passing a context). We FFI everything (no more hundreds of idiotic low-effort dependencies) We encapsulate exception throwing stuff inside a Monads.
Basically, we do a Bird-Wadler textbook style Haskell without dependencies.
We write our libraries as layered DSLs (a-la TeX or the Esher pictures DSL).
With some effort other decent languages (Ocaml, Scala 3, F#) can be used in the similar manner. Less is indeed more.
And this is it. This is Vedanta all over again.