Fusa: Model driven software development
FUSA is the glue between applications that saves you time, money and grey hairs.
Every client wants
· high performance
· rock solid reliable
· scalable in both CPU and IO bound spaces
· fully tested
· smartly monitored
· documented
· Fits to your organisation’s coding and production standards
at the lowest cost and time.
Can I do that as a single technical architect/developer? Yes I think so using model driven engineering toolkit I’ve created called Fusa.
Lazy programmer syndrome
The best programmers are lazy programmers!
We want to automate everything. If it takes longer than 30m to code and if I have to do it more than once then it should be automated as a general rule.
With the lazy (but productive) programmer we want to write the least amount of code and get the best results. We want to solve interesting problem and not spend time writing test cases or documentation.
Low/No-code tools
In the early 2000s there was the rise of 4GL, the notion that you could visually describe your system and you’d get your code, fully working and bug free. Well, that didn’t quite happen, as then StackOverflow wouldn’t be popular. Technology has moved on and now we have website builders like Wix, and slightly more functionality focused low-code tools like Bubble.io and Webflow. They are excellent tools for the front end, as long you are working within their boundaries.
What about the server side?
Given the plethora of adverts and jobs looking for server side developers in all industries we can be fairly sure that writing code is pretty complex undertaking and until AI can do this we will be secure in our jobs.
However just like the front end tools mentioned previously we can make the case that similar code generation techniques can be used to to glue together components in order to build bigger, better tested and more reliable systems faster.
Developer buy in
I’m a developer and I know how hard it is to sell a tool to other coders who typically have a Not Invented Here approach to almost anything. I understand it, it’s fun to solve complex problems and off the shelf tantalisingly almost but not quite solve your problem requiring you to delve into the code and almost re-engineer it to your environment and situation.
What problem are we trying to solve?
The reason you move processing onto the server is because you have
Significant data IO
Intensive compute requirements
Scalability
FUSA
I created the FUSA tool to be a domain specific language that you could define specific to the customer. i.e. not be tied into a model or language.
My use cases tend to be that I’m working with a client (usually a trader or a quant) and they want a lot of data from multiple sources, do some kind of calculation (pricing or risk) and then produce results to be sent along their way.
So we have to describe the following
data-sources
data-flow
compute
Let’s go through the steps
Step 1: Define a way to describe data entities
Let’s use a notation that everyone is familiar with : a simple enum
Declare your data objects with the business.
Let’s see what we get for Python.
With a declarative definition in our own language, we get all the python code generated, and all the tests.
Of course we aren’t restricted to python, we can generate anything we want, code, documentation, diagrams.
Step 2: Add constraints
This has the class definition along with some constraints.
This is a major feature; in the real world every piece of data has a constraint. Developers usually code with the “happy path” in mind and ignore the failure cases which lead to the most issues. The tool forces all data fields to be constrained at compile time forcing both the business and developer to think about the domain.
This reduces string data to enumerations. It reduces numbers to be constrained by limits. It reduces free form strings to well-formed regular expressions.
By just doing this step we can generate close to 100% of all behavioral test cases and ensure your data behaves in production exactly as you expect and can trap with context any failures. Instead of pushing the onus onto developers to write comprehensive logs to be able to fix a problem in production (usually after the fact), this is all done upfront automatically.
All documentation describing your data, where it’s from, where it goes, and what the constraints are is generated ready for Confluence or just a simple HTML page. It’s always in sync with your code as part of your continuous integration process of choice.
Step 3: Where do you want to get it from and in what format?
The above example specifies a json source, but we are free to specify
Databases: Oracle, SQLServer, Mongo, KDB, or any custom format
Messaging Queues: Tibco Rendezvous, IBM MQ, ZeroMQ, or any inhouse protocol
Files: CSV, Binary proprietary formats,
Message formats: JSon, MsgPack, Protobuf, CapnProto
This generates all the serialisation code to/from the various data sources. Robust solid code, that is type checked, and fail-fast error checking.
It creates test cases to read and write data from those stores, testing robustness.
Of course, some places use stored procedures and they may be in a particular company policy format so those can be generated too, along with DDLs to create tables, migration scripts to update tables, whatever you want.
It goes without saying we create all the test cases to ensure we aren’t invalidating key constraints, that we are loading/saving data efficiently. Unlike the current model of using introspection to create entity-relationship-models which are restricted to runtime costs, we can create correctly typed code at compile time along with company specific error handling for every piece of data.
We naturally get documentation updated continually and consistent with the code. This is an uml diagram but equally HTML pages, confluence, even Sparxx Enterprise Architect have been created.
Step 4: Define your dataflow and compute components
So far you have all your IO taken care of, the code, the tests and documentation.
We can now define the flow through the system again with constraints on both inputs and outputs.
This generates all your functional classes along with comprehensive test cases.
Those classes are stateless, hence are scalable across CPU cores and machines by design. They cannot access data outside their inputs. They cannot take in, nor produce output that is outside your specified domain. We are using “design by contract” as a methodology here where parameters are checked for adhering to the contract at all times.
If they fail, without any developer effort, we have explicit logging and notification of where, when, and why with precision.
More documentation is generated, now describing how your system behaves between components. This is excellent for technical presentations, demos, and general handover between teams.
Design by contract
As a side note, design by contract allows us to use property based testing such as QuickCheck. This was made popular in the Haskell language but is an excellent way to generate and isolate failure cases through fuzzing your code. As developers don’t need to write all the infrastructure themselves, it saves time and effort.
Step 5: Finally, the developers take over
They just fill in the computation logic which tends to be trivial to code and test since they do not have to write the test cases. True business driven TDD and BDD implemented at design time.
Non-functional benefits
Separation of dataflow and compute
By following this process, you will have naturally separated out compute and data flow of your system from each other.
With this premise, 2 benefits are realised
Traditionally to scale a process, you run multiple instances of it. But really you have two competing goals.
· IO bound
· CPU bound.
If you are seeing more IO traffic scale that out on cheap CPU with a fast network and disk machine. Once you see a corresponding increase in CPU usage, you only need to scale that part on a high core, fast CPU machine but not necessarily pay for disk or network bandwidth.
This enables you to optimise your usage of machines.
Zero effort smart monitoring and logging
As you have been forced to define your data entities, the constraints and your flow upfront, the developer does not need to write any logging code of his own.
Yet he or she gets the benefit of detailed fail-fast error capture and reporting. Errors are automatically specified to the exact reason they failed — they have violated a well specified business constraint. This is invaluable in production environment where 1st line support normally can only see a failure but do not know the cause. This is free root cause analysis before it even gets to your 2nd line support developers.
SIMD by design
Most developers are not comfortable with assembler coding or using SIMD intrinsics. Unfortunately, when your CPU is showing 100% and is not using vectorised code it’s only really using 12.5% of the machine. You have paid 8x more for the piece of silicon so you should be using it.
The process is designed to facilitate SIMD code generation by the compiler; or make it easy for the developer to do so.
Quantitative CPU bound code tends to simplify significantly and perform faster as all the data the process needs is right there sitting on the L1, L2 or L3 cache rather than in a data base somewhere. If you have followed the process well, they should also be amenable to SIMD calculations thus giving 4x-16x speed up per core anyway saving loads of compute dollars.
For the CTO: Cost savings
The focus of model driven engineering and my particular realisation of it is to act as way to glue components together to build an enterprise wide system where costs savings and time savings are measured in man months or years, not a single application.
Python is known as a glue language for libraries, FUSA is a glue language for applications.
Key man risk
The model language is custom to your organisation and process domain. It describes your whole system in as succinct way as possible. It should be something that a non-technical but domain specialist person can look at and reasonably understand how things work.
If a single document can describe your complete system, creates all the documentation, creates 90% of all the unit , system and integration tests, creates the fundamental code for all the data flow and compute areas, it frees your developers to work on the actual business value add.
They don’t need to spend time on things that add no value to the business but are merely an artifact of the development process.
Wholesale changing of code
Have you sometimes needed to change from one technology stack to another? Maybe switch databases from Oracle to SqlServer? Or perhaps from C++ to Java or C#? What’s the cost of that? What are the areas we need to work on? By following the FUSA process you have all information quantified in a document.
You can explicitly say how many data entities, stored procedures, or IO based code pieces you need to transfer. And funnily enough you only need to do it once, at the level of the generator templates and your complete code base is updated.
Outsourcing
Outsourcing is often like stepping into a quagmire of missed deliverables and specifications lost in translation. By having a well specified domain specific language in house you can generate interfaces directly from the specification. Give it to the outsource team. If their code doesn’t compile then they haven’t delivered to the specification you asked for! It really does drive home a great point.
Integrate as slowly or as fast as you wish into your code base
Because it’s a tool, it can be as comprehensive or as unobtrusive as you want. You don’t have to go wholesale and switch to this. Use it in a small test piece first and start to migrate you data objects first. Then start to add constraints and get free test cases and documentation. Expand the remit and get deployment and migration scripts for free and so on.
Final thoughts
FUSA model is a refinement of the natural need for developers and architects to bridge the gap between the needs of the business and the coders actually doing the work. It’s not a code generator to replace the innovation developers bring to your workplace, but a code generator to remove the drudgery of coding the boring bits.
Contact me
I’m Ronnie Chowdhury, author of FUSA. If you want to get in touch with me email me at ronnie.c995@gmail.com to see if I can help you build out your technical software architectural vision.