What I Infer As 10 Keys To Effectively Modularize Cloud Infrastructure Firmware

Architecture of firmware defines the life cycle of a product. What initially starts as a simple product could expand into many subprojects and quickly devolve into a portfolio of products. And all of these products could be derived from the same initial code base. So having a monolithic code base as the initial code will bog down execution and delay product launches and thus their viability. This clearly shows the need for having a modular architecture for the firmware that will allow for a quicker and easier expansion and thus adoption.

Now that we have set the tone for modularity in firmware, let us look at the key attributes that need to be maintained for achieving this effectively:

  1. Gross function modularity – There are several gross functions in a system like server hardware initialization through system BIOS, server health management through Baseboard Management Controller function, Chassis Health Management in case there are shared resources at the chassis level with multiple server nodes, and Rack level management of the resources at the rack level.
  2. Fine function modularity – Within each of the gross functions mentioned above, modularity needs to be emphasized in their respective code architecture. Instead of one single application that has several lines of codes without any boundaries, defining different modules with well-defined boundaries, clear responsibilities, and protocols for the exchange of data is essential.
  3. Dependency – While defining in modules is key, making sure that there is low coupling or dependency between the modules is also essential. What this means is that the architecture need to be planned up front taking into account compile and run time dependencies between the different defined modules and necessary coding methods needs to be applied so they are kept to a minimum. If this rigor is not adhered during the design process, it will lead to redundant code bases and thereby increased time, cost, and eventually leads to excess waste.

In Linux Foundation OpenBMC for example, there are many different system services including sensor monitoring, redfish, web server, ipmi, etc, that are managed by Systemd.  While each system service is to enable a special feature, they might share some libraries with other services, but they also maintain their independence to each other.

  1. Encapsulation – How and who accesses each modules needs to be clearly defined using encapsulation rules. Encapsulation allows for accessing data from a particular module without knowing the internal details of a module. It restricts any arbitrary access of data from other modules that do not have the necessary privilege access rights. Thus use of encapsulation in the architecture reduces the impact of changes in one module to others.
  2. Extensibility – Extensibility is the ability to add a new feature to the application in a seamless and low touch approach. If features are implemented as firmware modules, that adhere to the principles of modularity, low dependency, and encapsulation, then it is easy to add a new feature.

In Linux Foundation OpenBMC for example, Encapsulation and Extensibility is achieved through D-Bus.  Process A is responsible for updating some sensor reading to D-bus and Process B checks if there is any abnormal reading from D-Bus and takes action.

  1. Scalability and Portability – Firmware architecture needs to be scalable to various hardware configuration, sizes, feature etc. and should have a path to easy migration to other hardware if needed. Hardware Abstraction Layer (HAL) is used to provide that isolation to the hardware specific firmware modules’ initialization and control from other modules. Access from modules into the hardware registers need to always go through Application Programming Interface (API) of the HAL so any hardware register changes are isolated from the code.

In Linux Foundation OpenBMC for example, it separates the code to the common phosphor layer and the project-related layer, so the common phosphor layer could be leveraged for all platforms. OpenBMC community also provide guidelines to create a new project with some customizations. More information on this can be found in the link below:

https://github.com/openbmc/docs/blob/master/development/add-new-system.md

  1. Re-usability – Architecture of firmware should promote Re-usability to keep amount of changes to a minimum from one product version to the next.
  2. Fault tolerance – Scope of faults and desired actions on recovery from various fault conditions need to be defined and clearly instrumented in the architecture for every module. In Linux Foundation OpenBMC for example, because all system services are managed by Systemd, we could define what actions it should upon the detection of a particular service that crashes.
  3. Testability and Verifiability – Modules should be testable and verifiable. Customer requirements should be broken down into firmware functional requirements that can be implemented as features of the modules and the modules then should be tested to verify correctness of functionality or features to see if those external requirements are met. Unit test driven firmware development provides the tool for verification of functionality as long as the requirements are well decomposed and distributed to modules. Once that is achieved, then, integration testing should be carried out to verify performance as well. Nightly regression or continuous integration testing using build system tools such as (Jenkins, Bamboo, etc), build system sanity checks (to see if the interface definitions break the rules), static code analysis tools, and unit test automation should be part of the firmware development cycle to catch the issues early.
  4. Maintainability – In the product life cycle, maintenance of a module is the single most critical aspect and takes the most amount of resources as opposed creation. So care needs to be applied in strict coding conventions, good documentation practices, naming conventions, visualization of relations between modules using dependency graphs, inheritance and collaboration diagrams etc. In the case of an open source project like Linux Foundation OpenBMC, while most of the source code could be leveraged, the key is how to manage / integrate / and validate those that are the customized portions

There you have it – The 10 keys that that are used to optimize the Cloud Infrastructure firmware are Gross function modularity, Fine function modularity, Dependency, Encapsulation, Extensibility, Scalability & Portability, Reusability, Fault tolerance, Testability & Verifiability, and Maintainability.

As always, it would be great if you can share some of your thoughts, your own experience, and insights on the keys to achieving optimization on Cloud infrastructure firmware. We are keen to know whether the tenets that we are considering for a modular platform are relevant to you and if so, in what ways? We would love to hear your feedback in the comments section below. In the next blog we will test the core tenets of modularity that we have discussed in this series against our existing product roadmap.