MetaBooks™ is an end-user development environment that lets business experts (usually an accountant or someone with a similar operational background, i.e. not a programmer) develop a business application without relying on programmers. It is a platform on which many kinds of (transaction based) applications can be built.
MetaBooks™ both competes with and complements spreadsheets. The kind of people who would use MetaBooks™ are similar to those who build large spreadsheets, and people often use spreadsheets to do things that MetaBooks™ will do better. MetaBooks™ is not going to be as good as spreadsheets at doing “what-if” analysis, but is much better at building scalable, reliable and controlled applications. MetaBooks™ applications can import and export spreadsheets, so MetaBooks™ is designed to work with spreadsheets. But we expect that much of the appeal of MetaBooks™ is that it can reduce the number of spreadsheets that a business uses, especially in its operations.
MetaBooks™ is designed for “accounting-like” applications. MetaBooks™ applications are characterized by a set of transactions, each of which changes some set of balances. The transactions are not just actions, they are also data. Customers will want reports where they can see a list of transactions. They might argue about transactions, and want them corrected. Auditors will want to see the transactions, too, and will want to know who performed each one. The applications deal with money, so security is important. MetaBooks™ should be usable by large as well as small companies, so scalability is important. Applications built with it should be reliable, accessible from the web or from a laptop, and easy to connect to existing applications.
There are currently two environments for executing a MetaBooks™ application; bFAST™ Modeling Tool and MetaWeb. bFAST™ is used by a single user our a team of collaborators to create and change the rules that make up the application. A set of rules designed to be used together is called a “model” in bFAST™, because it is a model of the business. Because bFAST™ also executes the rules, a bFAST™ user can test new rules and can set up the initial state of the model before giving it to others. MetaWeb deploys the model on the web, and enables it to run with many users.
Types
bFAST™ includes a set of browsers that are used to build the model. One is the type “browser.” It is used to define the various types of objects in the system, such as securities, transactions such as the purchase and sale of securities, or positions (the amount of a security owned at a particular point in time). Each type defines attributes and associations. Attributes are properties with immutable values like strings, numbers and dates. Associations are properties whose values are objects. Types form an inheritance hierarchy; each type has a super-type and inherits the attributes and associations of its super-type. Types also have calculated attributes, which are defined by an expression and can take parameters.
Those who are familiar with object-oriented programming will notice that these types are similar to the classes of an object-oriented language, and calculated attributes sound a lot like methods. But one of the main purposes of methods is to change the state of their object, while calculated attributes are usually not supposed to change the state. A calculated attribute is more like the expression in a spreadsheet, which only gives the value of a single cell, and does not have side effects. MetaBooks™ objects can change state, but usually do so in a controlled fashion. For example, a security changes state only when someone edits it. A transaction can be edited when it is new, but once it is executed it cannot be changed, except for being canceled. Positions are changed when transactions are executed, but the calculated attributes in a position use a SUM operation to add up transactions (like a spreadsheet would add up a column of numbers) so there are no explicit side effects in those calculated attributes.
Each type can define a set of presentations for the web, also called “screens”. A screen for a security might be a simple editor, or (if the user is not allowed to edit securities) just a display. A screen can call up other screens, and a screen can display data from many objects, not just one. Screens are defined abstractly, making them easy for business experts. The type browser of bFAST™ allows screens to be edited as part of the definition of a type.
bFAST™ also has an editor for the main menu. The menu will open up screens, which then can open other screens. The menu and the screens together define the user interface for an application built with MetaBooks™.
Behaviors
Most types in MetaBooks™ have a specialized “behavior”. One of the most common is “Search”. The objects in MetaBooks™ are like a database, and every database needs a way to search it. In MetaBooks™, a search is an object, and objects that are searches are given the behavior “Search”. When an object is given the behavior “Search”, the type browser will let you define the various parts of a search, such as the types of the objects being searched and the criteria for the search. The result of a search is a table. The layout of the table can be defined by a screen of the search. The attributes of the search are the parameters of the search. A search can also have a screen that gives a UI for entering the parameters and causing the search to execute.
Another common behavior in MetaBooks™ is “import”. A data import is an object that converts data that is outside the system into objects in MetaBooks™. For example, there are imports that read activity files from a broker and turn them into trades, which are transactions. But imports can also be used to read in prices from a pricing service, or to read in a position report from a broker so that it can be compared against the calculated positions. Like searches, imports use their attributes as parameters, and can have screens for setting up the import.
A transaction in MetaBooks™ is an object. It can be executed, which means that it is "posted" to one or more books. A book will contain "accounts" (or "positions") that are similar to a spreadsheet page with a row of transactions. The process of posting a transaction to a book results in a row being added to one or more of the accounts in the book. MetaBooks™ labels each transaction with the person who created it. Once a transaction has been posted, it is not possible to delete it. Instead, it can be "cancel/corrected", which will cause new entries to be posted. Transactions have behavior “postingTransaction”, while books, accounts and positions either have behavior “Forwarder” or behavior “Repository”, depending on whether they contain other objects and forward the transactions to them (i.e. books and accounts that contain subaccounts or positions) or whether they make a list of all the transactions that get posted to them (i.e. the subaccounts, or accounts without subaccounts, or positions). The type browser allows editing of the posting rules, so the entire process can be controlled from within bFAST™ by business experts in their own language.
Most transactions are created by importers. Transactions can also be created manually, through screens. A third way to create a transaction is using a search. If a search is defined to make a table whose columns match the attributes of a type then the result of that search can be converted into a set of objects of that type. If the type is a transaction then the objects created by the search are executed. This feature is used for the Mark To Market transaction. Once a market price is established for a security, a position in that security can be “marked to market” by calculating the change in unrealized gain and then posting a Mark To Market transaction that records the price, the fx rate, the unrealized gain, and so on. The process for marking an entire portfolio would be a search that takes the portfolio and the date, searches for positions in that portfolio that have not been marked, and produces a list of Mark To Market transactions that mark them.
Searches are versatile. They are often used to create tables to be displayed on the web. They can also be used to produce reports, using JasperReports. They can be used to produce objects, especially transactions. And they can be used for exporting data. At the moment, they can only be exported in spreadsheet formats, but we plan to export to RDBMS soon.
Other browsers
bFAST™ has several browsers, but so far the only one that has been mentioned is the type browser. It is the most complex browser, but it is only one of several. The instance browser lets the business experts inspect and change objects in the database in an environment outside of production. The execution browser lets the business expert create and execute the kind of objects that have an “execute” behavior, i.e. transactions, searches, and imports. The menu browser edits the definition of the main menu. The initialization browser edits the script that initializes the database. Finally, the attribute type browser edits the set of attribute types. Attribute types are typically low-level types like Boolean, BigDecimal, or Date. Most of them are subtypes of String and are essentially lists of possible values. For example, TradeType is “Buy”, “Sell”, “Short” or “Cover”. Business experts have control.
Applications in MetaBooks™
The biggest application developed in MetaBooks™ is a portfolio / shadow back office system for a small to medium sized hedge fund. It is a middle-office/back-office system, not an OMS/EMS trading system at this point. It assumes that the alternative asset manager is using a front-end system provided by the broker or another vendor.
The name of this model is “Prime.” Like all MetaBooks™ models, it is centered around a set of books and accounts. Prime has 12 books; the four we will discuss are the Inventory Book that keeps track of positions, the Price Book that keeps track of prices of securities, the Accounting Book that provides a double-entry, multicurrency view of the hedge fund, and the Investor Book that keeps track of the capital investment of each investor in the fund. There is a type for each book. There are also a variety of “entities”, classes whose objects rarely change but which are often arguments of transactions, such as Security, Currency, Broker, BrokerAccount, and Trader. Most of the complexity of the system comes from the large number of transactions, such as EquityTrade, SecurityPricingTransaction, and Liquidate. Each transaction has posting rules for some set of books. EquityTrade and Liquidate both have posting rules for the InventoryBook and for the AccountingBook, while SecurityPricingTransaction just has posting rules for the PricingBook. New books can be created as needed by business experts.
Data comes from a variety of sources. Usually a set of securities, currencies, brokers, broker accounts, etc are created when the system is initialized. There are importers that read files in an “init” directory and create these objects. More can be created, of course, either by importers or by UI screens. Importers that read files from brokers will create trades, such as EquityTrade, and these tend to be run every day. Importers that read files from pricing services create SecurityPricingTransactions. Liquidate transactions are not usually created by importers, though. A Liquidate transaction matches a purchase with a sale, and indicates that the sale “liquidates” the purchase. Sometimes they are created by processes that automatically match sales to purchases using algorithms such as FIFO or LIFO, but sometimes they are created by UIs that allow a clerk to select specifically which purchases to liquidate.
From this description, it might seem that an application like Prime is complex. And it is. A similar system like Prime written in Java or another programming language could easily be several million lines of code and take hundreds of programmers to support. Prime on the other hand is a “Low Code” offering with just a few thousand lines of code, so by any reasonable measure it is significantly less costly and easier to support. Only a handful of business experts and programmers could support thousands of clients. Each part of Prime is relatively easy to use and simple to understand, but there are many parts and someone trying to master all of what Prime has to offer will take some time to master.
Using bFAST™Modeling Tool
Although it is possible to start with an empty list of types and build a model from scratch, almost all users of bFAST™ will start with a model and modify it. New users will probably just add reports or searches to an existing system, or fix an import when the data source changes. They might change a UI screen or add something to the menu.
When someone wants to change a running system, they take a snapshot of what is running on the server (using MetaWeb) and work with the snapshot in bFAST™. They make changes and test them and then save the metadata. When they are confident of their changes, they can restart the web server in a controlled fashion and have it import the changes and migrate the database. Many changes, such as adding a new report or changing the UI, have no effect on the database. The point is that people do not actually modify the metadata of the running web server in real-time, but they modify the metadata in a sandbox and then apply the changes to the server once everything is thoroughly tested.
Someone using bFAST™ is working with a powerful universe of objects. They not only change the metadata, but they can create objects and (if they are transactions or searches) execute them. They can test new posting rules by executing transactions, and they can test an import by executing it. Although it is possible to browse the objects and inspect each one, this gets tedious when there are many of them. Often it is easier to look at them with a search. Searches are currently the best way to test calculated attributes. So, it is common for someone using bFAST™ to create a search, use it for a while, and then delete it once everything is verified.
Since business experts are almost always working within an existing system, copy and paste is very important to them. Originally bFAST™ did not support copy and paste very well, but it has become easy and reliable. Of course, copy and paste leads to duplication, and often there is a way of using calculated attributes or inheritance to reduce duplication. In the future, bFAST™ will need to make it easy to eliminate unnecessary duplication.
How we designed MetaBooks™
MetaBooks™ was designed by a close collaboration between a programmer and a business expert. The goal has always been for all of the features of Prime to be controllable and understandable by the business expert. The initial design of MetaBooks™ quickly changed as the business expert used it. Many of the features were too complicated for the business expert and had to be redesigned.
For example, data import is an important feature that has been redesigned several times. The first few imports were written by a programmer, and the business expert would only edit them after they were written. But after a couple of revisions of the imports, the business expert was finally able to do a complete set of imports for several brokers on their own. There are still some file formats that seem to always take help from a programmer, so we are planning another revision to make them all easy for the business expert.
MetaBooks™ is implemented in Groovy, and we use Groovy scripting for rules and expressions. The business experts use a small subset of Groovy. We discourage using assignment and control structures. However, when a feature of Prime cannot be built using the existing features, sometimes a programmer will just write a Groovy script that does it, using features that are otherwise discouraged. Our label for these scripts is “looks too much like programming”. We then redesign MetaBooks™ so that they are not necessary which makes everything easier for business experts.
The future of MetaBooks™
Prime proves that MetaBooks™ is capable of supporting large projects. However, there are many ways it could be improved, and we are constantly improving it. Most of the ideas come out of our experience using it. For example, the transaction processing system was designed early on, and now that we have modeled a few hundred types of transactions with it, we can tell that the transaction processing system is a bit too general, because some of its features are never used, and because a more specific user interface would probably be easier for business experts to use. But other ideas come from the fact that the business experts, though they do not think of themselves as programmers, are engaged in software development much like using spreadsheets. bFAST™ needs to support them better. We have in general postponed addressing these issues until we had figured out a way of modeling that was suited for business experts. However, now that we have a good way of modeling the business, we can focus on issues more related to application development. Three of these are version control, debugging, and testing.
Version control: We now have several years experience using MetaBooks™ and bFAST™ with a small team. It is easy to imagine some of the problems that would occur if we had more people using it. The most important is version control and managing components. MetaBooks™ stores metadata as text. We include this metadata in our version control system. The version control tools are outside of bFAST™, so committing changes and updating to get changes from other people requires another tool, and sometimes interferes with bFAST™. More importantly, the text form is not very modular. It needs to have some sort of package system or module system so that more people can work together without interfering with each other. We have a preliminary design based on the module system of Newspeak.
Debugging: There are several places in bFAST™ that should be changed to make it easier to detect and fix mistakes. One of the advantages of MetaBooks™ is that changes are usually recorded with a transaction. So, when Prime produces a wrong result, it is possible to go back and figure where it came from. However, it would be easier to find the source of wrong numbers if more intermediate results were available. For example, when a business expert sees a wrong number in the results of a search, it should be possible to click on it and see where the number came from. Similarly, imports need to show intermediate steps. It should be possible to click on the result of an import and see each of the earlier steps in producing it. It should also be possible to click on an entry in a book and see not only the transaction that created it (that is currently possible) but the rules that transformed the transaction into the entry. All of these are examples of the same principle; making the connection between values more obvious, like a spreadsheet does, but they will require changes to different parts of the system.
Testing: There needs to be a way to write automated tests inside bFAST™. We use JUnit for automated testing and it is good for testing the components of MetaBooks™, such as dios (the core), bFAST™ and MetaWeb. We also have tests for the metadata in JUnit, but JUnit is designed for programmers, not business experts, and business experts need to write automated tests, too. bFAST™ now has a testing browser, a tool that lets business experts write and run tests against the system. However, it is still a little awkward to use and needs improvement. We are going to keep improving it until business experts wouldn’t dream of making changes to the system without writing tests for all their changes.
On the one hand, we want the business experts to say “It is wonderful how much I can do without programming”. On the other hand, we know they really are programming much like they do in spreadsheets. We are just trying to keep making what they do seem natural to them, and not require skills that are outside their comfort zone. So, we need to make testing and version control be natural and the easiest way for them to get their job done, not something new that they have to learn.
Metaficient™ is in the process of selecting early collaborators. Our focus is to help our partners achieve their goals by giving them access to the extra hands and automation they need to advance their business platform. To explore the possibility of being a partner, please send email to This email address is being protected from spambots. You need JavaScript enabled to view it. or call 914-241-1106.
