In this guide, we explore the 6 most common ways investment and asset managers integrate systems, data sources and service-providers. By the end of this article, you will know which types might be the best choices for you, depending on the type of data/systems for each integration you are considering.
Why did we create this post? As former investment managers ourselves, we would like to be helpful to you and share our knowledge accumulated after (possibly too much) research and testing of various approaches to asset management system integrations.
As former investment managers, we set out to build, deliver and service a generation 4 Investment Management Platform. For most clients, we cover the entire front-to-middle office workflow and hence the system has many integrations for every client. When clients want us to help with integrations and connectivity, we can offer 4 of the 6 types discussed here which we’ll point out through the article.
If you would like to directly explore some of the challenges with Limina’s solution, we’ve written an article exploring when we’re not a good fit for clients. You can also find out when we might be a great fit by watching our explanatory video.
Before we get started, below is a list of core systems that might serve as the single source of truth for integrations.
- Portfolio Management System(s),
- Order Management System(s),
- Investment Books of Records (IBOR)
- Investment Management Software:
Below is an illustration to which sources such a system might connect:
Integration type #1: Database or SQL based integrations
Let’s start with the most problematic integration. Direct database connectivity is (unfortunately) a very common asset management system integration approach. Some reasons why include:
- It usually offers fast time to market for initial use cases
- With older systems, it was the only way offered to get data in and out of the systems
- Some newer systems have also exposed their databases to clients
Below is an illustration of how it could work in an old system architecture:
There are many problems with this integration type, but the most significant is the dependency on the database tables. You don’t need to know what database tables are to understand the effect.
The database is the core of an investment management platform. As a system develops, changes to the data models in the database are required (for performance reasons or to add/change functionality). Once clients have been given access to the database, changing it can affect production in unknown ways.
The problem: if the vendor changes the database, they introduce problems that will be uncovered in the next upgrade. Your processes that are dependent on the database will cease to work and must be altered. This also means testing needs to be managed by you and full end-to-end tests conducted on every upgrade.
The result: upgrades become more expensive and riskier for you.
In this case the negatives (significant operational risks and costs) outweigh the pros (faster time to market) by so much we can only offer one piece of advice: always avoid direct database table integrations.
Limina note: as you might suspect, this is not one of the integration types Limina offers.
Integration type #2: Vendor managed connectivity
This is when the vendor takes full responsibility for a specfic asset management system integration, which means:
- Building the initial integration,
- Monitoring the integration in real-time and reacting to any issues, and
- Maintaining the integration over time as specifications change.
These are surprisingly expensive for vendors, especially since maintenance work is often grossly underestimated.
Managed integrations are great for time-sensitive and high-risk integrations that are shared by many clients. Connectivity to brokers and execution systems are some examples. The vendor is in a good position to monitor these on behalf of multiple clients and the result is usually a cost-efficient and stable integration.
Limina note: we do offer this type of integration as we believe it’s the best option for some system or service integrations (but not all).
The primary downside with managed integrations is dependency on the vendor. If you would like new integrations to be set up or replaced, you need to wait for the vendor. This could directly slow your change management or growth plans down. Worse yet, the vendor might not even want to support the integration you’re asking for, which puts you in a situation where you can’t take the business in the direction you would like.
Integration type #3: Custom files, usually CSV over SFTP
While this is not really a "type" of integration, it still deserves its own section because it's so common. The entire finance industry sometimes feels like it’s built on file transfers. Flat files (defined below) are often used in communication from a system such as Investment Management Systems, Investment Book of Records (IBOR), Portfolio Management Systems and Order Management Systems to and from custodians and fund administrators.“Flat files”, as they are also called, include:
- CSV files
- Excel files
- XML files
… and more
Files are usually delivered over Secure File Transfer Protocol (SFTP) from one machine to another. The transfer is made at a predetermined time of day (e.g. market prices after exchanges close) or as a trigger on events (e.g. after a trade has been confirmed).
The approach to use SFTP files works well for pushing data, i.e. when a sender wants to deliver data to a recipient, but the recipient hasn’t necessarily asked for or triggered the data (think push notifications on your phone). Catching the file, parsing, and transforming the data and inserting it into a system can be done either as a managed integration (#2) or via ETL tools (#5 and #6).
The primary benefit of such file transmission integrations is its simplicity – it’s platform independent and has no specific requirements or limitations around data format, so it’s a pattern that most systems will support. In addition, it also opens the possibility to augment the process by manual steps if needed since the files are both machine- and human-readable. These are the main factors for the widespread adoption for file-based integrations.
The main drawback of the file-based approach to asset management system integrations is also a direct consequence of its benefits. Since it involves no standards on data format, most such integrations are built ad-hoc in a custom format. While time to market can be quick, the consequence is that the overall level of standardisation in the industry remains low. It's for example generally not possible to re-use an integration for multiple service providers, since each custodian has its own format requirements for trade affirmation files, deposit/redemption files etc. Over time, the cost for maintenance and integration of additional service providers quickly adds up, which makes upgrades more complex due to the number of unique integrations involved.
In addition, there are some types of integrations where file-based approaches are not suitable:
- Low latency or real-time integrations such as for streaming market data and order workflows, due to performance requirements.
- Two-way integrations (posting data and then expecting response data back), such as analytics calculations for a position. This is because files are not suitable for two-way connectivity.
Limina note: we offer this type of integration both because it’s sometimes the only option available but we also believe it is an acceptable solution for many use cases, much more so than it’s bad reputation indicates.
Integration Type #4: APIs, e.g. REST and websockets
APIs are a good choice if data communication is real-time, for example real-time market prices. It’s also a good choice if data is requested by a system, for example when requesting static data on instruments (or when ordering a sandwich, you are “requesting” a sandwich).
APIs can technically be leveraged directly by your team if you have in-house developers. More common however, especially among mid-sized asset managers, is that the APIs are leveraged by the vendor for managed integrations (#2) or via ETL (#5 and #6).
Limina note: We believe in an API first approach to solve the most challenging integration problems (we offer APIs). At the same time, we don’t believe it’s a silver bullet to all integration problems, which is why we offer more approaches.
Integration type #5: Generic Extract, Transform, Load (ETL) tools
Below is an overview of what ETL means:- Extract: To get data out of a system, from a data source, service provider etc. This can be from databases (#1), files (#3) or through APIs (#4). For example, requesting a set of positions (or ingredients for a sandwich).
- Transform: This is logic to change the data in some way. It could for example be aggregation of market value over a set of positions (or making a sandwich out of ingredients).
- Load: The final stage is to insert the data into its destination, which could for example be saving the aggregated market value into a data warehouse (or deliver the sandwich to a hungry portfolio manager).
.png?width=836&height=523&name=ETL%20dark%20background%20(1).png)
ETL tools can work in “reverse”, i.e. used to get data from system(s) to external stakeholders (systems or service providers).
Conceptually (but maybe not in practice), an ETL platform provides great benefits in that it allows for high automation levels over a wide range of integration types (database, file-based etc). Off-the-shelf ETL products require an upfront investment (implementation work) but then provide value that scales quite linearly with the number of integrations and use cases that are automated.
However, there are in our experience three main drawbacks of off-the-shelf ETL platforms that for many investment managers result in the ROI equation not adding up:
- ETL tools are often quite technical and complex to use, and thus implementing require specific technical expertise
- Data sets to be integrated are often domain-specific and complex, and therefore requires the business users to spend time (that they don’t have) to help analyse, specify and test
- Aligning ETL tools with your change management is often a challenge. Whenever an involved system is upgraded, it's difficult to ensure any affected ETL integrations are a) not impacted, or b) upgraded correctly and timely
Limina note: we don’t offer this, but here are some of the top ETL tools.
Integration type #6: Configurable purpose-built import/export engine
This is a purpose-built application, like ETL, tailored for investment data. It does for example recognise what cash is and offers specific transformation settings useful for import of cash data (no more sandwich metaphors).
The benefits compared to a generic ETL tool is that it’s much easier and faster for business users to manage. It also comes bundled with the Investment Management System which means it’s not a separate solution you need to procure, integrate (!) and configure. The best native import/export engines will also include support for change management processes, for instance in the form of automatic regression testing capabilities of the configured import/export projects, to automatically capture any regression breaks when the underlying platform or external data format changes.
Being an embedded/native solution is also the main drawback since the integration engine is most likely limited to being used to get data into or out of the Investment Management Software.
Limina note: The reason we created a purpose-built ETL app was to give business users the ability to configure integrations without being dependent on vendors. If you would like to explore our import / export engine further, we’d be more than happy to show it to you.
What great integrations enable
By now you should (hopefully, otherwise let us know and we’ll try to explain better) have a better understanding of the various types of asset management system integrations and connections. Whether you are connecting an Investment Book of Record (IBOR), Investment, Order or Portfolio Management System to data sources, service providers or other systems, there is usually one type of integration that is best suited for that integration. In the end, there is no one type of integration that is always superior, the choice depends on requirements in each case.
To wrap up, strong integration support can enable the following for you:
- More efficient workflows, by reducing overhead when sending/receiving data
- Very robust processes, even spanning multiple systems and providers
- Increased confidence in data, by always knowing the status of each integration in production
Which is why it’s important to have the flexibility to choose the best type of connectivity – in all cases!
.svg)
