EFPF project envisions a connected and smart ecosystem of the future where multiple tools, services and platform can interact with each other through an open, interoperable and secure medium. In EFPF, we call this medium the Data Spine. The Data Spine is envisioned to support interoperability at the level of data, communication and security protocol levels.
The architecture of the EFPF Data Spine is shows in the adjecent figure.
The realisation of the promised Data Spine took a lot of effort, starting from the comparison of candidate technologies for the central component of the Data Soine i.e the Integration Flow Engine.
The search for the candidate technologies for the Integration Flow Engine resulted in the identification of two possible solutions:
1. WSO2 Carbon Infrastructure Stack: The WSO2 Platform includes an integrated set of components that can realise the environed Data Spine functionality. These components include the Enterprise Integrator (WSO2 El), API Manager, Governance Registry, identity Server and a Message Broker to support seamless data flow
2. Apache NiFi with other components: Apache NiFi is a Data-Flow management tool based on flow-based programming. NiFi is suitable to realise the Integration Flow Engine of Data Spine but it needed other components to realise the envisioned Data Spine functionality, such as the service registry, security infrastructure and message broker
The EFPF technical partners carried an extensive experimental evaluation of the above two technologies. The outcomes of these experimental evaluations are described below:
License: Both the WSO2 Platform and Apache NiFi come with Apache License v2.0. WSO2 Platform also comes with a commercial license for receiving support from WSO2 development/support teams and access to WSO2 products’ update service, etc.
Usability: Both the WSO2 EI and Apache NiFi provide a drag-and-drop style GUI to the developers to create the integration flows with minimal effort. WSO2 EI’s GUI is Eclipse-based whereas NiFi’s GUI is Web-based. For designing integration flows in WSO2 EI the developers needs to install the Eclipse-based EI Tooling development environment and configure the WSO2 EI server. On the other hand, the developers can directly use NiFi’s Web-based GUI through a Web browser. In this respect, the collaboration of work concerning a particular workflow among different developers would be difficult in the case of WSO2 Platform as each developer needs to use the EI Tooling desktop application installed in the local environment, whereas in the case of NiFi, such a collaboration would be easy as NiFi provides a Web-based GUI for creating workflows and a Multi-tenant authorization capability that enables different groups of users to command, control, and observe different parts of the dataflow, with different levels of authorization. Moreover, the learning curve for WSO2 EI was observed to be much steeper than NiFi.
Built-in Functionality: Both the WSO2 EI and Apache NiFi provide a decent development environment for creating the integration flows with a drag-and-drop style GUI.
Built-in Protocol Connector: Both the WSO2 EI and Apache NiFi provide connectors for standard communication protocols such as HTTP, MQTT, AMQP, etc. that are widely used in the industry.
Built-in Data Transformation Tools: WSO2 EI provides several components, which in the architectural solution of WSO2 EI are called mediators, for mapping one data model to another using translation rules. The WSO2 EI mediators that have been tested are: Data Mapper, Payload Factory, XSLT and Script. All the aforementioned mediators achieved the intended functionality but each one with some minor drawback that depended on the use case in which they were used. The major drawback with WSO2 EI was lack of flexibility that WSO2 EI has, in fact it seemed complicated to extend the functionality provided by the system. In comparison, NiFi provides less number of components, called processors, for the same purpose of data mapping. Apart from the steeper learning curve, the processors look more flexible and manageable. NiFi with its less graphical UI, seems to be aimed at a more technical user, giving more power and responsibility to the user.
Moreover, all the aforementioned NiFi components are able to achieve the data transformation tasks in a flexible way, rather than the higher number of WSO2 EI mediators that are mostly non applicable to the identified purposes because of a lack of flexibility.
Extensibility: NiFi is at its core built with extensibility in consideration. Points of extension include: Processors, Controller Services, Reporting Tasks, Prioritizers, and Customer User Interfaces. For example, it is possible to write a custom processor for NiFi in order to connect to an OPC-UA server (based on OPC-UA Java Stack) and read the data. WSO2 also has support for extension through custom mediators; however, WSO2 Platform’s documentation recommends to avoid using the custom mediators as they incur a high maintenance overhead and they might also introduce version migration complications when upgrading WSO2 EI to a new version.
Performance and Scalability: Heavy resource utilisation was noticed for WSO2 EI and WSO2 Message Broker whereas NiFi was observed to work seamlessly with the same amount of resource allocation. NiFi is also able to operate within a cluster.
Identity and Access Management: NiFi supports a pluggable OpenID Connect based authentication provider such as Keycloak. Alternatively, NiFi also supports user authentication via client certificates, via username/password with pluggable Login Identity Provider options for Lightweight Directory Access Protocol (LDAP) and Kerberos or via Apache Knox. WSO2 Platform offers its own solution for identity and access management called WSO2 Identity Server.
Component Integration Effort: It was experienced during experimental evaluation that the integration of WSO2 EI with WSO2 Governance Registry needs high integration effort, even though these components belong to the same software stack. NiFi provides connectors for integration with external components. For example, for integration with Kafka, NiFi has 20 built-in processors. During the experimental evaluations, the integration of NiFi with REST APIs of other components such as EFPF Security Server was done with minimal effort.
Maintainability and Documentation: During the experimentation, the WSO2 EI was evaluated with focus on Message Brokering. The WSO2 EI was installed on Ubuntu 16.04 TLS over Openstack. During the tests, a few times the software became slow, stopped working and crashed; since troubleshooting and maintaining is difficult, usually reinstallation was needed. As for Message Brokering for MQTT protocol, the tests were successful but connecting the WSO2 components to an external broker failed. For this problem, referring to the documentation wasn’t helpful because at the first glance, the documentation looked very detailed and completed, but at the moment of need, it was very difficult to find solutions. The documents are not up to date; the structure is complicated and different versions of WSO2 Platforms have overlapping documents. Since the WSO2 Platform has so many components and features the learning process needs to be easier and smoother.
Unlike the WSO2, NiFi’s GUI is very simple, drag-and-drop style and easy to manage and as for documents, they are simple and useful.
Other Factors: Some inconsistencies were observed with WSO2’s Management Console. For example, when simple workflows were created using the Management Console’s UI, they were not listed on the page that displays all workflows. Also, WSO2 EI internally makes use of SOAP (Simple Object Access Protocol) which incurs unnecessary overhead as most of the synchronous communication in EFPF ecosystem happens using REST over HTTP. No such issues were observed with NiFi.
Based on this experimental evaluation for quality assessment, Apache NiFi was selected to be the central Integration Flow Engine of the Data Spine. The details of the other components that make up the Data Spine will be elaborated in future blogs … keep visiting the Blog section of EFPF website.
Comentários