Hazelcast + Ignite
When deploying docuflow behind a load balancer with multiple instances running, distributed caching and locking mechanisms become essential for maintaining data consistency and preventing race conditions. Without a distributed solution, each docuflow instance would operate with its own independent cache and locking mechanism. This independence can lead to potential conflicts.
For example, if one docuflow instance modifies data in SAP via SAPLink, another instance might be completely unaware of the change. This lack of awareness could result in the second instance using stale data or causing processing errors due to conflicting updates.
Hazelcast and Ignite address this problem by providing a shared, distributed cache and locking service. This ensures that all docuflow instances behind the load balancer have a consistent view of data. Furthermore, operations, especially those interacting with external systems like SAP, are synchronized correctly across all instances. This synchronization guarantees reliable and predictable behavior in a high-availability, load-balanced environment. Document Metadata Caching (Major performance improvement):
In addition to distributed locking and general caching, docuflow selectively uses Hazelcast and Ignite to cache document metadata upon creation. This means that when a document is created, its metadata is stored in the distributed cache. Subsequent retrievals of that document and it's metadata can then be served significantly faster from the cache, rather than requiring a potentially slower fetch from the underlying content repository (e.g., FileNet, Documentum, Box). It's important to note that this metadata caching is not universally applied to all document operations; it's strategically used in specific areas of the application to maximize performance gains. Confirm with your docuflow architect for more information on which specific operations utilize this metadata caching.
Starting with docuflow version 6.8.0, both Hazelcast and Ignite are offered as options for distributed caching and locking. This dual offering provides significant flexibility. The two platforms have different strengths: Ignite boasts a broader range of in-memory computing features beyond just caching and locking, while Hazelcast is often considered simpler to deploy and manage.
This choice accommodates users who may already have existing Hazelcast or Ignite infrastructure or in-house expertise. It also allows for performance benchmarking in specific environments to determine which solution performs optimally. The different licensing models (Hazelcast has commercial options, whereas Ignite is fully open-source) also cater to varying budgetary and licensing preferences. Finally, offering two distinct technologies reduces reliance on a single vendor and provides a fallback option, enhancing the overall robustness of the docuflow deployment. Ultimately, users can select the best fit for their particular needs and avoid vendor lock-in. I. Introduction: Distributed Caching and Locking
The Enterprise Connector utilizes distributed caching and locking mechanisms to improve performance and ensure data consistency across multiple instances (nodes) of the connector, especially in high-availability or clustered environments. You can choose between two primary providers for these features:
- Hazelcast: A popular in-memory data grid that provides distributed caching, locking, and other features.
- Ignite: A more comprehensive in-memory computing platform that also offers distributed caching, locking, and advanced features like compute grids and SQL support.
This document explains how to enable and configure either Hazelcast or Ignite within the config.properties file. You should generally choose only one; do not enable both simultaneously.
II. Enabling and Disabling Hazelcast/Ignite
The most fundamental configuration is enabling or disabling these features:
- HazelCast.Enabled: Controls whether Hazelcast is used.
- YES: Enables Hazelcast.
- NO (or any other value, or if the property is missing): Disables Hazelcast.
- Ignite.Enabled: Controls whether Ignite is used.
- YES: Enables Ignite.
- NO (or any other value, or if the property is missing): Disables Ignite.
Example (Enable Hazelcast, Disable Ignite):
Example (Disable Hazelcast, Enable Ignite):
Example (Disable both - default behavior if neither is specified):
III. Hazelcast Configuration
If HazelCast.Enabled=YES, the following properties are used to configure Hazelcast:
- Basic Cluster Setup:
- HazelCast.GroupName: The name of the Hazelcast cluster. All connector instances that should work together must use the same group name. This is crucial for them to discover each other and form a cluster. Choose a descriptive name, but avoid spaces or special characters.
- Example: MyConnectorCluster
- HazelCast.GroupPassword: A password for the Hazelcast cluster. All connector instances must use the same password to join the cluster. This adds a basic layer of security, preventing unauthorized nodes from joining.
- Example: MySecurePassword
- HazelCast.networkIpAddress: The primary IP address or hostname that this connector instance will use for Hazelcast communication. This is how other nodes will find this one.
- If running on a single machine (for testing), you can often use 127.0.0.1 (localhost).
- In a production environment, use the actual IP address of the machine, preferably a static IP. Do not use 0.0.0.0 (which binds to all interfaces) unless you fully understand the implications.
- Example: 192.168.1.100
- HazelCast.networkIpAddress2: An optional second IP address or hostname. This can be used for redundancy or if the connector instance has multiple network interfaces. If not needed, leave it blank or remove the property.
- Example: 192.168.1.101
- 
- HazelCast.networkPort: The port Hazelcast will use for communication. The default is usually 5701. You generally don't need to change this unless there's a conflict with another application. All nodes in the cluster must use the same port.
- Example: 5701
- If you change it, make sure the new port is open in your firewall.
- HazelCast.isPublic: Indicates whether the Hazelcast instance is accessible from outside the local network (e.g., on the public internet). This has significant security implications.
- YES: The instance is publicly accessible. Extreme caution is required. Ensure strong security measures (password, potentially TLS/SSL) are in place.
- NO (or blank, or property missing): The instance is only accessible within the local network. This is the recommended setting for most deployments.
- Hazelcast and SAPLink Locking (SAPLINK.UseHazelcastLocks):
- This property, if set to YES, tells the connector to use Hazelcast's distributed locking mechanism for SAPLink operations. This is important for ensuring data consistency when multiple connector instances are interacting with SAP.
- YES: Enable Hazelcast-based locking for SAPLink.
- NO (or blank, or property missing): Use the default locking mechanism (which might not be distributed).
- Important: If you have multiple connector instances and are using SAPLink, you should enable either Hazelcast or Ignite locking.
Example Hazelcast Configuration:
IV. Ignite Configuration
If Ignite.Enabled=YES, the following properties configure Apache Ignite:
- Basic Cluster Setup:
- Ignite.internalIp: The primary IP address or hostname that this connector instance will use for Ignite communication. This is analogous to HazelCast.networkIpAddress.
- Example: 192.168.1.100
- Ignite.internalIp2: An optional second IP address or hostname for redundancy or multiple network interfaces. Similar to HazelCast.networkIpAddress2.
- Example: 192.168.1.101
- Ignite.clusterName: Specifies the name of the Ignite cluster. Similar to HazelCast.GroupName, all nodes in the same cluster must have the same name.
- Default: defaultClusterName (if the property is missing)
- Example: MyIgniteCluster
- Ignite SSL/TLS Configuration (Optional but Recommended): Ignite can be configured to use SSL/TLS for secure communication between nodes. This is highly recommended in production environments.
- Ignite.SSLEnabled: Enables or disables SSL/TLS for Ignite.
- true: Enable SSL/TLS. You must also configure the keystore and truststore.
- false (or any other value, or property missing): Disable SSL/TLS. Not recommended for production.
- Ignite.KeystorePath: The absolute path to the Java Keystore (JKS) file that contains the connector's SSL/TLS certificate and private key. This is required if Ignite.SSLEnabled=true.
- Example: /opt/ewave/connector/config/ignite_keystore.jks
- Ignite.TruststorePath: The absolute path to the Java Truststore (JKS) file that contains the trusted certificates of other nodes in the cluster, or the Certificate Authority (CA) that signed those certificates. This is required if Ignite.SSLEnabled=true.
- Example: /opt/ewave/connector/config/ignite_truststore.jks
- Ignite.ssl.keystore.password: The password for the keystore specified by Ignite.KeystorePath.
- Ignite.ssl.truststore.password: The password to the trustore specified by Ignite.TruststorePath
- Ignite and SAPLink Locking (SAPLINK.UseIgniteLocks):
- This property, if set to YES, tells the connector to use Ignite's distributed locking mechanism for SAPLink operations, similar to the Hazelcast equivalent.
- YES: Enable Ignite-based locking for SAPLink.
- NO (or blank, or property missing): Use the default locking mechanism.
Example Ignite Configuration (with SSL/TLS):
Example Ignite Configuration (without SSL/TLS - NOT RECOMMENDED for production):
V. Key Considerations and Best Practices
- Choose ONE: Use either Hazelcast or Ignite, not both.
- IP Addresses: Use static IP addresses for production environments to ensure consistent connectivity.
- Firewall: Ensure that the chosen ports (default 5701 for Hazelcast) are open in your firewall between all connector instances.
- Security:
- For Hazelcast, always set a strong HazelCast.GroupPassword.
- For Ignite, strongly consider enabling SSL/TLS (Ignite.SSLEnabled=true) and configuring the keystore and truststore.
- If HazelCast.isPublic=YES, be extremely careful about network security.
- Testing: Thoroughly test your configuration in a non-production environment before deploying to production. Verify that the connector instances can discover each other and form a cluster. You can often check the logs for messages related to cluster formation.
- Keystore/Truststore Management: If you use Ignite with SSL, carefully manage your keystore and truststore files. Ensure they are properly generated, securely stored, and have appropriate passwords.
- Absolute Paths: Use absolute, not relative paths for Ignite.KeystorePath and Ignite.TruststorePath.
- SAPLINK Locking: Enable distributed locking (either Hazelcast or Ignite) if you have multiple connector instances interacting with SAP via SAPLink.
This comprehensive guide should enable your administrator to configure Hazelcast or Ignite effectively within the config.properties file. Remember to consult the official documentation for Hazelcast and Apache Ignite for more advanced configuration options and troubleshooting.