Progress DataDirect Hybrid Data Pipeline - Yearly Subscription

rootcert.pem 7. Extract server certificates from the PKCS12 file. openssl pkcs12 -in servercert.pfx -clcerts -nodes -nokeys > servercert.pem 32 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Deployment scenarios 8. Concatenate the certificates and private key in a single PEM file. In this example, the Linux/UNIX cat command is used to concatenate root certificate, server certificate, and private key. cat rootcert.pem servercert.pem privatekey.pem > server.bundle.pem 9. Confirm that the PEM file has the private key and the required certificates as described in PEM file format on page 31. The resulting server.bundle.pem file should be specified during the installation of the Hybrid Data Pipeline server. Converting a Java jks keystore file to a PKCS12 file A Java jks keystore file must first be converted to a PKCS12 file. The PKCS12 file can then be converted to a PEM file. 1. Use the following Java keytool command to convert the jks file into a pfx file. keytool -importkeystore -srckeystore keystore.jks -srcstoretype JKS -deststoretype PKCS12 -destkeystore target.pfx 2. Enter the keystore password and keystore file alias when prompted. 3. Use the resulting target.pfx file to create a PEM file by following the instructions in Converting a PKCS12 (pfx) file to a PEM file on page 32. Converting PKCS7 (p7b) file certificates to PEM file certificates These instructions assume that the private key is already available as a PEM file. 1. Use the following OpenSSL command to convert PKCS7 file certificates to PEM file certificates. openssl pkcs7 -print_certs -in certificates.p7b -out certificates.pem 2. Concatenate the certificate and private key files. In this example, the Linux/UNIX cat command is used. cat certificates.pem privatekey.pem > server.bundle.pem 3. Confirm that the resulting PEM file has the private key and the required certificates as described in PEM file format on page 31. The resulting server.bundle.pem file should be specified during the installation of the Hybrid Data Pipeline server. Converting PKCS7 file certificates to PKCS12 file certificates and adding the private key to the PKCS12 file After the certificate and private key files have been converted to the PKCS12 format, the PKCS12 file can then be converted to a PEM file. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 33Chapter 1: Welcome to DataDirect Hybrid Data Pipeline 1. Use the following OpenSSL command to convert a PKCS7 file to a PKCS12 file. openssl pkcs7 -print_certs -in certificate.p7b -out certificate.cer 2. Use the following command to add the private key to the PKCS12 file. openssl pkcs12 -export -in certificate.cer -inkey privatekey.key -out target.pfx -certfile CACert.cer 3. Use the resulting target.pfx file to create a PEM file by following the instructions in Converting a PKCS12 (pfx) file to a PEM file on page 32. Converting DER certificates to PEM file certificates The DER extension is used for binary DER files. These files may also use the CER and CRT extensions. These instructions assume that the private key is already available as a PEM file. 1. Use the following OpenSSL command to convert DER certificates to PEM file certificates. openssl x509 -inform der -in certificates.cer -out certificates.pem 2. Concatenate the certificate and private key files. In this example, the Linux/UNIX cat command is used. cat certificates.pem privatekey.pem > server.bundle.pem 3. Confirm that the PEM file has the private key and the required certificates as described in PEM file format on page 31. The resulting server.bundle.pem file should be specified during the installation of the Hybrid Data Pipeline server. Creating a PEM file from a private key and Base64 encoded certificates PEM files use Base64 encoding. Therefore, no conversion process is required. However, the Base64 encoded certificates and the private key must be concatenated in a single PEM file. These instructions assume that the private key is already available as a PEM file. 1. Concatenate the certificate and private key files. In this example, the Linux/UNIX cat command is used. cat Base64rootcert.pem Base64servercert.pem privatekey.pem > server.bundle.pem 2. Confirm that the PEM file has the private key and the required certificates as described in PEM file format on page 31 The resulting server.bundle.pem file should be specified during the installation of the Hybrid Data Pipeline server. Application and driver configuration for standalone deployment Client applications must be appropriately configured. In conjunction with ODBC and JDBC applications, ODBC and JDBC drivers will also need to be configured. OData applications will need their own modifications. For the most part, configuration of the ODBC and JDBC drivers is handled during the installation of the drivers. If the drivers are installed using the configuration files generated by the Hybrid Data Pipeline server installation, then they will use the DNS of the host machine. Nevertheless, you may wish to configure the drivers in other ways. 34 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Deployment scenarios OData applications must be modified to use the DNS of the host machine for HTTP or HTTPS requests. In addition, OData applications should be configured for SSL as appropriate. Firewall and port redirection using iptables for standalone deployment Hybrid Data Pipeline Web UI and API endpoints are exposed by default on port 8080 for HTTP connections or port 8443 for HTTPS connections. The iptables firewall utility can be used to route connections from the standard HTTP port 80 and HTTPS port 443 to these endpoints. In this scenario, ports 80 and 443 will be accessible to everyone, while ports 8080 and 8443 are only accessible to processing running on the server. The instructions in the following topics can be applied to RedHat 7, Oracle 7 and Centos 7 distributions of Linux. Please see the documentation for your Linux distribution for more information about configuring the firewall. Note: If you are using a Suse 12 distribution of Linux, use the YaST2 Firewall setting GUI to configure your firewall. In Suse 12 you can find the firewall setting under Applications > System Tools > YaST > Adiministrator Settings/Security and Users/Firewall. Disabling firewalld If you are using a later version of Linux, it may have come configured with the newer firewalld software. Consult the documentation for firewalld to determine how to configure it in a similar way, and how to disable firewalld and use iptables. To disable firewalld, use the following commands in a console window. systemctl disable firewalld systemctl stop firewalld Installing iptables Installing iptable requires root privileges. 1. Log in with an admin account. 2. Run sudo -s 3. Use yum to install the iptables services: a) yum install iptables b) yum install iptables-ipv6 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 35Chapter 1: Welcome to DataDirect Hybrid Data Pipeline Creating the iptables configuration file Create the file /etc/sysconfig/iptables containing the content displayed here (your configuration may be slightly different). This will require root privileges. # Generated by iptables-save v1.4.21 on Thu Jun 23 09:05:43 2016 *nat :PREROUTING ACCEPT [1100:133346] :INPUT ACCEPT [1:48] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080 -A PREROUTING -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 8443 -A PREROUTING -p tcp --dport 8080 -j MARK --set-mark 1 -A PREROUTING -p tcp --dport 8443 -j MARK --set-mark 2 COMMIT # Completed on Thu Jun 23 09:05:43 2016 # Generated by iptables-save v1.4.21 on Thu Jun 23 09:05:43 2016 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [378:34583] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -m mark --mark 1 -j DROP -A INPUT -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT -A INPUT -m mark --mark 2 -j DROP -A INPUT -p tcp -m state --state NEW -m tcp --dport 8443 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT # Completed on Thu Jun 23 09:05:43 2016 Starting the iptables service Start the iptables service using the service command. service iptables start Load balancer deployment Hybrid Data Pipeline configuration depends in part on whether you are deploying the service on a standalone node or deploying the service on one or more nodes behind a load balancer. A load balancer deployment offers high availability and scalability, and is therefore the best option for production environments. In a load balancer deployment, the service is installed on one or more nodes behind a load balancer. Requests are handled by the load balancer which distributes requests across nodes. Hybrid Data Pipeline is largely configured during the installation process.When installing the service on multiple nodes behind a load balancer, the initial installation of the Hybrid Data Pipeline server is used as a template for installations on additional nodes.The following configuration details should be addressed before installation to ensure a successful load balancer deployment. • Login credentials for load balancer deployment on page 37 Passwords for the default administrator and user accounts must be specified during installation of the Hybrid Data Pipeline server. When initially logging in to the Web UI or using the API, you must authenticate as one of these users. 36 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Deployment scenarios • Load balancer configuration on page 38 Hybrid Data Pipeline can be deployed on one or more nodes behind a load balancer to provide high availability and scalability. Hybrid Data Pipeline supports two types of load balancers. • Network load balancers that support the TCP tunneling protocol (such as HAProxy) • Cloud load balancers that support the WebSocket protocol (such as the AWS application load balancer and the Azure application gateway) • System database for load balancer deployment on page 44 A system database is required for storing user and configuration information. For load balancer deployments, an external database is required to serve as the system database. As a best practice, the external system database should be replicated, or mirrored, to promote the continuous availability of the service. • Shared files and the key location for load balancer deployment on page 48 The specification of a key location is required during installation. The installation program writes shared files used in the operation of the data access service to this directory. As a matter of best practices, the key location should be secured on a machine separate from the machines hosting the Hybrid Data Pipeline service or the machine hosting the system database. • Access ports for load balancer deployment on page 49 The access ports used for Hybrid Data Pipeline should be enabled for incoming traffic and unallocated for other purposes. • SSL certificates for load balancer deployment on page 49 SSL/TLS encrypted communications between client applications and the load balancer are supported. In addition, all communications between the On-Premises Connector and the load balancer are SSL/TLS encrypted. SSL connections between the load balancer and the Hybrid Data Pipeline nodes are currently not supported. • Client application configuration for load balancer deployment on page 50 Applications and drivers must be properly configured to ensure a successful deployment of the service. • Browser configuration for load balancer deployment on page 51 For load balancer deployments, the browser you use to connect to the Web UI must have cookies enabled. Login credentials for load balancer deployment You must specify passwords for the default d2cadmin and d2cuser accounts during installation of the Hybrid Data Pipeline server. The default password policy is not enforced during installation of the server. However, best practices recommend that you follow the default password policy when specifying these account passwords. When initially logging in to the Web UI or using Hybrid Data Pipeline APIs, you must authenticate as one of these users. Hybrid Data Pipeline default password policy After installation, Hybrid Data Pipeline enforces the following password policy by default. • The password must contain at least 8 characters. • The password must not contain more than 12 characters. A password with a length of 12 characters is acceptable. • The password must not contain the username. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 37Chapter 1: Welcome to DataDirect Hybrid Data Pipeline • Characters from at least three of the following four groups must be used in the password: • Uppercase letters A-Z • Lowercase letters a-z • Numbers 0-9 • Non-white space special characters Load balancer configuration The Hybrid Data Pipeline product package does not include a load balancer. However, Hybrid Data Pipeline can be deployed on one or more nodes behind a load balancer to provide high availability and scalability. Hybrid Data Pipeline supports two types of load balancers: network load balancers that support the TCP tunneling protocol and cloud load balancers that support the WebSocket protocol. In turn, the load balancer must be configured to support the Hybrid Data Pipeline environment according to the following criteria. • The load balancer must be configured to accept HTTPS connections on port 443 and unencrypted HTTP connections on port 80. • The load balancer must be configured for SSL termination to support encrypted communications between clients and the load balancer. The configuration of the load balancer depends in part on the type of SSL certificate supplied. See SSL certificates for load balancer deployment on page 49 for details. • The load balancer must support session affinity. The load balancer must either be configured to supply its own cookies or to pass the cookies generated by the Hybrid Data Pipeline service back to the client. The Hybrid Data Pipeline service provides a cookie named C2S-SESSION that can be used by the load balancer. For ODBC and JDBC applications, the ODBC and JDBC drivers automatically use cookies for session affinity. OData applications should be configured to echo cookies for optimal performance. • The load balancer must pass the hostname in the Host header when a request is made to an individual Hybrid Data Pipeline node. For example, if the hostname used to access the cluster is hdp.mycorp.com and the individual nodes behind the load balancer have the hostnames hdpsvr1.mycorp.com, hdpsvr2.mycorp.com, hdpsvr3.mycorp.com, then the Host header in the request forwarded to the Hybrid Data Pipeline node must be the load balancer hostname hdp.mycorp.com. • The load balancer must supply the X-Forwarded-Proto header to indicate to the Hybrid Data Pipeline node whether the request was received by the load balancer as an HTTP or HTTPS request. • The load balancer must supply the X-Forwarded-For header for IP address filtering. The X-Forwarded-For header is also required if the client IP address is needed for Hybrid Data Pipeline access logs. If the X-Forwarded-For header is not supplied, the IP address in the access logs will always be the load balancer''s IP address. • The load balancer may be configured to run HTTP health checks against nodes with the Health Check API. • Additional configuration is required for the following scenarios. • If you are using the On-Premises Connector with a network load balancer such as HAProxy, see Configuring a network load balancer with the On-Premises Connector on page 39 for additional configuration requirements. • If you are using the On-Premises Connector with a cloud load balancer such as the AWS Application Load Balancer or the Azure Application Gateway, see Configuring a cloud load balancer with the On-Premises Connector on page 42 for additional configuration details. 38 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Deployment scenarios Configuring a network load balancer with the On-Premises Connector When running Hybrid Data Pipeline behind a network load balancer with an On-Premises Connector, the load balancer must be configured to route requests for on-premises data sources to the correct server nodes. There are two general steps involved in configuring your load balancer to support on-premises data access. First, a custom Access Control List must be created to direct requests for the On-Premises Connector to cluster nodes. Second, a backend notification pool that specifies the on-premises port for each cluster node must be created.The following instructions explain how an HAProxy load balancer can be configured to support Hybrid Data Pipeline access to backend data sources using the On-Premises Connector. These instructions may be adapted for other load balancers, such as NGINX and F5. The Hybrid Data Pipeline installation program automatically generates an HAProxy configuration file for each installation of the server. These HAProxy configuration files are written to the HAProxy subdirectory in the key location directory specified during installation. These files must be merged to create a single HAProxy configuration file for a load balancer deployment of Hybrid Data Pipeline. Take the following steps to create an HAProxy configuration file for a load balancer deployment using the On-Premises Connector. 1. Create an Access Control List (ACL) to direct requests for the On-Premises Connector to each Hybrid Data Pipeline server. Note: Options 1 and 2 below may be used in combination. • Option 1. Use a custom header to direct requests. Each entry should be prefaced with acl. In this example, the custom header X-DataDirect-OPC-Host is used to direct requests to the server service2.myserver.com through the default On-Premises Port 40501. acl is_opa_hdr_service2_myserver_com_40501 hdr(X-DataDirect-OPC-Host) -i opa_service2_myserver_com_40501 use_backend opa_service2_myserver_com_40501 if is_opa_hdr_service2_myserver_com_40501 • Option 2. Use URL routing to direct requests. Each entry should be prefaced with acl. In this example, URL routing is used to direct requests to the server service2.myserver.com through the default On-Premises Port 40501. acl is_opa_url_service2_myserver_com_40501 path_end -i /connect/opa_service2_myserver_com_40501 use_backend opa_service2_myserver_com_40501 if is_opa_url_service2_myserver_com_40501 2. Add each Hybrid Data Pipeline server to the backend notification pool section using the server keyword. In the following example, the server server2.myserver.com has been added to the backend hdp_notification_pool section, and health checks have been enabled at the root with the option httpchk property. backend hdp_notification_pool mode http option http-tunnel balance roundrobin option httpchk HEAD / http-check expect status 200 #HDP Notification Server Definitions server server1.myserver.com 11.22.111.105:11280 check server server2.myserver.com 11.22.111.106:11280 check Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 39Chapter 1: Welcome to DataDirect Hybrid Data Pipeline 3. Create a backend pool that specifies the On-Premises Port for each Hybrid Data Pipeline server that supports the On-Premises Connector by adding a backend section to the configuration file. For example, the following backend section is for a node on the service2.myserver.com server using the default On-Premises Port 40501. Health checks have been enabled at the root with the option httpchk property. backend opa_service2_myserver_com_40501 mode http option http-tunnel option httpchk HEAD / http-check expect status 200 server service2.myserver.com 11.22.111.106:40501 check 4. Add each Hybrid Data Pipeline server to the default backend pool using the server keyword. In the following example, server2.myserver.com has been added to the backend hdp_default_backend pool, and health checks have been enabled by specifying the /api/healthcheck endpoint with the option httpchk property. backend hdp_default_backend mode http balance roundrobin option httpchk HEAD /api/healthcheck http-check expect status 200 cookie HDP_SESSION insert nocache #HDP Server Definitions server service1.myserver.com 11.22.11.105:8080 check cookie service1.myserver.com server service2.myserver.com 11.22.111.106:8080 check cookie service2.myserver.com Example The following example demonstrates an HAProxy configuration file for using the load balancer with two server nodes that have the On-Premises connector enabled, server1.myserver.com and server2.myserver.com. To create this file, the required sections were copied from the generated configuration file for service2.myserver.com into the generated file for service1.myserver.com. Copied sections are indicated with comments. global log 127.0.0.1 local0 chroot /var/lib/haproxy daemon defaults log global mode http option httplog option dontlognull timeout connect 5s timeout client 15m timeout server 15m ############################################################################## # Configuration for OPC with load balancer. ############################################################################## frontend lb_opc_nodes bind *:80 #Replace /common/hdpsmoke/shared/redist/ddcloud.pem with the location of the #loadbalancers SSL certificate bind *:443 ssl crt /common/hdpsmoke/shared/redist/ddcloud.pem #In production port 80 should be a permanent redirected to 443 by uncommenting the 40 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Deployment scenarios #following line #redirect scheme https code 301 if !{ ssl_fc } mode http default_backend hdp_default_backend #Define rules for HDP Notification Servers acl is_hdp_notification2 path_end -i /connect/X_DataDirect_Notification_Server use_backend hdp_notification_pool if is_hdp_notification2 acl is_hdp_notification hdr(X-DataDirect-OPC-Host) -i X_DataDirect_Notification_Server use_backend hdp_notification_pool if is_hdp_notification #Rules for on-premises connection to service.myserver.com acl is_url_opa_service1_myserver_com_40501 path_end -i /connect/opa_service1_myserver_com_40501 use_backend opa_service1_myserver_com_40501 if is_url_opa_service1_myserver_com_40501 acl is_hdr_opa_service1_myserver_com_40501 hdr(X-DataDirect-OPC-Host) -i opa_service1_myserver_com_40501 use_backend opa_service1_myserver_com_40501 if is_hdr_opa_service1_myserver_com_40501 #Rules for on-premises connection to service2.myserver.com. These rules were copied #from the service2.myserer.com configuration file. acl is_url_opa_service2_myserver_com_40501 path_end -i /connect/opa_service2_myserver_com_40501 use_backend opa_service2_myserver_com_40501 if is_url_opa_service2_myserver_com_40501 acl is_hdr_opa_service2_myserver_com_40501 hdr(X-DataDirect-OPC-Host) -i opa_service2_myserver_com_40501 use_backend opa_service2_myserver_com_40501 if is_hdr_opa_service2_myserver_com_40501 backend hdp_notification_pool mode http option http-tunnel balance roundrobin option httpchk HEAD / http-check expect status 200 #HDP Notification Server Definitions server service1.myserver.com 11.22.111.105:11280 check #The following server argument was copied from the service2.myserver.com #configuration file server service2.myserver.com 11.22.111.106:11280 check backend opa_service1_myserver_com_40501 mode http option http-tunnel option httpchk HEAD / http-check expect status 200 server service1.myserver.com 11.22.111.105:40501 check #The following section was copied from the service2.myserver.com configuration file. backend opa_service2_myserver_com_40501 mode http option http-tunnel option httpchk HEAD / http-check expect status 200 server service2.myserver.com 11.22.111.106:40501 check backend hdp_default_backend mode http Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 41Chapter 1: Welcome to DataDirect Hybrid Data Pipeline balance roundrobin option httpchk HEAD /api/healthcheck http-check expect status 200 cookie HDP_SESSION insert nocache #HDP Server Definitions server service1.myserver.com 11.22.11.105:8080 check cookie service1.myserver.com #The following server argument was copied from the service2.myserver.com #configuration file server service2.myserver.com 11.22.111.106:8080 check cookie service2.myserver.com Configuring a cloud load balancer with the On-Premises Connector Hybrid Data Pipeline can be deployed on a web service, such as Amazon Web Services or Microsoft Azure, behind a cloud load balancer that supports the WebSocket protocol. When using an On-Premises Connector, the cloud load balancer must be configured to route requests for on-premises data sources to the correct server nodes. The instructions in this section describe how an Amazon Web Services load balancer must be configured to support Hybrid Data Pipeline. These instructions assume that you have completed the following deployment tasks. • Created a Virtual Private Cloud (VPC) to host a Hybrid Data Pipeline environment. • Created AWS compute instances in the VPC for each node that will be used to support the Hybrid Data Pipeline environment. • Provisioned an RDS database instance to operate as a system database for storing user and configuration information. • Created a file system on a node in the VPC to be used as the key location for shared files. • Installed the Hybrid Data Pipeline server on each node that will be hosting the service. • The key location specified during the initial installation must reside on a node in the VPC. • The system database specified during initial installation must be the RDS database instance for storing user and configuration information. • Created an AWS Application Load Balancer in the VPC to connect to Hybrid Data Pipeline. The following general steps must be taken to configure routing and listening rules in the AWS Application Load Balancer. The corresponding topics provide detailed instruction for each step. 1. Create a target group for default routing to the Hybrid Data Pipeline service API on page 42 2. Create a target group for notifications on page 43 3. Create a target group for on-premises access on page 43 4. Configure target routing on page 44 Once the Application Load Balancer has been configured with listener and target group rules, you can install On-Premises Connectors. Create a target group for default routing to the Hybrid Data Pipeline service API Take the following steps to create a target group for default routing. 42 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Deployment scenarios 1. Use the AWS console to create a load balancer target group. 2. Specify target group details. Name - Protocol – HTTP Port 8080 Target type – Instance VPC 3. Set up health checks. Protocol: HTTP Port: 8080 Path: /api/healthcheck 4. Save the target group. 5. Register each Hybrid Data Pipeline instance as a target on port 8080. 6. Set the stickiness attribute for the target group to 5 minutes. Create a target group for notifications Take the following steps to create a target group for notifications. 1. Use the AWS console to create a load balancer target group. 2. Specify target group details. Target Group Name: Protocol HTTP Port 11280 Target type instance VPC 3. Set up health checks. Protocol: HTTP Path: / Port: Select traffic port 4. Save the target group. 5. Register each Hybrid Data Pipeline instance as a target on port 11280. 6. Disable stickiness via the stickiness attribute. Create a target group for on-premises access Take the following steps to create a target group for on-premises access. 1. Use the AWS console to create a load balancer target group. 2. Specify target group details. Target Group Name: Protocol HTTP Port 40501 Target type instance VPC Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 43Chapter 1: Welcome to DataDirect Hybrid Data Pipeline 3. Set up health checks. Protocol: HTTP Path: / Port: Select traffic port 4. Save the target group. 5. Register the first Hybrid Data Pipeline instance as a target on port 40501. 6. Disable stickiness via the stickiness attribute. 7. Repeat steps 1 through 6 for each Hybrid Data Pipeline instance. Configure target routing Take the following steps to configure target routing. 1. Create a rule to route to the notifications target group by setting Path is to /connect/X_DataDirect_Notification_Server. Note: For load balancers that support routing with HTTP headers, the header X-DataDirect-OPC-Host:X_DataDirect_Notification_Server should be used. 2. For each node running the Hybrid Data Pipeline service, create a rule to route to the corresponding on-premises access target by setting Path is to /connect/. Note: The format of the is opa__ where is the hostname specified during installation with dot characters replaced by underscores, and is the On-Premises Access port number. For example, the routing key for nc-d2c02.americas.test.com on port 40501 would be opa_nc-d2c73_americas_test_com_40501. 3. Create a default routing rule. The Forward to attribute should be set to the Hybrid Data Pipeline service API target group. Important: Setting the default rule for routing requests to the Hybrid Data Pipeline service API must be completed after creating the rules for routing to the On-Premises Access and Notifications servers. System database for load balancer deployment Hybrid Data Pipeline requires a system database for storing user and configuration information.When deploying the service behind a load balancer, you must use a supported external database. An external system database ensures that user and configuration information is consistent across multiple nodes behind the load balancer. These nodes use the system information on the external system database to access data and return successful queries. In addition, an external system database provides better security and more flexibility for backing up system information. As a best practice, the external system database should be replicated, or mirrored, to promote the continuous availability of the service. Configuring Hybrid Data Pipeline to use a system database occurs during installation. 44 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Deployment scenarios External system databases Hybrid Data Pipeline requires a system database for storing sensitive information used in the operation of the data access service. For a standalone node deployment, you can opt to use either the embedded internal database or a supported external database. For a load balancer deployment, you must use an external database. Depending on the external database you are using, certain requirements must be met. See the following sections for details. • Supported databases on page 45 • Oracle requirements • MySQL Community Edition requirements on page 46 • Microsoft SQL Server requirements on page 47 • PostgreSQL requirements on page 47 Supported databases Note: Hybrid Data Pipeline supports Amazon RDS instances that are compatible with these supported database versions. Database Version Microsoft Azure SQL Database Microsoft Azure SQL Database 11 Microsoft SQL Server Microsoft SQL Server 2016 Microsoft SQL Server 2014 MySQL Community Edition Support based on MySQL Connector/J 5.12 Oracle Database Oracle 12c R1, R2 (12.1, 12.2) Oracle 11g R2 (11.2) PostgreSQL PostgreSQL 11 2 Hybrid Data Pipeline does not provide a driver for MySQL Community Edition. MySQL Connector/J 5.1 must be used to support the use of MySQL Community Edition as an external system database. Therefore, you should refer to the MySQL Connector/J 5.1 documentation for information on supported versions of MySQL Community Edition. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 45Chapter 1: Welcome to DataDirect Hybrid Data Pipeline Oracle requirements If you plan to store system information in an external Oracle database, you must provide the following information. • Hostname (server name or IP address) • Port information for the database. The default is 1521. • SID or Service Name • Administrator and user account information • An administrator name and password. The administrator must have the following privileges: • CREATE SESSION • CREATE TABLE • CREATE ANY SYNONYM • CREATE SEQUENCE • CREATE TRIGGER • A user name and password for a standard account.The standard user must have the CREATE SESSION privileges. MySQL Community Edition requirements If you plan on to use a MySQL Community Edition database as an external system database, you must provide the following. • A MySQL Connector/J driver, version 5.1, and its location To download the driver, visit the MySQL developer website at https://dev.mysql.com/. • Hostname (server name or IP address) • Port information for the database. The default is 3306. • Database Name • Administrator and user account information: • An administrator user name and password. The administrator must have the following privileges: • ALTER • CREATE • DROP • DELETE • INDEX • INSERT • REFERENCES • SELECT • UPDATE • A user name and password for a standard account.The standard user must have the following privileges: • DELETE 46 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Deployment scenarios • INSERT • SELECT • UPDATE Microsoft SQL Server requirements If you plan to store system information in an external SQL Server database, you must take the following steps when setting up the SQL Server database. 1. Create a database schema to be used for storing Hybrid Data Pipeline system information. 2. Create an administrator who can access the newly created schema. The administrator must have the CREATE TABLE privileges. 3. Create a user who can access the newly created schema. The user must have the CREATE SESSION privileges. After the SQL Server database has been setup, you must provide the following information during installation: • Hostname (server name or IP address) • Port information for the database. The default is 1433. • Database Name • Schema Name • Administrator and user account information • An administrator name and password. The administrator must have the CREATE TABLE privileges. • A user name and password for a standard account.The user must have the CREATE SESSION privileges. PostgreSQL requirements If you plan to store system information on an external PostgreSQL database, you must take the following steps when setting up the PostgreSQL database. 1. Enable the citext PostgreSQL extension. 2. Create a database schema to be used for storing Hybrid Data Pipeline system information. 3. Create an administrator who can access the newly created schema.The administrator must have privileges to create tables. 4. Create a user who can access the newly created schema. The user must have privileges to select, insert, update, delete, and sequence tables. After the PostgreSQL database has been setup, you must provide the following information during installation: • Hostname (server name or IP address) • Port information for the database. The default is 5432. • Database Name • Administrator and user account information • An administrator name and password. The administrator must have privileges to create tables. • A user name and password for a standard account. The user must have privileges to select, insert, update, delete, and sequence tables. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 47Chapter 1: Welcome to DataDirect Hybrid Data Pipeline Shared files and the key location for load balancer deployment Hybrid Data Pipeline requires the specification of a key location during installation. The installation program writes shared files used in the operation of the data access service to this directory. For a load balancer deployment, the key location must be accessible to the node or nodes running the service. Shared files The following files are stored in the key location for a load balancer deployment. • .backup: A backup copy of the contents of the install directory from the previous install. This is used to restore the contents of the directory if there is an error during an upgrade. • key: Reference to the file containing the encryption key for the Hybrid Data Pipeline database. • key00: Encryption key for the system database. This key is used to encrypt sensitive information such as data source user IDs and passwords, security tokens, access tokens and other user or data source identifying information. If this is not present, or was over written during the installation, then you will not be able decrypt any of the encrypted information in the system database. • key-cred: Encryption key for credentials contained in Hybrid Data Pipeline configuration files. Examples of credentials in the config files include the user ID and password information for the system database. • db/*: Encrypted information about the system database. The contents of these files are encrypted using the key-cred key. Used by the installer when performing an upgrade or installing on an additional node. If these are not present, or do not have valid encoding, the installation or upgrade will fail. • dddrivers/*: A directory of internally supported drivers that have been updated after a product upgrade. • drivers/*: The directory used for integrating third party drivers with Hybrid Data Pipeline. • plugins/*: JAR files for external authentication plugins. • authKey: Authentication key for the On-Premises Connector. This key is used to encrypt the user ID and password information in the On-Premises Connector configuration file.The key in this file is encrypted using a key built into the On-Premises Connector.This encrypted key is included in the OnPremise.properties configuration file distributed with the On-Premises Connector. If this is overwritten or incorrect, the On-Premises Connector will not be able to authenticate with Hybrid Data Pipeline. • ddcloud.jks: Sun SSL keystore. This keystore contains the Hybrid Data Pipeline server SSL certificate if the SSL termination is done at the Hybrid Data Pipeline server. • ddcloud.bks: Bouncy Castle SSL keystore. This keystore contains the same SSL certificate as the ddcloud.jks keystore.This keystore is in the Bouncy Castle keystore format and is used when the server is configured to run in FIPS compliant mode. Should only be present with FIPS enabled. • ddcloudTrustStore.jks: Sun SSL truststore. This trustore contains the root CA certificate needed to validate the server SSL certificate. This truststore is distributed with the On-Premises Connector and with the ODBC and JDBC drivers, allowing these components to validate the Hybrid Data Pipeline server certificate. • ddcloudTrustStore.bks: Bouncy Castle SSL truststore. Should only be present with FIPS enabled. This truststore contains the root CA certificate needed to validate the server SSL certificate in the Bouncy Castle keystore format. The Bouncy Castle SSL library does not use the default Java cacerts file, so this truststore is populated with the contents of the default cacerts file and the root certificate needed to validate the Hybrid Data Pipeline server certificate. Should only be present with FIPS enabled. • key-opc: Contains the unencrypted encryption key. The authKey above contains the encrypted version of this key. This key is not shipped with the On-Premises Connector. • global.properties: Stores properties and other information shared between nodes in a cluster. 48 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Deployment scenarios • redist/*: Redistributable files. These files are used to install the On-Premises Connector and the ODBC and JDBC drivers. Access ports for load balancer deployment Multiple access ports on nodes hosting the Hybrid Data Pipeline server must be opened and unassigned to other functions. The following tables document the required ports and default port numbers. The installation program for the Hybrid Data Pipeline server confirms that default ports are available and allows new port values to be assigned when needed. Port values are passed during the installation of Hybrid Data Pipeline servers. Server Access Port A Server Access Port must be opened for the load balancer. As a matter of best practices, the load balancer should be configured for SSL/TLS termination. Name Default Description HTTP Port 8080 Port that exposes Hybrid Data Pipeline Server Internal Ports The Shutdown Port must be opened. However, as a matter of best practice, the Shutdown Port should not be available outside the firewall of the Hybrid Data Pipeline server. For a load balancer installation, the Internal API Port on any node must be open to all the other nodes in the cluster. The Internal API Port should not be available outside the firewall. Name Default Description Internal API Port 8190 Non-SSL port for the Internal API Shutdown Port 8005 Shutdown port On-Premises Access Ports The Message Queue Port must be opened. For a load balancer installation with the On-Premises Connector, the On-Premises Access Port and the TCP Notification Server Port must be opened for the load balancer. Name Default Description On-Premises Port 40501 Port for the On-Premises Connector TCP Port 11280 Port for the Notification Server Message Queue Port 8282 Port for the message queue SSL certificates for load balancer deployment The following SSL encrypted communications are supported for a load balancer deployment. • Communications between the browser and the Hybrid Data Pipeline Web UI when the load balancer is configured for SSL. • Communications between applications using the REST API, including the OData API, and the load balancer. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 49Chapter 1: Welcome to DataDirect Hybrid Data Pipeline • Communications between the JDBC or ODBC drivers and the load balancer. • Communications between the On-Premises Connector and the load balancer. Important: SSL connections between the load balancer and the Hybrid Data Pipeline nodes are currently not supported. The following guidelines should be used when implementing SSL in a Hybrid Data Pipeline environment. • The load balancer needs to be configured with the root certificate and any intermediate certificates necessary to establish the chain of trust to the root certificate. • The root certificate must be specified as the SSL certificate during installation of the Hybrid Data Pipeline server. When intermediate certificates are required for the trust chain, then the SSL certificate must be supplied in a PEM file format. When there are no intermediate certificates, then the SSL certificate can be supplied in either DER or PEM file format. • The SSL certificate specified during installation is used to generate the trust stores for the ODBC driver, JDBC driver, and On-Premises Connector.These files are written to the redist directory of the key location upon installation. Before installing the ODBC driver, the JDBC driver, or the On-Premises Connector, the trust store and properties files in the redist directory must be copied to the installer directory of the component you are installing. Client application configuration for load balancer deployment Client applications must be appropriately configured. In conjunction with ODBC and JDBC applications, ODBC and JDBC drivers will also need to be configured. OData applications will need their own modifications. For the most part, configuration of the ODBC and JDBC drivers is handled during the installation of the drivers. If the drivers are installed using the configuration files generated by the Hybrid Data Pipeline server installation, then they will use the hostname of the load balancer or machine hosting the server. However, you may wish to configure the drivers in other ways. OData applications must be modified to use the hostname of the load balancer for HTTP or HTTPS requests. Additionally, for optimal performance, OData applications should be configured to echo cookies for session affinity. OData applications must also be configured appropriately for SSL. See Node-to-node communication in OData Hybrid Data Pipeline load balancer environment on page 50 for details on communication between nodes when an OData client cannot be configured to echo cookies. Node-to-node communication in OData Hybrid Data Pipeline load balancer environment In an OData Hybrid Data Pipeline load balancer environment, the load balancer and OData clients should be configured to handle cookies to achieve session affinity and optimize OData query performance. The load balancer should supply its own cookies or pass the cookies generated by the Hybrid Data Pipeline service back to the OData client. In turn, the OData client should echo cookies to allow the load balancer to direct query requests to the node that initially received the query. However, it is not always possible to configure an OData client to echo cookies. In such cases, Hybrid Data Pipeline uses an internal mechanism called the distributed file persistence manager.When a query is executed that requires file persistence, execution results are stored temporarily on the node that initially received the query. The manager associates the query with the node and the execution results stored there. If a request from the same query is routed to a different node in the cluster, the manager obtains the persisted execution results from the original node. The query results are then returned to the client by the node that received the request. 50 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Deployment scenarios The distributed file persistence manager requires node-to-node communication using the HTTP protocol to achieve session affinity. The Internal API Port specified during Hybrid Data Pipeline server installation is the port used for this node-to-node communication. Data remains secure in the following respects. First, the Internal API Port (8190 default) is not exposed externally to the public facing network. Each node registers itself using this port, and communications are restricted. Second, a UUID is generated during the node registration process. This UUID is passed in as an HTTP header to confirm the validity of node-to-node communications. Third, the service stores persisted files on only a temporary basis. Browser configuration for load balancer deployment For load balancer deployments of Hybrid Data Pipeline, the browser you use to connect to the Web UI must have cookies enabled. Exposing on-premises data sources to cloud-based applications This scenario describes a deployment where on-premises data sources are exposed for secure access by cloud-based applications. For this deployment, a Hybrid Data Pipeline server is installed in the cloud, and the On-Premises Connector is used to perform secure connections through the firewall to the backend data store. The cloud-based application is located in a separate cloud but connects with Hybrid Data Pipeline through an API such as OData, ODBC, or JDBC. This deployment could be suitable for an independent software vendor who wants to embed Hybrid Data Pipeline services in the cloud to give the cloud application users access to their data that resides in the data center or other on-premises systems. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 51Chapter 1: Welcome to DataDirect Hybrid Data Pipeline Connecting an application in the cloud to on-premises data sources This scenario describes a deployment where the Hybrid Data Pipeline server is installed behind a firewall with on-premises data sources while a number of applications reside in the cloud. With the Hybrid Data Pipeline server behind a firewall, a cloud-based service does not need to be maintained, and SSL can be used to secure your data. This deployment scenario could be suitable when using cloud-based OData applications, for example, creating a real-time connectivity between Salesforce and an on-premises database. External JRE support and integration Hybrid Data Pipeline uses an embedded JRE at runtime. However, you can integrate an external JRE with a standing deployment of Hybrid Data Pipeline. The following JREs are currently supported. • Oracle Java 8 JRE • OpenJDK 8 JRE Hybrid Data Pipeline must be installed on at least one server before you proceed with integrating an external JRE. Files associated with the embedded JRE can then be used to modify the external JRE you wish to use with the Hybrid Data Pipeline server or the On-Premises Connector. Note: Using an external JRE with the server is exclusive from using an external JRE with the On-Premises Connector. That is, the server can run on an external JRE while the On-Premises Connector runs on the embedded JRE, and vice versa. The following work flow outlines the procedure for integrating an external JRE. See the corresponding topics for details. 1. Modify the external JRE. • Option 1. Non-FIPS environment. 52 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Deployment scenarios • Option 2. FIPS environment. Note: FIPS is not supported for the On-Premises Connector with either embedded or external JREs. 2. If integrating the external JRE with the server, configure the server to use the JRE. 3. If integrating the external JRE with the On-Premises Connector, configure the connector to use the JRE. See also Importing data store SSL certificates on page 55 Modify the external JRE for a non-FIPS environment Take the following steps to modify an external JRE for a non-FIPS environment. Note: • is the installation directory of the Hybrid Data Pipeline server. • is the home directory of the external JRE. 1. Enable the Unlimited Strength Jurisdiction Policy according to the JRE vendor documentation. Depending on the vendor and version, the Unlimited Strength Jurisdiction Policy may be enabled by default. Note: Enabling the Unlimited Strength Jurisdiction Policy is the only modification required for using an external JRE with the On-Premises Connector. Therefore, the remaining steps can be ignored if the JRE is to be used only with the On-Premises Connector. 2. Copy the /ddcloud/utils/jre/lib/ext/bc-fips-1.0.0.jar file to the /lib/ext directory. 3. Merge the contents of the embedded JRE /ddcloud/utils/jre/lib/security/java.policy.sun file into the external JRE /lib/security/java.policy file. Note: • Any previously made customizations to the /lib/security/java.policy should be preserved. • Any permissions for data sources in the embedded JRE java.policy.sun file should be carried over to the external JRE java.policy file. 4. Merge the contents of the embedded JRE /ddcloud/utils/jre/lib/security/java.security.sun file into the external JRE /lib/security/java.security file. Note: • Any previously made customizations to the /lib/security/java.security should be preserved. • Any properties enabled in the embedded JRE java.security.sun file should be carried over to the external JRE java.security file. What to do next: Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 53Chapter 1: Welcome to DataDirect Hybrid Data Pipeline • Configure the server to use the external JRE. • Configure the On-Premises Connector to use the external JRE. Modify the external JRE for a FIPS environment Take the following steps to modify an external JRE for a FIPS environment. Note: FIPS is not supported for the On-Premises Connector with either embedded or external JREs. Note: • is the installation directory of the Hybrid Data Pipeline server. • is the home directory of the external JRE. 1. Enable the Unlimited Strength Jurisdiction Policy according to the JRE vendor documentation. Depending on the vendor and version, the Unlimited Strength Jurisdiction Policy may be enabled by default. 2. Copy the /ddcloud/utils/jre/lib/ext/bc-fips-1.0.0.jar file to the /lib/ext directory. 3. Merge the contents of the embedded JRE /ddcloud/utils/jre/lib/security/java.policy.bcfips file into the external JRE /lib/security/java.policy file. Note: • Any previously made customizations to the /lib/security/java.policy should be preserved. • Any permissions for data sources in the embedded JRE java.policy.bcfips file should be carried over to the external JRE java.policy file. 4. Merge the contents of the embedded JRE /ddcloud/utils/jre/lib/security/java.security.bcfips file into the external JRE /lib/security/java.security file. Note: • Any previously made customizations to the /lib/security/java.security should be preserved. • Any properties enabled in the embedded JRE java.security.bcfips file should be carried over to the external JRE java.security file. What to do next: Configure the server to use the external JRE. Configure the server to use the external JRE Once you have modified the external JRE, you can configure the server to use the external JRE by performing an upgrade installation of the server. During the upgrade, you will be prompted specify whether you are using the embedded JRE or an external JRE. If you select external JRE, you must specify the path to the external JRE. 54 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Deployment scenarios If you are using a response file to perform a silent upgrade, best practices recommend that you use the installation program to generate the response file. However, you may opt to edit the response file manually. If editing the response file manually, you must add Java configuration options to the response file. The options and values depend on whether the response file is based on the GUI installation template or the console mode installation template. GUI mode #Java Configuration# ------------------ SPECIFY_JAVA_HOME_NO=0 SPECIFY_JAVA_HOME_YES=1 HDP_JAVA_HOME_DIR=/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 SPECIFY_JAVA_HOME_NO indicates whether you are using an external JRE. If you are using an external JRE, specify 0. SPECIFY_JAVA_HOME_YES indicates whether you are using an external JRE. If you are using an external JRE, specify 1. HDP_JAVA_HOME_DIR specifies the path to the external JRE to be used at runtime. Console mode #Java Configuration# ------------------ SPECIFY_JAVA_HOME_YESNO=\"Yes\",\"\" HDP_JAVA_HOME_DIR_CONSOLE=\"/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-4.b14.el7.x86_64/jre\" Important: The escape characters, as shown in this example, are required for a response file based on the console mode template. SPECIFY_JAVA_HOME_YESNO indicates whether you are using an external JRE. If you are using an external JRE, specify Yes. HDP_JAVA_HOME_DIR_CONSOLE specifies the path to the external JRE to be used at runtime. What to do next: If integrating the external JRE with the On-Premises Connector, configure the connector to use the JRE. Configure the On-Premises Connector to use the external JRE To use an external JRE with an On-Premises Connector, the JRE''s Unlimited Strength Jurisdiction Policy must be enabled. No other modifications to the JRE are required to use it with an On-Premises Connector. Depending on the vendor and version of the JRE, the Unlimited Strength Jurisdiction Policy may be enabled by default. Once the Unlimited Strength Jurisdiction Policy has been enabled, you can configure the On-Premises Connector to use the external JRE when installing or upgrading the connector. During installation or upgrade, you will be prompted to specify whether you are using the embedded JRE or an external JRE. If you select external JRE, you must specify the path to the external JRE. Importing data store SSL certificates The Hybrid Data Pipeline server and On-Premises Connector use a JRE at runtime. When connecting to a data store secured with a self-signed certificate, you must import that self-signed certificate into the truststore of the JRE used at runtime.You may also need to import a certificate if you are using a certificate from a less-well-known certificate authority. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 55Chapter 1: Welcome to DataDirect Hybrid Data Pipeline Note: To view the certificates in a JRE truststore, navigate to the truststore directory and use the keytool utility to list supported certificates. For example: JAVA_HOME/bin/keytool -list -v -keystore cacerts See the following sections for instructions on importing SSL certificates into JRE truststores. • Importing certificates into the Hybrid Data Pipeline server JRE truststore • Importing certificates into the On-Premises Connector JRE truststore Importing certificates into the Hybrid Data Pipeline server JRE truststore If you are connecting from the Hybrid Data Pipeline server to the data store, you must update the truststore on each node running the server. The location of the truststore depends on whether you are using the default, embedded JRE or an external JRE. • Embedded JRE trustore location: hdp_install_dir/jre/lib/security/cacerts, where hdp_install_dir is the Hybrid Data Pipeline installation directory. • External JRE truststore location: jre_install_dir/jre/lib/security/cacerts, where jre_install_dir is the installation directory of the external JRE used by the server. Take the following steps to import an SSL certificate into the Hybrid Data Pipeline server JRE truststore: 1. From your console, navigate to the JRE trustore directory. For example: cd hdp_install_dir/jre/lib/security 2. Use the keytool to import the certificate file. In the following example, the certificate file is in the PEM file format. JAVA_HOME/bin/keytool -importcert -file full_path/selfsignedcert.pem -keystore cacerts -storetype JKS -storepass changeit Note: The default password for the JRE truststore embedded with the Hybrid Data Pipeline server is changeit. 3. Restart the Hybrid Data Pipeline service. a. Run the stop service script. ./install_dir/ddcloud/stop.sh Note: Shutting down Hybrid Data Pipeline can take a few minutes. Wait until you see the Shutdown complete message displayed on the console before taking any additional actions. b. Run the start service script. ./install_dir/ddcloud/start.sh 4. Follow steps 1-3 for each node running the Hybrid Data Pipeline service. 5. Test connectivity to the data store by setting up a Hybrid Data Pipeline data source and running a query against it. 56 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Deployment scenarios Importing certificates into the On-Premises Connector JRE truststore If you are connecting to an on-premise data store with the On-Premises Connector, you must update the truststore of any On-Premises Connector that is being used to connect to the data store. The location of the On-Premises Connector truststore depends on whether you are using the default, embedded JRE or an external JRE. • Embedded JRE trustore location:opc_install_dir\OPDAS\ConfigTool\ddcloudTrustStore.jks, where opc_install_dir is the On-Premises Connector installation directory. • External JRE truststore location: jre_install_dir\jre\lib\security\cacerts, where jre_install_dir is the installation directory of the external JRE used by the On-Premises Connector. Take the following steps to import an SSL certificate into the On-Premises Connector JRE truststore: 1. From your console, navigate to the JRE trustore directory. For example: cd opc_install_dir\OPDAS\ConfigTool\ddcloudTrustStore.jks 2. Use the keytool to import the certificate file. In the following example, the certificate file is in the PEM file format. JAVA_HOME\bin\keytool -importcert -file full_path/selfsignedcert.pem -keystore ddcloudTrustStore.jks -storetype JKS Note: There is no default password for the JRE embedded with the On-Premises Connector. If you are updating the embedded JRE, press Enter when prompted for the truststore password to continue. 3. Restart the On-Premises Connector. a. Select Stop Services from the Progress DataDirect Hybrid Data Pipeline On-Premises Connector program group. b. After the service has stopped, select Start Services from the Progress DataDirect Hybrid Data Pipeline On-Premises Connector program group. c. Select Configuration Tool from the Progress DataDirect Hybrid Data Pipeline On-Premises Connector program group. d. Select the Status tab and click Test to verify that the On-Premises Connector configuration is correct. 4. Follow steps 1-3 for each On-Premises Connector that may be used to connect to the data store using the new certificate. 5. Test connectivity to the data store by setting up a Hybrid Data Pipeline data source and running a query against it. See also External JRE support and integration on page 52 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 57Chapter 1: Welcome to DataDirect Hybrid Data Pipeline 58 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.12 Administering Hybrid Data Pipeline The administration of Hybrid Data Pipeline involves the management of three basic resources common to any Hybrid Data Pipeline environment: tenants, user accounts, and data sources. A Hybrid Data Pipeline system administrator can develop either a single-tenant or multitenant architecture. In a single-tenant architecture, the system administrator creates user accounts in the default system tenant. In a multitenant architecture, the system administrator first creates one or more child tenants in the system tenant. Then, the system administrator may create user accounts in either the system tenant or any one of the child tenants. The user accounts that reside in one tenant are isolated from those in other tenants. Once a tenant architecture has been established, a system administrator can provision user accounts in two general ways. First, an administrator can provision an account such that the user has direct access to the Hybrid Data Pipeline service. In this case, the administrator can provision the user to create, manage, and query data sources. The administrator can also promote or restrict access to Hybrid Data Pipeline features, such as the Web UI, the SQL Editor, and the Management API. Alternatively, an administrator can provision an account such that user access is limited to queries against a data source. For example, an administrator may want to provision user access such that a user can run an OData application against a backend data store. In this scenario, the administrator creates the data source and supplies the end user with connection information for the data source. The end user can query the data store with the connection information supplied, but he or she does not have access to the connection information stored in the data source definition or to Hybrid Data Pipeline itself. Beginning with information on initial log in, the topics in this section provide information on administering Hybrid Data Pipeline and configuring Hybrid Data Pipeline features. For details, see the following topics: • Initial login with default user accounts • Permissions and default roles • Logging in to the Web UI Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 59Chapter 2: Administering Hybrid Data Pipeline • Using Hybrid Data Pipeline APIs • Using the Web UI • Tenant architectures • User provisioning • Authentication • Password policy • Enabling and disabling the password policy • Configuring change password behavior • Implementing an account lockout policy • Transactions • Implementing IP address whitelists • Throttling • Configuring CORS behavior • FIPS (Federal Information Processing Standard) • Data source logging • SQL statement auditing • Using third party JDBC drivers with Hybrid Data Pipeline • Configuring Hybrid Data Pipeline to authorize client applications using OAuth 2.0 • Integrating Hybrid Data Pipeline with a Google OAuth 2.0 authorization flow to access Google Analytics • Troubleshooting Initial login with default user accounts You must specify passwords for the default d2cadmin and d2cuser accounts during installation of the Hybrid Data Pipeline server. Best practices recommend that you follow the Hybrid Data Pipeline default password policy when specifying these account passwords. When initially logging in to the Web UI or using Hybrid Data Pipeline APIs, you must authenticate as one of these users. The d2cadmin account has the default System Administrator role.The System Administrator role has all Hybrid Data Pipeline permissions.The d2cuser account has the default User role.The User role has a set of permissions associated with standard user tasks. (See Permissions and default roles on page 61 for details.) These default roles cannot be deleted. However, the users associated with them can be modified through the Web UI or Hybrid Data Pipeline APIs. As a matter of best practices, you should consider removing the default d2cadmin and d2cuser accounts. To remove the default d2cadmin account, you must create at least one other user with the Administrator permission. When you log in through the new account that has the Administrator permission, you can then remove the default d2cadmin account. Hybrid Data Pipeline requires that at least one user have the Administrator permission. However, as a matter of best practices, more than one user should have Administrator permission at any time. For more information on provisioning users, see Tenant architectures on page 87 and User provisioning on page 112. 60 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Permissions and default roles See also Logging in to the Web UI on page 63 Using Hybrid Data Pipeline APIs on page 64 Permissions and default roles on page 61 Tenant architectures on page 87 User provisioning on page 112 Permissions and default roles Hybrid Data Pipeline user accounts are required to have at least one role. A user account with a given role inherits all the permissions associated with that role. Roles can be assigned and managed either through the Users API or the Web UI. However, the users API must be used to associate a permission directly with a user account. The permissions for a user account are the sum of the permissions granted to the role(s) associated with the account and the permissions granted explicitly on the account. Hybrid Data Pipeline provides three default roles in the system tenant: System Administrator, Tenant Administrator, and User. As detailed in the table below, the System Administrator role has all permissions, the Tenant Administrator role has tenant and user permissions, and the User role has only user permissions.These roles cannot be deleted, and only the users associated with them can be modified. When building out a Hybrid Data Pipeline environment, it can be useful for administrators to consider permissions in terms of the following categories. • User permissions support the ability of users to create and manage their own data sources directly through the Web UI, the Management API, or both. • Tenant permissions support the ability of administrators to provision and manage users on a tenant-by-tenant basis. The OnBehalfOf permission allows administrators to create and manage resources on behalf of users. This on-behalf-of functionality allows administrators to obscure or conceal the service from users. • Elevated permissions support the ability of administrators to use administrative features, such as throttling and logging. The operations associated with these permissions can affect all users of the system and may not be isolated on a tenant-by-tenant basis. Important: To administer user accounts and other resources that belong to a tenant, a tenant administrator must be given explicit administrative access to the given tenant. In the Web UI, administrative access to a tenant can be granted by editing a user account via the Manage Users view on page 67. With the API, administrative access can be granted either by updating the tenants administered for a user via the Users API or by updating the list of administrators for a tenant via the Tenant API. Note: A subset of permissions can be set on data sources. See Data source permissions on page 1350 for details. Permission System Tenant User Category ID Description admin admin CreateDataSource x x x user 1 May create new data sources ViewDataSource x x x user 2 May view the details of any data source they own Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 61Chapter 2: Administering Hybrid Data Pipeline Permission System Tenant User Category ID Description admin admin ModifyDataSource x x x user 3 May modify or update any data source they own DeleteDataSource x x x user 4 May delete any data source they own UseDataSourceWithJDBC x x x user 5 May connect to any data source they own with the JDBC driver UseDataSourceWithODBC x x x user 6 May connect to any data source they own with the ODBC driver UseDataSourceWithOData x x x user 7 May make OData requests to any data source they own WebUI x x x user 8 May use the Web UI with data sources they own. Operations on the data source through the Web UI will be limited based on the permissions they have been granted ChangePassword x x x user 9 May use the Web UI to change their password SQLEditorWebUI x x x user 10 May query the data sources they own with the SQL Editor in the Web UI MgmtAPI x x x user 11 May use the Management API CreateUsers x x tenant 13 May create users in administered tenants ViewUsers x x tenant 14 May get lists of users and their information in administered tenants ModifyUsers x x tenant 15 May modify user information in administered tenants DeleteUsers x x tenant 16 May delete users in administered tenants CreateRole x x tenant 17 May create roles in administered tenants ViewRole x x tenant 18 May get lists of roles and their information in administered tenants ModifyRole x x tenant 19 May modify role information in administered tenants DeleteRole x x tenant 20 May delete roles in administered tenants OnBehalfOf x x tenant 21 May use ?user= to manage user''s data sources in administered tenants 62 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Logging in to the Web UI Permission System Tenant User Category ID Description admin admin Configurations x elevated 22 May view and modify system configuration values CORSwhitelist x elevated 23 May view and modify the CORS whitelist Logging x elevated 24 May view and modify logging settings TenantAPI x elevated 25 May use the Tenant API to create, view, modify or delete tenants RegisterExternalAuthService x elevated 26 May create, view, modify, or delete authentication services in administered tenants Limits x elevated 27 May see and modify limit values for administered tenants, users in administered tenants, and data sources of users in administered tenants OAuth x elevated 28 May specify and update OAuth information that a data source uses for authentication IPWhiteList x elevated 29 May create, view, modify or delete IP whitelists Administrator x system 12 May use the Administrator API. A user admin with the Administrator permission has all permissions and access privileges across the system. This permission can only be granted to a user in the system tenant. See also Tenant architectures on page 87 User provisioning on page 112 Logging in to the Web UI Logging in to the Web UI is a two step process. First, you must enter the URL of your Hybrid Data Pipeline instance in the address field of a supported browser. Then, you must enter your username and password at the Hybrid Data Pipeline login screen. A URL includes the Web protocol, a server name, and a port number. For example: https://MyServer:8443/hdpui The syntax for this URL can be described as follows. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 63Chapter 2: Administering Hybrid Data Pipeline webprotocol://servername:portnumber where webprotocol is the Web protocol, such as HTTP or HTTPS, used to connect to your Hybrid Data Pipeline instance. servername is the name of the machine hosting the Hybrid Data Pipeline service, or the name of the machine hosting the load balancer used to route requests to the Hybrid Data Pipeline service. portnumber is the port number of the machine hosting the Hybrid Data Pipeline service, or the port number of the machine hosting the load balancer used to route requests to the Hybrid Data Pipeline service. For a standalone installation, the port number is specified as the Server Access Port during installation. For a load balancer installation, the port number must be either 80 for http or 443 for https.Whenever port 80 or 433 are used, it is not necessary to include the port number in the URL. See also Initial login with default user accounts on page 60 Using the Web UI on page 65 Using Hybrid Data Pipeline APIs on page 64 Using Hybrid Data Pipeline APIs Hybrid Data Pipeline provides a representational state transfer (REST) application programming interface (API) for managing Hybrid Data Pipeline connectivity service resources. Hybrid Data Pipeline APIs use HTTP Basic Authentication to authenticate user accounts. The Hybrid Data Pipeline user ID and password are encoded in the Authorization header.The Hybrid Data Pipeline user specified in the Authorization header is the authenticated user. To execute REST calls, you must pass a valid REST URL and pass a valid username and password to authenticate with basic authentication. A REST URL must include a base and resource-specific information. The base includes the Web protocol, a server name, and a port number, while resource-specific information provides a path to a particular resource necessary for performing an API operation. For example: https://MyServer:8443/api/mgmt/datasources Note: The port number is only required if the Hybrid Data Pipeline server or load balancer is configured to use a port other than 443 for SSL or 80 for non-SSL connections. The syntax for a REST URL can be described as follows. webprotocol://servername:portnumber/resourceinfo where webprotocol is the Web protocol, such as HTTP or HTTPS, used to connect to your Hybrid Data Pipeline instance. 64 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Using the Web UI servername is the name of the machine hosting the Hybrid Data Pipeline service, or the name of the machine hosting the load balancer used to route requests to the Hybrid Data Pipeline service. portnumber is the port number of the machine hosting the Hybrid Data Pipeline service, or the port number of the machine hosting the load balancer used to route requests to the Hybrid Data Pipeline service. For a standalone installation, the port number is specified as the Server Access Port during installation. For a load balancer installation, the port number must be either 80 for http or 443 for https.Whenever port 80 or 433 are used, it is not necessary to include the port number in the URL. resourceinfo is resource-specific information that provides a path to a particular Hybrid Data Pipeline resource necessary to perform an API operation. See also Hybrid Data Pipeline API reference on page 1065 Initial login with default user accounts on page 60 User provisioning on page 112 Logging in to the Web UI on page 63 Using the Web UI The Hybrid Data Pipeline Web UI consists of views which can be selected from the navigation bar to the left. Access to these views, and the ability to execute operations they support, depend on permissions granted to the user (see Permissions and default roles on page 61 for details). These views include: • Manage Tenants • Manage Users • Manage Roles • Data Sources • SQL Editor • Manage External Authentication • Manage IP WhiteList • Manage Limits • System Configurations See the following topics for details on these views and other features of the Web UI. • Manage Tenants view on page 66 • Manage Users view on page 67 • Manage Roles view on page 69 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 65Chapter 2: Administering Hybrid Data Pipeline • Data Sources view on page 71 • SQL Editor view on page 77 • Manage External Authentication view on page 79 • Manage IP WhiteList view on page 80 • Manage Limits view on page 82 • System Configurations view on page 85 • User profile on page 87 • Changing your password in the Web UI on page 87 • Product information on page 86 Manage Tenants view The Manage Tenants view provides a list of tenants with description and status information for each tenant. With the appropriate permissions, you can add, modify, and delete tenants using this view. The Manage Tenants view is available to users with either set of the following permissions. • Administrator (12) permission • WebUI (8) and TenantAPI (25) permissions, and administrative access on tenants the user administers The following table provides permissions and descriptions for each action in the Manage Tenants view. Note: Any user with Administrator (12) permission may perform all actions. 66 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Using the Web UI Action Permissions Description Create new WebUI (8) To create a new tenant, click + New Tenant. Define the tenant tenant with settings under each of the following tabs. TenantAPI (25) • General tab. Enter values in the given fields.The tenant name is required. • Roles tab. Import roles from the parent tenant, if desired. • Limits tab. Set limits as desired. Edit a tenant Administrative access for the To edit a tenant, select a tenant from the list of tenants. tenant Then, select Edit from the Actions dropdown. Edit the tenant settings as desired. WebUI (8) TenantAPI (25) Delete a tenant Administrative access for the To delete a tenant, select the tenant you want to delete. tenant Then, select Delete from the Actions dropdown. Confirm or cancel the delete operation in the dialog. WebUI (8) TenantAPI (25) View tenant Administrative access for the To view the users of a tenant, select the tenant from the list users tenant of tenants. Then, select View Users from the Actions dropdown.You are directed to the Manage Users view WebUI (8) where a list of users belonging to the tenant is displayed. ViewUsers (14) See Manage Users view on page 67 for details. TenantAPI (25) Transfer tenant Administrative access for the To transfer users from the system tenant to a child tenant, users system tenant and the tenant select the child tenant from the list of tenants. Then, select to which user(s) will be Transfer Users from the Actions dropdown.You are transferred directed to the Transfer User From System Tenant page. Select each user you want to transfer to the child tenant, WebUI (8) and choose a role for each user from the role dropdown. ViewUsers (14) ModifyUsers (15) Note: Users can only be transferred from the system tenant to a child tenant. TenantAPI (25) Manage Users view The Manage Users view provides a list of users with roles and status information for a given tenant. With the appropriate permissions, you can add, update, and delete users using this view. The Manage Users view is available to users with either set of the following permissions. • Administrator (12) permission • WebUI (8) permission, ViewUsers (14) permission, ViewRole (18) permission, and administrative access on the tenant to which the users belong Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 67Chapter 2: Administering Hybrid Data Pipeline The following table provides permissions and descriptions for each action in the Manage Users view. Note: Any user with Administrator (12) permission may perform all actions. Action Permissions Description Filter users by Administrative access to An administrator with administrative access to multiple tenant multiple tenants tenants will have the option of selecting the tenant for which he or she wants to view or manage users. Select the tenant Web UI (8) for which you want to view users from the Select Tenant ViewUsers (14) dropdown. ViewRole (18) Create a new Administrative access for the To create a new user, click + New User. Define the user user tenant with settings under each of the following tabs. Web UI (8) • General tab. Enter values in the given fields. User name CreateUsers (13) and role are required. ViewUsers (14) • Authentication Setup tab. The required information depends on the type of authentication you are using. ViewRole (18) See Authentication on page 148 for details. • Limits tab. Set limits as desired. • Tenant Admin Access tab. Grant the user administrative access to tenant(s), if desired. 68 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Using the Web UI Action Permissions Description Edit a user Administrative access for the To edit a user, select a user from the list of users. Then, tenant select Edit from the Actions dropdown. Edit user settings as desired. WebUI (8) ViewUsers (14) ModifyUsers (15) ViewRole (18) Delete a user Administrative access for the To delete a user, select the user you want to delete. Then, tenant select Delete from the Actions dropdown. Confirm or cancel the delete operation in the dialog. WebUI (8) ViewUsers (14) DeleteUsers (16) ViewRole (18) View the data Administrative access for the To view the data sources owned by a user, select a user sources owned tenant from the list of users. Then, select Data Sources from the by a user Actions dropdown. A list of data sources owned by the WebUI (8) user is displayed. ViewUsers (14) ViewRole (18) OnBehalfOf (21) Manage Roles view The Manage Roles view provides a list of roles for a given tenant. With the appropriate permissions, you can add, update, and delete roles using this view. The Manage Roles view is available to users with either set of the following permissions. • Administrator (12) permission • WebUI (8) permission, ViewRole (18) permission, and administrative access on the tenant to which the role(s) belong Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 69Chapter 2: Administering Hybrid Data Pipeline The following table provides permissions and descriptions for each action in the Manage Roles view. Note: Any user with Administrator (12) permission may perform all actions. Action Permissions Description Filter roles by Administrative access to An administrator with administrative access to multiple tenant multiple tenants tenants will have the option of selecting the tenant for which he or she wants to view or manage roles. Select the tenant Web UI (8) for which you want to view roles from the Select Tenant ViewRole (18) dropdown. Create a new role Administrative access for the To create a new role, click + New Role. Provide a name tenant and description for the new role. Then, select permissions to define the role. Web UI (8) CreateRole (17) ViewRole (18) 70 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Using the Web UI Action Permissions Description Edit a role Administrative access for the To edit a role, select the role from the list of roles. Then, tenant select Edit from the Actions dropdown. Edit the role as desired. WebUI (8) ViewRole (18) ModifyRole (19) Delete a role Administrative access for the To delete a role, select the role you want to delete. Then, tenant select Delete from the Actions dropdown. Confirm or cancel the delete operation in the dialog. WebUI (8) ViewRole (18) DeleteRole (20) Data Sources view The Data Sources view allows you to manage data sources and data source groups. The Data Sources view is available to users with either set of the following permissions. • Administrator (12) permission • WebUI (8) and ViewDataSource (2) permissions The Data Sources view consists of the following pages. • Data Sources • Data Source Groups Data Sources The Data Sources page enables you to create, edit, delete, and share data source definitions. A data source definition configures the connection between Hybrid Data Pipeline and a data store. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 71Chapter 2: Administering Hybrid Data Pipeline The following table provides permissions and descriptions for basic actions in the Data Sources page. For detailed information on creating data sources, see Creating data sources with the Web UI on page 240 and How to create a data source in the Web UI on page 240. Note: With the appropriate permissions, administrators can view data sources owned by other users through the Web UI. However, administrators cannot create, modify, delete, or share data sources owned by other users through the Web UI. To create, modify, delete, or share data sources that belong to other users, administrators must use Hybrid Data Pipeline APIs. See Data Sources API on page 1306 and Managing resources on behalf of users on page 1310 for further details. Action Permissions Description Filter data Administrative access to An administrator with administrative access to multiple sources by multiple tenants tenants will have the option of filtering by tenants to view tenant data sources owned by a given user. Select the tenant in WebUI (8) which the user resides from the Select Tenant dropdown. ViewDataSource (2) ViewUsers (14) Note: Any user with the Administrator (12) permission can view the data sources of any user across all tenants. 72 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Using the Web UI Action Permissions Description Filter data Administrative access to the A user with administrative access to a tenant can filter data sources by user tenant sources by user. Select the user whose data sources you want to view from the Select User dropdown. WebUI (8) ViewDataSource (2) ViewUsers (14) Note: Any user with the Administrator (12) permission can view the data sources of any user across all tenants. Search for a data Use the search field in the upper right to filter data sources WebUI (8) source by name, data store, and description. ViewDataSource (2) Create a new WebUI (8) To create a new data source, click + New Data Source. data source See How to create a data source in the Web UI on page CreateDataSource (1) 240 for details. ViewDataSource (2) Modify a data WebUI (8) To modify a data source, select the data source from the source list of data sources. Then, select Edit from the Actions ViewDataSource (2) dropdown. Edit the data source as desired. ModifyDataSource (3) Delete a data WebUI (8) To delete a data source, select the data source you want source to delete. Then, select Delete from the Actions dropdown. ViewDataSource (2) Confirm or cancel the delete operation in the dialog. DeleteDataSource (4) Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 73Chapter 2: Administering Hybrid Data Pipeline Action Permissions Description Share a data Administrative access to the To share the data source: source tenant 1. Select the data source from the list of data sources. MgmtAPI (11) 2. Select Share from the Actions dropdown. WebUI (8) 3. Select the user or tenant with which you want to share ViewUsers (14) the data source. CreateDataSource (1) 4. Select the permissions you want to grant the user or ViewDataSource (2) tenant. 5. Click Save. Note: Any user with the To stop sharing the data source: Administrator (12) permission can share a data source he 1. Select the data source from the list of data sources. or she owns with any tenant 2. Select Share from the Actions dropdown. across the system. 3. Select the user or tenant with which you want to stop sharing the data source. 4. Click Remove. Test a data WebUI (8) To run queries against a data source through the Web UI, source select the data source. Then, select SQL Testing from the ViewDataSource (2) Actions dropdown.You are directed to the SQL Editor SQLEditorWebUI (10) view where you review schema and execute a SQL statement against the data source. At least one of the following query permissions: • UseDataSourceWithJDBC (5) • UseDataSourceWithODBC (6) • UseDataSourceWithOData (7) Sync OData WebUI (8) OData enabled data sources maintain an OData model. Schema The OData model should be refreshed whenever the ViewDataSource (2) schema of the data source has been changed. To refresh ModifyDataSource (3) the OData model, click the sync icon . For details, see MgmtAPI (11) Configuring data sources for OData connectivity and working with data source groups on page 646. 74 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Using the Web UI Action Permissions Description Obtain OData URI WebUI (8) To obtain the OData URI for an OData enabled data source, ViewDataSource (2) click the link icon to copy the link associated with the data source. Configure data WebUI (8) To configure data source logging, click the settings icon source logging ViewDataSource (2) .You are directed to the Logging Settings page. Set Logging (24) logging and privacy levels as desired. Data Source Groups The Data Source Groups page enables you to combine OData enabled data sources into a single data source group.You can create, edit, delete, and share data source groups from this page. The following table provides permissions and descriptions for basic actions in the Data Source Groups page. For detailed information on creating OData enabled data sources and data source groups, see Configuring data sources for OData connectivity and working with data source groups on page 646. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 75Chapter 2: Administering Hybrid Data Pipeline Action Permissions Description Filter data source Administrative access to An administrator with administrative access to multiple groups by tenant multiple tenants tenants will have the option of filtering by tenants to view data source groups owned by a given user. Select the WebUI (8) tenant in which the user resides from the Select Tenant ViewDataSource (2) dropdown. ViewUsers (14) Note: Any user with the Administrator (12) permission can view the data source groups of any user across all tenants. Filter data source Administrative access to the To filter data source groups by user, select the user whose groups by user tenant data source groups you want to view from the Select User dropdown. WebUI (8) ViewDataSource (2) ViewUsers (14) Note: Any user with the Administrator (12) permission can view the data source groups of any user across all tenants. Search for a data Use the search field in the upper right to filter data source WebUI (8) source group groups by name, data store, and description. ViewDataSource (2) Create a new WebUI (8) To create a new data source group, click + New Group. data source See Creating a data source group on page 659 for details. CreateDataSource (1) group ViewDataSource (2) Modify a data WebUI (8) To modify a data source group, select the group. Then, source group select Edit from the Actions dropdown. Edit the group as ViewDataSource (2) desired. ModifyDataSource (3) Delete a data WebUI (8) To delete a data source group, select the group you want source group to delete. Then, select Delete from the Actions dropdown. ViewDataSource (2) Confirm or cancel the delete operation in the dialog. DeleteDataSource (4) 76 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Using the Web UI Action Permissions Description Share a data Administrative access to the Note: Sharing a data source group requires that the source group tenant member data sources of the group also be shared. MgmtAPI (11) To share the data source group: WebUI (8) 1. Select the data source group from the list of data ViewUsers (14) sources. CreateDataSource (1) 2. Select Share from the Actions dropdown. ViewDataSource (2) 3. Select the user or tenant with which you want to share the data source group. Note: Any user with the 4. Select the permissions you want to grant the user or Administrator (12) permission tenant. can share a data source 5. Click Save. group he or she owns with any tenant across the To stop sharing the data source group: system. 1. Select the data source group from the list of data sources. 2. Select Share from the Actions dropdown. 3. Select the user or tenant with which you want to stop sharing the data source group. 4. Click Remove. Obtain OData URI WebUI (8) To obtain the OData URI of a data source group, click the ViewDataSource (2) link icon to copy the link associated with the data source group. SQL Editor view The SQL Editor view allows users to browse schemas3 and to query data associated with a data source. The SQL Editor view is available to users with either set of the following permissions. • Administrator (12) permission • WebUI (8) permission, ViewDataSource (2) permission, SQLEditorWebUI (10) permission, and, to query data sources, at least one of the following query permissions: • UseDataSourceWithJDBC (5) • UseDataSourceWithODBC (6) • UseDataSourceWithOData (7) 3 For backend data stores that support schemas, the Metadata Exposed Schemas option can be used to restrict the exposed schemas to a single schema. Metadata Exposed Schemas only affects the metadata that is displayed in the Schema navigation pane. SQL queries can still be executed against tables in other schemas. For details, see the parameters topic for your data source type. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 77Chapter 2: Administering Hybrid Data Pipeline The following table provides permissions and descriptions for actions in the SQL Editor view. To perform any action from this view, begin by selecting a data source from the Select a Data Source dropdown. Action Permissions Description Explore the WebUI (8) To begin, a data source must be selected from the Select schema and a Data Source dropdown.To view schema tables, click the ViewDataSource (2) tables associated a schema carrot in the Schema Tree panel. Click on a table with the data SQLEditorWebUI (10) to view the details of a table in the Table Details panel. source Views and procedures that reside in the schema may also be listed. Execute a SQL WebUI (8) To begin, a data source must be selected from the Select statement a Data Source dropdown. To run a query against the data ViewDataSource (2) against the data source, enter the SQL statement in the field provided in the source SQLEditorWebUI (10) Editor panel. Then click Execute to run the query. SQL query results will be returned in the Results panel. At least one of the following query permissions: Note: Queries made via the SQL Editor view time out after • UseDataSourceWithJDBC 6 minutes.Therefore, to validate a data source connection, (5) you should execute queries that require less processing • UseDataSourceWithODBC time. For large queries, only the first 200 results are shown. (6) • UseDataSourceWithOData (7) 78 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Using the Web UI Manage External Authentication view The Manage External Authentication view allows you to add, update, and delete an external authentication service.The external authentication service must first be implemented by a system administrator as described in Authentication on page 148. Once the service has been implemented, it can be added to a tenant. The Manage External Authentication view is available to users with either set of the following permissions. • Administrator (12) permission • WebUI (8) permission, RegisterExternalAuthService (26) permission, and administrative access on the given tenant The following table provides permissions and descriptions for actions in the Manage External Authentication view. Note: Any user with Administrator (12) permission may perform all actions. Action Permissions Description Filter Administrative access to An administrator with administrative access to multiple authentication multiple tenants tenants will have the option of selecting the tenant for which services by he or she wants to view or manage external authentication WebUI (8) tenant services. Select the tenant for which you want to view RegisterExternalAuthService authentication services from the Select Tenant dropdown. (26) Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 79Chapter 2: Administering Hybrid Data Pipeline Action Permissions Description Register an Administrative access for the To register an authentication service with the tenant, click external tenant + New Service. Provide the following information, and then authentication click Save. WebUI (8) service RegisterExternalAuthService • The name and description of the service (26) • The service type • For Java plugin service provide: • The class name • Attributes • For LDAP service provide: • Target URL • Service Authentication • Security Principal • Other Attributes Edit an external Administrative access for the To edit an authentication service, select the service. Then, authentication tenant select Edit from the Actions dropdown. Edit the service as service desired. WebUI (8) RegisterExternalAuthService (26) Delete an Administrative access for the To delete a service, select the service you want to delete. external tenant Then, select Delete from the Actions dropdown. Confirm authentication or cancel the delete operation in the dialog. WebUI (8) service RegisterExternalAuthService (26) Manage IP WhiteList view The Manage IP WhiteList view allows you to create and manage IP address whitelists. (See Implementing IP address whitelists on page 169 for details on the IP address whitelist feature.) The Manage IP WhiteList view is available to users with either set of the following permissions. • Administrator (12) permission • WebUI (8) permission, IPWhiteList (29) permission, and administrative access on the given tenant 80 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Using the Web UI The following table provides permissions and descriptions for actions in the Manage IP WhiteList view. Note: • Any user with Administrator (12) permission may perform all actions. • See Configuring IP address whitelists with the API on page 170 for step-by-step instructions on creating and applying IP address whitelists in the Web UI. • IP address whitelists are enabled by default. Unless you have disabled this feature, any IP address whitelist you create will immediately be enforced. For how to enable or disable IP address whitelists, see Enabling and disabling the IP address whitelist feature. Action Permissions Description Select level Administrative access to the An administrator with the permissions to set IP address system tenant whitelists at more than one level will have the option to select the level. From the Select Level dropdown, select WebUI (8) the level at which you want to apply IP address whitelists. IPWhiteList (29) • System applies the whitelist across the system. • Tenant applies the whitelist to a selected tenant. • User applies the whitelist to a specified user. Select tenant Administrative access to An administrator with administrative access to multiple multiple tenants tenants will have the option of selecting the tenant to which he or she wants to apply IP address whitelists. From the WebUI (8) Select Tenant dropdown, select the tenant to which you IPWhiteList (29) want to apply IP address whitelists. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 81Chapter 2: Administering Hybrid Data Pipeline Action Permissions Description Select user Administrative access to a An administrator with administrative access to a tenant will given tenant have the option of selecting the user to which he or she wants to apply IP address whitelists. From the Select User WebUI (8) dropdown, select the user to which you want to apply IP IPWhiteList (29) address whitelists. Configure an IP Administrative access to a To configure an IP address whitelist, click New IP Range. address whitelist given tenant From the New IP Range window, select the resource for which you are creating the whitelist. Then, enter the IP WebUI (8) address or IP address range you want to apply to the IPWhiteList (29) resource. IPv4 and IPv6 formats are supported. For details, see Implementing IP address whitelists on page 169. Manage Limits view The Manage Limits view allows you to view and set limits for features such as throttling, logging, and SQL auditing. In the Manage Limits view, limits can be set at either the system or tenant level. System limits apply to behavior across Hybrid Data Pipeline and override default behavior, while tenant limits apply to the resources of a given tenant and override default behavior and system limits. Most limits can only be configured at the system level. However, some limits, such as MaxFetchRows and MaxConcurrentQueries, can be configured at any level. Note: • Tenant limits can also be set via the Manage Tenants view on page 66. • Limits can also be specified for users and data sources. User limits can be set either through the Manage Users view on page 67 or the Limits API on page 1099. User limits override default, system, and tenant limits. Data source limits can only be set via the Limits API on page 1099. Data source limits override all other limits. The Manage Limits view is available to users with either set of the following permissions. • Administrator (12) permission • WebUI (8) permission, Limits (27) permission, and administrative access on the given tenant 82 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Using the Web UI The table below provides descriptions for limits that may be set via the Manage Limits view. Note: • Throttling limits can be set either for the system tenant or any child tenant across the system. • Log Management, Data Usage Meter, and Security limits can only be set for the system. • SQL Auditing can be set for the system tenant or for a child tenant. However, the SQLAuditingRetentionDays and SQLAuditingMaxAge limits may only be set at the system level. • To set system limits, the system tenant must be selected from the Tenant dropdown. The user must have the Administrator (12) permission. • To set tenant limits, the child tenant must be selected from the Tenant dropdown. The user must have either the Administrator (12) permission, or WebUI (8), Limits (27) permissions, and administrative access on the given tenant. Category Limit Description Throttling MaxFetchRows Maximum number of rows allowed to be fetched for a single query. Throttling ODataMaxConcurrentPagingQueries Maximum number of concurrent active queries per data source that cause paging to be invoked. Throttling TransactionTimeout The number of seconds the system allows a transaction to be idle before rolling it back. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 83Chapter 2: Administering Hybrid Data Pipeline Category Limit Description Throttling XdbcMaxResponse Approximate allowed maximum size of JDBC/ODBC HTTP result data in KB. Throttling ODataMaxConcurrentRequests Maximum number of simultaneous OData requests allowed per user. Throttling ODataMaxWaitingRequests Maximum number of waiting OData requests allowed per user. Log Management LogRetentionDays Number of days log files should be retained. Log Management MonitorRetentionDays Number of days monitor details should be retained Data Usage Meter UserMeterRetentionDays Number of days user meter details should be retained Data Usage Meter UserMeterWriteInterval The number of seconds the system waits before scanning sessions for current metrics. A lower setting will result in more rows written to the meter table Data Usage Meter UserMeterMaxAge The number seconds the system waits before writing out meter records. A lower setting will result in the rows written to meter table to occur more frequently Security PasswordLockoutInterval The duration, in seconds, for counting the number of consecutive failed authentication attempts. Security PasswordLockoutLimit The number of consecutive failed authentication attempts that are allowed before locking the user account. Security PasswordLockoutPeriod The duration, in seconds, for which a user account will not be allowed to authenticate to the system when the PasswordLockoutLimit is reached. Security OAuthAccessTokenDuration The duration, in minutes, for which a Access token is valid. Security OAuthAccessTokenCacheSize Number of oauth access tokens to be cached in memory for OAuth Authentication. By default up to 2000 tokens will be cached in memory. Security CORSBehavior Configuration parameter for CORS behavior. Setting the value to 0 disables the CORS filter. Setting the value to 1 enables the CORS filter. Setting the value to 2 enables the CORS filter with the whitelist option. 84 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Using the Web UI Category Limit Description SQL Auditing SQLAuditing Configuration parameter for SQL statement auditing. Setting the value to 0 disables SQL statement auditing. Setting the value to 1 enables SQL statement auditing. SQL Auditing SQLAuditingRetentionDays The number of days auditing records are retained in the SQLAudit table. SQL Auditing SQLAuditingMaxAge The maximum number of seconds the service waits before inserting the auditing records into the SQLAudit table. A lower setting will increase the frequency with which records are written to the SQLAudit table. System Configurations view The System Configurations view can be used to set a number of configurations across the Hybrid Data Pipeline system. This view is only available to users with the Administrator (12) permission (system administrators). The following table provides descriptions of the options available via the System Configurations view. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 85Chapter 2: Administering Hybrid Data Pipeline Option Permissions Description Delimiter Administrator (12) Specifies a delimiter to be used between the user name and authentication service name. In the following example, the | symbol delimits user437 and the LDAP1 service: user437|LDAP1. See Authentication on page 148 for details. Secure Password Change Administrator (12) Specifies whether the current password is required in order to update the password of the logged-in user. The default value is ON. Default OData Version Administrator (12) Sets the default OData version for new data sources. Default Entity Name Administrator (12) Sets the default entity name mode for OData V4 data sources. For details, see Configuring data sources for OData connectivity and working with data source groups on page 646. JDBC DataStore Administrator (12) Enables the third party JDBC data store feature. The default value is ON. For details, see Using third party JDBC drivers with Hybrid Data Pipeline on page 197. Password Policy Administrator (12) Enables the default password policy.The default value is ON. System Monitor Details Administrator (12) Determines how the system persists monitor details. IP WhiteList Filtering Administrator (12) Enables the whitelist filtering feature.The default value is ON. See Implementing IP address whitelists on page 169 for details. Product information Users can access product information by clicking the question mark icon and selecting About. The About Hybrid Data Pipeline window displays installation and version information. 86 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Tenant architectures User profile The down arrow next to the username in the upper right hand corner of the Web UI opens a dropdown menu. Users can change their passwords by selecting the Change Password item, or log out by selecting the Log Out item. Changing your password in the Web UI Take the following steps to change your password in the WebUI. Note: You can also change your password using the Change Password API. 1. Select the arrow next to your username in the right hand corner of the Web UI. 2. Click Change Password to open the Change Password window. 3. Enter your current password in the Current Password field. 4. Enter your new password in the New Password field. Note: The password must be a maximum of 32 characters in length. 5. Retype your new password in the Confirm Password field. 6. Click SAVE. Tenant architectures A Hybrid Data Pipeline system administrator can develop either a single-tenant or multitenant architecture. In a single-tenant architecture, the system administrator creates user accounts in the default system tenant. In a multitenant architecture, the system administrator first creates one or more child tenants in the default system tenant. Then, the system administrator can create user accounts in either the system tenant or any one of the child tenants. The user accounts that reside in one tenant are isolated from those in other tenants. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 87Chapter 2: Administering Hybrid Data Pipeline When establishing a tenant architecture, the system administrator should consider the roles users and other administrators will assume in the Hybrid Data Pipeline environment. As detailed in Permissions and default roles on page 61, Hybrid Data Pipeline provides three default roles: System Administrator, Tenant Administrator, and User. These roles can be used in either a single-tenant or multitenant architecture. In the context of these roles, the system administrator has full permissions and administrative access across the system, while the tenant administrator can assume responsibility for provisioning and managing user accounts in tenants for which he or she has administrative access. Important: To administer user accounts and other resources that belong to a tenant, a tenant administrator must be given explicit administrative access to the given tenant. In the Web UI, administrative access to a tenant can be granted by editing a user account via the Manage Users view on page 67. With the API, administrative access can be granted either by updating the tenants administered for a user via the Users API or by updating the list of administrators for a tenant via the Tenant API. The following topics describe single-tenant and multitenant architectures in greater detail, including how administrative roles can be applied in each. • Single-tenant environment • Multitenant environment Single-tenant environment Tenancy is mostly transparent in a single-tenant environment where users and features are managed from the default system tenant. Nevertheless, tenant and elevated permissions were introduced with support for multitenancy. Tenant permissions support the ability of administrators to provision and manage users, while elevated permissions support the ability of administrators to execute other administrative tasks, such as throttling and logging. By granting such permissions to other users, the system administrator can delegate administrative tasks and responsibilities. Note: As an alternative to using the default system tenant, a system administrator could create a child tenant in the system tenant. This child tenant could function as a single, dedicated tenant from which users and features are managed. See the following topics on how to set up a single-tenant environment. • Using the Web UI to set up a single-tenant environment on page 88 • Using the APIs to set up a single-tenant environment on page 89 See also User provisioning on page 112 Permissions and default roles on page 61 Using the Web UI to set up a single-tenant environment The following steps show how you can set up a single-tenant environment using the Web UI. Note: It is assumed that users and features will be managed from the default system tenant. Therefore, there is no step to create a child tenant. 1. Create administrator roles. 88 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Tenant architectures a) Navigate to the Manage Roles view by clicking the manage roles icon . b) Click + New Role. c) Provide a name and description for the role. d) Select permissions to define the role. e) Click Save. 2. Create administrator users. a) Navigate to the Manage Users view by clicking the manage users icon . b) Click + New User. c) Define the user with settings under each of the following tabs. • Under the General tab, enter a user name and assign the role you have created for the user. • Under the Authentication Setup tab, configure authentication settings. • Under the Limits tab, configure limits as desired. Note that user limits override system limits. • Under the Tenant Admin Access tab, grant the user administrative access to the system tenant. d) Click Save. 3. Set system configurations. a) Navigate to the System Configurations view by clicking the system configurations icon . b) Configure options as desired. See System Configurations view on page 85 for option descriptions. c) Click Save. 4. Set limits. a) Navigate to the Manage Limits view by clicking the manage limits icon . b) Set limits as desired. See Manage Limits view on page 82 for limit descriptions. c) Click Save. Using the APIs to set up a single-tenant environment The following operations show how you can set up a single-tenant environment using Hybrid Data Pipeline APIs. Note: It is assumed that users and features will be managed from the default system tenant. Therefore, there is no step to create a child tenant. • Retrieving valid roles in the system tenant • Create a user with the Tenant Administrator role • Grant the administrator user administrative access to the system tenant • Create a new role with tenant and elevated permissions Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 89Chapter 2: Administering Hybrid Data Pipeline • Assign the new role to the administrator user • Retrieving and setting system configurations • Retrieving and setting limits Retrieving valid roles in the system tenant The following GET operation retrieves the valid roles and their IDs for the system tenant in a single-tenant environment. Role IDs can then be used to assign roles to users. Request GET https://MyServer:8443/api/admin/roles Response Payload { "roles": [ { "id": 1, "name": "System Administrator", "tenantId": 1, "description": "This role has all permissions. This role cannot be modified or deleted." }, { "id": 2, "name": "User", "tenantId": 1, "description": "This role has the default permissions that a normal user will be expected to have." }, { "id": 3, "name": "Tenant Administrator", "tenantId": 1, "description": "This role has all the tenant administrator permissions." } ] } Create a user with the Tenant Administrator role The ID for the Tenant Administrator role (3) can then be used to create a user with the Tenant Administrator role, as shown in the following POST operation. The user inherits the permissions associated with this role. Request POST https://MyServer:8443/api/admin/users Request Payload { "userName": "TenantAdmin", "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "password": "", "passwordStatus": 1, "passwordExpiration": "2020-01-01 00:00:00" }, "permissions": { 90 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Tenant architectures "roles": [ 3 ] } } Response Payload { "id": 87, "userName": "TenantAdmin", "tenantId": 1, "tenantName": "Root", "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "passwordStatus": 1, "passwordExpiration": "2020-01-01 00:00:00.0" }, "permissions": { "roles": [ 3 ] }, "authenticationInfo": { "authUsers": [ { "authUserName": "TenantAdmin", "authServiceId": 1 } ] } } Grant the administrator user administrative access to the system tenant In addition to being granted the Tenant Administrator role, the tenant administrator must be granted administrative access to the system tenant. The following Users API request grants user account 87 administrative access to the system tenant. Note: Administrative access to the system tenant can also be managed by updating the list of administrators via the Tenant API. Request PUT https://MyServer:8443/api/admin/users/87/tenantsadministered Request Payload { "tenantsAdministered": [ 1 ] } Response Payload { "tenantsAdministered": [ 1 ] } Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 91Chapter 2: Administering Hybrid Data Pipeline Create a new role with tenant and elevated permissions The following POST request creates the new Tenant Admin Plus role. The new role has all user and tenant permissions plus the Logging (24), Limits (27), and OAuth (28) permissions. Request POST https://MyServer:8443/api/admin/roles Request Payload { "name": "Tenant Admin Plus", "description": "This role has all the tenant administrator permissions plus elevated permissions.", "permissions": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21, 24, 27, 28 ], "users": [] } Response Payload { "id": 42, "name": "Tenant Admin Plus", "description": "This role has all the tenant administrator permissions plus elevated permissions.", "permissions": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 92 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Tenant architectures 20, 21, 24, 27, 28 ], "users": [] } Assign the new role to the administrator user The following PUT assigns the new Tenant Admin Plus role to the administrator user. The user inherits the permissions associated with this role. Note that the ID of the Tenant Admin Plus role (42) was provided in the response payload when the role was created. Also, note that any existing roles and permissions are removed by this operation. Request PUT https://MyServer:8443/api/admin/users/87/permissions Request Payload { "roles": [42], "permissions": [] } Response Payload { "roles": [42] } Retrieving and setting system configurations The following GET operation retrieves a list of system configurations. Request GET https://MyServer:8443/api/admin/configurations Response Payload Note: See System Configurations API on page 1152 for a complete list of system configurations and their descriptions. { "configurations": [ { "id": 1, "description": "Delimiter between user name and authentication service/configuration name", "value": null }, { "id": 2, "description": "Enable Secure Password Change, when value is set to true, the change password api will require a valid old password in order to update the logged in user password.", "value": "true" }, ..., Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 93Chapter 2: Administering Hybrid Data Pipeline { "id": 8, "description": "Configure whitelist filtering. Enables filtering when value is set to ''true''. Default value is "true" ", "value": "true" } ] } The following PUT operation disables IP address whitelists. The number 8 is the ID of the IP address whitelist feature. Request PUT https://MyServer:8443/api/admin/configurations/8 Request Payload { "value":"false" } Retrieving and setting limits The following GET operation retrieves a list of limits. Request GET https://MyServer:8443/api/admin/limits Response Payload Note: See Limits API on page 1099 for a complete list of limits and their descriptions. { "limits": [ { "id": 1, "name": "MaxFetchRows", "description": "Maximum number of rows allowed to be fetched for a single query", "minValue": 1, "maxValue": 9000000000000000000, "defaultValue": 9000000000000000000, "validForLimits": 15 }, ..., { "id": 6, "name": "ODataMaxConcurrentQueries", "description": "Maximum number of concurrent active queries per data source", "minValue": 0, "maxValue": 9000000000000000000, "defaultValue": 0, "validForLimits": 15 }, ... ] } The following POST creates a system-level limit of 50000 queries. The number 6 is the ID of the ODataMaxConcurrentQueries limit. The payload passes 50000 as the value for this limit. 94 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Tenant architectures Request POST https://MyServer:8443/api/admin/limits/system/6 Request Payload { "value": 50000 } See also User provisioning on page 112 Users API on page 1174 Roles API on page 1140 System Configurations API on page 1152 Limits API on page 1099 Multitenant environment Multitenancy allows system administrators to isolate groups of users, such as organizations or departments, hosted by the Hybrid Data Pipeline service. The system administrator maintains a physical instance of Hybrid Data Pipeline, while each tenant (group of users) is provided with its own logical instance of the service. To create a multitenant environment, the system administrator creates child tenants in the default system tenant. The system administrator can then proceed with setting up administrative and support structures for maintaining the Hybrid Data Pipeline environment.The administration of tenants follows two general patterns: system-level administration and tenant-level administration. In system-level administration, a system administrator may want to delegate or share user provisioning and other administrative tasks with a tenant administrator who can manage user accounts and enable supported features across multiple tenants. In this instance, the system administrator creates tenant administrators in the system tenant with user management permissions and administrative access to the tenants they will manage. These tenant administrators are able to manage users, data sources, and other resources across multiple tenants. In tenant-level administration, the system administrator delegates user provisioning and other administrative tasks to tenant administrators who belong to one of many tenants. For example, a Hybrid Data Pipeline provider may host several external organizations where it is appropriate for the organizations themselves to provision users and administer data access. In this scenario, the system administrator would create tenant administrators who reside in the tenants they administer, thus isolating administrative tasks such as user provisioning from one tenant to another. For tenant-level administration, tenant administrators must have administrative access to the tenants in which they reside, as well as user management and other permissions as needed. Note that system-level and tenant-level administration are not mutually exclusive. For example, a system administrator might want to delegate and isolate the administration of tenants, but also provision support personnel to work with resources across multiple tenants. The following topics provide information on creating multitenant environments. • Setting up a multitenant environment with system-level administration on page 95 • Setting up a multitenant environment with tenant-level administration on page 104 Setting up a multitenant environment with system-level administration A system administrator can take the following general steps to set up a multitenant environment with system-level administration. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 95Chapter 2: Administering Hybrid Data Pipeline 1. Create child tenants. 2. Create administrator roles in the system tenant. 3. Create tenant administrators in the system tenant. 4. Set system configurations and limits. See the following topics for details. • Using the Web UI to set up a multitenant environment with system-level administration on page 96 • Using the APIs to set up a multitenant environment with system-level administration on page 97 Using the Web UI to set up a multitenant environment with system-level administration Take the following steps to set up a multitenant environment with system-level administration using the Web UI. 1. Create tenants. a) Navigate to the Manage Tenants view by clicking the manage tenants icon . b) Click + New Tenant. c) Under the General tab, enter a name and description for the tenant. d) Under the Roles tab, select any roles that you created in the system tenant that you want to import to the new tenant. e) Under the Limits tab, specify any limits that you want to set for the tenant. These limits will override limits at the system level. f) Click Save. 2. Create administrator roles. a) Navigate to the Manage Roles view by clicking the manage roles icon . b) For Tenant, select System from the dropdown. c) Click + New Role. d) Provide a name and description for the role. e) Select permissions to define the role. f) Click Save. 3. Create administrator users. a) Navigate to the Manage Users view by clicking the manage users icon . b) For Tenant, select System from the dropdown. c) Click + New User. d) Define the user with settings under each of the following tabs. • Under the General tab, enter a user name and assign the role you have created for the user. • Under the Authentication Setup tab, configure authentication settings. • Under the Limits tab, configure limits as desired. Note that user limits override system limits. 96 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Tenant architectures • Under the Tenant Admin Access tab, grant the user administrative access to any tenants they will be administering. e) Click Save. 4. Set system configurations. These configurations apply to all tenants across the system. a) Navigate to the System Configurations view by clicking the system configurations icon . b) Configure options as desired. See System Configurations view on page 85 for option descriptions. c) Click Save. 5. Set system limits. a) Navigate to the Manage Limits view by clicking the manage limits icon . b) For Tenant, select System from the dropdown. c) Set limits as desired. Limits set on tenants will override limits set at the system level. d) Click Save. Results: You have created child tenants in the system tenant. In addition, you have created tenant administrators who reside in the system tenant. These administrators can perform administrative tasks, based on the permissions associated with their roles, in any tenants to which they have administrative access. System configurations and limits have been set as well. Using the APIs to set up a multitenant environment with system-level administration The following operations show how you can set up a multitenant environment with system-level administration using Hybrid Data Pipeline APIs. • Creating tenants with the Tenant API • Retrieving roles with the Roles API • Provisioning a user at the system level with the Tenant Administrator role • Granting administrative access to the tenant with the Users API • Granting administrative access to the tenant with the Tenant API • Setting system configurations and limits • Setting tenant limits • Creating users and roles at the tenant level Creating tenants with the Tenant API In this example, a system administrator creates TenantA, TenantB, and TenantC using the Tenant API. The User (2) role has been specified as an imported role. As new tenants are created, the imported role becomes unique and is given a new ID. Request to create TenantA POST https://MyServer:8443/api/admin/tenants Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 97Chapter 2: Administering Hybrid Data Pipeline Request Payload { "name": "TenantA", "description": "This is the HDP tenant for organization A.", "parentTenant": 1, "status": 1, "importedRoles": [ 2 ] } Response Payload { "id": 61, "name": "TenantA", "description": "This is the HDP tenant for organization A.", "parentTenant": 1, "status": 1, "roles": [ 81 ] } Request to create TenantB POST https://MyServer:8443/api/admin/tenants Request Payload { "name": "TenantB", "description": "This is the HDP tenant for organization B.", "parentTenant": 1, "status": 1, "importedRoles": [ 2 ] } Response Payload { "id": 62, "name": "TenantB", "description": "This is the HDP tenant for organization B.", "parentTenant": 1, "status": 1, "roles": [ 82 ] } Request to create TenantC POST https://MyServer:8443/api/admin/tenants Request Payload { "name": "TenantC", "description": "This is the HDP tenant for organization C.", "parentTenant": 1, "status": 1, "importedRoles": [ 2 98 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Tenant architectures ] } Response Payload { "id": 63, "name": "TenantC", "description": "This is the HDP tenant for organization C.", "parentTenant": 1, "status": 1, "roles": [ 83 ] } Retrieving roles with the Roles API The system administrator must have the role ID to create a user with the Tenant Administrator role.The following GET operation retrieves the roles across the system. Request GET https://MyServer:8443/api/admin/roles Response Payload The first three roles in the payload are roles tied to the system tenant ("tenantId": 1). The remaining roles are the roles copied to the new tenants. { "roles": [ { "id": 1, "name": "System Administrator", "tenantId": 1, "description": "This role has all permissions. This role cannot be modified or deleted." }, { "id": 2, "name": "User", "tenantId": 1, "description": "This role has the default permissions that a normal user will be expected to have." }, { "id": 3, "name": "Tenant Administrator", "tenantId": 1, "description": "This role has all the tenant administrator permissions." }, { "id": 81, "name": "User", "tenantId": 61, "description": "This role has the default permissions that a normal user will be expected to have." }, { "id": 82, "name": "User", "tenantId": 62, "description": "This role has the default permissions that a normal user will be expected to have." }, Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 99Chapter 2: Administering Hybrid Data Pipeline { "id": 83, "name": "User", "tenantId": 63, "description": "This role has the default permissions that a normal user will be expected to have." } ] } Creating a user at the system level with the Tenant Administrator role With the following User API operation, the system administrator creates a user at the system level with the Tenant Administrator role. The tenant administrator must then be given administrative access to the tenant either through the Users API or the Tenant API, as described below. Request POST https://MyServer:8443/api/admin/users Request Payload { "userName": "SysTenantAdmin", "tenantId": 1, "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "password": "TempWord", "passwordStatus": 1, "passwordExpiration": null }, "permissions": { "roles": [ 3 ] } } Response Payload { "id": 1001, "userName": "SysTenantAdmin", "tenantId": 1, "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "passwordStatus": 1, "passwordExpiration": null }, "permissions": { "roles": [ 3 ] }, "authenticationInfo": { "authUsers": [ { "authUserName": "SysTenantAdmin", "authServiceId": 1 } ] 100 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Tenant architectures } } Granting administrative access to the tenant with the Users API In addition to user management permissions, a tenant administrator must be granted administrative access to the tenant. This can be done either through the Users API or the Tenant API. The following Users API request grants user account 2001 administrative access to TenantA (61). Request PUT https://MyServer:8443/api/admin/users/2001/tenantsadministered Request Payload { "tenantsAdministered": [ 61 ] } Response Payload { "tenantsAdministered": [ 61 ] } Granting administrative access to the tenant with the Tenant API In addition to user management permissions, a tenant administrator must be granted administrative access to the tenant. This can be done either through the Users API or the Tenant API. The following Tenant API request adds user account 2001 to the list of administrators who can administer the TenantA (61). PUT https://MyServer:8443/api/admin/tenants/61 Request Payload { "admins": [ 391, 502, 2001 ] } Response Payload { "admins": [ 391, 502, 2001 ] } Setting system configurations and limits Setting a system configuration Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 101Chapter 2: Administering Hybrid Data Pipeline The following PUT operation disables IP address whitelists across all tenants. The number 8 is the ID of the IP address whitelist feature. PUT https://MyServer:8443/api/admin/configurations/8 { "value":"false" } Setting a system limit The following POST creates a limit of 50000 concurrent OData queries across all tenants. The number 6 is the ID of the ODataMaxConcurrentQueries limit. The payload passes 50000 as the value for this limit. POST https://MyServer:8443/api/admin/limits/system/6 { "value": 50000 } Setting tenant limits The following POST creates a limit of 10000 concurrent OData queries on TenantA. The number 61 is the ID of TenantA, and the number 6 is the ID of the ODataMaxConcurrentQueries limit. This tenant limit will override the system limit. POST https://MyServer:8443/api/admin/limits/tenants/61/6 { "value": 10000 } Creating users and roles at the tenant level The new system-level tenant administrator (SysTenantAdmin) can now provision users and create roles for TenantA, TenantB, and TenantC. The first request creates a new user in TenantA (61). The second request creates a new role in TenantA. Request to create user in TenantA POST https://MyServer:8443/api/admin/users Request Payload { "userName": "User1A", "tenantId": 61, "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "password": "TempWord", "passwordStatus": 1, "passwordExpiration": null }, "permissions": { "roles": [ 81 ] } } 102 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Tenant architectures Response Payload { "id": 2601, "userName": "User1A", "tenantId": 1, "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "passwordStatus": 1, "passwordExpiration": null }, "permissions": { "roles": [ 81 ] }, "authenticationInfo": { "authUsers": [ { "authUserName": "SysTenantAdmin", "authServiceId": 1 } ] } } POST operation to create new role With the following POST request, a new role is created in TenantA (61) for OData-only access to data sources. No user is specified in this example, but a user can subsequently be assigned the new role either through the Roles API or the Users API. Request to create role in TenantA POST https://MyServer:8443/api/admin/roles Request Payload { "name": "ODataOnly", "tenantId": 61, "description": "This role allows only OData access.", "permissions": [7], "users": [] } Response Payload { "id": 102, "name": "ODataOnly", "tenantId": 61, "description": "This role allows only OData access.", "permissions": [ 7 ], "users": [] } See also User provisioning on page 112 Users API on page 1174 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 103Chapter 2: Administering Hybrid Data Pipeline Roles API on page 1140 System Configurations API on page 1152 Limits API on page 1099 Setting up a multitenant environment with tenant-level administration A system administrator can take the following general steps to set up a multitenant environment with tenant-level administration. 1. Create child tenants. 2. Create administrator roles in the child tenants. 3. Create tenant administrators who reside in the child tenants. 4. Set system configurations and limits. See the following topics for details. • Using the Web UI to set up a multitenant environment with tenant-level administration on page 104 • Using the APIs to set up a multitenant environment with tenant-level administration on page 105 Using the Web UI to set up a multitenant environment with tenant-level administration Take the following steps to set up a multitenant environment with tenant-level administration using the Web UI. 1. Create tenants. a) Navigate to the Manage Tenants view by clicking the manage tenants icon . b) Click + New Tenant. c) Under the General tab, enter a name and description for the tenant. d) Under the Roles tab, select any roles that you created in the system tenant that you want to import to the new tenant. e) Under the Limits tab, specify any limits that you want to set for the tenant. These limits will override limits at the system level. f) Click Save. 2. Create administrator roles. a) Navigate to the Manage Roles view by clicking the manage roles icon . b) For Tenant, select the child tenant for which you want to create the new administrator role. c) Click + New Role. d) Provide a name and description for the role. e) Select permissions to define the role. f) Click Save. 3. Create administrator users. a) Navigate to the Manage Users view by clicking the manage users icon . b) For Tenant, select the child tenant for which you want to create the new administrator user. 104 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Tenant architectures c) Click + New User. d) Define the user with settings under each of the following tabs. • Under the General tab, enter a user name and assign the role you have created for the user. • Under the Authentication Setup tab, configure authentication settings. • Under the Limits tab, configure limits as desired. Note that user limits override system limits. • Under the Tenant Admin Access tab, grant the user administrative access to the child tenant. e) Click Save. 4. Set system configurations. These configurations apply to all tenants across the system. a) Navigate to the System Configurations view by clicking the system configurations icon . b) Configure options as desired. See System Configurations view on page 85 for option descriptions. c) Click Save. 5. Set system limits. a) Navigate to the Manage Limits view by clicking the manage limits icon . b) For Tenant, select System from the dropdown. c) Set limits as desired. Limits set on tenants will override limits set at the system level. d) Click Save. Results: You have created child tenants in the system tenant. In addition, you have created tenant administrators who reside in the child tenants. These administrators can perform administrative tasks, based on the permissions associated with their roles, in the tenants to which they belong and have administrative access. System configurations and limits have been set as well. Using the APIs to set up a multitenant environment with tenant-level administration The following operations show how you can set up a multitenant environment with tenant-level administration using Hybrid Data Pipeline APIs. • Creating tenants with the Tenant API • Retrieving roles with the Roles API • Provisioning a tenant user with the Tenant Administrator role • Granting administrative access to the tenant with the Users API • Granting administrative access to the tenant with the Tenant API • Setting system configurations and limits • Setting tenant limits • Creating users and roles at the tenant level Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 105Chapter 2: Administering Hybrid Data Pipeline Creating tenants with the Tenant API In this example, a system administrator creates the following tenants with the Tenant API: OrgA, OrgB, and OrgC. The default User (2) and Tenant Administrator (3) roles are being imported from the system tenant. As the system tenants are created, the imported roles becomes unique and are given a new IDs. Request to create OrgA POST https://MyServer:8443/api/admin/tenants Request Payload { "name": "OrgA", "description": "This is the HDP tenant for organization A.", "parentTenant": 1, "status": 1, "importedRoles": [ 2, 3 ] } Response Payload { "id": 71, "name": "OrgA", "description": "This is the HDP tenant for organization A.", "parentTenant": 1, "status": 1, "roles": [ 103, 104 ] } Request to create OrgB POST https://MyServer:8443/api/admin/tenants Request Payload { "name": "OrgB", "description": "This is the HDP tenant for organization B.", "parentTenant": 1, "status": 1, "importedRoles": [ 2, 3 ] } Response Payload { "id": 72, "name": "OrgA", "description": "This is the HDP tenant for organization A.", "parentTenant": 1, "status": 1, "roles": [ 105, 106 ] } 106 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Tenant architectures Request POST https://MyServer:8443/api/admin/tenants Request Payload to create OrgC { "name": "OrgC", "description": "This is the HDP tenant for organization C.", "parentTenant": 1, "status": 1, "importedRoles": [ 2, 3 ] } Response Payload { "id": 73, "name": "OrgC", "description": "This is the HDP tenant for organization C.", "parentTenant": 1, "status": 1, "roles": [ 107, 108 ] } Retrieving roles with the Roles API The system administrator must have the role ID to create a user with the Tenant Administrator role.The following GET operation retrieves the roles across the system. Request GET https://MyServer:8443/api/admin/roles Note: The ?tenantID= and ?tenantName= query parameters can be appended to the URL to limit the roles returned to a specific tenant. Response Payload The first three roles in the payload are roles tied to the system tenant ("tenantId": 1). The remaining roles are the User and Tenant Administrator roles copied to the new tenants. { "roles": [ { "id": 1, "name": "System Administrator", "tenantId": 1, "description": "This role has all permissions. This role cannot be modified or deleted." }, { "id": 2, "name": "User", "tenantId": 1, "description": "This role has the default permissions that a normal user will be expected to have." }, Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 107Chapter 2: Administering Hybrid Data Pipeline { "id": 3, "name": "Tenant Administrator", "tenantId": 1, "description": "This role has all the tenant administrator permissions." }, { "id": 103, "name": "User", "tenantId": 71, "description": "This role has the default permissions that a normal user will be expected to have." }, { "id": 104, "name": "Tenant Administrator", "tenantId": 71, "description": "This role has all the tenant administrator permissions." }, { "id": 105, "name": "User", "tenantId": 72, "description": "This role has the default permissions that a normal user will be expected to have." }, { "id": 106, "name": "Tenant Administrator", "tenantId": 72, "description": "This role has all the tenant administrator permissions." }, { "id": 107, "name": "User", "tenantId": 73, "description": "This role has the default permissions that a normal user will be expected to have." }, { "id": 108, "name": "Tenant Administrator", "tenantId": 73, "description": "This role has all the tenant administrator permissions." } ] } Provisioning a tenant user with the Tenant Administrator role With the following User API operation, the system administrator creates a user in the OrgA tenant (71) with the Tenant Administrator role. The tenant administrator must then be given administrative access to the tenant either through the Users API or the Tenant API, as described below. Request POST https://MyServer:8443/api/admin/users Request Payload { "userName": "OrgAAdmin", "tenantId": 71, "statusInfo": { "status": 1, "accountLocked": false }, 108 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Tenant architectures "passwordInfo": { "password": "TempWord", "passwordStatus": 1, "passwordExpiration": null }, "permissions": { "roles": [ 103, 104 ] } } Response Payload { "id": 2001, "userName": "OrgAAdmin", "tenantId": 71, "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "passwordStatus": 1, "passwordExpiration": null }, "permissions": { "roles": [ 103, 104 ] }, "authenticationInfo": { "authUsers": [ { "authUserName": "OrgAAdmin", "authServiceId": 1 } ] } } Granting administrative access to the tenant with the Users API In addition to user management permissions, a tenant administrator must be granted administrative access to the tenant. This can be done either through the Users API or the Tenant API. The following Users API request grants user account 2001 administrative access to the OrgA tenant (71). Request PUT https://MyServer:8443/api/admin/users/2001/tenantsadministered Request Payload { "tenantsAdministered": [ 71 ] } Response Payload { "tenantsAdministered": [ 71 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 109Chapter 2: Administering Hybrid Data Pipeline ] } Granting administrative access to the tenant with the Tenant API In addition to user management permissions, a tenant administrator must be granted administrative access to the tenant. This can be done either through the Users API or the Tenant API. The following Tenant API request adds user account 2001 to the list of administrators who can administer the OrgA tenant (71). PUT https://MyServer:8443/api/admin/tenants/71 Request Payload { "admins": [ 391, 502, 2001 ] } Response Payload { "admins": [ 391, 502, 2001 ] } Setting system configurations and limits Setting a system configuration The following PUT operation disables IP address whitelists across all tenants. The number 8 is the ID of the IP address whitelist feature. PUT https://MyServer:8443/api/admin/configurations/8 { "value":"false" } Setting a system limit The following POST creates a limit of 50000 concurrent OData queries across all tenants. The number 6 is the ID of the ODataMaxConcurrentQueries limit. The payload passes 50000 as the value for this limit. POST https://MyServer:8443/api/admin/limits/system/6 { "value": 50000 } 110 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1Tenant architectures Setting tenant limits The following POST creates a limit of 10000 concurrent OData queries on the OrgA tenant. The number 71 is the ID of OrgA, and the number 6 is the ID of the ODataMaxConcurrentQueries limit. This tenant limit will override the system limit. POST https://MyServer:8443/api/admin/limits/tenants/71/6 { "value": 10000 } Creating users and roles at the tenant level The new tenant administrator (OrgAAdmin) can now provision users and create roles for the OrgA tenant (71). The first request creates a new user in OrgA. The second request creates a new role in OrgA. Request POST https://MyServer:8443/api/admin/users Request Payload { "userName": "OrgAUser1", "tenantId": 71, "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "password": "TempWord", "passwordStatus": 1, "passwordExpiration": null }, "permissions": { "roles": [ 104 ] } } Response Payload { "id": 3222, "userName": "OrgAUser1", "tenantId": 71, "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "passwordStatus": 1, "passwordExpiration": null }, "permissions": { "roles": [ 104 ] }, "authenticationInfo": { "authUsers": [ { "authUserName": "OrgAUser1", "authServiceId": 1 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 111Chapter 2: Administering Hybrid Data Pipeline } ] } } With the following POST request, a new role is created in the OrgA tenant for OData-only access to data sources. No user is specified in this example, but a user can subsequently be assigned the new role either through the Roles API or the Users API. Request POST https://MyServer:8443/api/admin/roles Request Payload { "name": "ODataOnly", "tenantId": 71, "description": "This role allows only OData access.", "permissions": [7], "users": [] } Response Payload { "id": 311, "name": "ODataOnly", "tenantId": 71, "description": "This role allows only OData access.", "permissions": [ 7 ], "users": [] } See also User provisioning on page 112 Users API on page 1174 Roles API on page 1140 System Configurations API on page 1152 Limits API on page 1099 User provisioning Once a tenant architecture has been established as described in Tenant architectures on page 87, a Hybrid Data Pipeline administrator can proceed with provisioning users. User accounts can be created and managed either through the Web UI or using Hybrid Data Pipeline APIs. User accounts must have at least one assigned role. A role is defined by the permissions that are associated with it. Users can be provisioned to have either direct access to the Hybrid Data Pipeline service or query-only access to Hybrid Data Pipeline data sources. Whether a user is a direct-access or query-only user depends on the role assigned and its associated permissions. • Direct-access user • Query-only user • Administrator permissions 112 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1User provisioning • User provisioning scenarios Direct-access user A direct-access user is a user the administrator has provisioned with direct access to the service to create, manage, and query data sources. The following work flow describes how access to data may be established with a direct-access user. 1. The administrator creates a role for a direct-access user. 2. The administrator creates a user account for the direct-access user. 3. The direct-access user creates a data source through either the Web UI or the Data Sources API on page 1306. Note: Alternatively, administrators can create their own data sources and share them with users or create data sources on behalf of users. 4. Data source connection information is integrated into a client-side application or BI tool. Query-only user An administrator can limit user access such that users can run applications against Hybrid Data Pipeline data sources, but not access Hybrid Data Pipeline directly. In this scenario, the administrator must not only provision user accounts, but also create the data sources against which queries will be made.The data source information may then be supplied either directly to the query-only user, or integrated into the client application such that data access is transparent to the application end user. Thus, client applications are given access to backend data stores, while users and developers on the client side do not otherwise have access to Hybrid Data Pipeline. When provisioning users for query-only access to Hybrid Data Pipeline data sources, administrators can manage data sources in two distinct ways. • First, they can create a data source themselves, and then share the data source with one or more user accounts. In this case, the data source information, including connection information, is the same for all accounts querying the data source. Hence, sharing data sources can be used to support general access to a backend data store when access to the data is the same across multiple end users. For example, an administrator might create a data source to support the use of a reporting tool. Multiple end users across the organization use the tool to run reports against the backend data store. In this case, connection information associated with the data source can be integrated with the reporting tool. Hybrid Data Pipeline may be entirely transparent to the users running the reports. However, the reporting tool uses the Hybrid Data Pipeline data source to access the backend data. Administrators can share data sources either through the Data Sources API or the Web UI. • Second, the administrator can create a data source on behalf of a user account. In this scenario, the data source is owned by the user account, and the data source information is unique to the account. Therefore, creating data sources on behalf of users should be used in scenarios where access to backend data must be unique for each user. For example, a backend data store might have row-level security measures on an Employee database such that managers are only able to access information for the employees they manage. In this case, an administrator would create data sources on the backend data store that are unique to each manager based on each manager''s credentials. Administrators must use the Hybrid Data Pipeline API to create data sources on behalf of users. The following work flow describes how access to data may be enabled for a query-only user. 1. The administrator creates a role for the query-only user. 2. The administrator creates a user account for the query-only user. 3. The administrator uses either of the following methods to create a Hybrid Data Pipeline data source for the query-only user. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 113Chapter 2: Administering Hybrid Data Pipeline a. The administrator creates a data source through either the Web UI or the Data Sources API on page 1306. The administrator then shares the data source with the query-only user based on the rules and guidelines in Sharing data sources on page 1308. b. The administrator creates a data source on behalf of the query-only user as described in the Data Sources API on page 1306 and Managing resources on behalf of users on page 1310. 4. Data source connection information is integrated into a client-side application or BI tool. Administrator permissions The ability of an administrator to provision users depend on the administrator''s permissions and administrative access to a given tenant. A system administrator – defined as a user with the Administrator (12) permission – can provision users across any tenant in the system. An administrator who does not have the Administrator (12) permission must meet the following requirements to provision users. • WebUI (8) permission must be granted if the administrator is using the Web UI to provision users. • Administrative access to the tenant. In the Web UI, administrative access to a tenant can be granted by editing a user account via the Manage Users view on page 67. With the API, administrative access can be granted either by updating the tenants administered for a user via the Users API or by updating the list of administrators for a tenant via the Tenant API. • The permission corresponding to the specific operation. For example, the administrator must have the CreateUsers (13) permission to create a user account, or the DeleteUsers (16) permission to delete a user account. User provisioning scenarios The following topics describe a number of Hybrid Data Pipeline user provisioning scenarios. • Provisioning users with the Web UI on page 114 • Provisioning users with Hybrid Data Pipeline APIs on page 119 • Managing permissions with Hybrid Data Pipeline APIs on page 137 Provisioning users with the Web UI The Web UI can be used to provision and manage Hybrid Data Pipeline user accounts. The Web UI can also be used to create, view, modify, and delete roles, and, more generally, manage roles and the users associated with them. Depending on the role assigned and its associated permissions, users may have either direct access to the service or query-only access to data sources. The following topics provide instructions for provisioning users with the Web UI. (See also Using the Web UI on page 65.) • Create user accounts on page 115 • Update user accounts on page 115 • Delete user accounts on page 116 • Create roles on page 116 • Update roles on page 117 • Delete roles on page 117 • View data sources or data source groups on page 118 • Reset user account password on page 118 114 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1User provisioning Create user accounts Take the following steps to create a user account through the Web UI. 1. Navigate to the Manage Users view by clicking the manage users icon . 2. Click + New User. 3. Under the General tab, provide the following information. • Tenant. Select the tenant to which the user will belong. Only the tenants for which you have administrative access will appear. • User Name. Enter the name of the user. • Role. Assign a role for the user. A role must be assigned for the user. • Status. Specify the user''s status. The user can be active or inactive. 4. Under the Authentication Setup tab, specify the Authentication Type. • If you select Internal, you must specify a user name and password. • If you select an external authentication service, you must specify one or more users whose credentials are maintained by the service.This action associates the user with the Hybrid Data Pipeline user account. Note: See Authentication on page 148 for information on implementing authentication services. 5. Under the Limits tab, set limits for the user as desired. 6. Under the Tenant Admin Access tab, administrative access to tenants may be granted if desired. 7. Click Save. Results: The user has been created. The user will appear in the list of users in the Manage Users view for the given tenant. What to do next: Depending on the application environment, either of the following actions may be taken. • The direct-access user creates a data source through either the Web UI or the Data Sources API on page 1306. • The administrator creates a data source through either the Web UI or the Data Sources API on page 1306. The administrator then shares the data source with the user based on the rules and guidelines in Sharing data sources on page 1308. • The administrator creates a data source on behalf of the user as described in the Data Sources API on page 1306 and Managing resources on behalf of users on page 1310. Update user accounts Take the following steps to update a user account through the Web UI. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 115Chapter 2: Administering Hybrid Data Pipeline 1. Navigate to the Manage Users view by clicking the manage users icon . 2. Select the user account you want to update. 3. Click the Actions dropdown. Then select Edit. 4. Under the General tab, update user information as desired.. • Tenant. The tenant field displays the tenant to which the user belongs. However, you may not select another tenant in order to transfer the user. To transfer a user from one tenant to another, you must create a new user account in the tenant to which the user is moving. • User Name.You may edit the user name field. • Role.You may assign a different or additional role to the user. • Status.You may change the user''s status. The user can be active or inactive. 5. Under the Authentication Setup tab, update authentication information as desired. • Authentication Type. Specify the method of authentication the user must use to login. In addition to the internal authentication service, an administrator can integrate an external authentication service such as Active Directory. See Authentication on page 148 for details. • Password. Enter a new user password. See Password policy on page 164 for password requirements. • Confirm Password. Re-enter the new password. 6. Under the Limits tab, set limits for the user as desired. 7. Under the Tenant Admin Access tab, administrative access to tenants may be granted or removed. 8. Click Save. Results: The user has been updated. Delete user accounts Take the following steps to delete user account through the Web UI. 1. Navigate to the Manage Users view by clicking the manage users icon . 2. Select the user account(s) you want to delete. 3. Click the Actions dropdown. Then select Delete. 4. When prompted confirm that you wish to delete the user(s) by clicking Delete. Results: The user(s) has been delete. The user(s) is removed from the list of users in the Manage Users view for the given tenant. Create roles Take the following steps to create a role through the Web UI. 116 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1User provisioning 1. Navigate to the Manage Roles view by clicking the manage roles icon . 2. Click + New Role. 3. Provide the following information. • Tenant. Select the tenant for which the role is being created. Only the tenants for which you have administrative access will appear. • Role Name. Enter the name of the role. • Role Description. Enter a description of the role. • Permissions. Select the permissions associated with the role. See Permissions and default roles on page 61 for details. 4. Click Save. Results: The role has been created. The role will appear in the list of roles in the Manage Roles view for the given tenant. What to do next: You can now create users with this role, or assign this role to users. Update roles Take the following steps to update a role through the Web UI. 1. Navigate to the Manage Roles view by clicking the manage roles icon . 2. Select the role you want to update. 3. Click the Actions dropdown. Then select Edit. 4. Update the role as desired. • Tenant. The tenant field displays the tenant to which the role belongs. However, you may not select another tenant in order to transfer the role. A distinct role must be created for the other tenant. • Role Name.You may enter a new name for the role. • Role Description.You may enter a new description for the role. • Permissions.You may modify the permissions associated with the role. See Permissions and default roles on page 61 for details. 5. Click Save. Results: The role has been updated. The permissions for the users to whom the role was assigned are modified accordingly. Delete roles Take the following steps to delete roles through the Web UI. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 117Chapter 2: Administering Hybrid Data Pipeline 1. Navigate to the Manage Roles view by clicking the manage roles icon . 2. Select the role(s) you want to delete. 3. Click the Actions dropdown. Then select Delete. 4. When prompted confirm that you wish to delete the roles(s) by clicking Delete. Results: The role(s) has been delete. The role(s) is removed from the list of roles in the Manage Roles view for the given tenant. View data sources or data source groups In the tenants they administer, administrators can view a list of data sources owned by the users that reside in the tenant. Take the following steps to view data sources in the tenant. 1. Navigate to the Data Sources view by clicking the data sources icon . By default, a list of data sources owned by the administrator will be shown. 2. Specify whether you want to view the user''s data sources or the user''s data source groups. • Select the Data Sources tab to view the user''s data sources. • Select the Data Source Groups tab to view the user''s data source groups. 3. Select the user''s tenant and then the user''s name from the Select Tenant and Select User dropdowns. Results: A list of data sources or data source groups owned by the user is displayed. Reset user account password Take the following steps to reset a user account password through the Web UI. 1. Navigate to the Manage Users view by clicking the manage users icon . 2. Select the user account you want to update. 3. Click the Actions dropdown. Then select Edit. 4. Select the Authentication Setup tab. 5. Enter a password in the Password field. See Password policy on page 164 for password requirements. 6. Re-enter the password in the Confirm Password field. 7. Click Save. Results: The user password has been reset. 118 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1User provisioning Provisioning users with Hybrid Data Pipeline APIs Administrators can use Hybrid Data Pipeline APIs to provision users for access to Hybrid Data Pipeline. The Users API on page 1174 can be used to provision and manage Hybrid Data Pipeline user accounts.The Roles API on page 1140 can be used to create, view, modify, and delete roles, and, more generally, manage roles and the users associated with them. The following topics detail API operations for provisioning users in a number of scenarios. (See also the Hybrid Data Pipeline API reference on page 1065.) • Providing direct access on page 119 • Providing query-only access by sharing a data source on page 123 • Providing query-only access by creating data sources on behalf of users on page 126 • Providing limited direct access to data sources and features on page 130 • Providing query access to an ODBC data source and limited access to the Web UI on page 133 Providing direct access The following operations show how you can provision a direct-access user with Hybrid Data Pipeline APIs. • Creating a user account • Creating new role • Assigning new role • Setting permissions on a user account • Resetting user account password • Changing user account status • Deleting a user account Creating a user account The following operation creates a user account in tenant 26 with role 86. The administrator must have the Administrator (12) permission, or the CreateUsers (13) permission and administrative access on the tenant. Note: An administrator cannot create users that have tenant or elevated permissions unless the administrator also has those permissions. Request POST https://MyServer:8443/api/admin/users Request Payload { "userName": "testuser", "tenantId": 26, "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "password": "TempPassword", Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 119Chapter 2: Administering Hybrid Data Pipeline "passwordStatus": 1, "passwordExpiration": "2020-01-01 00:00:00" }, "permissions": { "roles": [ 86 ] } } Response Payload { "id": 31, "userName": "testuser", "tenantId": 26, "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "password": "TempPassword", "passwordStatus": 1, "passwordExpiration": "2020-01-01 00:00:00" }, "permissions": { "roles": [ 86 ] }, "authenticationInfo": { "authUsers": [ { "authUserName": "testuser", "authServiceId": 1 } ] } } Creating new role The following operation creates a new role in tenant 26. The administrator must have the Administrator (12) permission; or the administrator must have the CreateRole (17) permission, any permissions specified in the new role, and administrative access on the tenant. Request POST https://MyServer:8443/api/admin/roles Request Payload { "name": "odata_ds_role", "tenantId": 26, "description": "This role allows users to create and work with OData data sources.", "permissions": [ 1, 2, 3, 4, 7, 8, 9, 10, 11 120 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1User provisioning ], "users": [] } Response Payload { "id": 94, "name": "odata_ds_role", "tenantId": 26, "description": "This role allows users to create and work with OData data sources.", "permissions": [ 1, 2, 3, 4, 7, 8, 9, 10, 11 ], "users": [] } Assigning new role The following operation assigns the odata_ds_role to the testuser user account. The user account ID 31 is specified in the URL.The administrator must have the Administrator (12) permission; or the administrator must have the ModifyUsers (15) permission, any permissions specified in the new role, and administrative access on the tenant. Request PUT https://MyServer:8443/api/admin/users/31 Request Payload { "userName": "testuser", "tenantId": 26, "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "passwordStatus": 1, "passwordExpiration": "2025-01-01 00:00:00" }, "permissions": { "roles": [ 94 ] } } Response Payload { "userName": "testuser", "tenantId": 26, "statusInfo": { "status": 1, "accountLocked": false }, Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 121Chapter 2: Administering Hybrid Data Pipeline "passwordInfo": { "passwordStatus": 1, "passwordExpiration": "2025-01-01 00:00:00" }, "permissions": { "roles": [ 94 ] }, "authenticationInfo": { "authUsers": [ { "authUserName": "testuser", "authServiceId": 1 } ] } } Setting permissions on a user account The following operation shows how permissions can be set explicitly on a user account. In this example, the administrator retains the odata_ds_role for the user, but adds the UseDataSourceWithJDBC (5) permission. The administrator must have the Administrator (12) permission; or the administrator must have the ModifyUsers (15) permission, any permissions specified in the new role, and administrative access on the tenant. Request PUT https://MyServer:8443/api/admin/users/31/permissions Request Payload { "roles": [ 94 ], "permissions": [ 5 ] } Response Payload { "roles": [ 94 ], "permissions": [ 5 ] } Resetting user account password The following operation shows how to reset a user account password. Making this request changes the password and sets the passwordStatus to 2 (reset). The end user must change the password when he or she next logs in. Users can change their passwords either through the Web UI or through the User Details API. The administrator must have the Administrator (12) permission, or the ModifyUsers (15) permission and administrative access on the tenant. Request PUT https://MyServer:8443/api/admin/users/31/resetpassword 122 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1User provisioning Request Payload { "newPassword": "tempsecret" } Response Payload Status code: 204 No Content Changing user account status The following operation shows how to change user account status from active (1) to inactive (0). An inactive user cannot log in to the Web UI, use APIs, or establish JDBC, ODBC, or OData connections.The administrator must have the Administrator (12) permission, or the ModifyUsers (15) permission and administrative access on the tenant. Request PUT https://MyServer:8443/api/admin/users/31/statusinfo Request Payload { "status": 0 } Response Payload { "status": 0 } Deleting user account The following operation shows how to delete a user account. The user account ID 31 is specified in the URL. The administrator must have the Administrator (12) permission, or the DeleteUsers (16) permission and administrative access on the tenant. Request DELETE https://MyServer:8443/api/admin/users/31 Response Payload { "success":true } Providing query-only access by sharing a data source The following operations show the provisioning of a query-only user for ODBC access to a SQL Server database. The administrator begins by creating a role for the user account, creates a user account, creates a data source, and then shares the data source with the user account. Note: A data source can also be shared with a tenant, in effect sharing the data source with all the users in the tenant. See Sharing data sources on page 1308 for details. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 123Chapter 2: Administering Hybrid Data Pipeline • Create role for query-only access • Create user account • Create data source • Share data source Create role for query-only access The administrator begins by creating a role for query-only access with the following operation.The administrator must have the Administrator (12) permission, or the CreateRole (17) permission and administrative access on the tenant. Request POST https://MyServer:8443/api/admin/roles Request Payload { "name": "Query access", "tenantId": 59, "description": "This role permits only query access.", "permissions": [ 5, 6, 7 ], "users": [] } Response Payload { "id": 62, "name": "Query access", "tenantId": 59, "description": "This role permits only query access.", "permissions": [ 5, 6, 7 ], "users": [] } Create user account The administrator then provisions a user account with the "Query access" role. The administrator must have the Administrator (12) permission, or the CreateUsers (13) permission and administrative access on the tenant. Request POST https://MyServer:8443/api/admin/users Request Payload { "userName": "QueryOnlyUser", "tenantId": 59, "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { 124 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1User provisioning "password": "TempPassword", "passwordStatus": 1, "passwordExpiration": null }, "permissions": { "roles": [ 44 ] } } Response Payload { "id": 921, "userName": "QueryOnlyUser", "tenantId": 56, "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "passwordStatus": 1, "passwordExpiration": null }, "permissions": { "roles": [ 44 ] }, "authenticationInfo": { "authUsers": [ { "authUserName": "QueryOnlyUser", "authServiceId": 1 } ] } } Create a data source The administrator then creates a data source. The administrator will be the owner of this data source, but will share the data source with ODBCUser in the next operation. The administrator must have the Administrator (12) permission, or the MgmtAPI (11) and CreateDataSource (1) permissions. Request POST https://MyServer:8443/api/mgmt/datasources Request Payload { "name": "SQLServer2", "dataStore": "46", "connectionType": "Hybrid", "description": "Test SQL Server access", "options": { "Database": "CustomerData", "User": "MySQLServerUserId", "Password": "MySQLServerPassword" } } Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 125Chapter 2: Administering Hybrid Data Pipeline Response Payload { "id": "6334", "name": "SQLServer2", "dataStore": "46", "connectionType": "Hybrid", "description": "Test SQL Server access", "options": { "Database": "CustomerData", "User": "MySQLServerUserId", "Password": "MySQLServerPassword" } } Share a data source The administrator then shares the data source with the QueryOnlyUser. The administrator limits access to ODBC-only queries by setting the UseDataSourceWithODBC (6) permission on the data source. The data source ID 6334 is passed in the request URL, while the user ID 921 and the data source permission are passed in the request payload. The administrator must have the Administrator (12) permission; or the administrator must have the MgmtAPI (11) permission, the ModifyDataSource (3) permission, the UseDataSourceWithODBC (6) permission, and administrative access to the tenant to which the shared user belongs. Request POST https://MyServer:8443/api/mgmt/datasources/6334/sharedUsers Request Payload { "sharedUsers": [ { "userId": 921, "permissions": [ 6 ] } Response Payload Status code: 201 Successful response { "sharedUsers": [ { "userId": 921, "permissions": [ 6 ] } Providing query-only access by creating data sources on behalf of users The following operations show the provisioning of a query-only user for OData access to an Oracle database. The administrator begins by creating a role for the user account, next creates the user account, and then creates a data source on behalf of the user. (See also Managing resources on behalf of users on page 1310.) • Create role for OData query-only access 126 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1User provisioning • Create user account • Create a data source on behalf of the user account • Retrieve data source information on behalf of the user account • User queries the OData endpoint Create role for OData query-only access The administrator begins by creating a role for OData query-only access with the following operation. The administrator must have the Administrator (12) permission, or the CreateRole (17) permission and administrative access on the tenant. Request POST https://MyServer:8443/api/admin/roles Request Payload { "name": "OData query", "tenantId": 56, "description": "This role permits only OData query access.", "permissions": [ 7 ], "users": [] } Response Payload { "id": 21, "name": "OData-only Users", "tenantId": 56, "description": "This role permits only OData query access.", "permissions": [ 7 ], "users": [] } Create user account The administrator then provisions a user account with the "OData query" role. The administrator must have the Administrator (12) permission, or the CreateUsers (13) permission and administrative access on the tenant. Request POST https://MyServer:8443/api/admin/users Request Payload { "userName": "ODataUser", "tenantId": 56, "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "password": "TempPassword", "passwordStatus": 1, "passwordExpiration": null }, Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 127Chapter 2: Administering Hybrid Data Pipeline "permissions": { "roles": [ 21 ] } } Response Payload { "id": 921, "userName": "ODataUser", "tenantId": 56, "statusInfo": { "status": 1, "accountLocked": false }, "passwordInfo": { "passwordStatus": 1, "passwordExpiration": null }, "permissions": { "roles": [ 21 ] }, "authenticationInfo": { "authUsers": [ { "authUserName": "ODataUser", "authServiceId": 1 } ] } } Create a data source on behalf of the user account The administrator then creates a data source on behalf of ODataUser. Since the only permission associated with the assigned role is UseDataSourceWithOData (7), the user will be able to access data through this data source with OData queries, but will not be able to view data source information or access other Hybrid Data Pipeline features. The user query parameter (?user) is used to specify the owner of the data source. The administrator must have the Administrator (12) permission; or the administrator must have the MgmtAPI (11) permission, the OnBehalfOf (21) permission, administrative access on the tenant to which the user belongs, and the CreateDataSource (1) permission. Request POST https://MyServer:8443/api/mgmt/datasources?user=ODataUser Request Payload { "name": "Oracle_OData", "dataStore": 43, "connectionType": "Hybrid", "description": "", "options": { "User": "OracleTest", "Password": "Secret", "ODataSchemaMap": "{\"odata_mapping_v2\":{\"schemas\":[{\"name\":\"D2CQA01\", \"tables\":{\"Dept_Emp\":{},\"Employees\":{},\"Departments\":{},\"Salaries\":{}, \"Titles\":{},\"Dept_Manager\":{}}}]}}", "ServerName": "TestServer", 128 Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1User provisioning "ExtendedOptions": "EncryptionMethod=noEncryption", "SID": "UNI", "ODataVersion": "2" } } Response Payload { "id": "1681", "name": "Oracle_OData", "dataStore": 43, "connectionType": "Hybrid", "description": "", "options": { "User": "OracleTest", "Password": "Secret", "ODataSchemaMap": "{\"odata_mapping_v2\":{\"schemas\":[{\"name\":\"D2CQA01\", \"tables\":{\"Dept_Emp\":{},\"Employees\":{},\"Departments\":{},\"Salaries\":{}, \"Titles\":{},\"Dept_Manager\":{}}}]}}", "ServerName": "TestServer", "ExtendedOptions": "EncryptionMethod=noEncryption", "SID": "UNI", "ODataVersion": "2" } } Retrieve data source information on behalf of the user account The administrator can then retrieve data source details on behalf of ODataUser. The administrator must have the Administrator (12) permission; or the administrator must have the MgmtAPI (11) permission, the OnBehalfOf (21) permission, administrative access on the tenant to which the user belongs, and the ViewDataSource (2) permission. (Note that ODataUser cannot retrieve this information because the user does not have ViewDataSource (2) permission.) Request GET https://MyServer:8443/api/mgmt/datasources?user=ODataUser Response Payload { "id": "1681", "name": "Oracle_OData", "dataStore": 43, "connectionType": "Hybrid", "description": "", "options": { "User": "OracleTest", "Password": "Secret", "ODataSchemaMap": "{\"odata_mapping_v2\":{\"schemas\":[{\"name\":\"D2CQA01\", \"tables\":{\"Dept_Emp\":{},\"Employees\":{},\"Departments\":{},\"Salaries\":{}, \"Titles\":{},\"Dept_Manager\":{}}}]}}", "ServerName": "TestServer", "ExtendedOptions": "EncryptionMethod=noEncryption", "SID": "UNI", "ODataVersion": "2" } } User queries the OData endpoint With the appropriate connection information as supplied by the administrator, the ODataUser can now query the OData endpoint.With the following request, ODataUser retrieves an XML document from the Oracle_OData data source. Progress DataDirect Hybrid Data Pipeline: User''s Guide: Version 4.6.1 129Chapter 2: Administering Hybrid Data Pipeline Important: The new user must authenticate using basic authentication to execute API queries. Request GET https://MyServer:8443/api/odata/Oracle_OData/Employees Response Payload https://MyServer:8443/api/odata/Oracle_OData/Oracle_OData/Employees 2018-03-29T17:58:44Z https://MyServer:8443/api/odata/Oracle_OData/Employees(10001M)