Operational Tech Notes
- Craig Appl (Unlicensed)
- Ben Leibert
This page provides technical notes on the process we went through to setup the technical systems.
Setting up the Nifi Truststore to be able to access DHIS2
The DHIS2 server's URL is https://www.dhis2.org.mz, but their SSL certificate has the baseUrl set to dhis2.org.mz (without the www). Nifi sees this difference and throws an error when trying to run an InvokeHttp processor to authenticate with DHIS2. In order to get around this, we had to create a StandardSSLContextService that uses a local certificate truststore that's on the Nifi server. The following code was used to create a truststore on the server with the DHIS2 Mozambique server.
The TLS version was found by accessing: https://www.ssllabs.com/ssltest/analyze.html?d=www.dhis2.org.mz&latest
Notes on Trusted Store - Downloaded the dhis2 Mozambique certificate and stored it in a file named dhis2moz.pem openssl s_client -showcerts -servername www.dhis2.org.mz -connect www.dhis2.org.mz:443 < /dev/null | sed -n -e '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p' > dhis2moz.pem - Verified I could access the dhis2 server with curl referencing that certificate curl --cacert dhis2moz.pem -u USERNAME:PASSWORD https://www.dhis2.org.mz/prod/api/28/me - Created a store file in the server cd /opt/nifi/nifi-1.5.0/conf sudo keytool -importcert -keystore cacerts.jks -alias CA-cert -storepass CREATEPASSWORD -file ~/dhis2moz.pem sudo chown nifi:nifi cacerts.jks - Got into nifi and created a StandardSSLContextService properties: Truststore Filename: /opt/nifi/nifi-1.5.0/conf/cacerts.jks Truststore Type: JKS TLS Protocol: TLSv1 - Saved and enabled the service
Setting up an automatic SSH connection to SELV's host machine
SELV’s instance of PostgreSQL is not configured to allow for external connections. Instead, SELV’s host provides an SSH server through which external clients may establish a tunnel. By tunneling an arbitrary local port to localhost:5432 on SELV, clients may access its instance of PostgreSQL via SSH. The following steps describe how this was setup on the VAN-Dashboard's server.
#Install autossh
sudo add-apt-repository ppa:eugenesan/ppa
sudo apt-get update
sudo apt-get install autossh -y
#Retrieve SELV's key
Save this file to /home/ubuntu/.ssh
chmod 400 ~/.ssh/selv
#Manually run SSH to test it and accept the server’s RSA fingerprint
ssh -i ~/.ssh/selv -L 5431:localhost:5432 ec2-user@selv.villagereach.org
exit
#Define our service and its environment variables
sudo vim /etc/systemd/system/selv-dashboard-autossh.service
Add the content of selv-dashboard-autossh.service to the above file.
sudo vim /etc/default/selv-dashboard-autossh
Add the content of selv-dashboard-autossh to the above file.
#Reload service, set it to run at startup (“enable” it) and then start it
sudo systemctl daemon-reload
sudo systemctl enable selv-dashboard-autossh.service
sudo systemctl start selv-dashboard-autossh.service
#Verify that, because there's an SSH connection to SELV's host machine, the following command prompts you for a password
psql -h 127.0.0.1 -p 5431 -d open_lmis -U olreporter
#Verify that the above still works after a reboot
sudo reboot
psql -h 127.0.0.1 -p 5431 -d open_lmis -U olreporter
Ensuring that nginx launches when Ubuntu starts
sudo systemctl enable nginx
Creating a database for PAV's Data
psql -d superset -U postgres CREATE DATABASE pavdata WITH OWNER = superset;
After executing the above commands, run initialize_pav_database.sql
Installing a local NexLeaf Server
#Install NodeJS sudo curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash - sudo apt-get install nodejs #Retrieve and install the webapp sudo git clone https://github.com/villagereach/van-dashboard-nexleaf.git sudo mv ./van-dashboard-nexleaf /opt/local_nexleaf_server cd /opt/local_nexleaf_server sudo npm install #Note that it will eventually be necessary to remove the test images in /opt/local_nexleaf_server/public/images/nexleaf_data rm /opt/local_nexleaf_server/public/images/nexleaf_data/*.png #To launch the webapp manually to troubleshoot #node index.js
Configuring the NexLeaf to Launch when Ubuntu Starts
Copy this file to /lib/systemd/system/local-nexleaf-server.service and then run the following commands.
#Reload the systemctl daemon and tell it to launch local-nexleaf-server upon system-start sudo systemctl daemon-reload sudo systemctl enable local-nexleaf-server #Note that, to troubleshoot, you can manually start/stop/check/disable the service as follows sudo systemctl stop local-nexleaf-server sudo systemctl start local-nexleaf-server sudo systemctl status local-nexleaf-server sudo systemctl disable local-nexleaf-server
Installing the Admin-UI App Server
Copy the files within van-dashboard-admin-app to /opt/settings-server/
Move /opt/settings-server/support_files/van-dashboard-admin.service to the /lib/systemd/system directory.
#Install the nodejs app cd /opt/settings-server/ npm install #Install the service for it sudo systemctl daemon-reload sudo systemctl enable van-dashboard-admin
Installing DHIS-2 ("SIS-MA") Report Parsing Scripts
Copy the dhis2_html_parser directory along with its contents to /opt and then run:
npm install
Installing Database Backup Scripts
Please note that files referenced below are stored in our repository's database_backup_scripts directory.
sudo mkdir -p /var/lib/pgsql/bin sudo chown ubuntu /var/lib/pgsql # Copy aws_s3_sync.sh, pg_backup.config, and pg_backup_rotated.sh into /var/lib/pgsql/bin/ chmod +x /var/lib/pgsql/bin/pg_backup_rotated.sh chmod +x /var/lib/pgsql/bin/aws_s3_sync.sh pip install awscli mkdir ~/.aws # Copy the file called "credentials" to ~/.aws chown ubuntu ~/.aws/ chown ubuntu ~/.aws/credentials #Add the following to the crontab for the ubuntu user: 5 21 * * * /var/lib/pgsql/bin/aws_s3_sync.sh #Note that NiFi is expected to call /var/lib/pgsql/bin/pg_backup_rotated.sh itself prior to changing database state.