Upgrading CKAN to a New Instance¶
The Official Instructions don’t have example code, so I wrote this doc. I am not doing the in-place
instructions.
You will want 3 terminals open - one logged into the old instance, one logged into the new instance, and one for your local machine.
# Old Machine
ssh -i {path/to/private_key} ec2-user@{old_instance_ip}
# New Machine
ssh -i {path/to/private_key} ec2-user@{new_instance_ip}
If you don’t have access to key-pair for the old or new machine, follow these instructions.
Restore databases from the old to the new instance¶
Export the
ckan_default
anddatastore_default
PostgreSQL databases using pg_dump. When prompted for the password, use the the postgres user’s password that was previously set. You can find this password in thempsotto.kdbx
KeePassXC database under theAWS CKAN
folder.pg_dump -U postgres -W -F t ckan_default > ~/ckan_default_dump.tar pg_dump -U postgres -W -F t datastore_default > ~/datastore_default_dump.tar
Copy the dump files from the old machine to your new machine (don’t forget to change the IP addresses and private keys below)
# RUN THE FOLLOWING COMMANDS FROM YOUR LOCAL MACHINE ################################################### # copy the files and directories from your old instance to your local machine scp -i {path/to/private_key} ec2-users@{old_instance_ip}:~/ckan_default_dump.tar ~/ scp -i {path/to/private_key} ec2-users@{old_instance_ip}:~/datastore_default_dump.tar ~/ # copy the files and directories from your local machine to your new machine scp -i {path/to/private_key} ~/ckan_default_dump.tar ec2-users@{new_instance_ip}:~/ scp -i {path/to/private_key} ~/datastore_default_dump.tar ec2-users@{new_instance_ip}:~/
Learn more about the
scp
command here.Drop and recreate the databases in the new machine
# enter postgres terminal sudo -u postgres psql
-- drop existing connections to ckan_default, drop the table, then recreate it SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'ckan_default' AND pid <> pg_backend_pid(); DROP DATABASE ckan_default; CREATE DATABASE ckan_default; -- drop existing connections to datastore_default, drop the table, then recreate it SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'datastore_default' AND pid <> pg_backend_pid(); DROP DATABASE datastore_default; CREATE DATABASE datastore_default; -- Hit Ctrl+Z to exit postgres terminal
Restore the databases from the old machine to the new machine using pg_restore.
# move dump files to root to make them easily findable sudo mv ~/ckan_default_dump.tar / sudo mv ~/datastore_default_dump.tar / # Change permissions to avoid permissions errors sudo chmod 777 /ckan_default_dump.tar sudo chmod 777 /datastore_default_dump.tar # restore the databases using pg_restore as the postgres user sudo -u postgres pg_restore --dbname=ckan_default --create --verbose /ckan_default_dump.tar sudo -u postgres pg_restore --dbname=datastore_default --create --verbose /datastore_default_dump.tar
Move the Filestore¶
In the old instance, if you have enabled the CKAN filestore, copy it’s directory to your home folder. (If you’re unsure, check
/etc/ckan/default/production.ini
for theckan.storage_path
variable to see if it is set. Copy that directory if it is present.)Copy the filestore to your new machine (don’t forget to change the IP addresses and private keys below)
# RUN THE FOLLOWING COMMAND FROM YOUR LOCAL MACHINE ################################################### # copy the files and directories from your old instance to your local machine scp -i {path/to/private_key} -r ec2-users@{old_instance_ip}:~/ckan ~/ # copy the files and directories from your local machine to your new machine scp -i {path/to/private_key} -r ~/ckan ec2-users@{new_instance_ip}:~/
Learn more about the
scp
command here.Extract the filestore archive to the same location on the new instance and set the
ckan.storage_path
value appropriately within the/etc/ckan/default/production.ini
file. If it is the same name, you don’t need to worry about changing the storage_path variable.# remove the existing filestore sudo rm -rf /var/shared_storage/ckan # replace with the old filestore sudo mv ~/ckan/ /var/shared_storage/
Set the ckan folder’s permissions to
apache:apache
(otherwise datasets won’t be able to be added)sudo -u root chown -R apache:apache /var/shared_storage/ckan/
Move the Promoted Image¶
In the old instance, copy the promoted image to your home directory.
sudo cp /lib/ckan/default/src/ckan/ckan/public/base/images/promoted-image.jpg ~/
Copy the promoted image to your new machine (don’t forget to change the IP addresses and private keys below)
# RUN THE FOLLOWING COMMAND FROM YOUR LOCAL MACHINE ################################################### # copy the files and directories from your old instance to your local machine scp -i {path/to/private_key} ec2-users@{old_instance_ip}:~/promoted-image.jpg ~/ # copy the files and directories from your local machine to your new machine scp -i {path/to/private_key} ~/promoted-image.jpg ec2-users@{new_instance_ip}:~/
Learn more about the
scp
command here.Move the promoted image to the images directory
sudo mv ~/promoted-image.jpg /lib/ckan/default/src/ckan/ckan/public/base/images/
Link to
promoted-image.jpg
in the html snippet containing the promoted image code.sudo nano /lib/ckan/default/src/ckan/ckan/templates/home/snippets/promoted.html
<!-- add the path to promoted-image.jpg --> <img src="{{ h.url_for_static('/base/images/promoted-image.jpg') }}" />
Move any installed extensions¶
Check your old and new ckan instances to see what extensions need to be transferred.
# anything with ckanext- in the directory name is an extension # Old Instance ################################################## ls /lib/ckan/default/src/ ... ckan ckanext-apihelper ckanext-harvest ckanext-pages # New Instance ################################################## ls /lib/ckan/default/src/ ... ckan ckanext-harvest ckanext-pages
Assuming your two directories looked like the two above, you would need to copy the
ckanext-apihelper
extension to your home folder. You will place this in the same spot in your new instance.sudo cp -R /lib/ckan/default/src/ckanext-apihelper ~/
Copy the extensions to your new machine (don’t forget to change the IP addresses and private keys below)
# RUN THE FOLLOWING COMMAND FROM YOUR LOCAL MACHINE ################################################### # copy the files and directories from your old instance to your local machine scp -i {path/to/private_key} -r ec2-users@{old_instance_ip}:~/ckanext-{extension_name} ~/ # copy the files and directories from your local machine to your new machine scp -i {path/to/private_key} -r ~/ckanext-{extension_name} ec2-users@{new_instance_ip}:~/
Learn more about the
scp
command here.Move the extensions to the proper directory
sudo mv -R ~/ckanext-{extension_name} /lib/ckan/default/src/
add the extension name to the end of the plugins variable in the CKAN config file
sudo nano /etc/ckan/default/production.ini
ckan.plugins = stats text_view image_view recline_view datastore datapusher resource_proxy geo_view pages harvest ckan_harvester apihelper
Final Steps¶
Perform a Solr re-index so the new Solr service contains records of the newly imported data. Use the commands below:
# activate the python virtualenv and go to the ckan directory . /usr/lib/ckan/default/bin/activate cd /usr/lib/ckan/default/src/ckan # rebuild the index for the database paster --plugin=ckan search-index rebuild --config=/etc/ckan/default/production.ini
Disassociate the Elastic IP Address from the old instance and Associate it with the new instance.
View your running instances on AWS. Remember the Elastic IP address (copy and paste it somewhere so you don’t forget it).
Select the old instance, click the
Actions
dropdown, clickNetworking
, then clickDisassociate Elastic IP address
.Select
Elastic IPs
from the sidebar menu underNetworking
.Select the same IP Address you just Disassociated from the old instance, click the
Actions
dropdown, then clickAssociate Elastic IP address
.Select
Instance
, then click into the instance input field and select the new instance.Click into the Private IP input field and select the Private IP of the instance (there will likely only be one option).
Click
Associate
.
Follow these instructions to configure HTTPS on the server.
Restart the Nginx and Apache servers if you didn’t already in the HTTPS setup.
sudo systemctl restart nginx sudo systemctl restart httpd
Check that data was transferred correctly¶
[ ] Check that the promoted-image on the main page is present.
[ ] Check that all ckan users are still present.
[ ] Check that all ckan datasets are still present.
Setup CodeDeploy¶
First, you will need to move the
production.ini
to the main CKAN codebase directory and create a symlink to it in the original directory. This is so that you will be able to directly edit the production.ini without having to ssh into the EC2 instance.sudo mv /etc/ckan/default/production.ini /lib/ckan/default/src/ckan/ sudo ln -s /lib/ckan/default/src/ckan/production.ini /etc/ckan/default/production.ini
The rest of the CodeDeploy instructions will come at a later date (when we finish the setup). You can view this Github Issue for details on progress.