Upgrade V4.0 to V5.0
This page explains how to transfer the data of your existing V4.0 ]project-open[ installation to the new version 5.0, smoothly, without loss of data and without the need to import, export or convert. The process basically consists of three steps:
- Install a new ]po[ V5.0 instance on a separate computer.
- Copy a database backup of your current system to the new V5.0 system.
- Run a number of "upgrade scripts" on the new V5.0 system in order to update the data-model of your current system.
The instructions below explain the details.
Please note:
- This procedure assume that you did not customize the TCL source code of your current installation.
- Configuration changes (performed via ]po[ administration screens) are OK and will be preserved.
- In any case, please [don't hesitate to contact] us for a quotation to update your server.
Major Release Upgrade
]po[ V5.0 is a "major" release that comes with new infrastructure components:
- Application Framework:
OpenACS 5.9 instead of OpenACS 5.7
- Application Server:
NaviServer 4.99.8 instead of AOLserver 4.5.1
- Database:
PostgreSQL 9.2 instead of PostgreSQL 8.4 or lower.
PostgreSQL 8.4 is not supported anymore.
- Operating System:
CentOS 7 instead of CentOS 6.4 and Windows 10 and Windows Server 2008 or higher instead of Windows XP, 7, 8, and Windows Server 2003.
There are a number of incompatibilities between the V4.0 and the new V5.0 infrastructure. These instructions have been designed to work around these problems.
Installation Preconditions
Required Hardware
You will need an additional server (virtual or physical) during the upgrade process:
Please see the benchmark section for the sizing of your new ]po[ server. The resources below are required for a small production installation with 30 "active users" (PMs and other employees actively working on projects and logging hours) and a few hundred "occasional users" (customers or externals) consulting their projects every few days.
- CPU: 4 cores of a recent AMD or Intel CPU
- Memory: >=8GB RAM
- Hard Disk: >=50GB space of a RAID1 partition with fast (>=10.000rpm or SSD) disks
On this hardware please install a standard "vanilla" ]po[ V5.0 system (the one with the "Tigerpond" demo-company).
]po[ V4.0 or V4.1 Installation
This document covers the upgrade from ]po[ V4.0 and V4.1. If you are still running on ]po[ 3.5 or older, you will first have to upgrade to V4.0. Please see V3.5 -> V4.0 instructions for details. Direct upgrades from V3.5 or earlier -> V5.0 are not supported and will not work.
"Upgrade Scripts"
"Upgrade scripts" are small pieces of SQL that modify the data-model of an existing old ]po[ installation in order to work with a newer ]po[version . Whenever the ]po[ core team modifies the application source code ("TCL" code), we create these upgrade scripts in order to modify the data-model in the database ("SQL" code"). This way, the versions of TCL and SQL stay in sync.
Up until 2015, we have provided these upgrade scripts together with the TCL code, so that every customer could upgrade their ]po[ installation to the latest version at any moment. We have changed this policy now with ]po[ V5.0 and provide the upgrade scripts more restrictively:
- Open-source developers, partners and other active ]po[ collaborators have already been notified and enjoy free access to the upgrade scripts.
- Customers with Basic and Silver support contracts have access to the upgrade scripts.
- Customers with Gold or Enterprise support don't need the upgrade scripts, because the ]po[ core team will upgrade their servers once ]po[ V5.0 "final" has been released.
Please
contact us if you don't belong to one of the groups above.
Download Additional Packages
Your ]po[ V4.x system may include "packages" that are not part of the default ]po[ V5.0 distribution anymore:
- Look into the ~/projop/packages/ directory of your V4.0 server
- Look into the ~/projop/packages/ directory of the new V5.0 server
- Identify any package available in V4.0 that is not included in V5.0
- For each package please execute:
# cvs checkout <package_name>
This will get you the latest version ("CVS HEAD") of this package. This should be OK in most cases, because the excluded packages usually change very slowly. In case of issues please use the SourceForge "Open Discussions" forum.
- Restart your ]po[ server (very important)
Please note:
- On Windows please check the permissions of the new packages. Please make sure the new packages are accessible by the AOLserver process.
- For more information on CVS checkout please see our developer CVS checkout page.
- After the checkout please compare the list of packages in your V4.0 server and your V5.0 server as a double-check.
- With the next reboot of your system please check the ~/log/error.log file for lines containing the string "Error:"
Upgrade Process - Step by Step
Copy Your V4.0 Database to Your V5.0 Server
In this section we will copy the database of your existing system to the new server that you have set up as explained above:
Create a V4.0 backup of your old server
- You can use the Admin -> Backup functionality in ]po[ V4.0 in order to create a backup of your old server and to download it to your desktop computer.
- You can also take an existing backup dump from /web/projop/filestorage/backup or /var/backup (Linux) or C:/project-open/servers/projop/filestorage/backup/ (Windows).
Load the V4.0 backup dump into the database of your V5.0 server
- Copy the V4.0 backup dump into /web/projop/filestorage/backup (Linux) or C:/project-open/servers/projop/filestorage/backup/ (Windows) of the new server. You can use the Admin -> Backup -> "Upload Backup" functionality or just manually copy the file.
- Now reload the Admin -> Backup page in order for the new file to appear.
- You can use the "Restore" button to load the dump into the database.
Most ]po[ V5.0 pages will continue to work with a ]po[ V4.0 data-model, particularly the administration pages that we will use in the following steps to finish the upgrade.
Please study the output of the import process in detail:
- A normal ]po[ backup contains a section in the beginning which deletes previously existing data structures. This section may cause some ~2.000 error messages which you can ignore.
- There may be issues related to a different implementation of the TSearch2 full-text search engine. You can safely ignore messages that are somehow related to "tsearch2" or the im_search_objects table (which can be very long).
A good way to check the integrity of the import is to compare the number of entries per table using pgAdminIII (there is a statistics section in the list of tables).
Please see the PostgreSQL 9.2 documentation for details about the PostgreSQL database.
Download and Install Upgrade Scripts
As part of your support contract, you have received a CVS user and password. Please use these credentials in order to download the upgrade scripts.
On the new V5.0 system please execute the following shell commands:
# su - projop (login as user "projop")
# cd ~/packages/
# cvs -d :pserver:cvs-user@cvs.project-open.net:/home/cvsroot login (... and provide password)
# cvs -d :pserver:cvs-user@cvs.project-open.net:/home/cvsroot checkout upgrade-4.0-4.1
# perl ~/packages/upgrade-4.0-4.1/install-upgrades.perl
# cvs -d :pserver:cvs-user@cvs.project-open.net:/home/cvsroot checkout upgrade-4.1-5.0
# perl ~/packages/upgrade-4.1-5.0/install-upgrades.perl
# chmod -R go=u ~/packages
# chmod -R go-w ~/packages
Please replace "cvs-user" with the CVS user name assigned to you or your company. Please
contact us if you don't have a CVS user.
AMS, News, ETP and other Cleanup
This cleanup removes outdated functionality that isn't used anymore by ]po[, but which will cause problems during the upgrade process further down. The "AMS" (=Attribute Management System) is an OpenACS packages that is not used by ]project-open[. However, some of it's data-structures still exist in the database, probably due to an error years ago when installing and removing AMS. These AMS data-structures will cause errors when updating the OpenACS framework to OpenACS 5.9 further below. A similar situation exists with the "News" and "ETP" packages.
In order to cleanly remove the AMS structures you need to execute a SQL script manually. Please open up a shell in your ]po[ system and enter (in this specific order):
# su - projop (login as user "projop")
# psql -f ~/packages/intranet-core/sql/postgresql/upgrade/cleanup-tsearch2.sql
# psql -f ~/packages/intranet-core/sql/postgresql/upgrade/cleanup-ams.sql
# psql -f ~/packages/intranet-core/sql/postgresql/upgrade/cleanup-bug-tracker.sql
# psql -f ~/packages/intranet-core/sql/postgresql/upgrade/cleanup-etp.sql
# psql -f ~/packages/intranet-core/sql/postgresql/upgrade/cleanup-events.sql
# psql -f ~/packages/intranet-core/sql/postgresql/upgrade/cleanup-file-storage.sql (ToDo: content folder maybe not deleted?)
# psql -f ~/packages/intranet-core/sql/postgresql/upgrade/cleanup-news.sql
# psql -f ~/packages/intranet-core/sql/postgresql/upgrade/cleanup-postgresql-92.sql
You should see several lines of output talking about deleted sequences or views etc.
There should be no error during this process (lines including "Error:"). Please contact us if errors did occur, including the full output of the script that produced the error.
Uninstall Obsolete Packages
Next please go to http://<your_server>/acs-admin/apm and uninstall the following packages:
- "intranet-bug-tracker" (helper package for bug-tracker)
- "bug-tracker" (please migrate any bugs to intranet-helpdesk)
- "file-storage" (has never been used as part of ]po[) (ToDo: produces an error in po50dev)
- "intranet-sencha" (has been replaced by "sencha-core" and "sencha-extjs-v421"
In order to remove these packages please:
- Please verify that you have restarted your server after the "Download Additional Packages" step above.
- Please see the section "Download Additional Packages" above and verify that these packages are available as sub-directories in your new V5.0 system below /packages/. We will need the code to uninstall them cleanly.
- Go to http://<your_server>/acs-admin/apm/, click on "intranet-bug-tracker" -> "Uninstall this package from your system" -> Uncheck the check-box in front of the drop-script -> "Delete Package". Then reload /acs-admin/apm/ and check that the package isn't listed anymore.
- Please proceed as in 3. with "bug-tracker".
- Please proceed as in 3. with "file-storage". There will be a warning that "attachments" and "xowiki" depend on "file-storage", but you can ignore this. ToDo: Error with content-folder already existing.
Upgrade OpenACS and ]project-open[
OpenACS is the application framework below ]project-open[. You have to upgrade OpenACS before upgrading any ]po[ packages. You will have to upgrade OpenACS in several "batches" and restart the server after each batch:
Batch 1: acs-kernel
- Go to the package update page at: http://<your_server>/acs-admin/apm/packages-install?operation=upgrade
- Please select only the package "acs-kernel" and press the "Next ->" button.
- The page will show a list of 20 upgrade-*-*.sql files. Please press "Install Packages" to execute all of them.
- The system will now perform upgrade operations. There should be no error messages.
- Click on "Click here to restart the server now" at the bottom of the page.
Batch 2: acs-content-repository
- Please go to http://<your_server>/acs-admin/apm/packages-install?operation=upgrade and use Ctrl-F5 or Ctrl-<Reload-Button> in your browser to force a reload of the page.
- Please select only the package "content-repository" and press "Next ->".
- The next page will show three "Additional Packages Automatically Selected" with green "Dependency satisfied" messages. Please press "Select Data Model Scripts".
- The page will show a list of selected upgrade-*-*.sql files. Please press "Install Packages".
- Click on "Click here to restart the server now" at the bottom of the page.
Please note:
- The "normal F5" key is sometimes not sufficient to reload the package update page and will lead to errors in the following pages.
Please use Ctrl-F5 in this case, holding down the Ctrl key while pressing F5.
Batch 3: acs-subsite
- Please proceed as above, selecting only the package "acs-subsite". This will select 10 "Additional Packages Automatically Added" as above.
Batch 3: All other acs-* packages
- Please proceed as above, selecting all acs-* packages.
Batch 4: calendar, categories, general-comments, notifications and rss-support
Batch 5: Reinstall file-storage
- Please go to http://<your_server>/acs-admin/apm/packages-install?operation=install
- Select "file-storage" and proceed as above.
- ToDo 2016-01-12: Currently there is a bug that the fs_urls_full view was already there / not removed during cleanup
Batch 6: attachments, auth-ldap-adldapsearch, xotcl-core and xowiki
- Please go to http://<your_server>/acs-admin/apm/packages-install?operation=upgrade
- Please proceed as above.
- There may be an error during the installation of XoWiki ("Package enabled. Error: ...", multiple lines of message), which you can ignore.
Batch 7: intranet-core and intranet-dynfield
Batch 7a: Script ~/packages/intranet-search-pg/sql/postgresql/intranet-search-pg-create.sql in the console.
su - projop
psql -f ~/packages/intranet-search-pg/sql/postgresql/intranet-search-pg-create.sql
Batch 8: Install all other intranet-* packages
Execute remaining upgrade scripts
After a restart please visit the "Home" page and observe a grey box at the right with "Upgrade Information".
- Please click on "Run Upgrade Scripts".
- Please reload the page using Ctrl-F5 and repeat until the "Upgrade Information" box has disappeared.
After yet another restart you should have a perfect ]po[ V5.0 server running.
Go to /intranet/admin/backup and create a new backup.
Additional Tasks
DynFields without "layout page"
Please go to Admin -> DynField and check for any warning about DynFields without "Pos-Y" value and follow the instructions
Dealing with Upgrade Errors
The upgrade process consists of adapting your V4.x data-model to the new V5.0 code. These changes are performed by a number of upgrade scripts that perform actions including adding new columns to tables, removing columns, modifying master data etc. The upgrade scripts have been tested with standard "vanilla" ]po[ systems, so that no errors should occur.
However, changes in the data-model performed by customers can not be anticipated by upgrade scripts, and any such change may lead to an error during the upgrade process. These errors tend to be relatively easy to diagnose and fix, because each error contains a reference to the upgrade script and the line number and contains an error message that can be frequently interpreted with common sense and some knowledge of SQL.
- Please send us an email with the full error protocol, if you have signed a support contract covering upgrades.
- Please post your error message to the "Open Discussions" forum on SourceForge.net otherwise.
Customized /packages/ Folder
Please contact us if you should have customized your V4.x ]project-open[ code. There are three options for dealing with customisations:
- Create a "diff" and apply the diff to V5.0:
We can determine the "diff" (=difference) of what you have changed with respect to your V4.0 base line. In a second step we can apply the diff to V5.0. This procedure will work in about 80% of all cases, while the remaining 20% require manual "conflict resolution"
- Move your custom code in a customer specific package:
We may be able to copy your modified pages into a specific package for you ("customer specific package") and modify the GUI linking in your system so that the new code will be used. This way your custom code will survive also the next upgrade.
- Integrate your code into the "product":
We may include your code in the next V5.1 release if your extensions could be useful for other customers.
Quality Control
Here are a few checks in order to determine if your new system is really working.
- Edit an existing project/ create a new one
- Edit an existing company/ create a new one
- Edit an existing user/ create a new one
- Enter a search query
- Perform a backup and restore a backup
- XoWiki: Go to /xowiki/ and click on "New Page" at the top with Name = "test", Section="" (empty), Page Title= "Test" and Content = "This is a test".
- File Storage: Go to /file-storage/ create a new file and perform some other operations
ToDo
- Install V5.0 specific packages:
- intranet-gantt-editor
- intranet-portfolio-planner
- senchatouch-timesheet(?)
- XoWiki produces an error in one includelet
References