Migration and Upgrade
特別是缺乏維護的擴充模組，通常只能做到安裝成功，卻不處理乾淨移除的問題。新版 Plone6 在 2017 年初起步，已經能在 Zope4 和 ZODB5 環境執行，雖然不是完全跳船到 Python3，但目標是先清除 Python3 的障礙。
準備工作
先閱讀昇級指南與相關文件，動手之前，有完整備份系統和資料，才能無後顧之憂。執行 pack 並移除舊的模組，讓影響因素降到最低。
Python 2.6 -> Python 2.7 可採 Steve McMahon 推薦的保守方式逐步確認
使用 bin/snapshotbackup 指令
plone.app.linkintegrity Event Handler Patching (added in Plone 5.1 rc2)
Plone4 之前的環境要了解 Python Script 和 External Method，必讀的文件包括 Command-line interaction and scripting 和 Upgrade Guide。Plone Conf 2014 提及 PLIP #13283 標示許多 plone.app.* 套件將併進 Plone5。Plone3 昇到 Plone4 可能遇到 Catalog Brain 錯誤狀況，利用 collective.catalogcleanup 來消除已不存在的內容。
Move ZMI / DTML configuration form into a Plone page, Move CMFFormController to BrowserViews
Products.PageTemplates 成為 Zope2 內建模組 five.pt 逐步被取消
skins/plone_scripts 在 Plone5 不再使用並被移除，像 displayContentsTab 在 Plone5.1 被移除，Products.TinyMCE 不再是 Plone5 的相依模組。像 toggleSelect 被移除了，需要改用新模組。
refactor buildout.cfg: collective.googlenews
Plone5
Plone5 將 InterfaceTool 移除，導致 getDottedName 無法呼叫，像 plone.app.layout 需要調整 import 方式。將 portal_calendar 移除，宣告 Event-like 型別的方式要調整。
In Plone 5.1, IFactoryTool is no longer defined in CMFPlone, so we need to handle when it is missing.
registry.xml 運作方式調整，JavaScript 和 CSS 採新的註冊格式: cioppino.twothumbs Resource Bundle eea.facetednavigation Widget from Archetypes to z3c.form
TypeError: argument of type 'NoneType' is not iterable
Importing Plone 4.3 Site Fails: No Module Named CMFDefault.DiscussionItem
Plone4
Plone 4.0.x 可能會遇到 Python lib_dirs 錯誤造成 zlib 找不到。
Plone 4.2 -> 4.3 getSite, setSite : from zope.app.component.hooks or zope.site.hooks to zope.component.hooks
Unresolved Dependencies Error: 本身不會造成損害
def fix_plonepas_dependencies(setup_tool):
"""Remove 'plonepas-content' as a dependency for 'plonepas' GS profile."""
from Products.GenericSetup.registry import _import_step_registry
for registry in setup_tool.getImportStepRegistry(), _import_step_registry:
registry._registered.get('plonepas')['dependencies'] = \
(u'componentregistry', u'memberdata-properties', u'rolemap', u'controlpanel')
examples: collective.dancing
早期常用 Products.contentmigration 來執行 Archetypes 的昇級，可參考 FalconImageMigrator plone.app.blob 的 migrations.py，Products.CMFPlone 的 MigrationTool.py 和 tests/testMigrationTool.py 範例，同時 Archetypes 也附有昇級範例，新的工具包括 Script 或 Framework，像是 collective.blueprint.jsonmigrator、zopyx.plone.migration、rt.atmigrator，但 api.plone.org 這類 API 並不成熟。改用 plone.api 並更新相依關係: collective.watcherlisr
Framework Team 提議 Core Package 不使用 plone.api
- search catalog for all items of the particular portal_type;
- get the object for each brain in the result;
- check that the class matches expectations, if not, continue;
- Create a replacement item; copy the object state from the old version of the item to the new one using obj.__dict__() on the source, and setattr for each key/value (follwed by setting _p_changed attribute to True).
- Reindex catalog;
- Commit transaction.
事後檢查
plone_deprecated sitemap
plone.app.upgrade 已把 Plone 2.5 Plone 3 昇級程式碼移除
Deprecation
Editor and HTML Filtering
Issues on Upgrade to Plone 4.3
Schema Class 從 plone.directives.form.schema 改到 plone.supermodel.model，可能造成 PicklingError。可考慮搭配的方式，包括 wildcard.fixmissing 和移除 Marker Interface。
KSS 不再是核心模組，圖示使用 PNG 取代 GIF。
KSS Removal: kss_field_decorator_view FolderContentsKSSView vs FolderContentsBrowserView
UserWarning: DirectoryView archetypes_kss refers to a non-existing path 'Products.Archetypes:skins/archetypes_kss'
UserWarning: DirectoryView plone_kss refers to a non-existing path 'Products.CMFPlone:skins/plone_kss'
('Topic', '') ValueError: unknown meta_type 當 plone.app.contenttypes 昇級後，Archetypes 的 Topic 不再存在，部份舊式模組造成錯誤，利用 'deprecated="True"' 來處理。範例: Solgema.fullcalendar
<object name="Topic" deprecated="True" />
Plone 5 Upgrade Without UUID Links Configlet 介面和圖示更新 plone.app.caching 的安裝與隱藏
# migrate-to-uuid.py
from AccessControl.SecurityManagement import newSecurityManager
from Acquisition import aq_parent
from Products.Archetypes.Field import TextField
from Products.CMFPlone.interfaces.siteroot import IPloneSiteRoot
from lxml.html import fromstring
from lxml.html import tostring
from plone.app.textfield.value import RichTextValue
from plone.dexterity.interfaces import IDexterityContent
from plone.uuid.interfaces import IUUID
import transaction
def getUID(ob):
try:
return IUUID(ob)
except TypeError:
return ob.UID()
def findObject(from_obj, path):
while not IPloneSiteRoot.providedBy(from_obj):
found = from_obj.restrictedTraverse(path, None)
if found:
return found
from_obj = aq_parent(from_obj)
def transform_links_to_uuid(ob, html):
if not html:
return html
changes = False
try:
dom = fromstring(html)
except:
print('Error parsing DOM from content: %s' % '/'.join(ob.getPhysicalPath()))
return html
for el in dom.cssselect('a'):
href = el.attrib.get('href', '')
if (not href or href.startswith('http://') or
href.startswith('https://') or href.startswith('/') or
'resolveuid' in href):
continue
link_obj = findObject(ob, href)
if link_obj:
uuid = getUID(link_obj)
el.attrib.update({
'href': 'resolveuid/%s' % uuid,
'data-linktype': 'internal',
'data-val': uuid
})
changes = True
for el in dom.cssselect('img'):
src = el.attrib.get('src', '')
if (not src or src.startswith('http://') or
src.startswith('https://') or src.startswith('/') or
'resolveuid' in src):
continue
src, _, scale = src.partition('/@@images/image/')
if not scale:
src, _, scale = src.partition('/image_')
img_obj = findObject(ob, src)
if img_obj:
uuid = getUID(img_obj)
new_href = 'resolveuid/%s' % uuid
if scale:
new_href += '/@@images/image/' + scale
attribs = {
'src': new_href,
'data-linktype': 'image',
'data-val': uuid
}
if scale:
attribs['data-scale'] = scale
el.attrib.update(attribs)
changes = True
if changes:
return tostring(dom)
else:
return html
def migrate_dexterity_to_uuid_links(ob):
try:
orig = ob.text.raw
new = transform_links_to_uuid(ob, orig)
if new != orig:
ob.text = RichTextValue(
new, mimeType=ob.text.mimeType, outputMimeType=ob.text.outputMimeType)
print('Fixed content not using UUID links: %s' % '/'.join(ob.getPhysicalPath()))
return 1
except AttributeError:
pass
return 0
def migrate_archetypes_to_uuid_links(ob):
for field in ob.Schema().fields():
if type(field) == TextField and field.getContentType(ob) == 'text/html':
orig = field.getRaw(ob)
new = transform_links_to_uuid(ob, orig)
if orig != new:
field.set(ob, new)
print('Fixed content not using UUID links: %s' % '/'.join(ob.getPhysicalPath()))
return 1
return 0
def migrate_to_uuid_links(site):
catalog = site.portal_catalog
count = 0
for brain in catalog():
ob = brain.getObject()
if IDexterityContent.providedBy(ob):
count += migrate_dexterity_to_uuid_links(ob)
else:
count += migrate_archetypes_to_uuid_links(ob)
if count % 100 == 0:
transaction.commit()
print('finished processing %i' % count)
user = app.acl_users.getUser('admin') # noqa
newSecurityManager(None, user.__of__(app.acl_users)) # noqa
for oid in app.objectIds(): # noqa
_obj = app[oid] # noqa
if IPloneSiteRoot.providedBy(_obj):
migrate_to_uuid_links(_obj)
transaction.commit()
ZEXP export/import
AttributeError: uid_catalog 直接從 Plone4 匯出再進 Plone5 時會發生
避免用 Copy + Paste 方式複製 Plone，通常 catalog 會索引重複的項目，工作流程會使用初始狀態，UID 會重新產生。用 Cut + Paste 更糟，會在新項目建立 Reference，但被參照的項目卻不存在了。
Migrating Zope to .Net Apps: Access ZODB in .Net ?
Setting the right permissions on your blobstorage directory
Remove Broken Transforms XML parse
AttributeError: displayContentsTab
intid 維護管理
Running Old Instance on Another Port
bin/plonectl path
bin/instance path
bin/buildout path
parts/instance/etc/zope.conf path and port
Archetypes to Dexterity
存取 Field 的差異: 前者使用 obj.getField('fieldname').get(obj) 或 obj.getFieldname() 之類的方式，後者使用 obj.fieldname 或 __getattr__ 來取得預設值。
archetypes.schemaextender 主要步驟
ATContentTypes Walkers and Migrators
在 portal_atct 工具裡，有 migrator 註冊，這樣可以搭配自製介面，不過，常見的方式是直接在 External Method 呼叫，反正昇級動作通常只做一次。
Skin Directory Registration
MyTheme/__init__.py
from Products.CMFCore.DirectoryView import registerDirectory
GLOBALS = globals()
registerDirectory('skins', GLOBALS)
plonetheme/mytheme/configure.zcml <cmf:registerDirectory name="mytheme_images"/> <cmf:registerDirectory name="mytheme_styles"/> <cmf:registerDirectory name="mytheme_templates"/>
Migration Scripts
Products/TinyMCE/upgrades.py
def meta_types_to_portal_types(meta_types):
"""Convert meta types to portal types"""
meta_types = meta_types.replace(u'ATTopic', u'Topic')
meta_types = meta_types.replace(u'ATEvent', u'Event')
meta_types = meta_types.replace(u'ATFile', u'File')
meta_types = meta_types.replace(u'ATFolder', u'Folder')
meta_types = meta_types.replace(u'ATImage', u'Image')
meta_types = meta_types.replace(u'ATBTreeFolder', u'Large Plone Folder')
meta_types = meta_types.replace(u'ATNewsItem', u'News Item')
meta_types = meta_types.replace(u'ATDocument', u'Document')
return meta_types
Products.contentmigration convert.py
# -*- coding:utf-8 -*-
from plone.app.blob.interfaces import IATBlobFile
from plone.app.blob.migrations import ATFileToBlobMigrator
from plone.app.blob.migrations import migrate
from Products.contentmigration.walker import CustomQueryWalker
from Testing.makerequest import makerequest
from zope.component.hooks import setSite
from zope.interface import noLongerProvides
import transaction
app = makerequest(app)
site = app.ieausp
setSite(site)
class BlobFileToBlobImageMigrator(ATFileToBlobMigrator):
''' Migrate File (ATBlob) to Image (ATBlob) '''
src_portal_type = 'File'
src_meta_type = 'ATBlob'
dst_portal_type = 'Image'
dst_meta_type = 'ATBlob'
# migrate all fields except 'file', which will be moved to 'image'
fields_map = {'file': None}
def migrate_data(self):
value = self.old.getField('file').getAccessor(self.old)()
self.new.getField('image').getMutator(self.new)(value)
def finalize(self):
ATFileToBlobMigrator.finalize(self)
# Remove IATBlobFile interface, otherwise schemaextender yells
# about primary field already set
noLongerProvides(self.new, IATBlobFile)
additionalQuery = {'path': '/Plone/images/'}
walker = CustomQueryWalker(site, BlobFileToBlobImageMigrator,
use_savepoint=False,
query=additionalQuery,
src_portal_type='File',
dst_portal_type='Image')
try:
migrate(site, walker())
except:
# If something goes wrong, let's find out
import pdb;pdb.post_morten()
# Commit the whole migration
transaction.commit()
# Sync zeo
app._p_jar.sync()
Zope4
Python3 Wonderland Products.CMFPlone Zope4 update
Linux 系統昇級
linux-x86_64: undefined symbol: PyUnicodeUCS2_AsEncodedString
ImportError: No module named _md5 底線符號開頭的模組，通常代表由 C 語言實作。MD5 是舊的實作，新的實作通常改用 OpenSSL 提供的 hashlib。
MacOS + Python 2.4 + build.python 需要用到 bitbucket 的 openpyxl
測試 Booking 模組
Booking Center - 目錄式物件，可內含 my bookable object my bookable - 目錄式物件，可內含 Booking，也可 export ...?
When upgrading from Plone 4.0 to Plone 4.1, you need to import the "Plone Discussions" profile in portal_setup before the collective.blog.star View will work.
plone.app.event Upgrade
Calendar Settings: plone.app.event
Plone 4.3.4 (plone.app.event 1.1.4) --> Plone 4.3.6 (plone.app.event 1.2.7) 出現下列錯誤，先刪除 /events/test-site-running 後，就能成功昇級。
INFO plone.app.event.dx.upgrades.upgrades set whole_day = false for event at http://140.109.161.62:8090/dgcn/events/test-site-running INFO plone.app.event.dx.upgrades.upgrades Set open_end = False for event at http://140.109.161.62:8090/dgcn/events/test-site-running ERROR Zope.SiteErrorLog 1443157898.190.789932194815 http://140.109.161.62:8090/dgcn/portal_quickinstaller/prefs_reinstallProducts Traceback (innermost last): Module ZPublisher.Publish, line 138, in publish Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 48, in call_object Module Products.CMFCore.FSPythonScript, line 127, in __call__ Module Shared.DC.Scripts.Bindings, line 322, in __call__ Module Shared.DC.Scripts.Bindings, line 359, in _bindAndExec Module Products.PythonScripts.PythonScript, line 344, in _exec Module script, line 11, in prefs_reinstallProducts - <FSPythonScript at /dgcn/portal_quickinstaller/prefs_reinstallProducts> - Line 11 Module Products.CMFPlone.QuickInstallerTool, line 101, in upgradeProduct Module Products.GenericSetup.upgrade, line 140, in doStep Module plone.app.event.dx.upgrades.upgrades, line 111, in upgrade_defaults_wholeday_openend Module zope.event, line 31, in notify Module zope.component.event, line 24, in dispatch Module zope.component._api, line 136, in subscribers Module zope.component.registry, line 321, in subscribers Module zope.interface.adapter, line 585, in subscribers Module zope.component.event, line 32, in objectEventNotify Module zope.component._api, line 136, in subscribers Module zope.component.registry, line 321, in subscribers Module zope.interface.adapter, line 585, in subscribers Module plone.app.event.dx.behaviors, line 488, in data_postprocessing Module pytz, line 167, in timezone AttributeError: 'NoneType' object has no attribute 'upper'
EventCreated or EventModified event and the data_postprocessing handler
缺少 plone.resources (portal_resources, IResourceDirectory) 可能造成 TypeError: argument of type 'NoneType' is not iterable 錯誤。
plone.batching/batch.py:109 collective.cover/browser/contentchooser.py #216 Using len() is deprecated. Use the `length` attribute for the size of the current page, which is what we return now. Use the `sequence_length` attribute for the size of the entire sequence.
AttributeError: 'module' object has no attribute 'get_tool'
資料移植
Transmogrifier 能夠處理資料匯入及匯出的工作，資料來源可以是文字檔或資料庫，常見的處理方式像是字碼轉換，依據型別、位置、狀態的條件來存取內容。設定檔裡要指定 Pipeline 內容，這些內容可以透過 Registry 或 Python 程式碼來載入。
<configure
xmlns=="http://namespaces.zope.org/zope"
xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier"
i18n_domain="collective.transmogrifier">
<transmogrifier:registerConfig
name="exampleconfig"
title="Example pipeline configuration"
description="This is an example pipeline configuration"
configuration="example.cfg"
/>
</configure>
使用 Python 程式碼時，是透過 'collective.transmogrifier.tests:exampleconfig.cfg' 格式來讀取設定檔，
multi-value CSV data for LinesField
mr.migrator
import static website content from the filesystem
collective.diversion copy paste plumi.migration
collective.migrator collective.jsonmigrator transmogrify.ploneremote
Migration Example
PRODUCTION_RESOURCE_DIRECTORY not in container plone.app.event: DateTime vs datetime
AttributeError: type object 'IIntIds' has no attribute '__iro__' AttributeError: 'NoneType' object has no attribute 'items': Can't upgrade Plone 3 to 4.x
DeprecationWarning: isDefaultPage is deprecated. Import from Products.CMFPlone instead.
RichText 欄位要搭配 RichTextValue 來輸入資料，不然會遇到 LocationError: (u'Some text\n', 'output', []) 之類的錯誤。
plone.app.textfield: RichTextValue object. (Did you mean .raw or .output?)
google eduCommons migrated to enpraxis.educommons
jQuery PloneFormGen collective.js.jqueryui
POSKeyError collective.uploadify
DataGridField TypeError: <class 'Products.DataGridField.Column.Column'> Pickle
PicklingError: porrtlet Issues Migration to Plone 4.3 import of module plone.app.kss.interfaces failed
remove portlet example: collective.newsflash
Archetypes News to Dexterity Container Type to Dexterity Solgema.fullcalendar: AT DX 並存
experimental.gracefulblobmissing
plone.app.imaging 的圖檔在 ZEO blob storage 的管理方式
This simple script and diff and manual changes against appropriate files.
#!/var/zope/python/bin/python
f = open("Data.fs.tr0", "r+ab")
c = f.read()
s='__name__q'
a = c.split(s)
i = 0
print(len(a))
for polozka in a:
i = i + 1
fn = 'vystup_' + str(i)
x = open(fn,"ab")
x.write(polozka)
x.close()
f.close()
zopyx.plone.migration: exporter.py
def export_members(options):
log('Exporting Members')
fp = file(os.path.join(options.export_directory, 'members.ini'), 'w')
acl_users = options.plone.acl_users
users = acl_users.getUserNames()
num_users = len(users)
pm = options.plone.portal_membership
try:
# Plone 2.5
passwords = options.plone.acl_users.source_users._user_passwords
except:
# Plone 2.1
passwords = None
for i, username in enumerate(users):
if username == "":
# possibly Membrane User Object whick will be exported
# later in structure_export
continue
user = acl_users.getUserById(username)
member = pm.getMemberById(username)
if member is None:
if options.verbose:
log('--> (%d/%d) INVALID %s' % ((i + 1), num_users, username))
continue
if options.verbose:
log('--> (%d/%d) %s' % ((i + 1), num_users, username))
roles = [r for r in member.getRoles() if not r in ('Member', 'Authenticated')]
print >>fp, '[member-%s]' % username
print >>fp, 'username = %s' % username
if passwords:
print >>fp, 'password = %s' % passwords.get(username)
else:
try:
print >>fp, 'password = %s' % user.__
except AttributeError:
print >>fp, 'password = %s' % 'n/a'
print >>fp, 'fullname = %s' % member.getProperty('fullname')
print >>fp, 'email = %s' % member.getProperty('email')
print >>fp, 'roles = %s' % ','.join(roles)
print >>fp
fp.close()
log('exported %d users' % len(acl_users.getUserNames()))
以上程式碼可處理 Plone 2.x 3.x 的資料。
Plone 2.x 時代
ATContentTypes migrations in early Plone 2.1.x releases did miss large folders Columbia River PUD Upgrade from 2.1 to 4.3
<persistent broken Products.CMFPlone.LargePloneFolder.LargePloneFolder instance ... >