Skip to content. | Skip to navigation

Personal tools

Navigation

You are here: Home / Tips / Migration and Upgrade

Migration and Upgrade

Plone 核心模組本身附有自動昇級的工具程式,想要無痛昇級內建的型別資料,並非難事,客製模組則要額外撰寫程式來執行移植工作,許多昇級挑戰,都來自於擴充模組。

特別是缺乏維護的擴充模組,通常只能做到安裝成功,卻不處理乾淨移除的問題。新版 Plone6 在 2017 年初起步,已經能在 Zope4 和 ZODB5 環境執行,雖然不是完全跳船到 Python3,但目標是先清除 Python3 的障礙。

準備工作

先閱讀昇級指南與相關文件動手之前,有完整備份系統和資料,才能無後顧之憂。執行 pack 並移除舊的模組,讓影響因素降到最低。

Python 2.6 -> Python 2.7 可採 Steve McMahon 推薦的保守方式逐步確認

使用 bin/snapshotbackup 指令

plone.app.linkintegrity Event Handler Patching (added in Plone 5.1 rc2)

Plone4 之前的環境要了解 Python Script 和 External Method,必讀的文件包括 Command-line interaction and scriptingUpgrade GuidePlone Conf 2014 提及 PLIP #13283 標示許多 plone.app.* 套件將併進 Plone5。Plone3 昇到 Plone4 可能遇到 Catalog Brain 錯誤狀況,利用 collective.catalogcleanup 來消除已不存在的內容。

Move ZMI / DTML configuration form into a Plone page, Move CMFFormController to BrowserViews

Products.PageTemplates 成為 Zope2 內建模組 five.pt 逐步被取消

skins/plone_scripts 在 Plone5 不再使用並被移除,像 displayContentsTab 在 Plone5.1 被移除,Products.TinyMCE 不再是 Plone5 的相依模組。像 toggleSelect 被移除了,需要改用新模組。

refactor buildout.cfg: collective.googlenews

ftw.upgrade Zope4

Plone5

Plone5 將 InterfaceTool 移除,導致 getDottedName 無法呼叫,像 plone.app.layout 需要調整 import 方式。將 portal_calendar 移除,宣告 Event-like 型別的方式要調整。

In Plone 5.1, IFactoryTool is no longer defined in CMFPlone, so we need to handle when it is missing.

registry.xml 運作方式調整,JavaScript 和 CSS 採新的註冊格式: cioppino.twothumbs eea.facetednavigation

TypeError: argument of type 'NoneType' is not iterable

自行編輯 import-step 來更新設定值

Importing Plone 4.3 Site Fails: No Module Named CMFDefault.DiscussionItem

Plone4

Plone 4.0.x 可能會遇到 Python lib_dirs 錯誤造成 zlib 找不到。

Plone 4.2 -> 4.3 getSite, setSite : from zope.app.component.hooks or zope.site.hooks to zope.component.hooks

Unresolved Dependencies Error: 本身不會造成損害

def fix_plonepas_dependencies(setup_tool):
    """Remove 'plonepas-content' as a dependency for 'plonepas' GS profile."""
    from Products.GenericSetup.registry import _import_step_registry
    for registry in setup_tool.getImportStepRegistry(), _import_step_registry:
        registry._registered.get('plonepas')['dependencies'] = \
            (u'componentregistry', u'memberdata-properties', u'rolemap', u'controlpanel')

examples: collective.dancing

早期常用 Products.contentmigration 來執行 Archetypes 的昇級,可參考 FalconImageMigrator plone.app.blob 的 migrations.pyProducts.CMFPloneMigrationTool.pytests/testMigrationTool.py 範例,同時 Archetypes 也附有昇級範例新的工具包括 Script 或 Framework,像是 collective.blueprint.jsonmigratorzopyx.plone.migrationrt.atmigrator,但 api.plone.org 這類 API 並不成熟。改用 plone.api 並更新相依關係: collective.watcherlisr

Framework Team 提議 Core Package 不使用 plone.api

更改 dexterity class 步驟

  1. search catalog for all items of the particular portal_type;
  2. get the object for each brain in the result;
  3. check that the class matches expectations, if not, continue;
  4. Create a replacement item; copy the object state from the old version of the item to the new one using obj.__dict__() on the source, and setattr for each key/value (follwed by setting _p_changed attribute to True).
  5. Reindex catalog;
  6. Commit transaction.

事後檢查

plone_deprecated sitemap

Purging Missing Interfaces

plone.app.upgrade 已把 Plone 2.5 Plone 3 昇級程式碼移除

Deprecation

Style Guide

Editor and HTML Filtering

Kupu 在昇級過程沒有完整被移除,需要手動移除

Kupu ZCML Condition

Issues on Upgrade to Plone 4.3

Schema Class 從 plone.directives.form.schema 改到 plone.supermodel.model,可能造成 PicklingError。可考慮搭配的方式,包括 wildcard.fixmissing移除 Marker Interface

Dexterity View Upgrade

Theme

KSS 不再是核心模組,圖示使用 PNG 取代 GIF。

KSS Removal: kss_field_decorator_view FolderContentsKSSView vs FolderContentsBrowserView

UserWarning: DirectoryView archetypes_kss refers to a non-existing path 'Products.Archetypes:skins/archetypes_kss'

UserWarning: DirectoryView plone_kss refers to a non-existing path 'Products.CMFPlone:skins/plone_kss'

('Topic', '') ValueError: unknown meta_type 當 plone.app.contenttypes 昇級後,Archetypes 的 Topic 不再存在,部份舊式模組造成錯誤,利用 'deprecated="True"' 來處理。範例: Solgema.fullcalendar

<object name="Topic" deprecated="True" />

Plone 5 Upgrade Without UUID Links Configlet 介面和圖示更新 plone.app.caching 的安裝與隱藏

# migrate-to-uuid.py
from AccessControl.SecurityManagement import newSecurityManager
from Acquisition import aq_parent
from Products.Archetypes.Field import TextField
from Products.CMFPlone.interfaces.siteroot import IPloneSiteRoot
from lxml.html import fromstring
from lxml.html import tostring
from plone.app.textfield.value import RichTextValue
from plone.dexterity.interfaces import IDexterityContent
from plone.uuid.interfaces import IUUID
import transaction


def getUID(ob):
    try:
        return IUUID(ob)
    except TypeError:
        return ob.UID()


def findObject(from_obj, path):
    while not IPloneSiteRoot.providedBy(from_obj):
        found = from_obj.restrictedTraverse(path, None)
        if found:
            return found
        from_obj = aq_parent(from_obj)


def transform_links_to_uuid(ob, html):
    if not html:
        return html
    changes = False
    try:
        dom = fromstring(html)
    except:
        print('Error parsing DOM from content: %s' % '/'.join(ob.getPhysicalPath()))
        return html
    for el in dom.cssselect('a'):
        href = el.attrib.get('href', '')
        if (not href or href.startswith('http://') or
                href.startswith('https://') or href.startswith('/') or
                'resolveuid' in href):
            continue
        link_obj = findObject(ob, href)
        if link_obj:
            uuid = getUID(link_obj)
            el.attrib.update({
                'href': 'resolveuid/%s' % uuid,
                'data-linktype': 'internal',
                'data-val': uuid
            })
            changes = True

    for el in dom.cssselect('img'):
        src = el.attrib.get('src', '')
        if (not src or src.startswith('http://') or
                src.startswith('https://') or src.startswith('/') or
                'resolveuid' in src):
            continue

        src, _, scale = src.partition('/@@images/image/')
        if not scale:
            src, _, scale = src.partition('/image_')
        img_obj = findObject(ob, src)
        if img_obj:
            uuid = getUID(img_obj)
            new_href = 'resolveuid/%s' % uuid
            if scale:
                new_href += '/@@images/image/' + scale

            attribs = {
                'src': new_href,
                'data-linktype': 'image',
                'data-val': uuid
            }
            if scale:
                attribs['data-scale'] = scale
            el.attrib.update(attribs)
            changes = True

    if changes:
        return tostring(dom)
    else:
        return html


def migrate_dexterity_to_uuid_links(ob):
    try:
        orig = ob.text.raw
        new = transform_links_to_uuid(ob, orig)
        if new != orig:
            ob.text = RichTextValue(
                new, mimeType=ob.text.mimeType, outputMimeType=ob.text.outputMimeType)
            print('Fixed content not using UUID links: %s' % '/'.join(ob.getPhysicalPath()))
            return 1
    except AttributeError:
        pass
    return 0


def migrate_archetypes_to_uuid_links(ob):
    for field in ob.Schema().fields():
        if type(field) == TextField and field.getContentType(ob) == 'text/html':
            orig = field.getRaw(ob)
            new = transform_links_to_uuid(ob, orig)
            if orig != new:
                field.set(ob, new)
                print('Fixed content not using UUID links: %s' % '/'.join(ob.getPhysicalPath()))
                return 1

    return 0


def migrate_to_uuid_links(site):
    catalog = site.portal_catalog
    count = 0
    for brain in catalog():
        ob = brain.getObject()
        if IDexterityContent.providedBy(ob):
            count += migrate_dexterity_to_uuid_links(ob)
        else:
            count += migrate_archetypes_to_uuid_links(ob)
        if count % 100 == 0:
            transaction.commit()
            print('finished processing %i' % count)


user = app.acl_users.getUser('admin')  # noqa
newSecurityManager(None, user.__of__(app.acl_users))  # noqa

for oid in app.objectIds():  # noqa
    _obj = app[oid]  # noqa
    if IPloneSiteRoot.providedBy(_obj):
        migrate_to_uuid_links(_obj)
        transaction.commit()

Plone5 Theme Upgrade

ZEXP export/import

AttributeError: uid_catalog 直接從 Plone4 匯出再進 Plone5 時會發生

整併不同 Plone Sites 內容的範例

避免用 Copy + Paste 方式複製 Plone,通常 catalog 會索引重複的項目,工作流程會使用初始狀態,UID 會重新產生。用 Cut + Paste 更糟,會在新項目建立 Reference,但被參照的項目卻不存在了。

複製 Plone Site 要注意的事項

Migrating Zope to .Net Apps: Access ZODB in .Net ?

Setting the right permissions on your blobstorage directory

zexp Products.PDBdebug

Remove Broken Transforms XML parse

AttributeError: displayContentsTab

intid 維護管理

Running Old Instance on Another Port

bin/plonectl path

bin/instance path

bin/buildout path

parts/instance/etc/zope.conf path and port

Archetypes to Dexterity

存取 Field 的差異: 前者使用 obj.getField('fieldname').get(obj) 或 obj.getFieldname() 之類的方式,後者使用 obj.fieldname 或 __getattr__ 來取得預設值。

archetypes.schemaextender 主要步驟

ATContentTypes Walkers and Migrators

Archetypes 型別的轉換

在 portal_atct 工具裡,有 migrator 註冊,這樣可以搭配自製介面,不過,常見的方式是直接在 External Method 呼叫,反正昇級動作通常只做一次。

Skin Directory Registration

MyTheme/__init__.py

from Products.CMFCore.DirectoryView import registerDirectory

GLOBALS = globals()
registerDirectory('skins', GLOBALS)
plonetheme/mytheme/configure.zcml

<cmf:registerDirectory name="mytheme_images"/>
<cmf:registerDirectory name="mytheme_styles"/>
<cmf:registerDirectory name="mytheme_templates"/>

Migration Scripts

Products/TinyMCE/upgrades.py

def meta_types_to_portal_types(meta_types):
    """Convert meta types to portal types"""
    meta_types = meta_types.replace(u'ATTopic', u'Topic')
    meta_types = meta_types.replace(u'ATEvent', u'Event')
    meta_types = meta_types.replace(u'ATFile', u'File')
    meta_types = meta_types.replace(u'ATFolder', u'Folder')
    meta_types = meta_types.replace(u'ATImage', u'Image')
    meta_types = meta_types.replace(u'ATBTreeFolder', u'Large Plone Folder')
    meta_types = meta_types.replace(u'ATNewsItem', u'News Item')
    meta_types = meta_types.replace(u'ATDocument', u'Document')
    return meta_types

Products.contentmigration convert.py

# -*- coding:utf-8 -*-
from plone.app.blob.interfaces import IATBlobFile
from plone.app.blob.migrations import ATFileToBlobMigrator
from plone.app.blob.migrations import migrate
from Products.contentmigration.walker import CustomQueryWalker
from Testing.makerequest import makerequest
from zope.component.hooks import setSite
from zope.interface import noLongerProvides
 
import transaction
 
app = makerequest(app)
site = app.ieausp
setSite(site)
 
 
class BlobFileToBlobImageMigrator(ATFileToBlobMigrator):
    ''' Migrate File (ATBlob) to Image (ATBlob) '''
 
    src_portal_type = 'File'
    src_meta_type = 'ATBlob'
    dst_portal_type = 'Image'
    dst_meta_type = 'ATBlob'
    # migrate all fields except 'file', which will be moved to 'image'
    fields_map = {'file': None}
 
    def migrate_data(self):
        value = self.old.getField('file').getAccessor(self.old)()
        self.new.getField('image').getMutator(self.new)(value)
 
    def finalize(self):
        ATFileToBlobMigrator.finalize(self)
        # Remove IATBlobFile interface, otherwise schemaextender yells
        # about primary field already set
        noLongerProvides(self.new, IATBlobFile)
 
 
additionalQuery = {'path': '/Plone/images/'}
 
walker = CustomQueryWalker(site, BlobFileToBlobImageMigrator,
                           use_savepoint=False,
                           query=additionalQuery,
                           src_portal_type='File',
                           dst_portal_type='Image')
 
 
try:
    migrate(site, walker())
except:
    # If something goes wrong, let's find out
    import pdb;pdb.post_morten()
 
# Commit the whole migration
transaction.commit()
 
# Sync zeo
app._p_jar.sync()

Linux 系統昇級

linux-x86_64: undefined symbol: PyUnicodeUCS2_AsEncodedString

Python 版本昇級

ImportError: No module named _md5 底線符號開頭的模組,通常代表由 C 語言實作。MD5 是舊的實作,新的實作通常改用 OpenSSL 提供的 hashlib。

MacOS + Python 2.4 + build.python 需要用到 bitbucket 的 openpyxl

測試 Booking 模組

Booking Center - 目錄式物件,可內含 my bookable object my bookable - 目錄式物件,可內含 Booking,也可 export ...?

KeyError with MountPoint

When upgrading from Plone 4.0 to Plone 4.1, you need to import the "Plone Discussions" profile in portal_setup before the collective.blog.star View will work.

plone.app.event Upgrade

Calendar Settings: plone.app.event

Plone 4.3.4 (plone.app.event 1.1.4) --> Plone 4.3.6 (plone.app.event 1.2.7) 出現下列錯誤,先刪除 /events/test-site-running 後,就能成功昇級。

INFO plone.app.event.dx.upgrades.upgrades set whole_day = false for event at http://140.109.161.62:8090/dgcn/events/test-site-running
INFO plone.app.event.dx.upgrades.upgrades Set open_end = False for event at http://140.109.161.62:8090/dgcn/events/test-site-running
ERROR Zope.SiteErrorLog 1443157898.190.789932194815 http://140.109.161.62:8090/dgcn/portal_quickinstaller/prefs_reinstallProducts
Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module Products.CMFCore.FSPythonScript, line 127, in __call__
  Module Shared.DC.Scripts.Bindings, line 322, in __call__
  Module Shared.DC.Scripts.Bindings, line 359, in _bindAndExec
  Module Products.PythonScripts.PythonScript, line 344, in _exec
  Module script, line 11, in prefs_reinstallProducts
   - <FSPythonScript at /dgcn/portal_quickinstaller/prefs_reinstallProducts>
   - Line 11
  Module Products.CMFPlone.QuickInstallerTool, line 101, in upgradeProduct
  Module Products.GenericSetup.upgrade, line 140, in doStep
  Module plone.app.event.dx.upgrades.upgrades, line 111, in upgrade_defaults_wholeday_openend
  Module zope.event, line 31, in notify
  Module zope.component.event, line 24, in dispatch
  Module zope.component._api, line 136, in subscribers
  Module zope.component.registry, line 321, in subscribers
  Module zope.interface.adapter, line 585, in subscribers
  Module zope.component.event, line 32, in objectEventNotify
  Module zope.component._api, line 136, in subscribers
  Module zope.component.registry, line 321, in subscribers
  Module zope.interface.adapter, line 585, in subscribers
  Module plone.app.event.dx.behaviors, line 488, in data_postprocessing
  Module pytz, line 167, in timezone
AttributeError: 'NoneType' object has no attribute 'upper'

EventCreated or EventModified event and the data_postprocessing handler

缺少 plone.resources (portal_resources, IResourceDirectory) 可能造成 TypeError: argument of type 'NoneType' is not iterable 錯誤。

plone.batching/batch.py:109 collective.cover/browser/contentchooser.py #216 Using len() is deprecated. Use the `length` attribute for the size of the current page, which is what we return now. Use the `sequence_length` attribute for the size of the entire sequence.

AttributeError: 'module' object has no attribute 'get_tool'

資料移植

Transmogrifier 能夠處理資料匯入及匯出的工作,資料來源可以是文字檔或資料庫,常見的處理方式像是字碼轉換,依據型別、位置、狀態的條件來存取內容。設定檔裡要指定 Pipeline 內容,這些內容可以透過 Registry 或 Python 程式碼來載入。

<configure
    xmlns=="http://namespaces.zope.org/zope"
    xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier"
    i18n_domain="collective.transmogrifier">

  <transmogrifier:registerConfig
    name="exampleconfig"
    title="Example pipeline configuration"
    description="This is an example pipeline configuration"
    configuration="example.cfg"
    />

</configure>

使用 Python 程式碼時,是透過 'collective.transmogrifier.tests:exampleconfig.cfg' 格式來讀取設定檔,

Pipeline, Blueprints, Sources

to specify alternative _path

example from collective.nitf

multi-value CSV data for LinesField

mr.migrator

import static website content from the filesystem

批次昇級

collective.diversion copy paste plumi.migration

collective.migrator collective.jsonmigrator transmogrify.ploneremote

Migration Example

PRODUCTION_RESOURCE_DIRECTORY not in container plone.app.event: DateTime vs datetime

AttributeError: type object 'IIntIds' has no attribute '__iro__' AttributeError: 'NoneType' object has no attribute 'items': Can't upgrade Plone 3 to 4.x

DeprecationWarning: isDefaultPage is deprecated. Import from Products.CMFPlone instead.

RichText 欄位要搭配 RichTextValue 來輸入資料,不然會遇到 LocationError: (u'Some text\n', 'output', []) 之類的錯誤。

plone.app.textfield: RichTextValue object. (Did you mean .raw or .output?)

News Items with Images

google eduCommons migrated to enpraxis.educommons

Creation Modification Date

puppet facter like automation

jQuery PloneFormGen collective.js.jqueryui

POSKeyError collective.uploadify

DataGridField TypeError: <class 'Products.DataGridField.Column.Column'> Pickle

PicklingError: porrtlet Issues Migration to Plone 4.3 import of module plone.app.kss.interfaces failed

www.nidelven-it.no

remove portlet example: collective.newsflash

Collection instead of Topic

Archetypes News to Dexterity Container Type to Dexterity Solgema.fullcalendar: AT DX 並存

experimental.gracefulblobmissing

plone.app.imaging 的圖檔在 ZEO blob storage 的管理方式

mxm-mad-science: via JSON

This simple script and diff and manual changes against appropriate files.

#!/var/zope/python/bin/python
f = open("Data.fs.tr0", "r+ab")
c =  f.read()
s='__name__q'
a = c.split(s)
i = 0
print(len(a))
for polozka in a:
   i = i + 1
   fn = 'vystup_' + str(i)
   x = open(fn,"ab")
   x.write(polozka)
   x.close()
f.close()

zopyx.plone.migration: exporter.py

def export_members(options):
    log('Exporting Members')
    fp = file(os.path.join(options.export_directory, 'members.ini'), 'w')
    acl_users = options.plone.acl_users
    users = acl_users.getUserNames()
    num_users = len(users)
    pm = options.plone.portal_membership
    try:
        # Plone 2.5
        passwords = options.plone.acl_users.source_users._user_passwords
    except:
        # Plone 2.1
        passwords = None
    for i, username in enumerate(users):
        if username == "":
            # possibly Membrane User Object whick will be exported
            # later in structure_export
            continue
        user = acl_users.getUserById(username)
        member = pm.getMemberById(username)
        if member is None:
            if options.verbose:
                log('--> (%d/%d) INVALID %s' % ((i + 1), num_users, username))
            continue
        if options.verbose:
            log('--> (%d/%d) %s' % ((i + 1), num_users, username))
        roles = [r for r in member.getRoles() if not r in ('Member', 'Authenticated')]
        print >>fp, '[member-%s]' % username
        print >>fp, 'username = %s' % username
        if passwords:
            print >>fp, 'password = %s' % passwords.get(username)
        else:
            try:
                print >>fp, 'password = %s' % user.__
            except AttributeError:
                print >>fp, 'password = %s' % 'n/a'

print >>fp, 'fullname = %s' % member.getProperty('fullname') print >>fp, 'email = %s' % member.getProperty('email') print >>fp, 'roles = %s' % ','.join(roles) print >>fp fp.close() log('exported %d users' % len(acl_users.getUserNames()))

以上程式碼可處理 Plone 2.x 3.x 的資料。

Plone 2.x 時代

ATContentTypes migrations in early Plone 2.1.x releases did miss large folders Columbia River PUD Upgrade from 2.1 to 4.3

<persistent broken Products.CMFPlone.LargePloneFolder.LargePloneFolder instance ... >

修訂 AttributeError: kss_generic_macros 錯誤

Related content
Transmogrifier
Performance