Skip to content. | Skip to navigation

Personal tools

Navigation

You are here: Home / Tips / ZODB

ZODB

Plone 預設使用 Zope Object Database (ZODB) 儲存內容資料和設定值,ZODB 屬於物件導向資料庫,支援交易和版本管理功能,能夠有效地管理 Python 物件,進階用法是利用分散式架構提昇延展性,或是介接傳統的 SQL 資料庫。

ZODB 3.8 版本之後,提供 blob 大型檔案支援,透過 plone.app.blob 讓 Plone 模組能夠享用這項功能,原本使用 File 或 Image 型別的模組,可以把資料轉存到檔案系統裡。ZODB3 後來分成 persistent, BTree, ZODB 三個模組。

文章 Zope Secrets 說明 ZODB 核心運作機制。LxBTree 使用 Py_LongLong (a signed type) 做為 key,功能之一是處理 forward index。現有的 ZODB 和 ZEO 實作方式,假定 OID 和 TID 都是 64-bit 數值,通常可用 64-bit 整數來表示。

效能簡介

實戰經驗達 100GB 資料規模和 SQLite 比較

ZODB Benchmarks revisited by Roché Compaan

The truth is that the ZODB is faster than your RDBMS. My previous post about ZODB benchmarks incorrectly confirmed the general opinion people had of the ZODB which is that it is not very fast when you want to insert millions of objects into it. Many developers will tell you, the ZODB is a low-write high-read database. Many more developers will reduce this to a slating "the ZODB is slow".

Fortunately this is not true! I overlooked a basic thing when I compared ZODB performance with Postgres. I only realised that the Postgres table did not have an index on the key field when I started testing lookup speed. After adding the index I neglected to re-run the insertion test on Postgres. I realised this soon after I wrote up my findings and ran the test again. Adding the index causes the insertion on Postgres to drop logarithmically at a higher rate than the ZODB.

For the most part of the test, insertion is faster in the ZODB than in Postgres. Wow, I didn't expect that! I don't know about you but this is a very comforting result for me. There was a bug in my lookup test as well. After fixing this bug the times for lookups were looking a lot better:

Number of ObjectsAverage Lookup Time in Seconds
100000 0.00000311
1000000 0.00000648
10000000 0.00230820

From the above table it is clear that lookups on a BTree with 10 million objects is very fast at around 2 milliseconds. On a Postgres table with 10 million records the average lookup time was 14 milliseconds.

It wasn't my original goal to compare the ZODB with Postgres - I simply used Postgres as a reference point. I do think that the comparison was necessary to "fix" the perception people have of the ZODB. I love the ZODB and were looking really hard for reasons to justify my extensive use of it but benchmarks weren't available. Nevertheless, this is just a starting point. The tests are still very superficial, almost deliberately so since it allows one to compare apples with apples. I think applications like Zope and Plone generally do a lot more with the ZODB than what an RDMBS allows. I mean, of how many systems have you heard that has per-object security and indexed meta data for the majority of the content. I think it would be worthwhile to start quantifying how many objects are modified with common actions in Zope and Plone in a single transaction. This should give us an appreciation of how hard the ZODB is working. Our work is not done.

完整的 ZODB 說明,可參考技術文件文件計畫

@vangheezy: 讀的效能很好,但多人同時寫的效能不好,大量資料情況下的索引效能也不好。

@paulweveritt: PostgreSQL JSONB

@JimFulton: Optimizations Featuring Prefetch

想要觀察 Plone 應用 ZODB 的效果,可以檢查 FileStorage 和 BlobStorage 目錄裡的檔案,前者的檔案位置通常是 var/filestorage/data.fs,後者的目錄位置通常是在 var/blobstorage。

想要管理 ZODB 儲存容量,可以執行 pack 並清除不再使用的 blob

zodbpickle 使用客製化的 pickle

Zope Replication Services (ZRS)

FileStorage

Data.fs

BlobStorage

新增檔案或圖檔後,預設會以 Blob 格式儲存在檔案系統裡,檔案會以原有大小儲存,圖檔還會視情況同步儲存不同尺寸的縮放檔案。

如果 BlobStorage 獨立掛載,利用 experimental.gracefulblobmissing 可以改善空窗時段的讀取問題。

下列是上傳 1.png (725,985 bytes) 的結果:

$ ls -lR var/blobstorage/
var/blobstorage/:
total 8
drwx------ 3 marr marr 4096 Jul 27 20:13 0x00
drwx------ 2 marr marr 4096 Jul 27 20:13 tmp

var/blobstorage/0x00:
total 4
drwx------ 3 marr marr 4096 Jul 27 20:13 0x00

var/blobstorage/0x00/0x00:
total 4
drwx------ 3 marr marr 4096 Jul 27 20:13 0x00

var/blobstorage/0x00/0x00/0x00:
total 4
drwx------ 3 marr marr 4096 Jul 27 20:13 0x00

var/blobstorage/0x00/0x00/0x00/0x00:
total 4
drwx------ 3 marr marr 4096 Jul 27 20:13 0x00

var/blobstorage/0x00/0x00/0x00/0x00/0x00:
total 4
drwx------ 3 marr marr 4096 Jul 27 20:13 0x00

var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x00:
total 4
drwx------ 5 marr marr 4096 Jul 27 20:13 0x11

var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x00/0x11:
total 12
drwx------ 2 marr marr 4096 Jul 27 20:13 0x55
drwx------ 2 marr marr 4096 Jul 27 20:13 0x6c
drwx------ 2 marr marr 4096 Jul 27 20:13 0x81

var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x00/0x11/0x55:
total 0
-r-------- 1 marr marr 0 Jul 27 20:13 0x03a04fdd73235e22.blob

var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x00/0x11/0x6c:
total 712
-r-------- 1 marr marr 725985 Jul 27 20:13 0x03a04fdd73235e22.blob

var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x00/0x11/0x81:
total 164
-r-------- 1 marr marr 164975 Jul 27 20:13 0x03a04fdd77d56e55.blob

var/blobstorage/tmp:
total 0
$ tree var/blobstorage
.
├── 0x00
│   └── 0x00
│       └── 0x00
│           └── 0x00
│               └── 0x00
│                   └── 0x00
│                       └── 0x11
│                           ├── 0x55
│                           │   └── 0x03a04fdd73235e22.blob
│                           ├── 0x6c
│                           │   └── 0x03a04fdd73235e22.blob
│                           └── 0x81
│                               └── 0x03a04fdd77d56e55.blob
└── tmp

933 x 512 是原圖尺寸,400 x 219 是 preview 尺寸。

$ file 
var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x00/0x11/0x6c/0x03a04fdd73235e22.blob:
 PNG image data, 933 x 512, 8-bit/color RGB, non-interlaced
var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x00/0x11/0x81/0x03a04fdd77d56e55.blob:
 PNG image data, 400 x 219, 8-bit/color RGB, non-interlaced
var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x00/0x11/0x55/0x03a04fdd73235e22.blob:
 empty

在 portal_catalog 裡查詢它的 UID 是 9110c1329a1c46a09a4c99eec2e156aa。

將 1.png 刪除後,會增加一個 200 x 109 的 blob。

$ tree
.
├── 0x00
│   └── 0x00
│       └── 0x00
│           └── 0x00
│               └── 0x00
│                   └── 0x00
│                       └── 0x11
│                           ├── 0x55
│                           │   └── 0x03a04fdd73235e22.blob
│                           ├── 0x6c
│                           │   └── 0x03a04fdd73235e22.blob
│                           ├── 0x81
│                           │   └── 0x03a04fdd77d56e55.blob
│                           └── 0x82
│                               └── 0x03a058decaf30daa.blob
└── tmp
$ ls -l 0x00/0x00/0x00/0x00/0x00/0x00/0x11/0x82/0x03a058decaf30daa.blob
-r-------- 1 marr marr 44952 Jul 29 10:38
$ file 0x00/0x00/0x00/0x00/0x00/0x00/0x11/0x82/0x03a058decaf30daa.blob
 PNG image data, 200 x 109, 8-bit/color RGB, non-interlaced

執行 pack 後,blob 檔案會被移除,只留下目錄:

$ tree
.
├── 0x00
│   └── 0x00
│       └── 0x00
│           └── 0x00
│               └── 0x00
│                   └── 0x00
│                       └── 0x11
└── tmp

重新上傳 1.png 的話:

$ tree
.
├── 0x00
│   └── 0x00
│       └── 0x00
│           └── 0x00
│               └── 0x00
│                   └── 0x00
│                       └── 0x11
│                           ├── 0x8b
│                           │   └── 0x03a058ec7afd4ebb.blob
│                           ├── 0xa3
│                           │   └── 0x03a058ec7afd4ebb.blob
│                           └── 0xb8
│                               └── 0x03a058ec7fb658aa.blob
└── tmp
$ ls -lR 0x00/0x00/0x00/0x00/0x00/0x00/0x11/
0x00/0x00/0x00/0x00/0x00/0x00/0x11/:
total 12
drwx------ 2 marr marr 4096 Jul 29 10:52 0x8b
drwx------ 2 marr marr 4096 Jul 29 10:52 0xa3
drwx------ 2 marr marr 4096 Jul 29 10:52 0xb8

0x00/0x00/0x00/0x00/0x00/0x00/0x11/0x8b:
total 0
-r-------- 1 marr marr 0 Jul 29 10:52 0x03a058ec7afd4ebb.blob

0x00/0x00/0x00/0x00/0x00/0x00/0x11/0xa3:
total 712
-r-------- 1 marr marr 725985 Jul 29 10:52 0x03a058ec7afd4ebb.blob

0x00/0x00/0x00/0x00/0x00/0x00/0x11/0xb8:
total 164
-r-------- 1 marr marr 164975 Jul 29 10:52 0x03a058ec7fb658aa.blob

刪除 File 的情況下,可能不會馬上移除 Blob。

Conflict Error

當一個以上的 request 同時要寫入 ZODB 時,會造成衝突,有些 request 被迫要重試寫入。這類狀況本身問題不大,但代表系統可能需要調整執行效能

@vangheezy: ZODB Blobs are nice but only nice to replicate them is ZRS. It gets tricky with RelStorage.

vangheem: making a change to the ZODB on a slave, would make the replication get out of sync and it won't replicate anymore.

plone.app.blob.utils.openBlob()

Handling multiple object creations at once from multiple clients and processes

Relstorage: Abstract ZODB's MVCC implementation into a storage adapter lastTransaction

etc/zope.conf

zc.beforestorage: View ZODB Storage Before a Given Time

Blob Directory Layout

在 var/blobstorage 目錄裡,找得到 .layout 檔案,裡面記錄目錄結構的管理方式,例如新版預設使用 bushy 格式。舊版使用 lawn 格式,建議昇級成 bushy 格式。

# ZODB3-3.10.5-py2.6-linux-i686.egg/ZODB/blob.py

    def getOIDForPath(self, path):
        """Given a path, return an OID, if the path is a valid path for an
        OID. The inverse function to `getPathForOID`.

        Raises ValueError if the path is not valid for an OID.
        """
        path = path[len(self.base_dir):]
        return self.layout.path_to_oid(path)

"Catalog" is the usual term in ZODB-land (because a catalog contains one or more indexes). These aren't provided by ZODB itself, but there are packages that provide them on top of ZODB. E.g. one of those is repoze.catalog.

BTree 4.4.0 重新啟用 None 當作 Key Products.ExtendedPathIndex 同時被修訂

persistent dependency: ZODB3 包含 persistent 相依關係,但早期版本有相容性問題。

ZODBConvert: 加密 encrypt ZRS cipher.encryptingstorage

沒設定 UTC 系統時間 historical_connections time.time 會造成錯誤

TempStorage: RAM based storage for ZODB DemoStorage: Conflict Resolution and Fix history()

Traversal

goodie #1

response = traverse(root, HTTP_PATH.split('/'), request)
def traverse(context, path, request):
  if len(path) == 0:
    return context(request)
  elif len(path) == 1:
    view_method = getattr(context, path[0])
    return view_method(request)
  else:
    sub_context = context[path[0]]
    return traverse(sub_context, path[1:], request)

aq_base(obj) vs obj.aq_base

Find Large Objects

if not container.REQUEST.REMOTE_ADDR.startswith('141.233.'):
    return ""

plone_site_search_depth = 3 # depth it will search, from the zope root, for plone sites


def find_honking_files(plone):
    """
    Steps:
    1. look at Plone content types
    2. look in portal skins
    """

    pc = plone.portal_catalog
    brains = pc.searchResults({'portal_type' : ('Image', 'File', 'News Item') })
    brains = list(brains)

    #brains.filter(lambda x : "MB" not in x.getObjSize)
    #brains.sort(lambda x, y : x.getObjSize > y.getObjSize)

    for b in brains:
        size = b.getObjSize
        if "MB" in size or "GB" in size:
            url = b.getURL()
            print "%s %s
" % (size, url, url) ps = plone.portal_skins def find_big_ones(current): found = [] for x in current.objectIds(): obj = current[x] if hasattr(obj, 'getContentType') and str(obj.getContentType()).strip().startswith('image'): size = (float(obj.get_size())/1024.0)/1024.0 if size > 1.0: print "%s MB : %s
" % (str(size)[:4], obj.id()) if hasattr(obj, 'objectIds'): print find_big_ones(obj) return printed print find_big_ones(ps) return printed def get_plone_sites(): def find_it(current_context, depth): if depth == 0: return [] else: sites = [] for obj in current_context.objectIds(): site = current_context[obj] if hasattr(site, 'portal_type') and site.portal_type == 'Plone Site': sites.append(site) elif hasattr(site, 'objectIds'): sites.extend(find_it(site, depth - 1)) return sites return find_it(context, plone_site_search_depth) sites = get_plone_sites() for site in sites: print "Plone Site: %s
" % site.title print find_honking_files(site) return printed

檔案過大的處理方式

OSError: [Errno 12] Cannot allocate memory
/home/marr/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/Connection.py:660:
UserWarning: The <type 'BTrees.IOBTree.IOBucket'>
object you're saving is large. (24951980 bytes.)

Perhaps you're storing media which should be stored in blobs.

Perhaps you're using a non-scalable data structure, such as a
PersistentMapping or PersistentList.

Perhaps you're storing data in objects that aren't persistent at
all. In cases like that, the data is stored in the record of the
containing persistent object.

In any case, storing records this big is probably a bad idea.

If you insist and want to get rid of this warning, use the
large_record_size option of the ZODB.DB constructor (or the
large-record-size option in a configuration file) to specify a larger
size.

  warnings.warn(large_object_message % (obj.__class__, len(p)))
Killed

collective.firehose: for heavy-write scenarios