Skip to content. | Skip to navigation

Personal tools

Navigation

You are here: Home / Tutorial / Test, Debug, Profile

Test, Debug, Profile

測試、偵錯的工具及技巧

測試應該要花最少心力

1. Static Analysis (flow, eslint) 2. JavaScript Test (jest, mocha, ava) 3. Webdriver Test (end-to-end)

先備知識

https://medium.com/@jaroslavkubicek/cypress-setting-up-the-first-acceptance-tests-in-gitlab-ci-pipeline-54b1c53f478b Test Driven Development Using Django, Selenium, and JavaScript CPU Thread Dump

Equivalence Partition vs Exception Partition 依屬性把輸入值分群 測試各自該回應的輸出 通常會有穩定的輸出值空間, 如果輸入例外值 也該有穩定的合理回應

程式碼圖像化分析: pycallgraph

plone.app.testing 屬於 Testing 環境的相依模組,寫在新版環境 setup.py 的 extra_require 區段裡。

zope.globalrequest five.globalrequest - 讀取目前生效的要求

在 develop.cfg 裡 [buildout] test-packages 指定 my.theme 或 collective.cover 之類的模組名稱,執行 bin/buildout -c develop.cfg 後,就會產生 bin/test 工具程式。執行 bin/test 會顯示 egg 使用狀況,搭配 -D 參數,或是 (Pdb) pp self.__dict__ 有助除錯

pdb tips plone.testing 和 ZopeTestCase 不該混用 ZopeTestCase 過時不被使用: plone.app.i18n link_redirect_view.txt CMFPlone: ZopeDocFileSuite

建議使用 plone_log 不要用 print 在 doctest 會遇到問題

context.plone_log("That's a log message")

Logging In-Depth Tutorial Logger Example

import logging
logger = logging.getLogger(PROJECTNAME)

Troubleshooting Plone

Typing: Strong vs Weak, Static vs Dynamic integer, int, float

Functional Programming Howto

plone.app.openid: As outlined on this bug report, unittest2 is no longer needed in packages targeted for python 2.7 or 3 only. Plone 5.1 努力將 unittest2 相依關係移除 zope.testrunner

Index UnitTest

Pyunit collective.taxonomy

import unittest
class TestIndexer(unittest.TestCase):
  layer = INTEGRATION_TESTING
  def setUp(self):
    self.portal = self.layer['portal']
    self.portal.portal_workflow.setDefaultChain(
      'simple_publication_workflow')
    applyProfile(self.portal, 'plone.app.contenttypes:plone-content')
    self.document = api.content.create(
      container=self.portal, type='Document', title='Doc')
  def test_indexer_with_field(self):
    portal_catalog = api.portal.get_tool('portal_catalog')
    utility = queryUtility(ITaxonomy, name='collective.taxonomy.test')
    taxonomy = utility.data
    taxonomy_test = schema.Set(
      title=u"taxonomy_test",
      required=False,
      value_type=schema.Choice(
        vocabulary=u"collective.taxonomy.taxonomies"),
    )
    portal_types = api.portal.get_tool('portal_types')
    fti = portal_types.get('Document')
    document_schema = fti.lookupSchema()
    schemaeditor = IEditableSchema(document_schema)
    schemaeditor.addField(taxonomy_test, name='taxonomy_test')
    notify(ObjectAddedEvent(taxonomy_test, document_schema))
    notify(FieldAddedEvent(fti, taxonomy_test))
    index = portal_catalog.Indexes['taxonomy_test']
    self.assertEqual(index.numObjects(), 0)

plone.app.testing

plone.app.testing.PLONE_FIXTURE does not include content types. It's used as the basis for both plone.app.contenttypes- and ATContentTypes-based fixtures.

plone.app.testing 已取消 autoinclude 功能 移植成 plone.app.testing 範例: uwosh.pfg.d2c collective.searchandreplace

利用 Builder Pattern 建立測試資料 gist example

correct dependency for plone.app.testing: plone.app.imagecropping circular imports

Test Porting: Products.AutoUserMakerPASPlugin

plone.app.events: Do Not Validate Incomplete DateTime

Products.ZCatalog: Test for Added CompositeIndex

# added a new zopepy part to the buildout.cfg

parts = ... zopepy

[zopepy]
recipe = zc.recipe.egg
interpreter = zopepy
eggs = Products.ZCatalog
dependent-scripts = true

# And than ran:

$ bin/buildout -N
$ bin/mkzopeinstance -d .
...

$ bin/runzope -C etc/zope.conf -X "debug-mode=on"

Monitoring Tools

利用監控工具來收集 log 數據:

System Monitoring

  • Munin is simply an amazing tool, whether you have 1 machine or 10000. It measures system statistics over time and will help you grasp the concept of what a "normal" system state is. It also allows custom plugins, and the plone community has already responded to that with things like munin.zope and there is also one for zope thread watcher called ZopeHealthWatcher. Ganglia is a similar package that offers much of the same functionality. Others?
  • Monit and munin are best friends. Monit does the same thing as munin when it comes to monitoring except that it doesn't collect data over time, and if something looks fishy it takes corrective action. What kind of action you say? Anything you ask it too! You can email alerts, automatically restart downed processes, monitor disk space, run bash script and the list goes on. How many times have you forgotten to rotate logs and run out of disk space? Monit could have told you weeks befre that happened. What about zope using too much memory? No problem, just have monit restart zope when it reaches a certain percentage (you can get some sweet performance this way). I put some examples here, but please don't copy them word for word - they are just for ideas! Similar products include nagios and supervisor, but most people will agree that monit will win your heart here.
  • Zope Health Watcher is perfect for finding out exactly which pages are taking a super long time (i.e. did an addIndex operation tie up your zope for all eternity?). It's simple in that it just lets you know at any time, which threads are rendering which requests. You'll be surprised how useful this can be.
  • Just found out about this gem that monitors the length of requests in zope 2.12+ and the top like functionality that goes with it. Haven't tried it but it looks hot hot hot!
  • Zenoss includes a lot of the features of munin and monit, and includes a bunch of network monitoring too. Again I have not tried it but if there are opinions out there feel free to share.

Error Monitoring

  • PloneErrorMonitoring
  • Google Analytics
  • Soup up the logging module to send emails (or do something else) when an error is triggered. Check out the maillinglogger package for quick and easy setup. For those that want to roll their own: there are things to consider. You really need to think when you are coding, is this really an error worthy of ending up in my inbox? If not, downgrade that message to warning. The goal is to have a system so stable you get as few emails as possible, and it is possible! Also remember that sending an email is by no means free. If your system is hitting the crapper and triggers 1000 emails per minute, not only is your email admin going to kill you, but the system is going to double over on itself. Buffering in memory help ease that pain by chunking those emails so you know whats wrong, just not * 1000. The downside to this is that important errors may not get to you until either the buffer is filled or you have a restart. In my experience though, most really important errors come in 100's, if not 1000's. If you keep this code nice and clean, you can use this in all of your packages, not just zope and plone.
  • If you don't want to get into the code to filter through the logs, checkout Arecibo and the plone buildout plugin, which we believe has not been moved to Andy's Github space at: https://github.com/andymckay/arecibo.

collective.monitor

Deadlock Locking without Deadlock Products.signalstack Multithread Program 會一次要求多個 lock 解法是 1. 維持順序 2. 指定 lock 3. 循序給予 lock

Memory Leaks

Unit Test: KeyError: u'profile-my.common:default' unittest.skip to Skip Tests

plone.recipe.codeanalysis: collective.googlenews

Behavior Driven Test: Cucumber corejet.core corejet.testrunner corejet.jira

Regression Test Benchmarking: siege manual

$ bin/test # run tests for all packages
$ bin/test -s Products.CMFPlone # for specific package
$ bin/alltests # take about 45 minutes

There are a lot of test isolation issues, so you can't just run bin/test and expect it to work. You can do one of the following:

  1. Run tests for a single package, with bin/test -s dotted.path.to.package. This isn't thorough, but it's better than nothing and jenkins will run all the tests once you commit.
  2. Run tests for all packages with bin/alltests, which runs each package's tests in a separate process (except for some groups configured in tests.cfg as being safe to run together). This takes about 45 minutes.

prints all control names on all forms, this way you can easily see what's the name (of if it's really missing).

[[c.name for c in f.controls] for f in browser.mech_browser.forms()]

Continuous Integration

Best Practices CircleCI + kubernetes 工具清單

Jenkins

installation DigitalOcean Frontend Example Ubuntu Install Core Package 預設透過 Jenkins 其餘透過 Travis 測試

Jenkins + Ansible

Pull Request Testing: Video Demo

  • go to http://jenkins.plone.org
  • log in with your Github user
  • click on the Pull Request 5.1 job
  • click on the huge menu link "Build with Parameters"
  • paste the pull request URL into the text field (if multiple pull requests need to be combined in one run, then one PR URL per line)
  • click on Build

Jenkins Config with Git: plone.dexterity example collective.recipe.backup

Mechanism for Sharing Coverage Data between tox environment plone.recipe.alltests 建立 testrunner script 適合 buildout 批次執行測試

Mock - Mocking and Testing Library

dagger: trying to repeat errors

Set Dependency for Testing Only, Rather than for every Install

extras_require = {
    'test': [
            'hexagonit.testing',
        ]
},

coveralls.io coverage: Products.PythonScripts Exclude README from Coverage

endless recursion 從 plone.registry + collective.solr 追查原因

工具

bin/plonectl debug

app.mysite['front-page'].title  #使用 Title() 是配合 Dublin Core 規格
app.mysite['front-page'].description
app.mysite['front-page'].text.raw

bin/instance console 參數範例 使用 app.Plone 或 app['Plone'] 來存取 Instance

$ bin/instance -RLOPlone/front-page debug

plonectl 前置設定內容

from Testing import makerequest
root = makerequest.makerequest(app)
site = root.mysite

admin = root.acl_users.getUserById('admin')
admin = admin.__of__(site.acl_users)

from AccessControl.SecurityManagement import newSecurityManager
newSecurityManager(None, admin)

from zope.site.hooks import setHooks
from zope.site.hooks import setSite
setHooks()
setSite(site)
site.setupCurrentSkin(site.REQUEST)

沒有 setSite() 在 createContentInContainer() 會造成 ComponentLookupError

newSecurityManager

from AccessControl.SecurityManagement import newSecurityManager
user = app.acl_users.getUser(user_name_or_id)
newSecurityManager(None, user.__of__(app.acl_users))

testrunner: collective.z3cform.datagridfield

script with collective.dexteritytextindexer

$ bin/instance run -Ozodb/path/to/Plone script.py

pdb basic commands

import pdb; pdb.set_trace()
(Pdb) relation
<z3c.relationfield.relation.RelationValue object at 0x7fc4fbca2b90>
(Pdb) pp relation.__dict__
{'__parent__': <Document at community-preserves>,
'_from_id': 1526966540,
'from_attribute': 'isReferencing',
'to_id' 252159100}

continue 可以繼續執行,可參考更多設定技巧

PDBDebugMode 取得 OID

>>> app._p_jar.get(oid)

dm.pdb.zpdb is an extension of Python's "pdb" with a bit of special support for Zope. Its "w[here]" command displays Zope's "__traceback_info__" and "__traceback_supplement__" information (such as you see them in tracebacks) and thereby you can easily see in which templates and scripts you are -- or whatever other information has been provided by the "__traceback_*" feature.

Debug ZODB Bloat ZODB5 + plone.app.folder

Testing Extra Requirements

Integration Testing for Mail

z3c.form event handle IObjectModifiedEvent 區隔範例 IRelationBrokenEvent SearchableText

Plone5: PLONE_APP_CONTENTTYPES_FIXTURE collective.cover Calendar

Travis CI

PyPI

Travis 有 15 分鐘執行測試檔案的時間限制,利用 Plone Unified Installer 可以改善這個問題。整合新方法 舊方法耗時

setup.py 檔案裡應該設定 extra_require={'test': ...}

travis.yml: collective.catalogcleanup collective.solr boiler plate funkload collective.lineage bibliograph.core collective.themecustomizer covertile.cycle2 wildcard.lockdown 利用 env 設定版本變數 不同的 sudo 設定值

Problem is that it then runs after bin/test-plone_addon, which leaves behind a test.plone_addon directory that the nosetests then also get run on, causing ImportErrors. Ah, but of course this directory only gets left behind when the command fails, so then there is no need to call createcoverage.

Travis CI support in collective.googleanalytics plone.app.imagecropping collective.formcriteria collective.googleanalytics bobtemplates.plone alm.solrindex

Products.EasyNewsletter no root http://about.travis-ci.org/docs/user/how-to-setup-and-trigger-the-hook-manually

IPv6 http://danielnouri.org/notes/2012/11/23/use-apt-get-to-install-python-dependencies-for-travis-ci 有時需要 mo 檔案來協助測試

TypeError: 'NoneType' object has no attribute '__getitem__': collective.fingerpointing

documentation example:

$ mkdir -p source && ln -s ../../documentation source/documentation
$ mkdir -p buildout-cache/downloads
$ python bootstrap-buildout.py
$ bin/buildout -N -t 3 buildout:checkout=documentation sources:documentation="fs documentation egg=false"

cache example: bobtemplates.plone #1 #2

Travis CI: mr.scripty collective.recipe.pip plone.jsonapi.core plone.recipe.varnish Theme Jenkins Make travis cache the egg directory of the generated package

Create Test Site Structure Simplify Python 3.5 Testing

Get rid of travis.cfg configuration as its use is no longer considered best practice

transmogrify.wordpress

同時測試多個版本

language: python
matrix:
  include:
    - name: 'Plone Tests'
      python: 2.7.14
      env: TEST_SUITE=plone
    - name: 'Guillotina Tests'
      python: 3.7
      dist: xenial
      env: TEST_SUITE=guillotina
    - name: 'Unit Tests'
      env: TEST_SUITE=unit

BDD Behavior Driven Development

Tool: Selenium / Cucumber / Behave

Feature: My first behave feature

  Scenario: Add two numbers
    Given I have two integers a and b
    When I add the numbers
    Then I print the addition result

Robot Framework

Basic Test Example Robot Framework and SeleniumLibrary for Plone Developers How to Find Jenkins Report Selenium to work with Firefox

datakurre robotsuite plone.app.multilingual example

說明文件 collective.nitf: ${CMFPLONE_VERSION} ${CMFPLONE_SELECTORS}

plone.app.robotframework: supersede plone.act access from python library integrate with Plone 4.3 robotshots

Skip Tests and Expected Failures

custom style test Click Element css=button.browse

collective robotsuite sphinxcontrib-robotframework generates both log files and hard errors with nitpicky-mode

Cross-browser test your Plone add-on with Robot Framework, Travis-CI and Sauce Labs buildout.plonetest

testing CSS with Robot Framework Remote Libraries

reset robot test fixture

register a view for functional test

To set an alternative user (preferably one with the Manager role) on the console, use the following code:

from AccessControl.SecurityManagement import newSecurityManager

site = app['Plone'] # Adjust as needed
# Assuming your username is 'admin', adjust as needed again:
user = app.acl_users.getUser('admin').__of__(site.acl_users) 
newSecurityManager(None, user)

First type in:

site_id = '<id of Plone site>' # Adjust as needed

then paste:

import transaction, pdb
from zope.interface import implementedBy
from zope.component import getUtility, queryUtility, queryAdapter
from Zope2 import debug
from Acquisition import aq_inner, aq_parent, aq_chain
from zope.app.component.hooks import setSite, getSiteManager
from Testing.makerequest import makerequest
from AccessControl.SecurityManagement import newSecurityManager, getSecurityManager

try:
    import readline
except ImportError:
    print "Module readline not available."
else:
    import rlcompleter
    readline.parse_and_bind("tab: complete")

app = makerequest(app)
site = app[site_id]
setSite(site)
user = app.acl_users.getUser('admin').__of__(site.acl_users)
newSecurityManager(None, user)

Now I have readline completion and everything I need to do some real damage in my sites!

Local Site Manager

Dexterity 使用 Object Specification Descriptor 來動態查詢 Factory Type Information,而查詢之前,必須先指定 Local Site 資訊,才能正確查詢物件的 interface 資訊。

IContextAwareDefaultFactory

robotframework and plone.app.testing

https://lionfacelemonface.wordpress.com/2011/02/08/profiling-plone-4-well-profiling-zope-while-running-plone-4

Robot Framework TinyMCE plone.app.imagecropping

http://kevinormbrek.blogspot.com/2012/08/getting-robot-framework-results-in.html

Looping Through ZODB Broken Objects

Selenium

Selenium IDE Web Crawler Example

Upload File with Selenium in Python

plone.app.toolbar UI testing Robot Framework and Selenium : pytest

robot framework and selenium2library

Robot Framework Sphinx

HTML5 Validation: collective.nitf

Zope Component Architecture 主要的 component 是全域物件,在 test suite 場合裡,想要建立獨立的測試環境變得不容易,通常要透過 test layer 來確保物件順利完成註冊工作

Quoting Parameter Values: chars presence of which in parameter value will be cause the value to be enclosed in double-tuotes

查詢物件的類別繼承狀況 superclasses mro

The Python Method Resolution Order defines the class search path used by Python to search for the right method to use in classes having multi-inheritance.

Insufficient Privileges

Interface 忘記被加在 configure.zcml 的 allowed_interface 或 function 沒有寫進 Interface 都可能發生。

GDB

檢查效能瓶頸的步驟範例

$ sudo apt-get install gdb

$ top

$ gdb /home/marr/python-2.x/bin/python

(gdb) attach 16355

(gdb) info threads

(gdb) thread 1

(gdb) call PyRun_SimpleString("import sys, traceback; sys.stderr=open('/tmp/tb','w',0); traceback.print_stack()")

Probes into Plone and Zope

# buildout.cfg for five.z3monitor

[instance]
...
zope-conf-additional =
  <product-config five.z3monitor>
    bind 0.0.0.0:8888
  </product-config>
[instance]
...
eggs +=
  collective.monitor
zcml +=
  collective.monitor

$ bin/instance monitor dbinfo main
$ bin/instance monitor objectcount
$ bin/instance monitor stats
$ bin/instance monitor help

$ echo 'dbinfo main' | nc -i 1 127.0.0.1 8888

$ telnet 127.0.0.1 8888

gdbgui 使用 diamond 把資料放到 graphite

python_gettext

PyCharm import global modules Django Plone Buildout with PyCharm

http://bsuttor.herokuapp.com/2015/08/probe-into-plone-and-zope/

With Jean-François Roche, we started to have a look on Products.ZNagios.
This product allow you to have some probes from Zope, you can ask your instance (live):

    Number of unresolved conflict on Zope
    CPU usage
    DB sizes
    Memory percent
    Uptime of Zope
    ...

You can access to the probes with a thread which listen on Zope  on port 8888 (in this conf). You just have to add zope-conf-additional in your buildout like this:

[instance]
...
zope-conf-additional =
  <product-config five.z2monitor>
    bind 0.0.0.0:8888
  </product-config>

If you want more information on this, you can see documentation of five.z2monitor package.
collective.monitor

I created this package for adding some probes into Plone. I created probes as Products.ZNagios. We used a zope interface for registering all probes (zc.z3monitor.interfaces.IZ3MonitorPlugin). In this package, I added these probes:

    count users
    count valid users (user logged during 3 last months)
    check if smtp is set up
    last login time of a user
    last time a plone or zope object was modified

How use it

Adding collective.monitor in your buildout in eggs and zcml instance section

[instance]
...
eggs +=
    ...
    collective.monitor
zcml +=
    ...
    collective.monitor

And also adding zope-conf-additional as explain above.

After this little config, you can access to probes with different way
1. bin/instance

After starting instance (bin/instance fg) you can access to probes with

./bin/instance monitor dbinfo main
./bin/instance monitor objectcount
./bin/instance monitor stats./bin/instance monitor help

2. netcat

After starting instance (bin/instance fg) you can access to probes with

echo 'dbinfo main' | nc -i 1 127.0.0.1 8888

3. telnet

After starting instance (bin/instance fg) you can access to probes with

$ telnet 127.0.0.1 8888

Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
last_modified_zope_object_time
2015/08/11 11:49:48.540729 GMT+2
Connection closed by foreign host.

With this package, you can make stats on your instance. We use github/python-diamond to collect and put informations from probes on github/graphite-project/graphite-web. It's very helpful for having state of our infrastructure.

SQL injection: pdbpp

sql_str = "SELECT * FROM users WHERE (name='" + username + "') and (pw = '" + password + "');"
username = "1' OR '1'='1"
password = "1' OR '1'='1"

好的方式

    result_set = db.select(
        "USER",
        where=f"account=$account AND password=$password",
        vars={"account": account, "password": password},
    )

UI Testing with Puppeteer

https://medium.com/@florian.hopf/integration-tests-at-ninja-van-5b6abb0ff59d https://medium.com/@mikecronin92/test-driven-development-is-dumb-fight-me-a38b3033280c https://medium.com/@daaaan/comments-in-your-code-730cfd1dde02 https://medium.com/@aThinkingBusinessAnalyst/how-to-quickly-compare-data-sets-76a694f6868a https://medium.com/geekculture/snoop-on-your-python-eaa2f157743a https://www.facebook.com/groups/pythontw/permalink/10161703292208438/ = python open()

k6

https://medium.com/nerd-for-tech/installing-k6-and-running-a-load-test-b1fd07161b37