Test, Debug, Profile
1. Static Analysis (flow, eslint) 2. JavaScript Test (jest, mocha, ava) 3. Webdriver Test (end-to-end)
先備知識
https://medium.com/@jaroslavkubicek/cypress-setting-up-the-first-acceptance-tests-in-gitlab-ci-pipeline-54b1c53f478b Test Driven Development Using Django, Selenium, and JavaScript CPU Thread Dump
Equivalence Partition vs Exception Partition 依屬性把輸入值分群 測試各自該回應的輸出 通常會有穩定的輸出值空間, 如果輸入例外值 也該有穩定的合理回應
程式碼圖像化分析: pycallgraph
plone.app.testing 屬於 Testing 環境的相依模組,寫在新版環境 setup.py 的 extra_require 區段裡。
zope.globalrequest five.globalrequest - 讀取目前生效的要求
在 develop.cfg 裡 [buildout] test-packages 指定 my.theme 或 collective.cover 之類的模組名稱,執行 bin/buildout -c develop.cfg 後,就會產生 bin/test 工具程式。執行 bin/test 會顯示 egg 使用狀況,搭配 -D 參數,或是 (Pdb) pp self.__dict__ 有助除錯。
pdb tips plone.testing 和 ZopeTestCase 不該混用 ZopeTestCase 過時不被使用: plone.app.i18n link_redirect_view.txt CMFPlone: ZopeDocFileSuite
建議使用 plone_log 不要用 print 在 doctest 會遇到問題
context.plone_log("That's a log message")
Logging In-Depth Tutorial Logger Example
import logging logger = logging.getLogger(PROJECTNAME)
Typing: Strong vs Weak, Static vs Dynamic integer, int, float
plone.app.openid: As outlined on this bug report, unittest2 is no longer needed in packages targeted for python 2.7 or 3 only. Plone 5.1 努力將 unittest2 相依關係移除 zope.testrunner
Index UnitTest
import unittest class TestIndexer(unittest.TestCase): layer = INTEGRATION_TESTING def setUp(self): self.portal = self.layer['portal'] self.portal.portal_workflow.setDefaultChain( 'simple_publication_workflow') applyProfile(self.portal, 'plone.app.contenttypes:plone-content') self.document = api.content.create( container=self.portal, type='Document', title='Doc') def test_indexer_with_field(self): portal_catalog = api.portal.get_tool('portal_catalog') utility = queryUtility(ITaxonomy, name='collective.taxonomy.test') taxonomy = utility.data taxonomy_test = schema.Set( title=u"taxonomy_test", required=False, value_type=schema.Choice( vocabulary=u"collective.taxonomy.taxonomies"), ) portal_types = api.portal.get_tool('portal_types') fti = portal_types.get('Document') document_schema = fti.lookupSchema() schemaeditor = IEditableSchema(document_schema) schemaeditor.addField(taxonomy_test, name='taxonomy_test') notify(ObjectAddedEvent(taxonomy_test, document_schema)) notify(FieldAddedEvent(fti, taxonomy_test)) index = portal_catalog.Indexes['taxonomy_test'] self.assertEqual(index.numObjects(), 0)
plone.app.testing
plone.app.testing.PLONE_FIXTURE does not include content types. It's used as the basis for both plone.app.contenttypes- and ATContentTypes-based fixtures.
plone.app.testing 已取消 autoinclude 功能 移植成 plone.app.testing 範例: uwosh.pfg.d2c collective.searchandreplace
利用 Builder Pattern 建立測試資料 gist example
correct dependency for plone.app.testing: plone.app.imagecropping circular imports
Test Porting: Products.AutoUserMakerPASPlugin
plone.app.events: Do Not Validate Incomplete DateTime
Products.ZCatalog: Test for Added CompositeIndex
# added a new zopepy part to the buildout.cfg parts = ... zopepy [zopepy] recipe = zc.recipe.egg interpreter = zopepy eggs = Products.ZCatalog dependent-scripts = true # And than ran: $ bin/buildout -N $ bin/mkzopeinstance -d . ... $ bin/runzope -C etc/zope.conf -X "debug-mode=on"
Monitoring Tools
利用監控工具來收集 log 數據:
System Monitoring
- Munin is simply an amazing tool, whether you have 1 machine or 10000. It measures system statistics over time and will help you grasp the concept of what a "normal" system state is. It also allows custom plugins, and the plone community has already responded to that with things like munin.zope and there is also one for zope thread watcher called ZopeHealthWatcher. Ganglia is a similar package that offers much of the same functionality. Others?
- Monit and munin are best friends. Monit does the same thing as munin when it comes to monitoring except that it doesn't collect data over time, and if something looks fishy it takes corrective action. What kind of action you say? Anything you ask it too! You can email alerts, automatically restart downed processes, monitor disk space, run bash script and the list goes on. How many times have you forgotten to rotate logs and run out of disk space? Monit could have told you weeks befre that happened. What about zope using too much memory? No problem, just have monit restart zope when it reaches a certain percentage (you can get some sweet performance this way). I put some examples here, but please don't copy them word for word - they are just for ideas! Similar products include nagios and supervisor, but most people will agree that monit will win your heart here.
- Zope Health Watcher is perfect for finding out exactly which pages are taking a super long time (i.e. did an addIndex operation tie up your zope for all eternity?). It's simple in that it just lets you know at any time, which threads are rendering which requests. You'll be surprised how useful this can be.
- Just found out about this gem that monitors the length of requests in zope 2.12+ and the top like functionality that goes with it. Haven't tried it but it looks hot hot hot!
- Zenoss includes a lot of the features of munin and monit, and includes a bunch of network monitoring too. Again I have not tried it but if there are opinions out there feel free to share.
Error Monitoring
- PloneErrorMonitoring
- Google Analytics
- Soup up the logging module to send emails (or do something else) when an error is triggered. Check out the maillinglogger package for quick and easy setup. For those that want to roll their own: there are things to consider. You really need to think when you are coding, is this really an error worthy of ending up in my inbox? If not, downgrade that message to warning. The goal is to have a system so stable you get as few emails as possible, and it is possible! Also remember that sending an email is by no means free. If your system is hitting the crapper and triggers 1000 emails per minute, not only is your email admin going to kill you, but the system is going to double over on itself. Buffering in memory help ease that pain by chunking those emails so you know whats wrong, just not * 1000. The downside to this is that important errors may not get to you until either the buffer is filled or you have a restart. In my experience though, most really important errors come in 100's, if not 1000's. If you keep this code nice and clean, you can use this in all of your packages, not just zope and plone.
- If you don't want to get into the code to filter through the logs, checkout Arecibo and the plone buildout plugin, which we believe has not been moved to Andy's Github space at: https://github.com/andymckay/arecibo.
Deadlock Locking without Deadlock Products.signalstack Multithread Program 會一次要求多個 lock 解法是 1. 維持順序 2. 指定 lock 3. 循序給予 lock
Unit Test: KeyError: u'profile-my.common:default' unittest.skip to Skip Tests
plone.recipe.codeanalysis: collective.googlenews
Behavior Driven Test: Cucumber corejet.core corejet.testrunner corejet.jira
Regression Test Benchmarking: siege manual
$ bin/test # run tests for all packages $ bin/test -s Products.CMFPlone # for specific package
$ bin/alltests # take about 45 minutes
There are a lot of test isolation issues, so you can't just run bin/test and expect it to work. You can do one of the following:
- Run tests for a single package, with bin/test -s dotted.path.to.package. This isn't thorough, but it's better than nothing and jenkins will run all the tests once you commit.
- Run tests for all packages with bin/alltests, which runs each package's tests in a separate process (except for some groups configured in tests.cfg as being safe to run together). This takes about 45 minutes.
prints all control names on all forms, this way you can easily see what's the name (of if it's really missing).
[[c.name for c in f.controls] for f in browser.mech_browser.forms()]
Continuous Integration
Best Practices CircleCI + kubernetes 工具清單
Jenkins
installation DigitalOcean Frontend Example Ubuntu Install Core Package 預設透過 Jenkins 其餘透過 Travis 測試
Pull Request Testing: Video Demo
- go to http://jenkins.plone.org
- log in with your Github user
- click on the Pull Request 5.1 job
- click on the huge menu link "Build with Parameters"
- paste the pull request URL into the text field (if multiple pull requests need to be combined in one run, then one PR URL per line)
- click on Build
Jenkins Config with Git: plone.dexterity example collective.recipe.backup
Mechanism for Sharing Coverage Data between tox environment plone.recipe.alltests 建立 testrunner script 適合 buildout 批次執行測試
Mock - Mocking and Testing Library
dagger: trying to repeat errors
Set Dependency for Testing Only, Rather than for every Install
extras_require = { 'test': [ 'hexagonit.testing', ] },
coveralls.io coverage: Products.PythonScripts Exclude README from Coverage
endless recursion 從 plone.registry + collective.solr 追查原因
工具
bin/plonectl debug
app.mysite['front-page'].title #使用 Title() 是配合 Dublin Core 規格 app.mysite['front-page'].description app.mysite['front-page'].text.raw
bin/instance console 參數範例 使用 app.Plone 或 app['Plone'] 來存取 Instance
$ bin/instance -RLOPlone/front-page debug
plonectl 前置設定內容
from Testing import makerequest root = makerequest.makerequest(app) site = root.mysite admin = root.acl_users.getUserById('admin') admin = admin.__of__(site.acl_users) from AccessControl.SecurityManagement import newSecurityManager newSecurityManager(None, admin) from zope.site.hooks import setHooks from zope.site.hooks import setSite setHooks() setSite(site) site.setupCurrentSkin(site.REQUEST)
沒有 setSite() 在 createContentInContainer() 會造成 ComponentLookupError
from AccessControl.SecurityManagement import newSecurityManager user = app.acl_users.getUser(user_name_or_id) newSecurityManager(None, user.__of__(app.acl_users))
testrunner: collective.z3cform.datagridfield
script with collective.dexteritytextindexer
$ bin/instance run -Ozodb/path/to/Plone script.py
import pdb; pdb.set_trace()
(Pdb) relation <z3c.relationfield.relation.RelationValue object at 0x7fc4fbca2b90> (Pdb) pp relation.__dict__ {'__parent__': <Document at community-preserves>, '_from_id': 1526966540, 'from_attribute': 'isReferencing', 'to_id' 252159100}
continue 可以繼續執行,可參考更多設定技巧。
>>> app._p_jar.get(oid)
dm.pdb.zpdb is an extension of Python's "pdb" with a bit of special support for Zope. Its "w[here]" command displays Zope's "__traceback_info__" and "__traceback_supplement__" information (such as you see them in tracebacks) and thereby you can easily see in which templates and scripts you are -- or whatever other information has been provided by the "__traceback_*" feature.
Debug ZODB Bloat ZODB5 + plone.app.folder
z3c.form event handle IObjectModifiedEvent 區隔範例 IRelationBrokenEvent SearchableText
Plone5: PLONE_APP_CONTENTTYPES_FIXTURE collective.cover Calendar
Travis CI
Travis 有 15 分鐘執行測試檔案的時間限制,利用 Plone Unified Installer 可以改善這個問題。整合新方法 舊方法耗時
setup.py 檔案裡應該設定 extra_require={'test': ...}
travis.yml: collective.catalogcleanup collective.solr boiler plate funkload collective.lineage bibliograph.core collective.themecustomizer covertile.cycle2 wildcard.lockdown 利用 env 設定版本變數 不同的 sudo 設定值
Problem is that it then runs after bin/test-plone_addon, which leaves behind a test.plone_addon directory that the nosetests then also get run on, causing ImportErrors. Ah, but of course this directory only gets left behind when the command fails, so then there is no need to call createcoverage.
Travis CI support in collective.googleanalytics plone.app.imagecropping collective.formcriteria collective.googleanalytics bobtemplates.plone alm.solrindex
Products.EasyNewsletter no root http://about.travis-ci.org/docs/user/how-to-setup-and-trigger-the-hook-manually
IPv6 http://danielnouri.org/notes/2012/11/23/use-apt-get-to-install-python-dependencies-for-travis-ci 有時需要 mo 檔案來協助測試
TypeError: 'NoneType' object has no attribute '__getitem__': collective.fingerpointing
$ mkdir -p source && ln -s ../../documentation source/documentation $ mkdir -p buildout-cache/downloads $ python bootstrap-buildout.py $ bin/buildout -N -t 3 buildout:checkout=documentation sources:documentation="fs documentation egg=false"
cache example: bobtemplates.plone #1 #2
Travis CI: mr.scripty collective.recipe.pip plone.jsonapi.core plone.recipe.varnish Theme Jenkins Make travis cache the egg directory of the generated package
Create Test Site Structure Simplify Python 3.5 Testing
Get rid of travis.cfg configuration as its use is no longer considered best practice
language: python matrix: include: - name: 'Plone Tests' python: 2.7.14 env: TEST_SUITE=plone - name: 'Guillotina Tests' python: 3.7 dist: xenial env: TEST_SUITE=guillotina - name: 'Unit Tests' env: TEST_SUITE=unit
BDD Behavior Driven Development
Tool: Selenium / Cucumber / Behave
Feature: My first behave feature Scenario: Add two numbers Given I have two integers a and b When I add the numbers Then I print the addition result
Robot Framework
Basic Test Example Robot Framework and SeleniumLibrary for Plone Developers How to Find Jenkins Report Selenium to work with Firefox
datakurre robotsuite plone.app.multilingual example
說明文件 collective.nitf: ${CMFPLONE_VERSION} ${CMFPLONE_SELECTORS}
plone.app.robotframework: supersede plone.act access from python library integrate with Plone 4.3 robotshots
Skip Tests and Expected Failures
custom style test Click Element css=button.browse
collective robotsuite sphinxcontrib-robotframework generates both log files and hard errors with nitpicky-mode
Cross-browser test your Plone add-on with Robot Framework, Travis-CI and Sauce Labs buildout.plonetest
testing CSS with Robot Framework Remote Libraries
register a view for functional test
To set an alternative user (preferably one with the Manager
role) on the console, use the following code:
from AccessControl.SecurityManagement import newSecurityManager site = app['Plone'] # Adjust as needed # Assuming your username is 'admin', adjust as needed again: user = app.acl_users.getUser('admin').__of__(site.acl_users) newSecurityManager(None, user)
First type in:
site_id = '<id of Plone site>' # Adjust as needed
then paste:
import transaction, pdb from zope.interface import implementedBy from zope.component import getUtility, queryUtility, queryAdapter from Zope2 import debug from Acquisition import aq_inner, aq_parent, aq_chain from zope.app.component.hooks import setSite, getSiteManager from Testing.makerequest import makerequest from AccessControl.SecurityManagement import newSecurityManager, getSecurityManager try: import readline except ImportError: print "Module readline not available." else: import rlcompleter readline.parse_and_bind("tab: complete") app = makerequest(app) site = app[site_id] setSite(site) user = app.acl_users.getUser('admin').__of__(site.acl_users) newSecurityManager(None, user)
Now I have readline completion and everything I need to do some real damage in my sites!
Local Site Manager
Dexterity 使用 Object Specification Descriptor 來動態查詢 Factory Type Information,而查詢之前,必須先指定 Local Site 資訊,才能正確查詢物件的 interface 資訊。
robotframework and plone.app.testing
Robot Framework TinyMCE plone.app.imagecropping
http://kevinormbrek.blogspot.com/2012/08/getting-robot-framework-results-in.html
Looping Through ZODB Broken Objects
Selenium
Selenium IDE Web Crawler Example
Upload File with Selenium in Python
plone.app.toolbar UI testing Robot Framework and Selenium : pytest
robot framework and selenium2library
HTML5 Validation: collective.nitf
Zope Component Architecture 主要的 component 是全域物件,在 test suite 場合裡,想要建立獨立的測試環境變得不容易,通常要透過 test layer 來確保物件順利完成註冊工作。
Quoting Parameter Values: chars presence of which in parameter value will be cause the value to be enclosed in double-tuotes
The Python Method Resolution Order defines the class search path used by Python to search for the right method to use in classes having multi-inheritance.
Insufficient Privileges
Interface 忘記被加在 configure.zcml 的 allowed_interface 或 function 沒有寫進 Interface 都可能發生。
GDB
檢查效能瓶頸的步驟範例
$ sudo apt-get install gdb
$ top
$ gdb /home/marr/python-2.x/bin/python
(gdb) attach 16355
(gdb) info threads
(gdb) thread 1
(gdb) call PyRun_SimpleString("import sys, traceback; sys.stderr=open('/tmp/tb','w',0); traceback.print_stack()")
# buildout.cfg for five.z3monitor [instance] ... zope-conf-additional = <product-config five.z3monitor> bind 0.0.0.0:8888 </product-config>
[instance] ... eggs += collective.monitor zcml += collective.monitor $ bin/instance monitor dbinfo main $ bin/instance monitor objectcount $ bin/instance monitor stats $ bin/instance monitor help $ echo 'dbinfo main' | nc -i 1 127.0.0.1 8888 $ telnet 127.0.0.1 8888
gdbgui 使用 diamond 把資料放到 graphite
PyCharm import global modules Django Plone Buildout with PyCharm
http://bsuttor.herokuapp.com/2015/08/probe-into-plone-and-zope/ With Jean-François Roche, we started to have a look on Products.ZNagios. This product allow you to have some probes from Zope, you can ask your instance (live): Number of unresolved conflict on Zope CPU usage DB sizes Memory percent Uptime of Zope ... You can access to the probes with a thread which listen on Zope on port 8888 (in this conf). You just have to add zope-conf-additional in your buildout like this: [instance] ... zope-conf-additional = <product-config five.z2monitor> bind 0.0.0.0:8888 </product-config> If you want more information on this, you can see documentation of five.z2monitor package. collective.monitor I created this package for adding some probes into Plone. I created probes as Products.ZNagios. We used a zope interface for registering all probes (zc.z3monitor.interfaces.IZ3MonitorPlugin). In this package, I added these probes: count users count valid users (user logged during 3 last months) check if smtp is set up last login time of a user last time a plone or zope object was modified How use it Adding collective.monitor in your buildout in eggs and zcml instance section [instance] ... eggs += ... collective.monitor zcml += ... collective.monitor And also adding zope-conf-additional as explain above. After this little config, you can access to probes with different way 1. bin/instance After starting instance (bin/instance fg) you can access to probes with ./bin/instance monitor dbinfo main ./bin/instance monitor objectcount ./bin/instance monitor stats./bin/instance monitor help 2. netcat After starting instance (bin/instance fg) you can access to probes with echo 'dbinfo main' | nc -i 1 127.0.0.1 8888 3. telnet After starting instance (bin/instance fg) you can access to probes with $ telnet 127.0.0.1 8888 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. last_modified_zope_object_time 2015/08/11 11:49:48.540729 GMT+2 Connection closed by foreign host.
With this package, you can make stats on your instance. We use github/python-diamond to collect and put informations from probes on github/graphite-project/graphite-web. It's very helpful for having state of our infrastructure.
sql_str = "SELECT * FROM users WHERE (name='" + username + "') and (pw = '" + password + "');"
username = "1' OR '1'='1" password = "1' OR '1'='1"
好的方式
result_set = db.select( "USER", where=f"account=$account AND password=$password", vars={"account": account, "password": password}, )
UI Testing with Puppeteer
https://medium.com/@florian.hopf/integration-tests-at-ninja-van-5b6abb0ff59d https://medium.com/@mikecronin92/test-driven-development-is-dumb-fight-me-a38b3033280c https://medium.com/@daaaan/comments-in-your-code-730cfd1dde02 https://medium.com/@aThinkingBusinessAnalyst/how-to-quickly-compare-data-sets-76a694f6868a https://medium.com/geekculture/snoop-on-your-python-eaa2f157743a https://www.facebook.com/groups/pythontw/permalink/10161703292208438/ = python open()
k6
https://medium.com/nerd-for-tech/installing-k6-and-running-a-load-test-b1fd07161b37