ctrl+shift+p filters: :st2 :st3 :win :osx :linux
Browse

Unit​Testing

by SublimeText ALL

Testing Sublime Text Packages

Details

  • 1.5.8
    0.10.6
  • github.​com
  • github.​com
  • 4 months ago
  • 29 minutes ago
  • 9 years ago

Installs

  • Total 2K
  • Win 1K
  • Mac 790
  • Linux 594
Oct 5 Oct 4 Oct 3 Oct 2 Oct 1 Sep 30 Sep 29 Sep 28 Sep 27 Sep 26 Sep 25 Sep 24 Sep 23 Sep 22 Sep 21 Sep 20 Sep 19 Sep 18 Sep 17 Sep 16 Sep 15 Sep 14 Sep 13 Sep 12 Sep 11 Sep 10 Sep 9 Sep 8 Sep 7 Sep 6 Sep 5 Sep 4 Sep 3 Sep 2 Sep 1 Aug 31 Aug 30 Aug 29 Aug 28 Aug 27 Aug 26 Aug 25 Aug 24 Aug 23 Aug 22
Windows 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mac 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 0 1 0 0 1 0 1 0 0
Linux 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 0 0 0 0

Readme

Source
raw.​githubusercontent.​com

UnitTesting

test codecov

This is a unittest framework for Sublime Text. It runs unittest testcases on local machines and via Github Actions. It also supports testing syntax_test files for the new sublime-syntax format and sublime-color-scheme files.

Sublime Text 4

Sublime Text 4 is now supported and testing works for Python 3.8 packages. Though test coverage wouldn't work unless Package Control supports Python 3.8.

Preparation

  1. Before testing anything, you have to install UnitTesting via Package Control.
  2. Your package!
  3. TestCases should be placed in test*.py under the directory tests (configurable, see below). The testcases are then loaded by TestLoader.discover.

Here are some small examples

Running Tests Locally

UnitTesting can be triggered via the command palette command UnitTesting. Enter the package name in the input panel and hit enter, a console should pop up and the tests should be running. To run only tests in particular files, enter <Package name>:<filename>. <filename> should be a unix shell wildcard to match the file names, <Package name>:test*.py is used in default.

You could run the command UnitTesting: Test Current Package to run the current package. The current package will be first reloaded by UnitTesting and then the tests will be executed.

It is also possible to generate test coverage report via coverage by using the command UnitTesting: Test Current Package with Coverage. The file .coveragerc is used to control the coverage configurations. If it is missing, UnitTesting will ignore the tests directory.

GitHub Actions

Basic: put the following in your workflow.

name: test

on: [push, pull_request]

jobs:
  run-tests:
    strategy:
      fail-fast: false
      matrix:
        st-version: [3, 4]
        os: ["ubuntu-latest", "macOS-latest", "windows-latest"]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v2
      - uses: SublimeText/UnitTesting/actions/setup@v1
        with:
          sublime-text-version: ${{ matrix.st-version }}
      - uses: SublimeText/UnitTesting/actions/run-tests@v1
        with:
          coverage: true
          codecov-upload: true

Remarks: actions are released in the branch v1. Minor changes will be pushed to the same branch unless there are breaking changes.

Testing syntax_test files

name: test-syntax

on: [push, pull_request]

jobs:
  run-syntax-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: SublimeText/UnitTesting/actions/setup@v1
      - uses: SublimeText/UnitTesting/actions/run-syntax-tests@v1

Check this for an example.

Deferred testing

Tests can be written using the Deferrable testcase, such that you are able to run sublime commands from your test cases and yield control to sublime text runtime and continue the execution later. Would be useful to test asynchronous codes. The idea was inspired by Plugin UnitTest Harness.

DeferrableTestCase is used to write the test cases. They are executed by the DeferringTextTestRunner and the runner expects not only regular test functions, but also generators. If the test function is a generator, it does the following

  • if the yielded object is a callable, the runner will evaluate the callable and check its returned value. If the result is not None, the runner continues the generator, if not, the runner will wait until the condition is met with the default timeout of 4s. The result of the callable can be also retrieved from the yield statement. The yielded object could be also a dictionary of the form {"condition": callable, timeout: timeout} to specify timeout in ms.

  • if the yielded object is an integer, say x, then it will continue the generator after x ms.

  • yield AWAIT_WORKER would yield to a task in the worker thread.

  • otherwise, a single yield would yield to a task in the main thread.

An example would be found in here.

Options

UnitTesting could be configured by providing the following settings in unittesting.json

name description default value
tests_dir the name of the directory containing the tests “tests”
pattern the pattern to discover tests “test*.py”
deferred whether to use deferred test runner true
verbosity verbosity level 2
output name of the test output instead of showing
in the panel
nil
show_reload_progress self explained true
reload_package_on_testing reloading package will increase coverage rate true
start_coverage_after_reload self explained, irrelevent if reload_package_on_testing is false false
coverage_on_worker_thread (experimental) false
generate_html_report generate coverage report for coverage false
capture_console capture stdout and stderr in the test output false
failfast stop early if a test fails false

Others

Add Test Current Package build

It is recommended to add the following in your .sublime-project file so that c+b would invoke the testing action.

"build_systems":
[
  {
    "name": "Test Current Package",
    "target": "unit_testing_current_package",
  }
]

Credits

Thanks guillermooo and philippotto for their early efforts in AppVeyor and Travis CI macOS support (though these services are not supported now).