James Long
928 lines
94 KiB
Plaintext
928 lines
94 KiB
Plaintext
On the Evolution of Python Test Cases into
|
||
Property-based Tests
|
||
Cindy Wauters Ruben Opdebeeck Coen De Roover
|
||
Software Languages Lab Software Languages Lab Software Languages Lab
|
||
Vrije Universiteit Brussel Vrije Universiteit Brussel Vrije Universiteit Brussel
|
||
Brussels, Belgium Brussels, Belgium Brussels, Belgium
|
||
cindy.suzy.wauters@vub.be ruben.denzel.opdebeeck@vub.be coen.de.roover@vub.be
|
||
|
||
|
||
|
||
Abstract—Conventionally, unit tests exercise their unit under Scala2 , Python3 and JavaScript4 . Proposing domain-specific
|
||
test on predetermined input to validate its behavior. The advent input generation strategies and invariant properties, research
|
||
of property-based testing frameworks has led to unit tests that initiatives have brought property-based testing to challenging
|
||
exercise the unit under test on randomly-generated inputs, for
|
||
which the unit’s behavior has to satisfy developer-specified invari- application domains such as telecom software [2], distributed
|
||
ant properties. The promise of increased coverage and stronger synchronization services [19], neural networks [33], and cyber-
|
||
validation may lead developers to evolve existing conventional physical systems [13].
|
||
unit tests into property-based ones. This paper reports on the Despite these successes, property-based tests often consti-
|
||
results of an empirical study of 257 property-based unit tests tute only a fraction of the test suite. According to a 2023
|
||
(PBT) from 64 open-source repositories that have evolved from a
|
||
conventional unit test. The study examines each PBT-introducing JetBrains survey [22], the most common PBT framework
|
||
commit for change patterns, and for the type of features of for Python, Hypothesis, was only used by 5% of Python
|
||
property-based testing frameworks that are adopted. Next, it developers. To bring the benefits of property-based testing
|
||
investigates the impact of PBT adoption by running test cases to to more projects, researchers have proposed tool support for
|
||
compute changes in code coverage and test failures. Finally, the creating property-based tests [16], [38]. However, there is a
|
||
study investigates the evolution of the newly-introduced PBT by
|
||
comparing its first to its most recent version. The study’s findings gap in the research when it comes to the evolution of existing
|
||
include that 37 out of 257 evolutions required no changes to the test cases into property-based tests. Therefore, in this paper, we
|
||
body of the test. When changes are required, these are most investigate these evolutions, their impact, and the maintenance
|
||
likely to target setup code followed by oracle code. Over half of of the evolved test cases. By doing so, we aim to provide
|
||
the studied PBTs use off-the-shelf input generators, yet see an valuable insights for testers, tool builders, and researchers
|
||
improvement in code coverage by an average of 4 statements.
|
||
Additionally, for 5 test cases, the PBT found a counterexample alike. With this paper, we make the following contributions:
|
||
where the original unit test passed. Finally, 80% of the evolved • We collect a dataset of 257 test case evolutions across
|
||
test cases are still present in the current version of the project. 64 open-source GitHub repositories written in Python.
|
||
Index Terms—Empirical study, software testing, property-
|
||
based testing, code coverage Our data collection targets Python projects as, at the time
|
||
of writing, it is one of the most popular programming
|
||
I. I NTRODUCTION languages [21]. We provide a replication package [40]
|
||
Conventionally, unit tests take an example-based form in containing the resulting dataset and the implementation
|
||
which the unit under test is exercised on predetermined input. of all subsequent analyses.
|
||
For a unit test that should be run with several inputs, param- • An empirical analysis of the changes required to rewrite
|
||
|
||
eterized unit testing [35] frameworks support substituting the a conventional unit test into a property-based test, as well
|
||
example input by a parameter for which the developer provides as whether test cases evolved afterwards.
|
||
a predetermined collection of values. When enumerating these • An empirical analysis of the impact of these evolutions
|
||
|
||
input values becomes intractable, property-based testing [6] on the system under test in terms of changes in code
|
||
frameworks support generating them randomly according to coverage and test failures.
|
||
a developer-specified input generation strategy. For each of The remainder of this paper is structured as follows. Sec-
|
||
the generated inputs, the property-based test (PBT) will check tion II provides a motivating example and introduces the
|
||
that the behavior of the unit under test satisfies a developer- research questions. Section III details the dataset of real-world
|
||
specified invariant property. test evolutions we will be working with. Then, Section IV,
|
||
The QuickCheck framework [6] for the Haskell program- Section V, and Section VI present our research questions. For
|
||
ming language was the first to support property-based testing. each of the three research questions, we detail the research
|
||
Since then, more than 35 property-based testing tools [18]
|
||
have been introduced for various languages, such as Java1 , 2 https://github.com/typelevel/scalacheck
|
||
3 https://github.com/HypothesisWorks/hypothesis
|
||
1 https://github.com/jqwik-team/jqwik 4 https://github.com/dubzzz/fast-check
|
||
def test_encode_decode(): However, it can be difficult for developers to come up
|
||
txt = "hello world" with invariant properties to test [14]. The Hypothesis Quick-
|
||
assert decode(encode(txt)) == txt start [43] hints at existing parameterized unit tests (PUTs) and
|
||
existing sources of randomness in a test suite for inspiration.
|
||
Listing 1: An example-based test for encode and decode However, to our knowledge, no research has looked into real-
|
||
world test case evolutions such as the one described above. In
|
||
import pytest this paper, we answer the following research questions:
|
||
• RQ1: How do conventional unit tests evolve into
|
||
testdata = ["hello world", "test -", "(/test)"]
|
||
@pytest.mark.parametrize("txt", testdata) property-based tests? In this research question, we
|
||
def test_encode_decode(txt): investigate real-world commits for the changes required
|
||
assert decode(encode(txt)) == txt to rewrite a conventional example-based unit test or
|
||
parameterized unit test into a property-based one.
|
||
Listing 2: A parameterized unit test for encode and decode • RQ2: What is the impact of adopting property-based
|
||
testing? To investigate the impact of these test case
|
||
evolutions, we compute the difference in code coverage
|
||
method, the findings, and present an objective analysis of before and after the rewrites as well as the test failures.
|
||
these findings. Section VII discusses the actionable insights • RQ3: How are property-based tests maintained?
|
||
gained from the findings across the research questions. Here To assess the effort required to maintain the resulting
|
||
we also describe the threats to validity and their mitigation. In property-based tests, we investigate whether there are
|
||
Section VIII we position our work and compare it to current changes made to the test cases after their rewriting into
|
||
research into property-based testing. a property-based test, or if they are removed.
|
||
|
||
II. M OTIVATING E XAMPLE AND R ESEARCH Q UESTIONS III. DATASET
|
||
In order to study real-world unit test evolutions, we first
|
||
Test cases may evolve over time. To illustrate such an evo-
|
||
need a dataset of open-source repositories that contain at least
|
||
lution, consider an encode function and its inverse decode.
|
||
one property-based unit test (PBT) that started as either an
|
||
Listing 1 depicts an example-based unit test (EBT) checking
|
||
example-based unit test (EBT) or as a parameterized unit test
|
||
that decoding the encoding of "hello world" results in
|
||
(PUT). Our data collection will start from Python repositories
|
||
the same string. This unit test validates the behavior of the
|
||
that have a dependency on the Hypothesis [27] framework
|
||
unit under test for a single developer-provided example string.
|
||
for property-based testing. While other empirical studies have
|
||
Using Pytest [23], the most common Python testing frame- collected such repositories before (e.g., [32], [39]), we collect
|
||
work, the example-based unit test from Listing 1 can be gener- our own to ensure the dataset contains ample instances of test
|
||
alized into a parameterized unit test [36]. Listing 2 depicts an case evolutions.
|
||
example generalization in which a parameter txt substitutes 1) Data collection: We first use the GitHub API to find
|
||
for the earlier example input. Three possible values have been open-source Python repositories that have a dependency on
|
||
provided for the parameter, thereby validating the behavior on Hypothesis. In order to exclude toy projects, we only include
|
||
predetermined strings that contain special characters. projects that have at least three stars. We also filter out larger
|
||
Using Hypothesis [26], [27], the most popular property- projects with more than 10 stars. While these projects can
|
||
based testing framework for Python, the test case can be be valuable to study, they are also difficult for an outsider
|
||
generalized further. Listing 3 depicts the resulting property- to understand and reason about. We do not consider those
|
||
based test, specifying that the composition of decoding after projects suitable for a deep manual investigation, as required
|
||
encoding is the identity operation for all strings generated for this empirical study section. In total, we considered 1388
|
||
using st.text. Hypothesis will automatically generate mul- repositories that have a dependency on Hypothesis and satisfy
|
||
tiple strings, each of which will be encoded and decoded and our inclusion criteria.
|
||
compared to the original string. This way, the property-based Next, we use PyDriller [34] to find all PBTs in a repository
|
||
unit test might uncover defects for edge cases the developer did that were once either an EBT or a PUT, and are now a PBT.
|
||
not enumerate in the example-based nor in the parameterized To this end, we rely on the presence of @given decorators in
|
||
version of the unit test. a file with a PBT. The GitHub API alone does not suffice, as
|
||
noted by prior empirical studies into real-world Hypothesis
|
||
from hypothesis import given, strategies as st usage (e.g., [8], [32], [39]), due to the existence of other
|
||
libraries named Hypothesis. The @given decorator in a test
|
||
@given(st.text()) file, in contrast, denotes the entry point of a Hypothesis-based
|
||
def test_encode_decode(txt): PBT. If none exists, the repository does not contain a PBT, and
|
||
assert decode(encode(txt)) == txt is excluded. If we find at least one PBT in a file, we investigate
|
||
all previous versions of the file. In case the PBT exists in the
|
||
Listing 3: A property-based test for encode and decode initial commit of the file, PBTs have always existed in that
|
||
# LOC Python # evolved test cases # commits 1 def test_tolerance_sign_EBT():
|
||
Mean 7575 4.02 923 2 r = R.from_min_max(3, 2)
|
||
Std Dev. 17147 6.66 1996 3 assert r.tolerance == 0.2
|
||
4
|
||
median 2607 2 294
|
||
5
|
||
1st Quar. 1178 1 121.5 6 @given( unit=st.sampled_from((U, I, P, R)),
|
||
3rd Quar. 5346 4 680.5 7 a=st.floats(allow_nan=False),
|
||
Min 164 1 15 8 b=st.floats(allow_nan=False))
|
||
9 def test_tolerance_sign_PBT(unit, a, b):
|
||
Max 109516 39 10274
|
||
10 eu = unit.from_min_max(a, b)
|
||
assert eu.tolerance >= 0
|
||
TABLE I: Characteristics of the 64 repositories investigated 11
|
||
in this paper. Lines of Code computed by Cloc [10].
|
||
Listing 4: A real test case before and after becoming a PBT.
|
||
Some simplifications applied for readability. Case taken from
|
||
https://github.com/wese3112/eecalpy
|
||
file, and we do not consider this file relevant for investigating
|
||
test case evolution. In case there is a previous version of the
|
||
file in which the PBT did not exist, while it does in a newer
|
||
end, we categorize all changes to the body of the test case into
|
||
version, we manually investigate the last version before the
|
||
the following categories (four per type):
|
||
introduction with the first version after the introduction. We
|
||
• Changes to the oracle code: non-semantic changes, se-
|
||
perform this step manually to ensure that we also retrieve
|
||
evolutions of tests that have been renamed. Additionally, we mantic changes, additional oracles, removal of oracles.
|
||
• Changes to the setup code: non-semantic changes, se-
|
||
take note of whether the transformation in question is from an
|
||
EBT to a PBT, or from a PUT to a PBT. We recognize PUTs as mantic changes, additional setup code, removal of setup
|
||
unit tests adorned with the @pytest.mark.parametrize code.
|
||
decorator. This renders their recognition purely syntactic. To delimit test oracle code, we follow the definition of Barr
|
||
2) Characteristics of the dataset: Out of the 1388 et al. [3]. It is the part of the code that decides whether
|
||
Hypothesis-dependent GitHub repositories initially discovered, the system under test behaves as expected. This does not
|
||
575 have at least one PBT present. Upon filtering for test case necessarily mean that only (nor all) the code in an assert
|
||
evolutions, we identified 64 repositories that feature at least statement is part of the oracle code. Code that defines the
|
||
one evolution from an EBT or PUT into a PBT —amounting oracle is included too.
|
||
to 257 PBT before-after mappings. For each PBT, we have As setup code, we consider those parts of the body of
|
||
a mapping from the test before the PBT introduction to the the test case that create or retrieve values through which the
|
||
test after the PBT introduction. For a minority of PBTs, the behavior of the system under test will be validated. In other
|
||
test before its introduction was split into multiple PBTs. On words, it sets up the system under test. This is similar to the
|
||
the other hand, some tests were merged into a single PBT. In definition of a test fixture by Meszaros [28]. This is sometimes
|
||
these cases, we consider the same PBT multiple times (each also known as test context. However, we only consider the
|
||
time with a different mapping). Our 257 before-after mappings code inside the body of the test case.
|
||
therefore consist of 252 unique tests before PBT introduction As a non-semantic change to the body of a test case,
|
||
and 250 unique PBTs. Of the total 257 test case evolutions, we consider straightforward refactorings such as changing a
|
||
182 test cases follow the EBT to PBT evolution, whereas 75 constant to a value generated by the input generation strategy.
|
||
follow the PUT to PBT evolution. Table I presents summary For semantic changes, in contrast, we consider anything that
|
||
statistics of the distribution of the data across the most recent changes the semantics of a test case (e.g., extra calculations
|
||
version of the repositories. Included are the lines of code in within existing code). An addition or removal of setup code
|
||
Python (blank lines and comments not included), the number means any lines added removed, that can be considered setup
|
||
of investigated test case evolutions, and the total number of code. An addition or removal of oracle code is the addition
|
||
commits per repository. of, for example, a new assert statement, or the removal of an
|
||
old one.
|
||
IV. RQ1: H OW DO CONVENTIONAL UNIT TESTS EVOLVE A test case can evolve into a PBT through several of
|
||
INTO PROPERTY- BASED TESTS ? these types of changes. To illustrate, consider Listing 4 which
|
||
depicts one of the test cases in our dataset before and after
|
||
RQ1 investigates how PBTs can evolve out of conventional evolving into a PBT. Line 2, as well as the r.tolerance
|
||
test cases. To this end, we manually investigate the 257 afore- in line 3 is considered the setup code: these lines define
|
||
mentioned real-world before-after mappings in the dataset. what will be tested later. The assert statement, excluding
|
||
r.tolerance, is oracle code. This test case changes in two
|
||
A. Research Method
|
||
different ways. First, a non-semantic change to the setup code:
|
||
1) Changes to the test case body: First, we consider how on line 10 and 11, we see a renaming of r to eu. Additionally,
|
||
test cases are updated when they evolve into a PBT. To this hardcoded values R, 3, and 2 are replaced with the parameters
|
||
that will now be generated data. Second, a semantic change • Control: assume, @note, @event, target, used in
|
||
in the oracle code: as the values are not hardcoded anymore, the test case to modify how it is treated by the test runner,
|
||
an exact expected value is not known anymore, and the oracle or to further document it.
|
||
needs to be updated. The change from == to >= makes this a 3) Input generation strategies: As the input generation
|
||
semantic change. strategy is an important part of every PBT, we also categorize
|
||
Furthermore, it is also possible that the test case changed the strategies chosen at migration time as follows:
|
||
beyond recognition when it was rewritten into a PBT. During • Simple: strategies that generate standard values (e.g.,
|
||
our manual investigation, we mark such test cases as replaced, integers, floats, strings), and collections thereof (e.g., lists,
|
||
rather than updated, if we cannot recognize it within the PBT tuples). Strategies that compose standard values into a
|
||
anymore. Some large-scale studies into test evolution have user-defined data type are classified in the next category
|
||
used thresholds on automatically-computed similarity metrics instead.
|
||
instead (e.g., 66% used in [5] and [11]), but these make less • Custom, with reuse: strategies that are custom to the
|
||
sense when applied to small test cases instead of large files. project, but reused across multiple test cases.
|
||
We therefore rely on human judgement of the labelers instead. • Custom, no reuse: strategies that are custom to the
|
||
If the labelers cannot identify similar code in the test case project, but not reused across multiple test cases
|
||
before and after the evolution, and especially in their assert • Other: usage of third-party libraries for input generation.
|
||
statements, the test is tagged as different test cases. For example, to generate NumPy values.
|
||
The labeling was performed individually by the first and We assign each test case one of these four categories.
|
||
second authors of this paper. After a first round of labeling, Custom strategies encompass cases where developers define
|
||
we assessed inter-rater agreement using Cohen’s Kappa [7] their own, complex, input generators for the test cases. We also
|
||
calculated for each category, obtaining a mean Kappa of 0.475, consider any usage of builds and from_type in this cate-
|
||
signifying “moderate agreement” [25]. To avoid systematic gory, as developers use them to generate custom user-defined
|
||
disagreements, the two labelers discussed and resolved dis- data types. When a test generates both values of standard and
|
||
agreements on a random sample of 20% of the entire dataset to user-defined data types, we consider its generation strategy
|
||
establish a shared understanding. For instance, a recurring dis- custom overall.
|
||
agreement was the presence of setup code in assert statements.
|
||
After resolving these differences, the labelers individually B. Results
|
||
performed a second round of labeling on the remaining 80%, 1) Changes to the test case body: We first discuss the
|
||
resulting in a mean Kappa of 0.727, signifying “substantial changes that conventional unit tests undergo when rewritten
|
||
agreement” [25]. The remaining disagreements were discussed into a property-based test. Figure 1 shows the results for both
|
||
among the two labelers to obtain consensus. EBTs and PUTs that evolved into a PBT.
|
||
2) Features of the property-based testing framework that For test cases that evolved from EBTs in our dataset, it is
|
||
are adopted: Property-based testing frameworks often offer especially common (117/182) for the setup code to be changed
|
||
features to customize a test case. The @example decorator of in a non-semantic way (for example, replacing a hardcoded
|
||
Hypothesis, for example, enables specifying that a value needs value to instead refer to the input generation strategy, as
|
||
to be tested on every run of the test case. This can be used shown in Listing 4). The second most common change is for
|
||
with known edge cases that the developer is aware of. The the oracle code to be updated with a non-semantic change.
|
||
@settings decorator lets developers specify test budgets, Other common changes are additions of oracle code, semantic
|
||
the maximum number of examples to be generated, etc. It is changes to oracle code, and removal of oracle code or setup
|
||
also possible to reproduce test failures by re-using the random code. Additionally, 12% of cases changed beyond recognition.
|
||
input generation seed that found the counterexample (e.g., the Tests that evolve from PUTs are a bit different. Here,
|
||
input for which the property did not hold). we see fewer changes in the test cases overall. The most
|
||
We track the use of these features at the time the PBT was common change is the addition of oracle code, followed by
|
||
introduced. By doing so, we aim to discover whether devel- the addition of setup code, and semantic changes to the oracle
|
||
opers are using them from the onset. We use the following code. Notably, 35/75 test case evolutions from PUTs in our
|
||
categorization according to the Hypothesis API [42]: dataset required no change at all in the body of the test case
|
||
• Strategies: @given to generate inputs. The next para- (in comparison to 2 stemming from EBTs). Many PUTs might
|
||
graph categorizes the input strategies specified through already be testing a property, especially if they contain no
|
||
this decorator further. hardcoded expected values in the parameter. This means that
|
||
• Explicit inputs: @example, which enable the tester to evolving them into a property-based test can be less labor-
|
||
provide inputs to test each run. intensive for developers.
|
||
• Reproducing inputs: @reproduce_failure and 2) Features of the property-based testing framework that
|
||
@seed, to reproduce the data from previous test are adopted: The results are shown in Table II. All tests
|
||
failures. specified at least one input generation strategy, as expected
|
||
• Settings: @settings, to impose test budgets, a maxi- from a PBT. Hypothesis itself sees this decorator as the
|
||
mum number of examples to generate, etc. entry point of a PBT. Control (specifically assume), was
|
||
64.3% PBT origin
|
||
from example-based test
|
||
60 from parameterized test
|
||
|
||
|
||
|
||
50
|
||
Percent of tests
|
||
|
||
|
||
|
||
|
||
40
|
||
|
||
|
||
|
||
30 29.1%
|
||
|
||
|
||
22.5% 22.5%
|
||
20.0% 20.3%
|
||
20
|
||
17.6% 17.0%
|
||
16.0% 16.0%
|
||
14.7%
|
||
12.0% 12.1%
|
||
10 8.0%
|
||
6.7% 6.6% 6.7% 6.7%
|
||
|
||
|
||
0
|
||
tic
|
||
|
||
|
||
|
||
|
||
tic
|
||
tic
|
||
|
||
|
||
|
||
|
||
e
|
||
e
|
||
|
||
|
||
|
||
|
||
tic
|
||
|
||
|
||
|
||
|
||
e
|
||
dd
|
||
|
||
|
||
|
||
|
||
d
|
||
ov
|
||
|
||
|
||
|
||
|
||
as
|
||
ov
|
||
ad
|
||
an
|
||
|
||
|
||
|
||
|
||
an
|
||
an
|
||
|
||
|
||
|
||
|
||
an
|
||
:a
|
||
|
||
|
||
|
||
|
||
em
|
||
|
||
|
||
|
||
|
||
tc
|
||
em
|
||
m
|
||
|
||
|
||
|
||
|
||
em
|
||
em
|
||
|
||
|
||
|
||
|
||
em
|
||
p:
|
||
le
|
||
|
||
|
||
|
||
|
||
s
|
||
se
|
||
|
||
|
||
|
||
|
||
:r
|
||
|
||
|
||
|
||
|
||
t-u
|
||
|
||
|
||
|
||
|
||
r
|
||
c
|
||
|
||
|
||
|
||
|
||
te
|
||
s
|
||
:s
|
||
|
||
|
||
|
||
|
||
s
|
||
|
||
|
||
|
||
|
||
p:
|
||
ra
|
||
|
||
|
||
|
||
|
||
-
|
||
|
||
|
||
|
||
|
||
le
|
||
|
||
|
||
|
||
|
||
n-
|
||
Se
|
||
on
|
||
|
||
|
||
|
||
|
||
t
|
||
p:
|
||
|
||
|
||
|
||
|
||
t-u
|
||
O
|
||
|
||
|
||
|
||
|
||
le
|
||
|
||
|
||
|
||
|
||
en
|
||
c
|
||
|
||
|
||
|
||
|
||
no
|
||
t-u
|
||
ra
|
||
:n
|
||
c
|
||
|
||
|
||
|
||
|
||
Se
|
||
|
||
|
||
|
||
|
||
er
|
||
ra
|
||
|
||
|
||
|
||
|
||
O
|
||
|
||
|
||
|
||
|
||
p:
|
||
Se
|
||
le
|
||
|
||
|
||
|
||
|
||
iff
|
||
O
|
||
|
||
|
||
|
||
|
||
t-u
|
||
c
|
||
|
||
|
||
|
||
|
||
D
|
||
ra
|
||
|
||
|
||
|
||
|
||
Se
|
||
O
|
||
|
||
|
||
|
||
|
||
Fig. 1: The changes example-based (blue) and parameterized (orange) unit tests undergo when evolving into a property-based
|
||
test. Each test case can undergo multiple changes from different categories.
|
||
|
||
|
||
From EBT From PUT novice users of the framework, or because the test cases did
|
||
Feature # of test cases % # of test cases % not yet have the time to evolve and be tailored to the project.
|
||
Strategies 182 100 75 100 Therefore, we also investigate how the evolved test cases are
|
||
Explicit inputs 4 2.20 2 2.67 maintained in Section VI-B2.
|
||
Reproducing inputs 0 0 0 0 3) Input generation strategies: Table III depicts the results
|
||
Settings 11 6.04 8 10.67 for our analysis of the input generation strategies used. The
|
||
Control 2 1.10 14 18.67 majority of PBTs that evolve from an EBT have a simple input
|
||
generation strategy for standard data types. However, for 76 of
|
||
TABLE II: Features initially adopted from PBT framework. the 120 simple cases (a majority), more refined configurations
|
||
are used of these simple input generators (such as min/max
|
||
bounds for integers). 60 of the EBTs that evolved into PBTs
|
||
the second most common feature, followed by @settings now have a custom input generation, of which 24 are reused
|
||
(mostly imposing deadlines on the test cases) and explicit multiple times across different test cases.
|
||
inputs (@example). The fact that we see no PBT using When considering PBTs that have evolved from a PUT, just
|
||
failure-reproducing features is not surprising, as it is the first over half of the PBTs have a simple input generation strategy.
|
||
PBT-version of the test cases. They might not yet have run Of those 43 PBTs, 35 refined configurations. All custom input
|
||
enough to want to reproduce a specific failure-inducing input. generation strategies are reused across multiple test cases. It is
|
||
Compared to test cases that evolved from EBTs, test cases common for evolved PUTs from the same project to already
|
||
that evolved from PUTs are more likely to adopt these features share the same input parameter before their evolution into a
|
||
of the PBT framework. One reason for this could be that PBT. In our dataset, the use of libraries for input generation is
|
||
the developers are already more familiar with more advanced not common (only two instances). The only used library was
|
||
testing frameworks. the Hypothesis NumPy extension5 .
|
||
Corgozinho et al. [8] also performed a preliminary study Several test cases already contained some form of random-
|
||
into commonly used features across 86 PBTs. Their dataset ization before being rewritten into a PBT. Afterwards, this
|
||
does not consider the initial PBT introduction, and contains randomization is still reflected in the input generation strategy.
|
||
more well-established, big projects that make use of property- For example, if a test case previously contained a statement a
|
||
based testing. In their dataset, they observe a higher feature
|
||
adoption rate compared to ours. This could be because devel- 5 https://hypothesis.readthedocs.io/en/latest/reference/strategies.html#
|
||
opers evolving a PBT from an existing test might be more hypothesis-numpy
|
||
From EBT From PUT that define behavior in their test files, these should indeed
|
||
# % # % be considered. For others, including test files can lead to a
|
||
Custom, with reuse 36 19.78 32 42.67 wrong conclusion when the test cases themselves have shrunk
|
||
Custom, no reuse 24 13.19 0 0 or increased in size. We therefore report both on the absolute
|
||
Simple 120 65.94 43 57.33 and relative coverage twice, once with the test files included
|
||
Other 2 1.10 0 0 and once with the test files excluded.
|
||
We were able to run 219 of the test cases before PBT-
|
||
TABLE III: Input generation strategies used within PBTs introduction, and 219 test cases after PBT-introduction. How-
|
||
evolved from an EBT or a PUT. ever, not all were able to obtain code coverage. In the end,
|
||
we were able to run and obtain code coverage for 211 test
|
||
case evolutions before and after the introduction of the PBT
|
||
= np.random.randint(1, 10), a strategy of the test (meaning a total of 422 test cases). To ensure the test cases ran,
|
||
case would become @given(st.int(1, 10)). some minimal changes to the code were needed. For example,
|
||
one test case contained a broken import. Upon correcting this
|
||
RQ1 Non-semantic changes to the setup code are es- import, the test case did run. However, we made sure to not
|
||
pecially common in evolutions stemming from EBTs change the semantics of the system under test. In some cases,
|
||
(117/182). For evolutions from PUTs, it is more common tests failed but still resulted in some code coverage. Their
|
||
to require no changes in the body of the test case at all results are also included.
|
||
(35/75). The addition of settings is the most common
|
||
3) Test failures: Code coverage is not the only way to deter-
|
||
feature (19/257) early on in the adoption of the PBT
|
||
mine whether tests are effective. During the data collection of
|
||
framework. Most test cases use simple input generation
|
||
Section V-A2, not all test cases passed, and some test oracles
|
||
strategies (163/257).
|
||
threw an assertion error. For some test cases, this happened
|
||
before their evolution into PBT, whereas for others, the PBT
|
||
V. RQ2: W HAT IS THE IMPACT OF ADOPTING
|
||
did not pass. We investigate how often this occurs, and the
|
||
PROPERTY- BASED TESTING ?
|
||
failed assertions in each.
|
||
With research question two, we aim to find the impact
|
||
of introducing PBTs. We consider commit messages, code B. Results
|
||
coverage, and test failures of the 257 test evolutions. 1) PBT-introducing commits: First, we report on our anal-
|
||
ysis of the commit messages of the PBT-introducing commits.
|
||
A. Research Method Only eight commit messages mention anything positive or
|
||
1) PBT-introducing commits: We investigate and report on negative about Hypothesis or property-based testing (beyond
|
||
73 commit messages across 64 repositories that evolve at least simply stating that they have started using it). One commit
|
||
one conventional test case into a PBT. We aim to uncover message mentions improving code coverage to find more
|
||
reasons as to why the developer decided to evolve their test errors across the code. Two commit messages mention the new
|
||
case, and their potential opinions on this change. PBTs failing (a bug was found, but not yet fixed). Finally, five
|
||
2) Code coverage: We first explore whether code coverage commit messages use the words better, trickier, or improved.
|
||
improves once a conventional test case has evolved into a Four commit messages also mentioned bug fixes of some
|
||
PBT. Improving coverage can be a goal for some PBT- sort, meaning the PBT was introduced at the same time as a
|
||
introducing developers, even though it does not necessarily bug fix. This can indicate that the PBT at introduction might
|
||
improve test suite effectiveness [20]. Moreover, research has have found a bug that the initial conventional test case did
|
||
proposed variants of PBT with that explicit aim (i.e., coverage- not, after which the developer fixed the bug. Alternatively, the
|
||
guided PBT [24], [29], [30]). developer may also be introducing bug fixes and PBTs at the
|
||
We run each test case with statement coverage before same time to ensure their most recent bug fix is tested more
|
||
and after the introduction of Hypothesis while measuring in the future. We did not find any evidence in the commit
|
||
coverage using Coverage.py [4], a well-established Python message supporting either hypothesis, but Section V-B3 looks
|
||
library for test coverage. We only compute the coverage for into test case failures before and after the commit.
|
||
each individual test case, instead of for the entire test suite as a 2) Code coverage: To verify whether there is a difference
|
||
whole. For relative coverage (i.e., in percentage), we normalize in coverage before and after the evolution into PBT, we use a
|
||
the executed statements by those in the files executed by the Wilcoxon signed-rank test [41] as our data is non-parametric
|
||
test case rather than by those in the entire project. We opted and paired (code coverage of a test case before and after the
|
||
for this design as coverage across all statements of the entire PBT-introducing commit). We consider a p-value of <0.05
|
||
project would be very low for individual test cases. We also statistically significant. The test results are listed in Table IV.
|
||
report on coverage in absolute numbers. This can be useful For all 4 cases, we observe a statistically significant difference.
|
||
for test cases that lead to the execution of several files. Note First, we look at the change in number of statements
|
||
that, by default, coverage.py considers all executed Python covered. Figure 2 shows box plots for the distribution of the
|
||
files including the files that define the test case. For projects absolute statement coverage before and after the test case
|
||
Stmt (all) % (all) Stmt (no test) % (no test) coverage ∆ # Stmt (all) % (all) # Stmt (no test) % (no test)
|
||
p-value 0.006 2.497 × 10−20 1.477 × 10−13 0.045 Mean +13.09 +2.31 +4.01 -0.82
|
||
Std Dev. 27.79 6.06 16.81 5.5
|
||
TABLE IV: p-values of change in coverage, in terms of
|
||
median +11 +0.57 +0 +0
|
||
absolute and relative statement coverage.
|
||
1st Quar. +2 -0.06 +0 -1.26
|
||
total # ran tests # passing # failing with coverage 3rd Quar. +17 +2.46 +3 +0.28
|
||
Before PBT 219 199 20 211 Min -78 -10.09 -75 -21.90
|
||
After PBT 219 207 12 211 Max +223 +32.61 +77 +32.52
|
||
|
||
TABLE V: Number of passing and failing test cases. We could TABLE VI: Changes in code coverage, both in terms of
|
||
not obtain coverage for all tests (as shown by the last column). number of statements covered and change in the coverage
|
||
percentage, for both all files and all files without test files
|
||
included.
|
||
|
||
2500
|
||
evolution, taking into account all executed files (denoted “all”
|
||
in the plot) and all executed files without test files (denoted
|
||
2000 “no test”). While we see a slight improvement in median
|
||
and quartiles when all executed files are taken into account,
|
||
1500
|
||
the change becomes much more subtle when excluding the
|
||
executed test files. One reason could be that property-based
|
||
test cases execute more lines of test code. Even if the tests
|
||
1000
|
||
themselves are not necessarily longer, the inclusion of input
|
||
generators can add more test code to cover —especially for
|
||
500 projects that have defined custom ones in test files.
|
||
Figure 3 on the other hand shows the relative statement
|
||
coverage in percentage. As mentioned before, by default,
|
||
0
|
||
coverage.py only takes into consideration files that have at
|
||
Before (all) After (all) Before (no test) After (no test)
|
||
least one executed line of code, meaning files that are never
|
||
covered by the test case are not considered. Here we also see
|
||
an improvement when taking into account test files, but once
|
||
Fig. 2: Statements covered by test cases before and after PBT-
|
||
they are removed from the data, we notice a slight decrease
|
||
introduction. The first two plots show coverage across all files,
|
||
in code coverage on average.
|
||
the second two without test files included.
|
||
Table VI shows the deltas for each of the four measure-
|
||
ments. While the mean absolute number of covered statements
|
||
without test files included does improve slightly (4.01), the
|
||
relative coverage percentage goes down slightly. This could
|
||
100 be due to extra files being discovered, or more code being
|
||
introduced during the PBT-introducing commit. Most notice-
|
||
80 ably, one project’s covered statements decreases by 75. Upon
|
||
further investigation, the test case in question contained many
|
||
assert statements. While the conventional test case before
|
||
60
|
||
PBT-introduction passed, Hypothesis found a counterexample
|
||
on the first assert, thereby stopping the test case and not
|
||
40 executing the subsequent assert statements (leading to a drastic
|
||
decrease in coverage). Additionally, 26 test cases from the
|
||
same repository all had one additional change when evolved
|
||
20
|
||
into PBT: the removal of a call to the logging function (which
|
||
was three statements), leading to a decrease of coverage of
|
||
0 indeed 3 statements for these 26 test cases.
|
||
Before % (all) After % (all) Before % (no test) After % (no test) 3) Test failures: Table V shows the number of tests failed
|
||
before and after PBT-introduction. Two test cases did not pass
|
||
as a conventional unit test, nor after having been rewritten into
|
||
Fig. 3: Coverage percentage by test cases before and after a PBT. For 18 test cases, the initial conventional test did not
|
||
PBT-introduction. The first two plots show coverage across pass, but the updated PBT did pass. In another ten cases, the
|
||
all files, the second two without test files included. initial test passed but the resulting PBT did not.
|
||
In terms of the before, one project with one evolution • Updates to the test case encompassing both semantic and
|
||
had a test case that resulted in an assertion error. The PBT- non-semantic changes.
|
||
introducing commit also contained a bug fix. Another project We make one exception for 39 test cases originating from
|
||
had three failing test cases that returned a type error. After the one repository. In the most recent version of that repository,
|
||
introduction of the PBT, all three passed. However, the PBT- most Python code has been removed, and is scheduled to be
|
||
introducing commit does not introduce a bug fix (meaning added at a later stage. Because of this drastic rewrite affecting
|
||
the bug was in the test code itself). Another project had two the latest version in the repository, we instead consider its most
|
||
failing test cases, with an attribute error and a type error. For recent version with the test cases still present.
|
||
both, the PBT versions did pass. In one of the bigger projects Additionally, we report on the survival rate of the test cases
|
||
in our dataset, with 16 evolved test cases, ten tests did not by looking at the number of commits since introduction, or
|
||
pass before the evolution. In three cases this was because of the number of commits between introduction and removal.
|
||
an assertion error (with the other seven not passing due to a 2) Are more parts of the PBT framework adopted?: As de-
|
||
value not being defined). All ten of these test cases passed after velopers become more acquainted with property-based testing
|
||
being updated to a PBT. These results suggest that people are in their projects, they might discover features unbeknownst
|
||
introducing PBTs when fixing known errors, perhaps to ensure to them before. Therefore, similar to Section IV-A2, we
|
||
more thorough testing of those parts of the system. investigate whether the most recent version of their PBT
|
||
When looking at the failing tests cases after the introduction uses more features of the PBT framework. We use the same
|
||
of PBTs, we found five test cases for which Hypothesis categories of features from the Hypothesis API [42] to classify
|
||
reported a counterexample for which the original test case the additions. As all PBTs already had an input generation
|
||
passed. Further, as mentioned in Section V-B1, some commits strategy when they were introduced, we investigate whether
|
||
did contain additional bug fixes. This indicates that PBTs can this strategy has been updated in the meantime.
|
||
uncover bugs not found by conventional test cases validating
|
||
the same behavior, thus improving bug detection. For the B. Results
|
||
other 5 non-passing PBTs, we found other errors (such as 1) Do Property-based Test Cases Change?: To answer this
|
||
ValueErrors), even though the original test case passed before. question, we compare the most recent commit in the GitHub
|
||
repository to the PBT-introducing commit. Table VII shows
|
||
RQ2 8 PBT-introducing commit messages convey an the number of commits the test cases have survived (for those
|
||
intent to improve test cases to improve coverage or find that are still present in the current version of their project,
|
||
bugs. We observed a statistically significant change in i.e., the number of commits between introduction and the
|
||
statement coverage, with on average an increase of 4 most current commit), or how many repository commits they
|
||
statements. We also found evidence of PBTs being intro- survived before their deletion. Most test cases in our dataset
|
||
duced along with a fix for a bug in the code, and of PBTs (110) are unchanged. Their corresponding repositories have
|
||
finding counterexamples where the original corresponding also enjoyed the fewest commits since the PBT-introducing
|
||
conventional test passed. one (with 99 on average, with a standard deviation of 205).
|
||
Interestingly, one test case survived 1728 repository commits
|
||
VI. RQ3: H OW ARE PROPERTY- BASED TESTS without undergoing any changes itself. A total of 90 test cases
|
||
MAINTAINED ?
|
||
did undergo changes since their introduction, both semantics-
|
||
We consider the most recent version of the 250 unique PBTs preserving and semantics-affecting ones. We observe that these
|
||
from the in total 257 test case evolutions, and investigate the test cases also survived more commits, with on average 642
|
||
changes (or removals) they have been subject to since. commits and a higher standard deviation. This is expected as
|
||
PBTs need to co-evolve with their system under test.
|
||
A. Research Method
|
||
Finally, 50 PBTs were removed from the projects since
|
||
For each test case, we compare the version resulting from their initial introduction. These test cases also survived fewer
|
||
the PBT-introducing commit to the version in the most recent commits; 352 on average. However, one test case survived
|
||
commit in its repository. We consider the following: 3573 commits to the repository before being removed. Reasons
|
||
1) Do Property-based Test Cases Change?: We investigate cited for the deletion of these test cases were removal of the
|
||
whether the 250 PBTs in our dataset change over time, for functions they tested (5), unexpected behavior (3), and the tests
|
||
instance in response to changes in the system under test. For being slow (1).
|
||
each PBT, we manually compare against the same test case in 2) Are more parts of the PBT framework adopted?: Finally,
|
||
the most recent version of the GitHub repository. For each of we consider the changes made to PBTs in terms of the parts of
|
||
our test cases, we consider the following: the PBT framework that are adopted. Table VIII describes the
|
||
• No changes to the test case. The rest of the project and results. In 65 test cases, the input generation strategies were
|
||
other test cases might have changed, but the investigated updated after their introduction. One repository in particular
|
||
PBT has not. mentions doing so to improve the running time of the test
|
||
• Removal of the test case. In this case, we also investigate cases. Only one repository added an explicit input to test for on
|
||
possible reasons why. each run. Additionally, one test case added a seed decorator
|
||
No changes Changes Removal easier to evolve unit tests that exercise the unit under test on
|
||
Mean 99 642 352 inputs of standard data types (e.g., integers) into a PBT with
|
||
Std Dev. 205.31 1450.79 967.80 such a strategy. Our findings should motivate practitioners to
|
||
median 51 146 34 prioritize their generalization efforts on those unit tests first.
|
||
1st Quar. 48 64 17 Finally, a study by Goldstein et al. [14] reports that PBT
|
||
3rd Quar. 63 146 71 adopters have difficulty identifying the properties to test for.
|
||
Min 1 17 7 Using existing test cases, rather than creating new ones, might
|
||
Max 1728 5886 3573
|
||
help. This is especially true for parameterized unit tests (PUT),
|
||
and especially those that do not list expected outputs for every
|
||
TABLE VII: Number of commits since the introduction of the given input, as those might already be testing a property (but
|
||
PBT. Data split for the test cases that did not change at all, for not yet randomly generating input). Indeed, for 35/75 PBTs
|
||
those with changes, and for those removed in the meantime. that evolved from a PUT (Section IV-B), no changes to the
|
||
body of the test case were required at all. This suggests that
|
||
Feature # of test cases PUTs can be considered a stepping stone towards, or even an
|
||
Change in strategies 65 intermediate step in a planned introduction of PBTs.
|
||
Addition of explicit inputs 1 For tool developers: Recent years have seen the introduc-
|
||
Addition of reproducing inputs 1 tion of tool support for creating PBTs. From Hypothesis’
|
||
Addition of settings 19 own ghostwriter6 , over using LLMs to write PBTs [13], [38],
|
||
Addition of control 0 to tools that aim to aggregate similar test cases into one
|
||
Addition of exceptions 0 PBT [31]. Tools that take EBTs and transform them into PUTs
|
||
also exist [37]. However, there remains a need for tools that
|
||
TABLE VIII: Changes in used PBT features in the most recent support developers in evolving existing, dissimilar, test cases
|
||
versus the initial PBT-introducing commit of the test case. into PBTs. Our findings for RQ1 (Section IV-B), and our
|
||
supporting dataset, can inspire tool builders with examples of
|
||
to reproduce inputs. 19 of the test cases eventually got some real-world transformations that can be partially automated.
|
||
form of settings added, which doubles their total number. As shown in Section IV-B2, not all features of the property-
|
||
based testing framework are immediately adopted in newly-
|
||
RQ3 Most PBTs (200/250) in our dataset have survived evolved PBTs. In Section VI-B2 we found that features such
|
||
since their introduction, with 90 having gone through as PBT settings are often included only later in the life of
|
||
changes. Removed test cases survived on average 352 a PBT. We also saw that inside some PBTs, developers were
|
||
commits to the repository. 65 test cases saw a change using if-statements, where an assume might have been more
|
||
in their input generation strategy since their introduction. appropriate. Therefore, future tools could not only help with
|
||
Finally, 19 test cases received additional settings. the evolution of test cases into PBTs, but also aid with the
|
||
introduction of features to optimize existing PBTs.
|
||
VII. D ISCUSSION For researchers: There are still few empirical work that
|
||
We now distil the actionable insights from our findings, for compares conventional unit tests to PBTs. Ours is, to our
|
||
software testers, tool builders, and researchers alike. We also knowledge, the first to do so in a one-to-one comparison of test
|
||
discuss the threats to the validity of our findings. cases before and after their evolution into property-based ones.
|
||
Our comparison on test failures and code coverage has been
|
||
A. Actionable insights insightful, and could inspire other one-to-one comparisons on
|
||
For software testers: As evidenced by our findings for other metrics. For instance, mutation score (recently used in
|
||
RQ2 (Section V-B), evolving well-chosen unit tests into prior work [32], see Section VIII) has not yet been explored
|
||
property-based ones can increase their code coverage. We in a one-to-one comparative setting.
|
||
moreover found evidence of such real-world evolutions finding
|
||
B. Threats to validity
|
||
counterexamples where the original test passed without any
|
||
failures, leading to a deeper validation of the system under Following standard practice in empirical software engineer-
|
||
test. We even found instances of these evolutions being con- ing, we discuss potential threats to the validity of our results.
|
||
ducted alongside bug fixes, most likely to test for regressions For each threat, we describe the applied mitigation strategies.
|
||
more thoroughly. Our findings should motivate practitioners Construct validity: We considered test case evolutions that
|
||
to consider unit tests that have missed edge cases in the past are found in open-source GitHub repositories, which can
|
||
as candidates to be evolved into property-based ones. raise quality concerns. We mitigated this threat by applying
|
||
Prior studies have found that implementing input generation strict inclusion criteria, resulting in 575 high-quality candidate
|
||
strategies for a PBT can be tedious and effort-intensive [14]. repositories with at least one PBT. As most of those PBTs did
|
||
Nonetheless, our findings for RQ1 (Section IV-B) indicate not result from an evolution, we were left with fewer test case
|
||
that many evolved PBTs in our dataset use but a simple, 6 https://hypothesis.readthedocs.io/en/latest/reference/integrations.html#
|
||
built-in input generation strategy. Practitioners may find it ghostwriter
|
||
evolutions to study, but the study subjects still stem from 64 Goldstein et al. [14] interviewed 30 developers from a
|
||
different repositories, ensuring diversity. fintech company, to find insights about how people experience
|
||
Further, while false negatives cannot be excluded, all test PBT in practice. Another paper by Goldstein et al. [15] reports
|
||
case evolutions were manually inspected by both the first that developers experience difficulties in finding properties to
|
||
and the second author, ensuring the dataset is free of false test for. Our research looks at open-source projects rather than
|
||
positives. The same goes for the before-after mappings for the industry, and investigates existing test cases.
|
||
each test case evolution, which were created by the first author A study by Wauters and De Roover [39] looked into the use
|
||
and inspected and confirmed by the second author. of PBTs by 28 open-source machine learning projects. The
|
||
Manual labeling of test case evolutions was also used to study investigated common positive and negative sentiments
|
||
answer RQ1. We mitigated the threat of subjective bias by expressed in GitHub commits and code comments, which
|
||
having multiple labelers and multiple labeling rounds until parts of a project were tested, as well as the complexity of
|
||
a substantial inter-rater agreement was reached. To answer data generation strategies. Our study also investigates data
|
||
RQ2, we used tools to execute test cases and compute their generation strategies but does not focus on ML-intensive
|
||
code coverage. To mitigate the threat of measurement bias projects that often require more complex data.
|
||
introduced by the tooling, we only used mature open-source Ravi and Coblenz [32] recently performed a large-scale
|
||
tools that are well-established within the community. empirical study on PBTs in 426 Python programs. Their study
|
||
Internal validity: For RQ2, we measure changes in code uses a data flow analysis to automatically categorize real-world
|
||
coverage induced by the evolution into a PBT. However, the PBTs into 12 categories. Additionally, they used mutation
|
||
introduction of a PBT is not always the only change in a testing to investigate how many mutations a project’s existing
|
||
commit. If additional changes are present, the total number PBTs can find compared to the project’s existing conventional
|
||
of statements may change, which can confound the observed tests. Our research looks at the impact of evolving test cases,
|
||
coverage differences and lead to incorrect conclusions. To and compares a test case before its evolution into a PBT to
|
||
mitigate this threat, we manually inspected commits exhibiting the resulting PBT. This means we compare two test cases that
|
||
large coverage changes. In addition, we report both absolute are equivalent in terms of the behavior they are supposed to
|
||
and relative coverage, computed with and without the test files validate, and we do so one-to-one.
|
||
included, as test files are more likely to change due to the Unlike our study, the aforementioned papers do not inves-
|
||
introduction of PBTs. tigate the evolution of PBTs, but rather consider snapshots of
|
||
External validity: Finally, our study focuses on Python PBTs at one point in time.
|
||
test cases that use the Hypothesis framework. Hypothesis IX. C ONCLUSION AND F UTURE W ORK
|
||
is the most widely recognized PBT framework for Python, This paper studied the evolution of property-based unit tests
|
||
and Python is ranked among the most popular programming from conventional ones, the impact of such PBT-introducing
|
||
languages. Nevertheless, our findings may not generalize to evolutions, and how the resulting PBTs are maintained after-
|
||
other programming languages nor to other PBT frameworks. wards. We found that the evolution of parameterized unit tests
|
||
into a PBT commonly requires no updates to their test body
|
||
VIII. R ELATED W ORK
|
||
at all. For example-driven unit tests, in contrast, the evolution
|
||
Several studies have investigated the introduction and main- commonly required non-semantic changes to the test body.
|
||
tenance of various types of test cases, from tests in general [1], These account for the introduction and usage of a randomly
|
||
[9], [44] over GUI tests [5] to performance tests [11]. A generated input value. The input generation strategy of PBTs
|
||
few studies have also looked into changes in code coverage that evolved from existing unit tests is often simple. Impact-
|
||
induced by software patches, e.g., [12] and [17]. However, wise, we found that code coverage can improve when unit tests
|
||
our work is the first to study how PBTs can evolve from are replaced by PBTs. More importantly, we found evidence
|
||
conventional unit tests, and to study the impact on code of bugs found through or alongside the introduction of these
|
||
coverage —rather than changes in code coverage over time PBTs. Most of the PBTs in our dataset are still present in the
|
||
or induced by changes to the system under test. most recent version, with almost half having gone through
|
||
Zooming in on other studies targeting property-based test- some maintenance during their life time. The adoption of
|
||
ing, Corgozinho et al. [8] sampled 86 PBTs from GitHub, and additional PBT features is not uncommon either.
|
||
categorized each of them manually according to the testing One avenue for future work is investigating the long-term
|
||
patterns they adhere to, which stem from an influential blog impact of PBT-introducing test case evolutions. Another is
|
||
post7 . They also study the use of Hypothesis features in developer support in the form of a tool that identifies candidate
|
||
those PBTs, similar to us. However, while they investigate unit tests for such an evolution, and for partially automating
|
||
bigger, established projects, we investigate the adoption of the necessary test changes.
|
||
these features in test cases that were previously not a PBT.
|
||
ACKNOWLEDGEMENTS
|
||
We also study whether those features are adopted later on,
|
||
while the PBT matures. This research was partially funded by the Research Foun-
|
||
dation Flanders (FWO) Grant No. 1SHFI24N and by the
|
||
7 https://fsharpforfunandprofit.com/posts/property-based-testing-2/ Cybersecurity Research Program Flanders (CRPF).
|
||
R EFERENCES [18] John Hughes. Experiences with QuickCheck: Testing the Hard Stuff and
|
||
Staying Sane, pages 169–186. Springer International Publishing, Cham,
|
||
[1] Maurı́cio Aniche, Christoph Treude, and Andy Zaidman. How develop- 2016.
|
||
ers engineer test cases: An observational study. IEEE Transactions on [19] John Hughes, Benjamin C. Pierce, Thomas Arts, and Ulf Norell. Myster-
|
||
Software Engineering, 48(12):4925–4946, 2022. ies of dropbox: Property-based testing of a distributed synchronization
|
||
[2] Thomas Arts, John Hughes, Joakim Johansson, and Ulf Wiger. Testing service. In 2016 IEEE International Conference on Software Testing,
|
||
telecoms software with quviq quickcheck. In Proceedings of the 2006 Verification and Validation (ICST), pages 135–145, 2016.
|
||
ACM SIGPLAN Workshop on Erlang, ERLANG ’06, page 2–10, New [20] Laura Inozemtseva and Reid Holmes. Coverage is not strongly correlated
|
||
York, NY, USA, 2006. Association for Computing Machinery. with test suite effectiveness. In Proceedings of the 36th International
|
||
[3] Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Conference on Software Engineering, ICSE 2014, page 435–445, New
|
||
Shin Yoo. The oracle problem in software testing: A survey. IEEE York, NY, USA, 2014. Association for Computing Machinery.
|
||
Transactions on Software Engineering, 41(5):507–525, 2015. [21] Paul Jansen. Tiobe index for december 2025. https://web.archive.org/
|
||
web/20251218235222/https://www.tiobe.com/tiobe-index/. Accessed on
|
||
[4] Ned Batchelder and Contributors to Coverage.py. Coverage.py: The code
|
||
20 december 2025.
|
||
coverage tool for Python. https://web.archive.org/web/20251223093400/
|
||
[22] JetBrains. Python developers survey 2023 results. https:
|
||
https://github.com/coveragepy/coveragepy. Accessed on 20 december
|
||
//web.archive.org/web/20250920050020/https://lp.jetbrains.com/
|
||
2025.
|
||
python-developers-survey-2023/. Accessed on 10 december 2025.
|
||
[5] Laurent Christophe, Reinout Stevens, Coen De Roover, and Wolfgang
|
||
[23] Holger Krekel, Bruno Oliveira, Ronny Pfannschmidt, Floris
|
||
De Meuter. Prevalence and maintenance of automated functional tests for
|
||
Bruynooghe, Brianna Laugher, and Florian Bruhin. pytest
|
||
web applications. In 2014 IEEE International Conference on Software
|
||
9.0.0. https://web.archive.org/web/20250906051102/https:
|
||
Maintenance and Evolution, pages 141–150, 2014.
|
||
//github.com/pytest-dev/pytest/, 2004. Version 9.0.0 Contributors
|
||
[6] Koen Claessen and John Hughes. Quickcheck: a lightweight tool
|
||
include Holger Krekel, Bruno Oliveira, Ronny Pfannschmidt, Floris
|
||
for random testing of haskell programs. In Proceedings of the Fifth
|
||
Bruynooghe, Brianna Laugher, Florian Bruhin, and others.
|
||
ACM SIGPLAN International Conference on Functional Programming,
|
||
[24] Leonidas Lampropoulos, Michael Hicks, and Benjamin C. Pierce. Cov-
|
||
ICFP ’00, page 268–279, New York, NY, USA, 2000. Association for
|
||
erage guided, property based testing. Proc. ACM Program. Lang.,
|
||
Computing Machinery.
|
||
3(OOPSLA), October 2019.
|
||
[7] Jacob Cohen. A coefficient of agreement for nominal scales. Educational [25] J. Richard Landis and Gary G. Koch. The measurement of observer
|
||
and psychological measurement, 20(1):37–46, 1960. agreement for categorical data. Biometrics, 33(1):159–174, 1977.
|
||
[8] Arthur Lisboa Corgozinho, Marco Tulio Valente, and Henrique Rocha. [26] David R. MacIver and Alastair F. Donaldson. Test-Case Reduction via
|
||
How Developers Implement Property-Based Tests . In 2023 IEEE Inter- Test-Case Generation: Insights from the Hypothesis Reducer. In Robert
|
||
national Conference on Software Maintenance and Evolution (ICSME), Hirschfeld and Tobias Pape, editors, 34th European Conference on
|
||
pages 380–384, Los Alamitos, CA, USA, October 2023. IEEE Computer Object-Oriented Programming (ECOOP 2020), volume 166 of Leibniz
|
||
Society. International Proceedings in Informatics (LIPIcs), pages 13:1–13:27,
|
||
[9] Ermira Daka and Gordon Fraser. A survey on unit testing practices and Dagstuhl, Germany, 2020. Schloss Dagstuhl – Leibniz-Zentrum für
|
||
problems. In 2014 IEEE 25th International Symposium on Software Informatik.
|
||
Reliability Engineering, pages 201–211, 2014. [27] David R MacIver, Zac Hatfield-Dodds, et al. Hypothesis: A new
|
||
[10] Albert Danial. cloc: v1.92. https://doi.org/10.5281/zenodo.5760077, approach to property-based testing. Journal of Open Source Software,
|
||
December 2021. 4(43):1891, 2019.
|
||
[11] Sergio Di Meglio, Luigi Libero Lucio Starace, Valeria Pontillo, Ruben [28] Gerard Meszaros. xUnit test patterns: Refactoring test code. Pearson
|
||
Opdebeeck, Coen De Roover, and Sergio Di Martino. Performance Education, 2007.
|
||
testing in open-source web projects: Adoption, maintenance, and a [29] Agustı́n Mista and Alejandro Russo. Mutagen: Reliable coverage-
|
||
change taxonomy. In Proceedings of the 41st IEEE International guided, property-based testing using exhaustive mutations. In 2023 IEEE
|
||
Conference on Software Maintenance and Evolution (ICSME 2025). Conference on Software Testing, Verification and Validation (ICST),
|
||
IEEE, sep 2025. 41st IEEE International Conference on Software pages 176–187, 2023.
|
||
Maintenance and Evolution (ICSME 2025), ICMSE ; Conference date: [30] Rohan Padhye, Caroline Lemieux, and Koushik Sen. Jqf: coverage-
|
||
07-09-2025 Through 12-09-2025. guided property-based testing in java. In Proceedings of the 28th ACM
|
||
[12] S. Elbaum, D. Gable, and G. Rothermel. The impact of software SIGSOFT International Symposium on Software Testing and Analysis,
|
||
evolution on code coverage information. In Proceedings IEEE Inter- ISSTA 2019, page 398–401, New York, NY, USA, 2019. Association
|
||
national Conference on Software Maintenance. ICSM 2001, pages 170– for Computing Machinery.
|
||
179, 2001. [31] Hila Peleg, Dan Rasin, and Eran Yahav. Generating tests by example.
|
||
[13] Khashayar Etemadi, Marjan Sirjani, Mahshid Helali Moghadam, Per In Isil Dillig and Jens Palsberg, editors, Verification, Model Checking,
|
||
Strandberg, and Paul Pettersson. Llm-based property-based test genera- and Abstract Interpretation, pages 406–429, Cham, 2018. Springer
|
||
tion for guardrailing cyber-physical systems. In International Conference International Publishing.
|
||
on Bridging the Gap between AI and Reality, pages 18–46. Springer [32] Savitha Ravi and Michael Coblenz. An empirical evaluation of property-
|
||
Nature Switzerland Cham, 2025. based testing in python. Proc. ACM Program. Lang., 9(OOPSLA2),
|
||
[14] Harrison Goldstein, Joseph W. Cutler, Daniel Dickstein, Benjamin C. October 2025.
|
||
Pierce, and Andrew Head. Property-based testing in practice. In Pro- [33] Mohammad Rezaalipour and Carlo A. Furia. An annotation-based
|
||
ceedings of the IEEE/ACM 46th International Conference on Software approach for finding bugs in neural network programs. Journal of
|
||
Engineering, ICSE ’24, New York, NY, USA, 2024. Association for Systems and Software, 201:111669, 2023.
|
||
Computing Machinery. [34] Davide Spadini, Maurı́cio Aniche, and Alberto Bacchelli. Pydriller:
|
||
[15] Harrison Goldstein, Joseph W Cutler, Adam Stein, Benjamin C Pierce, Python framework for mining software repositories. In Proceedings of
|
||
and Andrew Head. Some problems with properties. In Proc. Workshop the 2018 26th ACM Joint Meeting on European Software Engineering
|
||
on the Human Aspects of Types and Reasoning Assistants (HATRA), Conference and Symposium on the Foundations of Software Engineering,
|
||
2022. ESEC/FSE 2018, page 908–911, New York, NY, USA, 2018. Associa-
|
||
[16] Harrison Goldstein, Jeffrey Tao, Zac Hatfield-Dodds, Benjamin C. tion for Computing Machinery.
|
||
Pierce, and Andrew Head. Tyche: Making sense of pbt effectiveness. In [35] Nikolai Tillmann and Wolfram Schulte. Parameterized unit tests.
|
||
Proceedings of the 37th Annual ACM Symposium on User Interface SIGSOFT Softw. Eng. Notes, 30(5):253–262, September 2005.
|
||
Software and Technology, UIST ’24, New York, NY, USA, 2024. [36] Nikolai Tillmann and Wolfram Schulte. Parameterized unit tests. In
|
||
Association for Computing Machinery. Proceedings of the 10th European Software Engineering Conference
|
||
[17] Michael Hilton, Jonathan Bell, and Darko Marinov. A large-scale study Held Jointly with 13th ACM SIGSOFT International Symposium on
|
||
of test coverage evolution. In Proceedings of the 33rd ACM/IEEE Foundations of Software Engineering, ESEC/FSE-13, page 253–262,
|
||
International Conference on Automated Software Engineering, ASE ’18, New York, NY, USA, 2005. Association for Computing Machinery.
|
||
page 53–63, New York, NY, USA, 2018. Association for Computing [37] Deepika Tiwari, Yogya Gamage, Martin Monperrus, and Benoit Baudry.
|
||
Machinery. Proze: Generating parameterized unit tests informed by runtime data.
|
||
In 2024 IEEE International Conference on Source Code Analysis and tation. https://web.archive.org/web/20251223092313/https://hypothesis.
|
||
Manipulation (SCAM), pages 166–176, 2024. readthedocs.io/en/latest/reference/api.html. Accessed on 10 december
|
||
[38] Vasudev Vikram, Caroline Lemieux, Joshua Sunshine, and Rohan Pad- 2025.
|
||
hye. Can large language models write good property-based tests?, 2024. [43] Hypothesis Works. When to use hypothesis and property-based testing
|
||
[39] Cindy Wauters and Coen De Roover. Property-based Testing within ML - hypothesis 6.148.3 documentation. https://web.archive.org/web/
|
||
Projects: an Empirical Study . In 2024 IEEE International Conference 20251223092910/https://hypothesis.readthedocs.io/en/latest/tutorial/
|
||
on Software Maintenance and Evolution (ICSME), pages 648–653, Los introduction.html#when-to-use-hypothesis-and-property-based-testing.
|
||
Alamitos, CA, USA, October 2024. IEEE Computer Society. Accessed on 11 december 2025.
|
||
[40] Cindy Wauters, Ruben Opdebeeck, and Coen De Roover. Replication [44] Andy Zaidman, Bart Van Rompaey, Serge Demeyer, and Arie van
|
||
package on the evolution of python test cases into property-based tests. Deursen. Mining software repositories to study co-evolution of pro-
|
||
https://doi.org/10.6084/m9.figshare.31424447, Feb 2026. duction & test code. In 2008 1st International Conference on Software
|
||
[41] Frank Wilcoxon. Individual comparisons by ranking methods. Biomet- Testing, Verification, and Validation, pages 220–229, 2008.
|
||
rics Bulletin, 1(6):80–83, 1945.
|
||
[42] Hypothesis Works. Api reference - hypothesis 6.148.3 documen-
|
||
|