#2016-01-2819:43paulspencerwilliamsHi all.#2016-01-2906:51lucasbradstreetHowdy#2016-01-2908:12paulspencerwilliamsHi @lucasbradstreet.#2016-01-2908:31paulspencerwilliamsSo I was hoping to get some advice about the properties I could test for the convert function found at https://github.com/paulspencerwilliams/gormanic/blob/master/src/gormanic/core.clj#2016-01-2908:38lucasbradstreettesting the string generated by convert will be a bit annoying#2016-01-2908:38lucasbradstreetbut you could test the functions that it uses#2016-01-2908:38lucasbradstreetsome easy ones, do any of them return a negative month, day, year#2016-01-2908:39lucasbradstreetor a month larger than 12 (or 11 if 0 indexed), day greater than 31, tec#2016-01-2908:39lucasbradstreetetc#2016-01-2908:41paulspencerwilliamsokay.#2016-01-2908:42paulspencerwilliamsI understand testing the boundaries aren’t crossed like you mentioned. If I returning a map rather than string, with day, month and year keys, it would be easier to ‘parse’ and thus test. Would quickcheck style testing help then?#2016-01-2908:44paulspencerwilliamsWould I use it to test the function actually returns the correct resultant date? I was thinking that I would have to model the gormanic calendar in the test, and for a random set of input dates, ensure convert would return the appropriate gormanic date. But this would be in effect duplicating SUT code in the test?#2016-01-2908:45paulspencerwilliamsApologies, if I’ve asked a common question, I’ve watched a few videos etc but many properties seem abstract, or away from the concrete problem if that makes sense?#2016-01-2908:48lucasbradstreetYeah, a map definitely seems easier#2016-01-2908:49lucasbradstreetI don’t know enough about the gormanic calender to provide some good properties there 😕#2016-01-2908:50lucasbradstreetThe wikipedia page isn’t all that much help, heh#2016-01-2908:55lucasbradstreetOne idea is if you could provide a comparator function for both date types, and make sure that if one random date is greater than another random date in the gregorian calender, then it’s also greater in the calculated gormanic calender date.#2016-01-2908:58paulspencerwilliamshehe, oh sorry, the gormanic calendar is made up for a kata. okay, I need to think about your idea for the comparator function. If I developed one, would you not simply say that for all inputs, the output compares exactly to the input?#2016-01-2908:58lucasbradstreetExactly#2016-01-2909:00paulspencerwilliamsokay. Cheers, let me think about that and I’ll have a play.#2016-02-0908:15lucasbradstreet@paulspencerwilliams: what solution did you end up with, and were you satisfied with the result?#2016-02-0908:20paulspencerwilliamsHi @lucasbradstreet, in the end, I used a combination of roundtripping tests for majority of coverage backed by example based tests for confidence. I will reduce these examples to a couple when I get back to tidying the code. I promised a blog post, and one is in draft although it maybe the weekend before I get chance to revisit it but I really enjoyed the exercise. Satified? I'm not sure, I liked the example based backups atm, but I think that will dissappear as I start to explore more. I can imagine using test.check more as I move my workflow from TDD, to repl driven.
https://github.com/paulspencerwilliams/gormanic/blob/master/test/gormanic/core_test.clj#2016-02-0908:22lucasbradstreet Cool. Was curious #2016-02-0908:22lucasbradstreetI usually pepper in a few example based tests as a sanity check#2016-05-1013:37lucasbradstreetGood article on different types of properties https://fsharpforfunandprofit.com/posts/property-based-testing-2/#2016-05-3101:25lgastakoIs there an idiomatic way to test that a function returns nil when given any value of any type other than the expected type? Or in general to generate something like “any value of any type exception this one”?#2016-05-3107:44lucasbradstreetlgastako: you can feed it gen/any, but it’ll be limited in the types that it can emit and you’d still need to test whether it sometimes returns a value when given the correct type. It doesn’t feel like the sort of thing I’d property test though.#2016-06-0813:51wilkerluciohello#2016-06-0813:51wilkerluciois it possible to run async code into a (props/for-all)
?#2016-06-0813:51wilkerlucioI would like to test some cljs code that requires to be async, any ideas on how to do that?#2016-06-0814:01lucasbradstreet@wilkerlucio: I’ve used property testing with core.async before. The only real way I could figure out to do it to make sure that there’s some kind of terminator on the channel (this could be that the channel was closed), and that you’ve hit it. You could also add a big enough sleep to make sure that everything should have finished, but that’s super hacky and will limit how many examples you can run in a given time#2016-06-0814:04lucasbradstreet@wilkerlucio: you may also find https://github.com/czan/stateful-check helpful#2016-06-0814:06wilkerluciohi Lucas, sorry, I didn't understand the idea you told me to use core.async, can you show me a code example of it please?#2016-06-0814:08lucasbradstreetSorry, I don’t really have an example handy. It really depends on what you’re trying to do#2016-06-0814:52wilkerluciothats fine, on my case I don't really need the for-all, I'm doing ok just sampling and testing from it, thanks for the assistance#2016-06-0818:53gfrederickslucasbradstreet: "big enough sleep" ⇐ are you talking about clj-jvm instead of cljs?#2016-06-0818:53gfrederickswilkerlucio: I was just wondering about async tests myself like two days ago; I think you'd need a separate test runner#2016-06-0818:53gfredericksi.e., would have to modify clojure.test.check/quick-check#2016-06-0818:54lucasbradstreetOh yeah, I am#2016-06-0818:54gfrederickswilkerlucio: you'd have to rewrite quick-check to listen for results on a core.async channel I think#2016-06-0818:55gfredericksit'd probably be one of those "quick-check returns a channel that will eventually receive the result" sort of situations#2016-06-0818:56lucasbradstreetYeah, that’s basically along the lines of what I was trying to get across#2016-06-0818:57gfredericksand then would need to rewrite prop/for-all to supply a channel for the body to write to somehow, or something#2016-06-0819:50wilkerlucioyeah, seems a lot of fun doing it, just need some time to learn how test.check works, hehe#2016-06-0819:50gfredericksI'm happy to advise#2016-06-0917:40agif I have something like this:
(defn uuid-gen []
(g/return (-> g/uuid g/generate str)))
{:table "my-table"
:key (uuid-gen)
:rows (-> (g/hash-map
:id (uuid-gen)
:external-reference-id g/string-alphanumeric)
(g/vector 1000) g/not-empty g/generate)}
how can I ensure that uuids generated for :key
and :id
are always unique?#2016-06-0917:45agI think I need to use something like gen/let
?#2016-06-0917:49lucasbradstreet@ag in cases where I wanted my uuids to be unique, and … thus uuid’y, I’ve just cheated by doing something like this: (gen/no-shrink (gen/fmap (fn [_] (java.util.UUID/randomUUID)) gen/int))
#2016-06-0917:50lucasbradstreetthe gen/int bit is especially more cheaty. There may be a nicer way to do it#2016-06-0917:51ag@lucasbradstreet: trying that, thanks#2016-06-0917:56agexcuse my complete noob question (I’m lazy). what is shrinking?#2016-06-0918:02lucasbradstreettest.check will shrink down your test cases so that it can find the smallest failing case e.g. integers might shrink from 232323976 to 0, a vector might shrink from a many element vector to a small one#2016-06-0918:02lucasbradstreetsince all we’re doing is generating a UUID, and not using the int generated by gen/int, there’s no point in shrinking this case#2016-06-0918:04lucasbradstreetOh, I can see what your problem is in that example#2016-06-0918:04lucasbradstreetHmm, maybe not#2016-06-0918:05lucasbradstreetwhat you’re doing wrong is actually generating a uuid sample via g/generate#2016-06-0918:05lucasbradstreetand then using gen/return#2016-06-0918:05lucasbradstreettry (gen/fmap str gen/uuid)#2016-06-0918:06lucasbradstreetThat’ll give you a uuid str generator#2016-06-0918:06lucasbradstreetYour problem was that you were essentially going (gen/return "5813d2ec-c486-4428-833d-e8373910ae14”)#2016-06-0918:06lucasbradstreetwhich will, of course, always return the same value#2016-06-0918:09agThank you for enabling my laziness and explaining in great detail 🙂#2016-06-0918:10lucasbradstreethappy to help spread the love and the word of the church of property test#2016-06-0918:28gfredericksag: lucasbradstreet: this might not be in the docstring but gen/uuid generates unique uuids#2016-06-0918:28gfredericksit's sort of in the docstring. It could be more clear.#2016-06-0918:29ag@gfredericks: yeah I think I’m doing something totally wacky here… no one to blame but myself.#2016-06-0918:29agstill learning#2016-06-0918:39lucasbradstreetYeah. I realised this which is why I investigated why the original solution didn't work. Thanks for saying that #2016-06-0918:41gfredericksI should also add things to the docstring for generate saying it's just a dev tool#2016-06-0918:41gfredericksag: ⇑#2016-06-0918:41gfredericksI bet people use it for real generators more often than I was expecting#2016-06-0918:42lucasbradstreetThe improved suggestion will be more reproducible than the one that I suggested. The original one will give different results when you use the same seed#2016-06-0918:43lucasbradstreetIt's easy enough to do starting out. Try some stuff out, and then you need a generator again so you go and stick it into return or elements without realising #2016-06-0918:43gfredericksyeah :/#2016-06-0918:44lucasbradstreetgen/sample does sound more devvy than gen/generate. I could see how people could get confused there#2016-06-0918:44lucasbradstreetA docstring note would be good anyway#2016-06-0918:45gfredericksyeah, probably on both of them#2016-07-2221:59zaneWhy is clojure.test.check.generators/make-gen
private?#2016-07-2221:59zaneWhat's the official way to make a generator from a thunk?#2016-07-2321:40gfrederickszane: what's the relationships between the thunk and the resulting generator?#2016-07-2400:06gfredericksif you just want to write your own (fn [rnd size] ...), the reason make-gen isn't public is that in most cases doing something so low-level isn't necessary#2016-07-2400:07gfredericksso I'm interested in what that function is doing so I can see if there's some higher-level way to accomplish the same thing#2016-07-2516:08zane@gfredericks: The thunk is literally producing the generated value. e.g. (fn [& _] (java.util.UUID/randomUUID))
.#2016-07-2516:09zaneI see now that make-gen
doesn't take a thunk (my bad!) but I'm still wondering what the right way to go about this is.#2016-07-2516:10zaneThe best thing we could come up with was: (clojure.test.check.generators/fmap (fn [_] (java.util.UUID/randomUUID)) (clojure.test.check.generators/return nil))
#2016-07-2516:10zaneBut that obviously feels pretty gross since the work of clojure.test.check.generators/return
is discarded.#2016-07-2516:16gfredericks@zane: would gen/uuid work for you or are you doing something else?#2016-07-2516:17zaneAh, I don't think gen/uuid
exists in the alpha we're using.#2016-07-2516:17zaneAssuming it doesn't, what would the right workaround be?#2016-07-2516:18zane(Maybe the answer is: Update your alpha. 😄)#2016-07-2516:18gfredericksYou could generate a collection of hex characters and use gen/fmap to convert then to a uuid#2016-07-2516:19zaneEek.#2016-07-2516:19lucasbradstreetYou can also generate two longs and build a uuid with it#2016-07-2516:20gfredericksIn general you want to use the randomness of the framework, not calling any random functions yourself, else you lose determinism and shrinking#2016-07-2516:20lucasbradstreetAh, my strategy doesn’t ensure uniqueness#2016-07-2516:20gfredericksNeither does mine#2016-07-2516:21lucasbradstreetThis is what you’d use if you don’t need uniqueness https://docs.oracle.com/javase/7/docs/api/java/util/UUID.html#UUID(long,%20long)#2016-07-2516:21lucasbradstreetI’ve also used with-redefs to override my internal random uuid generation function, and I just generated the seed via test.check#2016-07-2516:22lucasbradstreetA bit hacky, but it saved me from having to pass down a UUID seed and changing a lot of code#2016-07-2516:24gfredericksI'd just copy the code from master probably#2016-07-2516:29lucasbradstreetmmm#2016-07-2621:03agcan someone help me with creating a generator, that generates a map that contains keys like this
{
:amount-debit (gen/pos-int)
:amount-credit (gen/pos-int)
}
The problem is it has to be either debit or credit, and never both. And always one of them should have a value, they can't be null both at the same time#2016-07-2621:18metametadata@ag like this?
(def gen-card (gen/let
[card-type (gen/elements [:amount-credit :amount-debit])
value gen/s-pos-int]
(hash-map card-type value)))
(println "Sample:" (gen/sample gen-card 5))
;; Sample: ({:amount-debit 1} {:amount-debit 1} {:amount-credit 1} {:amount-debit 3} {:amount-credit 2})
#2016-07-2621:20ag@metametadata: hmmm interesting, my current code is slightly different, but I see what you’re suggesting. I’ll try to see if I can use this style without having to refactor a lot#2016-07-2621:21agthanks#2016-07-2621:38agAlthough it should generate a map with both debit and credit, yet only one of them should have a value#2016-07-2621:39agI started with gen/let
where I’m using is-debit gen/boolean
and then I think I want to try to use something like :amount-debit (g/such-that (fn [_] (= is-debit true)) g/pos-int)
#2016-07-2621:40agbut this fails with Couldn’t satisfy such-that predicate after 10 tries
#2016-07-2621:46agah nevermind using regular if
fork with pos-int and return 0 seems worked#2016-07-2622:03ageh, actually it didn’t, I need a need to have a let
block scoped insid gen/hash-map
block#2016-07-2622:04agor I have to assoc
two gen/hash-maps
somehow#2016-07-2622:07agis there a way to “merge” multiple hash-map
generators?#2016-07-2622:29agso this:
(g/let [debit? g/boolean]
(g/hash-map
:created-at (timestamp-gen)
:amount-debit (g/fmap #(when debit? %) g/pos-int)
:amount-credit (g/fmap #(when-not debit? %) g/pos-int)))
does not work 😞#2016-07-2622:49gfredericksum#2016-07-2622:49gfredericksI would do#2016-07-2622:51gfredericks(g/fmap (fn [[amount key created-at]] (assoc {:amount-debit nil :amount-credit nil :created-at created-at} key amount)) (g/tuple g/pos-int (g/elements [:amount-debit :amount-credit]) (timestamp-gen)))
#2016-07-2622:51gfredericks@ag ↑#2016-07-2702:13metametadata@ag @gfredericks your snippets both seem to work and produce similar results#2016-07-2702:47metametadatarefactoring @ag's version I came up with this (extracted amount
and gen-when
):
(defn gen-when
[x gen]
(gen/let
[x (gen/return x)]
(when x gen)))
(def gen-card
(gen/let
[debit? gen/boolean
amount gen/pos-int]
(gen/hash-map
:created-at gen/pos-int
:amount-debit (gen-when debit? amount)
:amount-credit (gen-when (not debit?) amount))))
Sample:
({:created-at 0, :amount-debit 0, :amount-credit nil}
{:created-at 0, :amount-debit nil, :amount-credit 0}
{:created-at 0, :amount-debit nil, :amount-credit 2}
{:created-at 2, :amount-debit 3, :amount-credit nil}
{:created-at 4, :amount-debit nil, :amount-credit 0})
I tend to eliminate bind
and fmap
because for me they're harder to read than let
.
I also like here how using gen/hash-map
allows to "inline" :created-at
generator.#2016-07-2702:50gfredericks@metametadata: your gen-when
ignores its first argument#2016-07-2702:50gfredericksoh I take it back#2016-07-2702:52gfredericksokay so if you want to use let then I think (gen/let [debit? gen/boolean, amount gen/pos-int, created-at gen/pos-int] {:created-at created-at :amount-debit (when debit? amount) :amount-credit (when-not debit? amount)})
would be a bit more direct#2016-07-2702:55gfredericksyou don't get to inline the created-at generator but on the other hand you don't have to create wrapper generators for credit/debit#2016-07-2702:56metametadatayeah, agree. I guess the approach will depend on the number of additional fields in the card map#2016-07-2702:57metametadataie if there are a lot of fields similar to created-at
than "inlining" is more preferable#2016-07-2703:05agactually my thing worked, I didn't know it. My ui was broken, I couldn't see it :)#2016-07-2703:25gfredericks@metametadata: you can also do both -- make two maps and merge them together#2016-07-2721:49mattlyhas anyone written about reducing generation time for creating large tree structures in test.check ?#2016-07-2721:50mattlyI added a new branch in my structure recently and it tripled the test run time#2016-07-2721:51mattlythe branch is basically (gen/list-distinct-by :name (gen/hash-map :name gen/string-alphanumeric))
and gets fmapped over with some other stuff#2016-07-2721:51mattlythe actual value of the string isn't important so much as that it's distinct#2016-07-2721:52mattlyso I'm looking at strategies for getting rid of the distinct clause in the list generator and making the value distinct some other way#2016-07-2721:53mattlyI'd also love to know if there's a way to peek a bit better at which parts of my tree are taking the most time generate#2016-07-2722:53gfredericksmattly: have you checked if the runtime increase roughly corresponds to an increase in the size of the generated data?#2016-07-2722:53mattlygfredericks: working on it#2016-07-2722:53mattlybut I think it does#2016-07-2722:53gfredericksif it does correspond, then my initial reaction would be "that sounds reasonable"; if not, then that could potentially be a problem worth looking into#2016-07-2722:54gfredericksunless I'm misunderstanding your question#2016-07-2722:54mattlyI don't think it's a problem of test.check per-se so much as the data structure I'm creating with it#2016-07-2722:55gfredericksmattly: how familiar are you with test.check's general sizing mechanism?#2016-07-2722:55mattlygfredericks: I've dug into the code a bit but haven't tried writing my own generator beyond composing from primitives#2016-07-2722:55gfredericksyou shouldn't need to normally#2016-07-2722:56mattlyyeah#2016-07-2722:56gfredericksI suppose I'm just wondering if you understand it well enough to know why (gen/vector (gen/vector (gen/vector (gen/vector (gen/vector gen/nat)))))
would OOM#2016-07-2722:56mattlyI'm pretty sure the problem stems from the need for distinct values where the value doesn't actually matter at a low point in the tree, and then having to generate all the child branches#2016-07-2722:56mattlyah#2016-07-2722:57mattlyyeah I do#2016-07-2722:57mattlyI've run into that quite a bit on Circle 😄#2016-07-2722:57mattlymost of my lists / vectors anymore I put max-elements on#2016-07-2722:58gfredericksI think the distinct generator is highly unlikely to be slow without also throwing exceptions#2016-07-2722:58mattlyok#2016-07-2722:58gfredericksi.e., if you think the problem is that it has to throw away a bunch of stuff because of collisions#2016-07-2722:59mattlythat was my guess#2016-07-2722:59gfredericksit'll throw an exception if it can't generate a unique thing after ~10 tries#2016-07-2722:59gfredericksso it's hard to imagine a scenario where it expends much effort on this without hitting 10 occasionally#2016-07-2722:59mattlyfair point#2016-07-2723:58mattlyI got some instrumentation up around the generators and my test runs#2016-07-2723:59mattlyeliminating the need to generate distinct values on one branch of my tree cut the test run time in half#2016-07-2723:59mattlyI'm basically putting those values in now after-the-fact similar to how you might do serial ids#2016-07-2800:00mattlyI found a few other places in the actual generation of things where I'm being wasteful#2016-07-2800:00mattlybut cutting that out wasn't nearly quite as effective as getting rid of distinct requirement#2016-07-2800:03mattlybtw, thanks for all the work on test.chuck, checking
and subsequence
are invaluable tools in my toolbox these days#2016-07-2800:03gfredericks:)#2016-07-2800:04gfredericksI don't quite understand what you mean by "putting those values in now after-the-fact"#2016-07-2800:04mattlyhm#2016-07-2800:04mattlyI'll do an example#2016-07-2800:05gfredericksFYI distinct collections are generated by maintaining the elements in a transient set: https://github.com/clojure/test.check/blob/master/src/main/clojure/clojure/test/check/generators.cljc#L546#2016-07-2800:05mattlyoh interesting#2016-07-2800:06gfredericksso it's not inconceivable that the overhead of adding and looking up elements is slowing things down#2016-07-2800:06gfredericksif that's really your issue I have a hard time imagining how to make it faster#2016-07-2800:07mattly(gen/fmap (fn [things] (map #(assoc %2 :name (str %1 "-" (:name %2))) (range) things)) (gen/list (gen/hash-map :name gen/string-alphanumeric)))
#2016-07-2800:07mattlyyeah#2016-07-2800:08mattlyI get that, and really for me and my use case it comes down to, am I using the generator for actual values I want to test or just random input?#2016-07-2800:08mattlyand in the case of these names, it's just random input that needs to be distinct#2016-07-2800:09gfredericksI suppose you probably have to compute hash values for the data when you wouldn't otherwise#2016-07-2800:09mattlyI won't get much value of shrinking#2016-07-2800:10mattlywhere I get the value of shrinking here is the number and depth of branches and the leaf values#2016-07-2800:10mattlybut not the id#2016-07-2800:12gfredericksFYI gen/uuid is evenly distributed and doesn't shrink, so you could use that for uniqueness without needing to check#2016-07-2800:13mattlyyeah, that occurred to me as well#2016-07-2800:13mattlybut this works better for my use-case#2016-07-2800:13mattlyit's... complicated#2016-07-2800:14mattlyI'm working on a system to let people do self-serve analytics on a data warehouse, but with a complicated permissions structure on top of it#2016-07-2800:15mattlyfrom my experience doing similar things in the past, i know that scenario-based testing has its own set of gotchas#2016-07-2800:16mattlyso I'm basically generating the dimension/fact graph and putting that into our data store, which, well, it's not what I would have chose#2016-07-2800:17mattlyproperty-based testing of it though has helped me catch a ton of bugs in the prototype I'm replacing that I don't think anyone would have ever thought to look for#2016-07-2806:04mattlyhaving gotten rid of the distinct name requirements across my entire gen'd tree, and doing some other things around the frequency of size and depth of branches, I cut the run for my test suite down to 1/10th of what it was before#2016-07-2806:04mattlyand it still shrinks awesomely#2016-07-2806:05mattlyI also included in the trunk generator some flags to turn off certain branches of the tree if they're not needed for a test#2016-07-2814:37gfredericks@mattly do you think parallelizing tests and/or generators would help out?#2016-07-2814:44gfredericksI've worked on parallelizing tests before, but it just occurred to me that slow generators could be parallelized even if the tests themselves can't#2016-07-2814:45gfredericksE.g., during the test run you have one or more background threads doing the generating while the main thread does the actual tests#2016-07-2814:48mattlyI'm not sure, tbh#2016-07-2814:49mattlyOne thing I'm looking into now, after a deeper branch ends up with 100k+ nodes when it starts to fail, is exponential scaling of the size of nodes#2016-07-2814:50mattlywhich would actually fit the shape of the data I'm trying to model well#2016-07-2816:13gfredericksYou're saying you like it being so large? #2016-07-2817:16mattlyeh, well, that's the count of something akin to (gen/vector (gen/vector (gen/vector gen/int)))
but flattened; and I've found a few bugs that have only manifested when the node size gets that large#2016-07-2817:17mattlyshrinking, of course, will end up narrowing that down to like the 2 or 3 end nodes that cause the failure#2016-07-2817:17mattlyand really it's more due to the nature of the data I'm working with / replicating, and the complex query sets I have to run on top of them#2016-07-2817:21mattlyas I find specific cases like that I tend to break out specific tests for that data scenario#2016-08-0212:59rickmoynihanis there a good resource to help understand shrinking; how it works, what to consider etc...#2016-08-0212:59rickmoynihanit seems like quite an involved topic#2016-08-0213:29lucasbradstreetI’m interested in this. I have a reasonable intuition about it, but I’d like something in more detail#2016-08-0214:01gfrederickswell I can answer specific questions about it at least#2016-08-0214:01gfredericksfor the most part the shrinking strategy mirrors the structure of your generator#2016-08-0214:02gfrederickse.g., tuple shrinks by shrinking its elements; collections shrink by deleting elements and shrinking elements#2016-08-0214:02gfredericksfmap shrinks by shrinking the underlying element and running it through the function again#2016-08-0214:09gfredericksbind is a little weird though, and therefore so is gen/let#2016-08-0215:04gfredericks@alexmiller: regarding the test.check issue described in CLJ-1997, I'm considering pasting all the relevant code from riddley into test.check as the most practical option. I'm wondering if you see any problems with doing something like that for a contrib library.#2016-08-0217:54mattlyI've started moving away from generators like gen/frequency
that don't shrink in favor of those that provide predictable shrinking behavior such as gen/one-of
#2016-08-0217:55mattlythe idea being that when there are multiple possible values that require different generators, it will shrink along an axis of my choosing#2016-08-0217:57mattlyfor example, I have a node on the tree structure I generate where most of the time the value is an element from a collection, but sometimes it's not – however for most of the problems I encounter, that doesn't matter#2016-08-0217:57mattlyso I setup the generator like so:#2016-08-0217:58mattlyinstead of using gen/frequency
to mimic the structure of my inputs#2016-08-0217:59mattlyit will definitely test the case where there's a thing, but will shrink towards there not being a thing#2016-08-0217:59mattlywhich reduces the noise I have to contend with when troubleshooting a failure#2016-08-0218:00mattlyin some cases where things
is a fixed size, I actually use gen/one-of
to force a shrinking preference, since gen/elements
is always random; you'd have to shrink the size of the input if you wanted predictability#2016-08-0218:01mattly(and feel free to correct me if I'm wrong – I'm going off of observation from usage and cursory readings of the code)#2016-08-0218:11gfredericks@mattly: what makes you say gen/frequency
doesn't shrink?#2016-08-0218:12gfredericksoh I see; I'm inclined to say that's a bug#2016-08-0218:12mattly@gfredericks: my experience of using it, that example before, the frequencies didn't seem to matter when shrinking#2016-08-0218:12gfredericksI would think it should shrink toward earlier generators like one-of
does#2016-08-0218:12mattlyso f.e. [[3 (gen/return nil)] [1 (gen/elements things)]]
it'll still pick gen/elements#2016-08-0218:13mattlythat'd be even better#2016-08-0218:13gfrederickswould you like to file the bug or shall I?#2016-08-0218:14mattlygo for it, i don't think I have an account on the jira#2016-08-0218:14mattlyI probably should#2016-08-0218:14mattlyand you'd know more about what to say about it 😄#2016-08-0218:18gfredericks@mattly: http://dev.clojure.org/jira/browse/TCHECK-114#2016-08-0218:18mattlyI'm planning to do a writeup on my usage of test.check and property-based testing on the system I'm working on#2016-08-0218:19mattlythere's a lot of material out there on what it is, but very little on how to use it effectively#2016-08-0218:21mattlythis has been such a success on my system, I'm teaching our QA guy how to write generators#2016-08-0218:22gfredericksnice#2016-08-0306:04lucasbradstreet@mattly cool, let me know if you want some eyes on it before you release it 🙂#2016-08-0414:22gfredericks@alexmiller: bumping the question above about pasting riddley into test.check#2016-08-0414:23gfredericksI just looked at the source and it's just a few hundred lines. I'm guessing licensing/CA issues would be the biggest thing?#2016-08-0420:07Alex Miller (Clojure team)sorry, I didn’t belong to this channel so didn’t get any notifications#2016-08-0420:13Alex Miller (Clojure team)haven’t read all the back chat yet but pulling riddley into test-check would require some things wrt license and contribution#2016-08-0420:14Alex Miller (Clojure team)maybe this isn’t a problem you should be solving in test.check#2016-08-0420:14gfredericksyeah, that's why I opened CLJ-1997; I'd rather not solve it :)#2016-08-0420:15gfredericksunless you're just objecting to it being a problem#2016-08-0420:15gfredericks(the gen/let
thing in particular)#2016-08-0420:17Alex Miller (Clojure team)if I read all this correctly, let allows you to handle both dependent and independent generators but because it can’t tell the difference it performs the more complicated dependent expansion#2016-08-0420:18gfredericksright#2016-08-0420:18Alex Miller (Clojure team)to which maybe I would suggest that you should add something that just lets you do the independent generator case with the less complicated expansion rather than trying to detect the difference#2016-08-0420:18Alex Miller (Clojure team)and/or to let users tell you which is which#2016-08-0420:18gfredericksI suggested a much more complex macro a while back (analagous to for
) but reid didn't like it :)#2016-08-0420:18gfredericksso I agree with you actually#2016-08-0420:19Alex Miller (Clojure team)sounds like a quorum to me :)#2016-08-0420:19gfrederickshaha#2016-08-0420:19Alex Miller (Clojure team)see how easy that was? :)#2016-08-0420:19gfredericks@alexmiller: do you think a fully-featured analog to clojure.core/for
would be a good fit?#2016-08-0420:20gfredericks(examples here: https://github.com/gfredericks/test.chuck#for)#2016-08-0420:20gfredericksthe recomendation would then be "if you care about optimizing shrinking, use for
instead of let
"#2016-08-0420:20Alex Miller (Clojure team)yeah, I’ve looked at it in the past#2016-08-0420:21Alex Miller (Clojure team)it seems fine, but I can’t say for
as a name had much resonance for me#2016-08-0420:21gfredericksah yeah; I was just making the analogy to clojure.core/for
I guess#2016-08-0420:21gfredericksmonads and whatnot :)#2016-08-0420:21gfrederickslet+
#2016-08-0420:22Alex Miller (Clojure team)over
#2016-08-0420:22gfrederickso_O#2016-08-0420:22Alex Miller (Clojure team)I’m not known for being very successful at suggesting good names :)#2016-08-0420:23Alex Miller (Clojure team)but I would prefer something that’s not a variant of let
#2016-08-0420:23Alex Miller (Clojure team)and I think it’s different enough from clojure.core/for that I’d be a little leery of overlapping that name#2016-08-0420:23gfrederickswhat sort of difference are you thinking of?#2016-08-0420:23Alex Miller (Clojure team)but maybe I could be convinced#2016-08-0420:24gfredericks:parallel
is an extra feature; but I think every aspect of clojure.core/for
has an analog#2016-08-0420:25Alex Miller (Clojure team)what does parallel do again?#2016-08-0420:25gfredericksit's exactly the thing you use to solve the problem with let
#2016-08-0420:25Alex Miller (Clojure team)just independent#2016-08-0420:25gfredericksit's how you say "these are independent"#2016-08-0420:25gfredericksyeah#2016-08-0420:26gfredericksyou could imagine clojure.core/for
having something analogous that would let you iterate over two seqs in parallel#2016-08-0420:28Alex Miller (Clojure team)I guess putting on my Rich hair, I would say: what problem are we trying to solve and what are some alternatives and tradeoffs between them?#2016-08-0420:28Alex Miller (Clojure team)this is one option#2016-08-0420:28Alex Miller (Clojure team)what are some others#2016-08-0420:28gfredericksthe problem is that gen/let
is very easy to use and expressing parallelism is comparably difficult (`gen/tuple` and destructuring)#2016-08-0420:29gfredericksand so people will either naively use gen/let
or else be grumpy that they can't#2016-08-0420:30Alex Miller (Clojure team)there’s a lot packed into for (some might argue too much :) - is there some way to take it apart into reusable pieces?#2016-08-0420:30gfredericksit's also possible to backwards-compatibly just add :parallel
to gen/let
#2016-08-0420:31gfredericksI literally read "hat"#2016-08-0420:31gfredericksso I needed you to point it out but now I enjoy it#2016-08-0420:31Alex Miller (Clojure team)good, thanks :)#2016-08-0420:31Alex Miller (Clojure team)hereby trademarking that#2016-08-0420:32Alex Miller (Clojure team)unfortunately, I have to go pick up my dog right now, but I think it’s worth considering the tradeoffs of adding :parallel to let#2016-08-0420:32gfredericksyeah I like that idea more now that you've made me think about it#2016-08-0420:33Alex Miller (Clojure team)pulling in either riddley or tools.analyzer seems like a large step so I would try to avoid it if possible#2016-08-0420:33gfredericksACK#2016-08-0716:05gfredericksI'm taking a twitter poll on options for independent bindings in gen/let: https://mobile.twitter.com/gfredericks_/status/762311329470447616#2016-08-0716:05gfredericksSo far people like the map syntax, which surprised me /cc @alexmiller #2016-08-1600:18gfredericksgoing to try to tackle parallel tests once and for all#2016-08-1702:52lvhwoop woop 🙂#2016-08-1702:55gfredericksit's more of a headache than it should be :(#2016-08-1714:41lucasbradstreet@gfredericks: re our twitter conversation, what’s your solution for generators with high variance? I think maybe it’s ok, because the common case where you’re using parallel testing will be tests passing#2016-08-1714:42lucasbradstreetif your tests are failing often, then generally parallelism isn’t really all that helpful anyway. Most of the time when I run test.check running overnight in #core JVMs, I do not expect to catch anything#2016-08-1714:47gfredericksIt can slow down even in the passing case#2016-08-1714:47gfredericksOh I forgot another goal#2016-08-1714:47gfredericksGiven a seed, the result should be deterministic#2016-08-1714:48gfredericksSo if you're preallocating work you have to give n/t trials to each thread#2016-08-1714:48lucasbradstreetYeah, I agree with that goal. That’s why I was saying that you have to partition the space and check whether all threads have passed X, where X is a failing iteration#2016-08-1714:48lucasbradstreetI can see how it’ll slow down passing cases though#2016-08-1714:48gfredericksIf one thread has lots of slow trials the others will be idle while it finished#2016-08-1714:48lucasbradstreetYeah, that’s troublesome#2016-08-1714:49lucasbradstreetEspecially if you have a bunch of tests to run through#2016-08-1714:49gfredericksSo my alg has a queue of trials to run#2016-08-1714:49lucasbradstreetGenerally I’m just running one test overnight, on X cores#2016-08-1714:49lucasbradstreetSo this isn’t a problem, but I see where it’s a problem for a test suite#2016-08-1714:50gfredericksAnd the threads just pull them off the queue#2016-08-1714:50lucasbradstreetI was going to say, a priority queue might be the answer#2016-08-1714:50gfredericksPriority? #2016-08-1714:50lucasbradstreetMaybe you don’t even need to prioritise, now that I think about it#2016-08-1714:50gfredericksYeah#2016-08-1714:51gfredericksThe other problem I have is deciding how to manage the threads#2016-08-1714:51gfredericksI suppose I could have it just use N futures...?#2016-08-1714:51lucasbradstreetYou also need to ID the test cases, in case there is some variance in when the cases finish?#2016-08-1714:51gfredericksMaybe that's best actually#2016-08-1714:51gfredericksYes they're ordered#2016-08-1714:52gfredericksSo a lot of complexity comes from when you find the first failure but have to wait for all earlier jobs to finish first#2016-08-1714:52lucasbradstreetYep, I can see that#2016-08-1714:54lucasbradstreetIf you use futures, make sure to check for (Thread/interrupted) in the processing loop, so you can cancel the thread reliably after the test is done#2016-08-1714:57gfredericksyeah cancelling things is a big deal too#2016-08-1714:58gfredericksI don't know how bad it is to .stop
a thread#2016-08-1714:58lucasbradstreetYeah, it doesn’t really work well#2016-08-1714:58lucasbradstreetIt’s really up to the thread to behave well#2016-08-1714:59lucasbradstreetif you check whether you’ve been interrupted in between test iterations, you’ll be fine with just (future-cancel fut)#2016-08-1714:59gfredericksC-c in nrepl calls .stop I think#2016-08-1714:59lucasbradstreetfuture-cancel may do too, but it’s really up to the thread to behave properly#2016-08-1715:00lucasbradstreetpretty sure thread.stop has been deprecated too#2016-08-1715:01lucasbradstreetyep, https://stackoverflow.com/questions/16504140/thread-stop-deprecated#2016-08-1715:01gfredericksyeah that's the entire reason for my hesitation#2016-08-1715:02gfredericksit's tough in this case to ensure the thread behaves properly though since it's running user code, and so might hang#2016-08-1715:02gfredericksso if I don't call thread.stop, the caveat to users is "if your test or your generator hangs, you're just going to have an extra thread on your jvm until you restart it, sorry"#2016-08-1715:03lucasbradstreetif user code hangs, it’s not your problem in my opinion#2016-08-1715:03lucasbradstreetbest you can do is exit between iterations#2016-08-1715:03gfrederickssure, but users have the expectation that they can C-c their tests and everything stops#2016-08-1715:03gfrederickswhat is "exit between iterations"?#2016-08-1715:04lucasbradstreetI agree, but there’s nothing you can do to guarantee that. User code can always cause situations where that is not true#2016-08-1715:04lucasbradstreetI mean, a thread will, at minimum, stop between an test iteration#2016-08-1715:05lucasbradstreetas far as I know, there may be other situations where their code can bail safely, but there can always be other situations where things are blocked and C-c is never going to help#2016-08-1715:06gfredericksthat's true when users are explicitly using threading constructs#2016-08-1715:06lucasbradstreetwhen I say “may”, I mean, it depends on their code#2016-08-1715:06gfredericksbut for user code on the main thread I think it's always true that C-c in nrepl stops it#2016-08-1715:07gfredericksI suppose this discussion overlaps with the "should tests run in parallel by default?" question#2016-08-1715:09lucasbradstreetAh yes, I can see how that is important #2016-08-1715:17gfredericksI think using futures and using an atom to aggregate the results will simplify that code a lot#2016-08-1715:18gfredericksso I'm glad about that; #2016-08-1715:19gfredericksI'm a little grumpy about how this interacts with the reporter-fn, since I think that has to be nondeterministic now#2016-08-1715:19gfredericksand the behavior when the reporter-fn throws an exception is potentially weirder#2016-08-1718:47Alex Miller (Clojure team)calling .stop on a thread is bad - it frees all monitors which can badly impact others using them#2016-08-1718:48Alex Miller (Clojure team)much preferred to interrupt#2016-08-1720:07gfredericks@alexmiller: do you think it's a problem that nrepl does it on ctrl-c?#2016-08-1720:08gfredericksshould we be resigned to forcing repl users to restart their jvm if they execute an infinite loop?#2016-08-1720:09gfredericksI see ctrl-c at the builtin clojure repl just kills the whole jvm#2016-08-1720:11gfredericksI'm also curious if you have any sense for how realistic the monitor problem is to a user who isn't explicitly using monitors#2016-08-1720:28Alex Miller (Clojure team)I don’t know that there is any better solution for nrepl, but that doesn’t make it good#2016-08-1720:29Alex Miller (Clojure team)the thread stop thing is a real problem, which is why its deprecated. any code that uses Java locks will be affected and that includes code that is calling into the Clojure runtime in places where synchronization is used (every lazy seq uses synchronization).#2016-08-1720:31Alex Miller (Clojure team)that said, typically at the repl, the person typing is the one likely to be affected and they will probably just start over with (mostly) fresh resources#2016-08-1720:32Alex Miller (Clojure team)Java Concurrency in Practice is probably the best resource on these problems and their recommended alternatives#2016-08-1721:44gfredericksYeah that's about what I thought. Seems like there's not much you can do in a general context where you don't control all the code. I'm surprised it hasn't been worth fixing (or is too hard to fix) on modern jvms#2016-08-1902:03lvhWhat’s the generator equivalent of “repeatedly”? I know return
is constantly
. Use case is: generate nested data structure with random (but no-shrink, because they don’t affect execution) byte arrays#2016-08-1902:04lvhI think the answer is (defn frepeatedly [f] (fmap #(f) (return nil)))
#2016-08-1902:04lvhor something#2016-08-1902:10gfredericksum#2016-08-1902:11gfredericksyou just want to generate collections of byte arrays?#2016-08-1902:11gfredericksit sounds like you just described (gen/list (gen/no-shrink gen/bytes))
maybe?#2016-08-1902:17gfredericks@lvh ⇑#2016-08-1902:19lvh@gfredericks: Well, sorta. I want byte arrays of a specific size (they’re cryptographic keys) — I already have a function for doing that, hence the phrasing of the question#2016-08-1902:20lvhI don’t actually care that they’re cryptographically random for these tests, but have a mild preference for just using the same tool#2016-08-1902:21gfredericksif you don't use test.check's randomness then you won't have the reproducibility you normally have#2016-08-1902:22gfredericksbut if you're convinced it won't affect your tests and don't want to bother reimplementing as a generator, I'd just "generate" that stuff as part of running your test and not in the generator#2016-08-1902:22gfredericksfixed-length byte arrays wouldn't be hard to implement though#2016-08-1902:26gfrederickse.g., (gen/fmap byte-array (gen/vector (gen/choose -128 127) 12))
#2016-08-1902:26gfredericksthat should give you a uniform distribution#2016-08-1902:26lvhOK, thanks 🙂 For clarity, the value doesn’t matter, but the length does; any byte array that isn’t exactly this length is wrong and won’t work#2016-08-1902:29gfredericks(gen/return (byte-array (repeat 12 0)))
:P#2016-08-1902:31lvhI do want them to be different; because I’m deriving keys, so seeing the value that got produced (if the inputs are different) tells me what really happened#2016-08-1902:32lvh(opaquely, of course :))#2016-08-1902:33gfredericksah#2016-08-1902:33lvhI think I might go with the approach you suggested of just generating them separately; or using test.check to generate steps rather than doing the operations directly 🙂#2016-08-2323:42gfredericksdiscussing parallelizing test.check on twitter is hard#2016-08-2403:09Alex Miller (Clojure team)s/test.check/anything/#2016-08-2917:06lvhIs there something in test.check for producing equal-but-internally-different data structures? (Trying to check if a serialization format is deterministic).#2016-08-2918:57lucasbradstreet@lvh where two data structures are equal but will serialize/deserialize to something equal but distinct? e.g. clojure maps and records can be equal, but a record should round trip back to a record#2016-08-2918:58lvhOh, right — that’s a good point 🙂#2016-08-2918:59lvhI guess I’m mostly thinking of issues like map key order or set order#2016-08-2919:02lucasbradstreetAh. Those two are interesting although, at least in clojure, I don’t think/don’t count them as part of the contract#2016-08-2919:03lucasbradstreethttps://github.com/ztellman/collection-check might be handy if you’re interested in stuff like maps/sets/etc, though it wasn’t really built for what you want#2016-08-2920:23Alex Miller (Clojure team)maps and sets are not ordered - equality does not consider them (and there are no guarantees around it)#2016-08-2920:34lucasbradstreetI remember from the google groups discussion that the order of keys and the order of vals, on an identical? map, are guaranteed to be in the same order, but that was it#2016-08-2923:24gfredericks@lvh I think if you could enumerate the sort of distinctions you're aiming at it wouldn't be hard to put together yourself #2016-08-2923:25gfredericks(defn gen-tweak "Returns a generator of variants of x such that (= x x')" [x] ...)#2016-08-3000:57lvhYeah; I was hoping that was implemented already because I’ll probably miss a bunch 🙂#2016-08-3001:04gfredericksit'd be a weird thing to include, because it's open-ended and also inherently suspicious of clojure itself#2016-08-3001:10lvhyeah#2016-08-3001:10lvhcanonical serialization is desirable in a lot of cases though 🙂#2016-08-3001:10lvhFortunately I can mostly deal with just local storage, so I don’t really need it#2016-08-3001:11gfredericksalso it depends on clojure impl details#2016-08-3001:12lvhright; which is why I want it as a test and not as some manual verification#2016-08-3001:12lvhI want a computer to tell me when it stops being true before it matters 🙂#2016-08-3001:15gfredericksfeel free to add something to test.chuck#2016-09-0320:32gfrederickswriting a quickcheck is hard#2016-09-0520:06lvhhm#2016-09-0520:08lvhI have a seq of generators; I’d like a generator that concats the seqs; how do I do that while keeping shrinking?#2016-09-0520:08lvh(IIUC sample would work, but would break shrinking.)#2016-09-0520:09lucasbradstreetCan you just do (gen/fmap #(reduce into [] %) (apply gen/tuple seq-of-generators)#2016-09-0520:10lvh(I also just realized that I can just make the generator later; i.e. first compute the concat’d thing, then chuck.gen/subsequence from that)#2016-09-0520:10lucasbradstreetYeah, that would also work#2016-09-0520:11lvhthanks 🙂 that was stumping me until I wrote it out 🙂#2016-09-0520:12lucasbradstreet🦆#2016-09-0520:12lvhexactly#2016-09-1520:31aengelbergIs there a way to get a variant of gen/any
that does not return NaN
?#2016-09-1520:32aengelbergMy use case is that I want to do an operation on two different types of datastructures and verify that the two results are equal. The problem is that, when the outputs are structurally equal but both have NaN
deeply nested inside of them, =
will return false.#2016-09-1520:34aengelbergSo, the generator can't just not return NaN, I would need it to not return NaN at any level of nesting.#2016-09-1523:28gfredericksman I needed that for test.check itself#2016-09-1523:29gfredericksI wonder what I did#2016-09-1523:29gfrederickshttps://github.com/clojure/test.check/blob/master/src/test/clojure/clojure/test/check/test.cljc#L1083#2016-09-1523:30gfrederickslooks like I used a custom recursive generator#2016-09-1523:32gfredericksIt was oriented at edn which afaik can't handle infinities either#2016-09-1523:33gfredericksBut NaN is pretty poisonous #2016-09-1523:33gfredericksYou could add something to test.chuck if you'd find that useful#2016-09-2118:39Alex Miller (Clojure team)@gfredericks curious if these failures are meaningful to you http://dev.clojure.org/jira/browse/CLJ-2026#2016-09-2118:39Alex Miller (Clojure team)in particular, if they are known things fixed since 0.9#2016-09-2917:32aghey guys… does anyone know why
(defspec foo-test 100
(prop/for-all [m (s/gen ::bool-kv)] ,,,
is failing when run in clojurescript with lein-doo in phantomjs? throws this:
ERROR in (foo-test) (
#2016-09-2917:54lucasbradstreet@ag my work mate had a similar problem to that and it was due to not requiring something#2016-09-2917:54lucasbradstreetI was surprised that cljsbuild didn’t warn about it#2016-09-2917:54lucasbradstreetYour mileage may vary, but it’s worth checking#2016-09-2917:55agactually same code works when compiled javascript directly passed to karma (without using lein-doo)#2016-09-2917:55agI want to use lein-doo though#2016-09-2918:05lucasbradstreetAre they using different compiler modes? i.e. advanced vs none?#2016-09-2918:07agit completely breaks with :simple. java.lang.AssertionError: Assert failed: No file for namespace my-app.test-runner exists
#2016-09-2918:09agwith :whitespace it fails with a different error:
Error: Cannot find module 'function (f) {if(typeof exports==="object"&&typeof module!=="undefined"){module.exports=f()}else if(typeof define==="function"&&define.amd){define([],f)}else{var g;if(typeof window!=="undefined"){g=window}else if(typeof global!=="undefined"){g=global}else if(typeof self!=="undefined"){g=self}else{g=this}g.React = f()}}’
#2016-09-2918:10lucasbradstreetSeems like a require error, but I don’t know too much more#2016-09-2918:11agI have created issue https://github.com/bensu/doo/issues/118#2016-09-2918:13lucasbradstreetI think it’s quite likely not to be a doo error#2016-09-2918:14lucasbradstreetTry requiring clojure.test.check.generators#2016-09-2918:15lucasbradstreetPossibly shouldn’t need to, but I think it might be a bug in the cljs compiler if it’s not warning about it#2016-09-2918:16agstill the same 😞#2016-09-2918:16lucasbradstreetI asked my co-worker and his was clojure.test.check.properties#2016-09-2918:17agI do have that require#2016-09-2918:17ag[clojure.test.check.properties :as prop]
#2016-09-2918:18lucasbradstreetOK, I’ve got nothing then 🙂#2016-09-2918:19agso I’m wondering if it has something to do with how I require clojure.spec.generators?#2016-09-2918:20agso here’s the thing to require clojure spec, in clojurescript you can either do require '[cljs.spec,,,
or require '[clojure.spec,,,
#2016-09-2918:20agbut when it comes to generators it’s not that smart#2016-09-2918:21agyou can’t do require '[clojure.spec.generators]
#2016-09-2918:21lucasbradstreetMaybe#2016-09-2918:21agI found a way to include them like this [cljs.spec.impl.gen :as gen]
#2016-09-2918:21agbut maybe that’s not completely right?#2016-09-2918:22agbut then why the same code compiled and run outside of doo works fine?#2016-09-3018:14liamdi was wondering how to write good properties for property based testing#2016-09-3018:14liamdfor instance i have this function:
(defn pixel->tile [position tile-size]
{:x (Math/floor (/ (:x position) tile-size))
:y (Math/floor (/ (:y position) tile-size))})
#2016-09-3018:14liamdwhich is pretty simple#2016-09-3018:15liamdi suppose i’d want to check that the output is non-nil and > 0? but that doesn’t seem to ensure the logic of it#2016-09-3018:15liamdis this too simple for property based testing?#2016-09-3020:24lucasbradstreetIt’s a pretty simple function, so it might be hard to come up with something much better. You could check whether the returned x is <= to the position x#2016-09-3022:17gfredericksyou could probably come up with a distance-scaling property#2016-09-3022:17gfredericksa relationship between (distance p1 p2) and (distance (f p1) (f p2))#2016-09-3022:18gfredericksor more generally the 2d-vector representing the offset between them#2016-10-0115:15gfredericksanother idea: "adjacent pixels are mapped to the same or adjacent tiles"#2016-10-2414:45nwjsmithHas anyone had experience adding 'famous' examples to their test.check runs?#2016-10-2414:47nwjsmithI think it would be valuable for regression testing, but I suppose that's what clojure.test is for.#2016-10-2415:28hiredmanI think the same thing that would let you do that, also makes it easier to run you test over and over again with a given set of inputs if you do hit a failure, which is useful when you do find a failure#2016-10-2415:29hiredmanthe way to do that, I think, is to have your, uh, assertions, as separate functions#2016-10-2415:29hiredmanwhich you can call from properties or other tests#2016-10-2415:31nwjsmithThat would work#2016-10-2415:36nwjsmithI'm experimenting with using spec/test.check to do TDD. It's proving more challenging than I thought it would be. Often it's easier to come up with an example than a property/invariant.#2016-10-2418:43gfrederickswriting properties takes practice#2016-10-2418:43gfredericks"I don't know what properties to write" is a common problem people run into at first#2016-10-2418:45Alex Miller (Clojure team)the only hard parts of property testing are the generators and the properties#2016-10-2418:45Alex Miller (Clojure team);)#2016-10-2418:56nwjsmithI better get practicing then 🙂 Hopefully in a few years, there will be much more guidance on how to do property-based testing. There's so much out there on example-based now.#2016-10-2713:31Yehonathan Sharvithttps://twitter.com/viebel/status/791633195016003584#2016-10-2713:32Yehonathan Sharvitwhat do you guys think of this interactive tutorial? Did you learn something about test.check
?#2016-10-2713:46lucasbradstreet@viebel: oh sweet! 👍I'll give it a go when I get home and will let you know if I have any comments #2016-10-2713:48Yehonathan Sharvit@lucasbradstreet It’s kind of the same idea as the onyx tutorial @michaeldrogalis shared on #onyx but it has much more content. It’s all about Code Interactivity in the browser.#2016-10-2719:32Yehonathan Sharvit@lucasbradstreet test.check official doc with klipse will look like this#2016-10-2719:32Yehonathan Sharvithttps://viebel.github.io/klipse/examples/test.check/doc/generator-examples.html
https://viebel.github.io/klipse/examples/test.check/doc/clojure.test.check.html#var-quick-check#2016-10-2806:15lucasbradstreet@viebel so great!#2016-10-2806:15Yehonathan Sharvit@lucasbradstreet which one? the blog post or the interactive doc?#2016-10-2806:19lucasbradstreetBoth are great. I love the docs versions with inline testable examples though. I want it everywhere 😄#2016-10-2806:21lucasbradstreetIn the example where the property is incorrect i.e.
(let [s (sort v)]
(< (first s) (last s)))))
it might be useful to say that both properties and the code they test can be incorrect. Usually you’re trying to start from a correct property and find a bug in your code, but sometimes your code ends up testing your properties instead. I’m not sure how obvious it’ll be to a new user#2016-10-2812:14Yehonathan Sharvitgood input#2016-10-2812:14Yehonathan Sharvithow would you fix it @lucasbradstreet ?#2016-10-2812:16Yehonathan Sharvit@lucasbradstreet you can live interactive doc with the codox klipse theme. It’s super easy to integrate in any lib that is cljs ported#2016-10-2812:16Yehonathan Sharvithttps://github.com/viebel/codox-klipse-theme#2016-11-0521:19gfredericksI'm messing around with a protocol for returning data with a trial: https://github.com/clojure/test.check/commit/a7860f25a4d6755e71d5fc13907279e16059c1f5#2016-11-0820:18mattlyis there any way to tell test.check to give up and just show me the smallest failing result so far after a certain number of iterations?#2016-11-0820:18mattlyI've got one failure that I'm having a really hard time tracking down, but it'll run for 100k iterations without settling#2016-11-0821:29gfredericks@mattly there are a couple tickets surrounding that#2016-11-0821:30gfredericks@mattly there is also a callback function you can register on the master branch, that would let you "listen" for things as it's shrinking#2016-11-0821:31gfredericksthere's a ticket for a max-shrink-time option, and for capturing C-c interruptions and returning the smallest value so far#2016-11-0821:31gfredericksI think the latter one is hairy though so I don't know if it will get implemented :/#2016-11-0821:32gfredericksC-c
would probably be the most useful, so if it turns out to be straightforward I would love to add it#2016-11-0821:33gfredericksif it's hairy it's probably because of varying details in different repls vs. unix signals when running test processes and that sort of thing#2016-11-0821:51mattlyoh fun#2016-11-0908:32lucasbradstreetmax-shrink-time would be nice. I sometimes resort to dumping the first shrink and then manually shrinking it via other strategies for some of my tests. The callback function would be awesome though - even selectively printing the failures as it shrinks would be helpful#2016-11-0913:20gfredericksI figured max-shrink time would be useful for CI#2016-11-0913:20lucasbradstreetAbsolutely#2016-11-0913:21lucasbradstreetEven just printing the seed before a shrink would be helpful, to be honest.#2016-11-0913:21lucasbradstreetOr giving us a way to do so. The callback idea would work though.#2016-11-0913:33gfredericksWell the callback is already implemented simple_smile #2016-11-0913:38lucasbradstreetLooking forward to trying it#2016-11-1105:57mattlyI have a wrapper around test.chuck's checking
that prints iteration count as it goes solely to prevent CI from timing out 😄#2016-11-1105:57mattlywell, and a nice guide for me#2016-11-1111:24lucasbradstreetCool#2016-11-1114:21gfredericksha#2016-11-1114:21gfredericksI hadn't thought about a CI that times out only when you stop printing#2016-11-1412:42gfredericksI'm trying to figure out if prop/for-all
could be changed to support clojure.test
assertions without breaking anything#2016-11-1412:42gfrederickseither all the time, or only when using defspec
(not sure which of those is better to aim for)#2016-11-1412:56nbergerThat would be awesome. At first glance I'd say that having it in defspec
and providing an easy way to use it when not in defspec
should be enough#2016-11-1413:04gfredericksI was hoping to make it just start working, without adding any new functions/macros#2016-11-1413:04gfredericksi.e., if you put assertions in the body of your prop/for-all
, it bases pass/failure on that instead of the expression return value#2016-11-1413:05gfredericksif new functions/macros are okay, then just adding a clojure.test.check.clojure-test/for-all
would work#2016-11-1413:07gfredericksactually I just thought about it a bit, and the edge cases involved in having both behaviors in the same macro are kind of nasty#2016-11-1413:08gfredericksedge cases meaning how things play out if you accidentally do or don't include assertions#2016-11-1413:08gfredericksI may have just talked myself out of it#2016-11-1512:28gfredericks@nberger I'm curious if there's a cleaner way to implement statistics using this: https://github.com/clojure/test.check/commit/4450152532eeaace449ca9241673428a19448eba#2016-11-1512:33nberger@gfredericks I think so. I'll play with it in the next few days. I already started rebasing the stats branch to the current master, playing with this will be my next task#2016-11-1512:42nbergerbut not "as is": we need a way to carry the labels over the quick check loop, but result is not passed over on passing trials (neither to the recur or to the reporter-fn :trial
). I'm looking at https://github.com/clojure/test.check/blob/4450152532eeaace449ca9241673428a19448eba/src/main/clojure/clojure/test/check.cljc#L101-L105#2016-11-1512:46nbergerWe especially need to pass it in some way in that recur call. But passing result
doesn't make much sense to me#2016-11-1512:47gfredericksright#2016-11-1512:48gfredericksit might not make any sense#2016-11-1512:48gfredericksjust struck me as a somewhat overlapping bit of functionality#2016-11-1512:48nbergeryeah#2016-11-1512:59nberger@gfredericks have you thought of turning the quickcheck loop into a reduce, where some "bigger" state is carried over, including the rng-state, the labels, test count... well all the state that is passed over in the recur but in a way that it would be easier to add stuff like the stats labels?#2016-11-1513:05nbergerI've been thinking around that idea for some time, but never got to play with it. At first sight seems like it should work.
I think haskell's quickcheck works that way, I'm looking at https://github.com/nick8325/quickcheck/blob/master/Test/QuickCheck/Property.hs#L370. Even though my haskell fu is very limited, and that the function is called mapTotalResult
, I think it's a reduce 🙂#2016-11-1513:07nbergerFor anyone wondering what statistics are we talking about, it's on http://dev.clojure.org/jira/browse/TCHECK-87#2016-11-1513:08gfredericksI was monkeying with the quickcheck loop when I was trying to do parallel tests#2016-11-1513:08gfrederickswhich changes the linearity of the reduce#2016-11-1513:08gfredericksbut I think it is a reduce, yeah#2016-11-1713:33nberger@gfredericks I just uploaded an updated patch for TCHECK-87 which is basically the old patch rebased to the current master + some polishing. Definitely not a final patch but in a better shape to experiment with it. And here's a nice diff: https://github.com/clojure/test.check/compare/master...nberger:stats-generator 🙂#2016-11-1713:43gfredericks@nberger you're not coming to the conj, are you? #2016-11-1714:26nberger@gfredericks I don't think so...#2016-11-1716:22gfredericks:/#2016-11-2112:58gfredericksjust added the result-data feature to master: https://github.com/clojure/test.check/commit/95112167af9636a48d3174ddd03409e8ac752179#2016-11-2113:01gfredericks@alexmiller ⇑ that should allow clojure.spec to not need to wrap failure data in an exception#2016-11-2113:07gfredericksactually taking advantage of that would mean that clojure.spec requires the next release of test.check; so I'm curious A) if that's reasonable, and B) if so, whether it's time for me to start panicking about finishing a test.check release before clojure-1.9 is ready#2016-11-2114:30Alex Miller (Clojure team)to be discussed#2016-11-2114:31Alex Miller (Clojure team)I’m working hard on the build server right now, hoping to get it up to the point where I can actually build test.check normally again#2016-11-2114:31Alex Miller (Clojure team)it’s not like 1.9 is imminent or anything#2016-11-2114:41gfredericksOk, I won't want to build anything for at least a week; I'm thinking of doing an alpha first, so people can use new stuff with the clojure 1.9 alphas#2016-11-2114:46Alex Miller (Clojure team)are you going to be at the conj?#2016-11-2115:32gfredericks@alexmiller yep#2016-11-2115:42Alex Miller (Clojure team)cool, we should chat at some point#2016-11-2116:15gfredericksI'll make sure that happens#2016-11-2119:25dangitAre people using test.check and midje together? Putting a tc/defspec
instead a midje/fact
doesn’t seem to do much.#2016-11-2119:32gfredericks@dangit could you share some code? I can't tell what you mean exactly#2016-11-2119:33gfredericksI dont' know how midje integrates with clojure.test, but presumably you could intermingle those integrations#2016-11-2119:33gfredericksdefspec looks just like a regular clojure.test/deftest
from clojure.test's perspective#2016-11-2119:41dangitYeah, I’m a bit of a newb, shoving a deftest inside a midje fact was just a guess. I tried this:
(facts "about numbers"
(fact "a number is anything"
4 => anything)
(defspec inc-test
100
(prop/for-all [v gen/int]
(= (inc v) (+ 2 v)))))
hoping midje would figure out what was going on but the result is:
>>> Output from clojure.test tests:
FAIL in (inc-test) (clojure_test.cljc:21)
expected: result
actual: false
1 failures, 0 errors.
>>> Midje summary:
All checks (1) succeeded.
Subprocess failed
So the failure triggers, but midje thinks everything is fine.#2016-11-2119:52gfredericksdefspec will only work at the top level; I don't know whether your test-runner is just clojure.test or if it's midje-specific, and that would determine whether your top-level defspec actually gets runs#2016-11-2119:52gfredericksbut if you like the nesting there's probably some way to run a generative test inside the facts
macro#2016-11-2119:52gfredericksmaybe by calling clojure.test.check/quick-check
directly#2016-11-2119:53gfredericksI don't know midje but I assume in the worst case you could do something like (fact "this property holds" (boolean (:result (quick-check 100 (prop/for-all ...)))) => true)
#2016-11-2119:53gfredericksso the only question is what's the best way to make that less tedious#2016-11-2119:54gfrederickswhich I can't guess at, not knowing midje#2016-11-2119:54dangitI’m just running lein midje/using midje’s autotest. The test is definitely getting run, it just seems like a hack what I’m doing now. Calling quick-check directly is a sane idea. 🙂#2016-11-2119:54dangitYeah, ok, thanks! I’ll play around with it a bit more.#2016-11-2119:55gfredericksnp#2016-11-2914:04nbergerI just created a ticket with a patch proposing a refactor to the quick-check loop that includes a clean way to provide feedback during the test run and also to augment/modify the state carried over during the process, making it easy to implement things like early abort, calculating statistics, adding timestamps at different steps of the process, etc. Feel free to give your feedback in http://dev.clojure.org/jira/browse/TCHECK-126#2016-11-2914:50gfredericksThat would be an interesting extension point for users#2016-11-2914:52gfrederickstransducers might have something to do with this#2016-11-2916:45nbergerI'd like to hear more about your transducers idea :)#2016-11-2917:42gfredericksIt might be thwarted by the irregularity of the shrinking phase#2016-11-2917:43gfredericksBut transducers are maybe related in that they separate the sequence processing logic from the orchestration, so that you can do things like swap in async orchestration#2016-11-2917:44gfredericksIt would be cool if parallelism where swappinable too, but I don't think transducers are amenable to that#2016-11-2918:02Alex Miller (Clojure team)not currently#2016-11-2918:21gfredericksI spent a lot of braintime trying to figure out how to do something transducerlike with the test.check shrink-tree but I think it seemed to hard to do while allowing the shrinking algorithm to do whatever it wanted#2016-11-2918:22gfredericksI forget what problem I was trying to solve exactly; probably just vague performance#2016-12-0815:59nwjsmithIs there a way to observe how a generator shrinks?#2016-12-0815:59nwjsmithLike sample
, but starting from a “big” generated value and then shrinking down?#2016-12-0818:45gfrederickstheoretically yes, there's just not an easy function included that does this#2016-12-0818:46gfredericksI've written it once or twice, it's only a few lines; I probably don't have one handy though#2016-12-0818:46gfredericksyour request can be interpreted a couple of different ways though#2016-12-0818:51gfredericksdefault shrinks are going to be less interesting; e.g., if your generator is (gen/vector gen-some-big-hairy-data-structure)
, then the easiest way of shrinking it consists of taking one step to []
#2016-12-0819:11nwjsmithMaybe a version of sample that returns small
, medium
, big
, medium
, small
, so that it’s easy to see how a value grows then shrinks?#2016-12-0819:11gfredericksthat might be weirder than it sounds#2016-12-0819:11nwjsmithI should probably get a much better understanding of test.check’s shrinking before bugging you furthur#2016-12-0819:12gfredericksit's tempting to think that the growing you see in sample
and in a test run is related to the shrinking you get in a failure, but they're almost entirely unrelated processes#2016-12-0819:12gfredericksI don't mind discussing it, the whole thing is currently underdocumented so I deserve to have to explain things live :)#2016-12-0819:13gfredericksanother thing you could implement without too much trouble is a random shrink#2016-12-0819:13gfredericksit might not resemble too closely what would happen in a real test, but it could at least give you an idea of what sorts of steps are possible#2016-12-0819:16gfredericks@nwjsmith this part of this talk introduces the idea of a shrink tree, which is most of what you need to know to understand the shrinking mechanics: https://youtu.be/u0t-6lUvXHo?t=11m6s#2016-12-0819:17nwjsmithOh, excellent. Thanks!#2016-12-0819:27gfredericksI suppose it's a rather steep introduction, but it's at least better than me typing stuff #2016-12-1515:19gfredericksI think I figured out how to use transducers in the shrink trees#2016-12-1515:22gfredericksIt would save a lot of intermediate lazy seq allocations, but I think it would only apply when shrinking, so it's not obviously worth the effort. #2016-12-1515:45nbergerNice!#2016-12-1812:40nbergerUploaded an alternative patch to http://dev.clojure.org/jira/browse/TCHECK-126 (Refactor c.t.c/quick-check as a state machine to provide more extension points). The new patch refactors c.t.c/quick-check "in-place", instead of introducing a separate namespace like in the old patch.
Most important change is that it replaces the reporter-fn
with a step-fn
which allows feeding back the quick-check loop to drive/augment it by returning a modified quick-check state (for example, adding a timestamp, or dynamically changing the number of tests left). reporter-fn
was for side-effects only, step-fn
allows to do the side-effects AND modify the qc state.
I understand this is not a breaking change because the reporter-fn
has not been part of a release yet AFAIK#2016-12-1812:48gfredericksso step-fn
is part of the public API then?#2016-12-1812:53nbergeryes, through the new step-fn
option that replaces reporter-fn
. There's also a c.t.c.clojure-test/default-step-fn
now#2016-12-1812:53nbergerhere's the new default-step-fn
: https://github.com/clojure/test.check/compare/master...nberger:test.check2-reporter-fn#diff-fb4b33c15ed48f22f9f6c3655f16da02R29#2016-12-1812:54gfrederickshm#2016-12-1812:54gfredericksI'd want to be sure that creating a well-behaved and useful step-fn
isn't more difficult than a reporter-fn
#2016-12-1812:57nbergeryeah, that's a risk... we could check for nil, but not sure if much more than that... maybe check that we get a map with certain keys back? But I don't know, passing a step-fn
is a bit of "advanced usage", and reading a little bit about it or seeing an example is not such a big expectation, maybe... Here's an example of a step-fn: https://github.com/clojure/test.check/compare/master...nberger:test.check2-reporter-fn#diff-5eb045ad9cf20dd057f8344a877abd89R985#2016-12-1812:57nbergerand here's a more useful one: https://github.com/clojure/test.check/compare/master...nberger:test.check2-reporter-fn#diff-5eb045ad9cf20dd057f8344a877abd89R1042#2016-12-1812:58nbergerperhaps we could provide some specs for the step-fn...#2016-12-1813:02nbergeranother option is to keep separate functions for the side-effecty part (`reporter-fn`) and the modify-qc-state part (`step-fn`? will need help with the names 😄 )#2016-12-1813:04nbergerbut I don't know... maybe that's more appropriate for a layer over this (a step-fn
that composes a reporter-fn and a change-qc-state-fn or whatever)#2016-12-1813:16gfrederickswhat uses do you imagine a user having for a step-fn
that aren't possible with reporter-fn
?#2017-06-1615:11gfredericksEspecially if the source is online too #2017-06-1615:11carocadit is. Currently a pr but it should be enough#2017-06-1615:15carocadhmmm seems that I was wrong. The build is still failing but it doesnt show the same problem as before#2017-06-1615:15carocadhttps://travis-ci.org/n7a235/backend.routing/builds/243706150?utm_source=github_status&utm_medium=notification#2017-06-1615:15carocadthe pr is this if it helps: https://github.com/n7a235/backend.routing/pull/3#2017-06-1615:18gfredericks@carocad that looks like just an eastwood failure?#2017-06-1615:19carocadyeah it seems like it. I just changed my code and the example from the repo worked locally. I am testing in travis#2017-06-1615:24carocadyeah it worked#2017-06-1615:24carocadso the problem was the previous bindings 😄 sorry#2017-06-1615:25carocadI'm closing the issue#2017-06-1615:26gfrederickscool, thanks#2017-06-1615:28carocadoh it seems I cannot close the issue 😞
I put a comment explaining it was my failure though#2017-06-1615:29gfrederickshuh, strange#2017-06-1615:29gfredericksI'll close it#2017-06-1615:31carocadthanks :thumbsup:#2017-06-1913:21gfredericks@alexmiller I have a branch with proposed changes to collection sizing: https://github.com/clojure/test.check/tree/TCHECK-106#2017-06-1913:22gfredericksI'm not sure if you are interested in the subtleties or not, but let me know if you have any thoughts or any other way of testing whether it improves the situation with spec's usage#2017-06-1913:22gfredericksI think the biggest "risk" is in severely reducing the coverage of existing tests that make assumptions about the old distributions#2017-06-1914:52gfrederickscould easily just release another alpha and see how it goes#2017-06-1915:37Alex Miller (Clojure team)I would love to take a look, but won’t be today#2017-06-1915:40gfredericks👍 no rush#2017-06-2016:49mfikesThere is an evidently breaking change in the 0.10.0-alpha1
release, somehow related to the introduction of the result protocol (https://github.com/clojure/test.check/commit/95112167af9636a48d3174ddd03409e8ac752179). Was it intentional that this be a breaking change (does it require clients to update?)
Here is a related ticket in the ClojureScript compiler regarding it: https://dev.clojure.org/jira/browse/CLJS-2110#2017-06-2016:52gfredericks@mfikes the issue is spec returning an exception object?#2017-06-2016:53mfikesTo be honest, I haven’t sorted out why things are regressing.#2017-06-2016:53gfredericksSo spec used a hacky workaround that the result protocol was meant to obviate#2017-06-2016:53mfikesI only know empirically right now based on bisecting. I’ve read the code to at least attempt to grok why, but haven’t gained comprehension yet.#2017-06-2016:54gfredericksSo ideally spec should change to the better version, but I don't think I meant it to break. Though I might have#2017-06-2016:55mfikes(I also don’t know if this affects Clojure—my experience with it is with JVM and self-hosted ClojureScript, where it breaks in the same way for both.)#2017-06-2016:55gfredericksI don't have a sense for what parts are changing at what speed 😕 #2017-06-2016:55gfredericksAnd whether it would be bad for spec to require tc alpha#2017-06-2016:55mfikesCool. If there was no clear intent to be a breaking change, I’ll dig further to see if can suss out what is going on.#2017-06-2016:57gfredericksI expect making it nonbreaking would consist of having t.c continue special-casing return values that are Exceptions#2017-06-2016:59mfikesThat helps… I’ll dig deeper—t.c. 0.10.0 is still alpha, after all.#2017-06-2017:05Alex Miller (Clojure team)I would say it’s possible that spec.alpha could depend on an alpha#2017-06-2017:05Alex Miller (Clojure team)but that regardless of that, t.c shouldn’t introduce a breaking change#2017-06-2017:06Alex Miller (Clojure team)and if the semantics need to change, it should have a new name#2017-06-2017:36mfikesFWIW, t.c 0.10.0-alpha1
has a breaking change for Clojure 1.9.0-alpha17
for this case as well:
$ lein repl
nREPL server started on port 55971 on host 127.0.0.1 -
REPL-y 0.3.7, nREPL 0.2.12
Clojure 1.9.0-alpha17
Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)
Results: Stored in vars *1, *2, *3, an exception in *e
user=> (require '[clojure.spec.alpha :as s]
#_=> '[clojure.spec.test.alpha :as st])
nil
user=> (defn ranged-rand [start end] 0)
#'user/ranged-rand
user=> (s/fdef ranged-rand
#_=> :args (s/and (s/cat :start int? :end int?)
#_=> #(< (:start %) (:end %)))
#_=> :ret int?
#_=> :fn (s/and #(>= (:ret %) (-> % :args :start))
#_=> #(< (:ret %) (-> % :args :end))))
user/ranged-rand
user=> (st/check `ranged-rand)
({:spec #object[clojure.spec.alpha$fspec_impl$reify__1215 0x2af92bb9 "
#2017-06-2017:38Alex Miller (Clojure team)what’s the change?#2017-06-2017:39Alex Miller (Clojure team)at a glance I don’t see anything amiss there#2017-06-2017:40mfikesRight, it might not be a breaking change, but an internal bug. Here is what it used to do…#2017-06-2017:41mfikesHere is a gist… it is too big to really paste: https://gist.github.com/mfikes/1caa83751019495cfdda83e79ad7e3ed#2017-06-2017:43mfikesInternal bug because: It doesn’t return a falsey :result
Breaking change because the :result
used to be much richer than just a Boolean value, and clients might not know to look for :result-data
#2017-06-2017:44Alex Miller (Clojure team)it just looks to me like it found a failing example in the old one but didn’t in the new one#2017-06-2017:44Alex Miller (Clojure team)which might just be due to what the random seed turned up, not a real difference#2017-06-2017:44mfikesAhh. Right.#2017-06-2017:45mfikesI’ll increase the number of tests to see what occurs.#2017-06-2017:46Alex Miller (Clojure team)I guess maybe it’s surprising that t.c found 1000 successful cases with a hard-coded result in the new one#2017-06-2017:47mfikesTrue. Seems unlikely to have pulled that off 🙂#2017-06-2017:48Alex Miller (Clojure team)so maybe just looking at (gen/sample (s/gen (:args (s/get-spec user/ranged-rand))) 1000) might be interesting as a before / after#2017-06-2017:50mfikesuser=> (gen/sample (s/gen (:args (s/get-spec 'user/ranged-rand))) 10)
((0 1) (-1 0) (-1 2) (0 1) (-23 2) (1 3) (-16 -6) (-92 -10) (-1 2) (2 7))
#2017-06-2017:50mfikes(2 7)
should fail, right?#2017-06-2017:50Alex Miller (Clojure team)yeah#2017-06-2017:51Alex Miller (Clojure team)as should (-92 -10)#2017-06-2017:51mfikesYep… many don’t include 0#2017-06-2017:51Alex Miller (Clojure team)or (1 3)#2017-06-2017:51Alex Miller (Clojure team)or (-1 0) - which is the failing example in the old one#2017-06-2017:52Alex Miller (Clojure team)so the generator seems fine#2017-06-2017:52Alex Miller (Clojure team)and next we should suspect the prop checking#2017-06-2017:59mfikesI revised my ranged-rand
to prn
its args, and also revised the :fn
part of the fdef
to print its argument, and at least those parts look correct
user=> (st/check `ranged-rand {:clojure.spec.test.check/opts {:num-tests 2}})
-1 1
{:args {:start -1, :end 1}, :ret 0}
-4 -2
{:args {:start -4, :end -2}, :ret 0}
({:spec #object[clojure.spec.alpha$fspec_impl$reify__1215 0x77b18e9a "
#2017-06-2018:08gfrederickshi I'm back#2017-06-2018:08mfikesTL;DR of the above is just reproing it in Clojure#2017-06-2018:10mfikesI suppose there is the twist that there might be an internal bug in t.c (disregarding any potential interface changes)#2017-06-2018:11gfredericks@mfikes regarding the first issue, it definitely looks like I didn't retain the old behavior; an extra clause here should fix it: https://github.com/clojure/test.check/blob/23e1fcc1bb7f3f2d0202ed99eee202a0b1f2c652/src/main/clojure/clojure/test/check/results.cljc#L18#2017-06-2018:11gfredericksI'm assuming cljs has something like Throwable
for extending error types, but maybe that's not true?#2017-06-2018:11mfikesIt might be possible to extend js/Error or somesuch#2017-06-2018:19mfikes@gfredericks Evidently you can. Not sure what the implications might be.
cljs.user=> (defprotocol Result
#_=> (passing? [result])
#_=> (result-data [result] "A map of data about the trial."))
nil
cljs.user=> (extend-protocol Result
#_=> js/Error
#_=> (passing? [this] false)
#_=> (result-data [this] (ex-data this)))
#object[Function "function (this$){
var this$__$1 = this;
return cljs.core.ex_data.call(null,this$__$1);
}"]
cljs.user=> (result-data (ex-info "hi" {:a 1}))
{:a 1}
#2017-06-2018:20mfikes^ works in JVM and self-hosted ClojureScript#2017-06-2018:20gfredericks@mfikes does it work for (throw "some string")
?#2017-06-2018:22mfikes@gfredericks If you extract using ex-message
:
cljs.user=> (extend-protocol Result
#_=> js/Error
#_=> (passing? [this] false)
#_=> (result-data [this] (ex-message this)))
#object[Function "function (this$){
var this$__$1 = this;
return cljs.core.ex_message.call(null,this$__$1);
}"]
cljs.user=> (result-data (js/Error. "some string"))
"some string"
#2017-06-2018:24mfikesI’ll see what happens if I build a t.c that further extends Result
to exceptions#2017-06-2018:30gfredericksI was just concerned that if you throw a string, the thrown thing is not a js/Error
and so wouldn't match that clause#2017-06-2018:30mfikesAhh#2017-06-2018:43mfikesAt least for the exception thrown for the case above, further extending does the trick#2017-06-2018:44mfikescljs.user=> (st/check `ranged-rand)
[{:spec #object[cljs.spec.alpha.t_cljs$spec$alpha5600],
:clojure.test.check/ret {:result false,
:result-data {:a 1},
:seed 1497984200982,
:failing-size 0,
:num-tests 1,
:fail [(-1 0)],
:shrunk {:total-nodes-visited 0,
:depth 0,
:result false,
:result-data {:a 1},
:smallest [(-1 0)]}},
:sym cljs.user/ranged-rand,
:failure false}]
With this kind of extension:
(passing? [this] false)
- (result-data [this] {}))
+ (result-data [this] {})
+
+ #?(:clj Throwable :cljs js/Error)
+ (passing? [this] false)
+ (result-data [this] {:a 1}))
#2017-06-2018:48mfikesPerhaps :failure false
is not what you’d want. Hrm.#2017-06-2020:15mfikesCaptured the above with https://dev.clojure.org/jira/browse/TCHECK-131#2017-06-2021:34gfredericks@mfikes great, thanks#2017-06-2021:35gfredericksI'll make sure to get another alpha out shortly#2017-06-2021:43mfikesThanks @gfredericks !#2017-06-2619:44rabbitthirtyeightI'm having trouble doing some seemingly basic stuff with test.check. Can someone point out where I'm going wrong?
Basically bind
and large-integer*
don't seem to be working as I expected.
If I have the following in my property:
(let [greater (gen/large-integer* {:min 0 :max 10})
lesser (gen/bind greater (fn [x] (gen/large-integer* {:min 0 :max x})))]
(prop/for-all [less lesser
great greater]
(is (<= less great)))
when I run the test I always get the following failure:
expected: (<= less great)
actual: (not (<= 1 0))
#2017-06-2620:28gfredericks@rabbitthirtyeight I know what's wrong, but don't know enough about what you're really doing to recommend an alternative#2017-06-2620:29gfredericksthe reason it isn't doing what you think is that there's a different greater
being generated for each of less
and great
#2017-06-2704:33rabbitthirtyeight@gfredericks ah. I see what you're saying. Thanks! Given that, I think I need to re-think what I was trying to do. If I'm unable to figure it out I'll post back in here.#2017-06-2712:39gfredericksI'm thinking of creating prop/for-all2
, or prop/for-all-2
, whichever name is better#2017-06-2712:40gfredericksdifferences:
- doesn't have the TCHECK-131 interpretation of returned exceptions
- uses sequential bindings like gen/let
, and has the map-binding syntax of gen/let
as well for parallel bindings
- any other improvements anybody would want?#2017-06-2713:43gfrederickstest.check version 0.10.0-alpha2
was just released; changelog here: https://github.com/clojure/test.check/blob/master/CHANGELOG.markdown#0100-alpha2-2017-06-27#2017-07-0421:15mattlyso just as a testament to the power of property-based testing:#2017-07-0421:16mattlythe harness I have for testing calculations & queries against the data warehouse I manage an application front-end for just caught a super-rare edge case#2017-07-0421:16mattlydoesn't actually happen against production data#2017-07-0421:16mattlybut it could#2017-07-0421:17mattlymy test suite started running monday morning, and test.check generated this edge case, and cranked on it for over 30 hours, approx 200k iterations, until it isolated the problem#2017-07-0421:17mattlyand I look at it and go, oh, of course!#2017-07-0421:17mattlyand it was like one of these little 2 character fixes#2017-07-0421:18mattlybut had the problem occurred against production data, nobody ever would have known#2017-07-0421:18mattlyso, thanks#2017-07-0421:25gfredericks> doesn't actually happen against production data
> but had the problem occurred against production data, nobody ever would have known
So how do you know it didn't?#2017-07-0503:56mattlyI checked the source condition against our production set, it didn't exist#2017-07-0503:56mattlybut it could have#2017-07-1421:07peejaI don't suppose there's a core.async "mode" I can run quick-check
in in ClojureScript, is there?#2017-07-1421:07peejaHaving the entire page lock up while test.check runs itself 100 times is crimping my style a bit#2017-07-1421:08peejaI'd love to get it to report its progress as it goes and let the page interrupt it#2017-07-1421:08gfredericksAsync tests are a high priority; I have a hacky branch you could try if you like#2017-07-1421:08peejaOoh, I'd love to take a look, at least#2017-07-1421:09gfredericksIt's called async-tests-POC on the main github repo#2017-07-1421:10gfredericksWould be interested to know if it solves your problems#2017-07-1421:10peejaI have a niggling feeling that there's a transducer in here somewhere#2017-07-1421:10peejaThat might abstract the testing itself from the synchrony choice#2017-07-1421:12gfredericksYou should talk to @nberger about that, he's been working on refactoring the core algorithm#2017-07-1421:13gfredericksThe shrink algorithm might resist transducing#2017-07-1421:13peejaBTW, what's future
in CLJS?#2017-07-1421:13peejaThat's not built in, is it?#2017-07-1421:14gfredericksIt is. I think it just does settimeout#2017-07-1421:14gfredericks....I hope#2017-07-1421:17peejaI don't see it anywhere…#2017-07-1421:22gfredericksAre macros in a separate file?#2017-07-1421:38peejaThey are, but I don't see it there, and it's not documented#2017-07-1421:39peejaAh, looks like it's not out yet?#2017-07-1421:40peejahttps://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj#L6865-L6872#2017-07-1421:40peejaNo, that's clojure…#2017-07-1421:40peejaNot sure how I got crossed over there#2017-07-1421:41gfredericksDid you try running it?#2017-07-1421:41peejahttps://github.com/clojure/clojurescript/blob/master/src/main/clojure/cljs/core.cljc#2017-07-1421:41peejaNot yet#2017-07-1421:42gfredericksBecause I'm not sure I did, in cljs 😶 #2017-07-1421:42peejaHeh#2017-07-1421:44gfredericksSo I take back everything i said about future#2017-07-1421:45peejaI mean, it'd be pretty sweet#2017-07-1421:51gfredericksEasy to write#2017-07-1423:01gfredericksSorry about the code being even less in shape than I thought#2017-07-2521:29moxajquestion: is there any way to control how a recursive generator shrinks? for example, the following can return [[[[false]]]]
as the minimal failing example:
(tc/quick-check 100
(tc.prop/for-all [value (tc.gen/recursive-gen tc.gen/vector tc.gen/boolean)]
(every? true? (flatten value))))
#2017-07-2521:31moxajhowever, if a recursive generator could shrink towards the children of the actual compound generator, it could produce [false]
as the minimal failing value#2017-07-2521:31gfredericksthere's a ticket for that#2017-07-2521:31gfredericksI'm not sure if it's an easy fix or not#2017-07-2521:32gfrederickshttps://dev.clojure.org/jira/browse/TCHECK-110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel#2017-07-2521:32gfredericksaccording to the comments, my past self wasn't sure either#2017-07-2521:32gfredericksmy guess is that identifying subtrees in a generic way might be difficult or impossible#2017-07-2521:33gfredericksbut I'd be happy to be wrong about that#2017-07-2521:38moxajI see#2017-07-2522:01peejaWhen is [:shrink :smallest]
not a vector of one element?#2017-07-2522:01peejaOr: why is it a vector?#2017-07-2522:07gfredericks@peeja when the prop/for-all
has multiple clauses, the vector has more elements#2017-07-2522:08peejaAh, that makes sense. Thanks!#2017-07-2614:28nberger@peeja: I'd love to help finding that transducer. I've been working on a refactoring that might be of help. The main idea is to make it easier to see the quick-check loop as a state machine. I think we could be closer to extract a transducer from there. The changes are in the test.check.refactor
branch in my fork, let me know if you have any questions or comments#2017-07-2614:44nberger-- that's related to your message from a few days ago 🙂#2017-08-0220:49cddrHow might you go about writing a generator of sequences of numbers which add up to n
?#2017-08-0220:51gfredericks@cddr depends what kind of distribution you want#2017-08-0220:51gfredericksthe easy dumb way to do it is to generate a sequence of numbers and then append (- n sum)
to them#2017-08-0220:52gfredericksalso depends on what kind of numbers you mean#2017-08-0220:54cddrThis is related to my question over in #clojure-spec about a having relations between a top-level amount, and children that should add up to that amount
https://clojurians.slack.com/archives/C1B1BB2Q3/p1501686285791508#2017-08-0220:54gfrederickswould it be weird to not generate the top-level amount?#2017-08-0220:55gfredericksgenerate just the children, then set the top-level to the sum?#2017-08-0220:58cddrYeah that was my first thought too. But then I thought I needed to solve this type of problem anyway but looking back at my code I can't remember why now 🙂#2017-08-0220:59cddrIn the actual example there are a few more relationships between the top-level and the children#2017-08-0221:08gfredericksa very useful tactic as generating something related to what you want, and using gen/fmap
to transform it#2017-08-0221:08gfredericksi.e., generate the information, then use fmap
to get it into the right format#2017-08-0811:44gfredericksI've occasionally had the idea of mitigating the "some things don't shrink very well" problem by allowing users to provide a fallback shrink strategy, where you just supply a function from x
to a collection of smaller x
s.
Interestingly it looks like hedgehog has exactly that feature: https://hackage.haskell.org/package/hedgehog-0.5/docs/Hedgehog-Gen.html (see the shrink
function)#2017-08-0816:28lucasbradstreetCool#2017-08-1415:21frankitoxHi Frederick! Loved the lib, there's a couple of functions that I've invented too for my project lol, but I think you took a much more clean approach.#2017-08-1415:22frankitoxcom.gfredericks.test.chuck.generators/string-from-regex
can only be used in the JVM scope, right?#2017-08-1415:23gfredericks@franquito yes -- you'll see a pull request for a clojurescript port; I can't remember if it's usable or not#2017-08-1415:23gfredericksif you end up trying it out, do leave feedback on that PR about how it goes#2017-08-1415:24gfredericksI've been meaning to review that PR for months now 😭#2017-08-1415:25frankitoxOh, nice!#2017-08-1415:26frankitoxWell, then I'll give it a try and see what happens, I just need a simple regexp#2017-08-1415:27frankitoxHaha I see! Yeah, It must be a huge pain to maintain some open source project#2017-08-1415:27frankitoxApart from It's size#2017-08-1415:27frankitoxIts*#2017-08-1418:47lucasbradstreet@gfredericks did the fallback shrinking strategy end up going anywhere? I’ve actually used gen/return to prevent shrinking, and then shrunk manually myself in the past, so this is pretty interesting 🙂#2017-08-1418:48gfredericks@lucasbradstreet you prevented shrinking because it would spend a long time without getting very far?#2017-08-1418:50lucasbradstreetYeah, because testing my invariants was on the order of seconds, so I tried some more aggressive heuristics to reduce the search space.#2017-08-1418:50lucasbradstreetMaybe not a very typical example that you want to allow for though 😛#2017-08-1418:53gfredericksI've been in that situation before, I don't want to disregard it#2017-08-1418:54gfredericksmy initial thought is that a fallback shrink would never speed up shrinking since it would only try it after it's tried everything else#2017-08-1418:54gfredericksbut I suppose you could disable the default shrinking first, which you're already doing#2017-08-1418:54lucasbradstreetAhhh. I missed that aspect of the word “fallback”#2017-08-1418:55gfredericks@lucasbradstreet also FYI with a mild amount of assuming-implementation-details, you can do this already#2017-08-1418:56gfredericksso if you want to try it out that would be informative for me#2017-08-1418:58lucasbradstreetCan you point me to where I should look?#2017-08-1418:58gfredericks@lucasbradstreet are you familiar with the idea of a shrink tree?#2017-08-1418:58lucasbradstreetAt a high level yes.#2017-08-1419:00gfredericksthe simpler thing is writing replace-shrink
, that takes [g f]
where f
is a function from x
to [x]
#2017-08-1419:00lucasbradstreetCan’t say I’ve looked into exactly how rose trees work though.#2017-08-1419:00gfredericksit's just a tree of values; the root is the generated value, and the child of any node are the shrinks from that point#2017-08-1419:01gfredericksand it's lazy because it's huge#2017-08-1419:01lucasbradstreetOK#2017-08-1419:01gfredericksso writing replace-shrink
just involves figuring out the exact functions to call, and the algorithm that transforms x
and f
into a lazy tree#2017-08-1419:02lucasbradstreetOK, with you up to here.#2017-08-1419:02gfredericksI'll spit something out real quick that will probably work#2017-08-1419:04lucasbradstreetCool. I can set aside some time later today to give it some testing #2017-08-1419:04gfredericks⇑ just off the top of my head, I didn't run it or compile it or parse it or anything#2017-08-1419:05lucasbradstreetSure thing :)#2017-08-1419:05gfredericksbut that should at least point you to the right functions, which would be the hard part otherwise#2017-08-1419:05gfredericksfeel free to ping me if you try it and can't get it to work#2017-08-1419:06lucasbradstreetWill do. Thanks!#2017-08-1922:59nwjsmithIs there a way to prevent test.check from printing to standard out on a clojure.test test run#2017-08-1922:59nwjsmith?#2017-08-1923:02gfredericksThere's a ticket for that, I can't remember what state its in. But until that's through, only monkey-patching#2017-08-1923:03nwjsmithah, ok. Thanks Gary.#2017-08-1923:06gfredericksYou'll see an obvious prn call at the top of the clojure-test ns#2017-08-3021:22gfredericksI just realized you could parallelize the shrinking process using something like a branch predictor#2017-08-3021:25gfredericksbut it could be more complex than a traditional branch predictor since you can execute multiple possibilities in parallel#2017-08-3021:31gfredericksthe search strategies used in chess programs seem a bit related too, though maybe not helpfully#2017-08-3113:12nbergerThat sounds pretty cool, parallel shrinking would be a neat feature#2017-09-1310:54thedavidmeisterhey, is there something generated by clojure.test.check.generators/any
that can break the output of the test runner?#2017-09-1310:54thedavidmeisteri got this fail#2017-09-1310:54thedavidmeisterFAIL in (??round-trip) (:)
expected: (= e (parse (string d)))
actual: (not (= [["_Iq1WSpRhKoq+0M?1?!CV-I+GD-ul?s**R?!*!6arO*hmmz!" "6KnðåómÔ(OD{hl$g\tÉ\"²Å¹Þˆß’" "6
#2017-09-1310:54thedavidmeisterit's like the string somehow didn't finish or something#2017-09-1310:55gfredericksthe string not finishing smells more like your output was truncated somehow#2017-09-1310:56gfredericksthe worst thing about gen/any
that could cause problems is that it can generate NaN
s deep in the nested structure#2017-09-1310:56gfrederickswhich causes the whole thing to not equal itself, but maybe only under certain conditions#2017-09-1310:57thedavidmeistermmm#2017-09-1310:58thedavidmeisterFAIL in (??round-trip) (:)expected: (= e (parse (string d))) actual: (not (= [["_Iq1WSpRhKoq+0M?1?!CV-I+GD-ul?s**R?!*!6arO*hmmz!" "6KnðåómÔ(OD{hl$g\tÉ\"²Å¹Þˆß’" "6Testing wheel.stylesheet.hoplon
#2017-09-1310:58thedavidmeisterit just stops...#2017-09-1310:58thedavidmeisterand goes straight into the next test#2017-09-1310:58gfredericksyou could bypass the test runner & printing by calling clojure.test.check/quick-check
directly with your property and inspecting the return value#2017-09-1310:59thedavidmeisteryeah, also i think what you said about NaN
is going to derail me anyway#2017-09-1311:00thedavidmeisteri might need a different approach#2017-10-0601:54nwjsmithHow would one generate a byte array of a specific length?#2017-10-0601:59nwjsmithI’m a dingus, just found byte
.#2017-10-1416:28nwjsmithI have a function that takes as input a large set of uniformly distributed random numbers and a directed, acyclic graph. It returns a random topological ordering of the graph.
I’d like to hook this into test.check
such that if I run test.check/quick-check
twice with the same seed parameter then the same topological orderings will be generated.#2017-10-1416:29nwjsmithAs I understand it, test.check
’s generators and combinators are designed to keep me from having to see the underlying RNG#2017-10-1416:30nwjsmithand that’s worked really well until now, when I need a lot of uniformly distributed random numbers#2017-10-1416:32nwjsmithIs the test.check.generators/->Generator
function considered part of the public API?#2017-10-1416:33gfredericks@nwjsmith so if you had a generator for uniformly distributed random numbers, would that solve it?#2017-10-1416:36nwjsmith@gfredericks yes, that would work beautifully#2017-10-1416:37nwjsmithThen I could do a (gen/vector (gen/rand-int max) num-elements)
and fmap
my random-topological-ordering
over it#2017-10-1416:55nwjsmith(def gen-rand-double
(gen/no-shrink
(gen/->Generator
(fn [rnd _size]
(rose/pure (random/rand-double rnd))))))
?#2017-10-1416:58nwjsmithAh, there’s random/rand-long
.
(def gen-rand-long
(gen/no-shrink
(gen/->Generator
(fn [rnd _size]
(rose/pure (random/rand-long rnd))))))
(gen/generate (gen/vector gen-rand-long) 10 100) ;; => [6200754525179398429 -3542882520031353397]
(gen/generate (gen/vector gen-rand-long) 10 100) ;; => [6200754525179398429 -3542882520031353397]
#2017-10-1416:58nwjsmithThat seems to do the trick#2017-10-1417:01nwjsmithI’m not really sure what the effects of no-shrink
are here, or what rose/pure
is doing. I’ll have to watch you’re old talk again to brush up on the internals.#2017-10-1417:02nwjsmithYour talk from this week was fantastic by the way, really good guide to building custom generators.#2017-10-1417:04gfredericks(also, why do they have to be uniformly distributed?)#2017-10-1417:05nwjsmithhttp://alexander-karzanov.net/Publications/91_LEO11_N.pdf#2017-10-1417:06nwjsmiththe algorithm is on page 2. I’ve got to generate a large number of random swaps of a topological ordering.#2017-10-1417:06nwjsmith(the bottom of page 2)#2017-10-1417:07nwjsmithAt first I tried gen/choose
, but I ended up with really bad distributions on the generator.#2017-10-1417:09gfredericksChoose is uniform, unlike most other builtin generators#2017-10-1417:09nwjsmithHrm. Maybe I just thought I’d end up with really bad distributions#2017-10-1417:09nwjsmithI’ll try choose again.#2017-10-1417:09gfredericksno-shrink is a noop in your code I believe#2017-10-1417:11gfredericksYou need uniform ints or doubles?#2017-10-1417:11nwjsmithints#2017-10-1417:11gfredericksWhat range?#2017-10-1417:12nwjsmithbetween 0 and 2n-1, where n is the number of nodes in the graph.#2017-10-1417:13gfredericksOooh, yeah, choose is bad for large n#2017-10-1417:14gfredericksFoolproof is (gen/vector (gen/choose 0 1) n), then fmap the bits to a bigint#2017-10-1417:15gfredericksBut even if the algorithm demands uniform, I'm curious if testing with a nonuniform generator would actually be bad#2017-10-1417:19gfredericksBut either way, there's no builtin bigint generator yet, which makes it harder for you :(#2017-10-1417:23nwjsmithPutting it together with the approach you suggested should work though. I’m going to experiment a bit with all of the approaches above. Will report back.#2017-10-1417:23nwjsmithBTW do you have a strategy for testing generators? Specifically how they shrink, their distribution, performance etc.#2017-10-1417:26nwjsmithExperimenting at the REPL has been working for me, but now I have all of these useful graph-related generators and I’m hoping to open source them. Might have to come up with my own strategy.#2017-10-1417:41gfredericksNo, that's an interesting idea. You can see examples in the tests of testing distribution & shrinking a bit, but it's all ad-hoc#2017-10-1417:41gfredericksShrinking is particularly weird to test#2017-10-1417:42gfredericksMaybe...#2017-11-0216:49ghadiIs there any relationship between procedural generation techniques for computer graphics and test check generators? I'm thinking in relation to stuff like Perlin noise#2017-11-0223:26gfredericksI've generally been skeptical of using test.check generators for anything very far from testing (though I might be too deep into it to think more creatively)
The primary reason is that the distributions are generally skewed with edge-case heuristics#2017-11-0302:43ghadiOh, I meant the opposite: using techniques from procedural generation to inform the way generators explore the search space #2017-11-0302:44ghadiI know next to nothing about procgen, except that it is a large topic filled with interesting heuristics#2017-11-0302:45ghadiLike making trees look realistic. My coworker was describing Perlin noise today, and a lightbulb went off#2017-11-0614:23gfredericksHow do folks feel about test.check printing success summaries?
I'm processing a patch that makes this configurable, but didn't know whether it should stay on by default, since I don't know how many people like it.#2017-11-0617:56lucasbradstreetI think I’d like it.#2017-11-0618:23gfredericksTo be clear, it already does this.#2017-11-0618:30gfredericksAnd currently it's not configurable
I'm trying to decide between
A) make it configurable, defaulting to on
B) make it configurable, defaulting to off#2017-11-0618:34lucasbradstreetOh, you mean printing the current summaries? Ah, I misread it. I think I prefer defaulting to on, especially since it’s the current behaviour.#2017-11-0618:56mattly@gfredericks can you provide an example of what you mean? I’m not entirely clear by “success sumaries”#2017-11-0619:05gfredericksThis is a defspec
thing in particular
Right now when a defspec
test runs, and passes, it prints something like:
{:result true, :num-tests 500, :seed 1509994998441, :test-var "the-name-of-my-test"}
#2017-11-0619:05gfredericksyou can compare this to the behavior of a regular clojure.test
test, which prints nothing on success#2017-11-0621:29mattlyinteresting. Is defspec
new?#2017-11-0622:37gfredericksno, defspec
is the standard clojure.test
integration that's been around for quite a while#2017-11-0623:32mattlyah, I guess I’ve only really ever used test.check in the context of test.chuck’s checking
#2017-11-0623:33gfredericksTry to say "test.check in the context of test.chuck's checking" five times fast#2017-11-0623:34mattly😄#2017-11-0700:34aj taylorI'd say default to on, but it may be just because I'm used to it ha#2017-11-0700:38aj taylorAlso since each test is running multiple times it's nice to have an idea of what's currently being executed (even though I could just configure it I guess with that update)#2017-11-0700:41gfredericksIt's good to know that some people like it. I was mainly considering changing the default because I was assuming basically nobody liked it#2017-11-0700:41gfredericksIf it's roughly split I'll probably keep the default since that's less confusing#2017-11-1012:50borkdudeHow can I get a sequence of random numbers by providing a seed in clojure.test.check?#2017-11-1012:52gfrederickswhat sort of numbers?#2017-11-1012:53gfredericksif you want to just use the raw RNG code it's pretty easy
(map random/rand-long (random/split-n (random/make-random seed) 500))
#2017-11-1012:53borkdudethanks#2017-11-1012:54gfredericksif you are using this for PBT I would probably recommend using generators instead, or would be interested in why you can't#2017-11-1012:55borkdudeI would like to use this for some graphical algorithm which needs random numbers, but I want to make it deterministic#2017-11-1012:56gfredericksokay cool; yeah if it's not for testing then using the RNG directly is perfectly fine#2017-11-1016:08borkdudeI’m looking for examples online but can’t find it. Is there a way that I can express that I want ints in a range using the generators + a custom seed?#2017-11-1016:09borkdudeor do I have to convert longs to a range myself, which is also possible#2017-11-1016:13gfredericksthe rng only generates raw uniform longs; if your range isn't close to 64 bits, I'd just use mod
to get what you want#2017-11-1016:14borkdudeyeah, cool#2017-11-1016:18borkdudeIs there a way to get around the fixed size 500 in
(map (comp #(mod % 1000) r/rand-long) (r/split-n (r/make-random 2) 500))
so I can get a lazy sequence of these numbers or do I need to specify this up front always?#2017-11-1016:19borkdudeI guess I can calculate this up front#2017-11-1016:45gfredericksyou can make an infinite lazy seq#2017-11-1016:45gfredericksthere's a private(?) function in generators.cljc
that does this#2017-11-1016:45gfrederickslazy-random-states
or something to that effect#2017-11-1016:51borkdudeCool: https://clojure.github.io/test.check/clojure.test.check.generators.html#var-lazy-random-states#2017-11-1016:55borkdudeSo this would be it right? (take 2 (map random/rand-long (gen/lazy-random-states (random/make-random 2))))
#2017-11-1016:56gfredericksyep#2017-11-1615:27gfredericksif anybody is in the mood to do random tedious OSS work, I would like to upgrade the version of cljs that test.check depends on in its project.clj
but doing it causes some failures due to changes in the cljs compiler API#2017-11-1615:28gfredericks(if you plan to work on that, let me know so I don't do it too)#2017-11-1615:31nwjsmith@gfredericks I'm in, will take a look later today#2017-11-1616:13gfredericks@nwjsmith thanks!#2017-11-1714:17gfredericks@nwjsmith do you have a jira account?#2017-11-1714:20gfredericksIn any case I just created https://dev.clojure.org/jira/browse/TCHECK-135#2017-11-1715:02nwjsmith@gfredericks yes! I'm nwjsmith
in Clojure JIRA, and I've signed the CLA#2017-11-1715:07nwjsmithFeel free to assign that to me.
I got the tests running on the latest CLJS last night, but the approach I took might require a larger overhaul of how the tests are currently run. Do you mind if I make some substantial changes in that area?#2017-11-1716:02gfredericks@nwjsmith if you're confident that your changes are making the orchestration more idiomatic for lein-cljsbuild setups, then that's fine with me#2017-11-1716:02gfredericksI don't use modern cljs much at all so I can't easily judge those things#2017-11-1716:04nwjsmithI'm confident that's the case. Will be taking my lead from Om's test running and some stuff we have internally#2017-11-1716:04gfredericks@nwjsmith I see your user page, but the full-name-based dropdown that jira gives me on the ticket edit page does not have your name under 'Nate Smith' or anything similar; seems like a bug, but perhaps you can assign it to yourself?
/cc @alexmiller#2017-11-1716:05nwjsmithWeird. Not sure if this helps but here's a ticket I'm the reporter on: https://dev.clojure.org/jira/browse/CLJ-2079#2017-11-1716:06nwjsmithCan't assign myself unfortunately.#2017-11-1716:08gfrederickshaha great#2017-11-1716:09gfredericksI'll just keep it assigned to myself ¯\(ツ)/¯#2017-11-1716:21gfredericks@nwjsmith if there are any other contrib projects with cljs tests, those would be an interesting comparison point#2017-11-1716:24nwjsmithGood call, looks like tools.reader is a good example#2017-11-1716:27gfrederickscore.async and spec.alpha sound plausible also#2017-11-1716:29Alex Miller (Clojure team)@gfredericks @nwjsmith your user wasn’t in the right groups to do edit/assign - I fixed and assigned that ticket#2017-11-1716:29nwjsmithThanks Alex!#2017-11-1716:30Alex Miller (Clojure team)re cljs tests, note that the official build file for all contrib projects is the pom.xml (as that’s what the CI build uses)#2017-11-1716:30Alex Miller (Clojure team)and the pom.xml uses clojure-maven-plugin which doesn’t know or understand cljs (at least in the version we are forced to use due to jvm and maven versions)#2017-11-1716:31Alex Miller (Clojure team)many projects have parallel project.clj files which people use to run cljs tests locally#2017-11-1716:31Alex Miller (Clojure team)a few projects have started to experiment with ways to integrate these - I think data.xml is the most robust example of that#2017-11-1716:32gfredericks👍 thanks @alexmiller #2017-11-1716:32Alex Miller (Clojure team)it’s done by integrating the cljs tests into the clojure tests directly#2017-11-1716:33Alex Miller (Clojure team)in this area https://github.com/clojure/data.xml/tree/master/src/test/clojurescript/clojure/data/xml#2017-11-1716:33Alex Miller (Clojure team)uses nashorn I believe#2017-11-1716:37gfredericksOh interesting
@nwjsmith if you go the nashorn route I think it'd be good to maintain the ability to easily run with node as well#2017-11-1716:38nwjsmith@alexmiller oops I think I got assigned to the wrong ticket, should be TCHECK-135#2017-11-1716:39Alex Miller (Clojure team)oops! reading comprehension fail. fixed.#2017-11-1716:39nwjsmithThanks#2017-11-1716:40nwjsmithI'll look into integrating the CLJS tests with the Clojure ones as well.#2017-11-1716:44gfredericksgetting the cljs tests to run via maven is a nice-to-have; if it's much harder, just getting clojurescript upgraded is fine#2017-11-1818:18nwjsmith@gfredericks
The patch ended up being much smaller that I had thought https://gist.github.com/nwjsmith/edd1ce97b9e5334ac3cbc49a77ffb339
I'm trying to track down the other part of that ticket though. I can't seem to find any reader conditionals related to :/
in the tests. Did that already get fixed up?#2017-11-1818:19gfredericks@nwjsmith do you have the latest master?#2017-11-1818:20nwjsmithfacepalm#2017-11-1818:20nwjsmithI see it now#2017-11-1818:33nwjsmithI've attached the patch to the ticket. Do I need to fiddle with the JIRA workflow of the ticket at all?#2017-11-1818:36gfredericksNo -- you got both of the :/ things?#2017-11-1818:43nwjsmithYeah, both of the keyword-symbol-serialization-roundtrip
and edn-roundtrips
tests#2017-11-1818:44nwjsmithand they both pass#2017-11-1819:13gfrederickscool, taking a look now#2017-11-1819:25gfredericks@nwjsmith looks great! will push as soon as the tests pass on my end and then close the ticket
thanks again#2017-11-1819:37nwjsmithNo problem. Thanks for your help and let me know if there's anything else you'd like me to take a crack at#2017-11-1819:41gfredericksTCHECK-125 looks manageable, if you're interested#2017-11-1819:42gfredericksI'd wrap all the defmethod
s in an if
that checks if clojure.test/report
is a MultiFn
or whatever that class is, and if it's not then just print a warning to *err*
#2017-11-2823:27johanatanhi#2017-11-2823:28johanatanhave you considered the utility of a gen/let
which allows either generators or plain values on the right hand side?#2017-11-2823:31gfredericksthere are a lot of places in the API where that sort of thing could be done; my feeling is that it would cause more confusion than it would add value; and the ambiguity of it (a generator is also a value, sort of) is off-putting to me
a general rant about that sort of thing here: https://brehaut.net/blog/2013/duck_wrapping
that said, I did do exactly that in the body of gen/let
#2017-11-2823:31gfredericksbut I've never liked it 🙂#2017-11-2823:33gfredericksthe most useful thing I can imagine in this direction is being able to create a complex nested data structure where some of the leaves are generators and others aren't
but that sort of feature could be written as just a function that takes such a data structure and returns the generator you want, and calling an extra function wouldn't be much of a burden at than point since you by definition already have a big chunk of code#2017-11-2823:35johanatani'm not seeing how that feature could be written as just a function? since there can be dependencies between the nodes of the let#2017-11-2823:35gfredericksnot what you asked for exactly#2017-11-2823:35gfredericksI was talking about something else at that point#2017-11-2823:36gfredericksyou'd have to write a macro that expands to gen/let
if you wanted to implement the feature yourself#2017-11-2823:36johanatanwhat was the something else you were talking about though? perhaps that would interest me if I understood it 🙂#2017-11-2823:38gfredericksa function that you could use for your use case like so:
(gen/let [x (generatorfy something)]
...)
which is no easier than writing gen/return
; but it would have additional magical features so that you could also write (generatorfy [:foo gen/nat 42])
and get a generator for things like [:foo 12 42]
and [:foo 19 42]
and etc.#2017-11-2823:39gfredericksand you can imagine more complex examples where the generators are buried deep in some gnarly data structure#2017-11-2823:40gfredericksbtw if it would be useful for you to have anything of this sort in https://github.com/gfredericks/test.chuck so you don't have to maintain it yourself, I'm happy to accept PRs#2017-11-2823:41johanatanah, i see#2017-11-2823:41johanatanmakes sense.#2017-11-2823:41johanatanfor now i'll probably just use gen/return
as my need isn't too complicated#2017-11-3021:41johanatan@gfredericks are you aware that gen/list
does not seem to be "just like gen/vector
" in the sense that min and max arguments are not accepted?#2017-11-3021:42johanatanalso, why does gen/double*
accept its min
and max
as keys in an opts map rather than flat args as gen/vector
does ?#2017-11-3021:44gfredericks1) yes; gen/list
is a bit second-class; I'm happy to take a patch for the docstring if it's misleading. I'd consider enhancing it to be like gen/vector
, but that gets to
2) wherein the API is inconsistent, which is due to its being created in phases. I think opt maps everywhere would be ideal, and I suppose now that you have me thinking about it, gen/vector
in particular could be evolved that way backwards compatibly#2017-11-3021:54johanatanok, cool. just wanted to make you aware (if not already)#2017-11-3021:54johanatan[not a huge deal to me personally but yea of course consistency is good where possible]#2017-12-0100:20johanatanbtw, is it generally considered bad practice to call gen/sample
oneself nested within a huge gen/let
(or other combinator construct)?#2017-12-0100:21gfredericksYes#2017-12-0100:21gfredericksOr in a generator at all#2017-12-0100:21gfredericksSame with gen/generate#2017-12-0100:21johanatanhmm, I have through discipline been able to almost always avoid this but sometimes things get so hairy that the way to avoid it is not easy to see.#2017-12-0100:21johanatanis there any standard/general procedure one can apply to help escape that situation?#2017-12-0100:24gfredericksYou're passing to sample a generator that's based on other things you generated?#2017-12-0100:24johanatanyep#2017-12-0100:24Alex Miller (Clojure team)Use bind#2017-12-0100:24gfredericksI think it can be hard to talk about these things abstractly, but I'm happy to look at whatever code you can share#2017-12-0100:24johanatan@alexmiller would bind
work within a gen/let
?#2017-12-0100:25johanatanah, yep. that seems like it would work.#2017-12-0100:25johanatanthx!#2017-12-0100:26gfredericksNice#2017-12-0100:32johanatanis there a function that goes from list of generators to generator of a list?#2017-12-0100:32johanatan[there should be but i'm not seeing it]#2017-12-0100:33gfredericksTuple#2017-12-0100:33johanatanah, cool. thx#2017-12-0100:33gfredericksUsed with apply#2017-12-0100:33johanatanright#2017-12-0105:05johanatanwould it be possible to have a generator of an infinite/lazy sequence of arbitrary type?#2017-12-0105:05johanatan[like lazy-random-states
but with an arbitrary generator for the values]#2017-12-0105:07johanatan[the reason i'd like this is i have some logic deeply nested inside a gnarly structure of gen/let
, gen/bind
, gen/fmap
et al and realized in the very innermost loop that i need to pull N (dynamically determined) values of a particular type]#2017-12-0105:07johanatanit's probably possible to restructure the code to generate these on the outside but it's a bit difficult given that the amount to be generated is dynamic/ computed/ derived from other values#2017-12-0105:09johanatanso, if i could at the outermost level bind
to a lazy sequence, then inside the nests I could pull from that lazy sequence without violating "thou shall not call gen/sample
manually"#2017-12-0105:10johanatanpull via a regular take
that is#2017-12-0112:04gfredericksI've thought about use cases like that before
the trickiest part is thinking about how you shrink that structure#2017-12-0112:04gfredericksyou could setup a generator of infinite lazy sequences using a recursive gen/bind
, but I think it would never stop shrinking#2017-12-0112:05gfredericksand even with lower-level help from the library, it still needs to have a strategy for that
the tension comes from not easily being able to tell how much of the infinite source of data the user actually used#2017-12-0112:07gfredericksthere are certainly hacky ways to try to do that, and that might be the best answer, but the "hacky" part has made me hesitant to commit to anything#2017-12-0112:08gfredericksone less hacky trade-off you can make is to relax the "requirement" that there actually be an infinite number of things generated -- e.g., if you generate a large vector of items and then use (gen/fmap cycle ...)
#2017-12-0112:08gfredericksthen that will naturally shrink to an infinite repetition of one very small piece of data
so it only works if you can tolerate repeats#2017-12-0112:09gfredericksI remember making this recommendation for spec when it generates function stubs#2017-12-0118:38johanatanah, nice thoughts. would definitely be cool if something like this can be added in a not-too-hacky way but for now yea i'll go with a finite but repeating structure#2017-12-0119:17johanatanif i were to attempt a bind
to the (gen/fmap cycle ...)
structure you mentioned, that would be an infinite loop yea?#2017-12-0119:20gfredericksno I don't think so#2017-12-0119:22gfredericksunless the function that accepts the infinite sequence needs to consume the whole thing before returning a generator#2017-12-0122:48johanatani made these helpers to realize your idea above:
(defn- repeating-seq
([gen] (repeating-seq gen 1000))
([gen size] (with-meta (gen/fmap cycle (gen/vector gen size size)) {:size size})))
(defn- take-repeating-seq [n seq]
(gen/fmap #(take n (drop (%1 1) (%1 0))) (gen/tuple seq (gen/choose 1 ((meta seq) :size)))))
#2017-12-0123:23gfredericksHuh...#2017-12-0123:24gfredericksSo you'd use take-repeating-seq several times on the same generated sequence?#2017-12-0123:34johanatanyea#2017-12-0123:37gfredericksYou just don't know how many times you need to it until you're doing it?#2017-12-0123:38johanatanmm, well actually i've restructured the code since i did that so haven't had to use it yet. but i see it as a sort of escape hatch if one gets too deep#2017-12-0123:41johanatan[of course i could be totally confused 🙂 ] i always feel about halfway confused when deep in generator code#2017-12-0200:04johanatanthe above code is obviously not very efficient given that drop
is O(n) so yea probably better ways exist#2017-12-1114:24gfredericksDoes anybody have opinions about the idea of adding a generators-2
namespace that attempts to address lots of confusion points and inconsistencies in the current namespace?
I assume one big downside is the new confusion caused by having two namespaces forever#2017-12-1114:26gfredericksAnd even more work required for a good release 😕 #2017-12-1114:34Alex Miller (Clojure team)Maybe start with a list of problems?#2017-12-1114:50gfredericksyeah that's a good idea#2017-12-1304:34johanatandoes anyone know how to diagnose the following error?#2017-12-1304:35johanatan1. Unhandled java.lang.IllegalArgumentException
No implementation of method: :specize* of protocol:
#'clojure.spec.alpha/Specize found for class: nil
#2017-12-1304:37johanatan[sorry wrong channel. it's spec related; not test check related]#2017-12-1514:07gfredericksAs suggested, here is a list of problems that make me wonder about creating a generators-2
namespace:
https://dev.clojure.org/display/design/Generators+Reboot
Apparently my original question has already been lost to the black hole of slack, so that's pretty good.#2018-01-1812:44gfredericksI reverted the changed that added map-bindings to gen/let
#2018-01-2617:04ghadi@gfredericks can you check in a deps.edn ?#2018-01-2617:04ghadiI'd like to use test.check directly from git#2018-01-2617:22ghadi{:paths ["src/main/clojure"]
:aliases {:test {:extra-paths ["src/test/clojure"]}}}
#2018-01-2617:31Alex Miller (Clojure team)You don’t need one#2018-01-2617:31Alex Miller (Clojure team)We support Pom manifests now#2018-01-2617:32Alex Miller (Clojure team)Although I guess you’re missing :paths#2018-01-2617:32Alex Miller (Clojure team)Unfortunate#2018-01-2617:32Alex Miller (Clojure team)I’m going to add src/main/clojure automatically I think - then you won’t need one#2018-01-2618:36ghadiFancy#2018-01-2618:37ghadiWe're beating Maven at its own game @alexmiller#2018-01-2702:32gfrederickshuhwhat?#2018-01-2702:32gfrederickslike src/main/clojure
is always on the path for every project and you can't turn it off?#2018-01-2712:31Alex Miller (Clojure team)Just for git and local deps with a Pom manifest#2018-02-0802:57wilkerluciohello, I'm trying to limit the recursion of a generator, the problem is that the recursion is not "direct", it generates something, that generates something else, and ends up in circle, but this kind of recursion seems to not be detected by test.check recursion limit#2018-02-0802:57wilkerlucioI was thinking about using some dynamic var to track how many times my generator was called, and stop if goes after a number in a call stack#2018-02-0802:58wilkerlucioI was trying something like this:#2018-02-0802:58wilkerlucio(s/def ::query-root
(s/coll-of ::query-expr-root :kind vector?
:gen #(if (> *max-depth* 0)
(binding [*max-depth* (dec *max-depth*)]
(s/gen (s/coll-of ::query-expr-root :kind vector? :max-count 5)))
(s/gen #{[]}))))
#2018-02-0802:58wilkerluciobut this doesn't work, because the binding is running on the generator definition, and not when the actual gen is running#2018-02-0802:59wilkerlucioI was trying to find a way to wrap some generator with this logic, but can't figure how to hook it up#2018-02-0802:59wilkerluciohow can I make this? because otherwise my generator fail about 80% of the time because of stack overflow =/#2018-02-0814:30gfredericksby "test.check recursion limit" you're referring to the mechanism in clojure.spec that's used when generating generators?#2018-02-0814:31gfrederickshave you tried expressing the generator for the whole thing using gen/recursive-gen
?#2018-02-0900:14wilkerlucio@gfredericks sorry the delay, I'm not sure I can use recursive-gen on this case#2018-02-0900:15wilkerluciobut I was able to make it work#2018-02-0900:15wilkerluciothis how it ended up:#2018-02-0900:15wilkerlucio(s/def ::query
(s/coll-of ::query-expr :kind vector?
:gen #(let [g (s/gen (s/coll-of ::query-expr :kind vector? :max-count 5))]
(gen/->Generator
(fn [rdn size]
(if (> *query-gen-max-depth* 0)
(binding [*query-gen-max-depth* (dec *query-gen-max-depth*)]
(gen/call-gen g rdn size))
(gen/call-gen (gen/return []) rdn size)))))))
#2018-02-0900:15wilkerluciothis limits the depth of the generator by using a dynamic var, was the best solution I could find#2018-02-0900:20gfredericksyeah, I think it's technically possible with recursive-gen but maybe not easy
would definitely be better for spec to handle it without error#2018-02-2318:05pabloreHi, how can I make a generator for a clojure.lang.PersistentQueue
?#2018-02-2318:14pabloreAlready did it:
(defn gen-queue
[some-generator]
(gen/bind (gen/vector some-generator))
#(into (clojure.lang.PersistentQueue/EMPTY) %)))))
#2018-02-2318:29gfredericksThat shouldn't work#2018-02-2318:30gfredericksShould have to switch bind to fmap and swap the arg order#2018-03-0517:44ghadi@gfredericks what's going on with https://dev.clojure.org/jira/browse/TCHECK-2 ? we just got bit by that#2018-03-0517:45gfrederickswhat's a good solution w.r.t. not breaking things? deprecate the old poorly named things and make new things with good names?#2018-03-0517:48ghadii guess i'm confused on how it's poorly named vs broken#2018-03-0517:48ghadi(pos-int? 0) => false#2018-03-0517:50ghadiIIRC it just so happens that spec filters out by doing (such-that pos-int? gen/pos-int) -- but if you use test.check directly you'll see 0#2018-03-0517:55ghadiyeah:
user> (clojure.spec.gen.alpha/sample (clojure.spec.alpha/gen pos-int?))
(1 1 1 4 2 4 31 1 8 4)
user> (clojure.test.check.generators/sample clojure.test.check.generators/pos-int)
(0 0 1 1 0 5 1 2 1 4)
#2018-03-0518:16gfredericksI think it was always a case of a bad name; I have no idea where those names came from, but I think the docstrings have reflected the actual behavior#2018-03-0518:16gfredericksso changing the behavior & docstrings to match the names would potentially break anybody using them#2018-03-0518:16ghadiugh#2018-03-0518:16ghadiwhat does the s in s-pos-int mean?#2018-03-0518:17gfredericks"strictly" -- which points even more strongly to the bad names theory#2018-03-0518:17ghadilmao#2018-03-0518:18gfredericksthere's a relevant section on integer generators here: https://dev.clojure.org/display/design/Generators+Reboot#2018-03-0518:18ghadii see#2018-03-0518:22ghadiin rich's compatibility algebra of provides and requires -- is not returning 0 a breaking change? Is it possible that certain random seeds never see 0?#2018-03-0518:22gfredericksI've wondered about how the compatibility algebra applies to generators and distributions 😄#2018-03-0518:23gfredericksI do think reducing the distribution is a breaking thing#2018-03-0518:23gfredericksyou're silently causing users' generators to test less than they used to#2018-03-0518:23ghadimy 2c: this generator is busted and it's not that big of a deal to remove 0.
(There is an argument around people relying on behavior whether correct or not)#2018-03-0518:24ghadii can be persuaded either way -- but not for a crappy name like s-pos-int#2018-03-0518:25gfredericksno I don't think "leave everything as-is" is a good solution#2018-03-0518:26gfredericksgiven zero is such an important edge case a lot of the time, silently making people's tests less effective is also a bad idea I think#2018-03-0518:27gfredericksI think the bigger question that needs deciding is the one from the link above -- whether to start a new namespace or improve things by deprecating individual generators#2018-03-0518:27gfredericksI feel like rich's paradigm could use some good deprecation tooling#2018-03-0519:14gfrederickswoah the spec alias of the exact same name has different behavior?#2018-03-0519:14gfredericksgee willickers#2018-03-0519:18ghadiyup#2018-03-0519:19ghadimaybe we should ask Rich on the mailing list#2018-03-0519:21gfredericksabout deprecation or about generator aliasing?#2018-03-0519:23ghadiThe former#2018-03-0519:29gfrederickslike whether deprecation is a good idea at all, or the specific question of how we should do it exactly?#2018-03-0519:34ghadiwhat to do specifically#2018-03-0519:34ghadineed a third set of eyes#2018-03-2922:51grumpletIs there a reasonably simple way to implement generative tests on an async function? It looks like this might be impossible without reimplementing test.check or clojure.spec.test.alpha/check?#2018-03-2923:08hiredmanit depends how you do your async functioning#2018-03-2923:09hiredmanif you just call async functions with abandon, it is hard, if create a an Executor that you can shutdown and run them on that Executor it is pretty easy#2018-03-2923:12hiredmanbasically, if you can put a function like interface on it, it doesn't matter if internally it does stuff asynchronously#2018-03-2923:21gfredericksFor cljs it's impossible#2018-03-2923:22grumplet@gfredericks That's the conclusion I was coming to 😞#2018-03-2923:23gfredericksYeah, it will take some work to add that#2018-03-2923:23gfredericksDefinitely a priority#2018-03-2923:24grumpletI'm trying to test a remote call to R against a cljs call which should return the same values and was hoping to get test.check style behaviour.#2018-03-2923:24gfredericksremote call = ajax?#2018-03-2923:24grumpletyep#2018-03-2923:26gfredericksA couple heavy workarounds: test from clj, either by rewriting your logic in cljc or shelling out to node#2018-03-2923:26gfredericksNeither of those are nice obviously#2018-03-2923:27grumpletInteresting idea though. I suspect the stats model would port to clj without too many problems.#2018-03-2923:28grumpletThanks - I'll think about that approach.#2018-04-0820:47gfredericksWould it be possible to have a generic test assertions library that plugs into both test.check and clojure.test and other testing libraries?#2018-04-0820:48gfrederickstest.check and clojure.test both have an impoverishment of assertions, where testing-specific assertions are needed so that specific feedback can be given to the user about how it failed#2018-04-0820:48gfredericksand it would be sad for that problem to be addressed in one library without the other#2018-04-1016:04Joe LaneIs there an api in test.check to get the collection of inputs and outputs used when exercising a test?
For example, when running
(defspec first-element-is-min-after-sorting ;; the name of the test
100 ;; the number of iterations for test.check to test
(prop/for-all [v (gen/not-empty (gen/vector gen/int))]
(= (apply min v)
(first (sort v)))))
getting a collection of
({:input [1] :output [1]},
{:input [2 1] :output [1 2]}
...
I suspect it the data I’m looking for is :args
and :result-data
or :result
from the quickcheck
function in check.cljc
.
Is there a blessed way to get access to this information?#2018-04-1016:11ghadiyou can recreate it using the random seed that quickcheck gives you#2018-04-1016:20Joe Lane@ghadi I can get the data literals using the seed? I know I can re-run the test again, i’m curious about the data itself though#2018-04-1016:21ghadiright, I'm not sure the exact process#2018-04-1016:21ghadithe seed is there so that you can get reproducibility#2018-04-1016:24nberger@lanejo01 may I ask what do you plan to do with that data? I'm truly curious, mostly because a while ago I wanted to add a way to get a glimpse on what's the distribution of the generated values#2018-04-1016:32gfredericksargs is the generated data#2018-04-1016:33gfredericksThe return value isn't given, you can just call your function yourself once you have the args#2018-04-1016:34ghadi@gfredericks can you do something like sample-seq
while providing the seed?#2018-04-1017:06gfredericksIt could exist, but might not be immediately useful since you couldn't easily pick out the failing args. Do you have another use case?#2018-04-1016:34Joe LaneI had another instance of a recurring dream last night. The word cloud looks like this: clojure.spec generative testing + consumer driven contracts (like http://pact.io) except for libraries, not web apis + spec-ish type-system-ish code inspection/analysis (e.g. (-> (makes-random-int) expects-only-odds)
would be analyzed and the system would show there is a problem) + service for publishing specs to a public repository.
The big thing here is consumer driven contracts for function apis. A library author could know the expectations of its consumers based on properties they provide, and then know if they make a breaking change or not.#2018-04-1016:35Joe LaneThere are a LOT of conflated ideas here, please dont take this as a proper description of all goals.#2018-04-1016:39Joe LaneI suppose the ideas are 1. consumer driven contracts for public library functions 2. analysis of clojure programs using spec to determine areas for potential bugs 3. maven for specs.#2018-04-1016:40Joe LaneThe reason I’m asking is about inputs and outputs is to create a system like haskell’s hoogle (https://www.haskell.org/hoogle/) with spec.#2018-04-1016:41Joe LaneThanks @gfredericks!#2018-04-1016:55grzm@lanejo01 The way something like that could work is if consumers published specs that the upstream author could consume, and those specs would have to be more constrained versions of the specs the author already publishes. The upstream author could swap those more specialized specs in their own tests. That reminds me of "specialization by constraint" in Date and Darwen's "Databases, Types, and the Relational Model". I'm not well-versed in type-theory in general, so it might have other names, but that's where I came across it.#2018-04-1016:56grzmThe idea there is that any proper subtype is only a subtype if it satisfies the definition of its parent type plus some additional specialization constraint.#2018-04-1016:57Joe LaneThat’s exactly how http://pact.io does it, except with json and regex, not something more expressive like spec + edn.#2018-04-1017:01grzmHow would "maven for specs" be different from maven (or similar tool) in general? Specs are code and just like when you use a library, you still need to know the names of the functions (or specs) once you have it.#2018-04-1017:13Joe LaneIt was a bit of a mischaracterization of what I’m going after ( which, is admittedly, a moving target). It’s more about a place for consumer “contracts” to be placed. In addition to that, the idea of a central place that I can look up what functions take this input and produce this output could be really exciting for a variety of reasons. Not so much maven for specs. Sorry for the confusion.#2018-04-1417:52nwjsmithI think I've asked this before, but I'm writing some (non-shrinking) generators that rely on generating uniformly random numbers. Is there any way to get a uniformly random number generated?#2018-04-1417:52nwjsmithAre any of the provided generators uniformly random? (i.e. gen/bytes
?)#2018-04-1417:53nwjsmithIf so, I might be able to fmap
my way into a random number#2018-04-1417:58nwjsmithErr, never mind. I might not want a uniform random number here#2018-04-1418:01gfredericksIn any case, gen/choose is such a thing, for smaller ranges#2018-04-1418:01gfredericksI've thought about adding something more robust for integers and doubles#2018-04-1418:06nwjsmithFWIW I'm generating a probability between 0.0 and 1.0, a double.#2018-04-1418:08gfredericksBut not uniform?#2018-04-1418:10nwjsmithI don't think it needs to be. It's a generator for a graph's adjacency matrix. It is the probability that one vertex is adjacent to another.#2018-04-1418:11nwjsmithSo an argument could be made that the most simple graph is the fully disconnected one#2018-04-1418:11nwjsmith(i.e. the probabilities are all 0.0#2018-04-1418:11gfredericksYeah that makes sense#2018-04-1418:11gfredericksI wonder what gen/double does in that range#2018-04-1418:12gfredericksIt might bias toward 1 at first#2018-04-1418:12gfredericksIf so you can fmap with #(- 1 %) to get the opposite#2018-04-1418:14nwjsmithYeah, with a sample of 1000 I get 101 occurrences of 1.0
and 19 occurrences of 0.0
.#2018-04-1418:15gfredericksThat might be misleading#2018-04-1418:15gfredericksfmap to #(long (* 10 %)) and see what you get#2018-04-1418:16gfredericksIt probably considers 0 and 1 simpler than everything in between#2018-04-1418:16nwjsmith{1 39, 0 698}
#2018-04-1418:17gfredericksNothing from 2 to 10?#2018-04-1418:18nwjsmith{0 700, 7 25, 1 39, 4 23, 6 14, 3 26, 2 31, 9 6, 5 36, 10 87, 8 13}
#2018-04-1418:18nwjsmithAre the full frequencies#2018-04-1418:19gfredericksSo mostly 0, because that's simpler, and next is 1.0, then 0.5, ...#2018-04-1418:20nwjsmithLooks like gen/double*
will do just fine#2018-04-1418:27gfredericksAre you actually trying to generate probabilities, or trying to generate the graph itself?#2018-04-1419:48nwjsmithGenerate the graph itself. I generate an adjacency matrix like this:
[[0.0 0.0 0.0 0.0]
[P.1 0.0 0.0 0.0]
[P.2 P.3 0.0 0.0]
[P.4 P.5 P.6 0.0]]
#2018-04-1419:49nwjsmithWhere P1
-`P6` are probabilities. This adjacency matrix ends up representing a random DAG.#2018-04-1419:50nwjsmithI map this probability matrix into a matrix where the values can be true
/`false` based on whether or not the probability is above 0.5
#2018-04-1419:53nwjsmithI have another generator that takes a DAG and generates a random topological ordering from it. It's also based on random number generation using gen/choose
, but the range is small enough that it should be uniform.#2018-04-1419:57nwjsmithBoth are looking good:
(let [g (gen/generate (gen-directed-acyclic-graph (seq "ABCD")))
t (frequencies (gen/sample (gen-topological-ordering g) 1000))]
(when (not-empty g)
(println t)
( (loom.graph/digraph g))))
Produced: https://www.dropbox.com/s/q0rkqfy7z0vffs6/png223331814782198197.png?dl=0
and the topological orderings:
{[C B D A] 122,
[D B A C] 117,
[B D A C] 115,
[D B C A] 127,
[B D C A] 135,
[C D B A] 132,
[D C B A] 137,
[B C D A] 115}
#2018-04-1419:59gfredericksCool#2018-04-1613:54ghadivery cool. @nwjsmith could you share your code for the graph or topological-ordering?#2018-04-1613:55nwjsmith@ghadi I'll try to get a repo up tonight or tomorrow, they should be fairly general-purpose. I'll ping you here when it's up#2018-04-1703:54mathpunkI was trying to deal with this bug (https://github.com/clojure-emacs/cider/issues/1841), and in investigating it I discovered that test.check
refers to a namespace that it doesn't have in the definition of test-context-stacktrace
: https://github.com/clojure/test.check/blob/master/src/main/clojure/clojure/test/check/clojure_test/assertions.cljc#L12#2018-04-1703:55mathpunkI commented that line out, and my problem went away. It's not clear to me how to contribute or report that so, I'm mentioning it here#2018-04-1703:57mathpunkAs I look at it more closely, I doubt it's that exact spot that helped; but, there are definitely several references to a clojure.test.check.clojure-test
namespace. I see only a clojure_test
directory, but no namespace#2018-04-1704:18Alex Miller (Clojure team)that’s due to someone monkey-patching clojure.test#2018-04-1704:18Alex Miller (Clojure team)leiningen does that#2018-04-1704:19Alex Miller (Clojure team)maybe others#2018-04-1704:20Alex Miller (Clojure team)https://github.com/technomancy/leiningen/issues/2173#2018-04-1704:20Alex Miller (Clojure team)and https://dev.clojure.org/jira/browse/TCHECK-113#2018-04-1704:29mathpunkWhile I did see this, I don't quiiiiite know what it means, nor what it implies about my project being a boot project#2018-04-1704:30mathpunkI've heard the term but only the context of 1) Ruby objects and 2) disapproval. Sometimes at once~#2018-04-1704:32Alex Miller (Clojure team)well this may be something similar but different, not sure. Leiningen reaches in and replaces the implementation of part of clojure.test, which breaks test.check’s expectation.#2018-04-1704:32Alex Miller (Clojure team)that’s a pretty dirty trick imo :)#2018-04-1714:39nwjsmith@ghadi https://github.com/nwjsmith/generators.graph#2018-04-1714:41ghadiNice @nwjsmith. That will be very useful, also good learning material.#2018-04-1714:42nwjsmithI'll try to spruce it up a bit this week. I think there's a ton of performance improvements available too, so I'd like to get some benchmarks around it.#2018-04-2019:57devnIf i want to generate a double with limited precision, how do?#2018-04-2020:00gfredericksExpressed in base 10 or base 2?#2018-04-2020:01gfredericksAnd do you mean precision relative to the fixed decimal point, or in the floating point sense?#2018-04-2020:02gfredericks(like is 1e-100 high precision?)#2018-04-2020:02devnyes, relative to the fixed decimal point#2018-04-2020:02devnbase 10#2018-04-2020:03gfredericksI'd generate an integer and divide it by 100.0 probably#2018-04-2020:03gfredericksWith fmap#2018-04-2020:03devnthanks @gfredericks#2018-04-2020:03gfredericks👍#2018-04-2020:03gfredericksObligatory warning about how doubles can't precisely represent most of those numbers#2018-04-2116:25mathpunkI found something unclear in the documentation for test.chuck
. I guess it could be a bug but I think it's more likely I'm holding generators wrong. In particular, I'm trying out the checking
macro#2018-04-2116:26mathpunkWhen I tested this myself---#2018-04-2116:29mathpunkSo the surprising bit is, why does it work fine if I get my generator with (s/gen int?)
and fail if I get (what I think is) the same generator via gen/int
?#2018-04-2116:29gfrederickshow does it fail?#2018-04-2116:30gfredericksoh sorry#2018-04-2116:30gfredericksonly saw the first five lines of the snippet#2018-04-2116:31gfredericksI'm guessing gen/
refers to clojure.spec.gen.alpha
or something of that sort#2018-04-2116:31gfredericksand that this is the inevitable confusion that stems from the fact that spec generally deals with 0-arg functions that return generators#2018-04-2116:32gfrederickswhile http://test.ch[eu]ck deals with vanilla generators#2018-04-2116:32gfrederickstry (gen/int)
#2018-04-2116:32mathpunkyep, that did it#2018-04-2116:33mathpunki can attempt to fix that doc if you like, though I haven't forked-and-pulled much open source#2018-04-2116:34gfrederickswhich doc?#2018-04-2116:34gfredericksthe checking
docs are referring to the clojure.test.check.generators
namespace#2018-04-2116:34gfredericksif you used that namespace instead, gen/int
would work and (gen/int)
wouldn't#2018-04-2116:37mathpunkI'm looking here: https://github.com/gfredericks/test.chuck/blob/master/src/com/gfredericks/test/chuck/clojure_test.cljc#L108#2018-04-2116:38mathpunkMind you I've been working on untangling the differences among:
- clojure.test
- test.check
- test.chuck
- test.check.clojure-test
- clojure.spec.alpha.test
...so maybe I'm confusing the hammers I should be using#2018-04-2116:41gfredericksthe problem is that there are two different namespaces that you'll see aliased as gen/
#2018-04-2116:41gfredericksone in test.check and one in clojure.spec#2018-04-2116:42gfredericksthe namespace in clojure.spec mostly exists to alias the namespace in test.check#2018-04-2116:42gfrederickswith the added confusion that it has functions that return generators instead of generators#2018-04-2116:43gfredericksI would alias the clojure.spec namespace as sgen
just to keep things straight#2018-04-2116:43gfredericksso gen/int
is the same as (sgen/int)
#2018-04-2116:49mathpunkI see. Do they have different purposes? Or is clojure.spec an alpha because it's absorbing test.check but hasn't fully digested it yet?#2018-04-2116:50gfredericksthe purpose of the spec namespace is to lazily load test.check only when necessary, so you don't need to have it as a production dependency#2018-04-2116:50gfredericksthat's also the reason for the functions-that-return-generators thing#2018-04-2116:50mathpunkohhhhh huh, ok#2018-04-2116:50mathpunkbecause specs have some use in production code#2018-04-2116:53mathpunkThanks--- there's a lot of polysemy going on in the terms test
spec
gen
and check
as I try and develop a sensible workflow for spec-driving#2018-04-2116:54gfredericksit needs a good blog post or something#2018-04-2117:00mathpunkI may be able to contribute to the beginners' end of the topic. The comments I'm writing to myself may become a post on a repl-to-gen-tests workflow with cognitect/transcriptor
and test.chuck
#2018-04-2117:01mathpunkNot gonna touch when a gen is a gen and when a gen is a thunk#2018-05-2405:38flowthingIs there a better way to write this?
(gen/fmap (partial reduce merge)
(gen/vector
(gen/let [id gen/uuid
x (gen/hash-map :name gen/string)]
{id (assoc x :id id)})))
That is, generate maps where the key is the same as the :id
in the map in the value. That works fine and doesn't seem too bad to me — just wondering.#2018-05-2410:38gfredericksif it were me I would be using plumbing and so I'd have this function around to make it slightly cleaner: https://github.com/plumatic/plumbing/blob/6d713472f7324344545aa93597cb2a137b3404b3/src/plumbing/core.cljx#L91#2018-05-2410:39gfredericksso it becomes (gen/fmap #(map-from-vals :id %) (gen/vector gen-a-map-with-an-id))
#2018-05-2410:41gfrederickswhich I guess means it also seems natural to me to pull out the map generator and give it a name#2018-05-2721:15gfredericks0.10.0-alpha3
is out https://github.com/clojure/test.check/blob/master/CHANGELOG.markdown#0100-alpha3-2018-05-27#2018-05-2721:15gfredericksit's crazy how much stuff can be done without doing any of the stuff I want to do#2018-05-2721:16gfredericksthank you to everybody who's contributed#2018-06-2617:49ghadiWhat's the most idiomatic way to make a generator from a source function (like a constructor)?#2018-06-2618:19ghadi(gen/fmap (fn [f] (f)) (gen/return some-constructor))
Seems oblique to say fmap "apply thyself"#2018-06-2618:26gfredericksthat example might be too simple to express your problem, because you can rewrite it as (gen/return (some-constructor))
#2018-06-2618:26gfredericksunless the constructor is nondeterministic#2018-06-2618:26ghadiI want to generate fresh instances#2018-06-2618:26ghadi(the constructor returns a reify)#2018-06-2618:26gfredericksis this with spec?#2018-06-2618:27ghadiyes#2018-06-2618:28gfredericksin general it's not idiomatic in test.check to be generating non-values, so spec exposes a bit of a mismatch here#2018-06-2618:28gfredericksbut I suppose I've heard similar things from alex about spec#2018-06-2618:29gfredericksso due to all that, there's not a more idiomatic way to do what you're doing#2018-06-2618:29gfredericksyou might even have to be careful about the same instance getting reused during shrinking#2018-06-2618:30gfredericksI'm 87% sure that's what would happen if you compose this with other generators#2018-06-2618:31ghadiInteresting.... Thanks Gary#2018-06-2618:33gfredericks👍 this might be an interesting thing to point out in the spec channel#2018-07-0109:38jfacorroHi 🙂. I've been working on porting test.check
to Clojure on Erlang VM. I'm pretty much done but I was wondering what's the relationship between test.check
and test.generative
. I've noticed that spec.alpha
uses test.check
and that's partly why I ported the library (because I want to port spec.alpha
as well), but test.generative
seems to be used in tests for clojure-1.9.0
.#2018-07-0109:39jfacorroI guess what I'm trying to ask is if test.check
is a superset of test.generative
(which I think it is) or if they are different in some fundamental way I'm missing.#2018-07-0110:25jfacorroI can see there is no shrinking in test.generative
#2018-07-0113:21gfrederickstg predates tc#2018-07-0113:21gfredericksI don't think tg is used much anymore#2018-07-0114:44jfacorroI see that c.t.g
it is being used for tests in clojure/clojure
, so is c.t.c
it seems#2018-07-0114:44jfacorro¯\(ツ)/¯#2018-07-0114:54gfredericksyes, the tg tests were added before tc existed#2018-07-0114:54gfredericksI expect those could be converted, but changes to the core codebase aren't normally accepted unless they provide a lot of value#2018-07-0115:05jfacorroI see, thank you 🙂#2018-07-0612:29Hukkastrgen helpfully mentions test.chuck for more comprehensive regex support. But is there then any reason to not use test.chuck always?#2018-07-0612:48gfrederickstest.chuck isn't an alternative to test.check, it's a pile of miscellaneous utilities to go with it#2018-07-0612:48gfredericksso it's fine to have it around if you want, but if you're not using any of its parts then it's not providing any value#2018-07-0612:48HukkaSure. I'm mostly looking for helpers to generate the input#2018-07-0612:49Hukkastrgen is purely for making strings based on regexes, but mentions that test.chuck has more comprehensive support. I was just wondering is there ever any reason to use that, instead of test.chuck#2018-07-0612:50gfredericksoh I'm sorry#2018-07-0612:50gfredericksI missed the part about strgen#2018-07-0612:51gfredericksit looks like strgen targets portability?#2018-07-0612:51gfredericksi.e., it probably works in cljs#2018-07-0612:52gfredericksthere's a half-pull-request in test.chuck for adding cljs support, but it needs some more work#2018-07-0612:54HukkaI see, thanks! Good to know – though for now I'm just speccing the backend code, there's plenty of fiddly data munging on the frontend too#2018-07-0615:14HukkaNot sure if this belongs here, as it's really a spec problem... but if running (stest/check) without any arguments gives an exception when check can't create any input that passes the input predicates, is there any way to know which specced function is causing problems?#2018-07-0622:45gfrederickslot more likely to be answered in the spec room I'd wager#2018-07-0622:45gfredericksthey have over ten times more people#2018-07-2016:38ghadihttps://twitter.com/smashthepast/status/1020347051589144578#2018-07-2016:43gfredericks👍#2018-07-2018:10gfredericksnot to mention if you get a rare failure you'll have a hard time reproducing it unless you're logging the generated value#2018-09-1813:30arohnerIs there a good way to generate a lazy sequence? I want to reduce
the seq until a specific condition is met#2018-09-1823:01gfrederickswhat aspect of laziness is important to you in this case?#2018-09-1823:01gfredericksyou can make whatever you want with gen/fmap
#2018-09-1823:01gfrederickswere you hoping for an infinitely random sequence?#2018-10-1218:18KaylaIn this snippet, frequency
evals render
too early. I think I need mutually recursive generators, and don’t want to lose the ability to shrink. If it actually ran it wouldn’t go on forever, but instead it explodes the stack. Anybody have any ideas?
(letfn [(render []
(tcgen/frequency [[1 (gen/tuple (render))]
[15 (gen/return [])]]))]
(gen/generate (render)))
#2018-10-1219:04hiredmanthe problem isn't frequency#2018-10-1219:05hiredmanfrequency is a function, so all the arguments are evaluated before the function is called#2018-10-1219:06taylorwhat kinda output are you hoping to get out of that. nested, empty vectors? @sean#2018-10-1219:06hiredmanyou may want to checkout gen/recursive-gen#2018-10-1220:00Kayla@taylor I distilled my problem down to this.
I am generating a tree where any given node’s children’s generator is dependent on the node that produces it. Also, some nodes must have children, and some must not. Also some of the data in the child needs to be consistent with what was generated the parent. Then the thing that breaks it is some nodes can have children of the same type, which use the same generator.
I am trying to make a generator for the whole tree.
So a generator for a node might produce something like: {:id 1 :parent-id 0 :type :foo}
Nodes of type :foo
can have children of types :bar
and :foo
.
The generators for the potential children of this node would produce something that looks like: {:id 2 :parent-id 1 :type :foo}
or {:id 2 :parent-id 1 :type :bar :another-quality "junk"}
#2018-10-1220:18taylorare the child nodes associated inside the parent nodes? or are they all in one big sequence and only correlated by :id
and :parent-id
?#2018-10-1220:18KaylaOne big sequence #2018-10-1220:01Kayla:bar
nodes have no children.#2018-10-1220:16gfredericksYou could use recursive-gen to get the structure, then fmap the whole thing to fix up consistency issues#2018-10-1220:19hiredman(defn type-foo [g]
(gen/let [children (gen/vector g)]
(gen/return
{:id 1 :parent-id 0 :type :foo :children children})))
(defn type-bar [g]
(gen/return
{:id 2 :parent-id 1 :type :bar :another-quality "junk"}))
(defn foo-bar-tree [_]
(gen/recursive
(fn [g]
(gen/one-of (type-foo g)
(type-bar g)))
(type-bar nil)))
#2018-10-1220:27hiredmanI will say often times it is easier to generate instructions to build a complex datastructure then it is to generate the datastructure. so for example if you wanted a tree of files in directories, it might be easier to generate a list of create file, move file, create directory, delete, etc, then just interpret that list to get some tree, instead of generating the tree#2018-10-1220:55taylor@sean even though you want a flat sequence of maps, you could maybe do something like this with recursive-gen:
(gen/sample
(gen/recursive-gen
(fn [g]
(gen/let [{:keys [id type] :as m} (s/gen ::my-map)
children (gen/vector g)]
(if (= :bar type)
m
(update m :children concat (map #(assoc % :parent-id id) children)))))
(s/gen ::my-map))
100)
then you could fmap
over that generator to flatten and fix-up the ID relations:
(gen/sample
(gen/fmap
(fn [m]
(map #(dissoc % :children)
(tree-seq #(seq (:children %)) :children m)))
(gen/recursive-gen
(fn [g]
(gen/let [m (s/gen ::my-map)
children (gen/vector g)]
(if (= :bar (:type m))
m
(update m :children concat (map #(assoc % :parent-id (:id m)) children)))))
(s/gen ::my-map))))
although I think this still leaves you with a problem of non-unique IDs#2018-10-1220:56tayloroh and I used some specs for the map generator:
(s/def ::id pos-int?)
(s/def ::type #{:foo :bar})
(s/def ::parent-id ::id)
(s/def ::my-map (s/keys :req-un [::id ::type ::parent-id]))
#2018-10-1221:22KaylaThanks Everybody for all the help. I’m signing off for the weekend. #2018-11-1303:32steveb8nI find it tricky to quickly find the cause when I get “clojure.lang.ExceptionInfo: Couldn’t satisfy such-that predicate after 100 tries. {}” back from generators. I can find it using binary search but is there a better technique for debugging these generator errors?#2018-11-1312:52gfredericks@steveb8n is this a (gen/such-that ...)
that you wrote yourself, or that spec wrote for you?#2018-11-1321:02steveb8n@gfredericks using gens from specs (I hardly ever use such-that gens). To me this is the hardest part of using generators so any tricks (or context data in the error) would help#2018-11-1321:15gfredericks@steveb8n there's a ticket somewhere describing that problem I think -- I added extra optional args to gen/such-that
so that spec could add explanatory information to exceptions, but I don't believe that's been taken advantage of by spec.#2018-11-1321:15gfredericksand in any case it's only released in the alpha versions 😕#2018-11-1321:15gfrederickssomeday I should just give up and release whatever's on master so at least there's not that problem anymore#2018-11-1321:19steveb8nhah, I thought this might be the case. I also thought that providing useful debug data would be tricky since specs can vary wildly. in lieu of that, how do you hunt these down? binary search (with knowledge of the specs to guide you) or some other cleverness?#2018-11-1321:23gfredericksI haven't personally used spec a lot
I think knowing that s/and
is the most likely culprit is usually helpful#2018-11-1321:25gfredericksand thinking about generation whenever you write s/and
in the first place#2018-11-1323:23steveb8ntrue that. I am using s/and extensively. I’ll ponder that a little. Thanks.#2019-01-1307:35Ho0manHi everyone, is there a way I could fill the :result-data
field in test reports?
(So when they fail I could have a structured failure report)#2019-01-1315:30gfredericks@ho0man yes, you have to return something that implements the protocol in clojure.test.check.results
I think#2019-01-1315:31gfredericksyou could use (reify ...)
for that#2019-01-1316:15Ho0man@gfredericks , thanks I’ll try that#2019-01-2419:33pabloreHi. I need to generate some data based on a normal distribution. How can I do that with test.check?#2019-01-2419:37gfredericks@pablore you want to ignore the size
parameter, I imagine?
any desired shrinking behavior?#2019-01-2419:37pabloreI dont understand what you mean#2019-01-2419:38gfrederickssize
controls how generated values start simple and get more complex as you run more trials#2019-01-2419:38gfredericksmore details here: https://github.com/clojure/test.check/blob/master/doc/growth-and-shrinking.md#2019-01-2419:39pabloreOk this is more complex than I thought xd. I'll give it a read, thanks!#2019-01-2419:41gfredericksif you don't care all that much, I'd say the easiest thing is to use gen/choose
with 0
and Integer/MAX_VALUE
and the gen/fmap
with some function that converts a uniform distribution to a normal one#2019-01-2419:41gfredericksmight help to divide by Integer/MAX_VALUE
first I guess#2019-01-2419:41gfredericksto get it between 0 and 1#2019-01-2419:42pablorethanks#2019-01-2419:42gfredericks(the fact that my initial questions don't have obvious answers is part of why this isn't built-in)#2019-01-2421:26aisamuHi!
I'm trying to integrate test.chuck's checking
into a cljs.test project.
The readme points to a blog post where there is an example showing the full report on errors (vs. just one wrong result)
Was that ever integrated into the official checking macro? I can't seem to make it work here.
(The full report is terribly useful for fixing things that went wrong!)#2019-01-2421:43gfrederickswhat do you mean by "full report"? I.e., what information is there?
(I'm fuzzy on all this)#2019-01-2423:11aisamuOh, sorry for the delay!
Here's the linked page - on the section "Including Results" you can see an example:
http://blog.colinwilliams.name/clojure/testing/2015/01/26/alternative-clojure-dot-test-integration-with-test-dot-check.html
I'm only getting the second kind of output, instead of both (or just the first)#2019-01-2423:19gfredericksmaybe this commit removed it by accident? https://github.com/gfredericks/test.chuck/commit/b45fb4db83f66#2019-01-2423:19gfredericksit's probably easy to interpret the output as accidentally oververbose#2019-01-2423:25gfrederickslooking more closely, it's possible that the addition of the not-falsey-or-exception?
call meant that the output originally intended wasn't happening at that point, so it really did look redundant#2019-01-2423:26gfredericksI think a PR that adds it back in would be okay, but a PR that adds custom reporting so that there's only one FAIL
line printed instead of two would be even better, probably#2019-01-2512:25aisamuGreat, thanks for the guidance!#2019-01-2713:16dangercoderDo you guys use any external stateful things like a database (Redis) or do you create simulators for all of it? Lets say you want to create property based tests for a sequence of function calls that uses Redis.#2019-01-2713:18dangercoderI was thinking of just having a redis-test-instance and then I will "Clean" it after all tests are done running.#2019-01-2713:43gfredericksyeah you'd want to effectively reset the database as part of your trial#2019-01-2713:44gfredericksI've certainly done that before
you can find bugs in your database that way 🙂#2019-01-2713:44gfredericksbut having to reset something in a different process can definitely lead to fairly slow tests, so you have to have some tolerance for that#2019-01-2716:58hiredmana year (maybe two?) ago we switched out the redis library we used at work, and I took all the operations we did using an old redis library and defined them as a protocol, then implemented that protocol on a concurrent hashmap and on the new redis library, then I wrote the thunderdome, which would take a generated list of redis operations and two things that satisfied the redis protocol and run those operations against them comparing the results.#2019-01-2717:00hiredmanhttps://github.com/advancedtelematic/quickcheck-state-machine is super interesting#2019-01-2717:39gfredericks@hiredman did you find bugs in redis?#2019-01-2718:03hiredmanI did not, I wasn't really exercising redis, the kind of api layer I pulled out, we aren't (weren't?) using redis all that much, but I wanted to make sure the operations behaved the same when moving from a higher level library (lots of built in serialization magic) to a lower level library (doing all the serialization ourselves). I think the protocol I ended up defining to encapsulate our redis use was maybe 4 functions#2019-01-2818:10wilkerluciohello people, I just like to share something that I done and found cool, its about how to setup complex configurable generators, in my case I did a generator setup for building random EQL queries (its like the datomic pull syntax), the way I done is by sending to generators thenselves a map environment so the user could replace parts of it when is needed, I posted some docs about it here https://github.com/edn-query-language/eql#generation. @gfredericks I would love to know what you think about this approach, its you see as a good pattern or maybe you have a better way to do it, thanks all!#2019-01-2818:30gfredericks@wilkerlucio what's the arg to those functions that's being ignored?#2019-01-2818:31wilkerlucio@gfredericks you mean in the default generators? if the arg is ignored it means that generator is not relying on anything that can be customised, usually the leaves in the process, makes sense?#2019-01-2818:32gfredericksin the examples, you customize things with (fn [_] ...)
#2019-01-2818:32wilkerlucioah, take a look this: https://github.com/edn-query-language/eql/blob/master/src/edn_query_language/core.cljc#L10-L121#2019-01-2818:32wilkerluciohere you can see some usages of the params#2019-01-2818:33wilkerlucioin case you wanted to re-use something on the map, you could pull it from that argument#2019-01-2818:33gfredericksah I see#2019-01-2818:34gfredericksI think you could use (constantly ...)
instead of (fn [_] ...)
, which would be "more performant" in a maybe-doesn't-matter sense#2019-01-2818:34gfredericksdepends on if the function gets called multiple times or not#2019-01-2818:34gfrederickswhich I can't tell at a glance#2019-01-2818:35wilkerluciointeresting, I didn't knew constantly was faster, but really this is only used to build the generators, which is a rather quick operation compared to how much takes to run (and you can cache the built generator if needed to)#2019-01-2818:35gfredericksbut the general idea of passing a map of generator options down a whole tree of generators is a really good one I think, and one I've thought about for test.check#2019-01-2818:35wilkerlucioyeah, I also found that to be a good way to control for depth, it worked quite well#2019-01-2818:35gfredericksconstantly just means that the return value gets created once, compared to a fn
where it gets created every time#2019-01-2818:37wilkerlucioah, makes sense, so the generator will be only instantiated once per process#2019-01-2818:38wilkerlucioat same time it would be always instantiated, even if the user doesn't use anything, seems doesn't matter much in the end :man-shrugging:#2019-01-2818:45gfredericksyep#2019-02-0210:04cheatexHi. I try to write a test and stuck with generator/bind semantics. Say I want to write a test map's get (my actual goal is more complex, this one just a good showcase).#2019-02-0210:04cheatexHere is a generator for keys
(defn keys-from [map]
(gen/elements (keys map)))
#2019-02-0210:05cheatexand i can use it like this
(gen/sample (keys-from (gen/generate (gen/map gen/keyword gen/nat))))
#2019-02-0210:06cheatexNow I can write a property
(def prop-test
(prop/for-all [map (gen/map gen/keyword gen/nat)
key (keys-from map)]
(get map key)))
#2019-02-0210:06cheatexBut checking it fails, claiming map
isn't a seq#2019-02-0210:12cheatexMore precisely: https://gist.github.com/CheatEx/91659e49e5f6964aaf8a7a74529af287#2019-02-0210:14cheatexThis from (defspec ..), (tc/quick-check 1 prop-test) yields completely different failure.#2019-02-0210:27cheatexMy another attempt
(defn keys-from-gen [map-gen]
(gen/let [map (gen/such-that (complement empty?) map-gen)]
(gen/elements (keys map))))
(gen/sample (keys-from-gen (gen/map gen/keyword gen/nat))) ; works well
(def prop-test-gen
(prop/for-all [map (gen/map gen/keyword gen/nat)
key (keys-from-gen map)]
(get map key)))
(defspec map-gen-test 2 prop-test-gen)
fails with
Uncaught Error: Assert failed: Second arg to such-that must be a generator
#2019-02-0210:28cheatexSo symbols in for-all
bound to neither real values nor generators. I am totally confused. How do I write such property?#2019-02-0215:04gfredericks@cheatex the confusion comes from the fact that the binding names in prop/for-all
aren't visible at all to the expressions for subsequent bindings#2019-02-0215:04gfredericks(i.e., they're all done "in parallel")#2019-02-0215:05gfredericksnormally you'd get whatever error you normally get when you use a name that's not visible, but in this case you're getting more confusing errors because you're happening to use the name map
, which already means cljs.core/map
#2019-02-0215:05gfredericksso you're passing the map
function to get
#2019-02-0215:06gfredericksprop/for-all
doesn't have a way to get the bind
semantics, so you have to do this with an additional generator to help you compose things#2019-02-0215:06gfredericks(def gen-map-and-key (gen/let [m (gen/map gen/keyword gen/nat), k (keys-from-gen m)] {:map m :key k}))
#2019-02-0215:17cheatexThanks for the explanation.#2019-02-0215:19cheatexI've made this kind of generator for my problem, but i'm not perfectly happy with it. It basically needs to generate a huge data structure to create a key for it... and than throw it away and create a new one just to make another key.#2019-02-0215:20gfredericksFor each trial, you mean?#2019-02-0215:20cheatexIs there a way to make few keys out of single generated map?#2019-02-0215:20cheatexYes#2019-02-0215:21cheatexProbably, not sure of precise meaning of "trial" in test.check 🙂#2019-02-0215:23cheatexI've had idea of generating [map [&keys]] but that would make properties to be a huge and
and hard to find exact failing key.#2019-02-0215:24gfredericksyou don't have to spell out the and
, you could use every?
#2019-02-0215:24gfrederickse.g., you could check every key#2019-02-0215:24gfredericksand as long as the failing map entry is independent of the others, it should shrink to a singleton map#2019-02-0215:28cheatexThey won't be independent in my case.#2019-02-0215:28gfrederickswill it be somewhat independent? like should shrinking remove most of the entries?#2019-02-0215:30gfredericksif that's a big problem, you could use the subsequence
generator from this library https://github.com/gfredericks/test.chuck#2019-02-0215:30gfredericksto get a subset of the keys#2019-02-0215:30gfredericksand then that would shrink to a subset with only one failing key#2019-02-0215:36cheatexLooks like it could do the trick#2019-02-0215:40cheatexBut generally. Is it impossible to use some generated value to set up other generator, use both value and generator in a property and than go to new one?#2019-02-0215:41gfredericksI'm not sure what you mean by "and than than go to new one"#2019-02-0215:42cheatexI mean new value and derived generator#2019-02-0215:44cheatexSo in my example when I run property check with size 50 it (just sample numbers to show magnitudes) generates 5 maps and 10 keys for every map.#2019-02-0215:45cheatexe.g. number of calls to first generator << than number of calls to derived#2019-02-0215:46gfredericksthere's a lot of ways to control the size or complexity of a generator#2019-02-0215:46gfredericksit might help to look at https://github.com/clojure/test.check/blob/master/doc/growth-and-shrinking.md#2019-02-0215:47cheatexI've seen it...#2019-02-0215:47cheatexProbably I need to work out better example and problem statement.#2019-02-0215:48cheatexThank you anyway!#2019-02-0420:54dtsiedelIs there some sort of workaround to test private functions?#2019-02-0420:55gfredericks@dts.siedel #'
is the general workaround for that, not specific to test.check#2019-02-0420:59dtsiedelAh I was afraid of that, it is pretty hacky in my eyes. Thanks for quick response @gfredericks#2019-02-0820:21nwjsmithIf I upgrade test.check from 0.9.0 to 0.10.0, will the same seed still produce the same test values?#2019-02-0820:22nwjsmith(not specifically thinking about those versions, just checking my gut feeling. My gut feeling is that they won’t stay the same across test.check versions)#2019-02-0821:51gfredericks@nwjsmith that gut feeling is right, there's no attempt to preserve that, since sometimes changes involve improvements to a generator's distribution
but in this particular case I think a lot of the lower-level generators will stay the same#2019-02-0821:51gfredericksrecursive-gen is the only one that I know for sure would be different#2019-02-0821:51gfrederickswhich I guess extends to gen/any and friends as well#2019-02-0822:23nwjsmithgood to know, thanks!#2019-02-1115:36Ajayhi, is it possible to make a generator from a custom lazy sequence? For practice, I was trying to use test-check to test FizzBuzz problem - I want a generator that returns multiples of 3 but not of 5 or 15#2019-02-1115:38gfredericksgen/such-that would suffice here for casual use#2019-02-1115:39Ajaybut it gave up when I tried#2019-02-1115:45gfredericksGive it a higher max-tries, like 100#2019-02-1115:45Alex Miller (Clojure team)it’s help to invert your thinking for stuff like this#2019-02-1115:46Alex Miller (Clojure team)generate a number, then use fmap to multiply by 3 so you always generate multiples of 3#2019-02-1115:46Alex Miller (Clojure team)could then use such-that over that to exclude multiples of 5 if needed#2019-02-1115:47AjayOk I had tried generate a number, then use fmap to multiply by 3 so you always generate multiples of 3
but didn't apply such-that to exclude multiples of 5, let me try, thank you for all replies!#2019-02-1115:49gfredericksyeah I forgot to say the multiply part#2019-02-1115:49gfredericksyou could still use such-that without multiplying, it'd just be even slower and you'd probable need an even higher max-tries#2019-02-1115:51gfredericksif you want to be fancy you can use some number theory to generate a good distribution of such numbers w/o filtering#2019-02-1115:58Ajaycould get it to work! thank you!#2019-02-1116:00Alex Miller (Clojure team)you could gen a set of non-5 numbers, then multiply all of them and by 3 to generate only numbers divisible by 3 but not 5 :)#2019-02-1116:01gfredericks@alexmiller that's just moving the filtering to the first generator, right?#2019-02-1116:02Alex Miller (Clojure team)yeah#2019-02-1116:06gfredericksthe fancy thing I have in mind is basically (gen/let [fifteens gen/int, threes (gen/choose 1 4)] (+ (* 15 fifteens) (* 3 threes)))
I think that'd work#2019-02-1809:36borkdudeI wanted to generate arguments for subs
:
(def subs-args-gen
(gen/fmap
(fn [[str [start end]]]
(if end
[str start end]
[str start]))
(gen/bind
(gen/string)
(fn [s]
(gen/tuple (gen/return s)
(gen/bind
(gen/choose 0 (count s))
(fn [start]
(gen/one-of
[(gen/tuple (gen/return start))
(gen/tuple (gen/return start)
(gen/choose start (count s)))]))))))))
(comment
(gen/sample subs-args-gen)
;; (["" 0] ["k" 1 1] ["" 0] ["…_" 0 0] ["" 0] ["¥/" 0] ["µ" 0] ["Þ¦\n" 3] ["" 0] ["\tsÈaíå" 1])
)
It feels a bit verbose, maybe there was a nicer way to do this?#2019-02-1814:21gfredericksthat's a good one#2019-02-1814:21gfredericksI'm trying to think of something that shrinks better than all the gen/bind
s#2019-02-1814:22gfredericksand has a good distribution#2019-02-1814:22gfredericksthose things can trade off with readability/simplicity sometimes#2019-02-1814:25gfrederickshere's a relatively simple approach
(gen/let [[s start end use-end?]
(gen/tuple gen/string gen/nat gen/nat gen/boolean)]
(let [start (min start (count s))
end (min end (count s))
[start end] (sort [start end])]
(cond-> [s start] use-end? (conj end))))
#2019-02-1814:26gfredericksthere might be edge cases there with (count s)
, I'm not sure how subs
works#2019-02-1814:27gfredericksand you might complain that this will be more likely than you'd want to generate args that are equal to (count s)
, and if that bothers you then you could do something like use (gen/choose 0 1000000)
for start
and end
and then scale them down to the size of the string#2019-02-1814:27gfredericks@borkdude ⇑#2019-02-1814:34borkdude@gfredericks cool, I did not know about gen/let
#2019-02-1814:35borkdudethe edge cases for subs I’ve already worked out, 0 <= start <= end <= (count s)#2019-02-1814:35borkdude(yes even end equals to (count s) works)#2019-02-1814:36gfredericksSo I got it right then I think#2019-02-1814:37borkdudeyeah, the distribution might be less nice, like you said#2019-02-1814:52borkdudeI like the boolean idea:
(gen/bind
(gen/tuple (gen/string) (gen/boolean))
(fn [[s end?]]
(if end?
(gen/bind
(gen/choose 0 (count s))
(fn [start]
(gen/tuple
(gen/return s)
(gen/return start)
(gen/choose start (count s)))))
(gen/tuple
(gen/return s)
(gen/choose 0 (count s))))))
#2019-02-1814:54gfredericksthe sorting trick is also good for generating ordered numbers w/o having to do it in stages#2019-02-1814:55borkdudeoh yeah.#2019-02-1814:56gfredericksI probably go overboard on avoiding gen/bind
#2019-02-1814:56gfredericksyou can actually write this fairly simply by using multiple clauses in gen/let
#2019-02-1814:57gfredericks(which expands to gen/bind
)#2019-02-1814:58gfredericks(gen/let [s gen-string, start (gen/choose 0 (count s)), end (gen/choose start (count s)), use-end? gen/boolean] (cond-> [s start] use-end? (conj end)))
#2019-02-1814:58borkdudeif gen/tuple accepted nil, then you could write:
(fn [start]
(gen/tuple
(gen/return s)
(gen/return start)
(when end? (gen/choose start (count s)))))
#2019-02-1814:58gfredericksI just have this instinct to not use gen/bind
if it can be avoided, which is probably not worth the effort in a lot of cases#2019-02-1815:00borkdudeis gen/let a new thing? my REPL says it doesn’t exist#2019-02-1815:00gfredericksnot too new#2019-02-1815:00borkdudeoh wait, gen from test.check, I think I’m using the one from spec#2019-02-1815:00gfredericksmaybe new in 0.9.0?#2019-02-1815:02borkdudeyeah:
(g/let [s g/string
start (gen/choose 0 (count s))
end (gen/choose start (count s))
use-end? (gen/boolean)]
(cond-> [s start] use-end? (conj end)))
works. awesome.#2019-02-1815:02borkdudeso g/let
is the monad-ish thing in test.check#2019-02-1815:02gfredericksyeah; it's just sugar over gen/fmap
and gen/bind
#2019-02-1815:04borkdudetotally useful sugar#2019-02-1815:09borkdudethis also works and avoids generating end if not needed:
(g/let [s g/string
start (g/choose 0 (count s))
use-end? g/boolean]
(if use-end?
(g/let [end (g/choose start (count s))]
[s start end])
[s start]))
#2019-02-1815:10gfredericksYep#2019-02-2109:33borkdudeI get an error with let in clojurescript:
$ clj -A:test -Sdeps '{:deps {org.clojure/test.check {:mvn/version "0.10.0-alpha3"}}}' -m cljs.main -re node
ClojureScript 1.10.520
cljs.user=> (require '[clojure.test.check.generators :as g])
nil
cljs.user=> (g/let [n (g/choose 0 10)] n)
WARNING: Use of undeclared Var cljs.user/n at line 1 <cljs repl>
WARNING: Use of undeclared Var cljs.user/n at line 1 <cljs repl>
Execution error (Error) at (<cljs repl>:1).
Assert failed: First arg to gen/let must be a vector of bindings.
#2019-02-2109:52borkdudefixed:
https://dev.clojure.org/jira/browse/TCHECK-154#2019-02-2519:30borkdude@gfredericks responding to this one: https://dev.clojure.org/jira/browse/TCHECK-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=51397#comment-51397
We don’t have to do anything funny. The only thing we should do to make this “testable” is to remove e.g. this line from the ns form:
https://github.com/clojure/test.check/blob/master/src/main/clojure/clojure/test/check/generators.cljc#L16#2019-02-2519:30borkdudeLeaving that line in, will make it untestable in other namespaces as well, since it will be referred to as macro there too then (maybe depending on the order of loading of the namespaces, but that was the behavior I got when testing this in a different namespace with no change to the existing ones).#2019-02-2601:40gfredericksthat is super weird#2019-02-2607:32borkdudeIt seems like a global thing#2019-02-2612:18gfredericksyou're saying that whether or not the generators
namespace wants to include macros from clojure.core
determines whether or not other namespaces can implicitly include macros from generators
?#2019-02-2612:22borkdudeoh sorry, I was wrong there. I was looking for a test namespace that used gen/let
in CLJS, but mistakingly pointed at the wrong line/namespace even. I’ll see if I can find a better example#2019-02-2612:22borkdudethis is the better example:
https://github.com/clojure/test.check/blob/88eebf105fbb01db17ce12a3bff207cc395270b9/src/test/clojure/clojure/test/check/test.cljc#L18#2019-02-2612:23borkdudesorry for the confusion#2019-02-2612:23gfredericksso you're saying removing that line would allow you to write what kind of test?#2019-02-2612:24borkduderemoving the conditional cljs portions of these require clauses in the test namespaces would enable you test if the macros work correctly in CLJS (provided they add a self-require)#2019-02-2612:25borkdudeyou were asking for a way to test this behavior: this is a way to test it. another way would be to write a completely isolated test suite#2019-02-2612:25gfredericksokay I think I get it#2019-02-2612:26gfredericksI suppose that's probably good enough#2019-02-2612:26gfredericksI can imagine somebody removing the line from src
that looks redundant and then seeing the test failure and fixing it by adding :include-macros true
, but that's starting to sound rather less likely#2019-02-2612:27gfredericksthe thing I was imagining was something like (ns clojure.test.check.require-test (:require [clojure.test.check.generators :as gen])) (deftest can-still-use-let-without-include-macros (is (gen/let [x gen/nat] 42)))
#2019-02-2612:28borkdudeyes, but since it doesn’t seem a namespace local but a project-global inference thing, that doesn’t work if you combine it with the existing tests unchanged#2019-02-2612:29borkdudeso adapting the existing tests to remove the :include-macros option would already be the test#2019-02-2612:30gfredericksI don't understand that first statement#2019-02-2612:30gfrederickshow does it not work?#2019-02-2612:31borkdudeok. say you would add (ns clojure.test.check.require-test (:require [clojure.test.check.generators :as gen])) (deftest can-still-use-let-without-include-macros (is (gen/let [x gen/nat] 42)))
. this will already pass if clojure.test.check.generators :as gen
has been required with :include-macros
before.#2019-02-2612:31gfredericksI feel like I'm about to learn something terrible about clojurescript#2019-02-2612:31borkdudein other words: that test already passes right now, without changing anything#2019-02-2612:32gfredericksyou're saying that if namespace A
requires X
and namespace B
requires X
, then what's visible from X
in B
is affected by how A
required it?#2019-02-2612:33borkdudeI’m not an expert in CLJS macro resolution but that’s my understanding of it. If A requires X with include-macros first, then B requires X without include-macros, it will still be able to use the macros.#2019-02-2612:33gfredericksI'm willing to bet fives of dollars that this is not true, because if it's true then I'll be so sad that I won't think to miss my fives of dollars#2019-02-2612:34borkdudewell, it’s easy to try out. just add this test namespace of yours and run the tests.#2019-02-2612:35borkdudeI’m not saying that I want this to be true, just that I saw this behavior when adding a similar test 🙂#2019-02-2612:37borkdudeOn the other hand, I don’t see why this would be problematic. If you don’t add :include-macros true, the worst thing that will happen is that you will get a function instead of a macro resulting in an error (the original problem), so not getting the error as a side effect of some other namespace doing the require with include-macros, is a nice bonus.#2019-02-2612:37borkdude@mfikes or @dnolen could maybe explain how this works in detail#2019-02-2612:39gfredericksokay, I'm running such a test now#2019-02-2612:49gfredericks(ns clojure.test.check.test2
(:require [cljs.test :refer-macros [deftest is]]
[clojure.test.check.generators :as gen]))
(deftest can-use-gen-let-without-include-macros
(is (gen/let [x gen/nat] 42)))
I can't get this to fail even by removing every other test namespace#2019-02-2612:50gfredericksat first I got a compiler warning about x
not being defined, which I thought was a good sign, but haven't been getting that since#2019-02-2612:51borkdudeHave you cleared the target for?#2019-02-2612:51borkdudeDir#2019-02-2612:51borkdude(Damn spelling corrector)#2019-02-2612:51gfredericksyeah#2019-02-2612:52gfredericksand there's nothing else in the project that's requiring gen with :include-macros true
#2019-02-2612:53gfredericksI just pushed this to the branch tmp-f5667927-abed-48d9-b5a8-7c7f0295b001
if you want to take a look#2019-02-2612:56borkdudeI’ll check it out, be back in a couple of hours#2019-02-2614:17borkdude@gfredericks
Change your test ns to something like:
(ns clojure.test.check.test2
(:require [cljs.test :as t :refer-macros [deftest is]]
[clojure.test.check.generators :as gen]))
(deftest can-use-gen-let-without-include-macros
(is (gen/let [x gen/nat]
(number? x))))
(t/run-tests)
Then I’m able to reproduce the error with:
$ clj -Sdeps '{:deps {org.clojure/clojurescript {:mvn/version "RELEASE"}}}' -m cljs.main -re node -i src/test/clojure/clojure/test/check/test2.cljs
WARNING: Use of undeclared Var clojure.test.check.test2/x at line 6 test2.cljs
WARNING: Use of undeclared Var clojure.test.check.test2/x at line 7 test2.cljs
Testing clojure.test.check.test2
ERROR in (can-use-gen-let-without-include-macros) (Error:NaN:NaN)
expected: (gen/let [x gen/nat] (number? x))
actual: #object[Error Error: Assert failed: First arg to gen/let must be a vector of bindings.
(vector? bindings)]
Ran 1 tests containing 1 assertions.
0 failures, 1 errors.
#2019-02-2614:19borkdudeWe could just write a script for this to isolate it from the rest of the tests (if you still want this tested, I think it’s pretty rare that anyone does this in a CLJS lib)#2019-02-2614:21gfredericks@borkdude But is it true that the test then passes if you revert the changes to the other test namespaces?#2019-02-2614:22borkdudenot in this case, since I only load that one namespace here. I could test it by also loading those first#2019-02-2614:29borkdude@gfredericks This branch: https://github.com/borkdude/test.check/tree/tmp-f5667927-abed-48d9-b5a8-7c7f0295b001
$ clj -Sdeps '{:deps {org.clojure/clojurescript {:mvn/version "RELEASE"}}}' -m cljs.main -re node -i src/test/clojure/clojure/test/check/test2.cljs
WARNING: Use of undeclared Var clojure.test.check.test2/x at line 6 test2.cljs
WARNING: Use of undeclared Var clojure.test.check.test2/x at line 7 test2.cljs
Testing clojure.test.check.test2
ERROR in (can-use-gen-let-without-include-macros) (Error:NaN:NaN)
expected: (gen/let [x gen/nat] (number? x))
actual: #object[Error Error: Assert failed: First arg to gen/let must be a vector of bindings.
(vector? bindings)]
Ran 1 tests containing 1 assertions.
0 failures, 1 errors.
$ clj -Sdeps '{:deps {org.clojure/clojurescript {:mvn/version "RELEASE"}}}' -m cljs.main -re node -i src/test/clojure/clojure/test/check/test.cljc -i src/test/clojure/clojure/test/check/test2.cljs
WARNING: var: clojure.test.check.generators/gen-raw-long is not public at line 158 test.cljc
WARNING: var: clojure.test.check.generators/gen-raw-long is not public at line 158 test.cljc
Testing clojure.test.check.test2
Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
#2019-02-2614:29borkdudeI guess I’m pretty close to winning that 5 dollars…#2019-02-2614:31borkdudeIt seems that once the CLJS compiler knows something is a macro, it cannot unknow it?#2019-02-2614:36borkdudemaybe that is what happens here (although that’s just me guessing): https://github.com/clojure/clojurescript/blob/master/src/main/clojure/cljs/analyzer.cljc#L629#2019-02-2614:44borkdudeso when you add a self-refer to a macro in a .cljc namespace, you effectively tell the CLJS compiler: I want to use this var as macro in CLJS. Any subsequent require to that namespace does not have to help CLJS with that anymore by adding include-macros true or refer-macros, it already knows.#2019-02-2614:56gfredericksI just asked in #cljs-dev about this#2019-02-2615:04borkdudeok, so it’s confirmed that this behavior exists#2019-02-2615:09borkdudewhat about making a standalone script that just does this:
clojure -Sdeps '{:deps {org.clojure/clojurescript {:mvn/version "RELEASE"}}}' -m cljs.main -re node -i src/test/clojure/clojure/test/check/test2.cljs
That wouldn’t interfere with any other namespaces requiring these macros#2019-02-2615:12borkdudetest2 would be called test_macros.cljs then#2019-02-2616:04gfredericksBlergh#2019-02-2616:05gfredericksHow about we just remove all the include-macro clauses from all src/test nses, except for the self ones#2019-02-2616:10borkdudeyeah, that was my idea when I suggested that the tests already test this if they don’t add include-macros themselves#2019-02-2616:10borkdudebut we could not reproduce this with lein cljsbuild#2019-02-2616:10borkdudemaybe that build tool has some additional magic#2019-02-2616:11borkdudebut I’m fine with your proposal#2019-02-2616:20borkdudeI’m looking at the cljsbuild code, but it seems there is some magic around macros in there:
https://github.com/emezeske/lein-cljsbuild/blob/master/support/src/cljsbuild/crossover.clj#2019-02-2616:22borkdudeI can see why the CLJS authors ask for repros not using these tools 🙂#2019-02-2616:53gfredericksyeah I think a global policy of "no other namespace should need to require-macros from a t.c namespace because they all do self-requiring" is good enough#2019-02-2616:54gfredericksso the additions to your patch would be A) deleting redundant :include-macros
B) doing this trick for the prop
namespace too#2019-02-2617:00borkdudeok, will do#2019-02-2617:28borkdudealso in clojure.test.check.clojure-test
for defspec
I presume#2019-02-2617:41gfredericks👍 yep#2019-02-2617:41gfredericksthanks for tracking everything down#2019-02-2617:44borkdudeNew patch uploaded#2019-02-2617:47gfredericksokay, I should get to it sometime this week#2019-02-2617:48borkdudethanks 🙂#2019-03-0315:40borkdudeThe last stable release of test.check has been 12 NOV 2015. What are the alpha releases waiting for to become stable?#2019-03-0315:42gfredericksa set of milestones I was asymptotically approaching for a while; most of them I gave up on, so the only reason I haven't cut a release is because of the things that seem too half-baked to release#2019-03-0315:42gfredericksI think the tickets tagged 0.10.0 reflected what I hadn't yet given up on#2019-03-0315:43gfrederickshttps://dev.clojure.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+10470+AND+labels+%3D+%22release-0.10.0%22#2019-03-0315:43gfredericksnow that I look, it seems like it's just a documentation ticket#2019-03-0315:49gfredericksthe big thing I wanted to do was write some guides, but that was apparently too much work, and arguably isn't related to a release#2019-03-0315:51gfredericksI think I also wanted to sync up with alex w.r.t. spec, and see if there was anything that'd be super useful to change before a release#2019-03-0315:51gfredericksso that there aren't future compatibility issues#2019-03-0315:52borkdudedid the lazy loading of generators in spec require changes to tc?#2019-03-0315:52gfredericksno, but there was a change to gen/such-that
for spec#2019-03-0315:53borkdudeis spec itself still using 0.9.0?#2019-03-0315:53gfredericksI guess? but if you depend on 0.10.0 yourself, that's what it'll be using#2019-03-0315:53gfredericksso people will conceivably be combining arbitrary pairs of versions#2019-03-0315:54borkdudeso 0.9.0 already had a change to gen/such-that for spec in November 2015?#2019-03-0315:57borkdudeIt seems not.#2019-03-0315:58gfredericksyeah, I didn't mean to imply that#2019-03-0316:00gfredericksI'll put out another alpha release this week for the cljs changes, and then I will plan on talking to alex in a few weeks when my free time clears up#2019-03-0316:00borkdudeyeah, no hurry, just wanted to know the current state#2019-03-3015:24caio@brewmasterv I think edge cases are pretty common within cycles. eg: repeating something until you get a memory error, rounding errors, etc#2019-04-1514:37defndainesRecently learned that prn-str
and edn/read-string
are not symmetric, thanks to playing around with test.check.
(defspec prop-symmetric
;; NOTE This actually fails. prn-str and read-string are not symmetric.
100
(prop/for-all [s gen/any]
(let [encoded (prn-str s)]
(and (string? encoded)
(= s (edn/read-string encoded))))))
#2019-04-1514:38gfredericksthis sounds like a good trivia question#2019-04-1514:38defndaines(Note, there isn’t any official documentation that they are supposed to be, if you exclude the comment here https://clojuredocs.org/clojure.edn/read-string#example-542dc112e4b05f4d257a2993 from phreed)#2019-04-1514:39gfredericksthe first thing I can think of is things like (keyword " ")
#2019-04-1514:39gfredericksI can't remember if gen/any
tries to generate those though; I think it doesn't#2019-04-1514:39gfredericksI'm 99.7% sure it doesn't after thinking about it for two more seconds#2019-04-1514:40gfredericks(symbol "true")
would be problematic; I don't think it avoids that, but it's also probably highly unlikely to generate it by chance#2019-04-1514:40defndainesI know that a Double/NAN
is one thing that isn’t symmetric.#2019-04-1514:40gfredericksI'm sure there's something much more basic#2019-04-1514:41gfredericksoh, well that probably reads back in as the correct object but =
won't be true#2019-04-1514:41gfredericksthat says more about =
than it does about prn-str
and edn/read-string
#2019-04-1514:41defndainesAh, true. Would have to use a specialized =
to see what else pops up.#2019-04-1514:41gfredericks(symbol "nil")
is easier to generate than (symbol "true")
I guess#2019-04-1514:43gfredericksthere's a lot more stuff that would come up if gen/any
was more ambitious#2019-04-1514:43gfrederickse.g., queues, sorted sets, sorted maps, ...#2019-04-2921:50kennyIs there any documentation on what each of the keys returned from quick-check mean?#2019-04-2922:10gfredericksThe docstring mentions some of them, doesn't it?#2019-04-2922:41kennyThe API docs don't have it https://clojure.github.io/test.check/clojure.test.check.html. Looks like it's been added in the source in master. It'd be useful to understand when certain attributes are a specific type. i.e. when :result
is false
or an exception
. The :result-data
map is doc'ed as {...}
. Is there a doc on that one elsewhere?#2019-04-3000:15gfredericksyeah there's a lot of stuff there; the main reason it's confusing is because I was trying to add more useful and easy-to-use information without breaking stuff relying on the old keys#2019-04-3000:16gfredericks:result
is the worst part, since it has the truthy-or-exception thing going on; that's why I added the :pass?
key, so you can tell with a straightforward check whether the test passed or not
:result-data
is customizable, if you write a property that returns something that satsifies the clojure.test.check.results/Result
protocol#2019-04-3000:17gfredericksthe only specific use of it that's baked in is when a normal property throws an exception, the exception is added as a key in the result-data#2019-04-3000:17gfrederickswhich might currently have the unfortunate side effect of causing exceptions to be printed twice 😞#2019-04-3000:18gfredericks@kenny ^^#2019-04-3000:20gfredericksI can see both parts are fairly undocumented 😕#2019-04-3000:20gfredericksI tried to spec things at some point I think; maybe that'd be a good way to start#2019-04-3000:32gfredericksExcept they probably haven't added docstrings to specs yet have they#2019-04-3000:35Alex Miller (Clojure team)not yet#2019-05-0811:14benI have a really basic question: If I want to test multiple assertions in a property test, what is the most idiomatic way to do it?
Is it just to wrap each assertion in an all
? Or can I leave them as separate forms?#2019-05-0811:20benI suppose I could wrap each in a clojure.test/is
but that seems definitely wrong#2019-05-0811:24gfredericks@ben606 are you using defspec
?#2019-05-0812:11benyes#2019-05-0812:15gfredericksthe main two options are
1) Multiple defspecs
2) an and
inside a single defspec#2019-05-0812:15gfredericksthere's tradeoffs#2019-05-0812:15gfredericksthe and
could be made nicer if somebody wrote a version of and
that returned result-data about which branch failed#2019-05-0812:23benOkay, thanks @gfredericks. I guess this isn’t good practice, but I am testing things that depend on each other (db write and read).
Seems like the Right Way would be to use multiple defspecs with some additonal setup/teardown?#2019-05-0822:06gfredericks@ben606 if you're doing something like "generate data, write it to the db, read it back, check that it's equivalent", then you'd want to do all of that in one defspec#2019-05-0909:40benRight, so that’s some of them. I also have a couple with
• read from the db (inserted in test migration)
• call a fn to modify the data
• read from the db again
• check that it has been modified as expected
Do you think this should follow the same pattern, @gfredericks?
Thanks#2019-05-0910:44gfredericks@ben606 that's what I would do, yeah; do you reset the db from within the defspec also? I think you'd want to#2019-05-0910:46benYeah, it all takes place within a transaction, so none of the changes get persisted#2019-05-0910:47gfredericksthat sounds good; the main thing dictating granularity is what things are dependent on what generated data#2019-05-0911:02benmakes sense. Thank you#2019-05-2716:16Ho0manHi, everyone
I was wondering how can I use clojure.test/use-fixture
with test-check’s defspec
?#2022-07-2805:39ossosoHow would I emulate the use of fixtures for each trial? Macro/higher order function?#2022-07-2900:26colinkahnHigher order function is what I've used before.#2022-07-2908:59ossosoI ended up writing the following macro:
(defmacro db-fixture [& body]
`(with-open [node# (db/start-node {})]
(binding [*node* node#]
#2019-05-2716:28gfredericks@ho0man is your issue that you want the fixture to run around each trial?#2019-05-2716:29Ho0manNope, the whole test#2019-05-2716:30Ho0manBut separately for different tests in a single namespace#2019-05-2716:30gfredericksI think that should just work#2019-05-2716:30gfredericksDoes it not?#2019-05-2716:37Ho0man:))
Yeah
It does
Thanks @gfredericks
(Sorry to bother you)#2022-07-2805:39ossosoHow would I emulate the use of fixtures for each trial? Macro/higher order function?#2019-06-1006:48ShimaHi
I want to rerun the test with the same seed field if the test had been failed, so I need to somehow have access to seed inside of the test defspec.
Is there any way to do this?#2019-06-1010:30gfredericks@tajoddin.shima defspec should print the seed when there's a failure, and I believe you can call the test function that defspec creates with a seed, like:
(defspec some-test ...)
(some-test 100 {:seed 42})
#2019-06-1012:33ShimaYes but I want to run the test with "lein test" and my test scenario needs to rerun the wrong tests with the same seed again. so I want to get the seed exactly inside of test/defspec, Is this even possible?
Thanks for your answer @gfredericks#2019-06-1012:34gfredericks@tajoddin.shima do you mean that you want to set the seed as an arg to lein test
somehow? or to have it hard-coded in your test-file somewhere?#2019-06-1012:41Shimanone of them :)) I need something like this : @gfredericks#2019-06-1012:43gfredericksI don't understand that; it looks like an infinite recursive loop#2019-06-1012:43gfredericksLike, if the test fails, you just want to run it again?#2019-06-1012:46ShimaI have some randomness out of test-check context in my generator
I save the number of recursion in an atom and control this.
I just want to make sure that when the test failed is because of randomness inside of my generators#2019-06-1012:47gfrederickswould it work to run the individual trial multiple times instead of the whole run?#2019-06-1012:48gfrederickssomething like
(prop/for-all [test-suite (generator)]
(loop [failures-left 5]
(or (actual-test test-suite)
(and (pos? failures-left) (recur (dec failures-left))))))
#2019-06-1012:52Shimaexactly
just instead of test-suite inside the loop, each time generate the new suite with the same seed#2019-06-1012:54Shimawith generating the suite with same seed I want to bound the randomness to something that comes from out of test-check context#2019-06-1013:01gfredericksThis extra randomness is used in the generator?#2019-06-1013:03Shimayes#2019-06-1013:04Shimawhat about this? It would be enough that I have access to the seed inside of the defspec to save in DB and don't need to regenerate the test-suite in it.#2019-06-1013:05gfredericksI wonder whether you need the extra randomness at all
That's the nonstandard part
What is it accomplishing?#2019-06-1119:07ShimaYes, you're right. There is no need for this extra randomness.
but I still wondering is there any way to get seed of each run? maybe in use-fixture function ?( for example to save in DB)
sorry for latency @gfredericks#2019-06-1119:08gfredericksthe seed is printed after every defspec
run#2019-06-1119:08gfredericksif you want something more custom, you can always write your own code to call clojure.test.check/quickcheck
#2019-06-1119:08gfredericksdefspec is just a macro that does some stuff for you#2019-07-0115:32gfrederickstest.check version 0.10.0-RC1
was released yesterday
Details: https://github.com/clojure/test.check/blob/master/CHANGELOG.markdown#2019-08-0218:06kennyIs there a way to create a generator from a list of generators such that when it generates, it returns the same list with each element in the list generated?#2019-08-0218:08kennySomething like this:
(gen/let [my-things (map (fn [thing]
(get-gen-for thing))
my-things)])
#2019-08-0218:10gfredericks@kenny is this different from (apply gen/tuple ...)
?#2019-08-0218:11kennyWow, so obvious haha. That's exactly it. Thanks 🙂#2019-08-0218:28Alex Miller (Clojure team)or just s/gen of an s/tuple if you're using spec#2019-08-1218:55gfrederickstest.check version 0.10.0
has been released; no differences from 0.10.0-RC1
#2019-08-2019:27kennyIs there a good way to provide some initialization for a property when used with the clojure.test integration? For example, I have this code:
(defspec my-gen-test
100
(kafka/with-driver [d (make-driver {})]
(prop/for-all [record (my-record-gen)]
(kafka/pipe-input d record))))
It needs to initialize a Kafka test driver and then run the property test. with-driver
is basically the same as with-open
(i.e. it will close a resource after the form's body completes). The problem I'm running into is the property check doesn't appear to block until it's completed so the driver opens and then closes pretty quickly. Is there a way to work around this? Currently thinking to just make my own defspec
that can handle this sort of thing.#2019-08-2019:32kennyI could write my own quick-check
fn like this:
(defn quick-check
[num-tests property & opts]
(tc-test/assert-check
(apply tc/quick-check num-tests property opts)))
#2019-08-2019:34gfredericks@kenny you it to be opened/closed just once for the whole test run, across many trials?#2019-08-2019:34kennyYes. Open at the beginning, close at the end.#2019-08-2019:34kennyIt should run through all the properties using the same driver.#2019-08-2019:35gfredericksIf you did it in a fixture and bound a dynamic var, would that be good enough?#2019-08-2019:36kennyYes but fixtures are icky. They either run with each deftest
or only once. Neither of those choices are a good fit.#2019-08-2019:37gfredericksOh right#2019-08-2019:38gfredericksYeah I don't think there are any super clean options#2019-08-2019:41kennyThat quick-check wrapper fn should work for now#2019-08-2119:37johanatan@gfredericks any idea why gen'/string-from-regex
would be undefined when i have added test.chuck as a dep and required the chuck.generators namespace as gen'
?#2019-08-2119:37johanatantried both 0.2.8 (which used to work for me) and 0.2.10#2019-08-2119:37johanatanand this is with org.clojure/test.check v0.10.0#2019-08-2119:37johanatan[using deps.edn]#2019-08-2119:37gfredericksNot at all#2019-08-2119:37gfredericksIt's a regular ole function#2019-08-2119:37johanatanhmm 🙂#2019-08-2119:38johanatanwow, must have been some weird fluke. restarting figwheel a few times resolved it#2019-08-2119:38gfredericksWait#2019-08-2119:39gfredericksIt's not in cljs#2019-08-2119:39gfredericksIs that what you're using?#2019-08-2119:39johanatanoh, yea this is cljs#2019-08-2119:39gfredericksWell now I don't know why it works 😂#2019-08-2119:39johanatanhaha, yea same here#2019-08-2119:40gfredericksThere's s longstanding PR for porting it, but I don't think it's complete#2019-08-2119:40gfredericksIt's very nontrivial, because different regex engines#2019-08-2119:40johanatanhmm ok. right#2019-08-2119:58johanatanbtw, should everything in clojure.test.check be in the clojure.spec.alpha namespace now?#2019-08-2119:58johanatanor should we continue using the former until v2 comes out?#2019-08-2119:59johanatanasking because gen/let
doesn't appear to be there#2019-08-2120:07kenny@johanatan Several functions in test.check gen ns are not in the clojure.spec gen ns.#2019-08-2120:09Alex Miller (Clojure team)gen/let is a macro and only the primary functions are being dynaloaded#2019-08-2120:09Alex Miller (Clojure team)it's not impossible to do so but not high on our list of things to do, so you should just continue using the one from test.check if you need it#2019-08-2120:37johanatanok, thx#2019-08-2202:53kenny@alexmiller Adding gen/let to the spec gen ns would be quite helpful. It basically a copy paste (unless there's some smarter way to dynamically expose a macro). We use gen/let all over the place via a helper ns. We cannot require test.check because we don't want it on the classpath at runtime.#2019-08-2203:20Alex Miller (Clojure team)because you need macros at compile time, I don't think you can dynamically include it the way we dynamically load the other functions from gen#2019-08-2203:20Alex Miller (Clojure team)there is actually a ticket for this though with a patch, but I don't remember if it just recreates the macro or what#2019-11-0309:17leonoeldoes the empty generator exist ?#2019-11-0316:47andy.fingerhutmeaning, fails to generate anything?#2019-11-0316:47leonoelyes#2019-11-0316:48andy.fingerhutWhat would you expect it to do if you tried to cause it to generate a value? Throw an exception?#2019-11-0316:48gfredericksCan make it with such-that, if so#2019-11-0316:49andy.fingerhutCan you describe a situation where it would be useful to have a generator that doesn't generate anything? I'm lacking imagination there.#2019-11-0316:51leonoelI'm working on regular languages, and I'm willing to construct a generator for words of a given language#2019-11-0316:52leonoelso, does it makes sense to have a generator for words of the empty language ?#2019-11-0316:52gfredericksSure#2019-11-0316:52andy.fingerhutWould it just throw an exception when you tried to generate such a word?#2019-11-0316:53leonoelno, because it's not error#2019-11-0316:53gfredericksI'd hope so#2019-11-0316:53andy.fingerhutIf it returns a value, that would be interpreted as a generated valid value, wouldn't it?#2019-11-0316:53gfredericksWhat would it generate then?#2019-11-0316:54leonoelit would generate nothing#2019-11-0316:54andy.fingerhutWhat would a call to samples
return? (edit: I meant a call to gen/sample
)#2019-11-0316:54leonoelan empty list#2019-11-0316:55gfredericksThere's no such thing as generating nothing#2019-11-0316:55gfredericksSimilar to there's no such thing as a function that returns nothing#2019-11-0316:57andy.fingerhute.g. a generator that returns nil
is thereby indicating: nil
is a value that satisfies the desired predicate. Not that there is no valid value satisfying the predicate.#2019-11-0317:01gfredericksI guess a downside of using a throwing generator for the empty language is that test.check wouldn't let you do things like create a disjunction using gen/one-of like you'd want to#2019-11-0317:02gfredericksThat would just give you a generator that threw half the time, which is obviously wrong#2019-11-0317:02leonoelI don't fully understand why it doesn't make sense, and why e.g (g/one-of [])
is forbidden.#2019-11-0317:02leonoelthis is really a noob question, I'm not really familiar with this topic#2019-11-0317:02gfredericksThis might be a feature that could be added, but it would have to pervade a lot of the combinators#2019-11-0317:03gfredericksLike one-of would have to filter out empty generators from its input, for example#2019-11-0317:04gfredericksfmap and bind would have to short circuit#2019-11-0317:04gfredericksAnd on and on#2019-11-0317:04andy.fingerhutWell, if you used such-that with an arbitrary function predicate that always returned false, test.check would have no general way to identify all possible generate-nothing generators. It could only recognize ones that you made obvious were generating-nothing generators.#2019-11-0317:05gfredericksYes#2019-11-0317:05gfredericksWhich is why the pervasive support is important#2019-11-0317:05gfredericksTo track emptiness#2019-11-0317:06andy.fingerhutSo one-of would have to try invoking empty generators, and have a way of getting back "I didn't generate anything", wouldn't it? i.e. in order to enable arbitrary empty gens.#2019-11-0317:06gfredericksI guess you could do it that way#2019-11-0317:06gfredericksAnd this would still be a generator that throws when you try to generate, so if that can't be useful to you, then even this idea wouldn't help#2019-11-0317:06gfredericksI guess you could do it that way#2019-11-0317:07gfredericksNot sure how that would affect the usability of such-that#2019-11-0317:07andy.fingerhutNot recommending that approach, necessarily, but it seems like one approach, the other being to say something like "empty gens using such-that with arbitrary Clojure functions as predicates may throw exceptions, or some other behavior you might not like, if you try to use them"#2019-11-0317:07gfredericksThat's a pretty deep breaking change, though, so I doubt it would be added as a feature#2019-11-0317:12leonoelis (g/such-that (constantly false) generator)
a terrible idea ?#2019-11-0317:13gfredericksNo, but it throws, and you said you didn't want that#2019-11-0317:13gfredericksAlso it doesn't compose like I suspect you'd want for regular languages#2019-11-0317:14gfredericksBut if you use it without composing, it'd work fine#2019-11-0317:14leonoelthat matches my intuition#2019-11-0317:15leonoelI ended up working with nil
able generators, and it's not that clumsy so I'll keep it that way#2019-11-0317:16gfredericksMeaning a generator that generates nil, or a context where you either have a generator or you have nil?#2019-11-0317:16leonoelthe latter#2019-11-0317:16gfredericksSounds fine#2019-11-0317:17leonoelOK thank you for clarification#2019-12-1700:21jvtriguerosHi all, I'm debugging a failing test that uses test.check
's defspec
. When my test fails, I'm able to obtain the seed and I can see the value that caused the test to fail, however, neither the exact value or the reduced value produce the failure.
I reckon it's the sequence of values that cause the failure Dx
Is there a way that I can output the values so that I can execute them in order and see the failure? The only thing that comes to mind is spit
ing the values inside the for-all
closure.#2019-12-1700:29gfredericksThat would work. This isn't a "supported" scenario because it's assumed that your test passes or fails solely based on the generated values#2019-12-1700:29gfredericksSo if you don't get your test into that form you'll be fighting the system the whole time#2019-12-1700:35jvtriguerosRight. In this case, I'm doing a roundtrip test with a DynamoDBLocal instance so I bet I'm overwritting something causing failure in the tests.#2019-12-1700:48jvtriguerosThanks! That helped, I had an issue in my generator.#2019-12-1701:46gfredericksFor db stuff, I'd generally look for a way to reset the db at the beginning of each trial, or partition the data so it doesn't overlap#2020-01-0416:35bartukahi ppl, how do you usually integrate test.check with deftesting or midje?#2020-01-0416:36bartukaI am slowly building some property-based tests in my code base to learn more about it, but I currently use midje
to my tests and would like to have it more integrated#2020-01-0417:27gfredericksAt worst you can call the quickcheck function and assert that the pass? key (or whatever it's called) is true#2020-01-0417:28gfredericksdefspec is the normal integration point with clojure.test, I have no idea if that's usable with midje#2020-01-0418:42bartukaI had done this macro to help me... it's working but not ideal
(defmacro check-that [desc n property]
`(fact ~desc
(let [check# (tc/quick-check ~n ~property)
passed# (:result check#)]
(when-not passed# (clojure.pprint/pprint check#))
passed# => true)))
#2020-01-0418:43bartukathe message when error occur is not visible in the midje
stack trace.. but I still get it.#2020-01-0418:57gfredericksyou're checking the wrong return key#2020-01-0418:58gfredericksexceptions are exactly the edge case where that doesn't work#2020-01-0418:58gfredericksyou want the :pass?
key: https://github.com/clojure/test.check/blob/master/src/main/clojure/clojure/test/check.cljc#L108#2020-01-0418:59gfredericksif you want exceptions reported the way midje would, you can also do something like
(when (and (not (:pass? check#)) (:result check#))
(throw (:result check#)))
#2020-01-0418:59gfredericksor maybe midje has a less hacky way to do that#2020-01-0419:00bartukaoh.. didn't know!! thanks#2020-01-0419:00gfredericksthe keys are a bit confusingly named for legacy reasons#2020-01-0419:01gfredericksbut they're described in the docstring at least#2020-01-0419:02bartukathe :result key is not a boolean? So when an exception happen the result will be a keyword :exception something like that?#2020-01-0419:03gfredericksno it'll be an actual exception object#2020-01-0419:03gfredericks:pass?
is always a boolean#2020-01-0419:04bartukagot it! thanks for the hint#2020-01-0419:05bartukajust watched your presentation and started to write tests to a utils namespace I have in my current project#2020-01-0419:05bartukainteresting bits already pop out!! some edge cases dealing with dates 😃#2020-01-0419:07bartukaand a performance issue as well. lol. The prod version of the code took 12secs to run 1000 times.. after some refactor it went down to 2sec. And it still works! I used to slow one to validate the fast impl. (Y)#2020-01-0419:07gfredericksoh yeah, I love the before-and-after refactoring check#2020-01-0419:08gfredericks"test.check as incentive to make your code faster" is an angle I hadn't considered before 🙂#2020-01-0419:09gfredericksit's almost the opposite of the TDD people that put high effort into making the tests extremely fast#2020-01-0419:10bartukathat's true. I stumbled in test.check because I am really studying tests as a whole subject and experimenting it in my project here#2020-01-0419:10bartukathis subject has people with very very strong opinions hehehe#2020-01-0419:11bartukasaw some guy arguing that Golang was the standard language for TDD because of the incredible fast feedback you get from your entire suite running in ms#2020-01-0419:11bartukao.O#2020-01-0419:12gfredericks¯\(ツ)/¯#2020-01-0419:13bartukado you have more material about property-based tests to share? I am looking for something more conceptual about how to spot invariants#2020-01-0419:14bartukaI know it is very context based, but some general concepts must be applicable#2020-01-0419:14gfredericksthere's language-agnostic material on that subject#2020-01-0419:14gfredericksor rather, any material on that subject regardless of language would be applicable#2020-01-0419:15gfrederickse.g., a quick trip in the googlemobile suggested: https://fsharpforfunandprofit.com/posts/property-based-testing-2/#2020-01-0419:18bartukacool, thanks! I will keep my search as well o/#2020-01-0419:21Alex Miller (Clojure team)I spent a bit of time trying to talk about this in Clojure Applied, not sure I said anything revolutionary, but perhaps of interest#2020-01-0419:25bartukaI like when ppl suggest a book and I have it hehe. thanks for pointing out @alexmiller Part III on Practices (Y)#2020-01-0419:27Alex Miller (Clojure team)yeah, it's chapter 8#2020-01-0420:10bartukagosh, how can I work next monday o.O
I just got this error from a test:
There is insufficient memory for the Java Runtime Environment to continue.
repl_1 | # Native memory allocation (malloc) failed to allocate 312 bytes for AllocateHeap
repl_1 | # An error report file with more information is saved as:
repl_1 | # /app/hs_err_pid46.log
It is a simple code to add business days taking into account Brazilian holidays to an java.util.Date. But the generator is creating java.util.Date with any values for min,secs and millis. Then, this happened:
(.equals #inst "1980-02-02T00:01:01.000-00:00" #inst "1980-02-02T01:00:01.000-00:00")
#2020-01-0420:30bartukaI edit.. both were the same, year, month and day. But different hour.#2020-01-0420:11bartukait was suppose to be true in the prod code =(#2020-01-0420:11gfredericks... why?#2020-01-0420:12bartukapeople [me probably] was imagining this code as a Date not DateTime inputs#2020-01-0420:12bartukabut some day this extension happened without proper care rsrs#2020-01-0420:13gfredericksLocalDate?#2020-01-0420:13bartukayes#2020-01-0420:14gfredericksso many languages/libraries seem to encourage people to think of datetimes as localdates with more resolution, but that's not at all the case#2020-01-0420:15bartukayeah, first few minutes and a lot of lessons. This year will be nice. haha#2020-01-0420:16bartukaand funny enough, we had an example-based tests.. but the java.util.date passed around was all with T01:00:01.000-00:00
#2020-01-0500:10bartukamy snippet.. found some edge cases with interest-rate computation. I made a gist to show to others later, very compact and sums up everything I learned today. 1) Modeling my problem domain with generators (unfortunately, we have interest rates up to 400% an year for some loans here in Brazil.. i'm not too pessimistic in my model lol), 2) testing invariants [round-trips operations ] https://gist.github.com/wandersoncferreira/183c56da92c4fe388efef05ca51af4fa#2020-01-0500:10bartukathree namespaces all covered. thanks guys o/#2020-01-0712:11bartukaI need to generate a list of maps that have distinct :number key. For example [{:number 1 ..}, {:number 2 ..}] and I need these numbers to be sequential. I tried (gen/vector-distinct-by :number (gen/hash-map :number gen/pos-int ...)
and it does work but not as sequential numbers. I don't know now if I should try to create a generator to replace gen/pos-int
or the vector-distinct-by
. rsrs#2020-01-0713:27Alex Miller (Clojure team)test.check generators are generally for creating random things, not sequential things like that#2020-01-0715:53bartukaI was thinking about that when I got into this problem. But I want to model a situation where does not make sense to have this kind of behavior for example, a list of installments for a loan. They are generally numbered here.#2020-01-0713:29Alex Miller (Clojure team)there may be some way to do it (gen an int that's the size of the range, gen/fmap that with range, gen/bind that with something that makes coll of maps)#2020-01-0713:29Alex Miller (Clojure team)but you're going to be fighting the tide there#2020-01-0713:59gfredericksgen the collection (w/o numbers) and the minimum number separately; then gen/fmap
them both to set all the numbers#2020-01-0714:01gfredericks(gen/fmap (fn [{:keys [list-of-maps min-number]}] (map #(assoc %1 :number %2) list-of-maps (iterate inc min-number))) (gen/tuple gen-list-of-maps-without-number gen/large-integer))
#2020-01-0716:04bartuka@gfredericks thanks for the guidelines... I did some adaptations to work as I intended.
(def gen-list-of-maps-without-number (gen/not-empty (gen/list (gen/hash-map :testing gen/string-alpha-numeric
:values gen/double))))
(gen/fmap (fn [[list-of-maps min-number]]
(map #(assoc %1 :number %2) list-of-maps (iterate inc min-number)))
(gen/not-empty (gen/tuple gen-list-of-maps-without-number (gen/large-integer* {:min 1 :max 72}))))
#2020-01-0716:10gfredericks@iagwanderson the not-empty around tuple can be removed#2020-01-0716:11gfredericksOtherwise looks good#2020-01-0723:30bartukaWhen I ran a test using defspec
, I tried two kind of conditionals: 1) Only using a (and ..
form and returning true for all branches of the test and 2) using is
from clojure.test
inside the (and.. form. I did this because without the is
I cannot see which branch actually failed, I only get back the map with the current output and the smallest input that provides the failure.#2020-01-0723:30bartukaI would like to know the branch that failed#2020-01-0723:31bartukathe real problem is that I have 200+ assertions now being reported by clojure.test if I place the is
inside it and I lose the smallest input that provided the failure#2020-01-0723:33hiredmanmake each branch a separate defspec#2020-01-0723:35bartukaI thought about that, in this case here, I am validating the size of the returned list and them some content in the list.#2020-01-0723:35bartukait seems a lot of duplication of code to test only the count for example#2020-01-0723:36hiredmanthey are distinct properties however#2020-01-0723:43bartukaseems like @gfredericks had some ideas about that in this library: https://github.com/gfredericks/test.chuck#2020-01-0723:43bartukahttps://blog.colinwilliams.name/clojure/testing/2015/01/26/alternative-clojure-dot-test-integration-with-test-dot-check.html#2020-01-0800:58gfredericksyep, that's what that stuff is about#2020-01-0800:58gfredericksalso, the newer version of test.check has a more flexible API that would support some kind of library for giving more information about what failed#2020-01-0803:32bartukacool! I will tinker around that later#2020-01-2300:08kennyIs there a way to generate an alphanumeric string of a specific length?#2020-01-2300:20gfredericksI think all the string generators have corresponding char generators#2020-01-2300:20gfredericksSo you can use that with vector, fmap, and str#2020-02-2416:19kennyIs there a way to know if the output from quick-check was the result of a generator throwing an exception versus the function throwing an exception?#2020-02-2416:22gfredericksI don't think generator exceptions get caught#2020-02-2416:23kennyAh. I'm calling via spec's check
which does not throw in that case which explains the weird looking output.#2020-02-2416:23gfredericksDoes it catch exceptions?#2020-02-2416:24kennyIt appears so. When a gen throws, it attaches the generator exception on the :result
key and nothing else.#2020-02-2416:24gfredericksIs it calling the generator manually?#2020-02-2416:24gfredericksOr it just catches everything?#2020-02-2416:26kennyhttps://github.com/clojure/spec.alpha/blob/e89e294a3e28765c8758d6ac8569ff8c4eda76ef/src/main/clojure/clojure/spec/test/alpha.clj#L306#2020-02-2416:26kennyHmm actually that's not even a generator throwing. It's a gen throwing while being constructed.#2020-02-2416:26gfredericksYeah#2020-02-2416:27kennyBad test case. Will see what happens when a gen actually throws.#2020-02-2416:27gfredericksI don't like that return value#2020-02-2416:27kennyAgreed#2020-02-2416:27kennyI'm checking for it like this haha
(and (instance? Throwable (:result ret))
(nil? (:shrunk ret)))
#2020-02-2416:28gfredericksUckh#2020-02-2416:29kennySame thing when the gen throws#2020-02-2416:29gfredericksDifferent part of the code returns the same thing?#2020-02-2416:30kennyYep#2020-02-2416:31kennyI don't think there's any way for me to differentiate an exception from a generator throwing while construction and a generator throwing while generating.#2020-02-2416:32gfredericksExcept the stack trace 😉#2020-02-2416:33kennyYeah... Not sure where it's catching the gen throwing case.#2020-02-2416:33kennyIt's not in that ns. Must be doing it elsewhere.#2020-03-0422:31sparkofreasonIs there any way to limit the size of generated strings? I see the :max-size
stuff in the context of checking properties, but I want it for generating samples.#2020-03-0422:38gfredericksYou can control size for individual generators#2020-03-0422:39gfredericksSee gen/scale#2020-03-0422:46sparkofreasonIs there anything that would cause thread contention? I'm trying to generate samples of collections in parallel, behaving oddly.#2020-03-0422:50sparkofreasonHmmm, maybe it was just the scale
thing.#2020-03-0423:11gfredericksI can't think of any threading pitfalls#2020-03-0514:58sparkofreasonUsing scale
fixed it. Looked like threading because several threads would run and complete quickly, then appear to get increasingly bogged down, even if they weren't asking for a lot of samples. But I think it was just generating progressively large strings the longer it ran. Once I limited the size of the strings, everything went fine.#2020-03-3117:29Braden ShepherdsonI'm really struggling with an opaque such-that failure. the references for the failure are at generators.cljc
line 416, but I'm not explicitly calling elements
anywhere and I'm not sure what's up.#2020-03-3117:29Braden Shepherdsonif I run my own generators 100,000 times it works fine.#2020-03-3117:58gfredericksWhat version is this?#2020-03-3117:59gfredericks(So I can follow the line number)#2020-03-3118:02gfredericksNow that I think about it I bet you're getting misled by line numbers across versions#2020-03-3118:02Braden ShepherdsonI think I tracked down the source of that error; which is that (s/and int? #(< -30 % 30))
implicitly uses such-that
internally.#2020-03-3118:03gfredericksYeah that's the usual culprit#2020-03-3118:04Braden ShepherdsonI also apparently managed to break my REPL, and it kept telling me that something wasn't a generator when it clearly was. reloading it fixed that, not sure what caused it.#2020-03-3118:05Braden Shepherdsonanyway, now it's working nicely.#2020-03-3118:05gfredericksReloading the generators namespace somehow?#2020-03-3118:08Braden Shepherdsonnot sure. now that I've fixed the problem with my generators, it's not happening anymore.#2020-03-3118:08gfredericks🤷#2020-03-3118:13Braden Shepherdsonyeah, it's a mystery. I'm really loving the generative testing for fuzzing some complicated file encoding code, though.#2020-03-3118:14gfredericksYeah it's fun#2020-04-1116:48kennyIs there a way to get test-check to continue running a check even if one fails? I'm looking to create a large list of failure values for further analysis. Going to try using with-redefs on results/pass?
. Curious if anyone else has done something like this.#2020-04-1116:51gfredericksAre you using defspec in particular?
Do you want shrunk values or just original failures?#2020-04-1117:22kenny@gfredericks Just quick-check
on a prop/for-all
.#2020-04-1117:23kennyIn this case, I don't really care all that much on shrunk vs original.#2020-04-1117:23gfredericksWouldn't it be pretty easy to run it in a loop then?#2020-04-1117:24kennyLoop over quick-check N times, collecting failures? Would that cover a good amount of surface area if the iterations to failure is small?#2020-04-1117:25gfredericksYou're worried about there being some small likely failure that keeps getting found and obscuring other larger failures?#2020-04-1117:25kennyRight#2020-04-1117:26kennyI think it's likely to fail in < 5 tests.#2020-04-1117:26kennyI'm looking to find holes in my specs, verifying that the hole is correct or the spec is wrong.#2020-04-1117:27gfredericksSo the tc failure doesn't represent an actual bug?#2020-04-1117:28kennyYes - it may or may not.#2020-04-1117:28gfredericksGetting a PBT library to generate distinct failures without iteratively fixing the earlier ones is pretty hard/advanced#2020-04-1117:29gfredericksAny small failure is likely to be a component of larger failures#2020-04-1117:30kennyFor sure. Distinct can be impossible to define too 🙂#2020-04-1117:30gfredericksYep that's part of it#2020-04-1117:32gfredericksYou can get it to start at larger sizes if you think that will help#2020-04-1117:33gfredericksgen/scale#2020-04-1117:33gfredericksWith min#2020-04-1117:34kennyTo be more specific about my problem... We have a wrapper around some data pulled from an external API. We query data out using flat maps. Sometimes your query may have no results. This is the "problem". The solution is either: 1) Our generator for the query generated a map that should never be possible. We need to fix the generator (this can be hard) or acknowledge that this case will return no results 2) The external API is missing data for a particular query that should be there. We need to open a support case with the 3rd party, asking them why this is the case and what should be done.#2020-04-1117:35kennyHmm, that's interesting. I'll try your loop approach first and see what results it brings. If it seems to be missing some cases I can try scaling the gen. Perhaps even using a random scale with each loop.#2020-04-1117:36gfredericksSure#2020-04-1117:36kennyThanks 🙂#2020-04-1612:45Matheus Moreirahello! does anyone know if it is possible to use test.check’s let
generator to create a spec using spec’s with-gen
? basically i am spec’ing a function and one for one of the args i use let
to create the generator. the problem is that it doesn’t work if i try (spec/with-gen a-spec #(tcg/let…))
or (spec/with-gen a-spec (tcg/let…))
. it seems that something is missing to bridge from test.check to spec.#2020-04-1613:04gfredericks@matheus.emm that first one should work; what happens when you try it?#2020-04-1613:05Matheus Moreirathis is the error that i see:
Error printing return value (ClassCastException) at advent.code.2019.day4/fn$fn (day4.clj:48).
class clojure.test.check.generators.Generator cannot be cast to class clojure.lang.IFn (clojure.test.check.generators.Generator is in unnamed module of loader clojure.lang.DynamicClassLoader @717f07d1; clojure.lang.IFn is in unnamed module of loader 'app')
#2020-04-1613:06Matheus Moreiraand this is the function spec:
(spec/fdef repeated-consecutive-digits?
:args (spec/cat
:ds
(spec/with-gen
(spec/coll-of nat-int? :kind vector? :min-count 1)
(fn []
(check.gen/let [n (check.gen/elements #{2 3 4 5 6})
x (check.gen/nat)
d (check.gen/vector-distinct
(check.gen/such-that #(not= % x)
(check.gen/nat))
(- 6 n))
i (check.gen/elements (range (- 6 n)))]
(concat (subvec d 0 i)
(repeat n x)
(subvec d (+ i n))))))))
#2020-04-1613:08gfredericksnothing there looks weird to me#2020-04-1613:09gfrederickswell#2020-04-1613:09gfredericksmaybe the problem is just (?>
#2020-04-1613:09gfredericks#2020-04-1613:09Matheus Moreira#2020-04-1613:11Matheus Moreira#2020-04-1613:12gfredericks#2020-04-1613:12Matheus Moreira#2020-04-2121:00colinkahn@gfredericks is this ticket still in consideration? https://clojure.atlassian.net/browse/TCHECK-147 Ran into this today which prompted a discussion in the #clojure channel and that ticket was linked.#2020-04-2121:05gfredericksI don't know why that ticket is still open#2020-04-2121:05gfredericksSee gen/any-equatable#2020-04-2121:06gfredericksI'll update the ticket sometime soon#2020-04-2121:07colinkahnI see, thanks. I saw any-equatable but was kind of hoping the scope was broader and that perhaps the defaults would not produce non-equatable things#2020-04-2121:11gfredericksSee my comment in the thread in the other channel#2020-04-3014:07adamfreyWhen compiling an application including test.check in CLJS 1.10.597
I see this warning:
WARNING: cljs.core/<=, all arguments must be numbers, got [#{nil js/Number} number] instead at line 999 target/cljsbuild-compiler-2/clojure/test/check/generators.cljc
googling that message shows me that it's been appearing in people's stack traces for years, but I couldn't find any discussion about that warning in particular on the slack archives or on JIRA.
If anyone remembers prior discussion about this message would you mind pointing me to it or summarizing? Thanks.#2020-04-3014:10gfredericksI've never heard of it#2020-04-3014:12gfredericksWhat test.check version are you using?#2020-04-3014:13adamfreyI believe 1.0.0. This is all part of a big application, I think I'll try to recreate a minimal example#2020-04-3014:13gfredericksThat's not a version#2020-04-3014:13adamfreyoh, one sec#2020-04-3014:13adamfreyhttps://github.com/clojure/test.check/blob/master/CHANGELOG.markdown ?#2020-04-3014:14gfredericksWoah#2020-04-3014:15adamfreylol, I'm bringing all the news today#2020-04-3014:15gfredericksI guess Alex released it#2020-04-3014:24Alex Miller (Clojure team)surprise!#2020-04-3014:15gfredericksOkay well anyhow#2020-04-3014:15gfredericksLine 999 doesn't have a <= call#2020-04-3014:16gfredericksSo I have no idea what it might be referring to#2020-04-3014:17gfredericksThat line just defs a number to be the negation of another#2020-04-3014:17gfredericksI'd ask in #cljs#2020-04-3014:18adamfreythe difference in line numbers might be because of cljc
but this is the line that 999 points to in my target
directory: https://github.com/clojure/test.check/blob/master/src/main/clojure/clojure/test/check/generators.cljc#L1193#2020-04-3014:22ghadicljc
causes different line numbers?#2020-04-3014:23gfredericksYeah that's news to me#2020-04-3014:23ghadiit shouldn't, and whether it does is a verifiable assertion#2020-04-3014:24ghadi(i know you qualified it with "might" @adamfrey )#2020-04-3014:24adamfreyyeah, sorry, I threw that out there as an guess#2020-04-3014:24gfredericksLooking at the code it's pretty obvious via human analysis that the args can't be nil#2020-04-3014:25gfredericksIf those cljs warnings have false positives, maybe they have a way to suppress them?#2020-04-3014:26adamfreyAdding to the false positives likelihood I found a recent message of a similar vein: https://clojurians.slack.com/archives/C07UQ678E/p1587734805230700#2020-04-3014:26gfredericksI don't know what cljs wants us to do 🙂#2020-04-3014:26adamfreyso maybe this is a #cljs-dev question rather than #test-check#2020-04-3014:27gfredericksYeah#2020-04-3014:27gfredericksAt least initially#2020-04-3014:27adamfreyThanks for your help, Gary and Ghadi.#2020-04-3014:32adamfreythe difference in line numbers looks like it's because I'm not actually getting 1.0.0
, but an older version in my compilation output even though 1.0.0
is in my project.clj
. I'll figure that part out. Sorry for the wild speculation, ha#2020-05-1112:56arohnerWhat’s the best way to test properties on a collection? I typically write
(prop/for-all [in generator] (let [ret (foo in)] (every? (fn [r] ...) ret))
, and this works, but the output on failure isn’t very nice, because test.check doesn’t know how to show the offending element#2020-05-1112:57arohneroh, one other wrinkle, the spec for each item is dynamic:
(every? (fn [m]
(s/valid? (s/spec (::p/message-type m)) m)) msgs)
#2020-05-1112:58arohnerso I can’t easily use (s/valid? (s/coll-of ::foo) msgs)
#2020-05-1113:15gfredericks> test.check doesn’t know how to show the offending element
this can be remedied by returning a custom Result object#2020-05-1113:16gfredericksyou could imagine a helper for this, a variant of every?
, provided by a library or test.check itself#2020-05-1113:16gfredericksbut the plumbing is there for getting better output#2020-07-0509:25Vincent Cantin(I was unable to post on Jira, so I report here) in the docstring of tuple:
(def t (tuple gen/small-integer gen/boolean))
should be
(def t (gen/tuple gen/small-integer gen/boolean))
#2020-07-0516:37gfrederickslet's see if I still have commit rights#2020-07-0516:39gfredericksI do! it's fixed, thanks#2020-07-0516:46Vincent CantinYesterday I struggled with test.check’s API, I had to use a lot of internal functions and did not see a way to do otherwise.
Here is my function:
(defn- decreasing-sizes-gen
"Returns a generator of lazy sequence of decreasing sizes."
[max-size]
(#'gen/make-gen
(fn [rng _]
(let [f (fn f [rng max-size]
(when-not (neg? max-size)
(lazy-seq
(let [[r1 r2] (random/split rng)
size (#'gen/rand-range r1 0 max-size)]
(cons size (f r2 (dec size)))))))]
(rose/pure (f rng max-size))))))
#_(gen/sample (decreasing-sizes-gen 100) 1)
#2020-07-0516:49Vincent Cantin• I had to make my own #'gen/make-gen
because gen/randomized
is not public,
• gen/lazy-random-states
is not public either,
• I did not find an easy way to recursively iterate between using a generated value and using it to feed a new generator.#2020-07-0516:58Vincent CantinI am under the impression that, since the public API cannot be complete for sure, make-gen
and rng
should be exposed to the users.#2020-07-0516:59Vincent CantinIf one day I am writing a successor to test.check, I would build it based on that.#2020-07-0517:01gfredericksYou can't do this with the sizing combinators?#2020-07-0517:02Vincent Cantinwhich ones?#2020-07-0517:02gfrederickssized and with-size, I think#2020-07-0517:02gfredericksOne to get the size, the other to set them#2020-07-0517:04Vincent Cantinthen I would still need a way to build my lazy sequence#2020-07-0517:05gfredericksmap with range#2020-07-0517:05gfredericksAnd tuple#2020-07-0517:06gfredericksIt won't be lazy; is that important?#2020-07-0517:07Vincent Cantinin my case, I could live without, but in the general case, lazy sequence is something with zero support in test.check#2020-07-0517:08Vincent CantinI will keep my function as is, but I wanted to report the limitations I found.#2020-07-0517:10gfredericksYes. Should be easier to add laziness now that the rng is immutable#2020-07-1017:44Alex Miller (Clojure team)Released test.check 1.1.0, with https://clojure.atlassian.net/browse/TCHECK-155 (don't generate :/)#2020-07-1019:17gfredericksIt also doesn't generate :-) or :-( or :'( but I suspect it can generate :-P#2020-07-1019:17gfredericksAnd :-D#2020-07-1110:31sogaiui am confused by this line of documentation: https://github.com/clojure/test.check/blob/master/src/main/clojure/clojure/test/check/rose_tree.cljc#L90
> Takes a list of roses, not a rose
Is it possible what was meant was actually:
> Takes a rose, not a list of roses
as the functions that follow (i.e. permutations
, zip
, remove
, etc.) all seem to take a list of roses, but filter
does not?#2020-07-1112:31gfredericksyep, that sounds wrong#2020-07-1112:31gfredericksI pushed a commit that deletes that sentence.#2020-07-1112:31gfredericksaccording to git history, it was always wrong#2020-07-1112:52sogaiu@gfredericks thanks! (i didn't chase down the full history, but i also found that as far back as a i looked the sentence was the same.)#2020-07-1112:54gfredericksgit log -S "a list of roses"
is a good trick for finding where it came from#2020-07-1210:10Vincent CantinI think that the source code would have been easier to read if the rose tree was named after its purpose, for example shrink-tree
#2020-07-1210:19sogaiufwiw, quickcheck in haskell seems to use the term "rose" as well: https://hackage.haskell.org/package/QuickCheck-2.9/docs/Test-QuickCheck-Property.html#g:4 - i was initially puzzled but found a wikipedia page: https://en.m.wikipedia.org/wiki/Rose_tree#2020-07-1210:20sogaiualso the docs of the corr ns say: https://github.com/clojure/test.check/blob/master/src/main/clojure/clojure/test/check/rose_tree.cljc#L11#2020-07-1210:23sogaiuin this case i was initially lost but the name actually helped me to find relevant related info#2020-07-1212:29gfredericksyeah the initial draft of test.check was heavily based on quickcheck#2020-07-1522:38sogaiui was fortunate to find: https://www.youtube.com/watch?v=u0t-6lUvXHo -- it's been helping me to understand some of test.check's internals better.
i don't see this linked to from test.check's repository. if it isn't there somewhere, perhaps it would be worth mentioning it?#2020-07-1522:47Alex Miller (Clojure team)definitely!#2020-07-1522:49Alex Miller (Clojure team)https://github.com/clojure/test.check/commit/7e7c2116fa721211e8f5642a249e4a0f327445f4#2020-07-2007:45plexusWARNING: Use of undeclared Var goog.math.Long/fromBits at line 64 file:/home/arne/.m2/repository/org/clojure/test.check/1.1.0/test.check-1.1.0.jar!/clojure/test/check/random/longs.cljs
WARNING: Use of undeclared Var goog.math.Long/fromString at line 77 file:/home/arne/.m2/repository/org/clojure/test.check/1.1.0/test.check-1.1.0.jar!/clojure/test/check/random/longs.cljs
WARNING: Use of undeclared Var goog.math.Long/fromNumber at line 81 file:/home/arne/.m2/repository/org/clojure/test.check/1.1.0/test.check-1.1.0.jar!/clojure/test/check/random/longs.cljs
WARNING: Use of undeclared Var goog.math.Long/fromNumber at line 87 file:/home/arne/.m2/repository/org/clojure/test.check/1.1.0/test.check-1.1.0.jar!/clojure/test/check/random/longs.cljs
WARNING: Use of undeclared Var goog.math.Long/getOne at line 92 file:/home/arne/.m2/repository/org/clojure/test.check/1.1.0/test.check-1.1.0.jar!/clojure/test/check/random/longs.cljs
WARNING: cljs.core/<=, all arguments must be numbers, got [#{nil js/Number} number] instead at line 1193 file:/home/arne/.m2/repository/org/clojure/test.check/1.1.0/test.check-1.1.0.jar!/clojure/test/check/generators.cljc
I'm seeing these warnings when using test.check from ClojureScript. Not sure what's causing them, can't immediately see fault with the code.#2020-07-2007:45plexusdid some googling but only found this, which does seem to show at least one other person ran into this 🙂 https://github.com/AdamFrey/cljs-test-check-warning-example#2020-07-2013:39Filipe Silvaheya all#2020-07-2013:40Filipe Silvahow are people using test.check with async code?#2020-07-2013:41Filipe Silvais there any pretty obvious way to do it?#2020-07-2013:42Filipe SilvaI saw @gfredericks POC on https://github.com/clojure/test.check/commit/f90ee794dbe4dc766d91f9daa2160e8a8b0853be#2020-07-2013:43gfredericksHuh. Does that work? I don't remember it 🙃#2020-07-2013:43gfredericksIn any case there's nothing more official than that#2020-07-2013:43Filipe Silvaok, now I know#2020-07-2013:44Filipe SilvaI was asking because @wilkerlucio actually did something similar to what you did, but with core.async channels#2020-07-2013:44Filipe Silvahttps://github.com/clojure/test.check/compare/master...wilkerlucio:async-check#2020-07-2013:45Filipe Silvawilker and I were using it for test at http://roamresearch.com but now needed to extract it away from the official test.check#2020-07-2013:45Filipe Silvaso I was wondering if there was something around already that we should be using#2020-07-2013:47gfredericksIt's possible there are other libraries.
But I'd say this is the most severe hole in test.check at the moment. That POC could be usable if the cljs.test details are worked out, and if it doesn't hurt perf for synchronous jvm use#2020-07-2013:48Filipe Silvawhat are the cljs.test details?#2020-07-2013:52gfredericksjust "how should this be used from cljs.test?"; ideally it'd be similar or same as on the jvm
e.g., can defspec work?#2020-07-2013:53Filipe Silvahm I see...#2020-07-2013:53Filipe SilvaI was kinda focused on cljs so that didn't really come to mind#2020-07-2013:53Filipe Silvathere I'd just use an async deftest#2020-07-2013:56gfredericksIt might be easy. I just didn't know much about cljs testing when I made that commit#2020-07-2013:56Filipe Silvaok, will think about it some#2020-07-2013:56Filipe Silvathank you for the insight on this#2020-08-0422:25kennyIs there a facility for writing a generator that generates the next value based on the previous? e.g., similar to iterate
#2020-08-0422:27gfredericksGenerators aren't meant to be aware of previously generated values.
If this is part of a larger structure, you could describe that and there might be another way#2020-08-0422:30kennyThe objective is to generate a seq of dates increasing over time separated by a bounded, random amount of time.#2020-08-0422:30kennyTrying to generate data that imitates what actual data will look like.#2020-08-0422:31gfredericksGenerate the start date and the increments, and then fmap the result to get what you're describing#2020-08-2711:32sogaiui came across this: https://github.com/renatahodovan/grammarinator and started wondering if it is possible to take an antlr grammar of something and generate things based on it for use with test.check.#2020-08-2715:57gfredericksShould totally be possible.
test.chuck's regex generator is similar#2020-08-2721:55sogaiuthanks!
i was thinking about ashnur's email address problem being discussed on another channel and i noticed there is an antlr grammar for rfc822 email addresses: https://github.com/antlr/grammars-v4/tree/master/rfc822/rfc822-emailaddress#2020-08-3113:31sogaiutest.chuck's regex generator is quite neat!
i also found instacheck and instagenerator.
ha ha -- so late to the party 🙂#2020-08-3113:34gfredericksI hadn't heard of either of those#2020-08-3113:56sogaiumay be you have found them already, but fwiw:
* https://github.com/Engelberg/instaparse/issues/82#issuecomment-57892866
* https://github.com/Engelberg/instaparse/issues/82#issuecomment-426022741#2020-08-3114:17gfredericksCool. Apparently I must have seen the second one and forgot about it, since it mentions me#2020-08-2715:55arohnerBefore I embark on this journey, has anyone written a thing that combines prop/for-all
and gen/let
?#2020-08-2715:57gfredericksDoes test.chuck have something like that? I think so#2020-08-2715:58arohnerYeah, looks like test.chuck.gen/for
is what I want, thanks#2020-08-2716:00arohnerHrm, though I can’t stick it in the prop/for-all
, which is what I wanted to solve:
(prop/for-all [[a b c] (gen/let [a gen-a b gen-b c gen-c] [a b c] (do-test))
#2020-08-2716:01arohnerI’m trying to avoid ^^ that boilerplate#2020-08-2716:01gfredericksLemme see#2020-08-2716:01arohneroh, prop'/for-all
, I just have to keep reading#2020-08-2716:02gfredericksYep 🙂#2020-08-3011:41borkdudeI hit a issue in test.check where a global object in clojure.test.check.random is initialized at compile time in GraalVM, so everytime you invoke the binary, you get the same random number.
Strangely enough, I can't reproduce this with AOT-ing and running on the JVM#2020-08-3011:41borkdudeI've discussed this problem in #graalvm as well and posted a workaround for it#2020-08-3012:24borkdudeI summarized the issue in this gist: https://gist.github.com/borkdude/768eb5d01085944190e5eb485fd4737a#2020-08-3101:18gfredericksAOT wouldn't reproduce it because AOT doesn't run code at compile time#2020-08-3101:18gfredericksI mean maybe it runs it but it only compiles the code, not the result#2020-08-3101:19gfredericksa fix would be something that lazy-initializes the global rng the first time you use it, I guess?#2020-08-3101:20gfredericksI don't think that's in a perf-sensitive codepath so I don't think there are any downsides#2020-08-3101:21gfrederickssounds like a legitimate jira ticket#2020-08-3118:44MitchAre there resources available for testing async code in cljs via test.check
? I see that there is an open ticket in jira for support, just wondering if there are any known workarounds that might allow me to still use defspec
#2020-08-3118:55gfredericksDoes the ticket reference a branch with a proof-of-concept?#2020-08-3119:35MitchI don't think the ticket references it, but I have seen that branch. Haven't tried the branch out yet, though. I can give it a roll and share how it works for our use case if you think that would be helpful#2020-08-3119:37gfredericksI was suggesting a "workaround" would be to just use that branch, if it works.
Depends on your definition of workaround.
I'm not sure there are any other options short of somebody completing integrating that feature. Coverting to async is a pretty fundamental change.#2020-08-3119:42MitchCool, thanks for helping me out. I really appreciate it. I'll see how that goes. I'm a little surprised that this seems to be a rare scenario 🤷#2020-08-3119:45gfredericksI, also, am surprised#2020-08-3119:48gfredericksI haven't done much professional clojure web dev; maybe people tend to use frameworks where the user code is just functions on data, and so those user functions are amenable to synchronous testing?#2020-08-3119:58MitchYeah, I think that is a big part of it. Most of the time there is already a well-supported React component for whatever library you want to integrate, so you do not need to worry about testing its behavior#2020-08-3121:49borkdude@gfredericks
> a fix would be something that lazy-initializes the global rng the first time you use it, I guess?
https://gist.github.com/borkdude/768eb5d01085944190e5eb485fd4737a#workaround
I'll make a JIRA ticket tomorrow#2020-09-0114:29borkdude@gfredericks done: https://clojure.atlassian.net/browse/TCHECK-157 (includes patch)#2020-09-0114:49gfredericksLGTM
I can't open jira on my computer for some reason, so can't comment on it#2020-09-2716:49colinkahnIs it possible to make a vector generator that always generates X number of items but can shrink down to only 1 of those items?#2020-09-2716:50gfredericksWhat's the use case?#2020-09-2716:52colinkahnI have a test that has some expensive setup and I want to run the max items through it (each item of the vector) but still want to be able to get it to shrink to a single item if it fails#2020-09-2716:53gfredericksyou can get approximately this behavior by modifying the size
parameter
that won't specifically guarantee large vectors, but it does express the idea of "I want to generate big inputs for whatever reason"#2020-09-2716:54colinkahnBut will that affect the size of other gens or can I modify per-gen?#2020-09-2716:57gfredericksummmmmmhm
let's see
something like
(defn gen-vector-with-shallow-min-size
[min-size element-gen & other-args]
(gen/sized (fn [size] (gen/scale #(min min-size %) (apply gen/vector (gen/resize size element-gen) other-args)))))
#2020-09-2716:57gfrederickskind of hairy, but I believe that's what you're describing#2020-09-2716:58gfredericksagain, doesn't guarantee any minimum number of elements, but it will skew the distribution larger#2020-09-2716:58gfredericks(which is intentional -- it's nice to still test empty/small vectors occasionally just in case)#2020-09-2716:59colinkahnNice! Thanks! I’ll give it a shot!#2020-09-2812:31Saikyunis it possible to write a meaningful "not"-generator? in my case I have a bunch of specs and I use clojure.spec.gen.alpha/generate
for automatic testing. it just struck me that it might sense for some falsifying tests, i.e. checking that my code actually throws exceptions on invalid data#2020-09-2812:51gfredericksif you have a spec you could do something like
(gen/such-that (complement passes-the-spec?) gen/any)
#2020-09-2812:51gfrederickskind of sloppy/stupid but also it's one line#2020-09-2817:13Saikyunhaha, interesting. could be nice to just throw at the program to see what happens 🙂#2020-09-2817:13SaikyunI just started thinking that it might be too naive to only test for "valid" data (it's an input form)#2020-09-2817:15gfredericksYeah that's a tough use case I think.
There's a variety of options with different tradeoffs, and no holistic silver bullet thing#2020-09-2817:15Saikyunmakes sense. I guess I could try generating random strings and just filter away the ones that are actually valid#2020-09-2817:15gfredericksTwo things are
Just use examples
Generate valid things and then tweak them in specific ways#2020-09-2817:16Saikyunright#2020-09-2817:16Saikyunthanks for the input, will think a bit about it 🙂#2020-09-2817:17SaikyunI'm very happy because today I managed to create automatic gui / end-to-end testing by using cljs + spec generators 🙂#2020-09-2817:17Saikyunit looks very cool to have everything filled out for me. will be a nice demo to show when talking about clojure with my colleagues#2020-09-2817:18gfredericksNice#2020-09-2817:23andy.fingerhutAs a maybe-already-obvious example, generating strings of 10 characters where each character is uniformly chosen from the set of all Unicode code points, and then filtering only the ones that contain only ASCII characters, is extremely unlikely to get past the ASCII filter.#2020-09-2817:25Saikyun@andy.fingerhut was that aimed at me? 🙂#2020-09-2817:26andy.fingerhutIt was following up on your conversation, yes.#2020-09-2817:29Saikyunah, okay. yeah, then I get what you mean 🙂 I was thinking if there might be something "cleverer", or an obvious answer that I didn't know about. thanks ^^#2020-09-2817:47andy.fingerhutObviously for that simple example, if you want to guarantee all-ASCII strings, it is straightforward to generate them that way in the first place, and then no filter is needed. But when the condition you want to satisfy has more interdependence between the parts, it can get very tricky.#2020-09-2906:47Saikyunyeah, that makes sense. we'll see how I solve it then 🙂#2020-10-0607:27Steven DeobaldHey folks. I have a friend who's looking to dig into some generative testing for a web service/app we're volunteering on. All the generative testing examples I have at hand are closed-source, from work done in another life. Does anyone have an open source project(s) they'd recommend he read to see some examples of exercising a domain with test.check? Preferably apps or services, rather than libraries.#2020-10-0612:15gfredericksthat one conj keynote was about ostensibly testing dropbox
I think there's a paper to go with it#2020-10-0612:17gfredericksand there was a much earlier conj talk about testing a web UI; that probably doesn't qualify as a real code example though#2020-10-0722:24Hugh PowellMorning folks, does anyone know if there's a simple way to run a `test.check` `defspec` with only the shrunk value of a failed run and without trying to further shrink it? I know I can just call the test itself with a `:seed` argument, but that runs it with the original data and all the subsequent runs to shrink it. This can make debugging harder than it feels it should be. The only way I can think to do this would be to factor out the test code into a function that takes the generated data.#2020-10-0722:29gfredericksThis overlaps with the use case of wanting to add regression cases to a defspec.
Both of those would not be hard features to implement, just need to decide on an api.
But as of now it does not exist. I've done the factoring you describe a number of times, and it fits well with adding a deftest with regression cases.#2020-10-0722:33Hugh PowellAwesome, thanks for that :thumbsup:#2020-10-0722:57seancorfieldThat sounds like the same advice that Eric Normand gives in his Property-Based Testing course (on http://PurelyFunctional.tv).#2020-10-0722:58seancorfieldThat allows him to take the shrunk failure from the output of defspec
and just run that on its own through the test code in a Rich Comment Form in the same ns.#2020-10-1220:50colinkahnis it possible to have a generator that doesn’t shrink at one level, but its children do shrink? For example:
(commit-to (gen/one-of [gen-for-case-1 gen-for-case-2]))
So once it chooses either case 1 or 2 it won’t try the other, but will shrink the things gen’d in them?#2020-10-1220:53gfredericksNot using public functionality, I don't think#2020-10-1220:56gfredericksA hacky idea is to have a predicate on the generated values to tell you what branch matched, and then wrap that in a such-that that insists it is something from that branch#2020-10-1220:57gfredericksSuper goofy looking, and wastes a bit of cpu, but otherwise would do what you're describing#2020-10-1220:59colinkahnHaving a hard time imagining what that would look like, but I think if it’s not standard I can just duplicate my test 😅#2020-10-1221:35gfredericksYeah I couldn't tell what your appetite for Weird was 🙂#2020-10-1410:59zcljHi test.check users, I have released a library intended to provide insights on the values created by test.check generators. If that sounds interesting to you, I would appreciate any usage feedback you might have, with the hopes of making this useful to all test.check users.
https://github.com/zclj/test.check.insights#2020-11-2215:26mishahttps://akovantsev.github.io/corpus/clojure-slack/test-check - filterable and explorable channels log on a single offline html page, cheers#2020-12-0421:36andy.fingerhutI am using the collections-check library, which currently depends upon not-the-most-recent version of test.chuck and test.check libraries#2020-12-0421:36andy.fingerhutIn trying to verify that it can catch certain kinds of errors, I am intentionally introducing some small errors into a collection implementation. In one case, I make it throw an exception.#2020-12-0421:38andy.fingerhutcollections-check makes a call to test.chuck's checking macro, like this:
(chuck/checking "vector-like" num-tests-or-options
[actions (gen-vector-actions element-generator (transient? empty-coll) ordered?)]
(let [[a b actions] (build-collections empty-coll base true actions)]
(assert-equivalent-vectors a b)))
#2020-12-0421:38andy.fingerhutThe function assert-equivalent-vectors
is a series of clojure.test is
macro invocations.#2020-12-0421:39andy.fingerhutWhen I change the implementation so that it throws an exception, the exception comes not from calling assert-equivalent-vectors
, but calling build-collections
.#2020-12-0421:40andy.fingerhutI put a print statement with a counter inside of assert-equivalent-vectors
to see what parameters it was being called with, and how many times, and when there is no exception thrown, I see the checking
macro limiting itself to num-tests-or-options
iterations (100 or 200 in my short tests).#2020-12-0421:42andy.fingerhutWhen build-collections
throws an exception, something happens where I see assert-equivalent-vectors
continue to be called many, many times, perhaps an infinite loop, but a println I added within test.check's quick-check
macro (or function? I forget) no longer prints each time through the loop. Either the println gets messed up in some weird way (very odd if so), or somehow that quick-check
loop is somehow causing assert-equivalent-vectors
to be called in an infinite loop, without calling build-collections
again.#2020-12-0421:42andy.fingerhutThis may all be rubber ducking if I figure this out, but wanted to ask if:#2020-12-0421:43gfredericksShrinking is part of this, right?#2020-12-0421:43andy.fingerhut(a) Is test.chuck's checking
macro intended to be used with a checking function that calls clojure.test/is
one or more times, and its return value isn't important?#2020-12-0421:43andy.fingerhutIs there a way I can tell whether shrinking is involved? I'm happy adding extra println's anywhere here.#2020-12-0421:45andy.fingerhutOh, perhaps shrinking is causing many calls to assert-equivalent-vectors
, and they are all failing, and shrinking is "stuck" somehow?#2020-12-0421:45gfredericksis is the whole point of checking#2020-12-0421:46gfredericksI don't have any theories for why you'd see it stop printing#2020-12-0421:47andy.fingerhutok. Will keep looking a bit longer.#2020-12-0421:47gfredericksYou can money patch that to confirm it's shrinking#2020-12-0421:47andy.fingerhutI was hoping that checking
would quickly show some output of a failure if an exception was thrown, rather than going into an infinite loop (if it is infinite -- not sure whether it is infinite or just long)#2020-12-0421:48andy.fingerhutPhone handier than links today? 🙂#2020-12-0421:48gfredericksYep 🙂#2020-12-0421:48gfredericksThat's in the base t.c namespace#2020-12-0421:49andy.fingerhutI can try updating to latest versions of test.chuck and test.check, but initially when I tried that there seemed to be some incorrect args being passed around somewhere, and hadn't figured that out yet to make it work well enough to debug.#2020-12-0421:50andy.fingerhutThanks for the leads#2020-12-0421:52gfredericksn.p.; happy to help out if you run into something#2020-12-0421:53andy.fingerhutShould latest test.check and test.chuck be compatible with each other, as far as you know?#2020-12-0421:54andy.fingerhutI think collections-check may need some changes to work with latest test.chuck, but would prefer not to guess too many times at which pairs of test.chuck/test.check version combinations should work with each other.#2020-12-0421:54gfredericksYes#2020-12-0421:55gfredericksBoth libraries have generally avoided breaking changes#2020-12-0422:15andy.fingerhutWow. I think I found it. collections-check is passing a double for the num-tests-or-options, which test.chuck's times
function in the latest version returns nil
for.#2020-12-0422:15andy.fingerhutI mean, I found the thing that I thought would take 30 seconds in about 20 mins 🙂. Not the final answer.#2020-12-0422:17gfredericksThis thing returns nil???#2020-12-0422:20andy.fingerhutThis thing:
(defn times [num-tests-or-options]
(cond (map? num-tests-or-options) (:num-tests num-tests-or-options tc.clojure-test/*default-test-count*)
(integer? num-tests-or-options) num-tests-or-options))
#2020-12-0422:22gfredericksOh geez there's two of them#2020-12-0422:23andy.fingerhutIt was weird that collection-check was using doubles anyway -- probably Zach T liked 1e2, 1e3 for brevity on test counts.#2020-12-0422:24gfredericksThis is why we can't have nice static type systems#2020-12-0423:18andy.fingerhutThe shrink-loop is definitely going for far more iterations than I would have expected.#2020-12-0423:19gfredericksWhat sort of data is it?#2020-12-0423:20andy.fingerhutcollections-check generates sequences of operations on data structures, in this case on vectors, like assoc, conj, transient, persistent!, seq, etc.#2020-12-0423:20andy.fingerhutSo I think it must be attempting to shrink that sequence of operations down to a case where it passes, and then increase it a bit?#2020-12-0423:22andy.fingerhutI am printing out the length of the vectors produced in the function that does multiple is
calls to compare the results of two different vector implementations, each produced with two different vector implementations, and they do fluctuate a bit, but they are not getting consistently smaller.#2020-12-0423:22andy.fingerhutI guess I could figure out how to look at the sequence of operations being tried in the shrink-loop to see if it is getting smaller/simpler#2020-12-0423:24gfredericksThe pattern isn't necessarily simple#2020-12-0423:25gfredericksEspecially if the generator uses bind#2020-12-0423:26andy.fingerhutThis is a wild guess, but it also generates the values to be added to the vector via gen/int, and there are about 500 of them in the failing vector, so if it is spending any work trying to simplify those numeric values, most of those are going to give the same result, since the error I put into the bad vector implementation fails over about 520 vector elements, unless pretty much most of the vector elements are equal to each other.#2020-12-0423:28gfredericksWhat problem are you actually trying to solve?#2020-12-0423:29andy.fingerhutGood question. In this case, I wanted to verify that collections-check could detect a simple bug in an implementation, because I wanted to better trust the results when it returns no bugs found.#2020-12-0423:30gfredericksAh okay, and the fact that it's spinning instead of returning failure is concerning#2020-12-0423:30andy.fingerhutSo I'm really done verifying that it can find this example bug I've introduced, but only because I've added extra debug prints. If I don't have those, I just get what appears to be an infinite loop, with no failing test case given.#2020-12-0423:31gfredericksWrap the gen in gen/no-shrink 😛 😉#2020-12-0423:31andy.fingerhutI suppose I could eliminate the possibility of gen/int shrinking slowing things down, by forcing vector elements in the sequence of operations to be increasing numbers.#2020-12-0423:31andy.fingerhutOK, didn't know about that, so thanks.#2020-12-0423:31andy.fingerhutwill try#2020-12-0423:31gfredericksIf there's not a feature for limiting shrink time then there oughta be#2020-12-0423:31andy.fingerhutJust (gen/no-shrink gen/int)
where I currently have gen/int
I guess?#2020-12-0423:32gfredericksYou could. Or you could wrap the whole collection generator#2020-12-0423:33gfredericksYou get different effects for each#2020-12-0423:35andy.fingerhutIs there anything in shrink-loop that represents the value passed to the function being tested?#2020-12-0423:36andy.fingerhuti.e. that I could print and look at?#2020-12-0423:36andy.fingerhutI know it isn't head
and it is not result
because those have the wrong class#2020-12-0423:41gfredericksActually no#2020-12-0423:41gfredericksI thought so#2020-12-0423:41gfredericksBut the structure here is more function oriented than data oriented#2020-12-0423:43gfredericksYou know what I think there's a callback feature you'd like#2020-12-0423:43andy.fingerhutThe value of (:args (rose/root head))
might be what I'm seeking#2020-12-0423:43gfredericksUse reporter-fn#2020-12-0423:43gfredericksPass one in#2020-12-0423:44andy.fingerhutI'm comfortable hacking on my own modified copy of test.check and test.chuck if it helps, which is how I got to that result.#2020-12-0423:44andy.fingerhutThanks for the help. Will be away from keyboard for a bit here.#2020-12-0513:53gfredericks`:reporter-fn`
A callback function that will be called at various points in the test
run, with a map like:
;; called after a passing trial
{:type :trial
:args [...]
:num-tests <number of tests run so far>
:num-tests-total <total number of tests to be run>
:seed 42
:pass? true
:property #<...>
:result true
:result-data {...}}
;; called after the first failing trial
{:type :failure
:fail [...failing args...]
:failing-size 13
:num-tests <tests ran before failure found>
:pass? false
:property #<...>
:result false/exception
:result-data {...}
:seed 42}
It will also be called on :complete, :shrink-step and :shrunk. Many
of the keys also appear in the quick-check return value, and are
documented below.
☝️ an arg to clojure.test.check/quick-check
, which you could monkey-patch if you aren't actually calling it directly#2020-12-0515:13andy.fingerhutThanks for that tip.#2020-12-0515:13andy.fingerhutI know that shrink-loop in my case is not finishing within 10,000 iterations, since I put in some logging and let it run for a while.#2020-12-0515:15andy.fingerhutI guess I might let it run for a day or so, out of curiosity to see if it actually finishes. In the mean time, it does seem like some kind of option to force shrink-loop to return early would be useful in some cases, as I think you may have mentioned earlier.#2020-12-0515:16andy.fingerhutOr perhaps there is one, and I haven't read the docs yet.#2020-12-0515:16gfredericksI would have added that feature a long time ago, but I got stuck being unable to decide on a coherent way of configuring test.check given the >=2 different usage styles#2020-12-0515:17gfredericksalso once you start returning from shrinks early people will want a feature for resuming shrinks, and that's even more complicated#2020-12-0515:18gfrederickslooks like there's no error-handling on the reporter-fn
, so technically you could just throw an exception from there 😄#2020-12-0515:18andy.fingerhutThat is certainly one way to do it. Let me try that.#2020-12-0515:20gfredericksit might not be trivial to get it to accurately return a "smallest failing case so far"#2020-12-0515:21gfredericksshouldn't be hard to get it to return a "most recently failing case"; maybe that's the same thing, I'm not quite sure 🙂#2020-12-0515:21andy.fingerhutI see current-smallest
#2020-12-0515:21gfredericksoh, perfect#2020-12-0515:21andy.fingerhutwhich seems to be passed to reporter-fn whenever a new one is found#2020-12-0606:16andy.fingerhutThanks for pointing me in the direction of reporter-fn, and mentioning that having it throw an exception could terminate the shrinking early.#2020-12-0606:20andy.fingerhut@gfredericks Would you be interested in some kind of patch for the test.check library, perhaps documentation only, with around 60 lines of code that is a sample reporter-fn that when given to quick-check, will print progress on the number of tests run so far, and throw an exception if shrinking takes more than 5000 iterations through the shrink loop, saving the smallest args ever encountered in an atom before throwing the exception?#2020-12-0609:16andy.fingerhutIn case you are curious, this is what the code looks like: https://github.com/jafingerhut/clj-2594/blob/master/src/com/fingerhutpress/clj_2594.clj#L135-L193#2020-12-0702:56gfredericksI'm not maintaining test.check anymore, so not my call, officially
unofficially, documentation of a hack as a placeholder for a better-thought-out feature doesn't sound bad#2020-12-1012:31borkdude@gfredericks do you know who is maintaining test.check now? thanks for maintaining it for so long!#2020-12-1014:03gfredericksprobably de facto alex miller#2020-12-1014:05gfredericksat the very least he'd have a more official answer to that question#2020-12-1017:53seancorfield@gfredericks I guess I missed hearing you had stepped down from it until this discussion -- may I ask why?#2020-12-1018:00gfredericksI'm just really bad at design tradeoffs. I did all the no brainer low hanging fruit stuff, and everything that's left induces lots of analysis paralysis#2020-12-1018:01gfredericksI also have less time than I used to#2020-12-1018:02gfredericksSo those two things pushed it into "not personally worth it" territory#2020-12-1018:03seancorfieldTotally understand that! I've been staring at a V2 of HoneySQL for months and I am now at a point where I think parts of my design are just dead ends and I need to start from scratch... which is kind of daunting... 😞#2020-12-1018:05gfredericksYeah, it's not easy work#2020-12-1018:06gfredericksUnless you just yolo it and embrace "future people will hate this" 🙂#2021-01-2421:35borkdude#2021-03-0421:21hiredmanit seems unlikely, but I think you should be able to get by with a single fmap#2021-03-0421:22gfredericksEqual length strings?#2021-03-0421:24hiredmangenerate a string of more than n characters, generate a vector of n characters and indices, then fmap over both of those producing a pair of the string, and the string with characters swapped at the indices#2021-03-0421:24hiredman(the indices will require some fiddling to work out right, but doable)#2021-03-0421:25gfredericksYou could also generate edit operations#2021-03-0421:25hiredmanyeah, characters + indices are an edit operation#2021-03-0421:27murtazaWhat I've done so far is generate a string, and then edit out parts of it at random start and end positions using subs, joins and replace#2021-03-0421:27hiredmanno, I don't think you can do it without an fmap#2021-03-0421:28gfredericksRandom meaning calling rand-int or similar?#2021-03-0421:34gfredericksTied together with bind and fmap?#2021-03-0421:35gfredericksHa#2021-03-0421:38gfredericksSure#2021-03-0421:28hiredmanin order for the generator to work at all, you need to at the base to be able to turn 1 string in to a pair of strings#2021-03-0421:28hiredmanthat is gonna be an fmap#2021-03-0421:39gfredericksI think generating edits is pretty easy, though it might be harder to aim for exactly N differences if you need that#2021-03-0421:39gfredericksGenerating edits would shrink nicely I think#2021-03-0421:40gfredericksIf you want to shrink toward fewer differences#2021-03-0421:51hiredman;; written but not run
;; assuming strings of length 5 and differences of 3 characters
(gen/fmap
(fn [string edits]
(loop [[[shift index] & es] edits
s (vec string)
editted #{}]
(if e
(let [i (mod index (count s))]
(if (contains? editted i)
(recur (cons [shift (inc index)] es) s editted)
(recur es
(update-in s [i] (fn [c]
(char (+ 32 (mod (+ offset (- (int c) 32)) 126)))))
(conj editted i))))
(apply str s))))
(gen/tuple (gen/resize 5 gen/string-ascii)
(gen/tuple
(gen/tuple s-pos-int s-pos-int)
(gen/tuple s-pos-int s-pos-int)
(gen/tuple s-pos-int s-pos-int))))
#2021-03-0421:56hiredman(that mod 126 should maybe be mod 126 - 32)#2021-03-0421:57hiredmanand the fn has to destructure the vector, etc#2021-03-0421:58robertfwCan you recur
from there?#2021-03-0421:58hiredmaninside a loop? of course#2021-03-0421:58robertfwI meant from non-tail position#2021-03-0421:59hiredmanI am pretty sure those are all tails#2021-03-0422:00robertfwI thought I had run into not being able to recur from the first branch of if but I may well be getting mixed up with recur within try#2021-03-0422:01hiredmanyes, you can't recur across a try, logically a try pushes an exception handler on to a stack, so recuring across a try grows the exception handling stack#2021-03-0422:03robertfwThanks, that makes sense#2021-06-1400:22colinkahnI noticed a difference in using the for-all
and checking
forms available in com.gfredericks.test.chuck.clojure-test
, seems that for-all
will give one assertion error on the shrunk result where checking
will give one for each failing is
. Is that expected? I was hoping for-all
would give the same kind of reporting.
(defspec test-test
(for-all [x (gen/return 1)]
(is (zero? x))))
(deftest test-test
(checking "fails" 1 [x (gen/return 1)]
(is (zero? x))))
#2021-06-1400:41seancorfield@colinkahn It’s expected: for-all
is a single assertion — that the property holds for all values tested — so it will either pass or fail as a whole: is
just checks the expression; checking
runs the whole body it is passed — which can contain any number of assertions — and in that context each is
runs as a regular assertion. Does that help?#2021-06-1400:49colinkahn@seancorfield I knew for-all
in test.check
worked that way but figured the test.chuck
one wasn't limited to a single assertion and would report them all. https://github.com/gfredericks/test.chuck#alternate-clojuretest-integration
It is just a minor inconvenience if that's the case though, I like the fact that I can get the :smallest
from the return of defspec
in the REPL, but also wanted the reporting available in checking.
#2021-06-1717:21Adam Helins@colinkahn I was in search of "multiplexing" a single property into several one while still tracking exactly what fails. checking
does that using is
but I wasn't so pleased with the internals and wanted something that worked without is
, so I created this (in case you find that helpful): https://github.com/helins/mprop.cljc#2021-06-1717:31colinkahnCool, I’ll check this out!#2022-03-0819:33dnolenmaybe I'm not looking in the right places - but there seems to be not much information about writing customer generators?#2022-03-0819:34dnolenNot knowing that much about custom generators, after auditing some repos I have a trivial stateful generator for producing commands to drive some test#2022-03-0819:34dnolenit works by recursion by feeding the augmented state to produce the next command#2022-03-0819:35dnolenby doing so then it becomes easy to make the command sequence more meaningful, by i.e. decreasing the frequency that the last command will be chosen, etc. etc.#2022-03-0819:35dnolenbut before proceeding any further I did a quick check (ba-dum) and of course it did not shrink#2022-03-0819:36hiredmaninstead of generating a single command generate a list of commands, and statefully process each list, with a fresh state for each list#2022-03-0819:37hiredmanbasically you want to generate programs, instead of generating a single command and building programs out of the generated command#2022-03-0819:38hiredmanif you generate a whole program then the whole program can shrink#2022-03-0819:38dnolenthe recursive thing does indeed produce a list of commands - but that is what doesn't shrink#2022-03-0819:38hiredmanah, I thought you where doing the recursive thing outside of generating#2022-03-0819:39dnolenlooking at the built in generators - it appears that shrinking might be a bit manual#2022-03-0819:39dnolenlike you have to make the shrink tree yourself?#2022-03-0819:39hiredmanthere is https://github.com/clojure/test.check/blob/master/src/main/clojure/clojure/test/check/generators.cljc#L1641 which might help#2022-03-0819:40hiredmanI've never had to manually do shrinking#2022-03-0819:41hiredmanI usually generate programs then feed them into some interpreter, and that interpret might no-op some commands in certain states, instead of feeding the state through the generator#2022-03-0819:41dnolenhrm really, in the (repo docs) the first thing that it says is that if you build a stateful generator via bind - it will not shrink very well#2022-03-0819:44hiredmanoh yeah, I've had to massage things around to get good shrinking#2022-03-0819:45hiredmanbecause bind is opaque#2022-03-0819:46gfredericksI feel like @dnolen described this to me at a conference like five years ago#2022-03-0819:46hiredmanI think anything other than bind being a simple maping kind of operation (take a vector of numbers and produce a java.util.Date object) isn't going to shrink well#2022-03-0819:48dnolen@gfredericks haha we probably talked about it - but now I actually have a reason to use this approach at work#2022-03-0819:48gfredericksI feel like maybe the best thought I had on this was that it'd be useful to have a general framework for this style of generator
Something that could be implemented once against the lower level primitives#2022-03-0819:49dnolenonly tangentially related but I found this interesting paper - https://proper-testing.github.io/papers/icst2018.pdf#2022-03-0819:49gfredericksI assume the style is that you have a model of the state of the system at a given time, and can generate commands based on the current state, and each command transforms the current state to the next state?#2022-03-0819:49dnolenPropEr is a open source Erlang framework for testing stateful systems, and they have been implementing things to make it easier to test stateful system#2022-03-0819:50dnolenboth by providing a framework, but also integrated simulated annealing to control the search#2022-03-0819:50hiredmanI've done that sort of thing, but without feeding the state through command generation, and then ignoring invalid commands in the interpreter#2022-03-0819:51dnolenI don't really care one way or another about whether test-check provides such affordances#2022-03-0819:51dnolenbut rather some guidance on writing custom generators that might be influenced by above tactics#2022-03-0819:52dnolenso if I recur on this generator, do I need to manually populate the rose tree w/ children for the previous state#2022-03-0819:52dnolenso it can shrink later?#2022-03-0819:53gfredericksCustom shrinks aren't available at all in the main user facing api I don't think#2022-03-0819:54dnolenyes I know that 🙂#2022-03-0819:55dnolenbut the non-doc'ed but not private internal helpers can be used as far as I can tell (to accomplish the goal, not future-proof)#2022-03-0819:55gfredericksactually#2022-03-0819:57gfredericksBy "manually popular the rose tree" do you mean something like the haskell quickcheck api where you provide a function X -> [X]
X being a list of commands#2022-03-0819:57dnolen(oh yeah, re: using test-check, I think that was about testing stateful UIs - barely remember that so long ago)#2022-03-0819:57dnolen@gfredericks can you not just gen-fmap
and create new rose w/ what you know should be the children?#2022-03-0819:58gfredericksYou can#2022-03-0819:58Alex Miller (Clojure team)if you build out of fmap and bind, it will shrink#2022-03-0820:00Alex Miller (Clojure team)right? looks at gfredericks#2022-03-0820:01gfredericksThis kind of thing can definitely shrink, but often it's degenerate#2022-03-0820:01gfredericksIf you generate 200 commands#2022-03-0820:01gfredericksAnd each is logically dependent on all the earlier ones#2022-03-0820:02gfredericksThen shrinking command #7 invalidates the other 293 of them#2022-03-0820:02gfredericksSo they have to be replaced with freshly generated commands based on the new state after the new command 7#2022-03-0820:03gfredericksBecause TC can't do better than that generically#2022-03-0820:03gfredericksBut dnolen with his domain knowledge can describe more particularly when it's valid to shrink or delete intermediate commands#2022-03-0820:04gfredericksAnd so can get much better results#2022-03-0820:04Alex Miller (Clojure team):thumbsup:#2022-03-0820:05gfredericksSo the choice is
A) add a new feature for this situation that lets users express that domain knowledge
B) dnolen does this particular case by hand#2022-03-0820:05gfredericks(as I see it)#2022-03-0820:06gfredericksSometimes you can get by with tricks like this one https://clojurians.slack.com/archives/C0JKW8K62/p1646769032795009#2022-03-0820:07gfredericksWriting your own shrink tree can be laborious because it's hard to take advantage of the built in shrinking for the atomic types you're using#2022-03-0820:10gfredericksI also had an idea for a user-enhanced shrinking, but not sure that's a comprehensive solution here#2022-03-0820:13dnolen@gfredericks design is hard ... I'm curious as to why the shrinking stuff that's there is not just exposed and documented#2022-03-0820:13dnolenit seemed like one of the novel things about test-check was that shrinking was decoupled - but then somehow not a part of the api#2022-03-0820:15gfredericksI don't think that was too intentional
I mean you could ask Reid if you want historical context
I largely took the approach of not changing things without a really strong reason
So I think it's totally valid to suggest exposing it#2022-03-0820:19gfredericksI think the hope was that the built in shrinking was high enough quality that users wouldn't have to think about it
Probably not realistic, but it works much better than the QuickCheck approach I think#2022-03-0820:21dnolenI think you don't really hit this problem until you want to write stateful generators#2022-03-0820:21dnolenthen the domain starts to dominate and the defaults just aren't that useful - because it's mostly junk#2022-03-0820:23dnolenI will continue poke around at what's there for now - still barely know how to do anything so will probably say marginally more intelligent things later 😉#2022-03-0820:23gfredericksOh hey, I wrote something about this#2022-03-0820:24gfrederickshttps://github.com/clojure/test.check/blob/master/doc/old-confluence-notes.md#stateful-generators#2022-03-0820:24dnolenyes already read that 🙂#2022-03-0820:24dnolenseveral times#2022-03-0820:24gfredericksHah#2022-03-0820:25hiredmanit is iterate (unfold) for generators#2022-03-0820:27gfredericks@dnolen does the idea there under "general problem" seem like it would cover your use?#2022-03-0820:30hiredmannot entirely sure, but it seems like https://github.com/advancedtelematic/quickcheck-state-machine takes the approach of generating events without regard to state, but has you define preconditions which say if some event + state is well formed#2022-03-0820:32hiredman(I can tell the precondition is a thing, but my haskell is not good enough to immediate tell if the precondition stuff is somehow used as part of generation or just as a filter afterwards)#2022-03-0820:32dnolenbasically the problem that I've encountered so far is as @gfredericks alluded - steps depend on previous steps#2022-03-0820:32hiredman> The pre-conditions are used while generating programs (lists of actions).#2022-03-0820:33dnolenas the command grows expecting to randomly line anything up is unlikely#2022-03-0820:33dnoleninstead we want to look at the previous state to make intelligent but not totally predetermined steps#2022-03-0820:34dnolenat the same time, if something is found, we want to it to shrink#2022-03-0820:38dnolenok manually managing the rose tree seems like it leads to better results here#2022-03-0820:39dnolen@gfredericks I do see that one irritation w/ the rose tree versions is that mixing things (on accident) leads to very cryptic problems#2022-03-0821:02hiredmanAnother option would be, given a state machine description S, generate a vector of numbers, and then in a map bind, from your start state, take the sorted possible transitions, mod the first integer and take the nth, then iterate that for each state and integer#2022-03-0821:05hiredmanWhich would shrink based on vectors and int generators, which might not be the most intelligent shrinking, but avoids recursive generation passing the state through#2022-03-0821:06dnolenhrm, the problem has more constraints then it may appear, so I don't see that approach as being less cumbersome?#2022-03-0821:06dnolenin truth what I'm interested in is simulation generation#2022-03-0821:06dnolenthat is there is some actor A, and then they do something#2022-03-0821:06dnolenwell then maybe there is corresponding action for actor B#2022-03-0821:06dnolenand then also a corresponding action for actor C#2022-03-0821:06hiredmanSure, and I guess you'd get shrinking, but the behavior would be very odd#2022-03-0821:08dnolenbecause frequencies
is dynamic - I'm also interested in using the state at each step to dynamically bind new likelihoods#2022-03-0821:08dnolenso the tools in test.check at the moment seem sufficient to accomplish what I want, without going elsewhere#2022-03-0821:09dnolenI think probably only the command choice function (which generates the liklihoods and the command arguments) will be of any real interest#2022-03-0821:09dnolenthe rest will probably be boring#2022-03-0821:10Alex Miller (Clojure team)sounds like you are really doing simulation testing. we did several projects like this at Cognitect with Simulant (and other Simulant like things). there are a couple other open source things we made for those#2022-03-0821:10Alex Miller (Clojure team)https://github.com/candera/causatum#2022-03-0821:12dnolenbut we are not actually testing a whole system - we don't care about time#2022-03-0821:12dnolenand we're going to drive anything that would be external#2022-03-0821:12dnolenwe don't even care about concurrency#2022-03-0821:12hiredmanstate is time 🙂#2022-03-0821:13dnolenwe really don't care about time#2022-03-0821:13dnolenwe have a purely functional system - for what we want to test - no externalities are of interest#2022-03-0821:14dnolenvalue to value only - including all storage of interest (Datomic and custom data structures for cryptographic purposes)#2022-03-0821:14dnolenthus test.check is attractive#2022-03-0821:14hiredmansure, but if you have a succession of states, that is a ticking clock (how things like event clocks work)#2022-03-0821:14dnolenbut it's not of any interest at all#2022-03-0821:15dnolenin fact what we are interested is only the cryptographic results at the end of sequence#2022-03-0821:15dnolenthey must hold - nothing else really matters#2022-03-0821:21dnolenthe key is actually we are in fact looking for the shortest sequences that would break some invariant - I don't think the simulation stuff can do that @alexmiller ?#2022-03-0821:25Alex Miller (Clojure team)no, that's different use case#2022-03-0901:16hiredmanCame across this http://blog.guillermowinkler.com/blog/2015/04/12/verifying-state-machine-behavior-using-test-dot-check/ which is basically a retread (pretread?) of the discussion but includes some code and custom shrinking stuff#2022-03-0917:13dnolen@hiredman that's a nice write up!#2022-03-0917:13dnolenthanks#2022-03-2313:21jjttjjI want to make a generator for a string of an exact length where only certain letters are allowed. Is there an obvious better way than this that I'm missing:
(gen/generate
(gen/fmap (fn [x] (apply str x))
(apply gen/tuple
(repeat 10
(gen/elements
(mapcat (juxt identity str/upper-case) "abcdef"))))))
#2022-03-2313:32gfrederickstest.chuck can make a generator from a regex#2022-05-1106:17Oliver GeorgeHello. I'm trying to get this going: http://blog.guillermowinkler.com/blog/2015/04/12/verifying-state-machine-behavior-using-test-dot-check/
There's a repo which works
https://github.com/guilespi/fsm-test-check/
But when I update the deps (clojure 1.6.0-> 1.11.1 & test check 0.7.0->1.1.1) I get this error:
class clojure.lang.PersistentVector cannot be cast to class clojure.test.check.rose_tree.RoseTree (clojure.lang.PersistentVector is in unnamed module of loader 'app'; clojure.test.check.rose_tree.RoseTree is in unnamed module of loader clojure.lang.DynamicClassLoader @7d5508e0)#2022-05-1106:27hiredmanhttps://github.com/clojure/test.check/commit/d2378b37f1185cc20efc2dd364042521b9abc30e#2022-05-1107:01Oliver GeorgeThanks for the link. I think the implication is the code is old and the api has changed. I'll look more closely.#2022-05-1108:49Oliver GeorgeGot it. Thanks.#2022-08-1217:44dnolencircling way back - we sorted out how to do stateful property based testing a way that suited our needs#2022-08-1217:44dnolenhttps://t.co/aIZX1W6ZLH#2022-08-1310:22teodorlureally exciting talk. I love how you connect data orientation, immutability and toy implementations. Thank you!#2022-08-1217:45dnolengetting a stateful generator w/ shrinking wasn’t so hard in the end after assessing what had been done before - we didn’t like the various non-symbolic things out there so we just wrote the ~120 lines ourselves#2022-08-1217:46dnolenone interesting thing was that it became obvious that search is a big issue - and lot of the literature is about pretty fancy stuff#2022-08-1217:46dnolenone trick we employed that I think brings a lot of leverage is just bootstrap the initial world state so you don’t flounder in uninteresting spaces#2022-08-1217:47dnolenmaybe already covered by trivial unit tests (which never go very deep)#2022-08-2903:41Hugh PowellIs there any way to limit the number of shrinkages a failed test will do? On a fairly regular basis my REPL appears to hang, but I'm pretty sure it's just taking a while to shrink. I'd rather get a quicker response with a less than perfect shrink.#2022-09-0121:39colinkahnI don't think anything built in. I've made a "liveliness" reporter that will log the trial or shrink step the test is on so I get feedback. Also, judicious use of no-shrink around things can help.#2022-09-0423:18Hugh PowellAha, I didn't know about no-shrink
, looks that should work for me. Thanks 🙇#2022-09-0121:12Huahaiwhat’s the syntax for generating a bigint without long in for-all
?#2022-09-0214:55lread@huahaiy are you using size-bounded-bigint
? I find it a bit oddly named in that it does not always return BigInteger
s. I’m not sure about for-all
but you can use fmap
to force all to BigInteger
like so:
(gen/sample (gen/fmap bigint gen/size-bounded-bigint))
;; => (0N -41N -63N -58752N -421N -2818692N -6449877661N -111911828N 2341N 4730362N)
Does that help at all?#2022-09-0219:46HuahaiThanks. Yeah, it sometimes return longs. I just used such-that
to fill them out.#2022-09-2311:10markgdawsonDoes anyone know a simple way to have test.check change one parameter more slowly than another? So for example, parameter B changes every test whilst parameter A changes only every 10 tests.#2022-09-2316:38colinkahnI don't know of any way to finely control generation per-test, but you could look at frequency
#2022-09-2316:46markgdawsonYeah, frequency isn't really what I'm after here. What I'd really like yo do is to spin up a system with generated config and then fire a request against the system. But the system is expensive to spin up, and I'm happy with a small number of instances of it. So I want to bootstrap a single system, fire multiple samplings of the other params against it then move on to another generated system.#2022-09-2317:12colinkahnIn that case I would write a test where your generator generators a collection that you can apply in loop over a single bootstrapped instance. I would make sure that you wrote it so it short circuits on the first failure. Shrinking might take a bit longer, but test check does a good job shrinking collections so you'll most likely end up just a vector of a single item.#2022-09-2317:14colinkahnThere's also a trick to writing a collection generator that targets a certain size but shrinks to zero, I'll see if I can find it#2022-09-2317:15colinkahnhttps://clojurians.slack.com/archives/C0JKW8K62/p1601225845006100#2022-09-2317:23markgdawsonAh, i see. I was worried this approach wouldn't point me at the exact failure scenario. But you're saying I could achieve that by having the collection of failing tests shrink to the failing one. Smart idea, I'll give it a go!!#2022-09-2317:45colinkahnYep. That generator function above will let you tune how many to start the collection with (it won't always be exact, so you'll have to play around to see what the resulting item counts are), while still shrinking towards zero.#2022-09-3012:36arohnerIs there a way to get good logging about which property failed in a test? I have 4 conditions that all need to be true, so it would be nice to see after shrinking: “on the smallest failure, check 2 returned false”. I could println
, but then it logs before shrinking#2022-09-3013:01colinkahnYou can use https://github.com/gfredericks/test.chuck which supports is
assertions. That will report which ones failed.#2022-09-3013:02arohneroh very cool, thank you. I’m already using chuck, but didn’t know about that feature#2022-10-0314:25Brian BeckmanSOLVED
Hello, test.checkers. I am trying to understand the docs & source for test.check.generators.size-bounded.bigint
. It, and several of its sibling generators, mention a size
parameter, but I don’t see where it’s defined, how I can inspect its value, how I can input new values for it, etc. Example from the source:
#?(:clj
(def ^{:added "0.10.0"} size-bounded-bigint
;; 2^(6*size) was chosen so that with size=200 the generator could
;; generate values larger than Double/MAX_VALUE
"Generates an integer (long or bigint) bounded exclusively by ±2^(6*size)."
(fmap (fn [[n negate? force-bigint?]]
(cond-> n
negate? -'
;; adds some exciting variety
force-bigint? bigint))
(tuple size-bounded-bignat
boolean
boolean))))
#2022-10-0314:27gfredericks@bc.beckman https://github.com/clojure/test.check/blob/master/doc/growth-and-shrinking.md#2022-10-2119:41Brian BeckmanSELF-SOLVED
Ha! My question had its own answer in it 🙂 It may not be stylish, but (tgen/fmap #(apply list %) (tgen/tuple …))
works.
Hello test.checkers: I have a successful tgen/tuple
generator, but it produces vectors and i want to produce lists. The following does not work: (tgen/fmap list (tgen/tuple …))
. I’m not surprised because the appropriate postprocessing would be something like (apply list (tgen/generate (tgen/tuple …)))
, but I don’t want to exit the generators via something like tgen/generate
and then produce a list, I want a generator that produces a tuple-like list. I’d be grateful for advice!#2022-10-2119:44Alex Miller (Clojure team)does list*
work?#2022-10-2119:45hiredmanOr just comp apply list#2022-10-2119:46Alex Miller (Clojure team)there's not many good times to use list*, but this is maybe one of them#2022-10-2119:46Brian Beckmanyou guys are fast :)#2022-10-2119:46hiredmanlist* creates seqs 🤷#2022-10-2119:48Brian Beckmanmy installation can’t find list* in either test.check.generators or in spec.gen.alpha#2022-10-2119:48Brian Beckmanbut #(apply list #) works; (comp apply list) doesn’t quite because it only passes one arg to apply#2022-10-2119:48hiredmanclojure.core/list*#2022-10-2119:50hiredmanlist* is sort of comp apply cons if cons did that#2022-10-2119:55Brian Beckmanlist* seems stylish to me because it’s shorter than #(apply list %). The resulting seq is ok for my needs.#2022-10-2817:09Jared LangsonIf you're checking multiple related properties in one test, is better to use one =
or multiple =
? Does which method you use affect shrinkage? tree/node-healthy?
returns a boolean
(defspec add-remove-node-maintains-health
(prop/for-all [k gen/small-integer
size gen/nat]
;; +1 to prevent removing nodes from empty trees
(let [tree (make-integer-tree (+ 1 size))]
(=
(tree/node-healthy? (tree/node-add tree k))
(tree/node-healthy? (tree/node-remove-least tree))
(tree/node-healthy? (tree/node-remove-greatest tree))
(tree/node-healthy? (->> tree tree/node-random node/-k (tree/node-remove tree)))))))
or
(defspec add-remove-node-maintains-health
(prop/for-all [k gen/small-integer
size gen/nat]
;; +1 to prevent removing nodes from empty trees
(let [tree (make-integer-tree (+ 1 size))]
(= (tree/node-healthy? (tree/node-add tree k)))
(= (tree/node-healthy? (tree/node-remove-least tree)))
(= (tree/node-healthy? (tree/node-remove-greatest tree)))
(= (tree/node-healthy? (->> tree tree/node-random node/-k (tree/node-remove tree)))))))
or
(defspec add-remove-node-maintains-health
(prop/for-all [k gen/small-integer
size gen/nat]
;; +1 to prevent removing nodes from empty trees
(let [tree (make-integer-tree (+ 1 size))]
(= (and
(tree/node-healthy? (tree/node-add tree k))
(tree/node-healthy? (tree/node-remove-least tree))
(tree/node-healthy? (tree/node-remove-greatest tree))
(tree/node-healthy? (->> tree tree/node-random node/-k (tree/node-remove tree))))))))
#2022-10-2817:23hiredmandoesn't matter#2022-10-2817:24Jared LangsonStylistically, is one preferred?#2022-10-2817:24hiredmanthe body of your property is basically just a predicate#2022-10-2817:25hiredmanand (= x) is always true#2022-10-2817:25dpsutton(= false)
is true. And i’d worry if all of your predicates uniformly returned false and you misinterpreted this#2022-10-2817:25hiredman= doesn't do anything special#2022-10-2817:26dpsutton(for the (= (pred) (pred) (pred))
#2022-10-2817:26hiredmanso basically all 3 of your examples are bogus#2022-10-2817:27hiredmanthe first sort of works, but will also report a success if all your node-healthy? calls return false#2022-10-2817:27hiredmanthe second always is a success#2022-10-2817:27hiredmanand so is the third#2022-10-2817:29hiredmanif you remove the call to = the third is likely what you want#2022-10-2817:30Jared Langsonregarding the first, if I add false
or (> size 10)
inside the (=
block it fails So why would it pass when it should fail, if node-healthy?
is false?#2022-10-2817:31hiredmanbecause (= false false false)
is true#2022-10-2817:33Jared Langsonappreciate it. Thanks 🙂#2022-11-0322:52Jared LangsonIs there a way do this, but in a way that has shrinkage? I want to pull a random character from a random string and then do stuff. The random string is made from a generator. Is there a way to move the random character into the prop/for-all?
(defspec split-functionality-test
(prop/for-all [s (gen/not-empty gen/string-alphanumeric)]
(let [split-on (rand-nth s) ;can this be done in the for-all binding?
pattern (re-pattern (str split-on))]
(= (cp-str/split s pattern) (str/split s pattern)))))
#2022-11-0322:54hiredmanit is complicated#2022-11-0322:54hiredmanyou can, but the way to do it may make shrinking worse#2022-11-0322:57hiredmanI would try something like
(defspec split-functionality-test
(prop/for-all [i gen/nat
s (gen/not-empty gen/string-alphanumeric)]
(let [split-on (nth s (mod i (count s)))
pattern (re-pattern (str split-on))]
(= (cp-str/split s pattern) (str/split s pattern)))))
#2022-11-0322:58Alex Miller (Clojure team)sometimes it's helpful to build up the things you need rather than draw from one random thing#2022-11-0322:58Alex Miller (Clojure team)yeah, that's a decent way#2022-11-0322:58hiredmanpotentially moving the order of i and s if it isn't shrinking well (I forget which way test.check throws away shrinking when composing with bind)#2022-11-0323:00Jared LangsonThe other problem with my way is that because split-on is inside let
, I don't know the value of split-on when the test fails. So even if the shrinkage is worse, moving split-on to the prop/for-all could be a good tradeoff#2022-11-0323:02hiredmanin theory the ideal would be something like
(defspec split-functionality-test
(prop/for-all [s (gen/not-empty gen/string-alphanumeric)
i (gen/choose 0 (count s))]
(let [split-on (nth s (mod i (count s)))
pattern (re-pattern (str split-on))]
(= (cp-str/split s pattern) (str/split s pattern)))))
#2022-11-0323:03hiredmanso you only generate numbers from 0 to size of string, and those generated numbers also predictably shrink (throwing the mod in there can make things weird when shrinking)#2022-11-0323:04hiredmanbut this is sort of like (gen/bind ..gen for s... (fn [s] .. gen for i ...))
#2022-11-0323:05hiredmanand when you compose things with bind like that in test.check(and anything modeled after the original quickcheck) you lose shrinking information for one or the other, and I forget which choice test.check makes#2022-11-0323:06Jared LangsonWhen I ran your second version it complained about i (gen/choose 0 (count s))]
. Said "Unable to resolve symbol: s in this context"#2022-11-0323:08hiredmanoh, good, I guess test.check's for-all is optimized to get around the issue I mentioned, but as a result the generators cannot refer to each other#2022-11-0323:16hiredmannot optimized so the issue isn't there, but optimized to try and avoid it#2022-11-0323:09hiredmanfor-all* is the one that would let you I think#2022-11-0323:09hiredmannope#2022-11-0323:09hiredmanhuh, for some reason I thought for-all turned into a bind#2022-11-0323:10hiredmanI guess the second isn't directly expressible using for-all#2022-11-0323:14hiredmanI was looking at an old version of the code from before
> When there are multiple binding pairs, the earlier pairs are not
> visible to the later pairs.
was added to the docstring for for-all#2022-11-0323:33hiredmanhttp://clojure.test.check.ge/let is what I am thinking of that allows for the dependencies between generators#2022-11-0323:42Jared LangsonAre there pitfalls/gotchas with using gen/let?#2022-11-0323:44hiredmanthe above mentioned lose of shrinkage, which is https://clojure.atlassian.net/browse/TCHECK-112#2022-11-1403:14skylizeSeems like (despite the plural name) gen/elements
only returns 1 element per gen? Is that right? How do I gen a random quantity of elements?
(def map-and-1-key-from
(gen/let [m (gen/not-empty
(gen/map gen/keyword gen/any-equatable))
;; Surprisingly works as name above describes.
;; How would I make `map-and-some-keys-from` instead?
k (gen/elements (keys m))]
[m k]))
(def my-gen (map-and-1-key-from gen/keyword
#2022-11-1403:22hiredmanvector of elements#2022-12-1314:11skylizeI don't understand why this causes "Couldn't generate enough distinct elements!"
(defn foo [gen]
(gen/vector-distinct gen {:num-elements 10}))
(defspec foo-test
(prop/for-all [x (gen/let [n (gen/resize 10 gen/nat)]
(foo (gen/vector gen/any n)))]
true))
(foo-test)
Works fine if I replace the gen/let
wrapper with an inline constant.
(defspec foo-test
(prop/for-all (fgen/foo (gen/vector gen/any 5))
true))
Is there a better way to do this? or some way to work around the problem?#2022-12-1314:12gfredericksDoes it happen when n is zero?#2022-12-1314:13skylizeyes#2022-12-1314:36skylizeThis was stripped way down to highlight the issue.
Here is real code.
unary
, shown for easier explanation of the concept, takes gens for x
and y
, and generates a function f
mapping x
-> y
, (returning all 3 in a map).
n-aries
is the same basic idea. But the number of xs
is variable (i.e. generating n-arity of function instead of unary) and the number of xs
-> y
mappings is also variable, so the generated function can return different y
s depending on specific inputs.
The arity (or count of xs
) should be determined by xs-gen
(which is the problematic n
generated in the spec). The count of xs
-> y
mappings is determined by count-gen
. An empty xs
should be a valid result from xs-gen
. But xss
needs to be a list of unique xs
.
;; fgen.cljc
; unary
(defn- x-y->map [X Y]
{:x X
:y Y
:f (fn [x]
(if (= X x)
Y
(throw (ex-info "generated function called with incorrect arguments"
{:expected X
:actual x}))))})
(defn unary [x-gen y-gen]
(gen/let [x x-gen
y y-gen]
(x-y->map x y))
; n-aries
(defn- xss-ys->map [xss ys]
(let [xss-ys (zipmap xss ys)]
{:xss-ys xss-ys
:f (fn [& xs]
(if (some #(= % xs) xss)
(xss-ys xs)
(throw (ex-info "generated function called with incorrect arguments"
{:expected {:one-seq-of xs}
:actual xs}))))}))
(defn n-aries [count-gen xs-gen y-gen]
(gen/let [n count-gen
xss (gen/vector-distinct xs-gen {:num-elements n})
ys (gen/vector y-gen n)]
(xss-ys->map xss ys)))
; fgen_test.cljc
(defspec n-aries
200
(prop/for-all [{f :f {xs :xs ys :ys} :xs-ys}
(gen/let [n (gen/resize 10 gen/nat)]
(fgen/n-aries
(gen/resize 10 gen/nat)
(gen/vector gen/any n)
gen/any))]
(or (util/empties? xs ys)
(= ys (map f xs)))))
#2022-12-1314:39gfredericksWhy is uniqueness important?#2022-12-1314:40skylizeBecause the goal is to generate a function f
, which has deterministic behavior y
based on its input value(s) xs
.#2022-12-1314:41gfredericksThen why is generating a minimum number of things important?#2022-12-1314:41skylizeAdditionally, I am representing that determinism internally as a map where xs
is a key on a map of xs
-> ys
.#2022-12-1314:41skylizeWho said anything about a minimum number of things?#2022-12-1314:42gfredericksYour gen-vector-distinct call did#2022-12-1314:43skylizeThe gen-distinct
call asks for a specific number of things, where that number is generated by a user-provided generator. The user could choose to provide (gen/return 0)
or gen/nat
. That is not for me to decide.#2022-12-1314:44gfredericksWhat does n represent there, in the context of generating functions?#2022-12-1314:48skylizen
in n-aries
(not causing problems and replaced by 10
in the foo
example) is the number of mappings from xs
-> ys
.
n
in the spec (which is causing problems) represents the arity of the target generated function. i.e (defn foo [x y z])
is (= n 3)
, and (defn foo [])
is (= n 0)
. Again this will ultimately be provided by the user, when they define an xs-gen
to pass into n-aries
.#2022-12-1314:51gfredericksDo you at least agree that the error in your simplified example makes sense?#2022-12-1314:53skylizeYes. I see it now. In the case my test gens (= n 0)
, foo
cannot generate distinct data.#2022-12-1314:55skylizeBut that still leaves me at a loss how to put n-aries
through its paces, when (= n 0)
for should be a valid possibility. 😞#2022-12-1315:01skylizeHmm. I think with that in mind, the issue is not the test. Somehow n-aries
needs to be smarter about comparing the result of count-gen
with the results of xs-gen
to avoid nonsensical attempts at uniqueness.#2022-12-1315:04gfredericksA 0-ary pure function can only have one input/output pair#2022-12-1315:04gfredericksIt's effectively a constant#2022-12-1315:11skylizeYes. I don't know what that though.
The fact a 0-arity pure function is a constant does not obviate the need for someone testing a higher-order function check how it handles being given a 0-arity function, among others.#2022-12-1315:34gfredericksI haven't internalized the details of the situation, but if you're getting a "can't generate enough distinct elements" error because of a trivial generator, then you should probably either question the need for distinctness or minimum elements, or else consider that your api allows users to ask for impossible things#2022-12-1404:44skylizeThe problem was just the location of let
. It should be wrapping the generator passed to foo
to keep generating new n
values, rather than binding a single n
ahead of calling foo
.
(defspec foo-test
(prop/for-all [x (foo (gen/let [n (gen/resize 10 gen/nat)]
(gen/vector gen/any n)))]
true))
... instead of ...
(defspec foo-test
(prop/for-all [x (gen/let [n (gen/resize 10 gen/nat)]
(foo (gen/vector gen/any n)))]
true))
#2022-12-1314:13gfredericksIf I'm reading correctly, in that scenario you're asking it to generate ten distinct empty vectors#2022-12-1314:14gfredericksI can't tell what you're actually trying to do, so it's not clear what would count as a fix#2022-12-2003:24skylizeWould happily take feedback, especially ideas for getting out of the way of shrinkage.
https://clojurians.slack.com/archives/C06MAR553/p1671506574208579#2023-09-0217:12Ben SlessWhat's the best way to generate any
such that bigints are bounded?#2023-11-2419:53Caio CascaesI've once seen this kind of assertion:
(is (= (-> response :body (json/read-str :key-fn keyword))
[{:paper-quantity-standard 512
:nickname "Galactic printer"
:brand "EPSON"
:prints-color? true
:papers-quantity-photo 35
:prints-photo? true
:public-id logic.common/valid-uuid-str?
:model "GalaxyPrint 3000"}]))
That we could place a pure function to "nest" testing, such this part :public-id logic.common/valid-uuid-str?
How can I achieve this? Which lib? Or whatever?
(I'm not sure if it's (is (=...
, maybe something with ...equals...)#2023-11-2419:58hiredmantest-check doesn't really use is#2023-11-2419:58hiredmanis
is part of clojure.test#2023-11-2419:58hiredmantest-check is a generative testing library#2023-11-2421:07Caio CascaesSorry, maybe wrong place to ask?#2023-11-2421:10Caio CascaesI'm moving this question to #C03S1KBA2 channel, soon I'm removing this from here.